Facebook's new facial recognition efforts help blind users know exactly who's in photos

Facebook is working to make its platform even more accessible for blind users and people with low vision.

In a series of updates announced Tuesday, the company revealed that it will begin using its already-existing face recognition technology to identify people in photographs for Facebook users with screen readers.

Facebook's director of applied machine learning, Joaquin Candela, wrote in a blog post that the new feature will use face recognition alongside the platform's automatic alt-text tool, which launched in 2016.

Using artificial intelligence and the same face recognition technology that makes suggestions during the tagging process, the alt-text tool describes scenery, objects, animals, and people in photographs to those with vision loss. Prior to 2016, users could only hover their cursor over an image and simply hear the term "photo." However, the alt-text tool was limited to sharing the number of people present in the photo, rather than their identities.

Now, in addition to reading the photo's sharing details, caption, and a visual description, Facebook is able to analyze the pixels in photos on the News Feeds to let blind users know which friends are in each image — regardless of whether people are tagged.

"It was one of the most popular requests we had from people when we first showed them automatic alt-text a year and a half ago," said Matt King, accessibility specialist in UI engineering at Facebook and the man behind the technology.

When asked how this update accounts for any potential inaccuracies in face recognition, a Facebook spokesperson said the system performs a check to ensure it's not tagging someone who simply resembles another person.

"We compare the face that is being used for authentication against your profile picture, and other photos and videos that you're tagged in on Facebook," the spokesperson said. "We apply a certainty threshold to check if it's really you, not someone who looks like you."

A life-changing impact

For King, who is Facebook's first blind engineer, these advancements have drastically improved the way he communicates with others online.

"I signed up for Facebook back in 2009, before there was an accessibility team," he said. "At that time it was an extremely challenging experience because the basic foundation of accessibility wasn't present."

King recalled how difficult it was for him to use the site, explaining he once spent an entire Saturday morning trying to locate his list of friends to figure out if he needed to send a particular individual a friend request.

"That was a multi-hour process," he said. "Now? That task is like it is for everybody else — it’s a 30-second task."

Janni Lehrer-Stein, a disability rights advocate who is legally blind, recently went down to Facebook's offices to test the feature herself, and feels the efforts are a tremendous step toward inclusion.

"This is so exciting for people who are blind or low vision and it’s really hard to describe," said Lehrer-Stein, who was appointed by President Barack Obama to the National Council on Disability in 2011. "With this new innovation you know who the people in photographs are and you can put those people in context and then really be able to understand and participate."

Like King, Lehrer-Stein said that prior to Facebook's accessibility-focused technology, she felt isolated when using the platform and couldn't properly participate in online conversations.

"For me, it was virtually impossible to understand conversations that went around these visual images, whether they were of my family or friends or colleagues," she said. "It's important for us to understand that inclusion goes along with tolerance and compassion, and that this technology is going to open the door to millions and millions of people to be able to make contributions. That will make the world a much better place."

What's on the horizon?

King said that in the future, Facebook will work toward forming actual sentences in photo descriptions, rather than simply listing objects and people detected. To achieve this, he plans to continue developing recognizable concepts like actions, activities, people, objects, and scenes.

Another goal is to make the photographs interactive, so users can ask the system any lingering questions.

"If a photo tells me my friend Jason is in the photo, what you might be able to ask is, 'What's Jason’s hair color?' or, 'What shirt is he wearing today?'"

Finally, King hopes the technology will one day be able to clearly communicate the message of memes, or photos that have text on top of them.

"The ultimate goal is to give people who are visually impaired or blind equal access to visual information. That’s a long way off, but that’s the vision," he said.

Facebook tackles privacy issues

In addition to the increased accessibility, Facebook is also introducing new optional features related to safety and security. With the updates, Facebook says users will be given additional control over maintaining their identities, and can even turn off facial recognition.

If users choose to enable the features using the on/off control, they can be notified when photos of themselves are uploaded by someone in their "audience."

"You're in control of your image on Facebook and can make choices such as whether to tag yourself, leave yourself untagged, or reach out to the person who posted the photo if you have concerns about it," Candela wrote in Tuesday's blog post.

And in the future, facial recognition will be used to inform people when someone else uploads a profile photo of them, in an effort to avoid impersonations.

Original image has been replaced. Credit: Mashable

Original image has been replaced. Credit: Mashable

The features will roll out to most Facebook users, except those where Facebook doesn't utilize face recognition technology, such as Canada and the EU.

UPDATE: Dec. 19, 2017, 1:14 p.m. EST This post has been updated to include information about Facebook's process of confirming facial recognition.