Facial recognition firms are scrambling to see around face masks
Because of face coverings prompted by the coronavirus pandemic, companies are trying to ID people based on just their eyes and cheekbones.
In the age of the coronavirus, face masks have become a part of normal life. They're a safety requirement in many places, and for some people, a fashion statement. But for facial recognition technology, they pose a major challenge.
The US Centers for Disease Control and Prevention has recommended wearing face coverings to help fight the spread of COVID-19, the coronavirus-caused sickness that's killed more than 302,000 people around the world. And governments in more than half the US states are making masks mandatory in various public settings.
But don a mask and stare at your iPhone or Android device to unlock it, and you quickly see the problem for facial recognition.
Before the novel coronavirus hit, facial recognition providers were expecting to install their technology everywhere: in airports, casinos, restaurants and schools. Face masks threaten to change all that, but the industry is looking at the situation more as a speed bump than a roadblock.
Coronavirus updates
Some companies assert that their technology isn't affected by masks, and that artificial intelligence can still detect and identify people with a high accuracy rate, even when half the face is covered.
A public beta program for Apple's latest iOS release showed that the tech giant is updating its Face ID to account for people wearing masks. Google didn't respond to a request for comment on whether it's doing the same with Android.
Experts on facial recognition are skeptical about claims that the technology isn't fazed by masks. After all, even without masks, facial recognition can stumble -- studies have found that the majority of facial recognition algorithms had a higher rate of false positives for people of color by a factor of "10 to 100 times."
And because of the pandemic, these algorithms can't be properly tested with face masks by the US' National Institute of Standards and Technology, or NIST, which many consider the leading authority on facial recognition accuracy rates.
Still, facial recognition is being proposed as a solution for COVID-19, without any proof that the surveillance measure has any benefits, or even works properly with masks on.
"These workarounds are part of a larger effort to make an ever-expanding surveillance infrastructure a fundamental component of COVID-19 response governance," Evan Selinger, a professor of philosophy at the Rochester Institute of Technology, said in a statement.
Identifiable?
Masks have long been a method for avoiding facial recognition. Protesters in Hong Kong relied on them to beat the government's facial recognition, prompting a mask ban there.
"The greatest amount of biometric data that uniquely sets us apart resides in the central portion of the face, just above the brow line all the way down to the chin," said Eric Hess, senior director of product management for face recognition at facial recognition company SAFR. "When we put on face masks, we are blocking access to a significant amount of data points that help us differentiate one person from another."
SAFR says its facial recognition algorithm can identify people even when they wear masks.
With face masks now common, several facial recognition companies have said their technology can still identify people.
UK-based Facewatch said it's releasing an algorithm that can handle detection and identification based on just a person's eye and eyebrow region. The company is proposing its technology for retail stores and says the development will extend beyond masks to other coverings, such as the religious veil called a niqab that's worn by some Muslim women.
Facewatch had already been working on identifying people who are wearing hats and glasses, said company spokesman Stuart Greenfield. Its customers, mostly retail stores looking to keep shoplifters on a watchlist, didn't consider mask detection much of a concern, until the pandemic began.
"All we need is the government to insist on [face masks], and the whole sector will have to react very rapidly," Greenfield said. He added that Facewatch's new algorithm will be able to ID people because their eyes and eyebrows are fixed points on the face and don't change over time.
Still, Facewatch expects some complications because of face masks. Its algorithm typically identifies a person in half a second, and Greenfield said it could take longer because of the masks. But the company said it's doing everything it can to make the new algorithm effective.
"Everyone's working right now to ensure that we're fit for the market," Greenfield said. "Our future depends on having a product that works accurately."
SAFR, which promotes its technology for use in schools, also says its tools can handle face masks.
"Our algorithms are now being trained with images of people wearing face masks," Hess said. Until recently, the masks hadn't been very present in society, "so they were not really added as a training dynamic before," he said.
To train its algorithm, SAFR is relying on a hoard of photos of people wearing face masks, some shots that it creates on its own, and others its staff members have provided at the company's request. Hess said the company is training its algorithm on a diverse set of images, to account for differences in gender, race and age.
The accuracy rate of the tools is 93.5% when people are wearing masks, Hess said, but only under ideal conditions, such as when the subjects are depicted in a high-quality photo with proper lighting.
Skepticism
It's unclear how accurate these statements about facial recognition bypassing masks actually are. And it'll be a while until we get some definitive answers.
On May 1, NIST announced that it would be running tests to identify how accurate facial recognition is with people wearing face masks, by digitally adding masks to its existing database of photos. But testing is closed because of the pandemic, and there's no indication of when it'll resume.
Facewatch and SAFR said they intend to submit their respective algorithms to NIST when possible. Without the test, there's no way to effectively compare the accuracy to other facial recognition companies.
For now, people will have to take a company's word for it that its technology actually works despite face masks. Facial recognition specialists are skeptical.
Kate Rose is a digital security expert and the designer behind Adversarial Fashion. She makes clothes to trick surveillance tech, like dresses for fooling license plate readers and masks for thwarting facial recognition.
Rose tests the masks' effectiveness using open-source facial recognition tools at home, and she studies how surveillance technology recognizes people.
Facial recognition is designed to scan for and grab many data points on a person's face, such as how far apart the eyes are, and the structure of the nose and chin. For identification, the technology compares the face it's scanning with an image it already has in its database -- one that likely doesn't feature a face mask.
Rose doesn't doubt that it's possible for facial recognition providers to identify people from just their eyes and eyebrows, but she said this would possibly be ineffective in a real-world scenario.
"If you have perfect pictures of my eyes, I'm positive you can get them to match," Rose said. "But the real world offers this crazy variety of background, lighting -- and those things make it all really hard."
With the entire face, there's a greater number of distinguishing features for the AI to work from. When the features are reduced to just the eyes and eyebrows, a lot more similarities crop up that can trigger false positives.
The face masks would also play a role, said Liz O'Sullivan, co-founder of the AI monitoring company Arthur. The trained algorithm might be able to ID a person wearing a blue mask but could get tripped up by the same person wearing a red mask.
"With computer vision, so much depends on how it's being used," O'Sullivan said. "Most likely, they would need a data set that has the same person with and without masks, from different angles and lighting conditions. It might be possible to accomplish the same goal with just the masked and unmasked pairs, but the accuracy wouldn't be as high."
That's an issue SAFR has encountered in its testing, Hess said, describing how face masks can differ from country to country. The majority of masks used in Europe, for instance, are blue, he said, while in Japan, a few skin-toned masks have appeared in the company's data set. These deviations could confuse the system.
"There will be some masks that go undetected," Hess said.
Facial recognition has long had issues with accuracy rates, especially in regard to people of color and women. Adding face masks further complicates the task.
COVID-19 has hit minority groups especially hard, with "a disproportionate burden of illness and death" affecting their communities, according to the CDC. More than 80 percent of summonses handed out by the New York Police Department for social distancing violations from March 16 to May 5 were issued to people of color, according to the department.
Experts warn that flaws with facial recognition and masks are another problem minority groups may have because of the pandemic.
"The similarity of many different types of people is going to go up," Rose said. "We all like to think that we're very unique and distinctive, but odds are you can find many people in a data set with very similar eyebrows and eyes."
Beyond COVID
This new capability could have lingering effects long after the pandemic ends. Because of the public health crisis, companies are pushing for identification that can deal with masks. But that same capability could later be used by police to identify protesters wearing face coverings.
In January, members of the House Oversight committee warned that once facial recognition is perfected, it could have chilling effects on free speech and civil liberties. Police have already used facial recognition to monitor protests, and if people can be identified despite wearing masks, that creates a new level of privacy concerns for people exercising their First Amendment rights.
Both SAFR and Facewatch said that were it not for the pandemic, they wouldn't have been so quick to start work on dealing with face masks. But with how prevalent the masks have become, there's been a rise in demand from their customers.
"It is possible that you would have advancements that would not have been made if not for this," Rose said. "We should be aware that this may be a tide that raises all boats in terms of surveillance."