Amazon facial recognition mistakenly confused 28 Congressmen with known criminals
The ACLU says it's evidence that Congress should step in. Amazon says the ACLU didn't test properly.
Sean HollisterSenior Editor / Reviews
When his parents denied him a Super NES, he got mad. When they traded a prize Sega Genesis for a 2400 baud modem, he got even. Years of Internet shareware, eBay'd possessions and video game testing jobs after that, he joined Engadget. He helped found The Verge, and later served as Gizmodo's reviews editor. When he's not madly testing laptops, apps, virtual reality experiences, and whatever new gadget will supposedly change the world, he likes to kick back with some games, a good Nerf blaster, and a bottle of Tejava.
That's a lot of Congresspeople who may soon have some very valid questions about facial recognition and its potential to be abused -- particularly since Amazon thinks the ACLU didn't use it properly to begin with!
It turns out that the ACLU got its mugshot matches by using the Rekognition software at its default 80-percent confidence threshold setting, rather than the 95-percent plus confidence level that Amazon recommends for law enforcement agencies.
"While 80 percent confidence is an acceptable threshold for photos of hot dogs, chairs, animals, or other social media use cases, it wouldn't be appropriate for identifying individuals with a reasonable level of certainty. When using facial recognition for law enforcement activities, we guide customers to set a threshold of at least 95 percent or higher," an Amazon spokesperson told CNET by email.
But an ACLU lawyer tells CNET that Amazon doesn't necessarily steer law enforcement agencies toward that higher threshold -- if a police department uses the software, it'll be set to the same 80-percent threshold by default and won't ask them to change it even if they intend to use it to identify criminals. "Amazon makes no effort to ask users what they are using Rekognition for," says ACLU attorney Jacob Snow.
A source close to the matter tells CNET that when Amazon works with law enforcment agencies directly, like the Orlando Police Department, it teaches them how to reduce false positives and avoid human bias. But there's nothing to necessarily keep other agencies from simply using the tool the same way the ACLU did, instead of developing a relationship with Amazon.
But the ACLU worries that Amazon's false positives might bias a police officer or government agent to search, question or potentially draw a weapon when they shouldn't -- and we've all seen how those encounters can turn deadly. And the ACLU notes that Amazon's tech seems to have over-represented people of color.
The ACLU also provided CNET this statement:
Amazon seems to have missed, or refuses to acknowledge, the broader point: facial recognition technology in the hands of government is primed for abuse and raises significant civil rights concerns. It could allow – and in some cases has already enabled – police to determine who attends protests, ICE to continuously monitor immigrants, and cities to routinely track their own residents, whether they have reason to suspect criminal activity or not. Changing the threshold from 80 to 95 percent doesn't change that. In fact, it could exacerbate it.
Should Congress regulate facial recognition? Microsoft thinks so, and now 28 members of Congress have some very personal food for thought -- 95-percent confidence threshold or no.
In the hours since the ACLU's test was brought to light, five Congressmen have sent letters to Amazon CEO Jeff Bezos asking for answers and an immediate meeting. You can read the letters here.
Update, 12:44 a.m. PT: Added that five Congressmen have sent Amazon CEO Jeff Bezos letters with questions about the facial recognition tech. Also added that Amazon's tech appears to have over-represented people of color, according to the ACLU.
Correction, July 27: We mistakenly wrote that the ACLU used a database of 2,500 mugshots. The number was actually 25,000.