Amazon facial recognition mistakenly confused 28 Congressmen with known criminals
The ACLU says it's evidence that Congress should step in. Amazon says the ACLU didn't test properly.
Amazon is trying to sell its Rekognition facial recognition technology to law enforcment, but the American Civil Liberties Union doesn't think that's a very good idea. And today, the ACLU provided some seemingly compelling evidence -- by using Amazon's own tool to compare 25,000 criminal mugshots to members of Congress.
Sure enough, Amazon's tool thought 28 different members of Congress looked like people who've been arrested.
Here's the full list, according to the ACLU:
Senate
- Johnny Isakson (R-Georgia)
- Ed Markey (D-Massachusetts)
- Pat Roberts (R-Kansas)
House
- Sanford Bishop (D-Georgia)
- G. K. Butterfield (D-North Carolina)
- Lacy Clay (D-Missouri)
- Mark DeSaulnier (D-California)
- Adriano Espaillat (D-New York)
- Ruben Gallego (D-Arizona)
- Tom Garrett (R-Virginia)
- Greg Gianforte (R-Montana)
- Jimmy Gomez (D-California)
- Raúl Grijalva (D-Arizona)
- Luis Gutiérrez (D-Illinois)
- Steve Knight (R-California)
- Leonard Lance (R-New Jersey)
- John Lewis (D-Georgia)
- Frank LoBiondo (R-New Jersey)
- Dave Loebsack (D-Iowa)
- David McKinley (R-West Virginia)
- John Moolenaar (R-Michigan)
- Tom Reed (R-New York)
- Bobby Rush (D-Illinois)
- Norma Torres (D-California)
- Marc Veasey (D-Texas)
- Brad Wenstrup (R-Ohio)
- Steve Womack (R-Arkansas)
- Lee Zeldin (R-New York)
That's a lot of Congresspeople who may soon have some very valid questions about facial recognition and its potential to be abused -- particularly since Amazon thinks the ACLU didn't use it properly to begin with!
Did you see this? @amazon face surveillance technology FALSELY matched me w/ someone else’s mugshot. I’m outraged & worried by the impact this tool will have on #CommunitiesOfColor when put in the hands of law enforcement! @JeffBezos: We need to talk ASAP. https://t.co/xFOy8duef1
— Rep. Jimmy Gomez (@RepJimmyGomez) July 26, 2018
It turns out that the ACLU got its mugshot matches by using the Rekognition software at its default 80-percent confidence threshold setting, rather than the 95-percent plus confidence level that Amazon recommends for law enforcement agencies.
"While 80 percent confidence is an acceptable threshold for photos of hot dogs, chairs, animals, or other social media use cases, it wouldn't be appropriate for identifying individuals with a reasonable level of certainty. When using facial recognition for law enforcement activities, we guide customers to set a threshold of at least 95 percent or higher," an Amazon spokesperson told CNET by email.
But an ACLU lawyer tells CNET that Amazon doesn't necessarily steer law enforcement agencies toward that higher threshold -- if a police department uses the software, it'll be set to the same 80-percent threshold by default and won't ask them to change it even if they intend to use it to identify criminals. "Amazon makes no effort to ask users what they are using Rekognition for," says ACLU attorney Jacob Snow.
The ACLU says that even when it comes to facial recognition for security purposes, Amazon's website suggests that the 80-percent confidence threshold is sufficent.
A source close to the matter tells CNET that when Amazon works with law enforcment agencies directly, like the Orlando Police Department, it teaches them how to reduce false positives and avoid human bias. But there's nothing to necessarily keep other agencies from simply using the tool the same way the ACLU did, instead of developing a relationship with Amazon.
It's worth noting that false positives are (currently!) an accepted part of facial recognition technology. Nobody -- including the ACLU -- is saying police would arrest someone based on a false match alone. Facial recognition narrows down the list of suspects, and then humans take over. Recently, facial recognition helped ID the Russian assassins who poisoned a spy in the UK, as well as the Capital Gazette shooter.
And Amazon didn't actually create that many false positives even at the 80 percent confidence ratio, compared to, say, the UK Metropolitan Police's facial recognition tech.
But the ACLU worries that Amazon's false positives might bias a police officer or government agent to search, question or potentially draw a weapon when they shouldn't -- and we've all seen how those encounters can turn deadly. And the ACLU notes that Amazon's tech seems to have over-represented people of color.
The ACLU also provided CNET this statement:
Amazon seems to have missed, or refuses to acknowledge, the broader point: facial recognition technology in the hands of government is primed for abuse and raises significant civil rights concerns. It could allow – and in some cases has already enabled – police to determine who attends protests, ICE to continuously monitor immigrants, and cities to routinely track their own residents, whether they have reason to suspect criminal activity or not. Changing the threshold from 80 to 95 percent doesn't change that. In fact, it could exacerbate it.
Should Congress regulate facial recognition? Microsoft thinks so, and now 28 members of Congress have some very personal food for thought -- 95-percent confidence threshold or no.
In the hours since the ACLU's test was brought to light, five Congressmen have sent letters to Amazon CEO Jeff Bezos asking for answers and an immediate meeting. You can read the letters here.
Update, 12:44 a.m. PT: Added that five Congressmen have sent Amazon CEO Jeff Bezos letters with questions about the facial recognition tech. Also added that Amazon's tech appears to have over-represented people of color, according to the ACLU.
Correction, July 27: We mistakenly wrote that the ACLU used a database of 2,500 mugshots. The number was actually 25,000.