The tech giant has built technology to fight child exploitation and abuse, but advocates worry it could erode people's privacy. Here's why.
Apple has long presented itself as one of the only tech companies that truly cares about user privacy or security. But a new technology designed to help an iPhone, iPad or Mac computer detect child exploitation images stored on those devices has ignited a fierce debate about the truth behind Apple's promises.
On Aug. 5, Apple announced a new feature being built into the upcoming iOS 15, iPad OS 15, WatchOS 8 and MacOS Monterey software updates, designed to detect if people have child exploitation images or videos stored on their device. It'll do this by converting images into unique bits of code, known as hashes, based on what the images depict. The hashes are then checked against a database of known child exploitation content that's managed by the National Center for Missing and Exploited Children. If a certain number of matches are found, Apple is then alerted and may further investigate. Apple hasn't said when the software will be released, though on Sept. 3 it did announce a delay to make improvements and address privacy concerns.
Apple said it developed this system to protect people's privacy, performing scans on the phone and only raising alarms if a certain number of matches are found. But privacy experts, who agree that fighting child exploitation is a good thing, worry that Apple's moves open the door to wider uses that could, for example, put political dissidents and other innocent people in harm's way.
"Even if you believe Apple won't allow these tools to be misused there's still a lot to be concerned about," tweeted Matthew Green, a professor at Johns Hopkins University, who's worked on cryptographic technologies.
Nearly 100 policy and rights groups have since come out against the move, signing on to an open letter to Cook saying the benefits of Apple's new technology don't outweigh the potential costs.
"Though these capabilities are intended to protect children and to reduce the spread of child sexual abuse material (CSAM), we are concerned that they will be used to censor protected speech, threaten the privacy and security of people around the world, and have disastrous consequences for many children," the group said in the letter, whose signatories include the Center for Democracy & Technology, the American Civil Liberties Union, the Electronic Frontier Foundation and Privacy International.
Even the people who helped develop scanning technology similar to what Apple's using say it's too easily misused.
"We're not concerned because we misunderstand how Apple's system works. The problem is, we understand exactly how it works," Princeton assistant professor Jonathan Mayer and graduate researcher Anunay Kulshrestha wrote in a Washington Post opinion piece. "Apple is making a bet that it can limit its system to certain content in certain countries, despite immense government pressures. We hope it succeeds in both protecting children and affirming incentives for broader adoption of encryption. But make no mistake that Apple is gambling with security, privacy and free speech worldwide."
Apple's new feature, and the concern that's sprung up around it, represent an important debate about the company's commitment to privacy. Apple has long promised that its devices and software are designed to protect users' privacy. The company even dramatized that with an ad it hung just outside the convention hall of the 2019 Consumer Electronics Show, which said, "What happens on your iPhone stays on your iPhone."
"We at Apple believe privacy is a fundamental human right," Apple CEO Tim Cook has often said.
Apple's scanning technology is part of a trio of new features the company is planning for this fall. Apple also is enabling its Siri voice assistant to offer links and resources to people it believes may be in a serious situation, such as a child in danger. Advocates had been asking for that type of feature for a while.
It's also adding a feature to its messages app to proactively protect children from explicit content, whether it's in a green-bubble SMS conversation or blue-bubble iMessage encrypted chat. This new capability is specifically designed for devices registered under a child's iCloud account and will warn if it detects an explicit image being sent or received. Like with Siri, the app will also offer links and resources if needed.
There's a lot of nuance involved here, which is part of why Apple took the unusual step of releasing research papers, frequently asked questions and other information ahead of the planned launch.
Here's everything you should know:
The tech giant said it's been trying for a while to find a way to help stop child exploitation. The National Center for Missing and Exploited Children received more than 65 million reports of material last year. Apple said that's way up from the 401 reports 20 years ago.
"We also know that the 65 million files that were reported is only a small fraction of what is in circulation," said Julie Cordua, head of Thorn, a nonprofit fighting child exploitation that supports Apple's efforts. She added that US law requires tech companies to report exploitative material if they find it, but it doesn't compel them to search for it.
Other companies do actively search for such photos and videos. Facebook, Microsoft, Twitter and Google (and its YouTube subsidiary) all use various technologies to scan their systems for any potentially illegal uploads.
Apple in the past has followed a similar approach, telling 9to5Mac that it's scanned iCloud email and some other files since at least 2019.
What makes Apple's new system unique is that it's designed to scan our devices, rather than the information stored on the company's servers.
The hash scanning system will be applied only to photos stored in iCloud Photo Library, which is a photo syncing system built into Apple devices. It won't hash images and videos stored in the photos app of a phone, tablet or computer that isn't using iCloud Photo Library. So, in a way, people can opt out if they choose not to use Apple's iCloud photo syncing services.
Read more: Apple software head says plan to scan iPhones for child abuse images is 'misunderstood'
The question isn't whether Apple should do what it can to fight child exploitation. It's whether the company should use this method.
The slippery slope concern privacy experts have raised is whether Apple's tools could be twisted into surveillance technology against dissidents. Imagine if the Chinese government were able to somehow secretly add data corresponding to the famously suppressed Tank Man photo from the 1989 pro-democracy protests in Tiananmen Square to Apple's child exploitation content system.
Apple said it designed features to keep that from happening. The system doesn't scan photos, for example -- it checks for matches between hash codes. The hash database is also stored on the phone, not a database sitting on the internet. Apple noted that because the scans happen on the device, security researchers can more easily audit the way it works.
"If you look at any other cloud service, they currently are scanning photos by looking at every single photo in the cloud and analyzing it; we wanted to be able to spot such photos in the cloud without looking at people's photos," said Apple's head of software engineering, Craig Federighi, in an Aug. 13 interview with The Wall Street Journal. "This isn't doing some analysis for, 'Did you have a picture of your child in the bathtub?' Or, for that matter, 'Did you have a picture of some pornography of any other sort?' This is literally only matching on the exact fingerprints of specific known child pornographic images."
The company also said there are "multiple levels of auditability." One way is that Apple plans to publish a hash, or a unique code identifiable, for its database online each time it's updated by the National Center for Missing and Exploited Children. Apple said the hash can only be generated with the help of at least two separate child safety organizations, and security experts will be able to identify any changes if they happen. Child safety organizations will also be able to audit Apple's systems, the company said.
We've all seen some version of it: The baby in the bathtub photo. My parents had some of me, I have some of my kids, and it was even a running gag on the 2017 Dreamworks animated comedy The Boss Baby.
Apple says those images shouldn't trip up its system. Because Apple's program converts our photos to these hash codes, and then checks them against a known database of child exploitation videos and photos, the company isn't actually scanning our stuff. The company said the likelihood of a false positive is less than one in 1 trillion per year.
"In addition, any time an account is flagged by the system, Apple conducts human review before making a report to the National Center for Missing and Exploited Children," Apple wrote on its site. "As a result, system errors or attacks will not result in innocent people being reported to NCMEC."
Apple isn't applying its child abuse detection system to our text messages. That, effectively, is a separate system.
On iPhones logged in to childrens' iCloud accounts, the messages app -- which handles SMS and iMessage -- will "analyze image attachments" of messages being sent or received "to determine if a photo is sexually explicit." If it is, Apple will then alert the children that they're about to send or view an explicit image. At that time, the image will be blurred and the child will be presented with a link to resources about encountering this type of imagery. The children can still view the image, and if that happens, parents will be alerted.
"The feature is designed so that Apple does not get access to the messages," Apple said.
Because this system is merely looking for sexually explicit pictures, unlike the iCloud Photos setup, which is checking against a known database of child abuse imagery, it's likely Apple's text-related system would flag something like your legal photos of your kids in a bathtub. But Apple said this system will only be in place with phones that are logged in under a child's iCloud account, and that it's meant to help protect those iPhone users from unexpectedly being exposed to explicit imagery.
Apple maintains that its system is built with privacy in mind, with safeguards to keep the company from knowing the contents of our photo libraries and to minimize the risk of misuse.
In an interview with The Wall Street Journal published Aug. 13, Apple's Federighi, attributed a lot of the confusion to poor communication.
"It's really clear a lot of messages got jumbled pretty badly in terms of how things were understood," Federighi said in his interview. "We wish that this would've come out a little more clearly for everyone because we feel very positive and strongly about what we're doing."
He also sought to argue that the scanning feature is separate from Apple's other plans to alert children about when they're sending or receiving explicit images in the company's Messages app for SMS or iMessage. In that case, Apple said, it's focused on educating parents and children, and isn't scanning those images against its database of child abuse images.
Still, despite Apple's insistence that its software is built in privacy-protecting ways, the company announced a delay on Sept. 3, saying it plans to take extra time to "make improvements" based on feedback it's received. The company didn't say when its new target release date would be.