Your Instagram hashtags helped Facebook's AI get smarter at photo recognition
Facebook reaches new heights in image recognition with an approach that could help artificial intelligence learn from messy real-world data.
- Shankland covered the tech industry for more than 25 years and was a science writer for five years before that. He has deep expertise in microprocessors, digital photography, computer hardware and software, internet standards, web technology, and more.
Facebook's use of Instagram hashtags lets it train AI systems that are much better at identifying subjects in photos. Here, CTO Mike Shroepfer shows some examples at F8.
Facebook has claimed a new artificial intelligence victory in image recognition -- and all those Instagram hashtags helped it get there.
Usually, AI systems are trained using carefully labeled data -- this photo has a cat in it, that one has a steam engine. The result is an AI system that learns to spot patterns even without anyone telling it a cat has round eyes and pointy ears or a steam engine has a big smokestack.
It's an impressively useful technology, which is why it's spreading like wildfire across the tech industry for everything from speech recognition to spam filtering. But it's limited by how well-labeled the original training data is.
Facebook researchers, though, have found the sloppy and often ambiguous hashtags on Instagram can improve training data. The result is a new top score -- 85.4 percent accuracy -- on one photo-recognition test called ImageNet. That beat out a 2017 score of 79.2 percent that Google reported in 2017, Facebook said in a research paper about its approach.
It's an important development in the world of AI, and potentially a step toward systems that are better able to learn from the messy real world rather than carefully prepared input data. AI has delivered impressive goals, but so-called "unsupervised learning" that doesn't need that labeled input data remains an immense challenge.
Facebook's weakly supervised AI training
Facebook calls its approach "weakly supervised." The company, a major power in AI along with rivals such as Google, Apple and Microsoft, announced the development Wednesday at its F8 developer conference.
"We believe our study illustrates the potential of natural or 'wild' data compared to the traditional approach of manually designing and annotating datasets," Facebook researchers said in a paper about the approach.
But when asked how far Facebook has advanced toward truly unsupervised learning, the researchers were cautious.
Facebook thinks AIs that look at photo hashtags could be better at describing photos to people with impaired vision, for example by offering more detail.
"The fact that our models are able to learn from human communication may be a step towards unsupervised learning, but we think it's a baby step," said Manohar Paluri, leader of Facebook's computer vision work for the company's Applied Machine Learning group.
Baby steps can be important, though. Facebook also showed this week in a separate research paper that it's getting useful results in language translation with rougher input data instead of the usual carefully prepared sentences, Paluri said.
Learning from messy real-world data
Hashtags are messier data than carefully prepared captions, but they're still a big step from truly wild image data with no descriptions at all. Still, they're very useful, Facebook concluded, and not just for image recognition. Hashtag data also could be useful for deciding which photos to show you in your Facebook feed or for helping provide better photo descriptions to people with impaired vision, the company said.
One of the big difficulties of training AI systems is how computationally intense it is. Facebook spread the work across 336 graphics chips. The researchers tried various combinations of training data -- the biggest with a whopping 3.5 billion images and 17,000 hashtags.
Already today Facebook uses image recognition for things like figuring out who is in your photos and for automatically flagging photos with pornography, nudity and violence that violate Facebook policies.
Follow the Money: This is how digital cash is changing the way we save, shop and work.
CNET Magazine: Check out a sample of the stories in CNET's newsstand edition.