Google AI now can give YouTube videos a wacky background

Machine learning proves its worth for new video effects tech: distinguishing between faces and backgrounds at 100 frames per second.

Stephen Shankland principal writer
Stephen Shankland has been a reporter at CNET since 1998 and writes about processors, digital photography, AI, quantum computing, computer science, materials science, supercomputers, drones, browsers, 3D printing, USB, and new computing technology in general. He has a soft spot in his heart for standards groups and I/O interfaces. His first big scoop was about radioactive cat poop.
Expertise processors, semiconductors, web browsers, quantum computing, supercomputers, AI, 3D printing, drones, computer science, physics, programming, materials science, USB, UWB, Android, digital photography, science Credentials
  • I've been covering the technology industry for 24 years and was a science writer for five years before that. I've got deep expertise in microprocessors, digital photography, computer hardware and software, internet standards, web technology, and other dee
Stephen Shankland
2 min read

Google's AI software can detect the subject of a video and change the background.


Google's artificial intelligence researchers have a new trick up their sleeves: giving selfie videos a new background the way you can with still photos today.

Switching out background scenery is a decades-old technology, but it's generally complicated and time-consuming -- think of making movies with superheros acting in front of green screens so computers can later replace the uniform green color with digital effects like exploding buildings. Having 3D scene data, for example extracted from newer iPhones' twin cameras, makes the process only a little easier.

Google, though, trained a neural network with lots of carefully labeled imagery that helped it learn how to distinguish facial features -- eyes, hair, glasses, mouths and so on -- from everything else. The result is a system that can swap out backgrounds fast enough to keep up with video. Digital video frames whip by at 30 frames per second, but Google's technology works at 40 frames per second on a Google Pixel 2 phone and more than 100 frames per second on an Apple iPhone 7.

Want to try it out for yourself? Sorry. For now, it's available to only a pretty small number of high-profile YouTubers with access to the YouTube stories service, which offers an ability to share Snapchat-esque short videos.

Google detailed the research in a blog post Thursday.

"Our new segmentation technology allows creators to replace and modify the background, effortlessly increasing videos' production value without specialized equipment," Google programmers Valentin Bazarevsky and Andrei Tkachenka said in the blog post.

Google uses machine learning in its video segmentation technology, which can determine background imagery that can be replaced with something funny or eye-catching.

Google uses machine learning in its video segmentation technology, which can determine background imagery that can be replaced with something funny or eye-catching.


The technology shows how versatile AI and machine-learning technology is for solving complex computational problems. Neural network technology doesn't have to know endless rules, like "faces have two eyes, located above the nose, unless the face is viewed in profile, in which case there may only be one eye visible, unless the subject is wearing dark glasses." It just has to be trained on enough photos, labeled by humans who do actually know what a face is, that it eventually learns the patterns. It's proven a remarkably useful technology for everything from screening out email spam to predicting what word you're trying to type into your phone.

Google already uses AI in a related feature today on its Google Pixel 2 phones. The phones only have single cameras, but AI helps blur backgrounds for portrait-mode photos.

'Alexa, be more human': Inside Amazon's effort to make its voice assistant smarter, chattier and more like you.

Does the Mac still matter? Apple execs explain why the MacBook Pro was over four years in the making, and why we should care.