X

Can a Pikachu drink your coffee? MIT video magic might show the way

Pokemon Go, and other augmented-reality adventures, could get way more immersive, thanks to a crazy-cool new innovation that makes video responsive.

Gael Cooper
CNET editor Gael Fashingbauer Cooper, a journalist and pop-culture junkie, is co-author of "Whatever Happened to Pudding Pops? The Lost Toys, Tastes and Trends of the '70s and '80s," as well as "The Totally Sweet '90s." She's been a journalist since 1989, working at Mpls.St.Paul Magazine, Twin Cities Sidewalk, the Minneapolis Star Tribune, and NBC News Digital. She's Gen X in birthdate, word and deed. If Marathon candy bars ever come back, she'll be first in line.
Expertise Breaking news, entertainment, lifestyle, travel, food, shopping and deals, product reviews, money and finance, video games, pets, history, books, technology history, generational studies. Credentials
  • Co-author of two Gen X pop-culture encyclopedia for Penguin Books. Won "Headline Writer of the Year"​ award for 2017, 2014 and 2013 from the American Copy Editors Society. Won first place in headline writing from the 2013 Society for Features Journalism.
Gael Cooper
2 min read

Pokemon Go can make it seem like a cute Caterpie is sitting right there in your cup of coffee (eww). It's clearly just a cartoon image superimposed over the image your phone is capturing. But what if that Caterpie could react to the coffee, splash it around or drink it, as if it really were swimming in your Starbucks cup?

Two videos from a new MIT project exploring interactive dynamic video (PDF) show the way to that future. Though the videos have been live since August 2, attention from Reddit sparked fresh interest on Friday. Lead researcher Abe Davis, a computer science Ph.D. candidate working in computer graphics, even stopped by the site to chime in.

The first video from MIT's Computer Science and Artificial Intelligence Lab sketches out the process. The camera analyzes vibrations created when an object moves, even slightly, and uses that information to predict how the item will react in new situations, such as when a Pikachu is plopped down on top of it. Davis and crew explain it more thoroughly in the video below, and dub it interactive dynamic video.

There are practical purposes for the technology, of course -- the video mentions examining a bridge to judge its safety -- but it can also be used for fun, as in movie special effects.

As the team's second video points out, it can be used to upgrade augmented-reality games like Pokemon Go to dynamic augmented reality, where objects filmed separately seem to react with each other.

Redditors raised a troubling question: what if this technology someday allows for manipulation of images for sinister purposes, such as making it seem that someone committed a crime by putting them into a video?

"Actually, all of this work on photo and video manipulation has given rise to another line of research: photo and video forensics," Davis said on the thread. "The idea is to come up with algorithms that can tell when an image or video is real, or fake. So even if we can't tell the difference, there is hope that a clever algorithm will."