Pokemon Go can make it seem like a cute Caterpie is sitting right there in your cup of coffee (eww). It's clearly just a cartoon image superimposed over the image your phone is capturing. But what if that Caterpie could react to the coffee, splash it around or drink it, as if it really were swimming in your Starbucks cup?
Two videos from a new MIT project exploring interactive dynamic video (PDF) show the way to that future. Though the videos have been live since August 2, attention from Reddit sparked fresh interest on Friday. Lead researcher Abe Davis, a computer science Ph.D. candidate working in computer graphics, even stopped by the site to chime in.
The first video from MIT's Computer Science and Artificial Intelligence Lab sketches out the process. The camera analyzes vibrations created when an object moves, even slightly, and uses that information to predict how the item will react in new situations, such as when a Pikachu is plopped down on top of it. Davis and crew explain it more thoroughly in the video below, and dub it interactive dynamic video.
There are practical purposes for the technology, of course -- the video mentions examining a bridge to judge its safety -- but it can also be used for fun, as in movie special effects.
As the team's second video points out, it can be used to upgrade augmented-reality games like Pokemon Go to dynamic augmented reality, where objects filmed separately seem to react with each other.
Redditors raised a troubling question: what if this technology someday allows for manipulation of images for sinister purposes, such as making it seem that someone committed a crime by putting them into a video?
"Actually, all of this work on photo and video manipulation has given rise to another line of research: photo and video forensics," Davis said on the thread. "The idea is to come up with algorithms that can tell when an image or video is real, or fake. So even if we can't tell the difference, there is hope that a clever algorithm will."