This AI software knows what you're going to do next

Researchers take a key first step in developing artificial intelligence that can see into your near future.

Eric Mack Contributing Editor
Eric Mack has been a CNET contributor since 2011. Eric and his family live 100% energy and water independent on his off-grid compound in the New Mexico desert. Eric uses his passion for writing about energy, renewables, science and climate to bring educational content to life on topics around the solar panel and deregulated energy industries. Eric helps consumers by demystifying solar, battery, renewable energy, energy choice concepts, and also reviews solar installers. Previously, Eric covered space, science, climate change and all things futuristic. His encrypted email for tips is ericcmack@protonmail.com.
Expertise Solar, solar storage, space, science, climate change, deregulated energy, DIY solar panels, DIY off-grid life projects, and CNET's "Living off the Grid" series Credentials
  • Finalist for the Nesta Tipping Point prize and a degree in broadcast journalism from the University of Missouri-Columbia.
Eric Mack
2 min read

Prof. Jürgen Gall (right) and Yazan Abu Farha from the Institute of Computer Science at the University of Bonn are teaching computers to see your near future.

Barbara Frommann/Uni Bonn

For decades now we've been waiting for the robot butler or smart home that knows what we need and when, but so far we haven't come much further than digital reminders to leave for the airport and maybe follow up on an email. 

But researchers at Germany's University of Bonn say they've taken an important first step in creating a self-learning computer program that can anticipate our next actions -- in the kitchen, at least.

A team had two different neural networks "watch" four hours' worth of videos in which humans are shown preparing different salads. This data was used to train the system when each of about 20 actions required to prepare the salad started, as well as the duration of each.

"Then we tested how successful the learning process was," Jürgen Gall, co-author of a new paper detailing the research, said in a statement Wednesday. "For this we confronted the software with videos that it had not seen before." 

The program was shown more salad-making videos and told what actions it was watching for the first 20 to 30 percent of one of the videos. It then had to predict what would happen next for the rest of the clip.

The result? The algorithm was able to predict both which actions were coming up with 40 percent accuracy. That clearly leaves some room for improvement, but it's also a major first step.

"Our methods are the first to predict video content of up to several minutes length," the paper reads. It'll be presented at the Conference on Computer Vision and Pattern Recognition (CVPR), which is June 19-21 in Salt Lake City.

The system's predictive accuracy falls the further out an action is, but still gets it right 15 percent of the time even for activities that were more than three minutes in the future. 

Still, the researchers emphasize their work is a key start to the brand new field of activity prediction. In addition to salads, the system was also trained on videos of people making a wide variety of breakfast dishes with similar success. 

8 sci-fi ideas that might become science fact pretty soon (pictures)

See all photos

Once the technology is more dialed-in and refined it could lead to whole new ways for interacting with computer assistants and robots that anticipate when to lend just the right kind of helping hand, or whisk. 

"We want to predict the timing and duration of activities -- minutes or even hours before they happen," Gall explains.  

If that becomes reality, we could move beyond the era of smart homes that manage the thermostat and lights to helping homes that prep your ingredients and preheat the oven moments before you arrive home hungry after a long day. Decades later, Rosey the Robot is finally becoming feasible.

Crowd Control: A crowdsourced science fiction novel written by CNET readers.

Solving for XX: The tech industry seeks to overcome outdated ideas about "women in tech."