Researchers try Google approach to understanding photos

Carnegie Mellon researchers develop techniques that let computers automatically add or replace elements in a digital photo.

Stephen Shankland Former Principal Writer
Stephen Shankland worked at CNET from 1998 to 2024 and wrote about processors, digital photography, AI, quantum computing, computer science, materials science, supercomputers, drones, browsers, 3D printing, USB, and new computing technology in general. He has a soft spot in his heart for standards groups and I/O interfaces. His first big scoop was about radioactive cat poop.
Expertise Processors | Semiconductors | Web browsers | Quantum computing | Supercomputers | AI | 3D printing | Drones | Computer science | Physics | Programming | Materials science | USB | UWB | Android | Digital photography | Science Credentials
  • Shankland covered the tech industry for more than 25 years and was a science writer for five years before that. He has deep expertise in microprocessors, digital photography, computer hardware and software, internet standards, web technology, and more.
Stephen Shankland
2 min read
Carnegie Mellon University researchers are finding that a computer with enough information at its fingertips can act smart even if it doesn't really understand what it's doing.

It's an idea that should be familiar to anyone who's used Google, which doesn't understand content but which can often point people seeking particular information in the right direction. But the CMU researchers are applying the concept to the tasks of adding information to or removing it from digital photos.

Carnegie Mellon

The researchers plan to present their results in two papers at the Siggraph computer graphics conference in August. In one technique, called Photo Clip Art, the computer plucks images of people from a database, selecting only appropriate ones based on image parameters such as lighting. In the other, called Scene Completion, the computer can patch over unsightly elements in a photo by finding appropriate substitute material from a database of 2.3 million pictures stored at the Flickr photo-sharing site.

"We're trying to match the properties of the input photo with all the objects of our collection," said Alexei A. Efros, an assistant professor of computer science and robotics at CMU. But the researchers are doing so without the computer actually having to understand the photo's content, geometry, or other aspects.

Alexei A. Efros
Alexei A. Efros,
professor, CMU

"Just by having lots of data, we are implicitly understanding the picture even though we don't have any notion of what is there," Efros said.

However, Efros also has conducted research about giving computers a deeper understanding of image properties such as geometry, illumination, motion blur and perspective. His group now is working on uniting that analysis with the new work on matching images.

A graduate student, Jean-Francois Lalonde, led the Photo Clip Art research, and Microsoft Research employees contributed as well, Efros said. Another graduate student, James Hays, led the Scene Completion work.