​Microsoft puts some pizzazz into panoramic photos

Microsoft on Thursday added new creative options to its ICE software for creating panoramic scenes out of a collection of photos -- and now from a video, too.

ICE (Image Composite Editor) 2.0, last updated in 2011, is a somewhat experimental Windows program that's long been due for an upgrade. Fortunately for photo enthusiasts, there are some great new features in the new version:

• You can create panoramas from a video, with ICE stitching together still frames drawn from it.
• If you have a recurring figure in that video -- say, a skateboarder you've tracked traversing the scene -- you can designate where you'd like to have repeat appearances of that figure across the panorama.
• When you create a video that has awkward gaps in it -- a common consequence when you don't keep your camera level as you pan across a scene -- ICE 2.0 can "autocomplete" to fill in the missing pieces.
• And if you've taken shots for a 360-degree panorama, you can create a "small world" view that turns your shots into a diminutive planet.

By blurring the boundaries between single and multiple images and between still photos and video, ICE shows how much untapped potential there is in the domain of computational photography. That field of research applies computer processing techniques to extend well beyond just taking a single shot with a camera.

Computational photography

For example, computational photography has led to gigapixel images you can dive into with zooming and panning controls, to high-dynamic range (HDR) images that fill in shadow and highlight detail by combining multiple exposures, to cameras that can correct lens optical problems, and even to Lytro's light-field cameras that let people focus a shot after it's taken.

"There's a ton of opportunity to move the development of photography beyond the camera into the viewing experiences of users," said Matt Uyttendaele, manager of Microsoft Research's computational photography group.

Computational photography means new opportunities to challenge incumbents, as Lytro is trying to do with its cameras. A lot of computational photography sticks with the same old cameras, though, and adds post-processing software. Even though Adobe Photoshop has had an HDR tool for years, photographers still support specialist programs like HDRsoft's Photomatix Pro, Oloneo's PhotoEngine and Nik's HDR Efex Pro -- now part of Google.

For Uyttendaele, the most exciting frontier of computational photography is the smartphone -- especially with camera makers opening up access to the raw image sensor data before it's been processed into a JPEG.

"It's the first time we've really had a hackable camera. We're getting more and more access to the pixel development pipeline. That's super exciting," he said. "The level of computation we can apply at image formation time is going to open up lots of cool applications."

Panoramic imagery is increasingly common these days, with smartphone apps making it easier to sweep out a scene and instantly combine the shots into one single view through computational photography. The ICE software's abilities go a step beyond what smartphones typically offer, though.

ICE in action

ICE's core job is to examine images to figure out how they stitch together into a group. At that, I found ICE very capable for the most part. In more than a dozen tests from my own panoramic originals, stitched images were devoid of problems like discontinuities in the horizon, mismatched tree branches. Only in one case did I spot distracting tonal changes in the sky.

My biggest complaint was that in two cases, ICE couldn't figure out how some images fit into a whole, leaving one edge or the other of my panorama cropped off. Mostly, though, it handled vertical and horizontal panoramas fine, including some grid panoramas with a rectangular array of source images. My second gripe is that images seemed a little soft compared to the originals.

For still-photo fans, the biggest new feature is the ability to fill in panoramas' inevitable gaps. I found the autocomplete function to be hit or miss, but still very useful when it did work.

With featureless areas such as blue skies or a speckled granite foreground, autocomplete has an easy job. But when imagery patterns were complicated, ICE's copy-and-paste artifacts can make your shots eligible for publication on Photoshop Disasters. Most glaringly, slices of trees or buildings sometimes appear floating in the air, and big patches of duplicate scenery look like a glitch in the matrix.

But it's hard to get too sore about autocomplete. After all, it's hard to make up image data that isn't there in the first place. And several times, using autocomplete to add a little blue sky in one part of an image saved me from cropping off the top of a mountain peak elsewhere. A good approach to using it was to let it make its best guess then simply crop away what was unacceptable. You could also export the image and then bring it into a high-powered editor like Adobe Photoshop, whose content-aware fill feature might do a better job with the missing pieces.

Uyttendaele has ideas about how to improve autocomplete. One intriguing one -- assuming copyright issues were sorted out -- is to draw on the vast supply of images already available on the Internet.

"You could go to the Web and search for pixels to fill even better," Uyttendaele said.

The video techniques add a lot more spice, too. I love the idea of automating the process of pulling the same person multiple times into a panoramic image. That usually requires a lot of manual labor. Microsoft has a particularly compelling example with a juggler tossing clubs to himself from one side of the frame to the other.

The new ICE also gets a revamped user interface, the ability to export panoramas to Microsoft's Photosynth Web site, and 100 percent zoom for better previewing.

From Microsoft Research

ICE, from Microsoft Research's Interactive Visual Media Group, isn't a regular software product from the company. Instead, it serves as a vehicle to develop technologies that show up in other Microsoft products. ICE's technology, for example, is used in its photo-stitching Photosynth software and its Bing Maps service, where countless individual image tiles must be joined into a seamless whole.

ICE's purpose is to push what Microsoft Research can do -- and to encourage other parts of the company to adopt the technology.

"I've been in research for 15 years. When I started off, I had this rose-colored image of how product transfer would go from research into Microsoft products: we'd write a great paper, have some prototype code, and the idea would be so great that the project would sell itself," Uyttendaele said. "In reality, there's a lot of work left to do going from a research prototype to a product our customers can use. Microsoft ICE, where we finish the technology to the point where people can download and use it, forces us to polish it up so product groups can take it from us."

Photosynth for Android?

Given that mission, it's hard to imagine Microsoft devoting lots of resources to a version of ICE for Mac, for example. Moving the core engine is easy, but some user interface aspects are tied to Windows, Uyttendaele said.

But ICE-derived products are a different matter, particularly in the era of CEO Satya Nadella, who's more willing than his predecessor to bring Microsoft software to non-Windows operating systems like Apple's iOS.

So how about Photosynth for Android, for example?

"We might do that. I can't promise anything," Uyttendaele said. "Microsoft is writing more and more stuff for other platforms."