'Seam carving' photo resizing now for video

MONTEREY, Calif.--In August, researchers unveiled a new way of shrinking or expanding photos called seam carving. Now it turns out the technique applies to video, too.

Ariel Shamir, a senior lecturer at the Efi Arazi School of Computer Science in Israel and a visiting scientist with Mitsubishi Electric Research Laboratories, showed off the technique at the 6sight digital-imaging conference here last week. (Adobe Systems has hired another seam carving researcher, Shai Avidan.) The technology analyzes a picture for vertical or horizontal "seams"--the term the researchers use to describe a path traversing the photo where pixels are most like their neighbors and therefore least likely to be missed.

The effect is that important areas such as human faces remain intact, while relatively uniform backgrounds such as lawns or skies are squeezed. Seams zig-zag to an extent, for example detouring around clouds through interconnected patches of blue sky. Seam carving works best for photos with multiple subjects separated by an uninteresting background; a subject that occupies the entire frame is likely to be distorted as the image is scaled down.

A related technique called seam insertion reverses the process to add data, giving photos a more spacious look.

During Shamir's demonstration, he widened and narrowed a variety of photos. In addition, he showed how to select specific pixels for priority preservation or removal, in one case excising a girlfriend from a photo the way Soviet censors vanished Leon Trotsky.

And showing off a new twist on the technology, Shamir did the seam carving on a running video of a golfer taking a swing. The golfer remained intact even as the fairway changed from a narrow sliver to a broad tract of green grass. To see the demonstration, check the video above.

One area where seam carving could be useful is in resizing images along with the Web pages they're shown on, for example preserving important parts of a photo even when it's displayed on the tiny screen of some Internet-connected gadget. That could apply to video as well as to still images, though obviously it would require more computational horsepower.

Coming from a journalism background, my instinct is to keep photos generally true to the original, so these automated distortions at first ruffled my feathers.

But on further reflection, it occurred to me that at least when it's working well, seam carving does to an image precisely what my brain does as well: focus on the important bits.

What was that portrait I saw in my art class of youth, an elderly woman seated in black clothing? Her hands and face were painted with detail and care, but the rest of the picture was painted with rough, almost slapdash strokes. But the painting was fine: my mind naturally cared chiefly about the face and hands, the instruments of human expression, and all else was largely optional. It was like a good lossy compression algorithm that saves space by throwing away the data we're not going to miss.

Children, too, instinctively do the same thing. When they draw pictures of people, the features on the face often creep around to occupy the entire head. In reality, you can cover most of your facial features with the palm of one hand, so the significant parts don't actually take up much of the actual surface area. But in children's pictures, anything above the eyes or eyebrows evidently doesn't deserve much attention.

So perhaps seam carving is just an overt manifestation of what we already do ourselves.

Just don't use it to mess with any news sites!