Microsoft artist taps Kinect, Bing as his chosen media

At Microsoft Research Redmond -- one of 13 labs worldwide, with locations from Beijing and Banglaore to Cairo and Cambridge -- there exists Studio 99, an art space that mashes up data and coding with creative expression. It was a perfect place for James George, a computer scientist turned NYU video art professor, to embark on a three-month stint doing whatever he liked with the tools and staff available at one of the most advanced technology playgrounds on the planet.

"The first week I was here, I really had no plan," George explained in a Microsoft Research interview. "I met [senior researcher] Charles Loop and was excited to see he was working on a system that was a way of capturing people in three dimensions."

Inspired by Loop's work, George decided to take experiments with the Kinect motion sensor to a new frontier.

Now having produced a number a well-known projects using the Xbox accessory, George has become a pioneer of fusing motion-sensor technology and captured images to produce art. Under a collaboration with fellow Carnegie Mellon Fellow Jonathan Minard, George wrote software that would allow the Kinect to work in conjunction with a digital SLR camera to combine color images with motion-captured depth. The resulting effect is what the duo calls "point cloud portraits," which anyone can experiment with by using the open-source RGBDToolkit application.

George expanded that idea to the Eyebeams Art and Technology Center in New York City, where he curated an exhibit called "Wired Frames." George and Minard then launched a successful Kickstarter campaign to produce a documentary called "Clouds," which showcases the technology.

Thanks to Microsoft's incubator-like environment and George's ingenuity, the artist in residence has become an expert in manipulating three-dimensional objects.

"I took his [Charles Loop's] software and the way he worked with these cameras, and I worked out the engineering side of it to how it would work on a larger scale," he said. "I worked directly with him to adapt his software to scale it from a 5-foot ring into a 20-foot ring to capture two individuals at once."

George, who soon will finish his three-month residence at Microsoft Research, is the facility's first artist in residence since Studio 99 opened last year. Microsoft Research itself, however, is far older.

Founded in 1991, the research lab began at the company's headquarters in Redmond, Wash., and expanded to multiple countries. The main goal of the research labs is to push technology forward -- from machine learning and semantics to multimedia and quantum computing -- with the intention of one day bringing those technologies to Microsoft products. For instance, strides made at Microsoft's Redmond lab, combined with international research, have trickled down into products like the codename Cortana personal assistant.

The end result of George's expanded Kinect experiment is a video installation called "Grip." The two-column display features three-dimensional "human" figures that react as viewers come near them. George mounted Kinect sensors at the top of the installation and programmed the screens to display motion-captured movements of two bodies holding -- or gripping -- each other to create a mutually sustaining pose. A viewer's presence alters the nature of the image, forcing the figures to let go and then reconstruct themselves into another pose.

"The concept behind the video shoot was to film people who had just let go of a mutually sustaining pose, so they are falling," he said. "There's a lot of imagery of people falling down. I think this captures a visceral feeling of irreversible action that will get people to think about the way their work might reshape society.

George notes how Microsoft Research is just one of numerous institutions where people's fascinations lead to groundbreaking work. And that "work" eventually becomes an accepted part of everyday life.

"I'm very interested in technological determinism," George said, "in the notion that once a technology is suggested, it will inevitably become part of our culture."

George didn't stop with the Kinect project. In "Wall Queries," he created architectural-scale murals made from 10,000 images of Bing image-search results representing color and shape. He then programmed an algorithm to organize the images in such a way to illustrate how random in nature or eerily homogenous they can be.

"It's one example of how new ways of collecting and capturing visual information have become part of our daily experience," he said. "And I feel an urgency to reflect on the biases embedded within the formats and practices we are adopting so swiftly."