Microsoft tries building a smarter iPhone camera app in Pix
Tapping AI and other software tricks, Microsoft crafts a camera app that it says outdoes Apple's at taking pictures of people. An Android version is on its way.
- Shankland covered the tech industry for more than 25 years and was a science writer for five years before that. He has deep expertise in microprocessors, digital photography, computer hardware and software, internet standards, web technology, and more.
Your iPhone has a built-in camera app, but Microsoft hopes to lure you to a new alternative called Pix that's packed with computing smarts.
Pix combines several advanced technologies: artificial intelligence, computational photography and video stabilization. But it isn't like Photoshop, with hundreds of options and abilities for you to consider. Instead, it's designed to improve your photos and videos without you doing any work.
The app, released Wednesday in Apple's App Store, embodies Microsoft's new ethos under Chief Executive Satya Nadella. The software maker's products once served to reinforce the power of its core Windows software. But with Microsoft's failure to win over the smartphone market, the company now is bringing key programs like Office and Outlook to rival mobile products powered by Apple's iOS and Google's Android software.
When you shoot a photo with Pix, it actually takes 10 photos in a half second and presents you with the one it believes is best. If the scene changes significantly, it will present you with a second option and, rarely, three.
Pix works on getting faces right.
The app isn't just for still photos. It can speed up videos into smooth fast-forward versions with the approach used in Microsoft's Hyperlapse app for Android. It can also combine the images into distinctive shots called "cinemagraphs" in which parts of the image are still while other parts are in motion.
"Apple has a strong brand in photography," said Josh Weisberg, principal program manager in Microsoft Research's Computational Photography Group. But Microsoft believed it could help the iPhone take better photos. An Android version of Pix is on the way. "Now we feel we can do that on any platform," he said.
The sophistication comes at a cost. It takes up to two seconds to perform all the image processing, and people don't exactly like waiting for their photos to appear on screen.
How it works
Pix, at this stage, is geared toward taking pictures of people. It tracks faces and tries to set brightness to expose them properly. That's just the first phase, though. From the 10 photos it takes when you tap the shutter button, it deletes those in which eyes are shut, faces are blurred or people aren't smiling, Weisberg said.
Pix combines artificial intelligence, computational photography and video stabilization.
When you're shooting photos with Pix, it's actually taking photos continuously. Microsoft keeps seven photos from before you tapped the button and three from after, because its tests showed that people often missed the moment.
Even though Microsoft discards most of the shots, it first uses them to reduce image noise, or the colored speckles that degrade photos taken in dim or dark conditions. With several frames of the same scene, Pix can zero in on original colors.
In cases where Pix detects a combination of moving and stationary elements, it creates a cinemagraph. For example, it can show people standing still in front of a moving river, or a person's face unmoving except for blinking. Done right, cinemagraphs can be very eye-catching.
"We intelligently detect when to make them and only make them when it's interesting," Weisberg said.
Microsoft isn't the only company trying to blur the lines between photos and videos. Apple's newer phones take Live Photos that record a smidgen of video for a more immersive view. Google Motion Stills can stabilize videos and create cinemagraph-like looped animations. And the new Polaroid Swing app captures one second's worth of your life in what it calls "moving photos."
Bring on the brains
That intelligence stems from the hot field of machine learning, a type of AI in which computers operate with a digital relative of the human brain called a neural network. Training neural networks to produce good results can take a lot of time. But once a model for making the right choices is built, it can run on an iPhone 5S or later.
AI brings a new facet to a technology called computational photography that's been steadily gaining in importance. Camera makers have strived to improve their hardware for years, but computational photography processes images after the fact to go beyond the hardware. It combines multiple images into one, corrects lens flaws, gets rid of blurring from a shaky camera and changes the point of focus after the photo is taken.
Deciding when to build a cinemagraph is one situation where Microsoft takes the AI approach. It also uses that approach for understanding a scene to set exposure correctly, to pick the best of the 10 frames and to detect if a person has taken a photo of something stationary like an office whiteboard.
The app is headed for Android phones, though Microsoft won't say when it's due out. "Android has more complexity in the variety of camera modules, chipsets and versions of Android," Weisberg said.