In 2017, your phone's camera will have superpowers
Commentary: If you're wondering where phones are heading next year, think magic 3D-scanning cameras. Here's what I learned from using Google Tango for a few weeks.
- Nearly 20 years writing about tech, and over a decade reviewing wearable tech, VR, and AR products and apps
Virtual reality is a somewhat understandable concept in 2016. You put on a headset, you find yourself in 3D worlds. But augmented reality -- AR -- is still a bit less understood.
You could engage with augmented reality with a headset, and see 3D objects "projected" into your real world -- something that's usually called "mixed reality." Augmented reality doesn't need a headset, though. It can use your phone. In fact, it already does. 2016's summer smash hit Pokemon Go was the most widespread use of AR ever seen.
So, then, what is Google's Tango, and why should you care? Released at the end of the year in a large phone called the Lenovo Phab 2 Pro that most people will probably never be aware of, Tango is an advanced type of next-generation camera tech for phones. It's basically an array of cameras that enables depth sensing in 3D.
I've been using the Phab 2 Pro for several weeks, along with its assortment of launch apps. The phone itself isn't great. For a variety of reasons -- an older version of Android, no NFC, and the fact that it's just downright huge -- you shouldn't buy it.
But as the first Tango phone that's available at retail, it's still a big deal -- even if AR is still a niche technology. Here's why. Consider this a peek at what you can expect from more phones starting next year.
Depth sensing cameras let you place virtual things in the real world, sort of.
An infrared camera, wide-angle camera, and additional RGB camera track the immediate environment and can map 3D space, and also sense the edges of objects. In a virtual shopping app from Lowe's, I put a lamp on my floor, and an end table, and a big cozy armchair. I could walk around them, and they'd stay in place where they were supposed to be. You can do similar things with a standard phone camera, but they don't stay in place as persistently.
Virtual lamp.
AR with depth sensing means whole-room 3D scanning.
The most amazing thing you can do with Tango is whole-room 3D scanning. The Matterport app slowly scans everything around within a certain radius...slowly, as if painting in the world in many colored dots. The scans are time-consuming and take up a lot of storage, but when I was done I had a fuzzy but 3D model of my office, or the NJ Transit train, or my bedroom. I could spin around and zoom in on details, or even measure distances in that scanned space with a built-in tool. Tango can track larger spaces and fit together details scanned in piecemeal. Google is planning on using some of these ideas to create 3D indoor maps.
Objects don't rest in your hand, so forget about that fantasy.
You can't get too close, or too far.
Tango is finicky. It's a lot like an Xbox Kinect camera, shrunken down and put on a phone. It can only sense a certain distance, so far-off things down a hallway won't get measured...or even things all the way across a room. And it can't start tracking until something's at least a foot or two away. Lighting conditions can't be too dim, either. I wasn't able to make AR dominoes appear on a book held on my lap...I needed to stand back and paint them on a table or floor.
Apple's dual-camera Portrait mode on the iPhone 7 Plus has similar types of restrictions: I needed to line up my shots just right. This could end up being a defining trait of more advanced phone 3D AR cameras.
Measuring things works really well...and is hard to figure out.
Most of the apps are really hard to figure out, though.
The interfaces for many Tango AR apps I tried weren't intuitive. That's bad design, but it also suggests the challenges in explaining what a depth-sensing 3D camera means for using camera tools. In Lenovo's Measure app, which cleverly acts like a 3D measuring tape for your space, it took me a while to understand how to begin and end measurements. Many apps are exploring 3D content, but with a 2D screen and touch controls.
Most people can't tell the difference between depth-sensing AR and Snapchat on your phone.
A lot of people I showed Tango to seemed vaguely charmed. Hey, a virtual dog rolling on the real floor! A weird dancing gorilla in the corner of the room! Dominoes on the carpet! But the thing is, we've seen things like this on our phones. Look, a Pokemon on the lawn! Hey, my face is old and I'm wearing a sombrero! Snapchat's lens filters are so good, and Pokemon Go is so fun, that people have a hard time understanding why this phone is better. The improved tracking of virtual objects on real surfaces goes unappreciated. And that's a problem.
Best use cases so far? Room-scanning and home improvement.
The most practical functions are the ones that earn the biggest wows. The idea of placing accurately scaled furniture into a room to "test" out, or hang a picture, or change the carpet, makes sense (it's a shame the demo apps don't pull very deep catalogs of objects to try).
The Occipital Bridge adds Tango-like functions to an iPhone via a plug-in camera sensor.
Eventually, more phones will get tech like this...
Motorola's already hinted that its Moto Z phones might get Tango swappable modules in the future. Other phones are bound to get more advanced depth-sensing features: Intel's Realsense cameras could arrive in other devices, and many apps like Microsoft's upcoming 3D-scanning app for Windows 10 are learning to accomplish similar tricks on regular phones.
Companies like Occipital make depth-sensing cameras like Structure as separate add-on extensions for the iPhone. As computer vision technology becomes more advanced, phones will accomplish more augmented-reality tricks through software. Consider Snapchat's face filters once again, for instance.
...but you can wait.
Tango is undoubtedly a more impressive step forward for 3D scanning, but I didn't find it to be a jaw-dropping experience. Tango's killer features rely on good apps and use cases, and right now the tech is still too buggy and experimental. Shrunken down, and bolstered with more apps that can take advantage of its functions, depth-sensing cameras can help turn phone cameras into tiny magic scanners. Those days aren't quite here yet...or, your regular phone camera can probably do enough similar things to tide you over.
We might see more cameras like these from Apple, Samsung, or others in 2017. If we do, the apps need to be more helpful, the phones smaller, and the experience better. But if you're wondering where phones are headed next, look towards Tango's ideas. Smarter cameras are just getting started.