Matterport breathes life into fully immersive 3D models

Picture a 3D virtual representation of your living room, one you can fly over in a top-down view and even move through with the fluidity of a first-person video game. Now imagine having the ability to tinker with that space: Change the paint on your walls, drop in a new couch to see how it fits with the existing furniture, or perform accurate measurements of the room, all on a computer screen.

That's Matterport's vision for the future of 3D modeling, and it extends beyond home renovation. From architecture and construction to real estate and crime scene visualization, the scope of 3D models is expanding and the hardware and software that allow us to map our physical world are getting cheaper, faster, and better. Matterport is hoping it can be not only the Nikon and Canon, but also the Adobe of the burgeoning industry, offering a professional-grade camera and a cloud-processing platform for making 3D modeling exponentially easier and accessible.

If you've heard the name Matterport recently, it was likely in association with Project Tango, the mobile 3D mapping venture out of Google's Advanced Technology and Projects division, the research arm of Motorola Mobility that Google absorbed into Android before selling Motorola to Lenovo. Matterport got its hand on a prototype and last month released one of the first 3D models using Tango. Matterport, however, is aiming for a more pro-grade, less experimental market when it comes to its own breed of modeling.

On Thursday, the Mountain View, Calif.-based company -- created in 2011 and running on Y Combinator and VC funding amounting to $10 million -- launched its full platform suite, which includes the $4,500 Matterport Pro 3D camera alongside its subscription-based cloud service and Web player. Before that, an early-adopter program saw only a few dozen cameras get out into the wild to create more than 1,000 3D models. Matterport will sell the device and service directly, aiming primarily at contractors, real estate companies, and architects with the intention of drastically changing how those industries work with digital visualization.

Matterport modeling is all about automation

Although 3D modeling is of course not new technology -- it has been used for years in architectural modeling, video game design, film CGI, and computer animation -- it's still an incredibly intensive and multistep process. You need people to take measurements and draw out schematics to build an outline of a space, photographers and videographers to capture images for texture and depth, and designers and engineers working in 3D CAD software to construct it. Even then, there's still no easy way to cohesively stitch together all those parts into an easily transferable and reasonably sized file.

"If you spend enough effort and time, you can make a model that looks like something we can make," said Matterport CEO Bill Brown in an interview with CNET. "You'll have spent $20,000 or $30,000, but you end up with something you can't distribute. You can give it to someone if they have a CAD package."

It's no coincidence that Brown was tapped by Matterport's three co-founders -- Reactrix founder Matt Bell, PayPal alum Dave Gausebeck, and former SRI 3D guru Mike Beebe -- to be CEO. Brown came from Motorola Mobility, where he was general manager of a division called "converged consumer products" during its time under Google's wing, and where Project Tango would later be hatched. The combined expertise of Brown and Matterport's co-founders results in a highly focused effort that's more practical than flashy, and less intent on doing something radical -- like Tango -- than it is in tackling what the company thinks is a dormant technology ready for acceleration in select fields.

That's how Matterport was able to take the 3D modeling process and automate away the most tedious aspects of it. "In a matter of an hour, you can do something that takes two days for people to do now," Brown said.

What exactly makes Matterport's camera that much of a leap? For one, it has the ability to capture geometric and texture data simultaneously, while offloading much of the intensive computing to combine that data to the cloud to be done after the fact. "It's shooting at 30 frames per second, so it's in essence taking a video as it spins," Brown explained.

With three sensors relaying information between 2D and 3D, the camera is able to take in a near-360 degree scope of the room in one motorized sweep that takes less than half a minute to complete. Combine anywhere from as little as four to six sweeps and as many as 15 to 20 of an area and you have a detailed model that can be compressed down to an average file size of 50MB to 75MB. Even then, one can dig into the raw files of the sweeps and swap in stills, so that fuzzier portions of the model containing books on a shelf or a clock on the wall can suddenly have the fidelity of a hi-res photograph.

Even mirrors, which would have to show shifting reflections as someone moves through the 3D model, can be dealt with. "We identity areas that are mirrors and replace those with digital mirrors," Brown said. "A lot of the techniques and technologies we're using on the application side come from the computer gaming world." As in video games, mirrors and differences in lighting can be easily replicated with video effects tools.

But the key to Matterport's efforts lies not just in small file sizes, feature sets, and affordability, but also in ease of use. "We've automated that entire process and got it to the point where anybody can operate the camera and the cloud processing figures out how to do everything," Brown said. "It puts this thing together like a jigsaw puzzle."

And it is true that anyone can operate the camera. It's as simple as pressing a button; I did it myself, on an iPad, while a Matterport camera 3D-mapped a studio room at CNET's San Francisco office. You have to manually move the camera to desired areas for new sweeps, and you also have to walk around it as it's moving -- it pauses after each motorized rotation -- unless there's portion of the room that you can use to conceal yourself.

The result, after sending the data to the cloud service and letting it build the model for roughly 30 to 45 minutes, is a mix of awe and a strange sensation akin to animation's uncanny valley, thanks in part to the video game-styled movement you're employing within an unprecedented level of photorealism. Moving around can be done with directional arrows, including a jumping capability with the space bar. Or if you're on an iPad, which can run Matterport's 3D models via its Web player, you can use specialized touch commands like two fingers to strafe side to side or pinch-to-zoom to go from the top-down view into first-person mode.

The best description of the experience would be this: Google Street View meets the interactive panorama, the multimedia trend of the last few years that uses a series of shots from a singular point to create an immersive, drag-able photograph.

In that vein, Matterport is aware that its tackling more than just the current 3D modeling use cases. "We're trying to establish a new category of media," Brown said.

Next up: Movement modeling and mapping with mobile

Matterport's hardware and software have clear limitations. For one, the camera can't process glass and light-induced reflections, rendering any windows entirely translucent. And certain levels of light and drastic changes in those levels are also tricky, meaning Matterport can't be used outdoors except in select conditions.

Most importantly, though, the camera tech is hampered by movement, a shortcoming that Matterport will have to work out down the line. "It's going to be a specialized set of processing that we're going to need to figure out," Brown said of moving images. "I do think that at a high-level view, we can tell the difference between something that's moving and something that's not moving. We can isolate the things that are moving and handle them appropriately."

That means right now, not only must the camera remain still, but also anyone sitting anywhere within the camera's sight must sit still as well, or else be rendered into a kind of "2D plus," as Brown put it, resembling a paper cut-out.

That's an integral challenge for the 3D modeling field at large, and especially the large players in the 3D sensor industry, nearly all of which are communicating with Matterport, Brown said. Because the goal is not just to move 3D modeling capability to mobile devices, as Google is attempting with Project Tango, but to implement that idea as thoroughly as the integration of cameras was to early feature phones.

"We kind of say, down the road as you have 3D sensors in these mobile devices, you're not going to take 2D pictures anymore. You're always going to capture the 3D behind because there's a bunch of useful things that you can do with that 3D, even if it's just a still image," Brown said. "If you were going to walk through the space and shoot a video, you could build a 3D environment that in addition to that linear video could give a user the ability to not just pause, but back up on a path and say, 'I'm going to see what was to the sides as you were walking.'"

To achieve that, Matterport won't be focusing on drastic cost-cutting, multiple product tiers, or pushing to make its own mobile device, despite having originally started out attempting to develop a 3D mapping-capable handheld.

"One of the big pushes that we have is to work with the folks on the mobile side to make sure that these devices are going to capture the right data," Brown said. "We'll probably just make our software work on those devices. I'd never say never because there might be a point where a partnership with one of those company that makes one of those devices makes sense. But it's not Matterport's focus," he added.

The company's long-term view then is to build out its software platform to the point where Matterport applications become the primary way to process 3D models, ones captured from mobile all the way up to professional-grade, DSLR-level cameras like the one Matterport is now offering.

"There's a point, whether it's five years down the line or whenever it is, where every time someone captures a 2D image, they're capturing 3D data," Brown said. "We're going to be the company that makes that combination very useful."

Update at 11:38 a.m. PT: An earlier version of this story incorrectly spelled a Matterport co-founder's last name. It is Matt Bell, not Belle.