Inside Facebook’s plan to turn the world into the MoMA

On Facebook 's industrial-looking campus in Menlo Park, California, a mundane white wall makes up the exterior of one of the buildings, not far from Mark Zuckerberg's desk. In April, the social network's founder and CEO insisted someone put a plaque on the wall to commemorate the space.

All over Facebook's offices worldwide, the walls are decorated by work from artists commissioned by the company to liven up the environs for its more than 20,000 workers. Plaques with the artists' names and short biographies sit beside their creations. At Building 20, a huge Frank Gehry-designed office structure where all of Facebook's top brass works, the artwork includes a mural of polka dot flowers and a painting of the Rev. Martin Luther King Jr.

Then there's that big blank white wall.

To the naked eye, there's nothing there. But if you look at it through your phone's camera with a special app that Facebook created, a world of color suddenly appears. Streaks of blue and turquoise and white jut out in front of you, on the floor and zigzag up to the walls. A waterfall of virtual blue paint cascades down from the roof, dripping into a digital puddle on the floor.

The installation was created by Heather Day, a 28-year-old San Francisco-based artist. It's the first large scale piece of art created with Facebook Camera, a new platform for augmented reality , or AR, in which digital images are overlaid on top of what you're seeing in the real world. Zuckerberg showed off the art project in April at F8, its annual gathering of software developers and Facebook's biggest event of the year.

In August, I became one of the first reporters to see Day's invisible art installation in person. The wooden plaque on the wall says "FB AIR Collection," short for Facebook artist-in-residence. But if you glance at it quickly, you might misread it as "AR Collection."

If Facebook has its way, that would be more premonition than error. The kind of world where an entire collection of AR art exists is the kind of world Zuckerberg and the world's biggest social network are trying to create.

"You can just be walking and say, 'Oh my god, I'm standing in a painting right now,'" says Day from her sunny, three-story loft and work studio in San Francisco's Dogpatch district. "It's new territory."

Watch this: Inside Facebook's augmented reality art project

01:32

Augmented reality has become an obsession for Silicon Valley. Most tech companies see it as the next wave in computing, either as a bridge to virtual reality -- made up worlds that exist only on your digital devices -- or a destination all its own. Right now, AR is synonymous with Pokemon Go, the mobile game that spurred a lot of us last year to go outside and catch digital monsters in the world around us.

Snapchat, meanwhile, has pioneered much of the way young people currently use AR on their phones , with photo and video filters that superimpose a flower crown or a dog nose over your snaps. Apple has a platform called ARKit that lets software developers build AR apps for iPhones. Google followed suit last month with its own platform for Android-powered phones, called ARCore.

But the technology also opens up possibilities of sci-fi lore. Eventually, with a good pair of AR glasses -- something Facebook says it's working on -- you might be able to see if your kid has a fever just by looking at him. You could take your entire workspace with you on a flight, without ever opening up a laptop. You might never buy a new TV again, if all you need is a digital screen over your eyes.

But AR goes beyond the bounds of utilitarian. Part of the promise -- and peculiarity -- of augmented reality is that it muddles the line between the world made of atoms and the world made of zeros and ones. It's Alice and the Cheshire Cat.

Dear Mark

In March, Day wrote a letter to Zuckerberg, unprompted. Her request was simple: she wanted to work with Facebook on a virtual reality project. "To many, technology seems unapproachable," Day wrote. "But through art, technology becomes more human and even an artistic medium in its own right."

Day never sent the letter, but coincidentally, Zuck and Co. came calling anyway. An abstract expressionist painter, she'd already been making a name for herself as the go-to artist for tech companies.

Two years ago, Dropbox commissioned her for a mural at its Seattle office. It's got the same signature style she uses in the Facebook project -- lots of blue and white swatches of paint, like jumbled up Morse code. Airbnb and Facebook's rival Snap also bought paintings from her for their headquarters. So when it came time for Facebook to make a move into creating modern art, the company commissioned Day. (She won't say how much Facebook paid her.)

The whole project took two weeks in April -- leading right up to F8. Much of the process was meeting with Facebook engineers and designers to figure out what she wanted to create and visiting the corner of Facebook headquarters that would become her invisible mural. Facebook's engineers created a 3D computer-generated model of the space so they could map out how her artwork would interact with the environment.

Then, a crew of a half dozen people piled into Day's apartment, with a huge camera rig, to record her as she painted. With the camera positioned overhead, she'd make a brushstroke on a large piece of canvas in the middle of the room, then quickly move out of the way. Then she'd tilt and shoo the canvas, or splash water on the paint, so the camera captured the movement. That allowed the paint and animation to show up when you look at the wall through your phone's camera.

"I've been thinking about how attached everyone is to their phones," says Day, who's got freckles, glasses and bright blue eyes. "It's really interesting to think about how to tap into that audience -- people already looking at their phones."

The project is made possible by something called SLAM, or in tech-speak, simultaneous localization and mapping. It's a technique that computer scientists have been developing since the 1980s, originally meant to help robots navigate through unfamiliar terrain.

But the technique is also used in augmented reality. It takes digital graphics or animation and stitches them onto surfaces in the real world, in real time, while mapping out the objects in the room. Facebook is working on a version of SLAM to do all that computing on mobile devices.

The project originally began at Oculus, the virtual reality company Facebook bought for $2 billion in 2014. But since then, Facebook's artificial intelligence teams have taken it over. Facebook also has a few other AR demos in the works, like an app that lets you paint in the air, and another that brings posters to life.

"For the Heather Day project, we used all this complicated machinery," says Alvaro Collet, a researcher on Facebook's Applied Machine Learning team. "We have the opportunity with SLAM to do that just with a phone."

'G' is for good, 'E' is for evil

Augmented reality might be the latest trend in Silicon Valley, but it isn't a new concept. L. Frank Baum, author of "The Wizard of Oz," predicted a form of AR in his 1901 novel "The Master Key." In it, the protagonist is offered a magic pair of spectacles called a character marker. When he wears them, he sees letters over people's foreheads based on their moral fortitude: "G" for good, "E" for evil, "W" for wise, and so on.

The term "augmented reality" is said to have been coined as early as 1990 by a researcher at Boeing named Thomas Caudell, but it was used mostly to help workers with manufacturing. The tech has gained so much steam in the past few years because smartphones are finally capable of handling all the computing power to make the AR experience seem realistic.

But there are potential pitfalls in combining modern art with AR. One of them is the issue of permanence and reliability. Carla Gannis, assistant chairperson of the digital arts program at the Pratt Institute, used to be primarily an oil painter. Now she teaches a class on augmented reality and art. "I had to learn to deal with the instability of the platform," she says.

Gannis has done several AR art projects, and each time, she says it was like pulling teeth to get people to download a free app so they could view the installation.

There may also be problems when a company as big and powerful as Facebook, which has been battling critics over the past year for its handling of fake news, gets involved. "Is Facebook the right platform for artists that want to deal with provocative content? There's a lot of gray area there," she says. "What kind of data are they acquiring on us while we view the art?"

AR makes art -- in different ways -- both more and less accessible. On the one hand, art can be everywhere around you now. On the other, AR art can only be enjoyed by people who own smartphones. Around 5 billion people still don't.

Still, Gannis believes it's the way of the future. So much so that it's even altered the way she talks about life and consciousness. She tries not to put things in terms of "real world" or "reality" anymore. "There's physical space and virtual space, but it's all real life," she says. "That's where we are now."

Zuckerberg seems to feel the same way, though he puts it in different terms. At F8, with Day sitting in the audience, he unveiled the art project.

"One of the funny side effects of this is, at Facebook, we noticed that now there are just people gathering around looking at white walls," Zuckerberg said. "This is going to be a thing in the future."

And he's got the power to make it so.  

The Smartest Stuff: Innovators are thinking up new ways to make you, and the things around you, smarter.

Special Reports: CNET's in-depth features in one place.