LOS ANGELES -- It started like a Disney ride. Long lines, loud music. Microsoft's Halo 5 Experience booth looked like the entrance to a space station. A man in a white lab coat measured my eyes with a white box, writing the number on a tag around my neck.
is a headset that specializes in augmented reality (AR), but it's here at an E3 show that's . These are different technologies. How does HoloLens distinguish itself, and show what it can do best, to people who've never tried it before?
I never used HoloLens before today, but I've used every type of virtual reality (VR): the, , , , . Virtual reality uses lenses to wrap an immersive extra-wide 3D experience around you, but it's also closed off to the real world. Augmented reality layers virtual things into the actual world, usually using markers, or cards, to trigger positioning. Microsoft called HoloLens "mixed reality," in that it overlays those 3D images into the real world using head-mounted 3D cameras to keep the virtual stuff moving against reality, like it's an actual hologram -- from the point of view of the wearer, at least.
(As usual with HoloLens demos thus far, Microsoft banned photos and video; included herein are mostly HoloLens images from earlier events.)
HoloLens as a theme park experience: Halo 5
It was that mixed reality that I was seeing when I popped on the HoloLens and started the Halo 5 Experience at Microsoft's E3 booth.
Inside the "waiting room," I line up against a yellow line, and put a HoloLens helmet on my head: it rests on my head like a large baseball cap, with the dark visor dropping down in front of my eyes. I see arrows appear, pointing down a hallway. I get up and walk to a big Star Wars-like briefing room. As attendants wait, I look at a central table and see battle plans rise up like a Death Star holographic map, telling us what our upcoming 12-on-12 battle will be like. I try to pay attention, but I'm too busy examining what the HoloLens feels like.
The curved visor sometimes feels shimmery, like a rainbow. Is the HoloLens projecting into my eyes, or on the screen? As I get closer to the table, I can look around the base map, examine things up close. It's like a model assembled in front of me. Further details on our battle goals point to ships, and our drop point, and where supplies are. A card emerges from a table (a USB key), and virtual arrows tell me to grab it.
The whole HoloLens experience feels like a prop at a Disney ride. And sure enough, like any Disney ride, after the briefing was done my magic helmet was collected and I was taken to a side room to play...regular multiplayer Halo 5 on a regular TV screen.
Project X-Ray: Room-filling shooters
I was told that Kanye West was using HoloLens ahead of me. I had to wait for him. The upstairs demo rooms were small, office-like. Each HoloLens experience gets its own room.
Finally played HoloLens, but had to wait for @kanyewest to finish. I beat his high score in Project X-Ray, though.— Scott Stein (@jetscott) June 17, 2015
The Project X-Ray game involved alien spider-creatures that started popping out from the walls as I looked around. It's a demonstration of how mixed reality (or, augmented reality) can fill a room for a video game. I held an Xbox controller, and used the triggers to blast creatures as I ducked and walked around them. Another trigger turned on X-ray mode, where I could see through the walls where more creatures were hiding.
HoloLens could sense the room's walls and generate cracks, and even gaping holes where things crawled out. The cracked walls remained, and I could see pipes and pieces of virtual wall behind. What amazed me was how the display's brightness mixed with a normal room and looked convincing. HoloLens "holograms" have a glowing look, but feel very convincing. Except, of course, when things left my display's field of view. Then, all I saw was an empty sofa and walls.
Apparently, my high score beat Kanye West's. So, at least there's that.
Minecraft: Tabletop gaming
My final HoloLens game was the one that'll probably be its killer app:. I stood in front of a large, square wooden table in the center of the room, and as I put on the HoloLens I saw a cross-hair marker appear. I looked around the room, and the cross-hairs hugged the table, then the floor, then the cabinet and walls. HoloLens knew where to place the 3D marker, largely pretty perfectly. I asked how large the HoloLens camera's "seeing" distance was, and got no clear answer; enough to fill a normal living room, apparently. My guess it'll roughly match the Kinect's seeing distance.
I looked at the wall, and created a TV screen by using a voice command. Framed by Minecraft bricks, the screen stayed in proper place as I moved around, or I could make it bigger or smaller. I made the display 3D, then zoomed in close-up on a Minecraft world. I played with an Xbox controller, like I would on a big-screen TV.
From there, I aimed my cross-hairs back on the table, and used a voice-command to make the Minecraft level sprout up just like Microsoft's magical E3 press conference demo. Thousands of bricks grew up to form a bridge, trees, buildings. I saw pigs and crates. "Lightning strike," I said, and threw a bolt down on pigs to turn them into zombie pigmen. They attacked a little man controlled by someone else seated against the wall.
By pinching my fingers or snapping them, I could click or drag the world around forward and backward, or up and down to reveal levels and caverns underneath. I was guided how to create a pin using my voice, and dictate a voice message for a sign: "library here." I had fun looking around, peering in. I even created a second screen that floated on a wall next to me, that I could use while playing with the giant 3D model.
This is what mixed reality could be best: a way to throw magical models into the real world, even while using TV screens, or other displays. 3D tracking sometimes had hiccups, and I found controls to be a little rough at times. But that this worked, without wires, and using my voice as well, feels like a coup.
HoloLens' biggest limit: Field of view
As my colleagues at CNET have, the field of view on HoloLens is surprisingly narrow: it's like holding a cell-phone-size window up to your face and looking through it at the "holograms" that appear around you. That blindered display feels like a floating screen: when off, I still saw a vaguely purplish black. You could line up where you needed to keep the HoloLens display aimed at, and then magic 3D images appeared beyond it. Those 3D displays layered well, and looked pleasingly crisp. Images felt like magic Disney ghosts in the Haunted Mansion.
HoloLens, in this instance, is a clever creative prop versus a deep tool for gaming. It does work, without wires or any tethers... and surprisingly well, when you're looking where you need to look. That tetherless magic is no small feat, considering that the Oculus Rift and Project Morpheus use massive cables to link to a PC. But I what I missed the most was true peripheral immersion -- or, at least, the sense of it.
Virtual reality is so surprising because you can see things nearly from the corners of your eye: or at least, it feels that way. VR still has a "looking through a scuba mask" level of peripheral vision loss, but the sensation is like diving into another world. AR, via HoloLens, is in your real world -- except that in the real world, I have a huge range of vision. Looking through a smaller window while I can see everything else gives it an odd, Google Glass type of feel. I need to consciously line up my vision. Larger-scale things end up getting cut off, unless I move my head.
I was told something interesting during my demo by a Microsoft executive: how we perceive things in the front of our eyes is different than the periphery, which was an element considered by the HoloLens team. It's true, my periphery senses motion more than fine detail. But if there were a sense of a "hologram" object in my periphery that I could see more clearly by centering my gaze, that would help the illusion immensely. The cut-off, right now, being seeing and not seeing that's dictated by the HoloLens' magic rectangular window feels too harsh.
It's early days for HoloLens and augmented reality. Expanding the field of view will inevitably happen. After using HoloLens for the first time, it's surprising and novel but feels more like a piece of virtual theater, or a magic prop for a show, than a thing I'd want at home.
Virtual reality and augmented, mixed reality should eventually blend, be one device. But for now, you have to pick. Microsoft's mixed-reality road may cross paths with VR sometime soon. But it's a very different beast right now, and feels like it has its own unique possibilities -- and challenges.