Virtually alone: Why watching live events in VR feels so strange

I sat in the corner of an audience that I wasn't really in. I watched the back of a bald man's head as he snapped photos on the stage with his camera. I was high up, looking down on everyone as a virtual screen appeared in this virtual audience, with a countdown to the Democratic debate that was about to begin.

I couldn't talk to anyone, because these people couldn't see me. I couldn't tweet, because I couldn't see my phone. I was a ghost.

This was me in VR, trying to engage in a live event I cared about: the first Democratic debate being streamed in VR by CNN (via an app made by NextVR) onto a Samsung Gear VR headset with a Samsung Galaxy S6 phone. And it's also, largely, the state of what it means to be part of a virtual live event: it's an idea that's only halfway there. Watching the debate wasn't a great experience.

Virtual reality is in its early days. Right now, you can either use a PC or a phone strapped into a headset (like the Gear VR you see above). There are games and apps that show 3D worlds you can look around in. Or, alternatively, panoramic videos: 360-degree landscapes. And some of them, like the last Presidential debate, are being streamed live...to watch with a VR headset on. Which I did, for as long as I could bear.

After an hour or so, I stopped. I've been emotionally engaged in virtual reality before, swept up by its immersive magic. I felt transported to other worlds. So why did the streaming debate fail me? Because it was close, but not close enough. It distanced me, instead of pulling me in. It made lose focus on what was being said. It made me less aware of what others were thinking. It was the opposite of being present.

As I sat on the sofa next to my wife with a phone strapped to my face, I realized I was missing too much. And if VR is to be a true telepresence tool of the future, it's a challenge that needs to be fixed.

So close, yet so far

The type of panoramic 3D video captured by the cameras CNN used has already existed to capture concert footage, scenic flyovers and sports events, and its biggest advantage is a sense of "being there" in a space. But it loses a lot in the process: crisp, sharp visuals, and focus. In something as directed as a debate, it's a bad fit.

As the debate started, I was suddenly thrust onstage, looking at five podiums spread out in the middle distance. Anderson Cooper stood off to my left. A giant CNN logo, in gleaming red and black, spread across the floor. The audio quality was fine. The sense of depth worked. But the visual quality of VR is far lower than anything you'd get on a TV. There's a "screen-door" effect that happens to video when you look at a screen through giant magnifying lenses in front of your face, even if it is pixel-packed: it's the opposite effect of what HD and Ultra HD TVs can show in the fine details. Here, the details are lost. And there's an additional drop-off because of varying video streaming quality and the knit-together stitched nature of panoramic video.

Details disappear, and melt into a soft blur.

There's magic in suddenly being there, but it's almost like teleporting without your reading glasses. Until video resolution in VR improves, the shots need to be closer, more directed. I just couldn't see.

Maybe this is like the dawn of television, and VR techniques just need to be refined, shots invented. I just want an experience that's better than my TV, not worse. On my TV, I saw everyone's faces: clear, crisp. I finally saw the candidates again.

And I felt far removed in other ways: unlike those audience members around me, I had no way of really feeling present or involved. I ended up feeling more removed from the moment, distanced both literally and figuratively.

I paid attention to the wrong things

The joy of freely looking around a live-event shot for VR in panoramic mode means you can, for instance, see what one dancer behind you is doing while another dances in front of you, or examine different people in an orchestra. But when there's less to pay attention to -- like five people on a stage talking -- you're bound to start getting distracted. I could see five avatars in suits, faces little more than mush, off in the middle distance. I had to rely on voices to identify anyone. So I started to explore around me.

I stared at the floor. The textures of the shiny floor, the lights gleaming off it, were captivating. I stopped listening to the debate. Being in the room was cool, but it became the most interesting part of the debate...not the debate itself.

Virtual reality has that effect for me: I feel present, but obsess over textures and surfaces. The coolest part of the Oculus Cinema app isn't the movie on screen, but the hyper-real seats and the reflective glow of the movie against the virtual theater walls.

Also, small details distracted me. A cameraman in a dark outfit climbed behind the CNN logo at one point, a stealthy ninja sneaking to prep for the next shot. I watched him operating for a while, turning my head away from the debate. At another moment, as the camera position switched back to one that showed the audience, I stared at the people out there. Their movements and reactions as the candidates talked were more captivating than staring at the fuzzy dolls at the podiums.

Inconsistencies start to become the main attraction. 3D effect-to-distance-perceived ratio (if I should call it that?) also seemed off. As the camera view changed to one closer to Anderson Cooper, he seemed like a Barbie-sized doll standing near my face, while all the candidates appeared to be living toys. It was like watching the debate as a tiny 3D diorama.

Too long

Here's another problem: VR works best in bite-sized chunks. About 5 minutes, maybe. When a phone is strapped to your face, eye fatigue becomes an issue pretty quickly.

The debate ran for hours. There's no way anyone should, or could, watch the whole thing in virtual reality. I could only handle minutes at a time. The eyepieces occasionally fogged up, my face got sweaty. My eyes needed a break. My ears hurt from the straps. I lifted the goggles over my head from time to time, for a break. And to tweet, or try to tweet. I often wondered who else would tolerate this.

You can't do anything else, and you're alone

This is the biggest problem of all. In VR as it currently stands -- especially for live video streams -- you're little more than a hovering ghost. You see what the camera sees. No one else can see you. And you're completely alone. You can't use your hands to do things, and you can't look at anything else. The experience is pure isolation.

I wanted to tweet, do interact, to comment. But in VR (at least, on the Samsung Gear VR right now), I have no hands. All I can do is watch.

To tweet, I had to stretch my VR goggles over my head and pick up my phone. The stream stopped, and I had to restart again. Meanwhile, I realized, I was missing the New York Mets playoff game that was underway. Channel-flipping, and second-screen life: I've gotten so used to these for real-time events that their absence feels impossible to deal with. I tried sticking an earbud in my left ear attached to an iPad streaming the Mets game while the Gear VR broadcast the debate over speakerphone. Eventually, I grew sweaty, and the VR stream froze on Bernie Sanders mid-grimace, and I decided to just turn on the TV instead.

I looked at the Twitter stream on my phone for commentary. Suddenly I was connected again.

To be engaged, either by apps or with people around you, is part of having a live experience. Virtual reality strips these away right now, in exchange for a type of non-interactive telepresence.

VR right now: Better for empathy than interaction, but that will change

I've cried in virtual reality, on this same Samsung Gear VR headset, under very different circumstances. That documentary experience, also shot in panoramic 3D video, worked so much better because it was something I was meant to witness, not interact with. I was meant to be a silent observer, to absorb what was shown to me, to take it in and immerse myself.

And virtual reality works best that way. But for hot, living things like debates, or sporting events, it's a lot more complicated. I like to watch and to comment along -- most of us do, on Twitter and social media. Ironically, the very company that bought the technology that makes this VR headset, Facebook, is what I feel cut off from.

That connectivity will come someday, no doubt. Facebook's banking on it. But for now, it's not here. And it needs to be. Places where friends can appear. Avatars of others watching, who can see me too. Apps and notifications and other virtual screens inside this space, that I can summon or shut off.

And the use of your hands. Oculus Touch is one of several inputs on the horizon trying to make sure we can start doing more things in virtual worlds, instead of just watching. And while passive technology like streaming video won't allow for interaction like that yet, the whole magical dream of telepresence via virtual reality is that you can do things when you get where you're trying to be.

Bottom line is you didn't miss anything if you didn't catch the debate in virtual reality. But it illustrates the challenges that VR will be facing for a long time to come: how to make a personal virtual experience feel like telepresence, and how that can make meaningful interactions and viewership. I'm too much of a two-screen person to live with goggles over my head for major events, in fact most of us are. VR needs to figure out how to handle that challenge, sooner rather than later, perhaps by becoming all of our screens at once.