X

I picked the wrong day to try this NYC Hololens AR art installation

Mel Chin’s virtual flooding of Times Square is fascinating, but reveals the drawbacks of technology.

Scott Stein Editor at Large
I started with CNET reviewing laptops in 2009. Now I explore wearable tech, VR/AR, tablets, gaming and future/emerging trends in our changing world. Other obsessions include magic, immersive theater, puzzles, board games, cooking, improv and the New York Jets. My background includes an MFA in theater which I apply to thinking about immersive experiences of the future.
Expertise VR and AR, gaming, metaverse technologies, wearable tech, tablets Credentials
  • Nearly 20 years writing about tech, and over a decade reviewing wearable tech, VR, and AR products and apps
Scott Stein
4 min read
Sarah Tew/CNET

It's 95 degrees Fahrenheit -- 35 Celsius -- or at least it feels like it. I'm standing in the middle of Times Square, looking at my phone at an augmented-reality fleet of boats floating above my head. My phone is hot. I can't hear through the earphones well. A crowd of family tourists nearly collides with me as I try to stand near the marker painted on the ground. Nearby, the Statue of Liberty dances on stilts: A real person, I mean, in a costume.

Watch this: Art meets HoloLens augmented reality in Times Square

Next, I'm wearing a Microsoft HoloLens on my head, seeing the floating boats semivisible in a haze of bright sunlight and my own sweat. I try to crane my neck at the holographic effect, as representatives from Microsoft try to keep me in the shade. There's a reason why: my demo suddenly ends midway, going dark. The HoloLens has overheated.

Times Square in the summer is an awful place to experiment with augmented reality

02-mel-chin-ar-unmoored-hololens-art-times-square

The massive animatronic sculpture, Wake, is worth checking out even without the AR experience.

Sarah Tew/CNET

Still, the reason for all of this is good-hearted: Mel Chin, an acclaimed Houston artist who's had a year-long series of works at the Queens Museum and throughout New York City called All Over The Place, is debuting two immersive works in the middle of Manhattan's most crowded tourist spot, near the TKTS booth.

Mel Chin's Augmented Reality Artwork shows NYC underwater in statement about climate change

See all photos

"Unmoored" and "Wake" are a two-part installation: Wake is an animatronic ship-hull-meets-whale-skeleton, a sort of physical wreck erected in a pedestrian plaza. Unmoored is an AR app designed to layer over the sculpture that is Wake, and create a six-minute audiovisual immersion to shake visitors out of their everyday lives and become more aware of how climate change could change places like Manhattan in the future.

21-mel-chin-ar-unmoored-hololens-art-times-square

A little hard to see the AR in the bright sun of the city.

Sarah Tew/CNET

How things got too hot

It's a noble effort, and as a free installation running through September 5, it's a great thing to check out if you have time. As a demonstration of AR in public spaces, however, it revealed a lot of the problems the technology still needs to work through.

It was soooo hot. I really wanted to focus on the illusion of decaying ships hovering over Times Square, the mysterious boat skeleton sculpture, and how the HoloLens handled broad daylight. But I couldn't, because I was dying from the heat. That's the funny part. Mel Chin's work is made to raise awareness of ideas around global warming and flooding spaces. But mission accomplished: New York's scorching ambient temperatures were already doing the job.

Augmented reality and bright sunlight don't mix. Mostly, it had to do with heat. Both my phone and the HoloLens didn't do well at all with high-performance stuff like AR in the super-intense summer conditions. ARKit-enabled apps cause iPhones to get a bit warm on any day, but in direct sun it became a big challenge. My phone didn't conk out, but the HoloLens did. (Microsoft reps said that, once the sun cleared the buildings a bit and we were in shade, it would far better).

Ironically, virtual reality and bright sunlight don't mix, either. I accidentally damaged an Oculus Go earlier this year by leaving it face up in the sun -- the lenses fried part of the display like an evil kid burning an ant with a magnifying glass.

Crowds aren't great for AR, either. Staging AR in outdoor places is, frankly, a bit of a mess. There were too many people in the crowded pen at Times Square during the premiere. It's not a bad model of the average day in Times Square, though. There were other issues: AR effects make it seem like ghostly things are floating in the regular universe. But consumer AR can't do occlusion well yet, which is the special effect of making virtual objects "hide" behind the real ones. If someone walks in front of AR, the optical illusion is ruined.  In Unmoored, the boats are cleverly positioned high up, away from interference with pedestrians. But still, I found people breaking the effect a bit. Also, I nearly collided with lots of tourists.

AR can't really sync the real and virtual worlds as well as I'd like. To make Unmoored work, I had to stand in one of a handful of labeled spots marked by letters, and the orientation of the letter had to match between my phone's on-screen guided overlay and the real thing. I learned why: I started my experience imperfectly synced, and the real-world ship-sculpture didn't overlap properly with the virtual one that appeared and slowly floated into the sky. Sync the markers wrong, and ships will look like they're flying into buildings. AR can recognize walls and objects in close range and position objects properly, but long-distance effects don't always work as well.

I could barely see anything! Augmented reality's ghostly types of projections can look washed out on a HoloLens in super-bright daylight. On a phone, really bright light makes it hard to see the phone screen properly. Either way, I wished I was in a more sheltered space.

A promising exhibit you should try on a cooler day

I don't want to ruin the experience for you, and odds are that on a cooler, more well-shaded day, this could all be a lot more fun. I love the potential of immersive art experiences that blend technology with the real world. But Mel Chin's Microsoft collaboration ended up showing that the challenges of making this work well are still pretty daunting. Future AR headsets have a ways to go before they're really ready for serious outdoor use.

Apple's new ARKit has multi-user support: It's also partnering with Adobe on a new AR file format 

Microsoft's next HoloLens may cost less, arrive in early 2019: The next-gen HoloLens is reportedly code-named Sydney.