Apple iPhone Spatial Video Looks Amazing on Vision Pro
I tried the new iPhone 15 Pro camera feature on Apple's upcoming AR-VR headset. The 3D is undeniably vivid.
- Nearly 20 years writing about tech, and over a decade reviewing wearable tech, VR, and AR products and apps
Apple's iPhone-recorded 3D spatial videos can only be played back on next year's Vision Pro headset. But they do look great.
I'm looking at a plate of sushi hovering in front of me in 3D. The chef finishes off toppings on yellowtail rolls and tuna, talking to me as she works. It looks vivid. It looks real. The amazing part is that I just shot this video myself, moments earlier, on an iPhone 15 Pro. And now it's a VR experience I'm watching in beautiful 3D on Apple's Vision Pro headset.
I swipe with my fingers and watch other ghostly videos Apple provided. Families in a home, walking through grass. Cuddling together. All in lifelike 3D. I feel like I'm peeking in on their lives, which is weird and intimate. But the vividness is undeniable.
I'm experiencing Apple's new Spatial Video iPhone camera feature, now available in Apple's new iOS 17.2 update. It allows you to record 3D videos, but to use it you'll need Apple's best phone, the $1,000 iPhone 15 Pro. And to watch the videos in 3D you'll need the $3,500 Vision Pro headset, which launches early next year.
The videos look great and the 3D is compellingly realistic. They're also easy to record, and can save as videos that will play back in 2D in a normal video format. But ultimately this feature is made for a 2024 product that, at $3,500, it's safe to say most people won't buy anytime soon. Still, the experience is impressive.
Vision Pro impressed me once again
I first used the Vision Pro at Apple's WWDC conference in June, and once again I'm reminded of its extraordinary display capabilities. Photos and video look fantastic on Apple's headset. I'd even venture to say the Vision Pro beats any display I have anywhere in my home right now. Yes, I'd want to look at videos and photos -- 3D and otherwise -- if I had a Vision Pro.
My second dive into the Vision Pro, complete with fitted prescription lenses that matched my needs, reminds me how much smaller the headset was than I remembered. And also, how effortless the interface is.
Setting up eye tracking was a rapid process; I looked at dots around a circle and tapped them. I also tried a new set of zoom gestures, pinching my fingers and stretching them to expand photos. The gesture works with eye tracking, so wherever I looked, the image expanded. It felt like telepathy. I got to look through a few test photos in the Photos app, and play back photos in Memories. After having used a Quest 3 recently, Apple's passthrough cameras and display resolution are on another planet.
Panoramic photos were a surprise. They opened up and wrapped around me, felt like windows into other places -- almost 3D, in fact. And spatial videos look nice, too. Their 3D, almost ghostly playback quality feels like it's aiming for an immersive memory more than a straight-up video playback. But I found some limits even in my brief demo of the experience.
Unfortunately, there aren't any captures of what I saw or recorded here: the photos are all ones Apple provided, so you'll have to just follow along and read about my experience.
The side-by-side cameras record 3D video at 1,080p and 30fps, combining the info in a single file that can play in 2D everywhere else if you don't have a Vision Pro next year.
Spatial videos on iPhone 15 Pro: How they work
Spatial videos need to be recorded by holding the phone horizontally. The iPhone 15 Pro uses its main and ultrawide cameras side by side in landscape mode to create the 3D video. It records two 1,080p, 30fps videos at once, resolving the lens and distance differences with computational photography. The file is saved in HEVC format, similar to other Apple video files, with a storage size of about 130MB a minute. That's not too bad, likely because the files are limited to that relatively low 1,080p and 30fps resolution and frame rate.
Clips can be shared via Messages or AirDrop, but there could be cases where files are compressed down in other apps, losing the 3D data. Still, the video clips should be usable anywhere as an everyday video file, which is great news for anyone who wondered (myself included) whether you'd have to choose a "special" 3D video format.
You do have to toggle spatial video mode on, either inside Apple's Camera app settings or with a Vision Pro-mask-shaped toggle in the Camera app itself. That limits your video resolution and frame rate, and also means you have to shoot in landscape. Sorry, no 3D vertical videos.
The camera app makes recommendations on turning the camera sideways, and staying a certain distance from a subject. I was told to stay within 3 to 8 feet of what I was shooting for a good spatial video, but when I shot my test recording of someone making sushi at a table I got up closer and it looked perfectly fine. I also recorded in a well-lit room, but apparently the spatial video recording mode prevents adjustments on brightness and contrast, which means low-light recording may end up grainier than normal videos.
Apple has built in the spatial video recording toggle to the basic Camera app, but you need an iPhone 15 Pro to record in 3D.
How will spatial video evolve or be edited?
There's no API for Apple's 3D HEVC video format, which means the files are not designed so that third-party apps can recognize them yet. But it's possible app developers will figure out some sort of a solution. Also, it's disappointing that there's no other way to view the videos in 3D than on a Vision Pro.
The stereo 3D recording theoretically sounds like a process that could be adapted for other VR headsets like the $500 Meta Quest 3. It's straightforward stereo video, and doesn't use lidar or depth sensing for any sort of full-movement spatial capture. Unfortunately that's not currently in Apple's plans.
There's no recording limit on the videos, so you could theoretically make a spatial video 3D feature-length film. But the clips will only be able to be trimmed, either in the iPhone or Vision Pro. Final Cut Pro, Apple's video editing software, will get spatial video editing support, but not until sometime next year. Meanwhile if you do end up trying to edit these video files in a Mac or iOS video editor, they'll end up being converted over into 2D-only files.
It's also a tiny bit disappointing that the videos can't be recorded in 4K resolution. Apple's Vision Pro headset has astounding quality and resolution, which became clear once again as I looked at iPhone photos in the headset and zoomed in on them, or viewed panoramic photos in a wraparound mode that made it feel like I was in a vivid immersive re-creation of a location, similar to a 360-degree photo. The spatial videos look really nice, but I felt the desire to see them in more fluid 60fps, 4K or both. Maybe someday.
Memories for a 3D future
Apple often seems like it's trying to become not just a lifestyle, but a memory company. Automated photo collections are already called "Memories." And that's how 3D spatial videos feel on Vision Pro: hazily bordered, like digital recollections in some Minority Report or Blade Runner future. It reminded me, too, of some of the immersive 3D realism I had using Google's real-time 3D light field conversations in Project Starline, but in this case I was just watching videos.
Apple's choice to frame the spatial videos in a fuzzy sort of frame makes it look more like the videos are almost holographically being cast into the room, and the edges sort of dissolve a bit. But I'd love to see a way for the videos to have a standard frame around them, too.
Apple's spatial video format shows up in a new category of its own on the Photos app, and won't show up in Memories yet. Maybe this new spatial video tab in Photos becomes the go-to repository for new 3D experiences. Right now, Apple isn't allowing 3D spatial photos on the iPhone 15 Pro, just videos, although the Vision Pro can capture photos itself. But Apple's clearly aiming for everyone to hopefully start recording in the format before the headset arrives, building a library of capable videos.
That means you'll have to remember to toggle spatial video on, and care to use it. Future Vision Pro owners will have that feature toggled on by default on the iPhone 15 Pro, but that's a tiny portion of iPhone owners. I'll be turning it on, because I want to see what my videos will look like next year on a Vision Pro. But unless you're planning on buying a Vision Pro yourself, there's little reason to record spatial video now, even though it's possible, someday way down the road, you might regret not shooting that birthday party footage in 3D. I'm already thinking about how I shot video footage at my niece's bat mitzvah last week, and what it would have been like if I could see it in 3D, too. That new spatial video camera toggle is already playing on my future FOMO.
