Apple AR brings medical tricorder closer to reality
Butterfly Network's AR-assisted ultrasound can let doctors guide your own home scans.
- Nearly 20 years writing about tech, and over a decade reviewing wearable tech, VR, and AR products and apps
Butterfly Network's Teleguidance feature on an iPhone, with little arrows that direct using AR. A doctor can be somewhere else.
My wife fell down the stairs over a month ago and we didn't know if her foot was broken or not. We didn't go to the doctor to find out. We're not the only ones skipping in-person doctor visits. So when I saw Butterfly Network's Zoom demo of its handheld scanner -- and how a doctor could look for possible fractures from afar -- I thought: When can our house get one of these? It's like Star Trek's tricorder, but in the real world. That dream of a tricorder has been in motion in the tech world for years, but recent developments have brought it closer than ever.
Butterfly Network's handheld ultrasound device, the Butterfly iQ, has been available since late 2018. Connected to a phone or tablet, it can act like a little portable medical tool. But it's designated for "doctors or qualified practitioners only." I saw a demo last year at Apple's WWDC and I've thought about it ever since. What grabbed me was the device's promise of a future of remote patient assistance, using pop-up 3D instructions that could guide a patient to use the device properly, too.
Recent FDA changes allow medical imaging tools to be used remotely by doctors in emergency situations. This means Butterfly Network's tech could be used at home now, using remote AR-assistance software that taps into Apple's ARKit to render 3D directions as a person uses it.
I see the little hair trimmer-sized scanner point over someone's body, while in another window, I see the pulsing echoes of the ultrasound. I also see arrows and markers, which can be placed there on a patient's iPhone or iPad to remotely guide where to look, or to mark up areas of interest. Butterfly Networks' VP of Product, Matt de Jonge, thinks of it like a "magic mirror" that helps someone orient the scanner and follow a doctor's live guidance at the same time.
The Butterfly iQ's capabilities include looking at pregnancies, possible broken bones or even looking at lung conditions: According to Butterfly Network, doctors are currently using the tech to look remotely for signs of lung damage associated with COVID-19.
The TeleGuidance feature works on more recent iPhones and iPads (iPhone XR and later, fifth-gen iPad Mini and more recent Airs and Pros) that have the power to handle the AR and also room-scanning, machine-learning guidance needed to track the ultrasound device's position in relation to a human body. The handheld scanner also works with some Android phones , but not with the AR guidance yet. Butterfly Network's de Jonge sees more advanced phones with better computer vision and room scanning (the new iPad Pro's Lidar, or improving software efforts) as helping more down the road.
Butterfly Network is careful to make clear that right now, the FDA permission is temporary for the duration of the coronavirus emergency. Per the company, "We are working closely with FDA to bring safe, effective, remote-guided ultrasound imaging to the medical and patient community. We are grateful for the opportunity to advance clinical care during this challenging time."
The Expert View, where a doctor can send messages to guide the person using the ultrasound equipment and monitor results.
I've used AR headsets like the Microsoft HoloLens that have explored remote assistance and guidance via AR and Google Maps uses floating arrows to do the same. This is the first time I've seen this for an essential home need like telemedicine.
The ultrasound hardware is, to be clear, expensive: $2,000-plus subscription services, although the company says that doctors could prescribe them, or loan them. There's a possibility of insurance coverage. That's why I think it'll be some time before I get one of these in my own home.
The idea of AR assistance in general, though, is something that could keep being useful in all sorts of ways. A technician guiding you through home repair, or technical support. A contractor trying to understand your home and offer advice. Doctors. This type of home assistance isn't really here much yet, but it looks like a promising next step, maybe even after lockdown.
Correction, 8:49 a.m. PT: Clarifies that the Butterfly iQ can help doctors, specifically, look for signs of lung damage related to COVID-19.