When AR meets video chat, remote troubleshooting gets real
The Project Chalk app starts with video chat but lets you make notes on your screen that the recipient sees on an object itself through augmented reality.
Project Chalk creates a new way to interact on video chats.
Your dad doesn't know how to configure his router. Your grandma can't figure out how to program her TV remote. Your kid has questions about science homework while you're on a business trip. Or maybe you're the one who needs help.
We've all been there -- fielding requests for assistance when we aren't physically around.
Now, PTC's Vuforia augmented-reality software means you can at least show up virtually.
The company on Wednesday demonstrated a new Vuforia-based app called Project Chalk that lets you interact with others through video calling. The twist is that you can draw on your phone or tablet screen using your finger or a stylus to show someone where to plug in a cable or how to solve a math problem.
If you're the one giving assistance, what you see on your screen is what the other person is pointing a phone's camera at. You can create digital notes and drawings over an image. When your contact looks at the phone, the annotation show up as if it's stuck to the object itself, exactly where you made the scribble.
The service represents an easily understood and practical benefit of augmented reality , a hot trend that overlays digital images on the real world using special headsets or your own phone. Many of the early examples of popular AR include games like Pokemon Go or filters and lenses that go over your face on Instagram or Snapchat. But PTC's Vuforia is part of a new wave, including heavy hitters like Google, that want to make the technology more useful to you.
The idea of AR with Project Chalk is that the annotations are aware of your movement. No matter how your contact waves their phone or tablet, what you draw stays virtually fixed on the device you were annotating. If you draw a circle around a button on a TV remote, it will remain around that remote even if your grandma gets up and walks around. Your contact can even put the phone down, pick it up later and still see your notes (as long as you're still on the video chat).
"We allow two people to have interactions they couldn't normally have if they weren't in the same physical space," Jay Wright, president of PTC's Vuforia, said in an interview. CNET got an exclusive, early demo of the software a week before its unveiling at the Augmented World Expo, which runs Wednesday through Friday in Santa Clara, California.
"We think it can be used by everyone on the planet, from childhood to elderhood," Wright said.
As big as the iPhone
Augmented reality shouldn't be confused with virtual reality, which transports you into a different, digitally created world. By contrast, AR keeps you rooted in reality.
AR is expected to be a huge market. ABI Research predicts that the industry's revenue -- combining consumer and business software and hardware -- will total $96 billion in 2021, up from $1.4 billion this year.
An important factor in how big the market will get is content, said ABI Research analyst Eric Abbruzzese. "For AR on consumer devices, we haven't seen much yet," he said.
Even Apple CEO Tim Cook is excited about AR. Earlier this year, he compared the technology to the impact of the smartphone on the world's population.
"We don't have to think the iPhone is about a certain demographic, or country or vertical market: it's for everyone," Cook said during an interview with The Independent. "I think AR is that big, it's huge."
With Project Chalk, you can help your kids with their homework when you're not physically there.
Vuforia started as a project at mobile chip giant Qualcomm before being sold in late 2015 to PTC, an internet of things software maker based in Needham, Massachusetts. Vuforia software enables developers to create AR apps. More than 350,000 developers have registered to use Vuforia. They have built over 40,000 AR apps available today -- like Lego's Nexo Nights game or Mattel's View-Master Destinations app. There are another 45,000 apps currently in development.
Project Chalk is the first Vuforia-based app specifically for consumers, but PTC is opening up the project's capabilities to other developers, as well. This means an internet service provider, for instance, could offer a video chat app to troubleshoot your home broadband problems, or tech support could show you how to operate your cable box by sketching words and instructions that appear on your screen.
The app -- which doesn't yet have an official name -- will become available this summer to people who sign up for the early access program. The app is expected to hit the Apple App and Google Play stores this fall. There are plans to support Amazon's flavor of Android, as well as Windows.
The capabilities enabled by Project Chalk aren't completely new. Microsoft's Skype for Hololens is similar. But analysts say it's really the first app for consumers to use right on their phones and tablets.
"The type of functionality they're going to release is the start of something pretty significant," Gartner analyst Brian Blau said. "If that pathway they're making is successful, it really could change the nature in how we all interact with the physical world using our smartphone or digital technology."
Digital chalk
I stood in front of a fancy coffee machine that I had no idea how to operate. Luckily, J.J. Lechleiter, PTC's Vuforia senior director of product management, did know. He called me using the Project Chalk app to help me set the timer on the machine.
"Get in close on that control panel there," he said. I pointed a Samsung tablet at the box and watched as Lechleiter drew on the screen. A yellow circle appeared around a button on the front, as I heard him say, "This button here, you're going to push that until you get to the time you want." He then circled another button and another until I'd finally gotten the timer set. "Last one, select classic brew, and you're good to go," he said.
Behind the scenes, Vuforia builds a 3D model of your environment. (It doesn't save that information, so you don't worry about hackers stealing detailed images of your home or router.) The tech uses computer vision to recognize what you're looking at so those annotations can stick to whatever it is they're initially drawn on (that TV remote, for example). There's a video chat system on top of the technology that lets you interact with your contacts.
It's not aimed to be a full-blown FaceTime or Skype competitor. PTC envisions you using it for short periods of time -- for example, to quickly explain how to program a smart thermostat. PTC plans to offer a free version, as well as different price tiers based on the amount of time the video calling is used. Those tiers haven't been determined yet.
"It's for short-term use when you really need help," Wright said.
For now, all calls must be live. There's no recording/saving option, but that's something the company is looking at adding in the future. That would let businesses, like cable operators, record their conversations with customers or let your grandfather save your instructions to watch again later.
The person giving the support can take photos -- essentially screenshots -- during the chat and make annotations on them. This would be particularly helpful on a bigger item, like a car, where the person getting help is panning the camera. The person getting help doesn't see these photos but instead sees arrows showing they should move their camera in that direction to see more annotations.
The person helping also can remotely turn on the camera light on the other person's device, in case they need better lighting to see what they're doing.
Another possible future capability is object recognition. When you point the app at something like a router, it will know what you're looking at and can give you information about it. For now, Project Chalk relies on another person to help identify objects and annotate them.
"We really want this to feel as if one person is holding a piece of digital chalk," Wright said. "This is a new form of human communication where we're bringing people closer together."
Tech Enabled: CNET chronicles tech's role in providing new kinds of accessibility.
Batteries Not Included: The CNET team reminds us why tech is cool.