A quadriplegic's challenge: Make me a touch-free phone
The Sesame Enable app uses a phone's front-facing camera and voice commands to give the physically disabled a way to use the device.
This is part of CNET's "Tech Enabled" series about the role technology plays in helping the disability community.
Giora Livne missed out on the smartphone revolution.
In 2007, Apple CEO Steve Jobs strutted onto a stage and unveiled the iPhone, which would dramatically change how everyone interacts with mobile devices.
Well, not everyone. That same year, Livne, a former electrical engineer and Israeli navy commander, suffered a spinal cord injury that left him a quadriplegic.
As touchscreens went on to dominate every device imaginable, Livne watched the revolution from the sidelines. Unable to move his arms and legs, Livne couldn't use the phones. But after watching developer Oded Ben Dov show off a game controlled solely through head movements, he saw his chance to get in.
Livne tracked down Ben Dov and posed a challenge: "I can't move my hands or legs, could you make me a smartphone I could use?"
From there sprung a partnership that resulted in Sesame Enable, the company behind an app that uses voice commands and the phone's front-facing camera to let a user control a cursor on the device with head movements.
Though there are other ways for quadriplegics to use their phones, including voice activation or even a mouth stylus, Sesame Enable offers more-complete control. The technology opens the door for people with physical disabilities like Livne -- 5.6 million people in the US alone deal with paralysis -- to live their work and personal lives on a mobile device without ever lifting a finger.
Since Sesame Enable came out, in May 2015, Livne has been able to independently use his phone to call his wife (and order her flowers), as well as control his television, music, lights and the temperature of his home. It's a new freedom that he and hundreds of people worldwide have experienced.
Open Sesame works by scanning your head and letting you use head movements to control a cursor on the screen.
Sesame Enable started out as both hardware and software, with a custom program hard wired into a Nexus 5 phone. Earlier this month, the company released an app for compatible devices running on Android 7.0, meaning increased accessibility was just one download away.
"It was a clear problem that some people absolutely could not use their smartphones," Ben Dov said. "This was over five years after smartphones revolutionized our world, which was absurd."
Throughout it all, Livne, who is the co-founder and vice president of accessibility at Sesame Enable, served as the primary tester, giving notes and feedback on how the technology should be tweaked.
Open, Sesame
Sesame has helped Livne and many others take control of their lives again, even letting some go back to their jobs. So I decided to take a whirl with the program.
Using a Nexus 5X loaded with the app, I interviewed Livne over Skype using only Sesame, and I took notes along the way. Sesame uses the front-facing camera to scan your head. After 3 to 6 seconds of scanning, a cursor pops up on your screen, controlled by your head movements. You can click and swipe by holding your head -- and cursor -- still for a few seconds, until a menu appears. The default timing is 3 seconds, but you can adjust it; I changed mine to 1.5 seconds.
Giora Livne, using Open Sesame to control his phone.
The experience wasn't ideal. There were plenty of typos, and at times the camera lost track of my face and needed to sync up again. It was also uncomfortable to keep my neck straight for so long.
Still, the experience showed me what was possible with Sesame.
For Livne, it meant being independent and communicating with the world from anywhere.
"I don't need to call my caregiver all the time," he said.
Many others realized the potential to work was possible again, like Roni Moran, who ran a design firm in Israel before he passed away from amyotrophic lateral sclerosis in August.
"Although I'm disabled, I run a business, and the phone helps me do that," Moran said in a Google Israel video released in December 2015. "The pictures and emails too."
Others have seen Sesame as a way to work for social change. Alex Fridman, for instance, uses it to run the nonprofit organization "Disabled, Not Half a Person."
He's fighting for disability compensation in Israel, which is half the minimum wage and hasn't been raised in 15 years. Because of Sesame, Fridman, who has spinal muscle atrophy, has been able to lead the push for fair compensation without asking for help from a caregiver every few minutes.
"When you run a campaign this big, you obviously have to use the phone a lot to make and receive calls, and send text messages and emails," Fridman said in an email Ben Dov translated. "This is where Sesame's amazing technology comes into the picture. It enables a person with a severe disability, such as myself, to do all this."
Sesame Enable initially sold its app bundled with a customized Nexus 5 for $1,200.
From hardware to software
Sesame's initial plan to sell a customized Nexus 5 was a noble but limited effort. Not everyone can afford a $1,200 phone, after all.
But Google's latest update to its Android application programming interface was a big deal. It opened up the opportunity for Sesame Enable to run its face-tracking software as a simple app.
"I remember the exact email I got from Google," Ben Dov said. "I thought, 'this changes everything.'"
All of a sudden, Open Sesame was something anyone with an Android 7.0 device could download. (That's still a relatively small number, given that a vast majority of phones are stuck on an older version of Android.) In the future, the app will come as a subscription, with the first month free. The company hasn't figured out the exact pricing.
The app can't do everything yet. A common request: add the ability to take selfies and photos while using Sesame. Right now that's impossible because the app already uses the front-facing camera to track head movements.
Ben Dov and Livne are working on adding features like a joystick to control the cursor, which Livne said would help with neck movements and improve accuracy.
For Ben Dov it's been rewarding to watch people happily join the fun on phones, playing games like Angry Birds and Candy Crush for the first time with friends.
"What I learned over and over again regarding our users is that their needs and wants are the same as everyone else's," Ben Dov said.