Hacking for autism: Apps to help everyone on the spectrum

SAN FRANCISCO -- For people with autism, communication with other people doesn't always come naturally.

Recent high school graduate Shubhankar Jain understands -- his younger brother Paras is autistic. Watching Paras, who Jain said is intelligent but has difficulty interacting with people emotionally, inspired Jain to build an app, Audeo, that utilizes natural language processing and visuals to help autistic people learn how to communicate better.

"It's hard for them to parse the content and figure out exactly what the emotion is," Jain said.

Jain -- who started his own nonprofit to bring awareness to people living with disabilities -- and several former classmates from Cupertino High School (located in the same city as Apple's headquarters) built the app as part of a hackathon this weekend, held at communications company Twilio's headquarters. Audeo allows people to visualize a conversation and connect with the concepts through images. The app lets users record conversations, break them down into visual cues and concepts, and then go back to the recordings later to remember what they learned.

It was just one of several apps produced; all the apps were focused on helping autistic people transition into a more independent life, whether that means going to college, interviewing for a job, finding a place to live, starting a romantic relationship, or simply walking into a shop and buying a cup of coffee from a cashier.

"There's nothing that really shows a teen with autism how to apply for a job, apply for a school, how to find housing, or ask a girl out," said Lauren Elder, assistant director of dissemination science for Autism Speaks, the nonprofit that organized the hackathon. Elder said much of the technology that already exists is usually geared toward children with autism, but there are not a lot of resources for adolescents are adults.

Autism Speaks, which focuses are increasing awareness and advocacy, has been holding hackathons to challenge developers to use technology, like natural language processing, Kinect, Google Glass and big data crunching, to solve some of the more common communication issues for those with autism, a developmental disability. People with autism generally have difficulty picking up social cues, like facial expressions or body language, so it can be hard for them to hold a conversation. They also often have repetitive behavior and are detail-orientated.

Mike Wuebben, Autism Speaks' director of digital strategy, said creating apps for teens and adults is increasingly important as autistic children grow up. One in 88 children is diagnosed with an autism spectrum disorder by age 8. The rate is even higher for male children; 1 in 50 is diagnosed with the disorder. Autism Speaks' hope is that it will eventually fold the apps created at hackathons into its own app, creating a single resource for people.

The other applications created over the course of the weekend included:

• MindFlower, a site that matches autistic people with employers who need detail-orientated tasks completed. Similar to Amazon's mechanical turk, an online marketplace of tasks, the tasks are outsourced, allowing lots of people to help complete a big job through lots of little tasks. MindFlower breaks down the tasks into manageable steps and also has quality control in the form of feedback.

• MyMonitorGlasswear, a program that uses Google Glass to help people learn about social cues. The app tracked eye and voice level and rate of speech and gives feedback on whether you're making eye contact with the person you're speaking with or if you're speaking too loud or high. It also tracked how far the user was to a crowded area or if someplace is too bright (crowds, loud noises and brightness can be triggers that can use someone to be over stimulated and frustrated).

• VIP (Virtual Interactive Practice), an Xbox Kinect application (and the winner of the hackathon) that uses a video game format to help people learn concepts like how to act or what to say in a social situation. It uses facial recognition and detects body movement and scores the user depending on how well they listen to directions. John Fairchild, who helped develop VIP, demonstrated the app by going through a mock job interview with a borg-like character. He said he developed the program using the Epic Games Unreal Engine.

• Snuffy, a digital voice assistant. Although Snuffy's creators introduced the app as voice assistance for kids, adults probably won't mind the cute cat cartoon performing Siri-like tasks. Snuffy is a "virtual buddy" or answers questions by offering instructions, options and visual aids. For example, if you ask Snuffy about what to wear before leaving the house, he will compute the weather conditions and suggest some clothes, like jeans and a jacket, and bring up images of the clothes. If the user wants to buy coffee, Snuffy will break down the social interaction -- approaching the cashier, ordering the coffee and paying for the transaction -- into specific steps. It also offers the option of performing the transaction for you by speaking, so users can just play the message for the cashier.

Update, 10:48 a.m. PT: Clarified the number of male children diagnosed with autism.