Microsoft's AI-Powered Bing Google's ChatGPT Rival Hogwarts Legacy Review Ozempic vs. Obesity Best Super Bowl Ads 2023 Honda Accord Hybrid Review OnePlus 11 Phone Review Super Bowl: How to Watch
Want CNET to notify you of price drops and the latest stories?
No, thank you

Vocal Joystick controls PCs for those with hand injuries

University of Washington software project is aimed at, among others, Iraq war veterans who have returned with injuries preventing them from using computers.

An artist used a software program called Vocal Joystick to create this drawing of Mount Fuji. The software gives people with reduced or nonexistent use of their hands the ability to control a computer cursor by mouthing vowel sounds.
Daniel Terdiman/CNET News

SEATTLE--For many Iraq war veterans who have returned home with debilitating injuries that, for example, make it impossible to use their hands, doing anything on a computer is a hopeless task.

But a research project under way in the University of Washington's electrical engineering, linguistics and computer science departments could be the latest tool at such veterans' disposal, as well as for anyone who lacks the full use of their hands.

The project, known as the Vocal Joystick, is designed to allow someone to control a computer cursor using nothing more than their voice.

University of Washington graduate student Jon Malkin, who spoke at the Gnomedex conference here Saturday, described it is an extension of speech recognition technology.

It works by having a user train the Vocal Joystick software with his or her voice.

"We can do a lot with that," Malkin said. "Speech is a very complex signal."

Vocal Joystick is being developed by students and faculty in the electrical engineering, linguistics and computer science departments at the University of Washington. Users can move a cursor around by mouthing a series of vowel sounds that correspond to different directional movements. Daniel Terdiman/CNET News

Indeed, he explained that the software allows a user "four degrees of freedom," meaning that the cursor can be moved up, down, left and right, all with different vowel sounds.

For example, to move left, the user would sound out the "ee" tone in "feet." To go right, it would be the "aw" sound in "law." Hitting the equivalent of the mouse button would be the "u" sound in "but."

It might be tempting to think that only the most basic tasks are possible with this kind of software, but Malkin said that one artist who has used the Vocal Joystick had employed it to paint a picture of Mount Fuji that he then showed the audience.

The picture was very good, something that many people with full use of their hands would never have been able to create.

He said the artist had created the picture in just three hours.

One simple manifestation of the project is this game in which a person controls a fish that must swim around to eat other fish. Daniel Terdiman/CNET News

Malkin also demonstrated the software in real-time, showing how it is used in conjunction with a simple game where a player controls a fish swimming around trying to catch other fish.

He proceeded to sound out vowel after vowel, and sure enough, on-screen, his fish moved around dexterously, chomping up snack after snack. The Gnomedex crowd went wild.