Google Home's Assistant could one day know your mood

In an exclusive interview, a top Google exec says the AI could eventually recognize if you're frustrated or pick up conversations where you left off.

Richard Nieva Former senior reporter
Richard Nieva was a senior reporter for CNET News, focusing on Google and Yahoo. He previously worked for PandoDaily and Fortune Magazine, and his writing has appeared in The New York Times, on CNNMoney.com and on CJR.org.
Richard Nieva
9 min read
Ben Fox Rubin/CNET

From 550 feet in the air, Google's Scott Huffman peers down at the Las Vegas skyline. It's a sunny afternoon in early January and, at the moment, Google owns this town.

The search giant has come here for a marketing blitz at CES, the biggest tech showcase in the world. Its sole mission is trumpeting the Google Assistant, its digital concierge software for controlling your smart thermostat, getting your flight information and reading you headlines.  

Huffman, vice president of engineering for the Assistant, is standing with me in a big glass orb on the High Roller, Sin City's version of the London Eye observation wheel. We've just about reached the apex of the ride. Down below, you can see the words "Hey Google," the trigger phrase for the Assistant, plastered in gigantic letters on the Las Vegas Convention Center. The phrase is partly obscured by the corner of another building jutting into frame, so the sign looks like it just says "Hey Go."

It's a fitting vantage point because "go" seems to be the unwavering philosophy for Google when it comes to its Assistant. Since Google CEO Sundar Pichai released the software three years ago, the search giant has been hell-bent on catching up to Alexa, Amazon's own digital assistant. Alexa, which beat Google to market in 2014, is now, literally, a household name. It's also a product many considered the birthright of Google, whose raison d'etre is making the internet searchable and usable.

When it comes to gaining ground, Google still has work to do. Amazon's Echo devices, powered by Alexa, own almost 70 percent of the smart speaker market, according to the research firm eMarketer. Google Home devices, inhabited by the Assistant, own just shy of a quarter. But Google could take the crown from Amazon by 2023, predicts Canalys, another research firm.

"An assistant isn't useful if you don't have it," Huffman tells me. "So we've spent a lot of energy over the last couple of years really getting the Assistant out."

Ask most people what they use the Assistant or Alexa for, and they'll probably tell you voice assistants are great for playing songs, setting a kitchen timer or, if they're connected to a smart device, turning on a light. That's fine, but Google wants to beat Amazon by doing more than stuff like that. For the past year, Google's trajectory with the Assistant has been at the level of sci-fi lore, and it's only getting more aspirational from here on out.

Relying on its hardcore machine learning, natural language processing and artificial intelligence chops, it's introduced ambitious new features. To wit: With a project called Duplex, Google wants a freakishly human-sounding robot to schedule appointments on your behalf. And it wants the Assistant to translate conversations for you in real time with a new Interpreter mode that speaks 27 languages.

"Do we turn on a light? Yeah, that's table stakes," Huffman says as we hover above some of the world's most famous casinos. "But if AI is really our differentiator, what is the functionality that will take it to a level where it really changes people's lives?"

I've asked Huffman here because I was hoping the expansive setting might spark some introspective conversation. As if the great heights would spawn a moment like Mufasa and Simba on Pride Rock, talking about the Circle of Life. Or... something like that.


Scott Huffman, vice president of engineering for the Google Assistant

James Martin/CNET

Things are never that dramatic when you're talking to a media-trained tech exec. But it wasn't all just wishful thinking on my part. Huffman, a 14-year Google veteran with kind eyes and a bushy gray goatee, considers my questions about the future of privacy in the era of digital assistants. He says -- at first -- it might take legislation.

He's also game to talk about how the Google Assistant could evolve. It's illuminating insight into how Google is thinking about arguably its most important product.

In the next five years, Huffman suggests, the Assistant could achieve the basics of natural human conversation, which, from a computer science standpoint, are anything but basic. He says wake words like "Hey" or "OK" are "really weird." He wants the Assistant to understand your mood and tone, and detect if you're frustrated. He wants the software to remember an exact discussion you had with it yesterday, so that today you can pick up where you left off. 

I ask him about the vision for 10 years from now. Maybe, he muses, physical robots -- not just bots you can talk to, but robots that move and do stuff -- will become household products, and digital assistants could integrate with them.

As our ride on the observation wheel comes to an end, the cabin descends slowly back to earth. We pass a Las Vegas Monorail car docked on its track, which also has the words "Hey Google" slapped on its side. It's getting ready to leave the station.

Hey Go.

The Duplex moment

A few days before Pichai introduced the Assistant in May 2016 before a crowd of 7,000 developers at Google's annual I/O conference, I sat down with him in his office to hear the product pitch. The search giant was getting ready to announce Google Home, a smart home speaker to compete directly with the Amazon Echo. It was clear the Assistant would be seen as analogous to Alexa.  

But from the very beginning, Pichai insisted it was more than that. "It's Google asking users, 'Hi. How can I help?'" Pichai said at the time. "Think of it as building your own individual Google."

After some prodding, an annoyed Pichai eventually gave Amazon credit for igniting the market. "There are areas where we will be ahead, and there will be areas where someone points a way and we do it," he said.

In the last year, that ambition has become more apparent. In May, Pichai unveiled Duplex, a jaw-dropping, realistic-sounding AI that mimics human speech. The software uses verbal tics like "uh" and "um," and pauses while talking, as if thinking of what it's going to say next, even though its responses are preprogrammed. It's currently in limited public testing.

The point of Duplex is to enable the Assistant to make restaurant reservations and hair appointments for you. But almost immediately, industry watchers, AI ethicists and consumers worried about the software's ability to deceive the people it was talking to. Google later said it would build in disclosures so people would know they were talking to a robot.

It was a key moment for Google, Huffman says. "The strength of the reaction surprised me," he says. "It made it clear to us how important those societal questions are going forward."

The specter of legislation

One of those big questions: What does privacy look like in an age when digital assistants are getting smarter and smarter?

That's important as Silicon Valley faces more scrutiny over its privacy practices than ever before. Facebook has spent the last two years fending off crisis after crisis, from disinformation campaigns to massive data breaches. Last month, CEO Mark Zuckerberg had to defend the company's data-hungry advertising business model in an op-ed for The Wall Street Journal. Google has been under the microscope for its location data collection on Android phones, as well as what its stances on privacy could mean if were to re-enter an authoritarian country like China. Pichai was dragged in front of Congress in December to confront questions about the China project, called Dragonfly, and answer for Google's overall intake of people's personal information.

Put a device in people's living rooms with a microphone that's always listening for wake words -- that's where "Hey Google" comes in -- and things get even more complicated.

Google CEO Sundar Pichai Testifies Before House Judiciary Committee

Google CEO Sundar Pichai testified before Congress in December.

Alex Wong/Getty Images

"If you think about Google Home or Alexa, these are really the first devices ever that aren't personal devices," says Huffman, who has a doctorate in computer science. "They're really computing devices that live in a shared environment ... These things now live in this room with all of us, and we're all users. How does privacy work then?"  

Huffman points to the work Google has already done in setting the agenda around AI. Last June, Pichai released a set of AI ethical guidelines that govern how the company uses the technology. The guidelines came after employees protested against a contract Google signed with the Pentagon to help develop AI for the analysis of drone footage. Those guidelines include vows to never develop AI for weaponry and to only create technologies that are "socially beneficial." 

But self-regulation likely won't be the only check on the company.

"Honestly, I think it will also in the end probably take new legislation, as society figures out how these things fit in," Huffman says. "You look at things like the telephone; that's been around for a long time. There's a bunch of laws about how you could use a telephone, and what you can do."

For example, you can't wiretap a phone without a warrant, he says. "So that [rule] was for that generation of technology," says Huffman. "With AI, we're going to end up with society thinking through some of the rules of the road."

Pressed about what that legislation could look like, Huffman walks back his statement. "I don't know if it takes legislation or not. I'm not the right person to talk about that," he says. But somehow, it will be up to society to decide, he says.

Huffman won't speculate on the types of regulation, but Jen King, director of privacy at the Stanford Center for Internet and Society, has an idea of what legislation could look like. She's currently researching the types of data collected through smart speakers.

King says regulation could look similar to the restraints ushered in by the General Data Protection Regulation, or GDPR, a sweeping European law that went into effect in May 2018. It gives consumers more control over the personal information they hand over to tech companies. With regard to digital assistants, legislation could mean the government enforces deletion policies for data if consumers want it wiped. Or the law could require more specific consent on how the data is specifically used and to make sure it's not done so "in perpetuity."  

She says we should all pay attention to what defaults Google and other companies set on their devices in the future, to make sure people aren't unwittingly giving up personal information.

"Google, for most people, is the gateway to the internet. These assistants further fill that role," King says. "The company shapes your experience, and they're going do so in a way that benefits them."

'I get yelled at'

Of course, none of this matters if people decide they just don't need digital assistants.  

As Huffman and his team toil away developing features for the Google Assistant, Pichai is doing his part too -- even at the most granular level. Huffman says the CEO often reports bugs and inefficiencies with the software. For example, when he says "Hey Google" and the wrong device springs into action, he lets Huffman know about it. And when Pichai was trying to set up the software so the Assistant could understand the voices of his family members, he told Huffman the process was too complicated.

"I get yelled at sometimes, and rightly so," Huffman says with a laugh. "He's really pushing us."

Eager to prove that consumers are actually using their assistants, last month both Amazon and Google did something they rarely do: They released user figures.


Last month, Google unveiled an Interpreter mode for the Assistant, which is being piloted at the concierge desk at Caesars Palace.

James Martin/CNET

Amazon said more than 100 million Alexa devices have been sold. Not to be outdone, a few days later Google said the Assistant was close to hitting 1 billion devices. But neither figure tells the whole story. For example, the vast majority (Google won't say how much) of those billion Assistant devices are automatically included on Android phones because the software is preinstalled. Of course, Google also installs the Assistant on its own Pixel phones.

I ask Huffman when that figure will be dominated by devices that are not phones. He says he doesn't know, but Google is looking to two areas where the Assistant could live that could eventually add up -- cars and houses.

Google's battle plan for the smart home is well-documented. It wants to take on Amazon and its Echo devices in the market for smart home speakers. It also wants to make the Assistant work with as many smart TVs as possible, from manufacturers like Samsung, Sony and Hisense. But Google's push to get the Assistant into cars is less talked about. At CES, Google announced a few Assistant devices for automobiles, including a car phone adapter made by Anker Roav that plugs into a cigarette lighter.  

The idea is to spread the Assistant far and wide. It won't be easy. In laying out the odds, Huffman returns to the Vegas theme.

"The virtual assistants -- ours and everyone else's -- aren't quite there yet for most people, in terms of 'I can't live without that,'" he says. So Google has a lot of work to do. "No doubt it's a bet." ●

The Smartest Stuff: Innovators are thinking up new ways to make you, and the things around you, smarter.

Special Reports: CNET's in-depth features in one place.