AI Is Changing Our Phones, and It's Just Getting Started

Today's smartphones can do many things. But the process of accomplishing most tasks is generally the same: unlock your phone and open an app. More often than not, getting things done requires bouncing between multiple apps. 

Planning a vacation, for example? You're probably switching between a messaging app and Airbnb. Sharing an image from a concert? That likely involves opening the default camera app to snap a photo, editing it in a different app and sharing it on social media in a third app.

But with generative AI, tech companies large and small see that changing. Generative AI refers to artificial intelligence models trained on large volumes of data that create content when you provide a prompt (think ChatGPT, Microsoft Copilot or Google Gemini). It's different from other forms of AI that have played a crucial role in our phones for years by subtly blurring the background in our portrait photos and recognizing our speech when we give a command to virtual assistants like Siri and Alexa. 

Now tech leaders, startups and analysts alike believe generative AI has the potential to impact mobile devices on a more foundational level rather than just doing heavy lifting behind the scenes. Samsung and Google are providing a sense of what that looks like through the Galaxy S24 and Pixel 8, respectively, each of which has software that can generate new content in images after you press the shutter button. We should hear soon from Apple about what its AI plans are for the iPhone. And newer tech companies like Rabbit, Humane and Brain.AI are looking beyond the smartphone as we know it. 

"We are moving towards an era of intelligent smartphones," said Nabila Popal, a research director with the International Data Corporation who tracks the mobile phone market. "So we're going from smart to intelligent."

New Samsung and Google phones provide a glimpse into generative AI

Samsung's Galaxy S24 and Google's Pixel 8 are the strongest examples of how generative AI is starting to appear in today's phones. AI was at the forefront of the South Korean tech giant's Unpacked event in January, arguably its biggest product launch of the year. 

"Artificial intelligence will bring about great change in the mobile industry and in the way we live," T.M. Roh, president and head of Samsung's mobile experience business, said on stage during the event. "We believe Samsung Galaxy will democratize this change."

The Galaxy S24's headlining addition was Galaxy AI, a suite of AI-powered features that span areas such as photo editing, language translation, note taking, texting and even Googling -- in other words, almost everything you probably use your phone for. Galaxy AI launched on the Galaxy S24 and recently began rolling out to the Galaxy S23 lineup, Galaxy Z Fold 5, Galaxy Z Flip 5 and Galaxy Tab S9 through a software update, underscoring just how critical Samsung sees it being to its mobile devices. 

Among the most intriguing Galaxy AI features is Live Translate, which enables Samsung to essentially act as your personal interpreter during a phone call by translating speech in real time. When I called a coworker who speaks Korean, all it took was the tap of a button to tell Samsung's translator to kick in and help navigate the conversation. 

Circle to Search is another highlight. As the name implies, it allows you to launch a Google search for almost anything on your phone's screen just by circling or scribbling on it. Samsung's Galaxy S24 family and Google's Pixel 8 series were the first phones to support Circle to Search, but it later came to the Pixel 7. The feature more recently expanded to the other aforementioned Samsung devices as well as the Pixel 6 family and Pixel 7A. 

Dave Burke, Google's vice president of engineering for Android, tells me Circle to Search was the most popular topic of interest among the company's partners at this year's Mobile World Congress, the annual tech convention where device makers gather to set the agenda for the year ahead in mobile tech.

While he couldn't elaborate on specific plans to bring Circle to Search to other Android phones, he said to expect updates this year.

"I think this year you'll see expansion for sure in where it's available," he said when discussing Circle to Search. 

Photo editing is another example of how generative AI is having an immediate impact on phones. The Galaxy S24 and Pixel 8 include tools for erasing unwanted objects and resizing or moving objects in images. What makes Google's Magic Editor and Samsung's Generative Edit feel different from existing basic editing tools is their ability to generate content to fill the leftover empty space in the photo or change elements like the color of the sky.

"It's creating things," Popal said. "That's something that wasn't being done before."

But making sophisticated photo editing tools more readily available also raises questions about authenticity online, an issue that's been in the spotlight in the last month after a doctored photo of Princess of Wales Kate Middleton sparked backlash. Middleton said on social media that she edited the image herself, and there's no indication generative AI was involved. 

Still, the episode highlights ongoing concerns about what's real and what's fake on the internet, which are likely to escalate as generative AI gets better at creating convincing images. Samsung adds a watermark and a note in the image's metadata to signal that it was modified using Generative Edit. But in my experience, that metadata note has been visible only in Samsung's Gallery app, not Google Drive or Google Photos, although the watermark is present when sharing the photo to different apps. Google says information in a photo's metadata would indicate whether it was altered with Magic Editor.  

Algorithms also aren't infallible. Generative AI models can hallucinate, or produce answers that seem convincing but aren't accurate. Even when they don't hallucinate, they sometimes generate biased responses that can be offensive or simply don't make sense. Google's Gemini AI model came under fire in March after it created images of people that were historically inaccurate, for example.

As more smartphones come equipped with processors that are capable of running AI models locally on the device rather than through the cloud, hallucination may become less of a problem, says Ben Bajarin, CEO and principal analyst for Creative Strategies. 

"It's a much more tightly controlled [and] much smaller language model," he said. "And so because of that, they can put those guardrails on a little bit easier."

Google's Gemini Nano model can run on certain smartphones, and the latest chips from Qualcomm and Google are designed to handle on-device AI processing. Google's more powerful large language models that currently require an internet connection could arrive on smartphones as early as next year, according to CNBC, showing just how fast the technology is moving.  

Burke says the company is receiving a lot of "inbound excitement and requests" about bringing its on-device AI models to other Android phones.

"We're getting a lot of demand and doing our best to make that happen," he said. 

But perhaps the biggest unknown involves Apple's plans for generative AI. The company has been notably quiet about the technology, aside from CEO Tim Cook teasing AI-related announcements for later this year during a recent earnings call. Bloomberg also reports that Apple's next iPhone update, likely to be called iOS 18, will come loaded with AI features. 

Apple may even partner with Google on some of those additions, according to reports from Bloomberg, The New York Times and The Wall Street Journal, potentially giving Google's models an even stronger foothold on our phones. (Burke said he was knowledgeable only of the company's plans for Android when asked whether Gemini would expand to non-Android devices.) 

If Apple is developing new AI tricks and tools for the iPhone, we'll likely get a peek at them on June 10, which is the first day of the company's annual Worldwide Developers Conference. 

Moving from smartphones to intelligent phones

Like many new features on our phones, most new AI-based smartphone tools available right now are designed for specific individual tasks, like editing a photo or translating a text message. The larger question is whether the technology could change our smartphones in more sweeping ways that move beyond the app-based operating systems we know today. 

Circle to Search hints at that type of future. Unlike the new tools for summarizing notes, translating conversations and editing photos, Circle to Search introduces a new way of using your phone. Instead of exiting Instagram to manually search for nearby restaurants that serve those appetizing dishes you just encountered in your feed, you can simply circle them and let Google do the rest. The idea came about in 2022 as the company realized that most searches originated while users were using other apps, Burke says.

He sees a lot of opportunity for AI to be more helpful when it comes to changing how we navigate our phones on a daily basis. 

"We're just literally at the gate on that," he said. "And so, you're going to see a lot more evolution actually in both Circle to Search and the Gemini app as it pertains to everyday usage patterns."

Google isn't alone; startups are already brimming with ideas about how AI could power software experiences that are more efficient than today's smartphone apps. Rabbit AI and Brain.AI, for example, are developing software that leverages AI to accomplish tasks rather than requiring you to open an app. 

One demo from Brain.AI, which I witnessed over a Zoom call in March and my colleague Katie Collins experienced in person at Mobile World Congress, involved online shopping. After finding the desired product, all it took was pressing and holding on that search result to pull up additional information, like unboxing videos. A task like that would currently require you to switch from the Amazon app to the YouTube app to manually search for related videos. You can see how that's just a bridge beyond what Circle to Search already offers today.

Jerry Yue, Brain.AI's founder and CEO, says the company's software predicts and generates the interface based on context and the user's intention. 

"It's kind of creating a custom app for you every step of the way," he said.

That's similar to the goal of Rabbit AI, which made a splash with its AI-powered R1 handheld gadget during the Consumer Electronics Show in January. Instead of using apps, you push a button and recite a request, letting AI models handle the rest. Unlike a traditional voice-powered assistant like Siri or Alexa, Rabbit says its system learns how to use software so it can accomplish certain tasks on your behalf. Rabbit CEO and cofounder Jesse Lyu likens it to handing your phone to a friend to order takeout rather than doing it yourself. 

Humane's approach is a bit different, although it similarly involves using AI in favor of apps. Humane's AI Pin is a miniature wearable computer that can answer questions, translate languages and even project images on your hand using lasers at times when you need a visual answer. Humane co-founder Bethany Bongiorno doesn't see the AI Pin replacing your phone, but rather living alongside it.  

"The idea is that you can create a more focused relationship with your other devices, very similar to when you got an iPhone, you didn't get rid of your laptop," Bongiorno said. 

Apps probably aren't going anywhere for the foreseeable future. The past 15 years have conditioned us to tap, swipe and scroll our way through apps to get things done, and disrupting that system would likely make us feel like we're starting over. What seems more likely is that the underlying software on our phones will gradually become smarter and perhaps more automated, potentially executing on the original vision for virtual assistants like Siri, the Google Assistant and Amazon's Alexa. 

How big a shift this will actually be depends on who you ask. But for those who are invested in upgrading the smartphone as we know it, like Brain.AI's Yue, apps will one day feel like a relic of the past. 

While Yue says he doesn't expect AI interfaces to replace apps completely, he likens apps to MS-DOS, the command line interface from the 1980s that was used on computers before Windows came out. 

"You get what you want when you want it," he said when describing the benefit of AI-powered interfaces over using apps. "It's not taking out apps. It's splitting apps into more granular pieces that come to the user when they want it." 

Editors' note: CNET used an AI engine to help create several dozen stories, which are labeled accordingly. For more, see our AI policy.  

Correction, 1:25 p.m.: This report has been updated to correct the Circle to Search rollout timeline.