Inside Pixel 4's camera: Google improves portrait mode with help from Italian art
Other new camera features include better zoom, astrophotography and dual photo exposure controls. CNET has an exclusive look.
Stephen Shanklandprincipal writer
Stephen Shankland has been a reporter at CNET since 1998 and writes about processors, digital photography, AI, quantum computing, computer science, materials science, supercomputers, drones, browsers, 3D printing, USB, and new computing technology in general. He has a soft spot in his heart for standards groups and I/O interfaces. His first big scoop was about radioactive cat poop.
Expertiseprocessors, semiconductors, web browsers, quantum computing, supercomputers, AI, 3D printing, drones, computer science, physics, programming, materials science, USB, UWB, Android, digital photography, scienceCredentials
I've been covering the technology industry for 24 years and was a science writer for five years before that. I've got deep expertise in microprocessors, digital photography, computer hardware and software, internet standards, web technology, and other dee
Over the last three years, Google's Pixel phones have earned a well-deserved reputation for photographic strength. With the Pixel 4 and 4 XL, the company is flexing new camera hardware and software muscles.
The new flagship Android smartphone, which the search giant unveiled Tuesday, gets a second 12-megapixel camera, a key component in an overhauled portrait mode that focuses your attention on the subject by artificially blurring the background. The new portrait mode works more accurately and now handles more subjects and more compositional styles -- and Google is taking some new styling cues from Italian Renaissance painter Titian, too.
The additional camera, a feature Google itself leaked, is just one of the photography advances in the Pixel 4. Many of the others stem from the company's prowess in computational photography technology, including better zooming, live-view HDR+ for fine-tuning your shots and the extension of Night Sight to astrophotography.
Improving photography is something Google can do on its own, and photography is important. We're taking more and more photos as we record our lives and share moments with friends. No wonder Google employs a handful of full-time professional photographers to evaluate its products. So I sat down with the Pixel 4's camera leaders -- Google distinguished engineer Marc Levoy and Pixel camera product manager Isaac Reynolds -- to learn how the phone takes advantage of all the new technology.
Levoy himself revealed the computational photography features at the Pixel 4 launch event, even sharing some of the math behind the technology. "It's not mad science, it's just simple physics," he said in a bit of a jab at Apple's description of its own iPhone 11 computational photography tricks.
The Pixel 4's main camera has a 12-megapixel sensor with a f1.7 aperture lens, while the telephoto camera has a 16-megapixel sensor with an f2.4 aperture lens. The telephoto camera only produces 12-megapixel photos taken from the central portion of the sensor, though. Using a tighter crop from only the central pixels makes for a bit more zoom reach and sidesteps the greater processing burden required to handle 16 megapixels. Google is using Sony-manufactured sensors, Levoy said.
Two ways to see three dimensions
To distinguish a close subject from a distant background, the Pixel 4's portrait mode sees in 3D that borrows from our own stereoscopic vision. Humans reconstruct spatial information by comparing the different views from our two eyes.
"You get to use the best of each. When one is weak, the other one kicks in," Reynolds said.
Those two gaps are oriented perpendicularly, too, which means one method can judge up-down differences while the other judges left-right differences. That should improve 3D accuracy, especially with things like fences with lots of vertical lines.
Levoy, sitting at Google's Mountain View, California, headquarters, flipped through photos on his MacBook Pro to show results. In one shot, a motorcycle in its full mechanical glory spans the full width of a shot. In another, a man stands far enough from the camera that you can see him head to toe. The smoothly blurred background in both shots would have been impossible with the Pixel 3 portrait mode.
Google wants you to think of the Pixel 4's dual cameras as a single unit with a traditional camera's continuous zoom flexibility. The telephoto focal length is 1.85X longer than the main camera, but the Pixel 4 will digitally zoom up to 3X with the same quality as optical zoom.
That's because of Google's technology called Super Res Zoom that cleverly transforms shaky hands from a problem into an asset. Small wobbles let the camera collect more detailed scene data so the phone can magnify the photo better.
"I regularly use it up to 4X, 5X or 6X and don't even think about it," Levoy said.
The iPhone 11 has an ultrawide camera that the Pixel 4 lacks. But Levoy said he'd rather zoom in than zoom out. "Wide angle can be fun, but we think telephoto is more important," he said at the Pixel 4 launch.
HDR+ view as you compose photos
HDR+ is Google's high dynamic range technology to capture details in both bright and dark areas. It works by blending up to nine heavily underexposed shots taken in rapid succession into a single photo -- a computationally intense process that until now took place only after the photo was taken. The Pixel 4, however, applies HDR+ to the scene you see as you're composing a photo.
That gives you a better idea of what you'll get so you don't need to worry about tapping on the screen to set exposure, Levoy said.
One nice feature Google offers for photo enthusiasts is the ability to shoot photos not just with the traditional JPEG but also in the more flexible DNG "raw" format -- specifically its computational raw photos that the Pixel phones construct from the wide dynamic range of Google's HDR Plus files. Unfortunately, though, when using Super Res Zoom, you'll get a cropped version of that raw file.
More Titian, less Caravaggio
Google makes aesthetic choices about HDR+ styling, taking inspiration from Italian painters who rose to fame hundreds of years ago.
"We can learn from art," Levoy said in a Google video about the Pixel 4 launch. "I've always been a fan of Caravaggio," whose paintings have strong dark-light contrast and deep shadows. "That has always been the signature look of HDR+."
With the Pixel 4, though, Google has steered closer to the lighter shadows of Titian. "We've moved a little bit toward that this year, not crushing the shadows quite as much," Levoy said.
Separate camera controls for bright and dark
Live HDR+ lets Google offer better camera controls. Instead of just a single exposure slider to brighten or darken the photo, the Pixel 4 offers separate sliders for bright and dark regions.
That means you can show a shadowed face in the foreground without worrying you'll wash out the sky behind. Or you can show details both on a white wedding dress and a dark tuxedo.
The dual-control approach is unique, and not just among smartphones, Levoy says. "There's no camera that's got live control over two variables of exposure like that," he said.
Shoot the stars with astrophotography
In 2018, Google extended HDR+ with Night Sight, a path-breaking ability to shoot in dim restaurants and on urban streets by night. On a clear night, the Pixel 4 can go a step further with a special astrophotography mode for stars.
The phone takes 16 quarter-minute shots for a 4-minute total exposure time, reduces sensor noise, then marries the images together into one shot.
AI color correction
Digital cameras try to compensate for color casts like blue shade, yellow streetlights and orange candle light that can mar photos. The Pixel 4 now makes this adjustment, called white balance, based in part on AI software trained on countless real-world photos.
Levoy showed me an example where it makes a difference, a photo of a woman whose face had natural skin tones even though she stood in a richly blue ice cave.
The character of out-of-focus regions is called bokeh in photography circles, and with the Pixel 4 it's improved to be more like what an SLR would produce. That's because more of the portrait mode calculations happen with raw image data for better calculations. Point sources of light now produce white discs in the bokeh, not gray, for example.
All these features represent a massive investment in computational photography -- one Apple is mirroring with its own Night Mode, Smart HDR and Deep Fusion. Google has to "run faster and breathe deeper in order to stay ahead," Levoy acknowledged.
But Apple also brings more attention to Google's work. "If Apple follows us, that's a form of flattery."
Watch this: Pixel 4 and 4 XL hands-on: Dual rear cameras, radar face unlock
Originally published Oct. 15. Updates, Oct. 15, 16 and 17: Adds detail about Google's new computational photography abilities, Pixel 4 camera details and further comments from Levoy.
Pixel 4 and 4 XL: Industrially chic in glowing orange