How DxO Labs tests hot cameras like Canon's latest SLR

BOULOGNE BILLANCOURT, France -- So you think you're a camera expert? Well, when was the last time you evaluated a camera by spending days analyzing hundreds of test photos?

That's what DxO Labs does, over and over, publishing results so far for 185 cameras on its ="http: www.dxomark.com="" "="">DxOMark Web site. To see what lies behind those performance scores, CNET visited the company as it put one hotly anticipated camera through its paces: Canon's hot new video-capable SLR, the EOS 5D Mark III.

The results might not surprise close watchers of the camera business: Canon's score of 81 means the 5D Mark III, while respectable, utterly failed to dethrone the highly regarded Nikon D800, with a top DxOMark score of 95, when it comes to image sensor performance. (See this related story on the Canon 5D Mark III scores.)

But DxO Labs' scientific testing yields some other findings that might surprise people used to photography conventional wisdom.

Behind the scenes at DxO Labs (photos)

+8 More

See all photos

That testing is steeped in a mathematical approach to imaging technology that comes from unusual origins: DxO Labs was a 2003 spinoff from Vision IQ, a company that specialized in swimming pool safety. Its Poseidon products keep a digital eye out for potential drowning victims. To keep costs down, the system uses fisheye lenses with a very wide but distorted view; the DxO team developed technology to mathematically correct the view so a computer system could monitor the pool better.

That technology paved the way for DxO Labs' broader imaging businesses. Its biggest division -- about three quarters of its nearly 130 employees -- is designing image-processing technology incorporated into mobile phone cameras. A second group sells testing products that others can use to test their own camera gear.

And most familiar to consumers is the software group. It makes the DxO Optics Pro image-editing software, which can automatically correct optical problems for a large number of camera-lens combinations, and DxO FilmPack, which can precisely emulate the grain, contrast, and colors of several films. One customer is photographer Sebastiao Salgado, who uses it to keep his Kodak Tri-X film look though shooting digital now.

Upsetting the conventional wisdom
All this scrutiny of exactly how cameras work has led DxO Labs to some conclusions that might surprise those who think themselves knowledgeable about the technology.

On April 6, I visited DxO Labs' facilities in Boulogne Billancourt, a couple Metro stops south of Paris's swanky 16th arrondissement and about three miles downstream of the Eiffel Tower. The building itself is covered in rectangular white tiles separated by dark lines -- just the sort of pattern that gets a person thinking about his wide-angle lens' barrel distortion. Inside, it's your standard office building, except that a visitor gets the impression that the prints hanging on the walls get a lot more scrutiny than those at the average technology company. I was offered a rare opportunity to see how the company puts a brand-new camera through the wringer and how DxO Labs reaches its conclusions.

Frederic Guichard was one of the 10 employees who struck off from Vision IQ when DxO Labs got its independent start under the initial name Digital Optics Labs. Nine years into the company's independence, he's eager to share some unexpected conclusions from the company's testing work. Here are four of them:

Actually, megapixels do matter
Though the megapixel race has abated, with camera makers no longer relying on image sensor resolution as a proxy for camera quality, pixel-peepers are often upset at the drawbacks of more pixels. Shrinking pixels to fit ever more of them on sensors can produce noise when an image is viewed close up.

But higher resolution also means those noise speckles are smaller, thus benefiting a photo when viewed in its entirety instead of zoomed all the way in. The same improvements apply to color depth, he added

"Everybody says there is no need for more pixels, and we should reduce the number to a reasonable number so the quality will improve," Guichard said. However, DxO's aggregate measurements tell a different story: "If we look at the cameras, there are more and more pixels, and the quality is increasing in the meantime."

And Guichard said dynamic range, which measures how well a camera can record details in both dark shadows and bright highlights, has steadily improved, too: "Manufacturers have made enormous strides in handling ever-smaller pixel sizes."

There's no free lunch, of course -- more pixels means more hardware demands for processing and storing data. And lots of megapixels doesn't matter when lenses aren't good enough to transmit details. But think twice before you scoff at those pixels as a mere gimmick.

ISO isn't what it appears to be
ISO, the shorthand term for the sensitivity setting of a camera, has been surging in recent years, with top-end SLRs such as the Nikon D4 and the Canon EOS-1D X able to shoot at ISO 204,800. In comparison, only unusual film reached ISO 3,200. Doubling ISO means you can shoot photos in half as much light, but at the cost of more noise in the photo.

There's no question low-light performance has been improving. But take that ISO number with a grain of salt.

The ISO sensitivity test allows a certain latitude, so for example a Nikon D4 set for ISO 204,800 is actually shooting at 139,250, according to DxO's tests, and a Pentax K-01 set for ISO 3,200 actually is shooting at ISO 2,724. Pentax's ISO setting might give the camera an edge in a comparison to a rival's ISO 3,200 performance, since the Pentax is actually shooting at a lower ISO with lower noise. But at the same time, other camera makers could be playing the same game -- and in any event the photographer just has to push the ISO higher to keep a photo from being underexposed.

Camera makers also have shifted their standards for acceptable noise, so just because this year's camera goes to a higher ISO than an earlier model, don't assume that the image quality at the highest ISO setting is on par. Cameras can clean up photos as they're converted into JPEGs, but DxO's measurements of the raw image data shows how newer cameras produce more noise at the highest ISO before that processing.

Phone cameras are better than you think
Phone cameras necessarily have tiny image sensors that can't capture as much light as the bigger sensors on compact cameras or the even bigger sensors of full-frame SLR cameras. But for a given surface area of image sensor, mobile phone cameras actually do better.

"If you scale down the quality to the sensor size, today the [phone] cameras and sensors are better than the SLR sensors," Guichard said. "In the end, the image quality is not as good because it's smaller. But if Canon were able to put the technical quality of a 2012 phone camera on full-frame sensor, they would win about 1 stop more [in image quality]. It's a big difference."

In comparison, the average DxOMark image quality has risen by 2 stops from 2004 to the present, he said.

Unfortunately for photo enthusiasts, though, DxO Labs doesn't show scores for camera phones because those products only produce JPEG images for customers, not the unprocessed raw sensor data DxO Labs uses. "We have a lot of camera sensor data in raw, but we are not allowed to show the results," Guichard said.

In practice, sensors beat film's dynamic range
Most people aren't surprised to hear that high-end digital cameras offer higher resolution, lower noise, and better low-light performance than film. But there's a common belief that film still surpasses digital when it comes to another important attribute, dynamic range, which measures the spread from where a scene is too bright for a camera to capture detail to where it's so dark that details are lost in the image noise. Dynamic range is measured in "stops," with each stop representing a doubling of the amount of light; a wide dynamic range means a camera can capture details in both bright highlights and dark shadows.

The problem, Guichard said, is that the technical measurement of dynamic range is based on a signal-to-noise measurement that's not useful in practice. With that formula, "the dynamic range of film is eight stops more than any sensor on the planet," he said. However, "this threshold doesn't make sense. We have to define another threshold more related to a minimal quality acceptance threshold."

Basing dynamic range on this human-based quality level (technically, it means the threshold has a signal-to-noise ratio of 10, also known as 20 decibels), the tables are turned. "All digital SLR cameras are above film," Guichard said.

Running the DxO gauntlet
Judging images is necessarily a subjective matter, but technical tests are grounded in people's preferences -- no weird colors, exposure problems, or noise speckles. DxO's exhaustive test results for image sensors and lenses bring a little light to what is often a heated discussion. The Nikon vs. Canon debates aren't generally as vituperative as Windows vs. Mac debates or or iOS vs. Android flame wars, but there's plenty of passion.

DxO bases its judgments on a series of tests, all performed at a standard temperature and humidity, with carefully calibrated light levels and color.

When entering a testing lab, the first impression given is one of darkness. Most surfaces are absorbent black to keep stray light from interfering with results. In one corner is a box with a color chart to test the color response. Along a wall is a box with a ring of spots varying in light intensity to test dynamic range.

But dominating the far wall of the room is a giant black-and-white chessboard pattern used to check lens and camera image sharpness. Disconcertingly, the pattern is rotated 5 degrees compared to the rectangle on which it's printed. It's almost enough to induce a little seasickness, but it's what the test requires.

Here, technicians put cameras and lenses through their paces. Each camera test must be repeated across the full range of ISO settings, leading to a huge amount of data.

"It takes about 1,000 shots to characterize the camera," said Nicolas Touchard, the company's vice president of marketing and a veteran of imaging granddaddy Kodak.

When you consider that a high-end camera such as the 5D Mark III takes photos with file sizes on the order of 30MB, that means DxO gathers many gigabytes of data for each camera. It typically takes days to test a single camera in the lab.

Then comes even more work, though -- testing each camera to gauge how a range of lenses does on each camera.

That requires several thousand photos more per camera, because each lens-camera pairing must be tested at different apertures and focal lengths. And to ensure accurate sharpness measurements, DxO technicians take multiple photos at slightly different distances to figure out where the true sharpness maximum is reached -- often different from where the camera's autofocus system thinks it is.

The Dead Leaves test
DxO Labs thinks today's tests aren't good enough, however. In its work with the standards groups, It's proposing another, which it calls the "dead leaves" test.

Like many other tests, it involves judging photos of a standard pattern. But it forgoes the usual sharply defined black-and-white imagery for a range of intermediate gray circular splotches that are irregularly splattered atop one another.

Unlike standard test patterns, the dead leaves test mimics common natural structures. I immediately thought of speckled eggs or mud spots, but dead leaves is definitely an apt name.

"This pattern is tough to render. It's got low-contrast, medium-frequency pattern, like foliage, grass, fabrics, and stones," said Touchard. It's simple with monochrome test patterns to find edges and sharpen them for a better-looking photo, but it's much harder with the dead leaves pattern to distinguish what's actually there from image-sensor noise.

Getting a new test to catch on isn't easy, though -- especially a test that might spotlight flaws on particular cameras. DxO proposed the dead leaves test a year ago at the ISO's TC 42 group for photography standards, but has encountered some resistance.

"This is not perfect," critics tell Touchard, who attends the meetings. Touchard counters by pointing to the giant black-and-white checkerboard on the wall. "But is this perfect? If they can cope with [the dead leaves test], then we're going to have better pictures. A common objective for all camera makers is to have scenes that mimic real-world situations."

Putting the data to use
The result of all the testing is a lot of data, but it's not just academic. It's used to perform lens corrections in DxO Optics Pro. The software today has more than 6,000 modules available with various combinations of cameras and lenses.

Each test characterizes a lens' weaknesses, and the software mathematically backs out the problem. Among those problems are distortion, which can make parallel lines bow inward like a pincushion or bow outward like a barrel; chromatic aberration, which produces color fringes around high contrast areas because different colors of light pass through lenses along different paths; and vignetting, in which corners of a photo are darker than the center.

Other software can do that, too, including what probably is DxO Optics Pro's top competition, Adobe Lightroom. DxO Optics Pro adds another automatic adjustment, though: sharpening that varies across the image according to the performance of the lens. That means the software applies stronger sharpening to the soft-focus areas -- typically farther from the center of the image -- than to the sharper areas toward the center.

DxO needs every edge it can get.

"It's a very competitive market, for sure," said DxO Labs spokesman Constantin Foniadakis. Because DxO Optics Pro doesn't have the cataloging abilities of software like Lightroom or Apple's Aperture, but that means it can work in conjunction as an engine to process images in advance before importing them to those other programs.

"We have a scientific approach. We are very interested in how to get the best result in an automatic way with DxO Optics Pro," Foniadakis said. "We advise people who use Lightroom to use DxO Optics Pro at the beginning and then put the pictures into Lightroom."

DxO shares s big engineering challenge with its rivals: the difficulty of decoding raw image files. Raw files, typically available only on SLRs and other higher-end cameras, offer more flexibility and more quality. But photographers must process raw images manually to convert them into a more convenient format such as JPEG. Software from Apple, Adobe, DxO Labs, Phase One, or Corel is necessary for that conversion, but because the raw files are proprietary, those companies' programmers must figure out how to decode the images.

Camera makers offer varying degrees of help in figuring out the raw formats. "Some are very helpful, and some are just saying no information," Guichard said. "Sometimes 10 minutes, sometimes two or three weeks. For some formats we have a lot of pain."

For example, to correct an image's optical problems, DxO needs to know how the lens was focused, and that data is sometimes hidden away in a section of the image file called maker notes. "Typically ... it's not in meters or feet. It's written as step in the motor or something like that," and it takes engineering work to translate that into focus data.

It could make life easier for raw software companies if camera makers adopted Adobe's Digital Negative format, an openly documented raw format that Adobe is trying to turn into an industry standard. But the big-name SLR camera makers have shown little enthusiasm so far.

"I don't see Nikon or Canon working with Adobe," Guichard predicted.

Computational photography
DxO is at the heart of a major change sweeping the camera industry: the arrival of computers.

Every digital camera has at its heart an image sensor that works in concert with an image processor. The latter digests the raw data and does things like judge how much sharpening to apply, reduce noise speckles, compensate for various lighting conditions, and compress a stream of sensor data into video.

But the job description for image processors is getting longer and longer as microprocessors rewrite the camera industry rules at the uncomfortable pace of Moore's Law.

One of the newer jobs image processors are handling is correcting lens distortion, vignetting, and chromatic aberration -- and now doing that with video as well as still images. Another is combining multiple shots into a high-dynamic range (HDR) image. The extreme end of this spectrum is Lytro, whose radically different "light-field" cameras dispense with the difficulties of lens design and autofocus and instead rely on a built-in computer to produce an image from the raw data.

DxO's main business is selling this sort of technology to those building mobile-phone cameras, whose tiny sensors and lenses offer less leeway than point-and-shoots or SLRs.

"Because digital processing can correct flaws of hardware, with digital processing we can correct the defects" of cameras, Guichard said. "We can transfer some of the complexity of the sensor or lens to the digital processing." The result is that cheaper or smaller lenses become feasible.

Mobile phone photography
One case in point is automatic compensation for manufacturing problems in mobile phones' tiny camera modules. Each module is a package of parts sandwiched together, and DxO offers technology that compensates for lens alignment and manufacturing irregularities. It does so by checking for color-shading differences that reveal camera geometry issues and correcting accordingly -- every time a photo is taken. The result is that manufacturers don't have to test and calibrate the modules themselves, Touchard said.

Another is extended depth of field, or EDOF, which has shipped in tens of millions of camera phones so far. EDOF takes what's ordinarily a disadvantage of camera lenses -- that red, green, and blue light take different paths through a camera lens by virtue of the laws of physics -- and turns it into an advantage.

The different paths taken by light produce chromatic aberration, a problem which one frequency of light is focused but others aren't. But with DxO's EDOF approach, the image processor finds what color is sharpest and processes the out-of-focus colors accordingly.

"We take the sharpest color by computation, then transport the sharpness to the others," Touchard said. The result is cameras that don't need autofocus because everything is sharp, something that's important as sensor resolution of mobile-phone cameras increases. And no autofocus means faster camera response and cheaper products. It's not a perfect technology, but it's found a place.

DxO is just bringing to market another technology for when autofocus is needed. It speeds up the autofocus process.

"We embed in a lens optical flaws that we know how to correct. Thanks to those optical flaws, we have an idea of the object distance," Guichard said. "It's not very precise. But it can tell if the object is further or nearer. You always which way to go, and to an extent the magnitude."

And, he added, "This will ship very soon."

Given that mobile phone cameras are increasingly people's primary camera -- and given how painful their shortcomings are today -- it's a good market to be in.