DxO sheds light on camera sensor performance
Detailed tests of image sensors are raw meat for performance-obsessed pixel-peepers; DxO will add medium-format and lens tests later.
- Shankland covered the tech industry for more than 25 years and was a science writer for five years before that. He has deep expertise in microprocessors, digital photography, computer hardware and software, internet standards, web technology, and more.
A correction was made to this story. See below for details.
DxO Labs, a French company with deep experience measuring cameras' technical performance, has launched a Web site called DxOMark.com that features detailed information on the performance of the image sensor at the heart of many higher-end digital cameras.
Many Web sites and magazines measure camera image quality with varying degrees of rigor, typically examining either the JPEG that the camera produces or a processed version of the camera's raw. But with its DxOMark Sensor measurement, DxO takes a new approach by judging the sensor performance based on the unprocessed "raw" image file from higher-end cameras such as SLRs.
That's significant, because raw images typically must go through a conversion process called demosaicing before they're useful for viewing. Most digital cameras capture only a single color--red, green, or blue--for each sensor pixel. Demosaicing fills in the gaps in this colored checkerboard pattern so each pixel gets all three color components, but this processing stage can disguise sensor performance.
The detail-obsessed camera crowd has begun eagerly chomping on the new data. On Sunday, there were 220 mentions of DxOmark on the Digital Photography Review forums, a popular location for impassioned technical discussions.
New tests coming
More measurements are coming, added Nicolas Touchard, vice president of marketing for DxO Labs' image quality evaluation business. First, in two or three weeks, will come measurements for medium-format digital camera sensors from companies including Hasselblad, Mamiya, Phase One, and Leaf. Then will come more high-end compact "bridge" cameras.
DxOMark Image Processing for the camera's computer, whose job it is to perform tasks such as converting raw images to JPEG, and DxOMark Optics for lenses.
The latter measurement will go beyond most lens tests by showing how well each lens works on each camera rather than one or two reference models. DxO takes that approach because lenses behave differently because different cameras have different attributes such as the geometry of the microlenses that help each sensor pixel gather more light, Touchard said.
DxO makes a business out of detailed measurements of camera performance, selling the data to camera and chip companies and incorporating it into its own DxO Optics Pro raw-processing software for photographers. So why give some of the data away for free on a Web site? Publicity.
"It's a marketing program to communicate to the public as well to the industry our expertise in measuring cameras," Touchard said.
The DxOMark Sensor benchmark measures a sensor's color depth (useful for studio photographers), high-ISO performance (useful for photojournalists), and dynamic range (useful for landscape photographers)--the span between dark and bright. DxO rolls the three parameters up into a composite DxOMark Sensor score, but it provides other detailed measurements for each camera as well. (DxO also offers detailed descriptions of the three DxOMark sensor components.)
DxOMark Sensor doesn't measure many camera attributes--durability, autofocus, video and live-view modes, battery life, LCD quality, weatherproofing, user interface--and DxO doesn't pretend it does. Indeed, that's part of the reason for the later benchmarks for image processing and optics that the company envisions.
Intrigue
But even in its limited form, the tests are interesting. For example, the full-frame 12-megpixel sensors in Nikon's top-end D3 and D700 models currently rule the roost, beating out the full frame sensors in Canon's top-end 21-megpixel EOS 1Ds Mark III and Sony's top-end 24-megpixel Alpha A900. The D3 has a 80.6, the D700 80.5, the 1Ds Mark III 80.3, and the A900 78.9. Those differences are relatively narrow.
More interesting, perhaps, is the difference in performance lower down the line. With a score of 72.6, the sensor in Nikon's new 12-megapixel D90 is head and shoulders over the Canon 50D's 15-megapixel sensor, which scored 62.9, even though the D90 is a bit cheaper. Indeed, the D90's sensor scored sixth out of 50 cameras measured.
Sensor results can be compared two different ways: "screen," where performance is compared pixel-by-pixel, and "print," which compares the sensor's full-size images. In the print mode, to compensate for different sensor sizes and resolutions, the DxOMark sensor test normalizes results mathematically to the equivalent of an 8MP image.
Smaller pixels suffer from more noise, which shows up as speckles in the image. But having more pixels also improves the amount of information in an image, so high-resolution sensors get an edge in the "print" measurements. Thus, for example, the Canon 50D makes up some of the ground on the Nikon D90 when viewed in "print" mode to consider the whole image.
Comparing the Canon 40D to the newer 50D is also instructive. Though the 40D gets an overall score that's higher, presumably because of stronger weighting of the color depth, the 50D fares a smidgen better with dynamic range and significantly better with low-light performance.
Also worth a look
Also interesting at the site are a couple explanations, one about "the paradoxical evolution of sensor signal-to-noise ratio over time" and another that argues that "contrary to conventional wisdom, higher resolution actually compensates for noise." (Dust off your logarithms and statistical analysis for the latter.)
The diversity of camera sensor designs shines through in DxO's measurements. The SuperCCD sensors used in Fujifilm's SLRs, for example, use two sensors for each photo's pixel to increase the dynamic range of the images. That design shines through when it comes to dynamic range: the Fujifilm FinePix S5 Pro spans a range of 13.5 stops compared with 11.5 for the Nikon D200 and 12 for the Nikon D300. However, the Fujifilm camera's sensor pixels are larger, reducing resolution.
And there's still no result for the Sony Alpha A700, which performs some image processing on the sensor before the raw image is created. "It's not really raw. I'd say it's medium raw," Touchard said.
A final note: DxO provides some fodder for the enthusiasts who prefer to shoot raw images instead of JPEGs. That's because demosaicing algorithms get better over time, so more information can be recovered with newer software.
"For example, over the past two years, DxO Optics Pro raw conversion (with its denoising algorithm) showed a gain of 4 decibels signal-to-noise ratio, a much larger improvement than that achieved for sensor technology," DxO said (For reference, 4 decibels is about a 1-1/3 stop in exposure.) "These advances in raw conversion algorithms will continue, underscoring the advantages of shooting in raw, as raw images can always be reprocessed with every new release of a raw converter."
DxO's data is fun to pore through and should will be helpful for those considering upgrades or new purchases. Just don't let the technical analysis get in the way of actually taking pictures, too.
Correction on November 24: This story initially mentioned a Nikon camera with a 12-stop dynamic range. That camera is the D300.