DxO tests dig deep into camera lens performance

For photography gearheads who want detailed tests of lens performance, a significant new option is joining the likes of SLRgear.com, Photozone.de, and Digital Photography Review.

The new kid in town is from an established player in camera measurements: DxO Labs. The company, which performs detailed tests of cameras and sells software for editing raw photos, is revamping its DxOMark site to offer lens tests as well as the earlier image sensor tests.

It's got some significant differences over established reviews sites. First, it adds a parameter the others lack, transmission, which measures how much light actually makes it through the lens. For example, Canon's EF 50mm f/1.8 II lens actually has an effective maximum aperture more like f2.1, DxO's tests show.

Second--and this is the bigger difference--the tests show how each lens works on a wide range of camera bodies; existing lens sites typically run tests just on one or two bodies. Seeing how a lens works differently on different bodies can help inform prospective buyers whether a high-end lens is worth its premium on a lower-end camera, or how well an existing lens collection will work on a new camera body.

For example of the differences, the Nikon AF Nikkor 85mm f/1.4D IF rates "excellent" for both peak performance and resolution on the top-end Nikon D3X camera, but that drops to middling performance on the midrange D300 or the low-end D5000 camera. To pick another example, the high-end Sony 70-200mm F2.8 G gets a solid peak performance score on the top-end, full-frame Alpha A900 camera, but its score drops considerably on lower-end cameras.

There is, for now, a somewhat limited selection of lenses available--60 total, with 546 combinations with various cameras--but dozens more will be added monthly, said Nicolas Touchard, vice president of marketing for DxO's Image Quality Evaluation group. That will be a combination of existing data DxO already has and new data that will arrive as new cameras and lenses arrive. Also missing are Micro Four Thirds and related hybrid cameras and medium-format models.

Along with transmission, the DxO tests also measure more familiar characteristics: chromatic aberration, a color-fringing problem that results from the fact that different colors of light travel through a lens on different paths; distortion, in which lines that should be parallel bow inward or outward; vignetting, which darkens corners of images; and resolution, which measures the sharpness of details captured. Each lens-camera combination shows a collection of such scores, with each single number a composite score based on overall performance at a variety of focal lengths, apertures, and positions across the image frame.

The tests show results for a single copy of each lens, with serial number provided, but DxO factors in variability across multiple copies, Touchard said. Specifically, he said the score numbers are rounded to not show a misleading level precision. Lower-end lenses show more variability, he added, and typically only in resolution but not chromatic aberration, vignetting, and distortion.

The tests are certainly not complete reviews. They don't tackle value, build quality, weatherproofing, or any number of other factors important to buyers. And it doesn't tackle some optical properties that are hard to quantify, such as contrast or ghosting.

I've been digging around a beta version for a few days now, and find it takes some time to learn to use the complicated new site. But it's definitely useful.

I'm less keen on DxO's ratings of a lens for particular uses--photojournalism, travel, portrait, sports, and landscape--and would rather just judge a lens based on what attributes matter to me. But the overall goal to boil down some of the numeric complexity is laudable.

The ability to compare multiple lenses also is handy, but the site's inability to scale to larger monitors means you'll squint at the graphs. One hint: clicking on one of these miniature versions pops it up in an enlarged view.

For data junkies, it's fun to look at the collective camera-lens pair ratings, which offers a chart that ranks various combinations by total price on the one hand and by peak score on the other. The large gaggle of lower-end products toward the lower left of the chart is too cluttered to be really useful, but it's more interesting poking around the high-scoring, lower-cost region toward the upper left. Just remember that peak score isn't a full measure of the lens.

Peak score isn't a full reflection of a lens' optical quality boiled down into one number; instead it reflects the total amount of information gathered by a particular lens on a particular camera. It therefore can increase with improvements in resolution for either the camera or the lens or be held back with corresponding shortcomings in either.

Helpfully, DxO shows you where the peak score was achieved, a useful guide to a lens sweet spot. For example, Canon's EF 16-35mm f/2.8L II USM mounted on a Canon 5D Mark II has a peak score at 20mm and wide open at f2.8.

Again, don't consider the various scores a complete reflection of the products. But if you're curious about detailed lens performance, dive in.