Catch the falling stars: CNET's 2010 HDTV ratings overhaul explained

CNET editors provide the reasoning behind the overhaul of HDTV ratings from 2009 to 2010.

CNET's first review of a TV from the 2010 model year was the Sony KDL-NX800 series. Sarah Tew/CNET

If you're reading this, perhaps you just clicked through from an editors' note on a CNET review of a 2009 TV, looking for a detailed explanation of why we changed the rating. Here it is, in as much excruciating detail as you'd never want to know.

As the page linked from "Detailed editors' rating" below the stars on every CNET review explains, our five-star system is based on some simple math. The number of stars from zero to five correlates to a numeric overall rating from 1 to 10. That rating is determined by three or more subratings, again from 1 to 10, which are assigned weightings and averaged differently for each product category.

In the case of TV reviews those subratings are Design (30 percent), Features (30), and Performance (40). The Sony KDL-NX800, for example, received a 9, 8, 6 respectively, which, when weighted and averaged, works out to 7.5, or 3.5 stars.

An example of the subratings system used to determine the number of stars assigned to a CNET review.

So why lower the ratings of older products? The basic reason, which we gave in that editors' note, is "changes in the competitive marketplace." That means new products have been introduced that make the older ones deserve lower ratings.

We could simply leave the old products at the same ratings and rate the newer ones higher, but the obvious downside is that newer products will get ever higher ratings and eventually "break" the scale above. That's not acceptable to us, because we're interested in preserving the sanctity of the 10-point scale; it's much more intuitive than an 11-point, 20-point, or infinity-point scale. The main point of a ratings system, in our view, is to allow easy, intuitive comparison of the general relative merits of multiple products.

Those kinds of comparisons become more difficult as new products are announced while older ones remain on the market. Every spring, new TVs start hitting store shelves while older ones remain, and will likely remain through late spring and into summer, if not later. Meanwhile, at CES in January, most TV manufacturers outline their plans for improvements throughout the year, suddenly making those previously well-featured TVs, for example, seem skimpy in comparison.

The best solution--and we admit it's an ugly one--is to lower the ratings of TVs from 2009 to make room in the scale for new ones. That doesn't mean your 2009 TV is suddenly a piece of junk; it just means we raised the bar. The regular cycle of new TV introductions means we can do that now and (hopefully) be safe for the remainder of 2010, having to perform minimal ratings adjustments until this time next year. The goal is to allow comparisons between older and newer models on as level a playing field as possible.

We performed a big "re-rate" of TV reviews last year, but not to the extent we're doing now. We have lowered the rating of nearly every 2009 HDTV we reviewed. Those adjustments mainly took place in Features and Performance. We also didn't touch older reviews because, among other reasons, comparisons are largely irrelevant to current shoppers since those products aren't available anymore.

The Features adjustments are easy to understand--new features like 3D, built-in Wi-Fi, wireless HDMI and data-friendly remotes are in our opinion worthy of higher ratings--but reductions to Performance scores bear some more explaining. We lowered the Panasonic plasmas for reasons outlined here, but most of the other Performance changes were made to create more "headroom" in the scale.

We said in the Sony NX800 review from 2010, for example, that its picture quality was roughly equivalent to that of the Samsung UNB7000 from 2009, but we assigned it a 6. With improvements we've seen and anticipate this year, we feel that the Samsung also deserves a 6, so we lowered its previous 7. In other words, we fully expect at least a few new 2010 models to perform better than either TV, and thus deserve a 7 ("very good"), yet not be good enough to deserve an 8 ("excellent"). TVs with performance scores higher than 7 were generally left alone, but could conceivably be adjusted in the future if higher-performing models are reviewed. To that point, the dearly departed Kuro is still a 10 in picture quality on our scale, the Samsung 8500 is still a 9, and so on.

Below you'll find a spreadsheet that summarizes all of the ratings adjustments we made to 2009 TV reviews (changes to numbers are bold on the sheet). We believe the adjustments are fair and accurately reflect possible improvements in 2010, but with a scale as coarse as the one we use, some hard choices must be made, some differences smoothed over, and others exaggerated, and some readers might not agree.

Whether you agree or not, we'd love to hear your comments, and if you can think of a better way to do it, let us know.

Check out the full-page version of the spreadsheet here.
Featured Video

Tim Cook's blurry iPhone picture takes world by storm

What is the iPhone 6's "Error 53"? The new Apple tvOS brings new features and Tim Cook takes bad pictures.

by Brian Tong