Ever wondered what goes on behind the scenes of every CNET phone review?
The process is straightforward -- but that doesn't mean it's easy. We rate phones based on major categories like design, features, performance, camera quality, battery life, waterproofing and how well they compare to other devices in their field. We come to the final rating using a blend of benchmarks, procedural testing and real-life observation accumulated from using these devices day-to-day.
Things get a lot more complicated when we're deep in testing. During the review process, we test and tinker, and ask ourselves a lot of questions, including some of the recurring ones below. This isn't an exhaustive list of every consideration that comes in to a review, but it gives you a better idea of how we arrive at our final rating.
Our most important conclusions above all are if the phone's worth your time, money and attention. At the end of the day, we want to help you find the best phone for you.
Design and usability
You want your phone to last a long time, and rebuff scratches and scuffs. You want it to fit your hand and not get dirt or beach sand lodged into the grooves. So do we! We look at what the design gets right and wrong.
- Body: Durable or flimsy; modern or dated?
- Material: Is it made from quality material, or will it crack at the first drop? How quickly do your fingerprints pile up?
- Dimensions and weight: Too heavy or light; too thick, or sharp edges? We look for phones that are comfy to hold.
- Usability: Do the placement of buttons and jacks make sense? Do you have to stretch your hands to navigate and type?
- Screen: Is it clear, bright, contrasted and detailed? Can you read it outdoors in direct sunlight?
- Fingerprint reader, face scanning and/or iris-scanning: Are they accurate and fast?
- Special features: Do they make sense, and do they do what they say?
- Do any additional features or flaws stick out?
Water-resistance: Get it wet
If a phone has a rating of, which means that it's rated water-resistant for up to 30 minutes in up to 3 feet (1 meter) of water, we dunk it... twice.
We fill a 5-gallon bucket of water and gently place the phone at the bottom. After 28 minutes on the timer, we remove the phone and let it air-dry overnight. Then we inspect for condensation behind the camera lens, or water in the ports. We make sure we can take a clear photo.
We wait at least a day, and then run the test again. If a phone fails either test, we reach out to the manufacturer and publish our findings. Note, though, that we never dunk phones that have previously been used in a drop test -- or have been otherwise know to suffer physical stress -- since that could compromise the waterproofing.
Battery life: Drain, baby, drain
The longevity of your phone is such a big deal, we break it out as its own subrating. In addition to real-world tests, where we keep an eye on the battery life as we naturally use the phone, we also run multiple battery drain tests.
- Phone starts at 100 percent, turned on to airplane mode
- Screen brightness set to 50 percent (we use an app to keep the screen always-on)
- Media/headphone volume set to 50 percent
- Loop an HD video until the battery drains
- Run the test twice
- If the results differ by greater than 5 percent, we run the test again. And again, until we see consistency
The final score is based on a rubric that pegs battery longevity on a spectrum. Typically, phones last a work day before reaching critical levels, and reviewers will point out if battery life is especially impressive or poor.
Note, however, that battery mileage will always vary in the real world based on an individual's settings (screen brightness) and activities: If you're alternating between checking Facebook, streaming Netflix and gaming all day, don't expect even a great battery to last you through the dinner hour.
Let's say your phone has wireless charging. That's a feature. And so is a headset jack, a microSD card slot, a fingerprint reader, waterproofing, a digital voice assistant or programmable button.
There are software features, too, like face unlock in the iPhone X ($294 at Amazon), OnePlus 5T and Galaxy phones, and a digital assistant like Samsung's Bixby Voice. In fact, most custom software and hardware fall into this category. Just think of the snap-on Mods, BlackBerry KeyOne's ($399 at eBay) physical keyboard or Galaxy Note's ($240 at eBay) stylus.
During a review, we editors evaluate how well all this hardware and software is in theory and in practice.
But we also take into account missing features that we wish were there, like when a phone maker strips out the headphone jack and forces you to buy a dongle if you want to keep using your own headphones. Or maybe it's the only phone in its price category without waterproofing.
As a rule of thumb, the more quality features a phone offers, the better. But quality is the operative word. If the features aren't very good -- or if they don't add anything substantive to the product -- that score doesn't necessarily rise.
Whoo-boy, this is a big one. It's hard to overstate the importance of camera quality as a purchasing decision, especially in the age of Instagram.
So to test image quality, we take a ton of photos, just like you do. We take the majority of the pictures in automatic mode, often without a flash. In some cameras, auto settings kick in additional profiles like HDR or night or food mode, and that's OK. We want to fire off the shots that the majority of phone owners do when whipping out a device for a quick pic.
But we'll also take photos in specific modes, like macro, portrait or pro settings.
We always look at image quality with these conditions:
- Low light
- Photos of objects
- Photos of people
We evaluate photos both onscreen, and more importantly, on our larger computer screens. We compare pictures from the test phone with photos taken on other devices, and look at elements like detail and sharpness, color accuracy and color richness, and white balance and exposure.
We often take a step further, especially for marquee devices like or those with camera claims that go above and beyond the usual. For example, comparing the image quality of top handsets in a, or evaluating phones like this one .
Speed and performance
Nothing is more irritating than a laggy smartphone. The good news is that processors have gotten so good, we really don't see handsets stuttering the way we used to.
Some are still slower than others, which can make a difference when you're streaming video, processing photos or playing graphically rich mobile games that demand a quick refresh rates. Some cameras load faster than others, screens can scroll in the blink of an eye and app and browser load times also vary.
We test processor speed two ways: by comparing the results of benchmark tests with other phones in the category, and by paying attention to speed and hang times. We care more about how the speeds compare than the raw result.
We use Geekbench 4 and 3DMark Ice Storm - Unlimited. While Geekbench is more of a traditional benchmarking tests, 3DMark is a graphical and gaming benchmark that we also use for tablets. Gaming is such a large stressor and performance indicator, this test lets us predict how the phones will respond to a range of graphically intense and resource-heavy titles.
Editors might also decide to use additional benchmarking tests for certain reviews.
Benchmark results can be manipulated, which is why we also pay attention to how phones perform in the real world. Editors play a graphically demanding game, paying attention to frame rate and graphical quality. For example, a budget phone might tank on a benchmark, but play a game remarkably well.
- Call quality: We'll point out if a call sounds especially good or bad
- Data speeds: Like call quality, this fluctuates wildly around the world, depending on local carriers. We'll test claims of new features, like 5G speeds when those phones arrive
- New features: Any new hardware or software gets a thorough rundown so long as we have the testing tools. E.g., New voice assistant or wireless charging addition
Star rating and scoring
The written review tells you everything we really feel, but the overall score and star rating help nail it all in place.
Editors assign a rating out of 10 to these sub-ratings: design, features, camera, performance and battery life. The final score is a weighted average of those sub-ratings.
We have the discretion to slightly adjust that calculated overall rating to take into account special features. Cost and value fit in here, too. Let's say the handset costs half the price of every other phone in its field, that might see a 0.1 or 0.2 point bump. On the flipside, if the phone is terrific, but costs $500 more than every other phone in its category, we might lower the score if its specs fail to justify the sky-high price.
We also reserve the right to update the ratings in the short and near term based on a variety of factors, including software changes to the phone, quality control issues and changes in the competitive marketplace. In those cases where the ratings change, we mark them clearly in an editors' note at the top of the review.
First published July 1, 2015, 3:49 p.m. PT.
Update Dec. 22, 2017, 5 a.m. PT: Refreshed with our latest phone testing standards.