How we test: Phones

Sit tight as we break down how CNET determines what's good in the world of handsets, and what's not.

Like all good reviewers, CNET editors rate phones based on specific criteria, including their design, performance and camera quality, in addition to how well they compare to other devices in their field. Here, we walk you through our process for scrutinizing any phone we review. (And be sure to check out CNET's cell phone buying guide if you need more tips.)

Design and usability

First, we consider what type of device we're looking at, and what it's for. We mostly see smartphones these days -- and the occasional simple device like a flip phone -- but there are other specialized features we may need to consider, like resistance to water.

The Samsung galaxy S6 Edge has an unusual design worth noting. Juan Garzón/CNET

We take in the phone's physical attributes -- its size, weight, thickness, and the materials used to construct it. In general, handsets crafted from metal and glass feel more premium. Of course sophisticated fabrication techniques such as molding a phone from a seamless slab of sturdy plastic can lend an air of luxury as well.

Next, we examine the screen, accounting for such things as screen size, resolution, brightness, contrast, and color accuracy. Button layout plays a big role in a phone's usability, as do any special software enhancements (or hindrances) that manufacturers add on top of its stock interface. These points may all sound like minor quibbles, but they can really affect if you absolutely love your handset, or merely use it.

Speed and performance

For serious smartphone shoppers, nothing quite delivers purchase validation, or bragging rights, like a handset's processor performance. These days, octa-core chips are king, whether made by CPU specialists such as Qualcomm or MediaTek, or built in-house by handset manufacturers like Samsung.

To scientifically measure a phone's raw computing prowess, we run synthetic benchmark apps designed to test processor speed. Android smartphones make up majority we review. For those, we use the following diagnostics: Geekbench 3 gathers single-core and multi-core results that we often present alongside other phones in the contender's class. We're more interested in how the scores compare than we are in the test's raw result.

Geekbench 3 is one helpful diagnostic test. CNET

We also run 3DMark Ice Storm - Unlimited, a graphical and gaming benchmark we also use for tablets. Gaming is such a large stressor and performance indicator, this test lets us predict how the phones will respond to a range of graphically-intense and resource-heavy titles.

Devices can be manipulated to yield fantastic diagnostic results, which is why real-world tests form an important part of the review. Editors play a graphically-demanding game of their choice that they know well, paying attention to frame rate and graphical quality. For example, a budget phone might have a terribly low score, but decent gameplay regardless.

For the iPhone, Windows, BlackBerry OS and others, we may need to adjust the diagnostic tests we use, depending on the availability of compatible apps at the time of the review.

Battery life

A cell phone's battery life is just as critical as its processing power. To get a handle on a phone's longevity, we take a few approaches. On devices, we measure how long a phone can loop an HD video before it dies. To eliminate as many variables as possible, all wireless radios are switched off and volume and screen brightness levels are set to 50 percent. The screen remains always-on.

How long can your phone play video before conking out? Sarah Tew/CNET

The other option is to log a handset's talk time over its cellular connection. This test is handy for smartphones that can't run a video files, like a simple feature phone. Scores for both tests are reported in runtime by hours.

Editors test battery at least twice during the length of the review process, sometimes three or four times if results are varied.

Since phone owners don't exclusively watch video, editors also weigh in on anecdotal evidence, the battery life they witness during their review process. Typically, phones last a work day without needing a charge, but reviewers will point out if battery life is especially impressive or poor.

Call quality

It's the common joke: "Who makes calls?" Yeah, yeah, text-based communication trumps good ole' voice conversations, but we're still going to make sure you and your caller can be heard when you do need to dial.

CNET is a global site, and our readers use hundreds of carriers all over the world. While it would be impossible to predict just how a phone will sound on your specific carrier in your city (call quality can even fluctuate depending on the time of day), we can and will point out if a call sounds especially good or terribly wrong.

To test a device's call quality, cell phone calls are made in various indoor and outdoor locations to a landline, if possible. Editors will take note of volume level, distortion and clarity.

Data speeds

As any smartphone user can tell you, fast data speeds are essential for a satisfying experience. So to give you an idea of what to expect, we run tests on the fastest available network (LTE, 3.5 HSPA+, or even 3G) that the phone uses.

Quadrant and are just a few tests we use.
Quadrant and are just a few tests we use. Screenshot by Jessica Dolcourt/CNET

As a diagnostic test, we use the app by Ookla, a free download for iOS, Android, and Windows platforms, to measure download and upload speeds. We list scores for the test in units of megabits per second (Mbps) and complete at least five test runs from the same location. And if time allows, we'll also conduct more tests in other locations.

Editors may also note real-world experience, especially if the phone tends to often run into problems throughout the testing period.

Photo and video quality

How well a handset snaps pictures has become a crucial factor in making a buying decision.

At a basic level, any camera consists of three main components: a lens, a sensor, and the electronics that coordinate the whole show. The best phones, and cameras in general, quickly capture correctly exposed images with natural and lifelike color. They can also deal with myriad lighting conditions, be it low light indoors, weak winter sun, or strong backlighting with silhouetted objects. A phone should also not be fazed by fast-moving subjects like athletes, children or pets.

An example of a studio still-life shot. Brian Bennett/CNET

With every phone, we take at least three photos on automatic mode: one of an indoor studio still life, one outdoors in daylight, and another indoors under everyday office or home conditions. If a handset boasts a full-featured camera app with plenty of extras such as a wide range of shooting and scene modes, plus special color filters, that's icing on the cake.

Moving kids are tough to capture. Brian Bennett/CNET


When a phone introduces a new hardware or software feature, especially one that comes with a particular claim, we test that addition as much as possible. For example, new mobile payment system or wireless charging addition -- so long as we have the tools (i.e., local vendors accept the mobile payments).

Star rating and scoring

To complement the written review, an algorithm calculates the final phone score based on these sub-ratings: design, features, camera, performance and battery life. Although it's infrequent (and occurs only after much discussion), editors have the ability to slightly adjust the calculated rating to take into account special features, or perhaps an extra credit bonus if, say, the handset is worth more than the sum of its parts.

Featured Video
This content is rated TV-MA, and is for viewers 18 years or older. Are you of age?
Sorry, you are not old enough to view this content.

The one thing every refrigerator owner should know

One key factor determines how long your food stays fresh (and how much you end up wasting). Sharon Profis shares a few refrigerator organization tips everyone should know on "You're Doing it All Wrong."

by Sharon Profis