How we test: Cell phones
Sit tight as we break down how CNET puts phones through their paces and how we make evaluations of what's good and what's not.
From the most basic handsets to ultra-advanced smartphones that are essentially pocket computers, mobile phones are complex devices. They reach out to the cloud, surf the Internet, play music, run countless apps, connect to social-networking services, are messaging masters, and, yes, they even make voice calls.
With so many features, operating systems, and hardware designs available, it's no wonder choosing a new cell phone isn't easy. But before you select your next personal communication companion, find out what criteria we scrutinize most when we get our hands on a fresh phone for review. And be sure to check out CNET's cell phone buying guide.
Design and usability
The first things we notice about a cell phone are its physical attributes -- its size, weight, thickness, and the materials used to construct it. In general, handsets crafted from metal and glass feel more premium. Of course sophisticated fabrication techniques such as molding a phone from a seamless slab of plastic, sturdy not flimsy, can lend an air of luxury as well.
Then we consider what type of device it's meant to be, whether a flip phone designed primarily for voice calls, a rugged machine armored to withstand shocks and drops, or a full-fledged smartphone running apps and a true operating system.
Next, we examine the screen, taking into account such things as screen size, resolution, brightness, contrast, and color accuracy. Button layout plays a big role in a phone's usability, as do any special software enhancements (or hindrances) that manufacturers add on top of its stock interface. These points may all sound like minor quibbles, but they can add up to really affect whether you form a true bond with your handset.
For serious smartphone shoppers, nothing quite delivers purchase validation, or bragging rights, like a handset's processor performance. These days, quad-core chips are king, whether made by CPU specialists such as Qualcomm, or built in-house by handset manufacturers like Samsung.
To scientifically measure a phone's raw computing prowess, we run synthetic benchmark apps designed to test processor speed. Freely available for download from the Android Play market, Linpack has two ways to put a smartphone through its paces: multithread and single-thread options.
Reported in units of MFLOPS (mega-FLOPS, or million floating-point operations per second), quad-core handsets consistently score higher on the multithread version of the test than dual-core handsets. Another test we use is the tried-and-true Quadrant benchmark, also living as a free download in the Android Play store. We typically run the full form of Quadrant Standard Edition, and have found that it really puts mobile devices through the wringer.
Quadrant has many facets meant to challenge a device on multiple levels, these being subsystems for memory, both 2D and 3D graphics, input and output, along with CPU. Currently the highest Quadrant score we've logged was turned in by the(8,165 MFLOPS) which runs a 1.5GHz quad-core Snapdragon S4 Pro complemented by 2GB of RAM.
A cell phone's battery life is just as critical, or even more critical, than its processing power. To get a handle on a phone's longevity and how much staying power it offers we take a few approaches. On Android devices, we run our own custom app that records how long a phone can play an HD video before it dies. To eliminate as many variables as possible, all wireless radios are switched off and volume and screen brightness levels are set to 50 percent. On Windows Phone and iOS devices, we load a movie file and record how long the phone will last before the battery runs dry.
The other option is to log a handset's talk time over its cellular connection. This test is particularly handy for smartphones that can't run the CNET Labs video battery drain test app or for simple feature phones. Scores for both tests are reported in total hours and minutes a phone runs before it shuts down.
While we live in a world where text-based communication is on the rise, good old voice conversations and the way a cell phone conducts them still matter. To test a device's call quality, cell phone calls are made in various locations to a landline. First we place calls from a controlled indoor office environment to assess how clean the connection is. Other factors such as volume level, distortion, and audio compression are noted, too. In all reviews, we include an audio sample that we recorded on a landline.
We also listen for how voices sound through the handset's speaker phone, both ours and people speaking to us. Finally we make phone calls during the test period from noisy environments such as busy street corners and restaurants just to be sure. Additionally each phone reviewer records his or her own voice on a call placed through the test handset to a special landline, and we embed the recording in the review.
As any smartphone user can tell you, fast data speeds are essential for a satisfying experience. So to give you an idea of what to expect, we run tests on the fastest available network (4G LTE, 3.5 HSPA+, or even 3G) that the phone uses.
We use the SpeedTest.net app by Oookla, a free download for iOS, Android, and Windows Phone platforms, to measure download and upload speeds. We list scores for the test in units of megabits per second (Mbps) and complete at least five test runs from the same location. And if time allows, we'll also conduct more tests in other locations.
Overall speed and handling
Benchmarks aren't everything, so we'll also test a smartphone to see how it handles a number of operations under normal circumstances. These trials include checking boot time, how fast a phone powers up from a fully inactive state, and its camera boot time. We also record how quickly a device can fire up mobile and full desktop versions of Web pages, plus observe how long it takes to download a mobile application.
Photo and video quality
I often hear the phrase that "the best camera is the one you have on you." As someone who respects the power that dSLRs and even Micro Four Thirds cameras bring to the table, digging through the contents of my bag each time I need one is a pain. Grabbing my phone is usually way more convenient. And as the cameras that phones pack steadily improve, how well a handset snaps pictures has become a crucial factor in making a buying decision.
At a basic level, any camera consists of three main components: a lens, a sensor, and the electronics that coordinate the whole show. The best phones, and cameras in general, quickly capture correctly exposed images with natural and lifelike color. They can also deal with myriad lighting conditions, be it low light indoors, weak winter sun, or strong backlighting with silhouetted objects. A phone should also not be fazed by fast-moving subjects like athletes, children, or pets.
We take at least three photos with each phone we review, one of an indoor studio still life, one outdoors in daylight, and another indoors under everyday office or home conditions. If a handset boasts a full-featured camera app with plenty of extras such as a wide range of shooting and scene modes, plus special color filters, that's icing on the cake.