Meet the firm behind Verizon, AT&T's reliability, speed claims

Watch this: 4G networks: How they're tested

01:53

Apparently, it takes just two words to rile up wireless giants Verizon Wireless and AT&T.

Those words, "most reliable," set off the latest fracas between the two as both attempted to claim network superiority. With handset selection and phone plans so similar, the carriers are increasingly banking on the reputation of their service quality as the key competitive edge.

So Verizon was more than a little perturbed after AT&T co-opted its "most reliable" title. After initially taking the high ground and pointing to its own track record, Verizon added its own word to AT&T's new campaign: "misleading."

AT&T, unsurprisingly, stood by its boast.

So which side is right? CNET talked to Bill Moore, the CEO of RootMetrics, which is an analytics firm that tests the quality of each network. Verizon Wireless cites the firm's results in claiming the most reliable network, while AT&T points to its results as one of the tests confirming its speedier network. (AT&T hasn't publicly disclosed the firm that found it the most reliable.)

The world of network testing is anything but transparent. Some research firms will compile their data anonymously, which means they don't publicly disclose the results or how they came upon them. The different firms vary on how public they want to be, partially because they don't want to be seen publicly endorsing one carrier over another.

Each of the wireless carriers subscribes to individual network drive tests, partially to augment and verify their internal tests, but also to see where they stack up with each other. Firms such as J.D. Power, Nielsen, and RootMetrics conduct such tests, while publications such as PCMag also conduct their own speed tests. So when carriers do make their claims, they aren't idle boasts -- they are in fact backed by studies.

Read more: 5 things you didn't know about data testing

While RootMetrics is a younger firm that's more eager to generate a little public buzz, the large carriers take its results seriously. Moore complained about the lack of disclosure and consistency in the field, and pulled back the curtain to reveal his own firm's process.

Hitting the road
RootMetrics is nothing if not comprehensive. The firm employs dozens of workers who have driven a total of 130,000 miles, conducted 3.6 million tests, and hit thousands of indoor locations -- and that was all just in the first six months of this year.

The firm conducts tests 24 hours a day, as opposed to just working hours. When it can, it will select similar phones on different carriers and record their performance at the same time and location, providing accurate comparison data. It records everything -- including call failures, which Moore said some firms will throw out.

RootMetrics also buys its phones off the shelf, and doesn't change them aside from adding software to automate the test itself.

"There are no special privileges or plans," Moore said. "We don't alter them in any way."

It's important to test smartphones because that's how the mobile network is widely used, Moore said. That seems logical enough, but he noted that other firms would test the network using specific devices such as hot spots or USB dongles. Rather than a smartphone, which juggles multiple networks and technologies, and can drop from 4G to 3G at any moment, other firms will test specific networks -- one device for 4G and another for 3G, or one for voice and another for data.

Moore slammed that technique, calling it "very inaccurate."

"When you want to test reliability, the hardest thing is to switch from 3G to 4G or call vs. data," he said.

The firm even tests the reliability of text messages, measuring the speed of delivery and how often they failed to reach their destination. In terms of data speed, in which it crowned AT&T, the company looks at maximum speed, but also at factors such as upload speed, time it takes to get on the network, and e-mail task completion.

"Looking at the maximum speed is like looking at a car and its top speed or 0-60 speed," he said. "That's just one aspect of the car."

When evaluating, RootMetrics gives 45 percent weighting to phone calls and data connection, and another 10 percent to text messages.

The people's network test
RootMetrics offers an app that anyone can download to run tests in their own neighborhoods. The firm gets hundreds of millions of data points from hundreds of thousands of individuals.

Those results are available on the Web site. But when RootMetrics publishes a study comparing the different carriers, it only uses the results it gathers itself.

That's because the crowdsourced data is "messier" since there's no way of telling if a person is inside, outside, or in a crowd. Without phones from other carriers, there's no way to get an accurate comparison.

Moore said RootMetrics was started because there was no data that consumers could use to look at the coverage situation in their neighborhood.

"I've searched all over to find out," he said. "We're the voice of the consumer."

Moore criticized the network-testing industry for a lack of a consistency in measuring performance. That allows carriers to slice and dice and make declarations without having to share any backup information. Given that each firm has their own proprietary method of conducting tests, it's unlikely any will come to an agreement.

"There has not been and needs to be a standard," he said. "So you can't change the rules of the game."