Why flawed broadband speed tests have devastating consequences

If government policy is based on faulty data, everyone loses.

Marguerite Reardon Former senior reporter
Marguerite Reardon started as a CNET News reporter in 2004, covering cellphone services, broadband, citywide Wi-Fi, the Net neutrality debate and the consolidation of the phone companies.
Marguerite Reardon
6 min read

The question of just how fast your home internet service is seems pretty straightforward. Unfortunately, how the broadband industry gets at the answer is messy and complicated, and over the last few weeks, that's caused controversy. 

On Thursday, The Wall Street Journal published an investigative article accusing several broadband companies, including AT&T, of trying to make their services look better by manipulating information in the Federal Communications Commission's Measuring Broadband America report, which is supposed to show whether broadband service performance matches advertised speeds. AT&T has denied the claim. 

This comes less than two weeks after the FCC reworked a $4.5 billion subsidy program for rural mobile broadband service because it found that several wireless carriers had grossly overstated their coverage. Add to all this the FCC's ongoing struggle to produce maps that accurately show where fixed broadband service is and isn't, and you might reach the same conclusion some experts have, namely that much of the data the agency is using to set policy is questionable.

"This is yet another example that explains why the FCC's data has been so poor for so long," said Josh Stager, senior counsel for New America's Open Technology Institute. "It's really an issue of the fox guarding the hen house, with much of the data that the FCC relies on being supplied by broadband and wireless carriers themselves."

Watch this: How to solve the rural broadband problem? Fix the maps

The stakes are high. The FCC uses data it collects to produce reports, such as the Measuring Broadband America and the Broadband Deployment reports, to set policy and determine where to deploy resources to promote broadband adoption. Much of the data the FCC gets to populate these reports is supplied by the broadband and wireless companies themselves, or in the case of the speed test, a third party that also contracts with these companies. The result is information that often paints a rosy picture of wireless and broadband in the US. 

Though The Wall Street Journal article singled out the broadband speed test, there have long been complaints that the information collected to show where fixed and mobile broadband service is located is flawed. The issue around flawed mapping data has come to a head in the last several months in Congress, where Republicans and Democrats alike from rural regions of the US have lashed out at the FCC, demanding the issue be fixed. 

Some of the problems can be attributed to the methodologies used to collect the data. For instance, in mapping fixed broadband the FCC has been criticized for asking carriers to provide more granular data. But critics also charge that relying on carriers to self-report information can lead to problems. Earlier this month, the FCC found that three major US wireless carriers, Verizon, T-Mobile and US Cellular, had misstated their wireless coverage in several rural areas.

"So we've got carriers exaggerating coverage for mobile broadband, flawed methodology producing bad maps for fixed broadband, and unreliable numbers on the speed of broadband. What's left?" said Gigi Sohn, an advisor to former FCC Chairman Tom Wheeler and a distinguished fellow at Georgetown Law Institute for Technology Law & Policy. "If there is no cop on the beat, the carriers will just make it like they're doing awesome and no need for any regulation or oversight."

Measuring broadband speed is hard

The FCC began the Measuring Broadband America program in 2011 in an effort to compare actual speeds customers were receiving with advertised speeds promised by providers. A speed test was supposed to provide more-verifiable information than self-reported information from broadband providers.

But finding out what's actually happening on the ground isn't as straightforward as it sounds.

"Measuring broadband speeds is different from taking a ruler and measuring how long your desk is," said John Horrigan, formerly the head of research for the FCC's National Broadband Plan and a former research director with Pew Research Center's Internet and American Life Project. "It's not precise and it depends on the tool used, the sample choosing to participate, and where in the network you're measuring."

There are several ways in which you can try to measure broadband speeds. But none of them is perfect, since there are so many factors that determine network performance, such as time of day to account for network congestion as well as equipment used to access the internet. 

One popular approach to measuring network performance is crowd-sourcing, using apps from companies like M-Lab or Ookla, which let anyone on the internet, with a click of a mouse, determine the speed of the network they're on at any given moment. This data can be useful, but it also has drawbacks. Users connected to Wi-Fi instead of an Ethernet connection may report slower speeds. The processing power of one's computer or the browser used can also be a factor. 

Other tests from companies like Akamai, which records the speed of content delivery on its network, can offer some insight into network performance. Microsoft also recently examined this issue. Because its software is installed on millions of computers in the US, it was able to calculate broadband speeds by measuring the time it took for systems to update software. It found that 162 million Americans don't have broadband speeds that meet the 25Mbps download speed definition of broadband. 

For the FCC's testing, it contracted with a company called SamKnows, which places a "white box" in a consumers' home to measure the flow of traffic. The device is at the router, so it tests the connection right as it enters the home, before Wi-Fi, or before the service reaches an end user's computer. The device also measures other network attributes, such as latency and jitter. 

But this methodology is also far from perfect. For one, the 12,000 households participating in the sample aren't selected randomly. They're volunteers and aren't representative geographically of the US population. 

Second, as The Wall Street Journal reported, SamKnows has commercial relationships with many of the carriers it's reporting on to the FCC. Also, as part of the testing, it notifies broadband providers in advance which households the FCC will test, giving companies the opportunity to improve their service in certain markets ahead of the testing. 

The Wall Street Journal also reported that some broadband companies, like AT&T, convinced the FCC to omit certain unflattering data from its report. Specifically, it says AT&T asked the FCC not to include speed test data from its older DSL service in order to boost its overall performance numbers.

But AT&T says it asked the FCC to remove results from its first-generation DSL service because it was an outdated service and didn't meet the FCC's criteria for inclusion in the report. The company said it no longer marketed the service to consumers and that it accounted for less than 5% of the subscriber base, two requirements necessary for the FCC's report. 

AT&T has since said it'll drop out of the FCC's testing and instead do its own testing. 

"AT&T developed a best-in-class tool to measure its consumer broadband services," the company said in a statement. "This tool measures performance on all AT&T IP broadband technologies and is more accurate, versatile and transparent. For these and other reasons, our tool provides better and more useful information to our customers."

For its part, the FCC called the Measuring Broadband America program "unique" and said it provided useful information.

The program "relies on the voluntary participation of internet service providers and the expertise of the FCC's engineers," FCC spokesman Neil Grace said. "That open, transparent process makes the program unique and will continue to enable the program to improve, evolve, and provide meaningful results as we move forward."

Other metrics

The FCC's critics say that speed data is just one element of a more comprehensive view that's needed to ensure the agency has the right policies in place to promote broadband adoption. The FCC is still working on getting a clearer picture of where broadband and wireless service exists today and where it doesn't. 

Two other areas where the FCC still lacks insight are how much broadband costs and why consumers choose to subscribe to a particular service. Seeking information on the cost and affordability of broadband in addition to how service is used and where service is available and whether actual speeds match advertised speeds is just as important in crafting broadband policy. 

"It would be great if the FCC could dig deeper to really find out not only about the customer experience, but how people are using their broadband," Horrigan said. 

Stager said pricing information, which varies greatly market by market for fixed broadband, isn't something providers currently report to the FCC. But he said the information would be very helpful for policy makers and consumers. 

"You rarely hear consumers complain that their internet isn't fast enough," he said. "But they do complain that their service is too expensive and that there are too many outages."