X

T-Mobile down: Carrier says calling and texting outage was caused by 'IP traffic storm'

Calling and texting were restored after widespread network issues Monday.

Eli Blumenthal Senior Editor
Eli Blumenthal is a senior editor at CNET with a particular focus on covering the latest in the ever-changing worlds of telecom, streaming and sports. He previously worked as a technology reporter at USA Today.
Expertise 5G, mobile networks, wireless carriers, phones, tablets, streaming devices, streaming platforms, mobile and console gaming,
Eli Blumenthal
4 min read
tmobile-phone-2

T-Mobile had nationwide issues Monday. 

Graphic by Pixabay/Illustration by CNET

If your T-Mobile service was down on Monday, you were not alone. The third-largest wireless carrier had widespread issues across the country impacting the ability to make calls and send text messages starting shortly after 9 a.m. PT Monday and ending more than 12 hours later. 

At 10:03 p.m. PT Monday, Neville Ray, T-Mobile's president of technology, tweeted that texting and calling were once again working. 

Customers across the country took to Twitter to note the outage, with T-Mobile and #TMobiledown rising to the top spot on the site's US Trending Topics for several hours Monday. The main issue appeared to be with calls and texts, with users saying that data was working normally. 

After speculation throughout the day blaming the outage on a network configuration gone bad or a distributed denial-of-service (DDoS) attack, T-Mobile CEO Mike Sievert wrote in a blog post late Monday that the outage was caused by an "IP traffic related issue that has created significant capacity issues in the network core." 

T-Mobile confirmed to CNET that the issue wasn't a DDoS attack. 

"I can assure you that we have hundreds of our engineers and vendor partner staff working to resolve this issue," Sievert continued, adding that "our team will be working through the night as needed to get the network fully operational."

In an update Tuesday, Ray provided more details, saying the outage was triggered by a fiber circuit failure that T-Mobile leases from a third-party provider in the Southeast. The carrier has redundancies set up to handle this kind of issue, according to Ray. But in this case, the redundancy failed and created an "overload" situation.

"This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls," Ray wrote, referring to Voice over LTE calls. Ray said VoLTE and text in all regions were fully recovered by 10 p.m. PT Monday. 

Ray acknowledged in a tweet at 1:18 p.m. PT Monday that the carrier was having an outage and said that the carrier "hopes to have this fixed shortly." 

In a tweet sent shortly after 3 p.m. PT, Ray said the carrier was still working to fix calls and texts, while recommending that users use apps like FaceTime, WhatsApp and iMessage to communicate. In his post, Sievert echoed that recommendation. 

Those apps, unlike traditional SMS text messages or voice calls, send messages and calls through the data side of the network that was still operational. 

In tests on Monday afternoon, T-Mobile's data services appeared to be working normally in northern New Jersey, though I was unable to send text messages and had issues making calls on a OnePlus 8 5G phone. 

A CNET editor in New York City was able to send texts and iMessages using an iPhone , but calls were not working on either device. An editor in the San Francisco Bay Area noted that calls weren't working over Google Fi, the cell phone service offered by Google that relies on T-Mobile, Sprint and U.S. Cellular's respective networks. Data and texts over Fi, however, were working. 

In addition to Google Fi, T-Mobile is the underlying network providing service for several other carriers such as its Metro prepaid brand as well as for Mint Mobile and Simple Mobile.  

In tests early Tuesday, the OnePlus 8 5G in New Jersey was able to make and receive calls and text messages as well as browse the web and open videos on YouTube over 5G.   

Downdetector.com, a site where people can report outages, noted issues with all major wireless carriers -- AT&T, Verizon, T-Mobile and Sprint.

AT&T and Verizon each said that their respective networks were operating normally and without issues. Tests in northern New Jersey on AT&T and Verizon phones showed no issues with calls, texts or data except when trying to text or call a T-Mobile phone.  

"Verizon's network is performing well. We're aware that another carrier is having network issues," a Verizon spokeswoman told CNET in a statement. "Calls to and from that carrier may receive an error message." 

Verizon also took issue with Downdetector's spreading that its network is having outages. "Sites such as Downdetector.com utilize limited crowdsourced data drawn from sample social posts which are often statistically insignificant or factually incorrect," the spokeswoman said. 

"A lot of factors can contribute to a false report on a third-party website," adding that by simply aggregating this data "the result can be faulty reports of network performance interruptions causing wide-spread miscommunication for wireless users."

Sprint, which is now owned by T-Mobile, didn't respond to a request for comment.  

Speaking Tuesday, FCC Chairman Ajit Pai called the T-Mobile outage "unacceptable" and said the FCC will be investigating. "We're demanding answers -- and so are American consumers," Pai tweeted.