Twitter outage caused by human error, domain briefly yanked
CNET has learned a Twitter outage that left millions of users fuming when they couldn't click on links came from an unlikely source: a "phishing complaint" sent to an Australian firm.
An outage that broke hyperlinks on Twitter yesterday evening originated with a simple human error at a Melbourne, Australia-based hosting firm that was responding to an abuse complaint, CNET has learned.
Twitter last year began to abbreviate all hyperlinks using its t.co domain name -- which had the side effect of introducing a central point of failure where none existed before. That failure happened last night around 11:30 p.m. PT when t.co went offline, meaning millions of Twitter users received "non-existent domain" errors when trying to follow links.
A spokesman for Melbourne IT, a domain name registrar which Twitter uses for t.co, told CNET this afternoon that: "Yesterday in the process of actioning a phishing complaint, our policy team inadvertently placed the t.co domain on hold. The error was realized and rectified in approximately 40 minutes and t.co links again began working."
At first, it seemed as though the problem was caused by Dyn, a New Hampshire-based company that provides domain name system connectivity for Twitter's t.co link shorting service as well as Zappos and Etsy. But Tom Daly, Dyn's chief scientist, said it was "an issue with the upstream parent zone, .co, the country code domain for Colombia."
That would be .CO Internet, which hasas the hip domain suffix for the next wave of Internet companies. Some other adopters: Angel.co, Launch.co, Bumpercrop.co, Tailored.co, Gourmair.co, and Cardflick.co.
A spokeswoman for .CO Internet said that the t.co domain "did appear to be put on ClientHold status" last night -- something that registrars have the ability to do -- "meaning that the domain was removed from the zone during that time period."
In the rarefied world of people who keep the domain name system humming along, ClientHold is a special status usually reserved for customers who don't pay their bills on time. An Internet standard known as RFC 5731 specifies that if a domain falls into ClientHold status, connectivity "information MUST NOT be published," meaning it becomes unreachable.
Melbourne IT's spokesman, Tony Smith, added in e-mail:
Melbourne IT's domain policy team is active in investigating requests from registries and the public about potential abuse of domains (including phishing sites) registered via Melbourne IT or its resellers.
We take each request seriously and if our investigations show that there is clear evidence of phishing or malware we can shut the domain down to protect consumers... We have been in contact with Twitter, and are reviewing our investigation and verification processes.
A Twitter representative declined to respond to questions from CNET, saying "the issue was resolved last night." A status update early this morning from Twitter acknowledged that "t.co links were not resolving" for some users.
Twitter's original announcement about t.co last year predicted that wrapping all links with t.co will boost analytics and give "us a means to protect users from malicious sites and scams." What it didn't say is that, by routing all outbound links through t.co, it added a central point of failure that didn't exist before.
Mikko Hypponen, chief research officer for F-Secure, suggested last night that t.co's downtime "illustrates how shortlinks make the Web more fragile and harder to archive."
In the annals of Web site glitches and Twitter outages, last night's problem was a hiccup, not an epic failure. It lasted less than an hour and, while users in Asia and Australia may have been peeved, it happened in the middle of the night in North America and South America and before most of Europe was awake and online.
But the outage demonstrates the fragility of complex systems. It's a little like a recent outage in California that affected another single point of failure: Electricity failing at a single refinery caused gasoline prices to jump to, in some areas, over $5.
If t.co goes down again, there is a workaround. You can replace a dot or period in a link with its HTML entity equivalent (in this case, .) and Twitter will display the link without shortening it -- at least for now.