Sometimes the Internet just breaks

Last night and this morning I couldn't get to my personal website. Other websites and email worked just fine. The website itself wasn't broken ("down" is the official nerd term), the Internet was.

A great service for pinpointing a problem like this is available at siteuptime.com. Their free Quick Check (shown below) can be used to test the availability of a website from New York, Chicago, San Francisco and/or London. The HTTP (website protocol) tests of my site showed that it was fine when accessed from all four cities.

As a politician referred to it, the "tube" between New York (where I was) and Florida (where the site resides) had sprung a leak.

The path traveled between any two computers on the Internet can be long and convoluted. Amazingly so. Fortunately, the underlying transmission protocols (TCP/IP) include a debugging command for just this type of routing problem. On Windows it is called "tracert", on Linux it is called "traceroute". I'm not a Mac person, but, according to this Apple KB item, it's also called "traceroute" on OSX where it is part of the Network Utility.

Traceroute shows every router between you and another computer on the Internet. It also shows the time it took for data to get to these intermediate routers, but that's usually not an issue. Below is an edited sample of a Windows XP traceroute between my New York computer and CNET.

C:\Documents and Settings\userid>tracert cnet.com
Tracing route to cnet.com[216.239.122.102] over a maximum of 30 hops:
    ... [removed]
10 10 ms 10 ms 11 ms ae-13-69.car3.NewYork1.Level3.net [4.68.16.5]
11 11 ms 10 ms 10 ms att-level3-oc192.NewYork1.Level3.net
12 50 ms 51 ms 51 ms tbr1.n54ny.ip.att.net [12.123.0.90]
13 49 ms 48 ms 50 ms cr2.n54ny.ip.att.net [12.122.16.149]
14 51 ms 52 ms 49 ms cr2.wswdc.ip.att.net [12.122.3.38]
15 49 ms 50 ms 53 ms cr1.attga.ip.att.net [12.122.1.173]
16 51 ms 49 ms 50 ms cr2.dlstx.ip.att.net [12.122.28.174]
17 50 ms 50 ms 52 ms tbr2.dlstx.ip.att.net [12.122.18.214]
18 51 ms 51 ms 50 ms 12.122.100.97
19 64 ms 99 ms 52 ms 12.87.121.22
20 51 ms 50 ms 48 ms c18-sha-redirect-lb.cnet.com [216.239.122.102]

The first column is a sequence number, the next three columns are timings and the last column is the name and/or IP address of an intermediate router. All told, the test data traveling from me to CNET made 19 intermediate stops. At least it did this time, the route is not fixed and will change over time.

Note the first line of output that says tracing stops after a maximum of 30 hops. "Hops" refers to an intermediate router. On Windows XP, the assumption is that there normally aren't more than 30 routers between you and another computer.

When things go bad, traceroute shows asterisks. Below is an edited sample of the traceroute between my home computer and my Florida-based website at the time of the problem.

C:\Documents and Settings\userid>tracert www.michaelhorowitz.com
Tracing route to michaelhorowitz.com [208.84.150.101]
  over a maximum of 30 hops:
    ...
 9 13 ms 16 ms 14 ms ae-1-0.pr0.dca10.tbone.rr.com [66.109.6.165]
10 13 ms 15 ms 16 ms 64.132.69.61
11 76 ms 72 ms 73 ms 64.128.245.106
12 72 ms 73 ms 75 ms 64.128.245.106
13 76 ms 75 ms 83 ms core2.rapidvps.net [66.97.162.162]
14 72 ms 74 ms 73 ms moors.rapidvps.net [208.84.151.160]
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.

The router at moors.rapidvps.net is not necessarily the problem, it was the last normally functioning router. The real problem lied somewhere beyond it.

I sent the traceroute output and a description of the problem to the company hosting my website, and the problem turned out be with a firewall. For some reason, a firewall under their control was purposely blocking my IP address.

Every computer on the Internet is addressed by a number called an IP address. Large companies are given permanent IP addresses. Consumers, such as myself, normally share a pool of IP addresses given to our Internet Service Provider*. An ISP doles out their IP addresses to customers on an as-needed and rotating basis.

What probably happened was that another customer of my ISP was doing something bad or suspicious and that got them blacklisted in the firewall of my hosting company. But the IP address came from a shared pool and sometime yesterday it got assigned to me.

Go figure.

*I am referring here to public IP addresses, those visible on the Internet. Some IP addresses are reserved for internal use only and are referred to as "private" IP addresses. On a Local Area Network, the IP addresses assigned to each computer normally come from the private group. The most popular private IP addresses are 192.168.x.x and 10.x.x.x. Even if something gets mis-configured, routers on the Internet are programmed to throw away any packets they get with an private IP address.

See a summary of all my Defensive Computing postings.