Big iPhone headache: Waiting for AT&T activation

So I admit it: I bought an iPhone.

That was the easy part. I should have realized that if the customer service whizzes at AT&T could find a way to mar what was otherwise a perfectly pleasant experience, they would. After nearly nine hours over 36 hours, AT&T has yet to activate my iPhone, and it can't be used until they get around to it, hopefully sometime before 2010.

But I'm getting ahead of myself. Earlier on Friday, I had taken photographs of the throngs gathered outside the San Francisco Apple store at 6 p.m., and the far more sedate crowd outside the Market Street AT&T store, and decided not to bother standing in line for an iPhone that evening.

I figured I'd read some more early reviews and then pick one up sometime in the next few days after the lines were shorter. And those reviews were promising, including one that said audio quality was superior. Seeing the iPhone ripped apart was also educational.

Then, around 11:30 p.m., I read our News.com article by my colleagues Tom Krazit and Erica Ogg. They had stayed longer than I did and reported that: "Ninety minutes after Apple started ringing up sales of the iPhone at its 24-hour flagship store on 5th Avenue in New York, anyone could just walk into the store and pick up a device with a minimal wait."

Well, Apple stores were open until midnight, so why not? Around 11:40 p.m., I persuaded my wife to join me in a late-night dash to the Stockton Street store.

Yes, they had iPhones. Yes, the sales staff were exhausted after performing crowd control earlier. Yes, there were still two San Francisco policemen standing guard outside, looking slightly bored by now. But there were only two people in line in front of me, including one desperate fellow who had driven far too fast from Marin County north of San Francisco to make the midnight deadline after finding that an Apple store up there had run out of 8GB models.

So far, so good. When I got home, I plugged the 8 GB iPhone into our media-server iMac and typed my information into iTunes. I received an e-mail message at 12:10 a.m. saying: "AT&T is now processing your activation. You will receive an e-mail confirmation once your activation is complete." I had an existing AT&T account, so I figured that adding the $20/month iPhone wireless plan should take only one or two minutes.

Then I waited. And waited. And waited.

It's now 9:06 a.m., and still no change. Against my better judgment, I even took the God-help-me-now step of phoning AT&T customer support, something you should never do unless you're absolutely desperate and learn that some malcontent is running up calls to Zambia on your mobile account, and probably not even then.

Mary-Kay eventually answered. "Unfortunately, sir, you do have to wait," she wearily replied, presumably sick of having to answer this question once a minute since her shift began. "The iPhone will tell you when it is activated."

How long would this take, I asked. Five days? Two months? "I doubt that," Mary-Kay replied. "Believe you me, you're not the only one in those shoes. They did get jammed up last night. It's first-come first-served."

I began to ask her if my existing AT&T phone would continue to work, but the line seemed to go dead. It could be my sucky VoIP service, or Mary-Kay could have hung up on me. I really wouldn't blame her for iPhone ennui after dealing frustrated my-activation-hasn't-happened-yet customers all day.

Now, I've been a computer programmer longer than I've been a journalist, and I find it hard to imagine any system that should take nearly nine hours to perform a database query, do a credit check, and whatever other black box magic is necessary to make this gadget actually work. It's even less likely that the system should take this long in the middle of the night after the east coast iPhone binge should, in theory, have abated hours earlier. And I'm already an AT&T customer, too.

Occam's razor suggests that the more likely explanation is that AT&T has such antiquated computers that some poor saps in another secret AT&T division somewhere in a New Delhi office park are keystroking in my account update by hand. Seriously. We may never know, but it sounds about right.

I'm hardly alone. Some reports indicate AT&T activation is a recurring problem. One local Fox news channel article is titled "iPhone debuts with big headaches." A LiveJournal user entry buttresses my theory by reporting that the AT&T "transfer team" gets in to work at 9 a.m. PDT.

There's speculation that existing AT&T SIM cards may work with the iPhone, but I haven't tried that yet. Some posts in that thread are saying AT&T is telling customers they need to wait for 24 hours because of high volume. Some people are saying that their existing AT&T phones become unusable during the transition, though at least that hasn't happened to me so far.

It's important to stress that the iPhone can't be used for anything useful, not even iPoddish features such as playing music or movies, until activation happens.

The bottom line? Apple did everything it could to ensure that buying and setting up (and presumably using, though I can't attest to that yet) an iPhone is a pleasant experience. It succeeded magnificently.

But its key business partner, AT&T, has failed miserably. Computer companies know how to load-test a server to figure out how it will respond under unusually high demand. Why didn't AT&T do the same for its internal procedures to handle iPhone activation?

Update +14 hours @ Saturday 2:13 p.m. PT: I received an e-mail message at 12:35 pm PT from the customer-friendly address of do_not_reply@att.com saying: "We are currently processing your order... You will receive an additional e-mail when your order is complete that will provide further instructions to activate your iPhone." That's one informative message.

Update +20 hours @ Saturday 10:45 p.m. PT: I received another e-mail this evening from iPhone_Activation_Support@att.com saying: "We're sorry. AT&T has identified a problem with the information you provided. For more information, call 877-800-3701." So I did. And I waited on hold another 45 minutes. "For some reason it's not accepting the rate plan," the AT&T rep said once they actually answered the phone. And there was more bad news: AT&T Web Order Activation could do no more for me. I needed to call AT&T customer service at 800-331-0500. But, naturally, they were already closed for the evening.

Update +35 hours @ Sunday 11:34 a.m. PT: After I waited on hold the better part of an hour with AT&T customer service, Noah picked up the phone. I was his first iPhone customer. Great, I thought. He'll know exactly how to help me. My fears were justified: He was friendly yet clueless, and clung to his mistaken belief that iPhone itself had told me my existing rate plan was invalid. It took a while to disabuse him of this notion. He put me on hold and called the activation department at 877-419-4500. After some back and forth, the problem became clear: AT&T had realized, to its collective horror, I was not being charged enough under my current rate plan to permit me to add the iPhone's $20/month wireless plan to it. I had to choose one of the default iPhone rate plans, which I did. Noah cheerfully told me that "we've been told to expect 24 hours" for activation from this point on, which would make it a full 60 hours since I first plugged in my darling little iBrick.

Update +36 hours @ Sunday 12:15 p.m. PT: Noah called me back on my existing (elderly) AT&T mobile phone with news. After helping another customer with the same problem, Noah had learned that "I can actually process the activation myself right here." Whoo-hoo! He had me power-cycle the iPhone a few times, which did nothing. But then my existing (decrepit) AT&T phone stopped working and displayed the error "Unregistered SIM," which I suppose can be counted as progress, in much the same way that failure can be counted as success. Noah said I could use the iPhone's SIM card in my existing (ancient) phone. In Noah's defense, he was remarkably pleasant and went out of his way to call me back with more information. It's not his fault he's enmeshed in such a dysfunctional iPhone activation apparatus.

Update +38 hours @ Sunday 1:40 p.m. PT: After reading a few more iPhone horror stories, including one on MacSlash about horrific iPhone problems with business accounts, I began to doubt the reliability of my new friend Noah. In my mind's eye I was comparing him to an Afghan hound: eager to solve the problem but utterly lacking the cognitive skills to do so. So I popped a few Valium and called the AT&T Web Order Activation number again. After Yet Another iPhone Interminable Wait (YAIIW), Vanessa finally answered. "Because you do have a business account we cannot activate the iPhone under a business account," she told me. Ah, so poor Noah hadn't fixed things after all.

Update +38 hours @ Sunday 2:20 p.m. PT: I've been transferred to Aleecia from "business care" who promises to help me out but, as I expected, utterly failed to do so. It helps to keep your expectations low. I should say, by the way, I pay my own bill and am far too much of a peon to have a corporate account: Only CNET bigwigs have their phones billed directly to the company. But AT&T has been giving me a handsome discount of something like $1.25 a month off my phone bill because CNET is my employer, which is what apparently gummed up the works. I finally found an AT&T post buried deep in their site that says IRU accounts like mine may or may not qualify for the iPhone. SLBers and CRUs definitely can't, but some IRUs can. Get it? Anyway, it was Aleecia's bright idea to transfer me to Apple at 800-MY-IPHONE. "What I'm going to have to do is get ahold of the iPhone support themselves and to have them assist you," Aleecia said. But the Apple rep who answered had no idea what AT&T SLB, CRU, or IRUs are, and I figured was too polite to say what he really thought: "Why the hell is AT&T dumping its problems on me?" He did, however, provide me with yet another AT&T number, 866-539-1468, to call.

Update +38 hours @ Sunday 2:35 p.m. PT: YAIIWing. This must be what they mean by the "New AT&T."

Update +39 hours @ Sunday 2:45 p.m. PT: Ashley answers after something like 40 minutes, which is a record time by YAIIW standards. I resist the urge to compliment AT&T Chairman Randall Stephenson on reducing average YAIIW from a few hours to 40 minutes. This might count as an unqualified customer experience success -- except for the fact that, after 39 hours, my iPhone is approximately as useful a phone as a hunk of asphalt would be. Except the asphalt wouldn't cost $600+tax or require advanced studies in YAIIWing. Ashley replies: "I'm showing everything's OK with your account." Maybe Aleecia fixed things after all? He puts me on hold to call AT&T Web Order Activation, which is the same number I called over an hour ago in the latest round of YAIIWing.

Update +39 hours @ Sunday 3:00 pm PT: Success! Email arrives from Apple: "Activation Complete." I walk over to the media center iMac and, yep, the little iPhone is blipping away, just wanting to be picked up and fondled. I called my wife from the iPhone and asked her what I should do with Ashley, who was still YAIIWing himself, kind of recursively, and had me on hold on the VoIP line. She said to hang up (she's a ruthless corporate lawyer, you should know). I did.

Another update: We've heard back from Synchronoss, the AT&T activation contractor. They're "extremely pleased" with the way iPhone activation went over the weekend.