How I got burned by Twitter's API, why it matters, and how to fix it

Last week I discovered I was using Twitter too much. After an hour online with Twhirl, I got this message in the app: "Limit exceeded, paused 5 min." The error condition cleared up shortly, but the next morning, after just a few minutes, it came back and did not resolve. I had to go back to accessing Twitter via the Twitter.com site, where I still had access.

I had been bitten by a deficiency in Twitter's API (application programming interface), which allows alternate interfaces like Twhirl to work at all. The problem, it turns out, is temporarily fixable for end users, but Twitter is going to need to recode its API if it wants to make the Twitter platform for third-party apps and services more robust. And other Web 2.0 architects would do well to study this issue so they don't fall into the same hole.

I was schooled on the ins and outs of the Twitter API in part by my followers on Twitter, but also by Oren Michels, CEO of Mashery, a company that offers API services to Web 2.0 companies. Here's the lowdown:

The Twitter service limits the number of updates a user can get from it to 70 per hour. There's no limit if you're using Twitter.com, but if you want to use Twhirl or Friendfeed or Flock to read your Twitter account, the Twitter service keeps track of how many requests you're sending it and cuts you off if you exceed the limit.

The problem is that all the Twitter apps you use count to your total. It's cumulative. Once an app, or more importantly, a Web service, has your Twitter log-in credentials, it can keep requesting Twitter updates on your behalf even you aren't using the service anymore. And that's what happened to me: I use Twhirl heavily, but not that heavily. It's the other dozen or so services I've signed up to over the past few months that were pinging Twitter for me and using up my allotment of updates.

There's a temporary fix for people in my boat, and it's very simple: Change your Twitter password. That will cause all the previously-configured connections to Twitter to break, and they'll stop using up your API calls. Just reconfigure the apps you do want to use with your new password and you're back in business. Thanks to my Twitter friend Scott Mahan for this tip. But this is a hack, and an inelegant one.

A better solution would be for Twitter to authorize access to its API by user and application, so users, or the Twitter team itself, could throttle or disconnect just the apps that hog the API without cutting people off from apps that are working fine.

The new OAuth protocol is designed specifically to solve this issue for all services. It combines user credentials with application keys to let developers and users control access to online services not just based on user ID, but which app wants to use the service. Fire Eagle, the new location-caching service by Yahoo, uses OAuth in just this way, and it has some nice frills on top: It gives users a dashboard showing them which apps they've authorized to their Fire Eagle data. It also allows credentials to apps to expire after a certain time, so you're not left with a lot of forgotten apps pinging your personal data.

OAuth is a lot like OpenID (and in fact is modeled after OpenID in some respects) except it's designed to authorize program-to-program communication, not user-to-service log-ins.

Flickr, a pre-OAuth service, also makes for a good example. When you want to give an app access to your Flickr account, the app kicks off a process on Flickr itself in which you authorize the specific app to act on your behalf.

If Twitter is to survive (and potentially make money) as a service provider, it's going to need some way to let apps get priority access to data via an API, and its current log-in-based throttle won't cut it.

Since we're moving toward a Web world in which services and social networks are far more important than their interfaces--and where the best interface builders are often not employed by the companies that make the services they are writing apps for--getting APIs right is critical. Twitter, as much as I love it, got it wrong. I hope architects of new Web 2.0 services don't repeat the same errors.