Translating Twitter's business speak on outages

So intelligent life does inhabit Twitter headquarters after all. Say hallelujah. Say amen.

Breaking a prolonged silence, Twitter on Thursday issued a detailed post on its corporate blog explaining the repeated outages affecting the service and what the company plans to do to resolve them.

Until now, Twitter has proved impervious to everything from polite pleas to public goading--nothing, it seemed, was enough to convince these folks to engage in conversation with the Twittering public.

Maybe the prospect of finally getting their hands on a multi-million dollar investment put management in a better frame of mind. Up until now, these folks hunkered in the bunker each time the service went down. Unfortunately, Twitter interruptions have become so regular that they stopped being news weeks ago.

Whatever the reason, Twitter decided to turn the page, and the good news is that there's progress in building more scalablility into the system. The not-so-good news is that Twitter's only at the beginning of what sounds like a rewiring nightmare.

Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency's sake, Twitter was built with technologies and practices that are more appropriate to a content management system. Over the last year and a half we've tried to make our system behave like a messaging system as much as possible, but that's introduced a great deal of complexity and unpredictability. When we're in crisis mode, adding more instrumentation to help us navigate the web of interdependencies in our current architecture is often our primary recourse. This is, clearly, not optimal.

Translation: We've got a ton of work ahead. Flawed strategic thinking about what the service is and what it might become has put Twitter in a hole of its own making.

Our direction going forward is to replace our existing system, component by component, with parts that are designed from the ground up to meet the requirements that have emerged as Twitter has grown. First and foremost amongst those requirements is stability. We're planning for a gradual transition; our existing system will be maintained while new parts are built, and old parts swapped out for new as they're completed. The alternative--scrapping everything for "the big rewrite"--is untenable, particularly given our small (but growing!) engineering and operations team.

Translation: Get used to periodic brownouts. Rome wasn't built in a day--and neither was it torn down in a day. This is going to take time to fix, but the project will result a stronger infrastructure that can handle Twitter's furious success.

We keep an eye on the public discussions about what our architecture should be. Our favorite post from the community is by someone who's actually tried to build a service similar to Twitter. Many of the best practices in scalability are inapplicable to the peculiar problem space of social messaging. Many off-the-shelf technologies that seem like intuitive fits do not, on closer inspection, meet our needs. We appreciate the creativity that the technical community has offered up in thinking about our issues, but our issues won't be resolved in an afternoon's blogging.

Translation: Everyone's entitled to their opinion but you have no idea what we're up against. Try to walk around in our shoes for a while before shooting off your mouth about what we ought to do. And have a nice day!

We'd like people to know that we're motivated by the community discussion around our architecture. We're immersed in ideas about improving our system, and we have a clear direction forward that takes into account many of the bright suggestions that have emerged from the community.

Translation: Yes, we do feel your pain. It's not as if we're sitting around all day passing around a bong (even though it sometimes seems that way). Keep those cards and letters coming.

To those taking the time to blog about our architecture, I encourage you to check out our jobs page. If you want to make Twitter better, there's no more direct way than getting involved in our engineering efforts. We love kicking around ideas, but code speaks louder than words.

Translation: Hey, we just got a massive cash infusion. If there are any wannabe Ray Ozzies out there waiting to be discovered, let's talk.