Scaling Twitter redux--the ESB should be your best friend
Scaling a message-based system is a common enterprise problem. Web 2.0 needs to get a little enterprise-like to make things work better.
As we Twitt-iots sit around bemoaning the fact that we can't send each other useless junk on a flaky service, I thought I would take this chance to address the notion that this message-scaling problem is new.
It's not. It's very common, and it can be solved.
Scaling a messaging platform is why IBM sells a boatload of MQ series, why the AMQP protocol was developed, and why JMS is nearly ubiquitous. Pretty much every large enterprise has similar scale issues related to messaging, especially in financial services. But they don't have downtime, and if they did as frequently as Twitter, the people behind them would all get fired.
This is a topic I actually know something about. (Disclosure: my company develops an open-source enterprise service bus, or ESB, called Mule.) All of our use cases involve some kind of complicated messaging architecture, whether it be Web-service based, publish and subscribe, one to many, direct connection, etc. And most deal with data transformations and a vast array of protocols.
In my view, what Twitter needs is to adopt a bus-type of architecture that separates the transport from the application and uses a middleman to process the transactions. This is a very common enterprise scenario that needs to be applied. This is what an ESB does.
One example in the Web 2.0 world is OpSource, which is delivering billions of transactions a day as an SaaS provider using the ESB at the crux of the transactions. This is much larger than Twitter and has actual revenue impact. Maybe OpSource can host Twitter?