Scaling Amazon EC2 to 512 active nodes
Wondering how well Amazon handles cloud-computing demands? The answer, it seems, is very, very well.
Among all the cloud-computing hype, one thing that hasn't been evident is just how far you can scale across a provider. I haven't seen any other vendor come near Amazon.com's ability to reach the massive scale that the cloud itself connotes.
Max Gorbunov from Grid Dynamics ran a 512-node Monte Carlo simulation to find out how well Amazon EC2--short for Elastic Compute Cloud--would perform. He used GridGain, a Java-based open-source grid computing infrastructure for the test.
All in all, this test clearly shows that you can utilizefor high-end processing with an acceptable performance hit. And while I am sure I am oversimplifying the difficulty associated with getting this all set up, based on the development notes it seems like it was fairly easy (at least for Max.)
The test consisted of a custom setup based on open-source components including GridGain and Open MQ running on the default EC2 Fedora Core 8 distribution and using a custom test harness developed for this project.
The performance degradation of 3 seconds (about 20 percent) should be considered minor given roughly 250-fold increase in scale. The curve rises two times: in the ranges 2-8 and 256-512, while 8-256 remains almost flat.
While I don't have the math available to me about what this cost to run (I would guess well less than $5,000), the way you might have done something like this in the past would have involved expensive software like DataSynapse or very technical open-source tools like the Globus Toolkit.
It's a new world out there.
Thanks to Eugene at TSS for the pointer.