Scaling Amazon EC2 to 512 active nodes

Wondering how well Amazon handles cloud-computing demands? The answer, it seems, is very, very well.

Among all the cloud-computing hype, one thing that hasn't been evident is just how far you can scale across a provider. I haven't seen any other vendor come near Amazon.com's ability to reach the massive scale that the cloud itself connotes.

Max Gorbunov from Grid Dynamics ran a 512-node Monte Carlo simulation to find out how well Amazon EC2--short for Elastic Compute Cloud--would perform. He used GridGain, a Java-based open-source grid computing infrastructure for the test.

All in all, this test clearly shows that you can utilize Amazon's massive infrastructure for high-end processing with an acceptable performance hit. And while I am sure I am oversimplifying the difficulty associated with getting this all set up, based on the development notes it seems like it was fairly easy (at least for Max.)

GridGain deployment on EC2
GridGain deployment on EC2 Grid Dynamics

The test consisted of a custom setup based on open-source components including GridGain and Open MQ running on the default EC2 Fedora Core 8 distribution and using a custom test harness developed for this project.

The performance degradation of 3 seconds (about 20 percent) should be considered minor given roughly 250-fold increase in scale. The curve rises two times: in the ranges 2-8 and 256-512, while 8-256 remains almost flat.

While I don't have the math available to me about what this cost to run (I would guess well less than $5,000), the way you might have done something like this in the past would have involved expensive software like DataSynapse or very technical open-source tools like the Globus Toolkit.

It's a new world out there.

Thanks to Eugene at TSS for the pointer.

About the author

Dave Rosenberg has more than 15 years of technology and marketing experience that spans from Bell Labs to startup IPOs to open-source and cloud software companies. He is CEO and founder of Nodeable, co-founder of MuleSoft, and managing director for Hardy Way. He is an adviser to DataStax, IT Database, and Puppet Labs.

 

Join the discussion

Conversation powered by Livefyre

Show Comments Hide Comments