Scaling Amazon EC2 to 512 active nodes

Wondering how well Amazon handles cloud-computing demands? The answer, it seems, is very, very well.

Among all the cloud-computing hype, one thing that hasn't been evident is just how far you can scale across a provider. I haven't seen any other vendor come near's ability to reach the massive scale that the cloud itself connotes.

Max Gorbunov from Grid Dynamics ran a 512-node Monte Carlo simulation to find out how well Amazon EC2--short for Elastic Compute Cloud--would perform. He used GridGain, a Java-based open-source grid computing infrastructure for the test.

All in all, this test clearly shows that you can utilize Amazon's massive infrastructure for high-end processing with an acceptable performance hit. And while I am sure I am oversimplifying the difficulty associated with getting this all set up, based on the development notes it seems like it was fairly easy (at least for Max.)

GridGain deployment on EC2
GridGain deployment on EC2 Grid Dynamics

The test consisted of a custom setup based on open-source components including GridGain and Open MQ running on the default EC2 Fedora Core 8 distribution and using a custom test harness developed for this project.

The performance degradation of 3 seconds (about 20 percent) should be considered minor given roughly 250-fold increase in scale. The curve rises two times: in the ranges 2-8 and 256-512, while 8-256 remains almost flat.

While I don't have the math available to me about what this cost to run (I would guess well less than $5,000), the way you might have done something like this in the past would have involved expensive software like DataSynapse or very technical open-source tools like the Globus Toolkit.

It's a new world out there.

Thanks to Eugene at TSS for the pointer.

Featured Video
This content is rated TV-MA, and is for viewers 18 years or older. Are you of age?
Sorry, you are not old enough to view this content.

The WRT1900ACS is Linksys' new best Wi-Fi router to date

CNET editor Dong Ngo compares the new WRT1900ACS and the old WRT1900AC Wi-Fi routers from Linksys. Find out which one is better!

by Dong Ngo