Eliminate's 3G multiplayer: How'd they do that?

Ngmoco's Eliminate for the iPhone was a groundbreaking game in two ways. One was its pay-to-score business model, which offered the game for free, but required users to have virtual energy units (which could be bought) to gain rank and equipment bonuses. The second, and more important one, was the back-end server technology that let players hop onto online matches over 3G.

This feature was not so simple to implement, as Ngmoco's director of engineering Stephen Detwiler and lead engineer James Marr mapped out during a talk at Tuesday's Game Developers Conference in San Francisco. As Marr explained it, much of the success can be tied to spending around $10,000 to license the Quake 3 engine, which had about the same bandwidth needs and latency as what can be found on most 3G networks. It also saved the company at least a month's worth of work in trying to build an engine of their own.

The real challenge ended up being getting the thing to scale. Part of that revolved around writing the server code that set up game matches from scratch. Where Quake 3's server setup had it running one process per game being played, Ngmoco had to create a game manager app that could then run several at once without tying up the computing power. This was coupled with a distribution of data centers around the world that would offer people the lowest possible latency.

As Marr explained it, 200 milliseconds is the limit of what is acceptable, and noticeable in terms of game lag. Using a system that was distributed by location, Ngmoco was able to get it down to around 150 milliseconds over 3G. That's a pretty impressive number considering it's the time from the phone, to the data center, then back to the phone.

Another part of the equation was an emergency plan for when the game was getting too many users banging on the servers at once. Ngmoco's solution was to tap Amazon's Elastic Compute Cloud (EC2) to take on additional, virtualized game managers. The only problem there, Detwiler said, was that it ended up costing too much to keep that going as a full-time replacement. "It turns out compute time for EC2 is cheap, but bandwidth is not," joked Detwiler. Ngmoco compensated by collocating its own servers, which cut what the company would have paid for EC2 in half.

Ngmoco also cut bandwidth costs by expediting how it was setting up matches. Its system, which Marr said borrowed heavily from Bungie's Halo 2 on the Xbox, would weed out waiting players based on a number of factors, including estimated skill, quality of their connection, and the desired party size. The longer the process took, the looser these parameters got, which Marr said would keep the amount of time needed to find a match to about a third of what most other gamers were offering. For Eliminate that magic number is around three minutes from the time the user hits the play button to when the system hops them into a game.

So can other developers do this? Detwiler and Marr seem to think so, even though it ended up taking Ngmoco eight months to build and test Eliminate instead of the five it had planned. "It turns out this is a really hard problem," Detwiler said. "Using existing systems--where possible, is important."