Peek at a Big Game IT 'war room' -- at Domino's Pizza

On Super Bowl Sunday, Domino's Pizza is planning to deliver millions of pizzas (it estimates that a total of 11 million slices will be delivered) and 2.5 million chicken wings. According to Domino's execs, the national game day gorge has become more than just a manner of beefing up on drivers, dough tossers, and yes, beef -- it's also quite an undertaking in the information technology department.

According to the Michigan-based company, a third of Domino's orders come though a digital channel these days, and of course even the analog orders run through the corporate network. In 2007, for the first time, Domino's saw the need to put together a game day "defense" team on Big Game day to sit in a conference room and keep an eye on all the information systems.

That first year it was four people making sure Americans got the fuel they needed for their calorie- and beer-filled annual rite of senselessly shouting at a screen. (For the record for all future judgmental-type anthropologists, I feel no shame over the tear I shed when John Elway made that helicopter dive toward the end zone on a fateful day in January of 1998.)

Six years, millions of pizzas, and one iPhone and Android-powered smartphone revolution later, that team in a conference room in Ann Arbor now numbers more than 50 and the operation resembles something more like a NASA launch.

"It's all hands on deck," Lance Shinabarger, Domino's vice president for IT infrastructure and security tells me. "We have network operations, server administrators, network engineers, site reliability engineers, front-end developers, back-end developers, director of infrastructure, director of development, VP of global infrastructure, VP of online ordering development, VP of store operations, VP of POS development, CIO, and guest star appearances by both our CFO and CEO (all in the room)."

Just a few of the litany of items that the pizza mission control will be focusing on:

Hardware utilization (servers, firewalls, load balancer, routers, etc.)
Bandwidth utilization
Hardware failure
Application health
User experience degradation
Site response times
Security events
Social-media monitoring

Tradition holds that some of those highest-ups aren't actually there to bark out orders as chaos descends upon Domino's servers in the lead-up to kickoff. Last year the information system vice presidents and directors worked in the Domino's test kitchen making dozens of pizzas and wings for the crew.

Domino's representatives told me that preparation for this day actually begins as early as July or August. Load testing and other practice runs are just a few of the ways the team gets ready for a 300 percent spike in inbound traffic to the Domino's system just before kickoff. By the time most of those meals have been delivered, it starts all over again with another, smaller spike in orders at halftime.

"Normally you spread that order load over a four- or five-hour window, but during Super Bowl, it's those two peaks that you spend the entire year preparing for," Shinabarger explains.

Super Bowl XLVII might not be over yet (although the season was over for me when the Broncos lost in the divisional playoff round), but I'm already prepared to vote for MVP the network engineer who helps enable the delivery of millions of pizza pies in a span of just a few hours.