eBay's site began limping last night, and at 10:30 a.m. PT today engineers brought the site down for emergency maintenance. It went back in service at 11:42 a.m. PT, but its category listing function wasn't working until 1:27 p.m.
Today's outage followed a lengthy letter on Monday to eBay users from chief executive Meg Whitman and founder Pierre Omidyar, outlining steps the company has taken since a 22-hour outage June 10-11. The site has had at least three unscheduled outages since then.
The key measure was creating a backup system. eBay now runs two database servers simultaneously, with one designated as a "warm backup" to avoid lengthy downtimes. The backup system will be upgraded in the next few months for even faster transition in case of an outage, the executives said.
eBay also boosted its engineering staff by 75 percent last quarter and hired outsider Bob Quinn, a former Sun Microsystems executive, as chief information officer. Ex-IBMer Mark Ryan joined the company as chief technology officer.
eBay also implemented a more rigorous testing regime and vowed to change the site less frequently. It updated its Oracle database July 16, Whitman and Omidyar said.
Finally, the company said it spent more than $10 million on hardware during the last quarter, more than three times the sum it spent in the first three month of the year. It added eight 350-mhz processors to its database server and bought another Sun E10000 to handle search requests.
The letter also included most detailed explanation to date of what happened during last month's round-the-clock outage. That major problem was blamed in part on an update to the database software "that triggered a series of events that led to the corruption of our data and prevented us from restarting our system," the eBay execs wrote.