Chicago museum turns to open-source storage

Chicago's Museum of Broadcast Communications (MBC) collects, preserves, and presents historic and contemporary radio and television content with the purpose of educating, informing, and entertaining the public through its archives, public programs, screenings, exhibits, publications and online access to its resources.

MBC also runs Museum.tv--which stores and delivers terabytes of digitized radio and television content. Currently, they are featuring a 1984 senatorial debate including Roland Burris--whom you may recognize as the senator just appointed to fill Barack Obama's vacancy (check out the protests at the start of the debate and how the moderator handles it).

Through the years, the primary challenge the not-for-profit MBC faced was the lack of resources to put together a large enough storage infrastructure to handle the massive amount of digital data they had and needed to present. At one point, when their single-server storage setup didn't foot the bill, they had to suspend their online services.

All told, MBC has more than 100,000 hours of content that it needs to store and distribute. Due to its size and lack of structured data, video remains inherently difficult and expensive to store. And let's not get started on reliability and security issues--achieving data reliability and security at the terabyte level using traditional storage methods based on replication requires significant hardware capacity at multiple sites.

When Cleversafe CEO and MBC-member Chris Gladwin heard about this, he contacted MBC to introduce Cleversafe's Dispersed Storage technology that could potentially solve their problems.

Here's how Dispersed Storage works: instead of copying data, Cleversafe divides it into "slices" and disperses it across a secure network to different geographic locations. Each slice contains too little information to be useful, but any threshold of the slices can be used to perfectly re-create the original data. Manageability? Yup. The sum of all the slices is still less than maintaining multiple copies of the original data.

One interesting sidebar to this--in addition to data storage, MBC also relies on Cleversafe for distribution instead of a separate content delivery network (CDN). When users view content on MBC's site, the data is pulled directly from Cleversafe and displayed via a media server in front of the Cleversafe hardware, saving MBC money and physical space without sacrificing performance or scalability for their end users.