Q&A: Behind the BBC's iPlayer
iPlayer boss Anthony Rose offers a detailed look at the sheer numbers needed for the popular service's hardware, infrastructure, and operation as well as what's ahead.
During peak hours, BBC iPlayer pumps out 12GB of data every second, and 7 petabytes of data every month. It is insanely popular on Apple's iPhone--but mostly after midnight--and the next version of iPlayer will land this year with some exciting new features.
These are just some of the fascinating facts revealed by iPlayer boss Anthony Rose, in an exclusive one-on-one interview with sister site CNET UK.
Rose's official job title is controller, vision and online media group, and he heads up many of the BBC's Internet activities, including the BBC home page and the site's search system, as well as iPlayer. Previously, he was chief technology officer at the file-sharing service Kazaa, the music download service developed by the .
From what to expect over the next year of development to a detailed look at the iPlayer's hardware, infrastructure and operation, this interview takes you behind the scenes of the enormous on-demand video service.
Q: What will iPlayer 3.0 bring later this year?
Rose: At the moment, we've got the Web site--you go there and click play or you click download. But imagine a future where things come to you, where you can subscribe to things and have favorites and so on. Users are clearly telling us they'd like alerts and an online library, and having delivered HD and video quality that for the most part is similar in quality to TV, we now turn our attention to the personalization and socialization, and customization of iPlayer.
Although at the moment (iPlayer Desktop) is just a download system, in due course it's going to grow to become more part of the Web site. You'll be able to optionally log in to get enhanced iPlayer services: pre-booking, the equivalent of series link, and you'll be able to see which programs you're subscribed to for automatic download.
How much data is iPlayer really using?
Rose: is very popular and it's growing strongly month on month. Here's a fun stat: iPlayer usage, for streaming, peaks about 10 p.m.--just a little later from TV. But interestingly, iPlayer on the iPhone peaks at about midnight. So people are clearly going to bed with their iPhone and watching in bed. We also see on the weekends that there's a peak of Saturday and Sunday morning usage at about 8 to 10 in the morning on the iPhone.
I think that at the moment, just for streaming, iPlayer uses about 60Gbps of bandwidth (that's about 7.5GB downloaded every second) in an evening peak. I think about 15Gbps for downloads and about 1.5Gbps for iPhone. So overall on a particular peak day we may hit 100Gbps (about 12.5GB per second) although typically it'll be somewhat less than that. That turns out to be up to 7PB of data transfer a month.
Why did you abandon
Rose: I think the Internet world has changed in about the three years between our making a decision to use P2P and now. The P2P proposition was made in the day a long, long time ago, when distribution costs were really, really high, and it was felt that our servers wouldn't be able to cope with the load of a lot of people downloading.
But in the end, what's happened is that streaming is clearly the main proposition, and download is about 10 percent of the total consumption. So if the downloads came directly from our servers, it would add only a small extra on to the streaming piece, and in fact it made it more effective--essentially more cost-effective--to simply have a direct HTTP download of the download files, rather than maintaining the infrastructure for a separate P2P delivery network.
Additionally, some people didn't like their upload bandwidth being used. It was clearly a concern for us, and we want to make sure that everyone is happy, unequivocally, using iPlayer.
P2P did work very well for us, but times change and our saying we're not using P2P now doesn't mean we will never want to use it again. We may find, for instance, that we use multicast for live video or a live P2P in the future. The one thing that's constant in the tech world is that things change, and as of today, direct download makes the most sense for us. But things may change in the future.
Is the iPlayer completely automated?
Rose: Most content these days is ultimately on tape delivery, so BBC Vision might have filmed a program a year ago or someone may have sent in footage, and it's often on tape. We will create a program and it'll be saved on a digital tape, stored in a secure vault. Then a couple of days before a program goes out on TV, that tape is ingested and turned into a high-quality 50 to 100Mbps video file. So there there's a lot of manual handling of tapes.
Above the surface, you see a Web site that's clearly computer-generated and automated, but beneath the surface there are some trucks driving around town with tapes. We've been doing it this way since 1940 or earlier. There are a lot of systems which are operationally robust, but in a way a lot of Web people would consider legacy methods.
There's an operational team of probably five people who make sure that videos get encoded, and there are about a dozen people who work on the iPlayer Web site itself and the content management systems that power it. There's also about a dozen people who work on the video player and the media production systems.
What kind of hardware powers the iPlayer?
Rose: We've got about 60 encoding servers. They're typically dual quad-core Intel Xeon machines, and they run on a NAS backend architecture because the media that comes in is encoded at 50 or 100Mbps, and these files are many gigabytes in size. We make 400 hours or more a week.
We will have a system where our schedulers log in and say which platforms each program may be available on. Then our automated systems get the media file and encode it in all the ways that are allowed. After having been encoded, some files are set to download from BBC Web servers, sometimes they will be played out as Flash programs and they'll be distributed to content delivery networks like Level3, Akamai, or Limelight, and sometimes they'll go to special Real Helix servers which deliver content mostly to streaming mobile devices. Content is delivered broadly in one of the first two ways. When you download something directly from the iPlayer, those broadly come from the BBC's servers, which are operated by Siemens. There are two facilities in the U.K., and they're set up in a failover way, so if one of them goes down or the power goes off, we can continue serving from the other.
For streaming media we typically use the content delivery networks, and they've got edge servers located around the U.K. and in fact around the world. But typically we endeavor to make sure media is served from out of a U.K. location just because you get a better user experience.
However, it may turn out that perhaps on a big night where everyone is watching "The Apprentice," the content-delivery networks' local U.K. servers get maxed out, and they begin delivering from a server in France. These networks are set up to essentially load-balance, and it may turn out that there's no penalty to serving from France, than serving from London's Docklands.
Where possible we try and have the content encoded before it's needed to be available in iPlayer. Our rights framework says that we're allowed to make programs in iPlayer one minute after they finish on TV. So what we try to do is pre-encode the content so when it finishes, we can flip the switch and make it available.
In reality, for those programs it takes about 15 minutes to turn up in iPlayer because of the way pages are cached and so on. For programs we can't have pre-delivered, which is stuff like news and sports and live things, as well as some programs where the tapes are delivered only minutes before it goes on air, we do a live ingest.What happens there is it gets encoded off our 50i or 60i feed and then it gets manually trimmed because when you press play in iPlayer, it starts exactly on the frame where the program starts. So we manually trim and then it goes to the encode, and that takes a few hours. For some content it's available within minutes of it finishing on TV, and for some content it's typically one to two hours later.
We know the audience wants content instantly after it finishes, and in fact we're doing a lot of work to make those times faster.
How many versions of each program do you produce?
Rose: We create about 14 different formats, ranging from about 160Kbps for some mobile, over-the-air streaming, through to 1,500Kbps for our highest iPlayer SD quality stream, in H.264 played out as Flash. We also create 3Mbps (for standard definition) on Virgin Media, and now for our HD content we create 3.2Mbps HD. So it's about 14 or 15 flavors.
We typically make a 3GP format, which is really H.264. We make VP6, H.264, MPEG-2, and we make Windows Media Video. Then we deliver those in various ways. In fact, there are probably three aspects: the actual content encode itself, the transport mechanism and how it gets from our servers to the consumer--and then how it's played out.
Our Flash player can play the H.264 or VP6 files, and the mobile formats might play out often in RealPlayer on a mobile device. Then our Windows Media content we make available in a DRM format, typically where you can download or side-load to portable media players or Windows Media Center or Windows Media Extender devices.
One of the nice advantages of the Internet is that you can get people to install the latest software quite quickly. Whereas once you've got a box in someone's house, you need to be compatible with it, and so we make content in a format compatible with Virgin boxes that span several years in the market.
Why did you choose
Rose: We had to be on Mac, PC, and Linux, so we spent a lot of time analyzing solutions that we could use, including what I call "speed dating" companies that offer solutions in this area. But ultimately we chose Adobe Air for two key reasons: No. 1, it had a system that allowed our seven-day or 30-day playback to be enabled and controlled on PC, Mac, and Linux. That's a requirement of a DRM, not that we want to use (digital rights management), but we're forced to because we make content available for download. It's part of our rights framework.
The second reason we chose Adobe Air is because it provides a flexible scripting system. It also meant we could use the skill sets we've got in house with our Flash developers, so we could share the same resources across our Flash player and our download manager.
Is there a concern about relying on one company for this?
Rose: I don't think that's an issue. I think Adobe's a reasonably sized company; they're keen to make sure it works and works reliably. We've had really good feedback on the iPlayer Desktop, we had it in Labs for a while. Like all new platforms, one of the problems you've got is our existing platform had several years of development and actually worked very reliably on one platform and wasn't available on others.
We're now bringing out a new technology and clearly it's going to have bugs initially. I think we did remarkably well on a reliable product, but our servers on day one had an unexpected load on them, so we had some people saying the downloads were taking longer than on the existing one. But I think we've fixed that, and they should be at least as fast if not faster than the existing one.
Nate Lanxon of CNET UK reported from London.