File swapping shifts up a gear

Going by names like eDonkey and BitTorrent, many of the latest generation of file-swapping tools have been designed specifically to increase the efficiency and speed of transfer for large files such as movie files. Some of these tools have been in development for several years, but are just now reaching the critical mass needed to make a dent in the file-trading world.

Some in the copyright community say these new tools are finally starting to rival the piracy potential of the post-Napster generation of swapping services.

"We see people downloading like crazy," said Mark Ishikawa, chief executive of BayTSP, a Los Gatos, Calif., company that monitors file-swapping networks for movie studios and record labels. "eDonkey is passing Gnutella, and is even on its way to passing FastTrack." FastTrack is the technology behind Sharman Networks' Kazaa and Grokster.

File-swapping tools such as Kazaa and Streamcast Networks' Morpheus are still enormously popular--indeed, on Friday, Kazaa passed ICQ instant messaging software as the most-requested technology ever on Download.com, a software aggregation site owned by CNET Networks, the publisher of News.com. Kazaa has been downloaded more than 229 million times, according to that site.

The new generation of tools has been designed much like the old, by individuals or small teams of programmers working to correct the perceived shortcomings of earlier software.

In search of efficiency
eDonkey is largely the product of programmer Jed McCaleb of New York. His initial plan was to make file search and distribution more efficient, while keeping the same basic network-searching functions seen in Napster and its successors.

The final eDonkey product is different in two primary ways from earlier file-swapping services. The first is to do with decentralized search. When a file is shared on the network, the technology gives the file a "hash" identifier--essentially an address based on the characteristics of the file itself. Each computer logged onto the network has a certain range of addresses assigned to it, so it can act as an index.

This allows searches to be carried out more efficiently than in earlier decentralized systems. With Gnutella, for example, a search query for "Radiohead" or "Madonna," for example, ripples out through the network, asking each node if it has or is close to those files.

With eDonkey, the "Radiohead" query would be directed quickly to the computer that is temporarily responsible for keeping track of the location of files in that category, and a response would be returned more quickly.

The second--and main--advance is that the system can break up each file into tiny pieces, allowing them to be distributed independently. As soon as one person starts downloading these pieces, he or she starts offering them to the network at large. That means a movie does not have to be downloaded in its entirety before it can be offered to other people, making distribution of these and other larger files much more efficient.

McCaleb has been offering various versions of eDonkey for several years, but in the past few months in particular it has been catching on more quickly, especially in Europe, he said. The tool has been downloaded close to 50 million times, he added, but said as he updates it very frequently, the number of people who have tried the software is likely closer to 8 million.

Another tool, even more focused on distributing large files, has seen rapid adoption inside the open-source community in the past few months.

Dubbed BitTorrent, the tool was written by San Francisco programmer Bram Cohen, again as a way to make the distribution of large files more efficient. Unlike Napster, Kazaa or eDonkey, the tool concentrates on distribution rather than search. But what it lacks in search capabilities, it makes up for in speed, its supporters say.

BitTorrent works in the background of a Web browser, assisting with the uploading and downloading of files. If users--maybe a software company or a file swapper--want to distribute files, they have to set up a "tracker" Web site. A tracker is essentially a low-level server that keeps track of requests for a given file and directs the requests to the users offering the file. These users will have posted links to the tracker on a Web site, and these links will trigger the properly formatted BitTorrent downloads.

Like eDonkey, BitTorrent splits files into tiny bits. Once someone has started downloading a file, that person's computer immediately serves as an upload server for anyone else looking for the file. The technology automatically balances upload and download speeds, ensuring that people downloading give back to the network, Cohen said.

Unlike other file-swapping networks, if the number of people searching for a single file increases, it means faster downloads--not traffic jams--as the individual pieces get spread quickly around the community.

"I'm basically measuring my success by the amount of upload capacity which gets used, which would otherwise be sitting around wasted," Cohen said in an interview conducted over an instant messaging system.

Right side of the law
The BitTorrent technology isn't intended for movie piracy. Indeed, open-source advocates recently used it to distribute a new release of the Red Hat Linux software, making an end run around clogged company servers. One Web community that specializes in swapping high-quality recordings of live jam bands such as Phish, usually with the bands' permission, also uses the technology.

"It's definitely a technological marvel," said Wayne Chang, a Massachusetts college student and system administrator for PickATime.com who has used the technology. "The more people using it, the faster the whole system is."

People have quickly seen its potential for unauthorized online distribution of videos and movies as well as for lawful uses, however. Many popular Web sites already offer links to full movies, TV programs, games and music using the technology.

The tool has been the most popular download on the open-source software aggregation site SourceForge for some time, with more than 1.4 million downloads as of May 20.

Executives from the Motion Picture Association of America (MPAA), the movie studios' trade association, say they are well aware of and are monitoring the new file-swapping technologies.

"I would say that they represent a continuing threat," said Tom Temple, director of worldwide Internet enforcement for the MPAA. But just as with predecessors, the new technologies expose their Internet addresses of those who download files, allowing antipiracy enforcers to track them down, he said. "They still allow us to identify the IP address of a person."