Super-powered peer to peer

Morpheus developers are looking to recapture their onetime leading role in the file-trading world with a network technology called Neonet, written by a pair of former Harvard students.

Dubbed "distributed hash tables," Neonet's technology transforms the way that searches happen on peer-to-peer networks, potentially making it more efficient to search a much larger number of computers and more easily surface rare files. Similar technology is also used by eDonkey, a competitor that is on the verge of overtaking Kazaa as the most widely used peer-to-peer service in the world.

"Peer-to-peer technology to date is not good enough yet," said Michael Weiss, StreamCast's chief executive officer. "People ask, does the world really need another peer-to-peer network? I think the answer is, yes we do, because nobody's gotten it right yet."

The advances in peer-to-peer networking come as programmers are increasingly looking at using the technology most often associated with file trading for new applications such as Internet calling or instant messaging.

Even Internet service provider EarthLink recently released its own version of file-swapping software, saying that it was a prelude to more advanced applications such as Net phone calls.

Yet for most Net users, the draw of file swapping remains paramount. The threat of lawsuits from the Recording Industry Association of America--or even criminal investigations from the federal government--has dampened some swapping enthusiasm, but millions of people still download and use file-trading software every week.

Swapping's evolution
The release of Morpheus' software, along with eDonkey's rise, marks the ascent of a third generation of peer-to-peer networking technology. Each successive generation has decentralized more functions, making the networks harder to shut down and helping to expand the power of searches.

The first generation of file-swapping services, led by Napster, were built around big centralized indexes that would keep track of what was available everywhere on the network. These would serve as matchmakers, linking a person searching for a file with the computer where it was stored.

That was efficient, allowing access to a huge range of material--but it also proved to be illegal. Courts said that Napster was responsible for a network where a vast amount of copyright infringement was happening and ultimately shut the company down.

The second generation of decentralized services, led by Gnutella and the FastTrack technology underlying Kazaa, soon emerged to take its place. Neither of these had central servers. They relied instead on passing search requests from computer to computer until a file was found, and then passed that information back to the original searcher.

That technology proved initially unwieldy, as millions of search requests passed through every computer on the network, creating traffic jams at low-bandwidth bottleneck points. That improved over time as programmers figured out ways to hand off these search requests more efficiently, but usually resulted in searches that included only part of a network--say 100,000 people instead of 2 million.

A U.S Appeals Court recently ruled that this kind of decentralized network was legal, unlike Napster, in part because the software distributors did not have direct control over what was happening on the networks.

"The (record labels and movie studios) urge a re-examination of the law in the light of what they believe to be proper public policy," the court wrote in that decision. "Doubtless, taking that step would satisfy the copyright owners' immediate economic aims. However, it would also alter general copyright law in profound ways with unknown ultimate consequences outside the present context."

No more ripple effect
The third generation of networks, represented by eDonkey and now Morpheus, as well as a host of smaller independent developers, makes the tools even more decentralized than before.

Distributed hash tables are essentially a way of taking a snapshot of where every file on the network is at a given moment and scattering bits of that information around the entire network.

To find a given file, a search request goes first to any computer on the network. That computer will point to a different computer that has a little more information on how to find the file. The third computer might have information on the file itself--or it might take a few more hops to find the computer with the right information.

The process is analogous to asking a succession of increasingly informed tour guides for directions, rather than accosting random people on the street. The information about the network in each place is constantly being updated as new files or computers are added.

"The main benefit is that it allows you to search the entire network instead of just a local area," said Jed McCaleb, the chief programmer for the eDonkey project. "It's probably faster than the way Gnutella works, and it's definitely technically superior."

StreamCast acquired its technology last year from Harvard students Ben Wilkin and Francis Crick, the grandson of the DNA pioneer and Nobel prize-winner of the same name. Wilkin said he'd started the project after seeing early inefficiencies in Gnutella several years ago, while Crick joined the project later.

The pair says their technology will take just three or four hops to find any file, no matter how rare, on a network of up to millions of computers.

This kind of technology also holds promise for the newer applications such as Net calling. Neonet and eDonkey each are focused on file swapping, but the same efficient network routing could be used to connect calls quickly, even among computers that are constantly popping on and off the Net, they say.

"It can be used for all sorts of distributed computing tools, and that's where we're going to go with it," Wilken said. "It really eliminates the need to have any centralized infrastructure."