X

Server makers tout InfiniBand sequel

Seeking a better way to make high-end database servers from low-end parts, tech giants are working on a cheaper alternative to the InfiniBand high-speed networking technology.

Stephen Shankland Former Principal Writer
Stephen Shankland worked at CNET from 1998 to 2024 and wrote about processors, digital photography, AI, quantum computing, computer science, materials science, supercomputers, drones, browsers, 3D printing, USB, and new computing technology in general. He has a soft spot in his heart for standards groups and I/O interfaces. His first big scoop was about radioactive cat poop.
Expertise Processors, semiconductors, web browsers, quantum computing, supercomputers, AI, 3D printing, drones, computer science, physics, programming, materials science, USB, UWB, Android, digital photography, science. Credentials
  • Shankland covered the tech industry for more than 25 years and was a science writer for five years before that. He has deep expertise in microprocessors, digital photography, computer hardware and software, internet standards, web technology, and more.
Stephen Shankland
5 min read
Seeking a better way to make high-end database servers from low-end parts, Hewlett-Packard, IBM and others have begun working on a cheaper alternative to the InfiniBand high-speed networking technology.

The technology, called Remote Direct Memory Access (RDMA), borrows heavily from the years of work that went into creating InfiniBand, according to engineers involved in the work. While RDMA will arrive later and perform slower than InfiniBand, it will have the advantage of using conventional networking hardware based on the TCP/IP and Ethernet standards used to build the Internet.

"We're pushing hard on RDMA," Mary McDowell, the head of HP's Industry Standard Server group, said in an interview. (McDowell made her remarks just days before the announcement that she would be taking a sabbatical after years of running Intel-based servers for Compaq Computer and then HP.)

If it meets expectations, RDMA links will be used to connect small servers into a collection that can handle a large database that today must be housed on a much more expensive server that has dozens of processors. That vision, long touted by companies such as Microsoft and Intel that have strength in lower-end servers, is fast becoming a reality, as indicated by database software such as Oracle's 9i RAC and IBM's DB2 EEE. But many industry observers believe that vision for servers is still years away from broad use.

RDMA took a significant step closer to reality last week when the RDMA Consortium released the second of three sets of specifications needed to build the technology into hardware and software. And it will take another step this week when Microsoft announces additional backing at its Windows Hardware Engineering Conference (WinHEC).

"You're going to hear a lot more of the pieces of this from Microsoft," said Karl Walker, chief technology officer for HP's Intel-based server group. Linux programmers also are actively involved, Walker said.

On Wednesday, Microsoft plans to announce that RDMA will be necessary to take advantage of forthcoming 10-gigabit-per-second Ethernet networks, and it will discuss how RDMA supports a company plan, code-named Chimney, to enable easy use of network-accelerating hardware called TCP/IP offload engines (TOEs), according to the WinHEC agenda.

IBM is another backer. "If you line up RDMA and TOEs and 10-gigabit Ethernet, you have a very competitive technology. That's what InfiniBand does today," said Tom Bradicich, chief technology officer of IBM's Intel-based server line, and one of the key engineers behind the InfiniBand technology.

How best to be big
The RDMA technology opens a new chapter in a long debate about the best way to make big servers. Traditionally, systems that can handle hundreds of thousands of transactions per minute are made with single, expensive machines that contains dozens of processors--"scale up," in industry parlance. But many advocate a less-proven method, sharing the job over inexpensive machines joined with high-speed links--or "scale out."

The debate pits industry stalwarts such as IBM and Sun Microsystems--that have expertise in massive systems--against comparative upstarts such as Dell Computer and Microsoft, though most companies embrace both strategies in varying degrees.

RDMA, like InfiniBand, is a tool that could make the scale-out approach more feasible. "If you want no-compromise performance, InfiniBand is one of your best answers. But if you want scale-out for the masses, RDMA over TCP is an awfully compelling story," Walker said.

"The scale-out versus scale-up debate been going on in computer architecture circles for about 30 years," Walker said. "I don't think this debate is going to be over tomorrow, but we think the needle is moving...to shift to scale-out apps."

Companies vary on when they expect RDMA to arrive in the marketplace.

"I believe the deployment in product is in the 2005 time frame," Bradicich said. He believes the technology won't catch on until 10-gigabit Ethernet using copper wires arrives, something he expects in 2004.

But Walker said HP thinks RDMA-enabled products--operating systems and network cards--will be on sale in 2004. "We are architecting it so the technology is applicable to 1 gigabit," which is in widespread use today, not just 10-gigabit Ethernet coming later.

RDMA lets one computer directly read from or write to the memory of another without having to go through the time-consuming process required today, in which a request for data must move through the operating system and other layers of software. That delay, or latency, is the single most important issue for RDMA and the extent to which it will be able to succeed as a mainstream technology where InfiniBand failed.

The InfiniBand latency is about 8 to 12 microseconds, Walker said, and a reasonable hope for RDMA using 1-gbps Ethernet is in the range of 15 to 18 microseconds.

"If they can get 15 to 18 microseconds, that's pretty good," said Illuminata analyst Jonathan Eunice. "With the RDMA folks, the risk is that they shoot for 15 and get 35 or 40. The InfiniBand folks, for example, shot for 2 to 5 (microseconds), and they got about 10."

In any case, InfiniBand likely will have at least 18 months with the market to itself while RDMA details are worked out, Bradicich said.

Slow going
If RDMA catches on, it won't be the first time a slower technology that benefits from the widespread use of Internet Protocol challenges a faster but comparatively nonstandard technology. In storage, the iSCSI protocol for sending storage data over IP networks is gaining on the incumbent technology, Fibre Channel, despite the latter's maturity and speed.

"Fibre Channel had a 10-year lead, then woke up one morning, looked out the rear-view mirror and saw iSCSI coming up its tailpipe," Walker said.

RDMA matches up well with iSCSI, with the ability to make the storage transfer standard more efficient, Walker said.

Much of RDMA technology came directly from work pioneered by InfiniBand engineers, Bradicich and Walker said. For example, InfiniBand has a "verbs" specification to ensure that software such as an operating system is aligned with hardware features. The RDMA Consortium followed suit with its own very similar verbs specification last week.

InfiniBand also influenced other technology, such as the PCI Express expansion slots for next-generation PCs as well as high-speed cabling technology used in IBM's x440 Intel servers, Bradicich said.

InfiniBand has met with modest success in some high-performance niches, though it didn't become the popular technology early advocates had in mind. First Intel and then Microsoft retreated from InfiniBand plans. In addition, IBM backed off plans to produce standard InfiniBand chips that companies could employ in their products, although it has continued its business selling customized InfiniBand chips for specific customers, a company representative said.

InfiniBand is moving ahead, with IBM selling a database product using the technology and supercomputer makers interested as well. Several server makers rallied around InfiniBand in December; Sun in particular has high hopes for the technology.

The RDMA Consortium is building a specification intended to be mature enough for widespread use, but in the longer term, the group wants to hand over standardization to the Internet Engineering Task Force, which governs many Internet Protocol-related technologies and with which the RDMA Consortium already is working closely.

"The RDMA Consortium doesn't want to exist long term as another standards body," Walker said. The IETF is expected to charter its RDMA over Internet Protocol Suite Working Group in coming months, the RDMA Consortium said.

In addition to IBM, HP and Microsoft, the founding members of the RDMA Consortium are Intel, Dell, Cisco Systems, EMC, Adaptec, and Network Appliance.