Torvalds, developers at odds over Linux

On Monday, Rob Landley, a computer programmer, writer and Linux evangelist, posted a proposal to the Linux kernel development list calling for a "Patch Penguin"--a person who would help integrate fixes for the myriad of small problems that plague the current development kernel, Linux 2.5.

The proposal comes after many developers have grown frustrated with Torvalds for not keeping up with the slew of minor fixes hatched by volunteers, said Landley. A situation that, he added, that has become a source of underlying tension in the community.

"Right now, the patch process is manageable, but it's showing stress fractures, and I'm proposing to relieve that stress before an earthquake," said Landley, after his proposal set off a heated discussion on the list between Torvalds and several developers. "If the stress keeps growing, the more and more likely that something catastrophic will happen."

The debate has highlighted the fact that, while the complexity of Linux has grown, the task of managing additions to the operating system hasn't kept pace. The fear is that frustrated developers could strike out on their own, "forking" the Linux kernel and creating two distinct versions of the operating system.

Based on code written in the early 1990s by Torvalds, Linux has grown from a tiny, basic operating system to a set of software with features rivaling those of Microsoft's Windows. However, Torvalds still manages the single official version of the core operating system, known as the kernel, as well as architects the future direction of Linux.

Torvalds, a fellow at chipmaker Transmeta, argued that the current development organization is fine. Instead, he insists that developers are frustrated that he doesn't apply every patch that is sent to him.

"The basic issue is one of prioritizing," he said. "You can do one of two things: accept everything, including the crud, (or) being careful, and spending time on the patches you apply."

A matter of trust
Torvalds added that some of the tension comes from his refusal to apply patches that aren't properly submitted and that aren't from people he trusts. Those people, known as maintainers, are programmers designated to lead the development of certain Linux subsystems, such as networking, the help system, and graphics.

"In short, send patches to maintainers that you know I trust," he said. "If you cannot find a person to be a proponent of your patch, you should ask yourself if the patch might have some problem."

That keeps developers guessing, however, whether Torvalds refused the patch because it had a problem or because they didn't have time to get around to it.

"The problem is that the flow of good patches through the system is getting blocked," said Landley. "Part of the problem is that Linus' way of rejecting things is to simply ignore them."

And that's not just happening to unknown developers that have little respect in the community. Big-name developers are seeing their work go unused for long periods of time.

Eric Raymond, a well-known open-source evangelist and maintainer of the Linux Help system, said that he had to submit six patches to the system a total of 33 times to get them included. Each time the kernel changed without the inclusion of his changes, he faced extra work to make sure that his software fixes worked with the latest version of the kernel.

"Linux is not outgrowing Linus' capabilities as an architect, but right now it is outgrowing his capabilities as a manager," said Raymond. "If we are going to keep Linus as the architect, we have to find a way to replace him as a manager, or at least supplement his ability to deal with patches."

Mounting delays
Others argue that the patching problem is the leading cause of delays in starting development on the next version of Linux.

It took just over three months to stabilize Linux 2.2, a production kernel, and start development on Linux 2.3, a test kernel used only for development. However, it took developers almost 11 months to stabilize the latest production kernel, Linux 2.4, and move onto the newest test kernel, Linux 2.5.

Even people that have worked closely with Torvalds believe that he needs help to organize development efforts and keep the code updated.

Alan Cox, a well-known Linux kernel developer who, according to Landley, has unofficially acted as "Patch Penguin" for the current stable kernel, Linux 2.4, agrees that Torvalds needs a sidekick at the very least.

An indication of that, said Cox, is that companies that release their own Linux distributions, such as Red Hat, SuSE and Mandrake, patch the kernel themselves, fixing many problems to which Torvalds has refused to give priority.

"If you look at the vendors, they tend to ship kernels with fixes, changes and often lag markedly behind the leading edge--that's intentional," said Cox, a fellow at Red Hat. "The typical customer wants a solid, reliable platform and someone to stand up and say, 'We support this, we tested it, we say it works.'"

However, Cox downplays the rifts in the communities, noting that Linux developers tend to be a fractious lot.

"Think of this more like an office meeting to figure out what is going on and what needs to be tweaked in the processes," he said. "The difference being we hold our office meetings in public."

For his part, Torvalds said he isn't becoming overwhelmed by the work of keeping Linux development on track. However, he does allow for the possibility that an additional maintainer to keep track of the minor patches might have merit.

"A person who only takes the 'miscellaneous' patches--the stuff that falls through the cracks by virtue of being small and not in any clearly managed code--might be a fairly good idea," he said. "The problem with that is that there are very few people who want to just clean the stables and not do the big and 'exciting' stuff."