Why open source works

Trying to explain why open source works, you can of course point to the Cathedral and the Bazaar by Erik. But the kernel development process shows it happening 'in real time', every day, and that's a major reason why I so enjoy reading the weekly LWN.

Competition FTW

A recent article explained how two patch sets are developed, impacting the Huge TLB feature. One is about improving reference counting in the handling of huge pages to allow a huge page to be mapped in both 'huge page mode' and in 'normal page mode' by different processes. This is a first step towards being able to use transparent huge pages with the page cache, giving it access to the performance benefits of huge pages. This has been in the works for a while as it is a very complex change. But an alternative has recently surfaced, adding transparent huge page support to the tmpfs filesystem. This also has pieces needed to support huge pages in the page cache, but in a very different way.

Why on earth would you want developers to spend so much time on two competing solutions for a problem? Isn't there One to Rule Them All?

Finding that perfect solution

I am sure that there is a Perfect Solution. I would simply suggest that nobody knows it! Like in economics, where an equilibrium exists but no single person can define it, software development has become complicated enough that competition of ideas often leads to the best (or at least better) solutions.

Companies, building their own cathedral in-house, would put product managers and developers in a room. Let them come up with the Perfect Solution. But they won't, because of Joy's law:

"No matter who you are, most of the smartest people work for someone else"

Impressive results

This is the key to why open source works and works so well. No single development project in the world even approaches the size of the Linux kernel project - no less than 1400 people contributed in under six weeks. With millions of lines of code developed by tens of thousands of people over 20 years time and with an extremist attitude towards backwards compatibility, you wouldn't expect the trend to be "go faster". It is.

Bringing over almost 12,000 different people from 1,200 different companies together in the last decade, the kernel competes with Wikipedia for the title of humanity's largest collaborative effort. Oh, and I'm sure the masses building the pyramids were big, too, they just didn't integrate the experience of all their collaborators into its architecture as well as the kernel and Wikipedia do ;-)

Cookie licking

Many other projects struggle to learn from the kernel's success. Success which has many aspects, one of which exemplified here, a rule Frank sometimes talks about: no cookie licking!

In short, cookie liking is preventing others from working on something by claiming you are (but not really making progress). Extending it a bit, I would say that, while collaboration is good, if you feel you can do better -- or just want to sample a different approach:

"never be deterred to build an alternative solution to a problem"

Because it's space for competition within projects that makes open source work better.