Open-source Lucene threatens Microsoft, Google enterprise search

It must be depressing to be Microsoft these days.

You spend $1.2 billion to acquire enterprise search leader FAST in January 2008 and then another $100 million on semantic search vendor Powerset in July 2008, only to have the excellent Apache's Lucene, an open-source search project, and Solr, an enterprise search server based on Lucene, offer better performance at a 100 percent discount.

Not very sporting of the open-source community, now is it?

Granted, Lucene and Solr still lack some of the spit and polish that Microsoft FAST, Autonomy, Google's enterprise search appliance, and other proprietary competitors offer, but this isn't slowing its adoption. As CMS Watch's Kas Thomas notes, interest in Lucene and Solr is skyrocketing, as measured by job postings (among other data):

This was OK when Lucene stood alone, relatively rough and unadorned by Solr. But Solr makes Lucene much more palatable to enterprises that worry about getting mired in Microsoft (or Autonomy or Google or...), particularly with the uncertainty and unrest prior to its acquisition that may have led to an employee brain drain at FAST, and interest in Solr is improved further by the arrival of Lucid Imagination, a company started in 2007 to commercialize Lucene and Solr.

I talked Wednesday with Marc Krellenstein, co-founder and CTO at Lucid Imagination, and learned that while Lucene and Solr have been doing exceptionally well on their own, Lucid Imagination is in pole position to help advance the development of these open-source projects by offering dedicated development resources and to make a solid business for itself in the process.

Just having a company associated with Lucene and Solr may already be enough to get enterprises off the fence and behind the search project.

According to Krellenstein, Solr delivers significant performance improvements over proprietary alternatives. The goal is to continue to improve its functionality, which currently has roughly 80 percent of the total functionality of rival search products while also advancing innovation to surpass these rivals. The company and project have been making steady progress in these areas.

Did the Lucene/Solr community just upend billions of dollars in Microsoft, Google, Autonomy, and others' investments in proprietary search? Time will tell, but $1.2 billion for FAST is looking mighty expensive compared with the $0.00 Microsoft could have paid for Lucene.

Unfortunately, that's what its customers may be thinking, too. Like the U.S. intelligence community, for starters, which is now standardizing on Solr/Lucene. Microsoft and its peers must be hoping this will remain confidential.

Not a chance.

Disclosure: I am an adviser to Lucid Imagination.

Follow me on Twitter @mjasay.