I'm currently working on pricing models for several new open-source companies, and I keep running into a similar set of challenges. The primary issue is that when you shrink a market, as open source does, you must to find a pricing model that solves the equation, meaning that your costs must substantially lower in order for you to make money.
Customers assume that open source is free and that commercial open source is cheaper, but most companies aren't prepared to deal with the implications of having a lower-cost product. Even when you can clearly demonstrate value, you run into a scale issue sooner or later. (Sarah Lacy harangued me about this in an interview for Yahoo Tech Ticker.)
So, the answer appears to be that you have to provide more value for the dollar, but how do you do it in a way that makes a highly scalable, highly monetizable business? I like this quote from Alfresco's John Newton in the Times Online UK"
Newton sees the development of Japanese car brands in America as a good analogy for open source. "People admitted Japanese cars were cheap, but argued they would cost you more because they would break down more.
Japan, by focusing on quality and cost, was able to demonstrate that a Toyota does actually perform, gets you from A to B, and costs you less. That is how the Japanese became the biggest car makers in the world."
Japanese cars in the U.S. market is a good analogy. Former MySQL CEO Marten Mickos often said he wanted to be the Toyota of the software world--which didn't mean low cost as much as it meant that he wanted MySQL to be highly refined, to have mass-market appeal, and to have room for upgrades.
And while cars are not a great business to be in right now, the Toyota model still makes a great deal of sense. Open-source companies have to become models of efficiency in every area in order to keep costs down and revenues up. Customers expect open-source alternatives to be 10 percent to 20 percent of the cost of the proprietary product, which means that open-source companies need to be 80 percent to 90 percent more capital-efficient.