Eucalyptus open-sources the cloud (Q&A)

It's reasonably clear that open source is the heart of cloud computing, with open-source components adding up to equal cloud services like Amazon Web Services. What's not yet clear is how much the cloud will wear that open source on its sleeve, as it were.

Eucalyptus, an open-source platform that implements "infrastructure as a service" (IaaS) style cloud computing, aims to take open source front and center in the cloud-computing craze. The project, founded by academics at the University of California at Santa Barbara, is now a Benchmark-funded company with an ambitious goal: become the universal cloud platform that everyone from Amazon to Microsoft to Red Hat to VMware ties into.

Or, rather, that customers stitch together their various cloud assets within Eucalyptus.

I caught up with Eucalyptus founder and Chief Technology Officer Rich Wolski to learn more about how Eucalyptus hopes to displace industry leaders like VMware, and the role that open source plays in growing the cloud-computing market.

While Wolski made a big deal about the company's open-source credentials, I found his argument about the open architecture of Eucalyptus more compelling:

[Eucalyptus] is architected to be compatible with such a wide variety of commonly installed data center technologies, [and hence] provides an easy and low-risk way of building private (i.e. on-premise or internal) clouds...

Thus data center operators choosing Eucalyptus are assured of compatibility with the emerging application development and operational cloud ecosystem while attaining the security and IT investment amortization levels they desire without the "fear" of being locked into a single public cloud platform.

Code matters, in other words, but open architecture and open standards matter more. That's the same lesson that Tim O'Reilly argued back in July 2008 ("We can talk all we like about open data and open services, but frankly, it's important to realize just how much of what is possible is dictated by the architecture of the systems we use.")

Fortunately, Eucalyptus has time to figure this out, and the company appears to be on the right track. We're nowhere near saturation of cloud computing in the enterprise. We've barely just begun. We have the technology, but operations must evolve to embrace it, as Wolski speculates:

The economic incentives for deploying a private cloud are compelling, both in the near term and in the longer term, but the operational changes associated with any change in scalable platform are still being studied and implemented by IT organizations that plan to adopt cloud computing as a part of their critical infrastructure.

So, as Wolski told me, Eucalyptus sees a lot of proof of concepts today, but enterprises are still crawling toward full roll-outs of cloud computing. Gaming, finance, pharmaceutical, and manufacturing sectors have been early adopters of Eucalyptus technology.

Such companies are not, however, contributing back to the code. Not in terms of code, anyway:

Today we don't get much in the way of outside contributions to our code. The reason is that we have been evolving the code so rapidly (in response to open-source demand) that by the time an external contribution is submitted the code base that we are working from has made the contribution obsolete.

We discovered early in the Eucalyptus endeavor that because of cloud computing's newness, we would need to adopt a very aggressive and frequent release schedule to keep pace with industry innovation. Our early experiences with potential code contribution to the Eucalyptus core resulted in duplication of effort, a labor-intensive forward port, or both.

What Eucalyptus does get from its open-source community is feedback, encouragement, direction, and other "intangibles" that provide valuable insight into how to shape Eucalyptus' platform to meet current and future cloud-computing demand.

That's a potent force to have on one's side.

It's early days for Wolski and his Eucalyptus team. It remains to be seen whether a group of academics can build a successful IaaS business (though, as Wolski pointed out to me, the academic origins of the project allowed for relatively painless experimentation that has resulted in a better product), but the company's open-source approach is attracting increased amounts of attention in the industry.

Now it's time to turn that attention into revenue. Class dismissed.

For a full-text transcript of the interview, please read on.

Q: Tell me a little bit about Eucalyptus, and why it matters.
Wolski: Eucalyptus is an open-source software platform that implements IaaS-style cloud computing using the existing Linux-based infrastructure found in the modern data center. It is interface compatible with Amazon's AWS making it possible to move workloads between AWS and the data center without modifying the code that implements them.

Eucalyptus also works with most of the currently available Linux distributions including Ubuntu, Red Hat Linux (RHEL and CentOS), Suse Linux (SLES and OpenSUSE) and Debian. Similarly, Eucalyptus can use a variety of virtualization technologies including VMware, Xen, and KVM to implement the cloud abstractions it supports.

Eucalyptus is important for two reasons.

1. Because it is available as open-source and also because it is architected to be compatible with such a wide variety of commonly installed data center technologies, it provides an easy and low-risk way of building private (i.e. on-premise or internal) clouds.
2. Because it is implemented from first principles to be a cloud platform (e.g. it is not a refactored implementation of previously available technology) it is able to provide the scalable cloud abstractions in a form that is very nearly indistinguishable from those provided by the most popular public cloud provider: Amazon's AWS.

Thus data center operators choosing Eucalyptus are assured of compatibility with the emerging application development and operational cloud ecosystem while attaining the security and IT investment amortization levels they desire without the "fear" of being locked into a single public cloud platform.

There's a lot of hype around cloud computing right now. What's the reality?
Wolski: The reality is that cloud computing is in its infancy and as such its dominant usage scenarios have yet to emerge (both for the public clouds and for private clouds as well). Private clouds, in particular, represent an operational change for the IT organization that IT managers have yet to fully understand.

The economic incentives for deploying a private cloud are compelling, both in the near term and in the longer term, but the operational changes associated with any change in scalable platform are still being studied and implemented by IT organizations that plan to adopt cloud computing as a part of their critical infrastructure.

So how are organizations using your software today?
Wolski: The ideal customer is one with an IT organization that is tasked with supporting a heterogeneous set of user groups (each with its own technology needs, business logic, policies, etc.) using infrastructure that it must maintain across different phases of the technology lifecycle.

There are two prevalent usage models that we observe regularly. The first is as a development and testing platform for applications that, ultimately, will be deployed in a public cloud. It is often easier, faster, and cheaper to use locally sited resources to develop and debug an application (particularly one that is designed to operate at scale) prior to its operational deployment in an externally hosted environment.

The virtualization of machines makes cross-platform configuration easier to achieve and Eucalyptus' API compatibility makes the transition between on-premise resources and the public clouds simple.

The second model is as an operational hybrid. It is possible to run the same image simultaneously both on-premise using Eucalyptus and in a public cloud thereby providing a way to augment local resources with those rented from a provider without modification to the application.

For whom is this relevant technology today? Who are your customers?
Wolski: We are seeing tremendous interest in several verticals. Banking/finance, big pharma, manufacturing, gaming, and the service provider market have been the early adopters to deploy and experiment with the Eucalyptus technology.

Another way to measure interest is through downloads. From our Web site, excluding those who download from the Ubuntu Web site, we are approaching 100,000 downloads. Downloads are now tracking at greater than 15,000 per month. We have been downloaded in 74 countries and on every continent but Antarctica.

Our revenue in these early days has been driven by professional services as the enterprise continues to learn about the value proposition of cloud computing. Proof of concept deployments are by far where we spend most of our time and we have signed many support contracts for those customers deploying Eucalyptus in a production environment.

Ultimately, our revenue will be derived from the sale of our proprietary products, such as our support for VMware, which we launched in September as part of Eucalyptus Enterprise Edition (EEE). Demand has been brisk for this new product.

You face some big competitors like VMware and Microsoft, but also a host of start-up competitors that tackle cloud computing in different ways, among them VMOps and Cloudera. Can you give me an overview of the competitive landscape and why Eucalyptus' differentiation?
Wolski: To begin with, it is not clear that we are competing with these companies in the following sense. Eucalyptus is designed to be able to compose multiple technology platforms into a single "universal" cloud platform that exposes a common API, but that can at the same time support separate APIs for the individual technologies.

Moreover, it is possible to export some of the specific and unique features of each technology through the common API as "quality-of-service" attributes.

In this way, Eucalyptus users can choose the features they most want to combine in an application using the APIs they most want to use to achieve cross-platform integration. It is an ambitious goal, but one we decided to pursue early on, as part of the Eucalyptus research project and it remains technically feasible.

The business case for this approach, however, is less clear. For example, as an open-source platform, it is clearly beneficial for Eucalyptus to be an extensible, multi-technology platform.

Providing this form of technology aggregation where the individual technologies "compete" for usage among the users based on feature set, however, may prove unattractive to a single technology provider. The problem of integrating multiple platforms in this way becomes substantially more difficult if the constituent technologies are not partners in the endeavor.

Thus, we continue to discuss this approach with potential partners, but as of yet, we have not been able to achieve the consensus we believe is necessary for it to be widely successful.

The alternative, then, is that we are indeed competing with these companies. In the case of VMware, Microsoft, and VMOps our advantage lies in our availability as open source. In the same way that Linux penetrated the data center as a critical platform technology when commodity clusters became a technical reality, we hope that the openness and speed with which Eucalyptus can innovate as a platform will make it similarly ubiquitous.

Cloudera, however, is different in that its products complement rather than compete with Eucalyptus (at least, at the moment). That is, the implementation of Map-Reduce that Cloudera supports runs quite efficiently today on a Eucalyptus platform. Indeed, our expectation is that cloud services which operate above the IaaS level (such as Cloudera's Map Reduce) will be important components in a "cloud stack" and, as such, Eucalyptus will need to be able to support them.

Moreover, because Eucalyptus is compatible with public clouds, and many companies such as Cloudera are developing for the public clouds, the effort required to achieve this type of integration is minimal. For example, the Cloudera image available freely in Amazon ran almost immediately when it was moved to a Eucalyptus cloud because Cloudera had already made a (justifiable) effort to port their software to AWS.

And Red Hat?
Wolski: We do not yet have anything formal in place with Red Hat, but we plan to continue to support the use of its Linux distributions as a deployment platform for Eucalyptus.

A partnership with Red Hat is certainly possible. The goal for Eucalyptus is to allow its operator to combine a wide variety of data center technologies within a single, open cloud platform, and to export the salient features of the underlying technologies, both hardware and software, to its users. Thus we see a way to partner with Red Hat to allow its customers to combine the value it adds to Linux with other technologies within a single cloud platform.

But Red Hat isn't the only open-source company with which we can work. Clearly our experience with Canonical and Ubuntu indicates that partnering with Linux technology aggregation companies (e.g. Red Hat and Novell) can lead to a profitable outcome for both parties.

We also believe that there will be significant opportunities to partner with companies working at different levels of abstraction (e.g. Cloudera) as well as open-source projects (e.g. AppScale).

Finally, we anticipate that we will continue to expand our partnerships within the tooling and management ecosystem. Here we have partnerships with companies such as RightScale (which has a free SaaS offering) and Zmanda. We believe that these opportunities, particularly in open source, will continue to grow.

What is the value of open source to your business?
Wolski: Open source is particularly valuable as a way of rapidly identifying and categorizing potential innovations. It retains this value even though today we don't have many outside developers contributing to our code base.

The open-source community is generous with its criticisms, critiques, and suggestions for Eucalyptus. We learn a great deal from the open-source community about how to make the platform useful to a wide audience and for a wide variety of purposes.

This type of generality is especially critical for a platform such as Eucalyptus -- perhaps more so than for other software components higher in "the stack."

We also learn a great deal about how to make Eucalyptus configurable for a wide variety of execution environments. If it runs Linux, it seems to us that someone has tried to run Eucalyptus on it which, we believe, they would not have otherwise tried to do if it were not available as open source. The ability to configure Eucalyptus for almost any environment stems directly from these experiences.

Today we don't get much in the way of outside contributions to our code. The reason is that we have been evolving the code so rapidly (in response to open-source demand) that by the time an external contribution is submitted the code base that we are working from has made the contribution obsolete.

We discovered early in the Eucalyptus endeavor that because of cloud computing's newness, we would need to adopt a very aggressive and frequent release schedule to keep pace with industry innovation. Our early experiences with potential code contribution to the Eucalyptus core resulted in duplication of effort, a labor-intensive forward port, or both.

However, moving forward we plan to "slow down" the pace at which we drive the core code ahead, making external core contributions feasible.

Some suggest that private clouds are a transitory phase in the market; that we'll eventually migrate to public clouds once security concerns are overcome. Presumably you disagree. Why?
Wolski: We believe that the future IT infrastructure will be a mixture of public "utility" cloud usage and on-premise IT. Security and regulatory compliance are clearly near-term drivers for private clouds, but as these concerns dissipate, there remain several factors that we believe will make a hybrid attractive, at least in an enterprise setting.

First, enterprises often reflect policy in their configurations, particularly in the storage infrastructure. Much of this enterprise-specific specialization can be homogenized away so that a "generic" cloud utility becomes a suitable hosting venue, but what remains will require local infrastructure. Moreover, it will be important to be able to exploit that infrastructure in conjunction with what ever external cloud utilities are available.

Secondly, a complete understanding of the platform is often necessary when an enterprise is innovating. It may be possible to outsource operational hosting but we believe that innovation and the deep understanding it requires will drive the need to maintain infrastructure locally.

Finally, there is value in being able to define a technology lifecycle that is independent of the commodity lifecycle that utilities must necessarily adopt. The utility providers will need to focus their offerings on the technologies that can be most economically commoditized.

Again, innovation may require the use of technologies that haven't been commoditized or are on their way to being commoditized and over a lifespan that is not defined by utility demand.

Eucalyptus has a background in academia, similar to the Xen project before it. Is this an advantage?
Wolski: Realizing that an unbiased answer to this question is nearly impossible for us, our answer is clearly "yes."

First, as a research project we were free to make a nearly endless number of exploratory "mistakes" without regard for their impact on the viability of the project. We simply did not know if it was possible to build a private cloud of sufficient scalability and fidelity to be useful and as such we were able to probe the boundaries of feasibility pretty much at will.

Secondly, as experimental system builders in academia, we have ourselves relied on open-source software to provide the building blocks for our research products. In so doing, we garnered a great deal of experience with the use and management of open source, particularly in different data center settings, that would have been hard to duplicate outside of the academic environment.