Cloud, 'devops,' and 'shadow IT'

Last week, I attended the Cloud Connect Conference and Expo in Santa Clara, Calif., which is one of the biggest gatherings of cloud thought leaders and practitioners of the year. What I took away from that week was both a firm confirmation of the concepts I have covered in the past, and a surprising revelation of the maturity of some organizations with respect to those practices.

Most notably, there is a growing gap between the culture and practices of organizations that have embraced cloud as a primary IT model, and those that are trying to fit the cloud into their existing practices. The former has become very development- and application-driven, while the latter remains focused on infrastructure and data center operations models.

The developer-centric cloud model--a model based on the "devops" concepts I discussed in an earlier series--is one of working the code to meet the realities of the services available to it, including infrastructure services like computing and storage services. (The term "devops" is a combination of the words "developer" and "operations.")

The mantra "infrastructure as code" was spoken frequently during the conference, and the idea that infrastructure could be provisioned on demand, via an application programming interface has become a baseline assumption to these organizations. There is little argument about self-service versus controls and governance--developers are expected to develop apps that behave responsibly, and fix the bugs when they don't.

Its not just infrastructure that makes the cloud work for these organizations, however. It's also the variety of other services that can be found online, including coding platforms, payment services, notification services, data storage, and so on. These organizations tend to be focusing on being truly service-oriented, making the application (with some associated operations logic) be its own master.

These organizations also tend to be Web application-centric, and relatively small.

Meanwhile, the bulk of existing businesses with existing IT infrastructure and processes and budgets and...well, you get the idea...are working hard to figure out how cloud concepts are best adopted into their plans. And, as most of these IT groups tend to be infrastructure operations-centric, they are starting with the effects on that domain--not on application development or deployment, per se.

And, rightly so, there are some challenges to adopting cloud models from the infrastructure perspective. First, most existing infrastructure is not easily adopted into a cloud model. Differences in hardware profiles, network configurations, and even system density in the data center can make building a uniform cloud service from existing kit trend toward impossible. Or at least quite expensive.

So, most of these organizations begin with "greenfield" applications of cloud to their infrastructure. Sometimes that's specific applications, as in the case of a major financial firm that started with their business intelligence environment; and sometimes it's a "pilot" general purpose cloud for development and testing. But wholesale conversion to a private cloud model is rare. So, however, is wholesale adoption of public cloud as a primary model.

What is happening, however, is a realization that somehow, some way, culture and processes will have to change to make the cloud a powerful tool in the arsenal. It does no good to deploy a cloud with 5-minute provisioning times if there is still a committee with a 30-day backlog that has to approve the deployment. Having the illusion of infinite scale in terms of VMs or storage means nothing if your applications can't consume additional VMs or storage when they need to.

All of this brings me to a debate that took place on Twitter last week among some true practitioners of cloud, both in the "all in" sense and the "adoption into existing IT" sense.

See, what happened is this: Adrian Cockcroft is Netflix's "cloud architect," and a huge proponent of using Amazon Web Services or other public cloud services. Christian Reilly has built one of the first true private cloud infrastructures for a Fortune 500 company. The two of them sat down and had a long conversation one day last week, and Reilly walked away with a new appreciation for what the cloud could offer--and what it would take to get there.

When Reilly communicated this to his Twitter followers, a debate ensued with vendors, IT practitioners, and developers all chiming in. Reilly covers the core of the debate quite well in a recent post. The long and the short of it was that cloud adoption is different for those who's core business is software, versus those who use software to serve their core business. The former has the opportunity to change their culture and practices to get the most out of public cloud services, which is often the right thing to do for all but the biggest destinations.

The latter, however, must begin to adopt new practices--a painful process--as opportunity allows. The pace of adoption, technologies, and cloud models used and even evolution of development practices will be dictated by how quickly the organization as a whole can adopt --and adapt to--these changes. For most, that will take years.

Which brings me to the subject of "shadow IT"--the idea that business units working outside the realm of the IT organization and its processes cause problems for IT further down the line, and must be reined in. While the cloud certainly makes that easier--as one blogger suggests, check your employee's expense reports--I would contend that I wouldn't be so quick to squelch these projects.

The truth is, the lessons learned by teams working hard to adopt to cloud models and devops concepts will be invaluable to you regardless of what types of clouds IT ends up officially supporting--public, private, or hybrid. If you don't have the capacity to support experimentation with new models within IT, the initiative taken by "rogue" developers in this case might be the difference between getting ahead and falling behind.

Oh, and don't feel too badly if you never convert entirely to cloud infrastructure. Joe Weinman's cloud economics work demonstrates an advantage to mixed cloud and dedicated models for many applications. As my friend and colleague Chris Hoff notes in a classic post, "utility is in how you use things, not necessarily how it's offered."

You and your IT organization will be hearing a lot about online companies like Netflix, Good Data, and others building extremely agile businesses in the public cloud. If you think that you'll be asked to replicate some of that success, I bet you are right. If you think you can do that with technology alone, you are kidding yourself. Ask yourself, what are the ways in which these success stories can influence your existing IT environment.

That may be the multimillion dollar question.