The advent of the Web 10 years ago opened up vast banks of information to anyone with an Internet connection. Now, clever programming tricks that use data from public Web sites are letting developers mix up that information to suit consumers' particular needs.
Cheap Gas, a Google Maps-powered interface, is part of the phenomenon. Dozens of such nifty "mash up" programs, built by independent developers using tools provided by online businesses, provide services beyond those of the base sites.
"Mash-ups" that let people combine information from different Web sites are reshaping the Web experience, allowing independent developers to better control and customize the information consumers can get.
Experts predict Web site owners will increasingly resemble software companies: To generate traffic and sales, they will encourage add-on products and Web services.
They also portend big changes for site owners--at least, for those who want to take part in the next stage of the Web, called Web 2.0 by some. Instead of treating the Web just as a handy way to publish information, businesses need to start acting like software companies and encourage programmers to build services on top of their platforms, analysts say.
"The conclusion that many savvy Web presences had is very similar to what software companies have realized with open source: As creative as your organization may be, the community at large will always be more creative," RedMonk analyst Stephen O'Grady said.
By mimicking software companies and encouraging coders to build upon their data, Web site operators can give consumers more-tailored services--and themselves richer products.
Established Web companies have been onto the idea of wooing developers for some time. Amazon.com, for example, published its Web services APIs, or application programming interfaces, three years ago. Its annual report (see PDF) touts its "seller platform" for letting third-parties sell through its Web site, and it has regular software product releases.
At the same time, Google and Yahoo have been hiring luminaries in the software industry, such as well-known engineer Adam Bosworth, to help define how software services will be delivered over the Web.
Wild, wild Web ideas
Allowing individuals to play with their Web site data has resulted in programs that the companies might never have thought of. For example, Adrian Holovaty, a 24-year-old programmer, built a Web site called Chicagocrime.org that taps into Google Maps to display where crimes occur in Chicago.
Holovaty, whose day job was lead developer at the Lawrence Journal-World newspaper, said he wanted to provide a service to citizens of Chicago, and tackle a fun technical challenge. He spent about 40 hours on the job, spread out over a month of nights and weekends.
Another slick application, which taps into Amazon's book search service, is BookBurro which lets people compare book prices. This sort of Web service can be constructed pretty quickly: Instead of having to build a book search and e-commerce engine from scratch, one person can create something entirely new by combining Amazon's tool with other data sources.
Inviting third-party developers to build on top of a company Web site--much the way Microsoft woos outside programmers to its Windows operating system--creates a healthier business, advocates have argued.
eBay, for example, already gets more than 20 percent of its listings via programs created by outsiders to automate the auction giant's process. People can still use the regular eBay Web site to list items for sale, but the automated system enables sellers to move much higher volumes, said Greg Isaacs, the director of eBay's developer program. Isaacs noted that the program has mushroomed from 300 participants in 2003 to about 18,000 now.
"In the last two or three years, we've really seen working with developers as a competitive advantage," Isaacs said. "When I have to explain to management the importance of developers, it's very easy to do."
Mash-ups like these are emerging because a growing number of Web properties are releasing instructions, or APIs, on how to access their data. With these publicly available APIs--often published in standardized XML protocols--programmers get the documentation and tools needed to pull data from Web sites and to combine it with another information source to create something new. The effect is to put a great deal of power in the hands of outside individuals and to transform Web sites into programmable machines.
"Every site now is essentially fair game to go in, modify the site, change it, append it, trim it down," said Jeff Barr, a Web services evangelist at Amazon.
One factor in the burst of creativity is that publicly published APIs obviate the need for two Web properties to negotiate and cooperate in order to share information.In some cases, though, APIs are not well-documented or accessible, which means that programmers need to do more work. Still, advances in development technology, such as freely available , are lowering the skill level required to build Web applications. Today, the XML-based protocols required to access data are standardized and well known. and frameworks, many of which are available for free, give individuals access to powerful tools.
In addition, a Firefox Web browser add-on, called GreaseMonkey, lets script writers change a Web site's presentation. Developers are encouraged to share scripts and push the limits of customization.
"The really progressive companies will look at this and say, 'There must be something valuable in our organization, if developers are willing to reach inside and extract it for themselves. Let's figure out a way to work with those folks,'" Barr said.Opening up
With the success of Google and Amazon's programs before them, some Web start-ups are making developer involvement a central part of their business plan. Photo-sharing site Flickr and social-networking service 43 Things both publish APIs, for example.
In the case of Flickr, which was acquired by Yahoo in March, customers can find specialized tools to search through photos based on tags, or can sign up to get an e-mail notification when a change is made to a Flick page the customer subscribes to.
For its part, 43Things encourages developers to customize its service by giving them fine-grained control over the information the social networking site generates.
This all adds up to a shift in the Web. In effect, the nature of what a site can be has changed. Rather than being part of a publishing system, Web sites are becoming programmable, much like a PC's operating system.
Still, the experience of Holovaty at Chicagocrime.org indicates how the potential of the Web as a development platform is still in the early days. The GoogleMaps APIs were not fully documented when he started building the application, which created more work for him.
In addition, Holovaty has had to "screen scrape" data from the Chicago Police department Web site, or essentially cut and paste data from the site, since its data is not formatted nicely for third parties. "If the police department's site gets redesigned, my scraper breaks," he said.
For some, the programmable Web fulfills some of the long-held promises of the Internet boom. Consumer services, such as Microsoft's Hailstorm, touted the notion of giving consumers access to their data wherever they are. Business would be able to tap into the Internet "cloud" and procure and combine third-party Web services from a public directory, too.
Amazon's Barr, for example, sees the e-commerce giant's embrace of Web services as a sign that that shift to the next generation of the Web is under way--even if it's only the beginning.
"I don't think everyone has fully grasped the latent power of this yet," Barr said. "We're on the very, very leading edge of seeing this happen."