Algorithms everywhere: Can IBM automate decisions?

This was originally posted at ZDNet's Between the Lines.

IBM is outlining a vision--and of course a new services unit to go with it--that takes a little time to grok.

Big Blue speaks about the "information journey," about fact-based enterprises, and about nudging out gut calls in everyday management for decisions based on hard, cold facts. When you boil it all down, Big Blue is talking about providing a bag of algorithms that will automate many of your business decisions.

Sitting through IBM's series of presentations on Tuesday about how we'll all work for fact-based enterprises in the future left me with a few nagging questions (with a dearth of concrete answers): What's the role of intuition and gut calls in management? Are we all becoming quant black boxes of management? Can we see around corners for real? What's the prototype of a fact-based enterprise? How long will this journey take and what does the end state look like?

The answers after a few rounds with IBM execs: intuition will still matter, but for the bigger decisions. Some decisions will be automated (think a call center manager who will look for a simple yes/no answer when deciding on whether to extend a warranty). Predictive modeling will be everything. The journey will take awhile and the blueprint will be industry specific. (Rest assured, IBM will consult with you every step of the way).

"We're developing a bag of algorithms to be plugged together," said William Pulleyblank, chief of IBM's center for business optimization. Pulleyblank should know: he led the Blue Gene supercomputing effort. Remember Blue Gene? It might be your boss someday (only half kidding).

There's little question that IBM could hit a sweet spot for its consulting business. Simply put, risk management is on every manager's mind. Anecdotally, execs talk about risk management more. And I've seen it up close. ZDNet's risk management report about Goldman Sachs had a lot of interest, but it wasn't just the financial wonks on board--it was IT folks. Turns out everyone wants to see around the corner for the latest bogeyman. Risk management and mitigation were something that used to be tucked away in the corner. Today they're everyone's business.

In some respects, IBM's big pitch boils down to better data analytics. How can you take all that stuff you've been collecting and find some real intelligence? How can you account for systematic problems? Real-time reaction to customer needs? Altering pricing on the fly?

It would be easy to dismiss IBM's effort as another wrapper to sell software, hardware, and services, but the vision is big and makes a lot of sense. However, I couldn't help but feel a little uneasy. Algorithms are partly responsible for this financial mess. Sometimes the black box fails miserably. And sometimes a little human intervention is needed. And the biggest worry: if managers wind up just looking to a screen to make a call--yes or no--doesn't that make us a business equivalent of a GPS slave where no one will be able to read a map in a decade. Thankfully, IBM's vision doesn't include an artificial intelligence theme. (Maybe next decade.)

Meanwhile, IBM's Fred Balboni, head of the business analytics and optimizations services unit, says Big Blue will go easy on pitching the "intergalactic projects" that may spook clients. The game plan is to serve up valid business cases for being more analytic and deliver returns within a year. Over time, companies can become the so-called fact-based business.

Here's the big picture:

But are "applied semantics" better than "human insight?"

In Corporate America you can easily (predictively) model some cultural issues. CEOs will love this "fact-based" management, but the front line folks can resist. IBM notes the hurdles, but expects rapid adoption--at least something faster than the ERP revolution of the 1990s. Why? Younger folks already look at their PCs--and Google--as an answer machine, said Brenda Dietrich, vice president of business analytics and mathematical sciences at IBM Research.

This blur of data, computation, and decision-making looks great on paper. The reality is that data architecture is messy already and could use the clean up. Ask the Bill Eimicke, deputy commissioner of New York City's fire department. Eimicke got on this fact-based enterprise bandwagon in 2007 following the deaths of two firefighters as the Deutsche Bank building was being dismantled. (It was heavily damaged on Sept. 11, 2001.).

There were "flammable things" amid the demolition work. The problem: the FDNY didn't know there were flammables because it didn't have access to the data from the plethora of city agencies that inspect buildings.

Eimicke, a Columbia University academic on loan to the fire department, said that the data was there, but not in a form that was usable. His project: aggregate all of the data on buildings in New York so it can prioritize inspections. Maybe a few extra facts will save another Deutsche Bank building from happening. "That crisis triggered our project," Eimicke said.

Indeed, firefighting is an obvious fact-based business. Financial services, health care, and retail are other obvious verticals ripe for some algorithm love. Financial services firms are the farthest along on this algorithm utopia.

What can go wrong?

There are multiple industries that could benefit from a little fact-based decision making, but there are landmines ahead. When I asked Pulleyblank, he rattled off a few items. Given more time, a top 10 list would have emerged.

Here are Pulleyblank's landmines ahead:

• Data quality. Any automated process is only as good as the data being used. If the data has errors in it IBM's algorithms won't work as well. The solution will be better filtering to diminish "noisy data," says Pulleyblank. It's not like a company is going to go back and clean 30 years of data.

• Risk management techniques. This entire concept of managing systemic risk--and determining everything that could go wrong--is young.

• The need for real-time reactions: Are companies ready to respond and make decisions in real-time?

• An unexpected shift in the world. Pulleyblank acknowledges that predictive modeling only goes so far. What are the Black Swans ahead?

• A spectacular failure. "We can't afford a spectacular failure," said Pulleyblank. What would be a spectacular failure? Try a major Northeast blackout where the algorithms in the smart grid are at fault.

It's a bit early for those concerns, but it is something to think about. If IBM's vision plays out, algorithm management will be a key component of businesses everywhere.