Analysts as a lagging indicator of success
Gartner, Forrester, and other analyst firms tend to be great predictors of the past, probably because that's where they get their money.
A few months ago Sam Lawrence of Jive Software spent some time grading Forrester and Gartner as analysts. His verdict? Neither does a fantastic job, but Forrester is much better at servicing a small but growing vendor like Jive.
My own experience with analysts is mixed. Analysts tend to be great at predicting the past, but far less adept at predicting the future, which is actually what customers expect from them. If you look at such things as Gartner's Magic Quadrant, it is great at showing where the industry was, rather than where it's going.
The problem is that analysts like Gartner get their information from the vendors that subsidize their research, as well as from CIOs. Neither is a good indicator of where the market is going.
As Billy Marshall classically wrote, the CIO tends to be the "last to know" about new IT initiatives. As for the vendors, the only ones with enough cash to subsidize research are the same ones that have a vested interest in protecting existing cash cows. In other words, the past.
Analysts, then, are a lagging indicator of success. They tell an enterprise buyer from whom she should have purchased software and hardware a few years ago, not where she should invest IT dollars tomorrow. As an example, despite the massive influx of open-source vendors in the enterprise, Gartner persists in believing that open source is years away from making a dent in the enterprise, and you'll rarely find an open-source vendor in a Gartner Magic Quadrant. Here's a recent Magic Quadrant for Business Intelligence. No open source need apply.
Putting open source aside, some analyst research is so egregiously off that it's almost comical.
Take Gartner's premier placement of Xerox in a recent Enterprise Content Management report. Microsoft is a leader in ECM. Documentum is a leader. Interwoven is a leader. Xerox?
Let's put it this way. According to Gartner's own data (PDF), it's a rounding error in terms of market share. Nor is that market share growing. CMS Watch, a leading content management publication, has hardly mentioned Xerox in the past two years, which isn't good compared to how much it has tracked Microsoft SharePoint.
I'm not trying to criticize Xerox. Rather, I'm pointing out the abstruse logic that goes into Gartner's Magic Quadrant, as just one example of analysts' rear view-window analysis. The 451 Group, Redmonk, and other small analyst firms do a much better job at spotting the future, primarily because they actually spend time talking with customers and vendors involved in buying and selling that future.
Coming back to Sam Lawrence's original post, he mentioned that when he invited Gartner to attend a summit with a wide range of enterprises interested in buying into social collaboration software, the analyst firm balked (after suggesting it would cost Jive tens of thousands of dollars to have them attend). While we can perhaps excuse Gartner for not wanting to board a plane every time a small vendor points to a customer reference, I have a little less patience for analyst firms that dismiss smaller vendors without talking to any customers.
At Alfresco, an analyst firm interested in profiling us in a report asked us for references. We gave them several including two of the world's largest media companies and one of the world's largest financial services companies. The analyst firm never called the customers. It ended up writing a tepid review without ever talking with the people that matter most: customers.
It's time for analyst firms to get their research in different, better ways. It's time to look out the windshield, not the rear view mirror. Otherwise, the only thing enterprises will discover in analyst reports is a reminder of all the vendors that they should have bought from years ago.