January 16, 2012, 9:09 PM — Analyst firms often use surveys to make predictions sound more believable. However, there generally is no connection whatsoever between what a survey predicts and what the outcome eventually is. For example, according to the surveys the U.S. President should be Hillary Clinton and the person most likely to win the Republican nomination were a bunch of folks that aren't even in the race anymore.
Steve Jobs wouldn't use focus groups because he believed, and was right, that people have no idea what they are likely to do in a future event. Now surveys can be good at explaining why someone did something, but they aren't so hot at trying to predict what folks will do.
[ Free download: Windows 8 Deep Dive Report ]
The most recent annoying survey is the one that predicts Windows 8 Tablets will fail because there is no demand for the product. I'm sure a similar survey made a year before the iPhone or iPad would have likely concluded the same thing about those products.
Let's start out this year by talking about why you can't trust surveys.
My Experience With Surveys
When I first left IBM to join Dataquest my goal was to create the most accurate predictive operating system forecast ever created. Based on my graduate work on marketing studies, coupled with a number of years as a competitive analyst, and then as lead for the spinout of the IBM software business. Over the years I'd purchased millions of dollars' worth of predictive research and virtually all of it had been unreliable. This was because the typical projection was made based largely on historical data, or if a new offering, based on how much a critical vendor was contributing to the related service.
If something had been growing at 10% it likely would be projected to continue growing and, surprisingly, the most generous vendors would be predicted to succeed and the least generous predicted to fail.
Granted some of the folks doing this had a lot of influence and so the predictions became self-fulfilling at times. But the real problem was that upcoming problems, because no one wanted to a paying vendor angry, were often not reported even when known.