January 16, 2012, 9:09 PM — Analyst firms often use surveys to make predictions sound more believable. However, there generally is no connection whatsoever between what a survey predicts and what the outcome eventually is. For example, according to the surveys the U.S. President should be Hillary Clinton and the person most likely to win the Republican nomination were a bunch of folks that aren't even in the race anymore.
Steve Jobs wouldn't use focus groups because he believed, and was right, that people have no idea what they are likely to do in a future event. Now surveys can be good at explaining why someone did something, but they aren't so hot at trying to predict what folks will do.
[ Free download: Windows 8 Deep Dive Report ]
The most recent annoying survey is the one that predicts Windows 8 Tablets will fail because there is no demand for the product. I'm sure a similar survey made a year before the iPhone or iPad would have likely concluded the same thing about those products.
Let's start out this year by talking about why you can't trust surveys.
My Experience With Surveys
When I first left IBM to join Dataquest my goal was to create the most accurate predictive operating system forecast ever created. Based on my graduate work on marketing studies, coupled with a number of years as a competitive analyst, and then as lead for the spinout of the IBM software business. Over the years I'd purchased millions of dollars' worth of predictive research and virtually all of it had been unreliable. This was because the typical projection was made based largely on historical data, or if a new offering, based on how much a critical vendor was contributing to the related service.
If something had been growing at 10% it likely would be projected to continue growing and, surprisingly, the most generous vendors would be predicted to succeed and the least generous predicted to fail.
Granted some of the folks doing this had a lot of influence and so the predictions became self-fulfilling at times. But the real problem was that upcoming problems, because no one wanted to a paying vendor angry, were often not reported even when known.
So in 1995 I published what turned out to be my most accurate five-year forecast, it also became the reason I eventually stopped trying to do highly accurate forecasts. It started with a complex model that looked at historic trends, factored in known future events, and even took into account environmental changes and, modestly, became the benchmark for accuracy with one exception.
It accurately predicted the decline of UNIX, MacOS, OS/2 and Netware but it was overoptimistic of Windows 95. This last prediction was due to a bad assumption. We'll get to that in a moment. However, on these other declines it not only was accurate it showcased the downside to negative predictions. Because, suddenly a lot of executives appeared focused on making me an ex-analyst and the Dataquest CEO wasn't exactly my best friend (in fact that entire first year I seemed to constantly be on the bleeding edge of getting fired. The only thing that saved me is that this survey was getting a lot of press.)
One of the funniest was Novell where their then CMO told my boss that unless I changed the forecast she would cancel her subscription to our service. Upon being told that Novell had never subscribed, she sent over a check for $75K to subscribe and I never did change the survey.
As I mentioned above, where I missed however, was on Windows 95 sales projections and I missed by a lot. This was because after Windows 95 shipped Microsoft stopped demand generation, something we had not anticipated, and the negative news on problems with the product filled the marketing gap and folks stopped buying. Sold out stores with lines became unsold piles of Windows 95 boxes.
So even in a case where we tried to predict every element of a future world (and this took weeks of work) missing one assumption (granted none of us thought any company would be this stupid) resulted in a miss. Had we been aware of this variable we could have predicted more accurately the outcome and Microsoft might have avoided it.
However this brings up the second problem with surveys.
Predictive Surveys Can Change the Outcome
