The (non) wisdom of crowds

Some years ago I read an entertaining book by James Surowiecki entitled "The Wisdom of Crowds"[1]. The central thesis of the book is that the opinion of a large number of people, when averaged out, can compare favorably with the wisdom of the experts in any given field.

There are times when that absolutely works but equally - as the book acknowledges - there are times when it does not. Yes, you guessed it, I am going to say that IT is one of there areas. Actually, I am going to go further and suggest that in any technical field where a lot of complexity is hidden from non-specialists in the interests of product commoditization, crowds tend not to be not very wise at all.

Two examples will, I hope, serve to illustrate where I am coming from. Take digital cameras as an example. I know next to nothing about photography (something I am trying to remedy) but I do know that the more pixels you have in your digital camera the better. Right? At this point, fellow digital photography neophytes are most likely nodding whereas the much smaller number of digital photography aficionados are shaking their heads and mumbling phrases like "sensor size" and "ISO sensitivity equivalents". Without going into details which would take me out of my depth very rapidly, the best camera is not the one with the best pixel-per-dollar ratio. It is not that simple.

I know a lot more about computers than I know about digital cameras (not that that would be in any way difficult) so let us take an example from there. Everyone knows that when selecting a computer, the higher the clock speed of the CPU, the better the computer right? At this point any non-IT-technical readers of this column (if there are any) are most likely nodding their heads whilst all the rest of us are collectively shaking our heads and mumbling phrases like "memory latency" and "multiprocessing". Without going into details that are superfluous to requirements here, the best computer is not the one with the best clock speed-per-dollar ratio. It is not that simple.

Both of the above metrics - pixels per dollar (for cameras) and clock speed per dollar (for computers) - are, to specialists in the field, flat out wrong and misleading. So why do their respective markets continue to use them? They are used because markets need simple metrics to make comparisons between competing products/approaches/philosophies quickly.

They need to make comparisons because they need to make financial decisions without understanding the details. Thus the metrics need to be simple, based on a small number of variables: preferably one. Unfortunately it is a short hop from simple to simplistic.

I must be honest at this point and say that my game plan for this article was to reach this point with a clear idea of a positive as well as a negative perspective on this simplistic metric phenomenon. I am really struggling with the positive outcome. Here is how I see it at the moment. Clever technical things are developed - digital cameras, computers, whatever - a market for these things develops. The market forms around mechanisms for differentiating products that compete in the space. The market ("the crowd that might be wise") latch on to a number of rather simplistic metrics - pixel count, clock speed and so on - that overly simplify matters.

Marketeers latch on to the simplistic metrics that work well and focus on making sure their products fare better in the metrics comparison game than those of their competitors. This feeds back to engineering who then build products maximizing the values of the simplistic metrics in order to stay competitive - even if it means producing inferior products. The conversation goes something like this:

Marketing: "Customers want more foo-per-dollar. I think we should bring out a product with double the foo-per-dollar ratio for the Christmas market."

Engineering: "Increasing the foo-per-dollar won't really make the product much better. If instead, we increased the bar-induction-loop-invariant we could double the quality of the product for the same engineering cost."

Marketing: "Sounds interesting. Let's look at that later. Today, customers do not use bar-induction-loop-invariant in their buying decisions. How long will it take you guys to double the foo-per-dollar ratio?"

So, rather than competition driving true technical innovation to greater and greater heights, market forces can distort innovation necessitating a slavish attention to the simplistic metrics the market has adopted. The result? Is the result a highly efficient, market driven roller coaster ride to mediocrity? Thankfully, this is not the full picture. After all, genuine innovation happens all around us too. There are clearly other forces, other dynamics, at work. The one that immediately springs to mind is the "disruptive technologies" ideas of Clayton Christensen[2].

From some of the material I have read on the Web recently, it appears that digital cameras are in the grip of a pixel-per-dollar quality tailspin. Perhaps it will be a short term aberration. However, given the length of time that the clock-speed-per-dollar metric has survived in the computer business, I will not be holding my breath.

[1] http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds


[2] http://www.businessweek.com/chapter/christensen.htm