Implementing the un-implementable

by Sean McGrath

October 12, 2007 —

Item: To computerize anything you must understand what it is you are trying to computerize. Controversial? Probably not. I suspect most people would say "of course".

Item: The process of understanding something lends itself to a highly rigorous approach. You start at the top, think of the system as a black box and then iteratively move downwards. Down and down you go, adding more and more detail to your understanding until you have "bottomed" it all out. This is called "systems analysis".

Once the systems analysis phase is complete, it is a mere matter of design followed by implementation to realize a computerized version of the analyzed system. Controversial? Possibly. Possibly not. Most of the standard controversy relates to how the analysis is performed and how/when design and/or coding should begin.

There is a school of thought that holds fast to the principle that systems analysis has to be fully bottomed out before design/coding can sensibly proceed. The classic embodiment of this approach is the so-called waterfall method[1].

There is another school of thought that holds fast to the principle that systems analysis is a fool's errand. Better to design a little, code a little, deliver a little and then iterate the procedure over and over again. Extreme programming embodies this approach[2].

These probably approximate the extremities of the method panoply. There are many variations/fusions in between. These are surplus to present requirements so I will skip over them apart from mentioning that a full bookshelf would be required to house a comprehensive coverage.

Today, I have a different form of controversy in mind. Brace yourself for a real shocker...

Item: The process of understanding a real-world system can only succeed if the system itself can actually be fully understood.

I will give you a minute to pick yourself up off the floor and ask the obvious question, namely, "Is he saying that systems analysis might not work?".

Yes, that is exactly what I am saying. There are systems/processes out there in the real world that I do not understand. Not only that, but you don't understand them either. Nobody understands them except perhaps by labeling them as "complex", or "non-deterministic" or "chaotic" or some such. Systems of this form actively resist all attempts at classic top-down analysis.

Examples? How about a computer system to predict tomorrow's weather? Tomorrow's closing prices for pork belly futures? Tomorrow's ballgame scores? Controversial? Of course not. We all have at least an intuitive feel for the fact that these systems are too complex to be rendered into a form that a computer can understand because we humans find them too complex to understand. If we do not understand something in great detail we cannot hope to explain it to a computer.

And yet, when was the last time you heard of a systems analysis project that resulted in the analysts putting their hands up and saying "This cannot be computerized. It is beyond human understanding."?

That tends not to happen...

Item: Some computerization projects fail because computerization is actually impossible, but nobody knew that when the project started.

Item: Fundamentally unimplementable computerization projects are often huge projects. Huge projects can become huge runaway trains very quickly with everyone assuming (incorrectly) that somebody, somewhere knows how all the bits fit together.

Assuming for a moment that there is a glimmer of truth to all of this... What does this tell us about the waterfall versus iterative approaches to system development? Sadly, I think it tells us something and nothing at the same time.

On one hand, it could be argued that this is an argument in favor of the waterfall method. If a system is unimplementable then at least the waterfall gives you checkpoints and milestones to find this out and take remedial action.

On the other hand, iterative methods have a more organic feel to them. Complexity is grown over time rather than built in up front.

Complexity becomes an emergent property[3] of how individually implementable (and therefore simple) systems inter-operate. This seems to be how the universe churns out complexity for us to wonder at. Why should software be any different?

But then again, iterations can become circular. You can spend your life iterating without ever establishing if the end-goal is an achievable one. Design a little, deliver a little, tear it down, start again. Design a little, deliver a little, tear it down...

The picture is far from black and white.

Item: Be careful out there.

[1] http://en.wikipedia.org/wiki/Waterfall_model

[2] http://en.wikipedia.org/wiki/Extreme_Programming

[3] http://en.wikipedia.org/wiki/Emergence

ITworld.com