June 10, 2005, 3:37 PM — Imagine the brain of a software developer in which the left hemisphere
is devoted to pragmatism and the right hemisphere is devoted to
intellectual rigor and correctness. Let's travel through that brain,
surfing on a wave of thought, from left earlobe to right earlobe. The
thought we are surfing on is this: "How do I know my software is
correct?".
As we travel from left to right, neurons fire in all directions. Out of
the synaptic snowstorm that ensues, three answers solidify from
different sectors of the brain. They are, in order:
Sector 1 Answer: Working software is software that gets the job done
before my boss fires me.
Sector 2 Answer: Working software is software that passes all my tests
including the test of time. (i.e. not only has it passed all the tests
I have put it through but it has been running error free for a long
time.)
Sector 3 Answer: Working software is software that I can prove to be bug
free using a formal, mathematical theorem proving mechanism.
As an industry, we have spent a lot of time in sector 1. Software
development, especially commercial software development, is a high
pressure activity in which engineering requirements need to be traded
off with a heady concoction of requirements from other aspects of the
business. We need to get the stuff to work before the customer walks
down to our competitors, before the boss blows a gasket and so on.
As software development has matured over the years, we have codified
more and more best practice. This best practice lore is pushing us more
and more towards sector 2. This is good. Anything is better than the
'ship it, then worry', realities of sector 1.
Few would argue that in an ideal world, the science of computer science
would allow us to skip the empirical world of sector 2 and head straight
for the logical nirvana of formal correctness over in sector 3. After
all, as Edsgar Dijkstra famously observed, testing can only show the
*presence* - but not the *absense* of bugs. We can never say that we
know our software works, we can only say that, so far, we have failed to
show that it does not.
Personally, I think we will stay in sector 2 for a long, long time.
Perhaps even for ever. You don't need to spend much time contemplating
the enormity of the problem of formal correctness to realize that we
have a long way to go to get there. Indeed, we may never get there.
Formal correctness as a field of endeavor appears to have a history akin
to that of artificial intelligence. After a period of considerable
activity and excitement in the mid to late Eighties, activity has died
down and expectations seem to have been lowered as to what can
realistically be achieved.
So, it would appear, testing is where it is at. Certainly the field is a
hive of activity at the moment which phrases like 'extreme programming'
and 'test driven development' on many a software developer's lips.
I have always been a fan of testing. In the XML and EAI worlds where I
spend a lot of time, we make extensive use of test suites to test system
modules in isolation before plugging them together. XML is a big help
here as we can formally capture the structure of information to be
processed using a machine readable contract language known as a schema.
Common examples of XML contract languages include DTD, Relax NG or W3C
XML Schema. We can then instruct the software at runtime to check that
the data meets the expectations expressed in the contract. Note that I
said 'at runtime'. Put another way, we check things dynamically.
A testing focus has a way of veering your thoughts towards a dynamic
model of software construction. A model in which adding running tests is
a continuous activity, not an activity performed solely as part of a
beta test programmer. In such a model, it is the most natural thing in
the world to think of software itself as stuff that is as changeable as
the tests are. The concept of fixed, pre-compiled objects begins to
grate against the ear in this world. We don't want to have to drop our
testing tools to re-run some longwinded compilation process for the
system. We want to turn around software changes quickly. The quicker we
can turn them around, the more quality time we can devote to tests. The
more tests we have, the higher quality our software will be.
