Testing, testing, one to three, testing

June 10, 2005 —

Imagine the brain of a software developer in which the left hemisphere
is devoted to pragmatism and the right hemisphere is devoted to
intellectual rigor and correctness. Let's travel through that brain,
surfing on a wave of thought, from left earlobe to right earlobe. The
thought we are surfing on is this: "How do I know my software is
correct?".

As we travel from left to right, neurons fire in all directions. Out of
the synaptic snowstorm that ensues, three answers solidify from
different sectors of the brain. They are, in order:

Sector 1 Answer: Working software is software that gets the job done
before my boss fires me.

Sector 2 Answer: Working software is software that passes all my tests
including the test of time. (i.e. not only has it passed all the tests
I have put it through but it has been running error free for a long
time.)

Sector 3 Answer: Working software is software that I can prove to be bug
free using a formal, mathematical theorem proving mechanism.

As an industry, we have spent a lot of time in sector 1. Software
development, especially commercial software development, is a high
pressure activity in which engineering requirements need to be traded
off with a heady concoction of requirements from other aspects of the
business. We need to get the stuff to work before the customer walks
down to our competitors, before the boss blows a gasket and so on.

As software development has matured over the years, we have codified
more and more best practice. This best practice lore is pushing us more
and more towards sector 2. This is good. Anything is better than the
'ship it, then worry', realities of sector 1.

Few would argue that in an ideal world, the science of computer science
would allow us to skip the empirical world of sector 2 and head straight
for the logical nirvana of formal correctness over in sector 3. After
all, as Edsgar Dijkstra famously observed, testing can only show the
*presence* - but not the *absense* of bugs. We can never say that we
know our software works, we can only say that, so far, we have failed to
show that it does not.

Personally, I think we will stay in sector 2 for a long, long time.
Perhaps even for ever. You don't need to spend much time contemplating
the enormity of the problem of formal correctness to realize that we
have a long way to go to get there. Indeed, we may never get there.
Formal correctness as a field of endeavor appears to have a history akin
to that of artificial intelligence. After a period of considerable
activity and excitement in the mid to late Eighties, activity has died
down and expectations seem to have been lowered as to what can
realistically be achieved.

So, it would appear, testing is where it is at. Certainly the field is a
hive of activity at the moment which phrases like 'extreme programming'
and 'test driven development' on many a software developer's lips.

I have always been a fan of testing. In the XML and EAI worlds where I
spend a lot of time, we make extensive use of test suites to test system
modules in isolation before plugging them together. XML is a big help
here as we can formally capture the structure of information to be
processed using a machine readable contract language known as a schema.
Common examples of XML contract languages include DTD, Relax NG or W3C
XML Schema. We can then instruct the software at runtime to check that
the data meets the expectations expressed in the contract. Note that I
said 'at runtime'. Put another way, we check things dynamically.

A testing focus has a way of veering your thoughts towards a dynamic
model of software construction. A model in which adding running tests is
a continuous activity, not an activity performed solely as part of a
beta test programmer. In such a model, it is the most natural thing in
the world to think of software itself as stuff that is as changeable as
the tests are. The concept of fixed, pre-compiled objects begins to
grate against the ear in this world. We don't want to have to drop our
testing tools to re-run some longwinded compilation process for the
system. We want to turn around software changes quickly. The quicker we
can turn them around, the more quality time we can devote to tests. The
more tests we have, the higher quality our software will be.

The combination of the desire for dynamic, rapid coding/testing with
XML's emphasis on run time validation of data can color your assessment
of programming languages for projects quite significantly. Simply put,
there are programming languages that get you to 'declare' lots of stuff
in advance, putting instructions about the shape of data, for example,
directly into your programs which are then "compiled" into an
unchangeable, machine readable form. These are broadly known as static
programming languages. C++, Java, C#, Delphi fall into this category
(more or less). Then there are programming languages that let you do
this sort of stuff when your software is running and/or allow you to
change the software after it has been deployed. They are broadly known
as dynamic or interpreted languages. Python, Jython, Smalltalk, Scheme
are examples (more or less).

Today, there is a significant imbalance between the worlds of static and
dynamic programming language use in commercial IT. The balance has
firmly been on the side of the static languages with the occasional
pockets of dynamic language usage. I suspect that the increasing
emphasis of testing in/during and after software development is
beginning to tilt the balance the other way.

It would be wrong I think to attribute the recent growth in interest in
Python/Jython, Smalltalk, PHP, Scheme, Ruby etc. purely to the emergence
of testing as a key weapon in software development. These languages have
many other attributes that appeal to software developers too. However, I
believe that testing is a key contributor and will be an even stronger
contributor in the future.

Helping to tilt the balance is the growing realization that the extra
formalisms in statically compiled programming languages, designed to
help you feel confident in the correctness of your programs, come at a
price that may not be worth paying. The details are beyond the scope of
this article but I would recommend Bruce Eckel[1] and Tim Bray[2] on
this subject.

So, what is the takeaway from all this? If you don't have a dynamically
typed programming language in your bag of tricks, get one now. If you
are not a test-oriented software developer chances are you soon will be.

[1] http://www.mindview.net/WebLog/log-0025

[2] http://www.tbray.org/ongoing/When/200x/2003/05/08/FutureLanguage

ITworld.com, Ebusiness in the Enterprise