Tightly integrated? Just say no!

by Sean McGrath

February 4, 2003 —

As I write this, there is a barely audible, irregular thumping noise
emanating from a dusty 386 PC in the corner of the room. I fear a system
failure is on the way.

The noise reminds me of a recent echo cardiograph I undertook in which
the full extent of my advanced addiction to caffeine was transformed
into sound. Thumping sounds caused by palpitations emanating from the
most important muscle I possess. I no longer drink coffee by the way.

If the noises from the PC in the corner get any louder, I can think of a
number of people who work with me who may find themselves experiencing
stress - rather than caffeine induced - palpitations. You see, that
dusty PC in the corner is running a system last touched about seven
years ago. I kid you not. Seven years ago. It has given tireless, 24/7
service during those years with only occasional weeding, mulching and
the odd tender loving warm boot from tender, loving hands.

Sure we have backups. Weekly backups. We even have a machine in hot
backup mode ready to take its place if it goes belly up. Despite all
this, we are still nervous. Nervous because the system has not been
touched in years. Nervous that a number of its hardware and software
components are now antiques in IT terms. Nervous that while some of the
system was built in-house, some of it was outsourced and the people
responsible for its development have long since disappeared to find
themselves in the jungles of South America or snowboarding their way
around the USA.

We are nervous because we know in our hearts and souls, that under that
irregular thumping noise, lies a ball of tightly integrated software
applications.

For 'tightly integrated' in the above sentence, feel free to read 'ball
of mud'[1]. Unless you are extremely lucky, you have some balls of mud
in your sphere of influence too. Can you think of any systems you know
that you are afraid to touch? Systems that have become so tightly
knotted over the years that no one human brain understands it all both
at a conceptual and detail level? I thought so.

Now. Given that balls of mud are the harsh reality of many IT systems
and given that tight integration was the catalyst for creating them,
isn't it odd that the phrase 'tightly integrated' still sounds so
appealing? I have seen many a C-level executive nod approvingly as the
phrase 'tightly integrated' drops from the lips of the system vendor or
the system integrator. Tight is good.

Actually, tight is almost always bad.

To most business people, 'integration' means process integration. That
is, some applications are tightly integrated if they work as one to
perform some business function. It is a very unfortunate fact of life
that the quickest and cheapest way to get started with process
integration is by means of spot-welding systems together. This so called
point-to-point integration accounts for most of the world's "tight
integration" code developed by system integrators.

The downside of tight integration can be severe. First, the costs of
modifying the point-to-point integrations can be significant. With the
inevitability of change, this is more than just of mere academic
interest. Indeed, I have known systems where the point-to-point
integration code had to be redone *from scratch* after an
innocuous-sounding change request from the business managers.

Second, and perhaps most significant in these days of high application
counts, tight integration does not scale. By that I mean, as the number
of point-to-point integrations grows, the number of "spot welds"
increases exponentially. I'll leave it to your powers of deduction to
conclude what happens to the dollar cost!

So, tight integration sounds good but burns a hole in your wallet. What
is the alternative? Here things get interesting. The buzz phrase is
'loose coupling' achieved by means of middleware.

Loose coupling comes in a number of flavors - some of which do not
address the fundamental problems of tight integration at all.

For my money, the most useless form of loose coupling is to allow parts
of an application to run on machine A with parts of it running on
machine B. This really only gains you the ability to deploy your tightly
integrated applications across a bunch of loosely coupled wires. It is
still tightly integrated in terms of the domino effect of changes.
Furthermore, unless it is approached very carefully it does not
significantly reduce the scaling problems either. DCOM anyone? CORBA
anyone?

For my money, the most powerful form of loose coupling is temporal
decoupling. With this approach, two systems A and B send/receive
instructions/data to each other over a reliable messaging infrastructure
in which there is no assumption that any request will receive an
instantaneous answer. The reliable messaging infrastructure guarantees
you that an answer to any request will be provided - but does not tell
you when.

It has been my experience that if you build systems this way, true
decoupling can result. By true decoupling I mean the ability to
plug-replace any component piece without bringing the whole applecart
crashing down. If my dusty 386 in the corner were running a truly
temporally decoupled collection of applications, it would not be a
health risk to me or my staff. We would sleep easy at night because we
would know - on a component by component basis - what its interfaces
were to the other components. We would also know that should one
component fail, the reliable messaging plumbing interlinking the
components would allow us to replace it without losing any partially
completed business processes.

So, the next time a software or integration vendor intones the phrase
'tightly integrated' you might ask what that really means - especially
in terms of ability to change and evolve the tightly integrated system.
Also, the next time you hear the phrase 'loosely coupled' you might ask
what that really means. If the answer is RMI or RPC Web Services
allowing pieces of the system to be geographically separated - yet
temporally coupled - I suggest you dig deeper with your questioning to
see what the proposed loose coupling approach actually gains you.

Loose coupling is good for you, but only if it goes beyond the knee jerk
middleware layer approach that simply distributes the tight integration
over loosely coupled wires. Unfortunately, at the moment, there is a lot
of this going on in the form of RPC style Web Services. The good news is
that the advantages of temporal decoupling - and the reliable messaging
infrastructure that typically goes with it - are coming to the fore.

I suspect that temporal decoupling will form one of the key
differentiators of Service Oriented Architectures (SOA) from previous
approaches to middleware in the years ahead. That will not stop sales
guys pitching tightly integrated SOA architectures to you though. Old
habits die hard.

[1] http://www.laputan.org/mud/mud.html

ITworld