Outsourcing the middleware layer

September 11, 2006, 9:24 PM — Apparently (I am too young to remember) photocopier machines were greeted with suspicion in some quarters when they were first introduced. How can you know that the machine has made a perfect copy of your vital document? I'm told by someone who remembers those days that he used to proof read the copies coming off the photocopier to check for accuracy.



This sounds silly now, which isn't to say that mechanical copying no longer needs to be done with care [1]. Rather, we are now comfortable with the idea that all we need is a quick visual inspection of the photocopied page to see if it is all there. Once we have that, we do not need to inspect the individual letters to know we have a good copy. We trust the photocopier.



Today, we too are doing the metaphorical equivalent of proofreading the photocopies with some of our tools, technologies and techniques. Some of today's 'new' and untrusted things will become part of the furniture tomorrow. They will be taken for granted. We will learn to trust them. Our children will giggle while looking back at the time when these things were not trusted. So it goes.



The constant availability of web services out there in the cloud is one such idea. Today, we do not trust the cloud and the services on it to be always available. Few of us can remember any incidences in recent time when, say google.com or amazon.com or live.com was offline but we still do not trust them to be always there and available.



I predict that this day will pass. The day will come when outages of big commercial services on the cloud are as unusual as outages in the phone system or the electricity supply system. Sure, losing power will also lose you the services on the cloud but your business most likely has bigger problems to worry about when the power goes.



What if this is true? What if we come to treat the availability of services on the cloud the same way we treat availability of the phone system? Well, an interesting possibility presents itself in the enterprise application integration space. In any complex integration of disparate business processes and their concomitant computer systems, a lot of glue is required. Collectively this glue is known as 'middleware'. Middleware takes many forms including such commonplace phrases as RPC, ORB, asynchronous messaging, REST and so on. All this glue share two salient features. Firstly, in order to work, it must work all the time. All day every day. Secondly, nobody wants the pain of running the glue infrastructure themselves.



Middleware tends to be just plain hard. Middleware tends to be expensive. Middleware must work all the time which can make it very expensive indeed. Middleware tends not to be the core business of enterprises that depend on it (except of course, the vendors that sell it.). So, it is expensive, complex and not a core competency. What does that add up to? Well, we would all outsource it if we could right?



Wrong. The idea of outsourcing the middleware layer is a beautiful theory that falls foul of an ugly little fact. So it goes. Even if we did trust the infrastructure to always be there, we do not trust the infrastructure owners to ensure that our mission critical data is completely secure from eavesdropping. That really puts a spanner in the works...or does it?



All around us, services designed to provide middleware glue on the cloud are popping up. OSCS is one example. It builds on top of Amazon's S3 service[3] to provide developers with easy to use middleware services that sit out there in the cloud. Perhaps such services will be limited to those that exchange data that does not have critical security requirements? Perhaps traditional encryption mechanisms will save the day for the use of such services in enterprise scenarios?



Or perhaps something completely different will come along. A development as new and different as the photocopier was in its day. I have been reading about Owner Free Filing Systems[4]. Simply put, OFF systems store information by chopping it up into many separate, very large numbers. To create an intelligible message, you must combine particular numbers together. Unless you know how to combine the numbers, you cannot understand the message. Furthermore, any individual large number used on the OFF might be a constituent part of many separate messages.



The concepts underlying OFF may have major implications for the concept of copyright in the digital age. That is a whole new topic and one that I will not go into here.