November 29, 2004, 11:03 AM — Shhhsh! Keep the noise level down out there for a minute will you? Come along with me as we eavesdrop on a conversation happening right now between a business person and a systems analyst...
Analyst: "So Bob, I understand you are in charge of systems A and B. What do these systems do?"
Business person: "That's right Bill, I am. System A over there manages names, addresses and work telephone numbers of our customers. System B over there stores names, addresses and order history for the same customers".
Analyst: "And systems A and B are separate systems? They both store information about the same set of customers? They are not connected to each other or talk to each other?"
Business person: "That's right Bill. They are completely separate systems."
Analyst (external view as seen by Bob): "Ah, ok. I see. Great. Thanks".
Analyst (internal view unseen by Bob): "What a screwed up data model that is! Storing the same information in multiple places. Have these people never heard of data normalization? Sheesh!"
The urge to normalize a non-normalized data model to remove redundant data is very strong. The temptation is on a par with that felt by a 12 year old who is handed a sheet of bubble wrap and told not to burst any bubbles. Not an easy thing to do.
When I emerged blinking into the sunlight from my college years I was a fully trained up data normalization zealot. Nowhere in my college education did anybody point out, that data redundancy can be anything other than a hack introduced for performance reasons.
In recent years, I have seen a new side of data duplication. A side that deserves to get more air time than it currently does. Simply put, duplication creates resilience. Resilience is good. All around us are examples of Mother Nature building highly resilient systems based on enlightened data duplication. An Ant colony is perhaps the canonical example of a resilient, complex organization in which the unit of duplication is the humble ant. Instead of a small number of expensive highly reliable ants there are a large number of cheap, disposable ants that duplicate each others function in the colony. If you normalized an ant colony there would not be much left!
In business, we can think of enterprises as complex colonies in which computer systems play the role of ants. Performing tasks, storing information - the hive mind of corporations. Now, which makes the most sense :-
a) have a small number of expensive mission critical applications in which any individual data item or piece of business logic exists exactly once, or
(b) have a large number of non-mission critical applications that purposely duplicate each others data and business logic?
Bob the business man probably doesn't know it, but his data duplication may be one of the reasons his organization functions well and can continue to function in the face of failures of various kinds.
Bill the Data Analyst probably doesn't worry about the implication of failures on a fully normalized data model for Bob. After all, such considerations are merely implementation details.
I suspect that Mother Nature might beg to differ.
