Thursday, July 26, 2012

Life and Software–When The Obvious Is Not So Obvious

Some years ago, I was working on a project on Wall St. where one of the Big 4 consulting companies had just implemented a very expensive IT system.

Its architecture was glorious in theory, perfect in its purity and yet one day, this glorious manifestation took down the entire internal network of a major bank.

What happened?

Without getting too technical, the application was designed to get any message that needed to be displayed to its user from a centralized database of messages.

One day, someone inadvertently took the message database offline and the application realized that an error occurred when it couldn’t retrieve the messages it needed to display.

In order to display the error message describing what happened, the application naturally went looking for the error message in that database and it couldn’t find it, producing another error.

That error in turn caused the application to look for the same database to obtain the same error message to display and then ……

You get the picture.

The application architects and developers had failed to plan for the fact that the message database itself could disappear.

The application “panicked” and tried harder and harder to obtain the information it needed and in doing so, spun itself into oblivion while producing nothing in the way of a positive result.  Thousands of PCs started doing the same thing and the network came down.

A multi-million dollar system had been brought to its knees by a simple oversight.

Life Is Like That Sometimes

Sometimes Life is the same.  Something that we thought should be happening in Life is not happening and so we try again.

But instead of doing something different, we try the same thing and produce the same result.

Getting frustrated (or panicking) we do the same thing again.

It feels natural to do this until someone points out that in fact we are doing the same thing over and over. 

And if n0body points this out or they do and we ignore them, eventually our “network”; our brain, our strength, our courage or our faith in ourselves and others fails.

And then comes the inevitable crash just as the crash that occurred within the Bank I mentioned.

All it would take to avoid this crash is a slight change in our approach or a slight change in our understanding of the environment that we live within.

But sometimes when we are in the thick of things, the slight modifications needed in our actions or our environment are not easily visible – we’re too focused on rapidly finding a solution.

And when we get caught up in our ever-increasing need to find a solution, the solution we seek will evade us with ever increasing speed …. causing panic, pain or failure as a result.

The Objective Observer

In the case of the application I mentioned, the architects came to me and said “We’re stuck – what can we do?”.  They had spent weeks of meetings trying to sort it out.

I looked at the architecture and said “Perhaps if the error routine already had knowledge of the “Message database is not available” message without having to go get it, it could report the error without spinning itself into oblivion”.

A simple answer – easily discerned and seemingly obvious to me because I wasn’t buried in the weeds of the thing nor did I have any concern around admitting I had created the problem.

Life is the same.

Sometimes when we find ourselves buried in the weeds of something that doesn’t appear to be working, we need to find the objective observer who can point out what is obvious to them.

It took the Big 4 consultants with their $2500 per day bill rates a long time to suck up their ego before asking me what the issue was.

But eventually the embarrassment (and potential punishment) from failure was more powerful than the ego that was holding them back.

Einstein’s Law of Insanity applies here, the notion that we shouldn’t expect a different result from the same actions. However, I wonder if it should be renamed Einstein’s Law of Ego.

When we finally push our ego down enough to ask “Can you help me understand why this is not working?”, we open ourselves up to new results that are dramatically better.

Isn’t that better than spinning ourselves into oblivion?

Our level of perfection as human beings may be perfect in potential, just as this application’s architecture was perfect in theory.

But potential means nothing if the results don’t match the potential.

In service and servanthood,


PS  I was reminded as I wrote this about a conversation I had with an NTSB investigator years ago.  He mentioned that in some situations, a pilot was able to avert disaster when, as an unanticipated event occurred, he took 10 seconds to stop and ask “What is happening here?” instead of just instantly reacting to the situation and possibly making it worse if not fatal.

An interesting thought and somewhat related.

No comments:

Post a Comment