Showing posts with label software development. Show all posts
Showing posts with label software development. Show all posts

Thursday, July 26, 2012

Life and Software–When The Obvious Is Not So Obvious

Some years ago, I was working on a project on Wall St. where one of the Big 4 consulting companies had just implemented a very expensive IT system.

Its architecture was glorious in theory, perfect in its purity and yet one day, this glorious manifestation took down the entire internal network of a major bank.

What happened?

Without getting too technical, the application was designed to get any message that needed to be displayed to its user from a centralized database of messages.

One day, someone inadvertently took the message database offline and the application realized that an error occurred when it couldn’t retrieve the messages it needed to display.

In order to display the error message describing what happened, the application naturally went looking for the error message in that database and it couldn’t find it, producing another error.

That error in turn caused the application to look for the same database to obtain the same error message to display and then ……

You get the picture.

The application architects and developers had failed to plan for the fact that the message database itself could disappear.

The application “panicked” and tried harder and harder to obtain the information it needed and in doing so, spun itself into oblivion while producing nothing in the way of a positive result.  Thousands of PCs started doing the same thing and the network came down.

A multi-million dollar system had been brought to its knees by a simple oversight.

Life Is Like That Sometimes

Sometimes Life is the same.  Something that we thought should be happening in Life is not happening and so we try again.

But instead of doing something different, we try the same thing and produce the same result.

Getting frustrated (or panicking) we do the same thing again.

It feels natural to do this until someone points out that in fact we are doing the same thing over and over. 

And if n0body points this out or they do and we ignore them, eventually our “network”; our brain, our strength, our courage or our faith in ourselves and others fails.

And then comes the inevitable crash just as the crash that occurred within the Bank I mentioned.

All it would take to avoid this crash is a slight change in our approach or a slight change in our understanding of the environment that we live within.

But sometimes when we are in the thick of things, the slight modifications needed in our actions or our environment are not easily visible – we’re too focused on rapidly finding a solution.

And when we get caught up in our ever-increasing need to find a solution, the solution we seek will evade us with ever increasing speed …. causing panic, pain or failure as a result.

The Objective Observer

In the case of the application I mentioned, the architects came to me and said “We’re stuck – what can we do?”.  They had spent weeks of meetings trying to sort it out.

I looked at the architecture and said “Perhaps if the error routine already had knowledge of the “Message database is not available” message without having to go get it, it could report the error without spinning itself into oblivion”.

A simple answer – easily discerned and seemingly obvious to me because I wasn’t buried in the weeds of the thing nor did I have any concern around admitting I had created the problem.

Life is the same.

Sometimes when we find ourselves buried in the weeds of something that doesn’t appear to be working, we need to find the objective observer who can point out what is obvious to them.

It took the Big 4 consultants with their $2500 per day bill rates a long time to suck up their ego before asking me what the issue was.

But eventually the embarrassment (and potential punishment) from failure was more powerful than the ego that was holding them back.

Einstein’s Law of Insanity applies here, the notion that we shouldn’t expect a different result from the same actions. However, I wonder if it should be renamed Einstein’s Law of Ego.

When we finally push our ego down enough to ask “Can you help me understand why this is not working?”, we open ourselves up to new results that are dramatically better.

Isn’t that better than spinning ourselves into oblivion?

Our level of perfection as human beings may be perfect in potential, just as this application’s architecture was perfect in theory.

But potential means nothing if the results don’t match the potential.

In service and servanthood,

Harry

PS  I was reminded as I wrote this about a conversation I had with an NTSB investigator years ago.  He mentioned that in some situations, a pilot was able to avert disaster when, as an unanticipated event occurred, he took 10 seconds to stop and ask “What is happening here?” instead of just instantly reacting to the situation and possibly making it worse if not fatal.

An interesting thought and somewhat related.

Tuesday, June 26, 2012

Software–Complex Solutions to Simple Problems

I’m having one of those rare moments when my blog will actually be related to my line of work (strategy and IT architecture) and not my personal passion about what is going on in the world.  I’ll try not to make it a habit. ;-)

In recent years, I’ve been seeing a disturbing trend in the world of software development where the solutions being produced are far more complex than the problems requiring the solution.

Equally disturbing are the number of IT groups who have forgotten that they exist to enable the business needs of their user community.  Many of these IT groups believe that their user community exists to fuel the IT group’s dreams of fun and experimentation and for this reason, fiscal and schedule accountability is rapidly being eroded in such groups.

While this may not seem significant to those not in the IT industry, these trends affect everyone.  Allow me to explain.

There have been a couple of cartoons over the years that many of us in the industry used to laugh at but as I get older, I don’t find them as humorous as I used to.

Here they are:

You guys start coding cartoon

Solving the problem before we know what it is.

What the user really wanted versus what they were given – click on the image for a higher resolution version to read the captions.

 

The IT industry has at its disposal an incredible plethora of tools, methodologies, frameworks and best practices that never existed when I first entered the industry in the early 80s. 

When used appropriately, adopting what best aligns with how an organization works and what its needs are, this collection of tools can produce powerful, robust, effective solutions.

When abused or used inappropriately, these same powerful enablers can create some incredible disasters (or ticking time bombs with as-yet-unknown catastrophic consequences).

It’s like splitting an atom.  If I do it properly, I can create electricity for thousands of years.  If I do it in a slightly different way, the resulting explosion can kill millions of people in a microsecond.

While many software solutions that have been created are very cool, do amazing things and have a lot of wow factor, I am finding more and more of them to be so complex that the people who created them are unable to easily describe the original problem or the solution they created.  If you can’t describe either the problem or the solution, how do you even know that the solution is the appropriate one?  How would you fix it if it breaks?

What is equally disturbing are the increasing number of IT departments writing software to satisfy their own interests and curiosities without care as to how this “research” can ever pay dividends to their organization.  As one former client and senior IT architect said to me as he prepared to spend a considerable amount of money on a project: “I don’t give a damn what the user wants.  This is what I want.”.

For this particular project, no business case had been established for the money that was about to be spent.  No business value for the users or the organization-at-large had been identified.  The users in fact hadn’t even identified a problem and therefore hadn’t requested a solution.

Undeterred, the architect knew what he wanted for his own needs.  He also knew that he needed users to approve of his plan in order for him to keep budget approval so the message offered to the user community was “we will enable your business to be more effective”.  Unfortunately, no one could actually qualify or quantify the subjective phrase “more effective” but that didn’t prevent the project from proceeding.

I eventually walked away from the project, unable to balance two conflicting project mission statements (a public one and a private one) and always having to flip flop between two different project plans (depending on who was in the room).

You Can’t Run From The Facts

In a Harvard Business Review article last year (Why Your IT Project May Be Riskier Than You Think), the writers found that 1 in 6 IT projects surveyed had a cost overrun of 200% and a schedule overrun of 70%.

Frightening numbers indeed and ones that should make the average IT group want to understand how they can avoid becoming such a statistic.  Unfortunately, most don’t care to do so and thus are likely to become another case study demonstrating how not to run an IT project.

Not only are the measurable costs disturbing, other less-than-obvious “costs” exist which should be of great concern to the average citizen.  In the case of governments, the military and the financial services industry, data privacy, data security and yes, even national security, are so tenuous that many people in my industry lose sleep waiting for the catastrophe which we know to be inevitable.

At the root of this problem – ego, denial and lack of focus

Every new project sets out with good intentions but too often repeats the mistakes of previous projects within the same organization or other organizations despite the amount of material that exists that people can learn from and use to improve their results.

Unfortunately, many IT groups believe that they would never create such a failure. Without having an honest assessment of their strengths, weaknesses and past performance, how do they really know?

There seems to be, sadly, a direct correlation between the size of the collective ego within an IT group and how unnecessarily overly complicated their solutions are. Such a correlation also negates the perceived need to understand how one can improve one’s execution and one’s result since the potential for failure is often perceived to be the forte of “everyone but us”.

One person in the online retail space who was soliciting me recently was asking me hypothetical questions such as “How would you design a parking lot for cars, trucks and motorcycles"?”  When I would ask specific questions about the requirements of the parking lot, he would keep insisting that the details weren’t important.

And therein lies the problem.

As long as IT departments stay in the world of conceptual irrelevance instead of being pragmatic and focused about what is needed, who needs it, when they need it by, what knowledge can be leveraged, who would make good collaborators and what the best solution for the problem is, many IT projects will continue to run over budget with late delivery at best and with the potential for a catastrophic ending at worst.

We have the resources to deliver a much better result when it comes to producing software.

The challenge is – do we have the will, the humility and the focus to learn how to apply these resources more effectively?

In service and servanthood,

Harry

PS A humorous example of an over-engineered solution to a simple problem.

484470_10150944833467890_739779156_n