I was in the process of completing my annual report on security vulnerabilities yesterday when the news reported that an explosion in a communication hub in downtown Calgary had compromised landline and 911 service for 30,000 Shaw customers, including some municipal and provincial services.
As I write this this morning, service is almost completely restored.
No biggy …. they only lost service for 12 hours or so, right?
Well, maybe …. but where was the redundancy that should have prevented the failure from impacting those affected?
Here was the cause for the failure:
The system-wide outage was caused when a transformer exploded in an electrical room at Shaw Communications’ downtown headquarters Wednesday afternoon. Although the backup system was activated, when the sprinklers came on, they were also taken out.
I guess they didn’t think of or couldn’t afford a non-water-based fire suppression system, typical for rooms containing mission-critical computer or communication equipment nor did anyone consider the impact of a total site loss, given that they kept the backup system in the same building as the primary system.
Then I think about the time I was in Newfoundland when a fire in a communication hub took out land lines, cell phones, Internet and all forms of communication (thus knocking out any use of debit / credit cards). The outage was only hours in duration but while the event was in progress, spokespersons for Aliant (the communication company that owned the building) were saying they had no idea when the outage would be corrected, creating extra concern at the time.
Was there redundancy of technology in this situation to protect consumers against a catastrophic failure?
Yes, according to Aliant. They had full redundancy of all systems. Unfortunately, the primary and backup systems were in the same building and shared a common power supply.
Where did the failure occur?
You guessed it – the power supply.
So much for redundancy in either of these events.
Ironically, the Aliant redundancy mistake, which occurred six years ago, was studied by information and communication providers across Canada to make sure no one repeated the same mistakes in the future.
Ooops.
When the World Trade Center came down, some of the major communication providers had been using it as a communication hub. After all, they figured, what are the odds that we could lose the entire site?
Sadly, we know the answer and communication in the NYC area was compromised as a result of the WTC collapse and an excessive number of people using the system in the hours of terror that followed.
When we build communications systems such as these, we strive to strike a balance between need and cost, factoring in the probability of various external factors and events. We don’t build systems that can handle everyone and everything because, as we like to think, what is the likelihood of a worst case scenario occurring.
As we proved in NYC on 9/11, the likelihood is low but when we need it, the importance of having systems that can handle emergencies is critical.
But alas, I digress ….. on to my originally intended subject.
My Security Report
As part of what I do as a strategy advisor and global technology architect, I provide services to some clients in the areas of assessing security vulnerabilities.
Specifically, how secure are various client’s IT infrastructures, what can be done to enhance their security and should a compromise occur, how quickly can the compromise be neutralized while minimizing the impact of the compromise?
The contents of my report, which will be distributed to specific organizations, shows a number of interesting slices of society that are vulnerable to attack.
The list includes, but is not limited to:
- Specific large-scale banks and credit card providers
- Specific health-care providers
- Specific municipal, state and provincial governments
- Specific airlines
- Specific energy generation / distribution groups
- Specific infrastructure organizations, including some that govern water distribution and public transit
- A specific Roman Catholic Archdiocese that has been rocked by pedophile priest prosecution in the past and is alleged to be hiding a list of known pedophile priests (unknown to the public) who are still active priests
- Other large corporations in manufacturing and retail
- Other entities whose “commercial” nature I am not allowed to mention here.
The vulnerabilities range in nature and scale but the bottom line is this.
There is still way too much vulnerability in our infrastructure, whether it be in our communication infrastructure, in the security and privacy of our data and in national security overall.
Why Is This Happening?
Some folks do the best they can with the limited funding they are given by their leadership - leadership that downplays the risks of not having a thorough solution or who don’t understand the impact to their organization, public or private, and the people they serve should a compromise occur.
Some organizations, governed by greed, pour their efforts into maximizing return, assuming that creating secure, redundant architecture is just a money-wasting venture that impacts their bottom line unnecessarily.
Some organizations create solutions so complex that obvious vulnerabilities slip by them and they watch in dismay as the seemingly ultimate in technology falls to simple attempts to compromise them.
Some organizations have a lack of knowledge about the threats they face and what is needed to neutralize the threat. I saw with amusement (and concern) last year when a national retailer placed a classified ad looking for someone to take charge of the design and implementation of a security solution for their entire corporation.
Why was I concerned? The minimum requirement for the position was a high school diploma. No other experience, education or security solution background was required. I guess they will learn on the job.
All of this being said, I still believe that ego and an excessive amount of hubris is responsible for most of the problems we face today.
Beliefs such as “nobody can defeat my security solution” or “the likelihood that compromise or disaster will hit us is minimal” are responsible for many of our compromises, both the ones that make the press and the ones that people on “the outside” never hear about.
How much of a problem is this?
A significant one.
While billions of dollars go into airline security and border control annually, I believe we face a much larger threat when it comes to the security and redundancy of our infrastructure then we do from someone taking a plane out of the sky or sneaking something across the border.
Much of the knowledge of how to compromise, penetrate, steal from or cause the failure of communications and IT infrastructure is available in the public domain. We face multiple threats ranging from the seemingly benign example of kids trying to hack into the local high school to get the answers to an upcoming exam up to agencies (including foreign governments) attempting, sometimes successfully, to penetrate our critical corporate, government and military computer systems.
The head of the National Security Agency recently said we need to pour more resources into beefing up our cyber security, causing many people to cry foul that Big Brother was using this as a guise to exert even more control over us.
While I am wary of how much insight government has into our private matters, this is one area where we must not underestimate the need to invest more into protecting our technology assets.
I once asked a well known US / UK military advisor-turned-journalist how he dealt with his knowledge of our vulnerabilities and this was his reply:
“I try not to stay sober”
Now that’s a sobering thought.
The people in my industry (information and communication technology) need to do a much better job at enhancing the security of the citizens of the world at the personal, corporate, national and global levels.
The people who provide funding and make the go / no-go decisions that enable / restrict the people in my industry need to be better informed about the importance and impact of their decisions in supporting such ventures.
And each of us, while varying in levels of technical savvy, must do our best to hold all of these organizations responsible and accountable to do the best they can.
And we’re a long way from doing the best we can.
Many organizations, private and public, have knowingly or unknowingly created ticking time bombs that will impact all of us.
Acknowledging this is not “sky is falling” pessimism.
Acknowledging it is the only way it gets fixed before we get punished for not taking appropriate action.
This is not pessimism.
It is reality.
Most of us say we would do anything to protect the security of our families, our businesses, our nations and the world.
It is time to prove it … with a sense of urgency and appropriate action commensurate with the threats that exist.
In service and servanthood,
Harry
Addendum – July 12, 2012
This news story (about compromised Yahoo accounts) that broke an hour after I wrote the blog is a reminder of our personal responsibility to ensure the integrity of our personal information on the web.
And then a bank went down …..
I noticed that a Canadian bank, more than 24 hours after the previously noted fire in Calgary, still does not have an online presence as a result of this outage in one building.
Here is what Alberta Treasury Branch customers (both personal and corporate accounts) receive if they go to access their online accounts for bill paying and such (emphasis shown is theirs):
We're Sorry...
The fire at the Shaw Court building in Calgary yesterday caused our banking system to go down.
Overnight we moved our system to a back up location. We are working to resume normal services, and anticipate that it will take us a bit of time.
Meanwhile, ABM, debit cards and MasterCards are available, and our branch staff will also be able to assist customers.
We are currently working to restore ATB Online banking and ATB.com as soon as possible, please check back here or on our ATB Financial Twitter account (@atbfinancial) for updates.
We can not access emails right now, so please call your local branch directly if you have questions or require assistance. Our Customer Care Centre associates (1-800-332-8383) are also available to provide more information. And remember, we never contact you via text or email to ask for your personal or banking information.
© 2011 ATB Financial | All Rights Reserved. TM Trademark of Alberta Treasury Branches. Unauthorized access is prohibited. Usage may be monitored. Please visit our website at www.atb.com
So an electrical fire in one building has derailed the online processing for an entire bank for an entire province.
Not comforting nor an unacceptable architecture, in my opinion.
Addendum – 5:40 PM MDT
I received the following note which I couldn’t help but share :-)
Dear Mr. Tucker,
My name is Dxxxxx and I live in xxxxxxx, Alberta. I am a customer of ATB and because I am on the road, I need to pay some bills today using their online system and of course I cannot. When RIM went down last October, I had to deal with a lot of angry customers and almost lost one because of my inability to respond quickly to them. With all the firefighting I had to do with my customers because of RIM, I got some free games from RIM for my trouble.
Since I need to explain to some people why I can’t pay my bills today, do you think that ATB will offer me some free games also?
I guess on days like this you need a sense of humour.
Cheers,
Dxxxxx
Dear Dxxxxx,
I hear that the new Angry Birds is pretty cool and might be appropriate. :-)
Thanks for the note!
Harry
Addendum: July 17, 2012
This little ditty was announced on July 17, 2012. Info about up to 2.4 million voters may be compromised: Elections Ontario. Preventable and sadly …. predictable. We can do better and must do better.
Addendum: August 7, 2012
Here’s how easy one can be compromised. If we are in the IT industry, we need to demand better of ourselves. If we are not in IT, we need to demand better from those who are.
No comments:
Post a Comment