When It All Goes Pete Tong…

Murphy’s Law states:

“If something can go wrong, it will go wrong”

Many CISOs will also state:

“it is not a case of if you have been breached, but rather that you have, you just don’t know it yet”

Depressing as both statements sound by themselves, put them together, and you enter into a worldview of doom and gloom from which it is hard to crawl. It doesn’t matter what you do; there will always be a breach and multiple mistakes in your team. These factors create a perfect storm for finding a new job relatively quickly.

But there is hope that when you start a new role or join a new company, there is one thing that needs to be in place before anything else; the Incident Management Plan*. In all but the most security mature organisations, any improvements put into place by you will take months and years to bear fruit, during which time a disaster can strike without notice (the unknown unknowns hitting at an unknown time, if you will.) So making sure you have a plan to fall back on at a moment’s notice gives you space and time to respond appropriately while still being able to focus on the more fundamental changes you have in mind for the organisation.

But what to put into these plans? There are a few key points that should always be adhered to whenever writing a response plan;

Keep it Simple

Human beings are emotional sacks of meat and adrenalin when things go wrong. They can simultaneously be forgetful, angry, scared, sad, and even stupid. Therefore your plans, and by association, your writing and grammar, need to be as simple as possible. It’s not an easy task and will require many edits, reviews and rewrites, but simplicity is your friend during a confusing and rapidly changing situation. 

Keep it Flexible

Extending the first point, you also cannot create a prescriptive document. If you define every action based on a specific input, your plan will fail when that particular input isn’t happening. The plan needs to work on the principles of what must occur during an incident rather than the specifics of what needs to be done. It is useful, for instance, to focus on roles and responsibilities rather than activities; in this way, someone is accountable for “public communications”; how they achieve that is up to them, but the plan does not define it.

Know What’s Important

This is another way of saying, “Understand your critical services”. These services could be technology-based, process focussed or even role/person-specific. During an incident, the immediate focus is to get the bare minimum of services/capabilities/business operating again as quickly and safely as possible. Going back to Business As Usual is for later on. You need to know what the bare minimum is to achieve it.

The ISO 22301:2019 – Security & Resilience – Business continuity management systems standard is a great place to start to understand the mechanics of this element in more detail (and great for this topic as a whole).

Collaborate While Creating

It never ceases to amaze me how often plans like this get created in isolation across companies, divisions and departments. What that means, more often than not, is a competition for resources because they all assume they will have exclusive access to the resources required to see them through a crisis just because they have a plan.

Ideally, there should be a single master plan for the organisation that allows each discrete business area to manage their plans (essential in larger organisations). Then, all of these plans and their requirements are fed back into the overarching strategy to carry out capacity planning and coordination more effectively and efficiently.

Multi-channel Sharing and Education

This is the one time I will permit using a few trees to print out your plans. Electronic documents are still valuable and should be saved in different formats and on other devices and platforms (for redundancy, obvs). Having paper copies of the entire document, in addition to aide memoirs, laminated “cheat sheets”, credit card numbers and any other creative approaches to ensuring the needed information is always available. Remember, this is a time of crisis; your laptop may be burning down with your building, and your phone may be out of battery with nowhere to charge. Base your communication and distribution methods on the assumption of Murphy’s Law above.

Test the Plan, Learn and Review

You must test the plan as much as possible, especially when creating it. If you feel brave enough, you can have a tabletop walkthrough or pull the plug on a data centre. Some third-party services allow you to test your plan in a virtual space using specialised communications tools that are even more realistic. Whatever the case, every time you check it, review it and feed the findings back into the plan. Even a slight improvement could make all the difference.

Test the Plan Again

Did I mention testing? Even if you have a real-life crisis, use the learnings and feedback to improve the plan again. Every opportunity to stress the crisis plan, people and procedures must happen.

Test it Again

It must be tested, whatever happens, at least once a year, and reviewed yearly. You will be surprised at how much your business changes over a year; a process may be updated, people and roles change, and telephone numbers and email addresses frequently updated. If your plan doesn’t reflect even these simple changes, it is more likely to fail.

The Holy Trinity Mantra

Finally, if in doubt, remember these three elements of your plan. I like to ensure they are seen through in this order, but you may feel differently according to your business and how it operates. (If people don’t list as number one on your list, take a long, hard look at yourself.) Nonetheless, The Trinity remains the same.

  1. Focus on People – without your people, you have no business to speak of, recovered or otherwise.
  2. Focus on Facilities – even with just a pen, paper, telephone, and somewhere to work, your people can work miracles in keeping the business afloat. Keep them safe, secure and happy.
  3. Focus on Technology – get the systems running to take the strain off the people. This may have taken days or weeks, depending on the incident. Ensure your critical systems are running first, and that includes payroll. Paid people pull together in a crisis. Unpaid people don’t.

Hopefully, you will never have to use the plan, but if you do, feeling prepared for anything is a powerful way to ensure your best work on everything else on your list. Knowing that you have it ready to go is like remembering to take your umbrella with you when you leave the house. Because you have it, it isn’t going to rain; mildly annoying but so much better than getting caught in a monsoon in your best work attire.

*Also known as the Crisis Management Plan, Business Continuity Plan, When It Hits The Fan Plan, or any other variable that works for you, your company, and your business culture.

Links to other interesting stuff on the web (affiliate links)

How to Upskill Your Cybersecurity Team

The AWS Security Cheat Sheet

Think Before You Share The Link


Agile? Or FrAgile?

(I found this piece deep in the vaults at (TL)2 Towers, so I figured I would break my non-blogging streak.)

Sensitive client code has been discovered on a GitHub repository, and it looks like one of our developers put it there. The client is upset, and their Chief Information Security Officer (CISO) wants to meet with you in New York to discuss what happened.”

Not the best email start to a Tuesday morning, I grant you, but I did at least have two things in my favour; six days before the meeting with the client and an excellent Cyber Incident Response Team (CIRT) headed up by a very talented and focussed individual based out of Miami. So I felt confident we could get to the bottom of the incident.

Fast forward to the following Monday afternoon (don’t worry, this isn’t a blog about cyber forensic investigations), and I am walking out to the cab back to the airport with a thankful, happy client and the promise of more billable work in the form of security testing.

It sounds like the beginning and end of two different stories, but this is a real example from back in 2016 during my tenure as a CISO. What initially felt like a nightmare scenario was quickly turned around, and three things became apparent very quickly:

  1. Proper incident management is crucial because it is not a case of “if we have a data breach” nowadays but “when we have a data breach”.
  2. Agile working methods are a double-edged sword; we get work done quickly and effectively, but if we are not careful, boy, can it bite us in the ass.
  3. Focusing on the client, their perceptions, need for honesty, demands, and wishes are paramount.

I want to focus on 2 and 3 from above because let’s face it, incident management is often carried out by the cool-looking security folks who swoop in when it all goes wrong.

Is it Agile or FrAgile?

Agile working methodologies are, I am told, wonderful ways to get work done quickly and effectively. Flexible working, the heavy reliance on technologies, still often in early release stages and updated daily. You can reuse and share code through various repositories, test things out with the rallying cry of “Fail fast! Fail often!”, pull out your credit card and spin up servers on the other side of the world by the time the kettle has boiled or the coffee machine dripped its last drop.

The problem with this is that if we are not careful, the environment becomes the wild west, with code flying around various platforms containing hardcoded admin credentials and schematics shared openly as you struggle to squash that last bug with the help of the internet. (I am having palpitations just thinking about it.)

Sticking code up onto GitHub doesn’t make you agile; it makes you fragile. You must prepare and plan, work as a cohesive unit with your team and have established (and secure) ways of working with these new and funky tools that are not only effective but also in the best interests of the client.

The client is King AND Queen.

I recall another incident where a client contacted us to say they had found their code and credentials on a GitHub-type site. They were (unsurprisingly) very upset and demanded our response. Upon investigation, it turned out the data in question was uploaded there some two years earlier, was out of date, and the credentials were changed long ago anyway. It was severe, but we were off the hook with confirmation that third parties downloaded none of the data!

Famous last words. The client was livid.

The incident escalated to their board, and we were bought to task for the security lapse in no uncertain terms. However, we had forgotten one crucial thing in all of this; it wasn’t our data to lose in the first place. 

There is a saying in information security circles; “my risk model is not your risk model”. It means that what risks affect you are not the same as those that affect me. Precisely the mistake we made with this client. It didn’t matter that nothing had happened; it didn’t matter that the data and credentials were stale; we were sloppy and slapdash with their intellectual property. We had broken their trust.

In the middle of the high-pressure project environment, losing sight of what the client holds dear versus what you are trying to achieve in the here and now is easy.

Making the Leap

Do you need to address your agile working practises and understanding of your client’s motivations and risks? Here are two data points to consider:

  1. A constant increase in the inclusion of unlimited liability clauses in your contracts. Right or wrong, they aren’t going away, and as you access more of a client’s sensitive and confidential data to “make digital work” for them, it will continue to do so.
  2. When it happens, the cost of a data breach is increasing. According to the Ponemon Institute/IBM Security report “2019 Cost of a Data Breach Report – A comprehensive analysis of data breaches reported in 2018,” the average cost of data breaches has increased by 12%, a whopping $3.92M in the healthcare industry alone. You won’t get much change out of that for a satisfactory bonus pool.

When you leave your house in the morning, you lock your door, maybe enable an alarm system. When you leave your car, park it in a reputable garage and lock the door, immobilising the engine and setting the alarm.

Then you go to work and copy someone else’s confidential data and credentials to an open file share. Can you see the paradox?

Taking that leap in maintaining client confidence, assuring them of intelligent decision-making, agile (not fragile) working practises and mental rigour to maintaining their crown jewels is challenging. But it is essential.

Why did that first client give me more business after a breach? Demonstrating that we were in control of our processes and ensuring he and his risks were obviously the centre of our world helped: that and a high degree of transparency. But that initial breach was too high a price for extra business and a warm handshake.