Lies, Damned Lies, and Statistics

1 Comment

Pac-ManOriginally attributed to Mark Twain, who subsequently attributed it to Benjamin Disreali (although no evidence has been found that he actually said it), the above quote sums up how the use of statistics can blur the lines between powerful argument supporter and simple use of numbers to confuse and deceive.

When used properly, statistics in your risk management programme help support your recommendations, allow you to build effective business cases and even allow for a certain amount of self-analysis and performance reporting. When used badly, you run the risk of undermining the credibility of your entire risk management programme.

Consider the following two statements made by security awareness training companies:

“Reduce phishing click-throughs by 75%!”

KnowBe4 Internet Security Awareness Training

“…successfully trained over 7000 employees”  (Fox Entertainment)

TerraNova Security Awareness

In the first instance there is a bold claim in that click-through rates reduce by 75% which on the face of it sounds great. When reading in details there are some more impressive results but I can’t help thinking of the somewhat artificial nature of the test, i.e. “I have just taken anti phishing training and I am suddenly getting five phishing type emails, hmmmmmm”. Perhaps a more suitable test would have been to wait two months before sending the test emails? (The time between training and testing is unfortunately not specified however). There was also no mention of any feedback given in between each test. Security awareness training is such a hot topic however that I will leave that well alone for now!

In the second case the banner across the top of the website proudly announces how many people have been successfully trained; unfortunately it makes no mention of the other 5,500 employes who were not trained in Fox Entertainment (headcount checked at 12,500).

Now this is just standard sales patter and I certainly don’t mean to pick on these two companies specifically, but both statements illustrate the point perfectly. In both cases the products are probably very good in their own field, but when you “reverse” what it is they are saying they speak volumes. Some foods for instance are labelled as 90% fat-free, but in reality that means they contain 10% fat, and so here therefore there is still a 25% sized group of people who did click through and there are still 5,500 people who were not trained (and why not?). This is related to the fear, uncertainty and doubt that is often touted in the industry and can be used to scare and subsequently encourage people to buy products.

As risk professionals we need to take a more balanced, calmer route. We need to use statistics more carefully and responsibly, especially when what it is we are presenting makes its way into the core of the business, the leadership, the board, and ends up being used to make business decisions with serious implications. We can’t take sales statistics for instance on face value and use them to recommend a product or emphasise a point.

A Google search of “risk management statistics” produces over a billion results (in of itself a bad and useless statistic to present) so there is plenty of work out there on how to present your work, so I won’t be suggesting anything specific here. There are also plenty of other issues with statistics, for instance causation and inference which can be looked at in more detail at a later date.

i will however close on three key points I use whenever I am producing statistics for anything that comes out of the data gathered by a risk management programme:

  1. “Reverse” the statistic (see above). If you don’t like what you see, don’t use it.
  2. Be careful of your sample size; too small and the statistics are meaningless, too big and the resulting statistic you are focussing on is still a big and scary number even though you are potentially trying to emphasise quite the reverse.
  3. Look at what you come up with cynically; is it a lie, or a damn lie?

And to underscore how statistics can mess with your head, statistically there are six Popes per square mile in the Vatican; go figure.

Getting Your Hands Dirty

1 Comment

dirty-handsIn my last post I referred to ensuring that your risk management programme is producing the quality of output to ensure the business information it feeds into is of the highest quality; maintaining the integrity of your programme.

If there is one thing that can be done to improve the integrity of your risk assessments it is simply to get your hands dirty during them. I have had a number of conversations with people who have been on the receiving end of an assessment where the assessor simply sits at the table and asks for evidence in the form of documentation, verbal responses or even just PowerPoint presentations to confirm the effectiveness of the information security programme in question. Personally I have sat in a conference room for one or two days at a time and only left the room for a short thirty minute ‘walkabout’. Quite how the assessor felt they were getting a representative view of what we were doing was beyond me.

There are a number of problems with this hands off approach:

The ability of those being assessed to ‘play’ the assessor increases with their reluctance to physically move around the organisation. Pre-prepared evidences (the so called “audit box” as was once described to me) can be made available, the organisations SME’s can be wheeled in to ensure the right things are said at the right time and the people who never seem able to say the right thing at the right time (and every organisation has them!) can be told to work in a different building that day.

Secondly, unless the assessor is actually looking at the evidence first hand, even down to rifling through the physical pieces of paper or reviewing server logs, there is absolutely no way any kind of discrepancy will ever be found. Of course this is a sampling exercise, and of course there is no way every single piece of evidence, paper or electronic can be reviewed, but some kind of benefit can be gleaned from going though them. Quite apart form anything else it gives the clear impression that “no stone is unturned” during the assessment process. I have come up with a surprising number of findings from simply taking a few minutes to look through large piles of paper records.

Finally, and perhaps slightly more esoterically, the action of a walkabout can give a very good “feel” for a place. If the presence of the auditor brings hurried and furtive glances everywhere they go, it may give the indication of nervousness or unwillingness regarding the assessment (or of course just a healthy distrust of strangers). If there are rows of empty desks that are obviously normally in use but seem to be vacated for the day this may give the indication that special plans have been laid on for the assessment (or that the sales team are in a meeting). This last point is not so clear cut as the other two, and should only be used as an indicator of what is already coming out of your assessment, but it is a useful one nonetheless.

I have a colleague who every time he enters a “serious” meeting, he undoes his cufflinks and rolls up his cuffs a couple of times; this is his way of mentally preparing for the challenge ahead by literally rolling up his sleeves. When it comes to risk assessments that is exactly what you need to do, and then prepare yourself to get your hands dirty.

Certified Information Security [Insert Qualification Here] Post Nominals

6 Comments

exam_paperThe good news for me this last week was that I eventually took the CISSP exam and passed. I was obviously pleased and relieved, and I am currently going through the endorsement process. Despite the drubbing that the CISSP as a certification over the last year or so I have to admit that on the whole I was impressed with the depth and breadth of the subjects covered.

Of course the caveat to this is that I think this on the basis that the CISSP is an information security certification, not an IT security certification. There is plenty of content about fire extinguishers, foot candle illuminations of parking areas or even the legal constraints of transferring information outside of the EEA, all of which are important to my mind when taking into account the broader concepts of information security (especially when considering the Confidentiality, Integrity & Availability triangle). Much of the criticism I observed was around the relevance of topics like my previous three examples to IT security, to which I reply “It’s not”. There are sections that focus on these areas, but they quite rightfully don’t dominate the subject matter.

That said, there were areas that I thought were woefully under represented in the reference material that I used, for instance I disagreed with the definition of ISO 27001 versus ISO27002, their definition of an adequate security measure for WEP (hiding the SSID… really?) and other small points. I was however revising against the 2nd edition CBK which has now been updated to the third edition, so perhaps there have been updates in some of these areas.

The other area I struggled with was the relevance of some of the information required for the exam. The level of details required in areas like security architecture for models that actually aren’t in use any more or encryption techniques or even the finalists in the competition to decide what encryption method to use in what ultimately became AES… over twenty years ago! None of this is going to be useful to me in may day to day job at all.

But again, overall it really made me think about my “craft” and I have found it beneficial. There was an element of me taking this exam as a box ticking exercise given my current role, but this was mainly because I came to infosec quite late in my career and there were questions being asked as to why I didn’t have this qualification. It made sense to get it done now and out of the way as it were, and add to my CISM and CGEIT (and MBCS CITP… at this rate my business cards are going to have to be very wide.)

The big question for me now though is what’s next? CRISC or the CIPP/E? Risk or Privacy?

Don’t Put Baby in the Corner

2 Comments

5670_fullLast week I had the opportunity to do both a presentation at the BCS IRMA Specialist Group as well as take part in a drastically reduced panel with Javvad Malik (and only Javvad!) at the InfoSec Europe 2013 Press conference.

Firstly I want to recount the panel for the press conference. After some last minute drop outs (one of which I was replacing anyway!) there was just Javvad and me available to do it less than 24 hours before we were due to start. In his own inimitable style he proposed a double act Parkinson style to talk about the challenges faced by a CISO in the Enterprise. I was somewhat unconvinced by this but true to his word, the whole session went extremely well and was thoroughly enjoyable. Afterwards Javvad was told  by some of the journalists that the session was a great way to end the two days with the non vendor focus of the session, and the humour that Javvad and I of course used!

One of the main topics we discussed was that of the position of the CISO within the organisation and the influence that this subsequently brings. Ultimately my position is clear on this, that the CISO needs to be as high in the organisation, and as independent of vertical alignment as possible. What I mean by this is that if the CISO is on the board (or executive leadership team as appropriate) and does not report into the CFO, COO, CIO or any other C level executive there is a dramatically increased chance of security being a successfully managed activity in the enterprise. It ensures full representation of the security function at the most senior levels, free of conflicts of interest and able to vie for budget and attention on an equal footing with the rest of the business units.

I will caveat this however. If there is no security function in place or it is in its nascent stages, or the business itself is smaller, it makes absolute sense to have the security function perhaps initially reporting into the CIO; in all likelihood the staff building the team will come from IT anyway. However, as the team grows it needs to evolve its leadership and position in the organisation, perhaps moving away from the IT function, to the COO and then ultimately to the board.

This transition is something that I have never seen planned in advance, and this is probably one of the fundamental reasons why the CISO and security function is constantly under represented in the modern enterprise as it struggles to gain independence. This will always result in poor awareness and training, lack of budget and lack of true top down security adoption as they compete for ever diminishing resources from lower down in the organisation.

One fairly unique place I have seen the security function is reporting into the General Counsel/Legal function. This I have seen work well as it is the GC that is traditionally responsible for the tracking and management of risks for the enterprise, and frequently has the ear of the CEO. I rarely see a conflict of interest with the security function either. This is not common though, and is likely to only be likely in the larger organisations that have a formal role of GC.

Bottom line, if the newly appointed CISO (i.e. a senior level position for a mature security team) reports into the CIO, then in reality, security is not going to function effectively in that organisation.

And finally (although not in chronological order), the BCS. It was the final presentation of “An Anatomy of a Risk Assessment” and it was (as far as I can tell) well received. Unfortunately the weather and lack of sandwiches post the even meant there was little time to mingle afterwards, but I have since received a number of favourable comments and of course connection requests on LinkedIn which is always heartening. I did however  feel I didn’t answer one of the questions at the end, about India, particularly well, and may have come across as a little disingenuous when nothing could be further from the truth. I hope my friends and colleagues from india will forgive me if they make it to the end of the video when I get hold of a copy (and post it here). As an aside I found an extremely flattering write up of the very first time I presented this in January last year. To the author at Acumin, thank you! http://acumin.wordpress.com/2012/02/

All in all, a very enjoyable and engaging kick off to 2013.

 

2012 in review

2 Comments

Blogging can be seen as a very inwardly focussed activity, it is all about me, me, me. I have always tried to maintain a fairly balanced online presence, keeping it professional if a little informal, striving to only blog, or tweet quality rather than quantity. On the whole this has worked for me. The downside to this though has been a slow increase in my online presence (or brand, whatever term works for you) and therefore Twitter followers and blog visits. For example one of the primary reasons for blogging this year has been to “practise” writing about my profession in a way that I don’t get in my place of work and not to gain fans and followers (although that would be a nice by product!).

That said, the automated report that WordPress sends out prompted me to consider what I have achieved over the last year and realise how positive I feel about my online presence. To put it into context here are some very quick (and totally unscientific) stats: In 2011 (when I joined Twitter) I had four blog posts in a self managed blog page, attended one conference (RSA), had less than ten followers and tweeted maybe ten times. I had publicly spoken once, for two minutes, at the Christmas RANT forum. In short, I had no idea what the community had to offer or indeed how to engage with it.

It was at the aforementioned RSA conference that two things happened; firstly I realised that 80% of the presentations I watched were of a quality that I felt I could reproduce. Secondly I met a few folks on the last night that in all honestly changed my perception of the industry and how I could participate in it, namely Brian Honan (@BrianHonan), Kai Roer (@kairoer), Alex Hutton (@alexhutton) and Aaron Barr (@aaronbarr) amongst others. They showed me (unknowingly) how they worked with the community, staying in touch through Twitter, communicating through blogs, articles, podcasts etc.. I have since stayed in touch with Brian and Kai, both of whom I respect greatly and would like to thank for their openness and friendliness to me back in October 2011!

Fast forward to today and my stats are a little better: 26 blogs posts, nearly 500 tweets (not all of them are rubbish either!), 111 followers, six public speaking engagements including one panel and the RSA conference itself, a video blog with the almighty Javvad Malik (@j4vv4d) and contributed to two articles (for Tripwire and (In)Secure magazine). I attended in one capacity or another nearly twenty events/conferences/forums. The best part is that these stats don’t do the experience itself any justice. I have made friends and met many people for whom I have the most deep respect for and who I genuinely like and enjoy their company. I have submitted a joint CFP for a conference with one of them, and hope to continue my relationship with Acumin and the RANT forum (@Acumin & @GemmaPats) who gave me my first big break in public speaking (thank you!). In short, 2012 has been awesome as both a learning experience and a source of fun and enjoyment as regards my chosen profession. The blog stats below are of course modest by most peoples standards, but they are interesting and encouraging to me nonetheless in the context of the above.

I tweeted over the Christmas holidays that my word for 2013 is “growth” both professionally and personally; while I hope that my 2013 “stats” will continue to “grow” more importantly I hope that my new friendships and opportunities to learn in this odd, frustrating, challenging yet ultimately rewarding industry and community continue.

And before you ask, yes, New Year, New Theme for the blog; I’ve grown out of my dark goth and emo phase and now it is time for some colour and class!

Here’s an excerpt:

The new Boeing 787 Dreamliner can carry about 250 passengers. This blog was viewed about 1,200 times in 2012. If it were a Dreamliner, it would take about 5 trips to carry that many people.

Click here to see the complete report.