Originally attributed to Mark Twain, who subsequently attributed it to Benjamin Disreali (although no evidence has been found that he actually said it), the above quote sums up how the use of statistics can blur the lines between powerful argument supporter and simple use of numbers to confuse and deceive.

When used properly, statistics in your risk management programme help support your recommendations, allow you to build effective business cases and even allow for a certain amount of self-analysis and performance reporting. When used badly, you run the risk of undermining the credibility of your entire risk management programme.

Consider the following two statements made by security awareness training companies:

“Reduce phishing click-throughs by 75%!”

KnowBe4 Internet Security Awareness Training

“…successfully trained over 7000 employees”  (Fox Entertainment)

TerraNova Security Awareness

In the first instance there is a bold claim in that click-through rates reduce by 75% which on the face of it sounds great. When reading in details there are some more impressive results but I can’t help thinking of the somewhat artificial nature of the test, i.e. “I have just taken anti phishing training and I am suddenly getting five phishing type emails, hmmmmmm”. Perhaps a more suitable test would have been to wait two months before sending the test emails? (The time between training and testing is unfortunately not specified however). There was also no mention of any feedback given in between each test. Security awareness training is such a hot topic however that I will leave that well alone for now!

In the second case the banner across the top of the website proudly announces how many people have been successfully trained; unfortunately it makes no mention of the other 5,500 employes who were not trained in Fox Entertainment (headcount checked at 12,500).

Now this is just standard sales patter and I certainly don’t mean to pick on these two companies specifically, but both statements illustrate the point perfectly. In both cases the products are probably very good in their own field, but when you “reverse” what it is they are saying they speak volumes. Some foods for instance are labelled as 90% fat-free, but in reality that means they contain 10% fat, and so here therefore there is still a 25% sized group of people who did click through and there are still 5,500 people who were not trained (and why not?). This is related to the fear, uncertainty and doubt that is often touted in the industry and can be used to scare and subsequently encourage people to buy products.

As risk professionals we need to take a more balanced, calmer route. We need to use statistics more carefully and responsibly, especially when what it is we are presenting makes its way into the core of the business, the leadership, the board, and ends up being used to make business decisions with serious implications. We can’t take sales statistics for instance on face value and use them to recommend a product or emphasise a point.

A Google search of “risk management statistics” produces over a billion results (in of itself a bad and useless statistic to present) so there is plenty of work out there on how to present your work, so I won’t be suggesting anything specific here. There are also plenty of other issues with statistics, for instance causation and inference which can be looked at in more detail at a later date.

i will however close on three key points I use whenever I am producing statistics for anything that comes out of the data gathered by a risk management programme:

1. “Reverse” the statistic (see above). If you don’t like what you see, don’t use it.
2. Be careful of your sample size; too small and the statistics are meaningless, too big and the resulting statistic you are focussing on is still a big and scary number even though you are potentially trying to emphasise quite the reverse.
3. Look at what you come up with cynically; is it a lie, or a damn lie?

And to underscore how statistics can mess with your head, statistically there are six Popes per square mile in the Vatican; go figure.