Metrics: Be Clever, Tread Carefully.


In any professional project it is necessary to outline the way in which we assess both failure and success. Selecting a series of metrics provides an easily quantifiable method to carry out this assessment. The advantages of this are obvious, a solid metric reduces the complexity of the task: set a bar; hit it or don’t and react as decided.

The problems that arise come in two forms: vanity metrics and descriptive misattribution.

The first of these is fairly well documented on the web. A vanity metric is a measure which serves as little more than a demonstration that something is being measured, often to impress customers or boards, however they fail to portray any meaningful data or lead to action. A typical business example would be a start-up quoting “user base growth” as a percentage without context – 400% sounds great – is that 50 to 250? Perhaps just 2 to 8. An investor could not make a decision from the percentage alone: it is a vanity metric.

In InfoSec we’ve all heard the horror stories of the yearly information security staff training presentation. If a company were to take the number of names on their sign-in sheet against the number of staff in the company as a valuable metric of their information security it would be simply farcical. Neither could one go to the board and inform them of any real change in the security readiness of the company: the metric does not allow you to draw meaningful conclusions.

The second problem mentioned was ‘descriptive misattribution’ – when your metric doesn’t mean what you think it means. This is less often a case of misleading sales tactics and more often an oversimplification of the subject matter when trying to perform a comprehensive analysis of a subject more subtle than anticipated.

Firstly, consider the situation in which you have just started in your senior information security role. Your company has a sound technical set-up but, as is all too common, lacks a proper approach to the human aspect of information security. You are tasked with bringing the company into the position where you can robustly confront the business threats of the 21st century. With this in mind you focus on information security awareness making sure staff are aware of good habits and the threats out there. You find a suitable educational platform – a web search will always reveal more than a few. After a few months you can safely say information security awareness has increased, tutorials have been watched, a few tests done and so on. You have some well measured metrics to hand.

Job done? Sadly not. This is something talked about more extensively by HutSix previously (see The key point? Awareness is not behavioural change. These measurements on awareness are not wrong; they are not even misleading. They have simply been misattributed as an indication of actual improvements to security. The point is subtle because whilst awareness is an aspect of good security, it is inappropriate to use it as your only metric regarding the human side of information security, since you cannot attest to your employees behaviour only their education.

Clearly any professional must tread carefully when considering metrics, and this is particularly true in the information security world. On the human front where quantifiables can seem tricky, we must stay focused on what it is we really want. We must be clear about how far our metrics go to representing the change we need.

If this is an area you are interested in come talk to us at –


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s