Home / News / The Problem with Risk Scores and a Risk Matrix
The Problem with Risk Scores and a Risk Matrix
Increasingly year-on-year health and safety management is incorporating lots of statistics that are recorded, monitored and reported on a regular basis using health and safety software. This software improves management and ensures things are not forgotten, particularly if like Smartlog, risk assessments automatically assign tasks to individuals and remind them via email so that they take corrective action. On the other hand it can be considered that too much focus on statistics, in particular risk scores in a risk matrix can hide certain hazards by focusing on other hazards due to their numeric values. Moreover whilst statistics and reporting are great to measure progress and identify frequently occurring accidents, it is evident that the majority of time in health and safety management should be on prevention by design, planning and training.
Whilst risk assessments that feature risk scores can list a multitude of different levels of hazards for identification and review purposes, they create confusion over what is the most likely to happen and what is the highest severity of harm, as peoples’ perceptions of risk and severity differ. Risk assessment categorisation of risk focusing on likelihood and injury severity together under the term priority and then setting normal or high priority is a better system as issues are fixed quicker based on priority and not based on the subjective aspect of severity which is circumstantial in every case. For example tripping on the stairs because of a worn surface can be unlikely and can often only result in minor injury accidents however it can also result in death; this highlights the potential issue with risk scores in a risk matrix which might rate this with a low numerical value for both likelihood and severity.
An Outdated System?
In health & safety management risk scores and the risk matrix structure could be considered as a 20th century management tool originating from paper based risk assessments as a way of highlighting risk within the lines and columns on a paper risk assessment to decide which should be considered first. With 21st century cloud based health and safety software like Smartlog, risks are automatically moved to the top a priority action list and removed when they are resolved. Today’s quick priority identification and resolution via software means it is questionable what purpose risk scores achieve in today’s era of health and safety management with quick software that can facilitate quick corrective action if it’s designed that way.
The concept of risk and risk assessments has a long history. More than 2400 years ago the Athenians offered their capacity of assessing risk before making decisions and over the years it has become central to keeping people and operations safe. The introduction of The Health and Safety at Work etc. Act 1974 and The Management of Health and Safety at Work Regulations 1999 introduced documentation and management rules however there is no mention in these two pieces of legislation about the use of risk scores or a risk matrix in risk assessments.
Analysis – HSE
In fact risk scores and risk matrices are not mentioned in any health and safety legislation. Risk score matrices are mentioned on the HSE website but it’s only written that a risk matrix can be used, not that they must be used. In the Risk Management FAQs part of the HSE website here: www.hse.gov.uk/risk/faq.htm they have written:
‘Most businesses will not need to use risk matrices. However, they can be used to help you work out the level of risk associated with a particular issue. They do this by categorising the likelihood of harm and the potential severity of the harm. This is then plotted in a matrix (please see below for an example). The risk level determines which risks should be tackled first.
Using a matrix can be helpful for prioritising your actions to control a risk. It is suitable for many assessments but in particular to more complex situations. However, it does require expertise and experience to judge the likelihood of harm accurately. Getting this wrong could result in applying unnecessary control measures or failing to take important ones.’(Accessed 19/11/2018)
The HSE explicitly mention here that getting risk scores wrong can result in failing to take important control measures. This is the issue and danger with using risk scores.
The HSE do not state that a score or colour rating of risk about the likelihood of risk occurring or severity is needed or that score and colour needs to be recorded or have a matrix. There is no mention of it on HSE’s website, in risk assessment examples from the HSE or in health and safety legislation. Moreover there is no mention of it in their risk assessment guide, HSE document indg163. www.hse.gov.uk/pubns/indg163.pdf
The image of the risk matrix shown in the HSE’s mention of risk matrices on their website shows the complexity and subjectivity issue of deciding on a combined numeric value and colour for a risk. This is an example of a 3×3 grid matrix; when a grid gets larger e.g. 10×10 there can be the possibility of even more confusion, differences in opinion and lack of consistency in scoring amongst people.
An article in IOSH magazine from an independent health and safety consultant also mentions that ‘in most cases it is not possible to quantify either the likelihood or the severity with such accuracy, we make relative judgements’ furthermore she writes that:
‘To avoid confusion, ditch the numbers and replace “mostly harmful”, “unlikely” and so on with descriptions that match your organisation’s risk profile, and simply use the coloured areas to categorise the risk bands.’
Clearly it is also questionable as to what purpose the colours even serve if high and normal priority settings are used with health & safety management software that facilitates speedy corrective action in a priority list format. In summary she also concludes her article by writing ‘Poorly understood severity and likelihood categories and arbitrary risk bands will lead us to draw the wrong conclusions.’
Similarly the author of the article also presented a presentation named ‘What is significant risk?’ at the 11 February 2014 IOSH London Metropolitan Branch Meeting. This presentation sought to highlight the issues and over complexity of quantitative risk assessment (QRA) risk matrices based on risk scores amongst IOSH members.
Moreover the same author mentioned in another IOSH magazine article that ‘In most cases, we don’t have enough reliable data for QRA. Rather than sticking numbers on to categories and mistakenly calling assessments quantitative or semi-quantitative, we should be proud of producing high-quality qualitative risk assessments.’ Risk assessment that focuses on high and low priority allocation of corrective action and descriptive text from hazard assessment rather than numbers is qualitative rather than quantitative.
Regarding risk assessment methodology it is Safesmart’s view that you should choose a qualitative risk assessment so not to come across the issues of risk scores in the prioritisation of corrective action tasks based on over complexity via numbers in a risk matrix structure.
As you can see in the HSE’s risk assessment template there is no mention of risk scores or a risk matrix:
The HSE’s sample template can be found under resources here: http://www.hse.gov.uk/risk/
INDG163 is the HSE’s legal guidance for completing risk assessments.
In document INDG163, The HSE state that in a risk assessment you need to:
– Identify the hazards
– Decide who might be harmed and how
– Evaluate the risks and decide on precautions
– Record your significant findings
– Review your risk assessment and update if necessary.
Regarding HSE legal requirements for risk assessment it is important to remember that there is no required format for risk assessment so long as within the format you can achieve these aspects mentioned in INDG163.
Evaluate the risks and decide on precautions:
In this section on p2 of HSE document INDG163, it does not specify that a risk likelihood, severity score, colour rating or risk matrix needs to be recorded. It just mentions that you have to decide ‘how likely it is that harm will occur and what to do about it.’
On p2 of HSE document INDG163 in the ‘Evaluate the risks’ information section it states:
‘Having identified the hazards, you then have to decide how likely it is that harm will occur, ie the level of risk and what to do about it. Risk is a part of everyday life and you are not expected to eliminate all risks. What you must do is make sure you know about the main risks and the things you need to do to manage them responsibly.’
Record your significant findings
When recording a risk assessment, the HSE state on p3 of INDG163 that ‘any record produced should be simple and focused on controls’. They state you need to ‘record of your significant findings – the hazards, how people might be harmed by them and what you have in place to control the risks.’
The passage reads:
‘Record your significant findings, Make a record of your significant findings – the hazards, how people might be harmed by them and what you have in place to control the risks. Any record produced should be simple and focused on controls.’
Putting hazards in order
On p4 of HSE document INDG163 it’s stated that the hazards identified need to be put in order of importance to address the most serious risks first.
The HSE state on p4 of INDG163 that:
‘If your risk assessment identifies a number of hazards, you need to put them in order of importance and address the most serious risks first.’
By stating simply whether a hazard is normal or high priority it can be clearly and simply differentiated which hazards are high priority.
In addition to the HSE’s comments about the danger of using risk scores and a risk matrix the ‘Health and Safety Laboratory’ (HSL) the research arm of the HSE have also conducted research on behalf of the HSE about ‘Good practice and pitfalls in risk assessment’ in the 2003 research report 151 (RR151).
RR151 mentions the following pitfalls that seemingly could be the case with quantitative numeric focused risk assessment with risk scores and risk matrices:
• ‘Carrying out a detailed quantified risk assessment without first considering whether any relevant good practice was applicable, or when relevant good practice exists’
• ‘Making decisions on the basis of individual risk estimates when societal risk is the appropriate measure’
• ‘Inappropriate use of risk criteria’
Moreover on p32 under the title of individual risk measures RR151 mentions a problem with numerical risk scores in a matrix:
‘Each risk box in the matrix represents the combination of a particular level of likelihood and consequence, and can be assigned either a numerical or descriptive risk value (the risk estimate). If numerical consequence and likelihood category indicators are used, it is common to estimate the risk values as the product of the likelihood and consequence values, as a convenient way of ranking the risks. Care should be taken if such an approach is adopted as, for example, hazards of low severity and high likelihood will receive the same risk value as hazards with high severity and low likelihood. Although the risk values may be the same, the response to these different hazards in terms of priority for correction may be very different (St John Holt, 1999), and care therefore needs to be taken to ensure the method for estimating risk results in values or categories that can be interpreted appropriately.’
Furthermore where there is mention of quantitative numeric risk assessment on p14 in RR151 it is not mentioned as a legal requirement it is just written that:
‘Where the hazards presented by the undertaking are numerous and complex, and may involve novel processes, for example in the case of large chemical process plants or nuclear installations, detailed and sophisticated risk assessments will be needed, and it is appropriate to carry out a detailed quantitative risk assessment in addition to the simple qualitative assessment. Quantitative risk assessment (QRA) involves obtaining a numerical estimate of the risk from a quantitative consideration of event probabilities and consequences (in the nuclear industry the term ‘probabilistic safety analysis’ is used in place of QRA).’
Regarding RR151 it is important to remember that ‘This report and the work it describes were funded by the Health and Safety Executive (HSE). Its contents, including any opinions and/or conclusions expressed, are those of the authors alone and do not necessarily reflect HSE policy.’
But ‘it is hoped that this report will provide useful guidance for Inspectors involved in the assessment of industry risk assessments on the appropriateness of the adopted approaches, and also to practitioners in industry involved in the process of carrying out workplace risk assessments of how to avoid common pitfalls.’
www.hse.gov.uk/research/rrpdf/rr151.pdf
There is a lot of research online that criticises the use of a risk matrix and we encourage you to research and read this to understand the issues.
Examples of academic texts include Louis Anthony Tony Cox of the University of Colorado Department of Biostatistics and Informatics who wrote a journal article in 2008 for Risk Analysis the official publication of the Society for Risk Analysis in which risk matrix limitations are listed – the abstract of this journal article called ‘What’s Wrong with Risk Matrices?’ can be viewed here
As well as this, a well written piece called ‘The Risk of Using Risk Matrices’ was published in September 2013 for the Society of Petroleum Engineers, SPE Economics and Management Journal. It was written by Philip Thomas, SPE, and Reidar B. Bratvold, SPE, University of Stavanger; and J. Eric Bickel, SPE, University of Texas at Austin. They have written in this article about risk matrices (RMs) saying that:
‘Despite these claimed advantages, we are not aware of any published scientific studies demonstrating that RMs improve risk-management decisions. However, several studies indicate the opposite: that RMs are conceptually and fundamentally flawed.’
Moreover this journal article is concluded with the following:
‘In this paper, we have illustrated and discussed inherent flaws in RMs and their potential impact on risk prioritization and mitigation. Inherent dangers such as risk-acceptance inconsistency, range compression, centering bias, and category-definition bias were introduced and discussed by Cox et al. (2005), Cox (2008), Hubbard (2009), and Smith et al. (2009). We have also addressed several previously undocumented RM flaws: ranking reversal, instability resulting from categorization differences, and the LF. These flaws cannot be corrected and are inherent to the design and use of RMs. The ranking produced by RMs was shown to be unduly influenced by their design, which is ultimately arbitrary. No guidance exists regarding these design parameters because there is very little to say. A tool that produces arbitrary recommendations in an area as important as risk management in O&G should not be considered an industry best practice.’
A summative key point to take from HSE INDG163 is that it’s written there that:
‘Any record produced should be simple and focused on controls’
Risk assessment risk scores are not simple and as the HSE says themselves ‘could result in applying unnecessary control measures or failing to take important ones.’ The other benefit of keeping risk assessments simple is that they can be understood and conducted by all to help improve health & safety management and awareness across a whole organisation. Engagement and awareness of staff in health and safety is proven to reduce the likelihood of accidents.
Does Smartlog have numeric risk score scales, colour coding or a risk matrix structure?
No, Smartlog has been designed to keep risk assessment simple and efficient in order help you improve safety quickly. In HSE document INDG163 on page 3 it’s actually stated that ‘any record produced should be simple and focused on controls’. Safesmart believe in safety through efficiency.
Smartlog’s qualitative risk assessment structure focuses on actions to improve safety and lower risk. Actions to correct hazards. Rather than having the option to give a numeric score or colour rating for a risk, Smartlog focuses on pass/fail questions with comments & images to ensure compliance and clearly show what action needs to be taken to improve safety via the selection of normal or high priority.
Scoring a risk out of 5 for example may mean that lower numbered risks are ignored or forgotten about. As mentioned earlier in this blog post, The HSE write online that ‘Getting this wrong could result in applying unnecessary control measures or failing to take important ones’.
Low risk hazards that may or may not have significant severity are still important and Smartlog ensures that all risks are clearly visible putting high importance hazards at the top of the interactive to-do list called ‘due checks & tests’ automatically based on answers to risk assessment questions and the selection of high or normal priority. Based on risk assessment answers, reminders are also sent to remind individuals to take corrective action, there are also reminder escalation levels so seniors are notified if action hasn’t been taken by tasked individuals. When corrective action is taken the due check & test is moved to complete and the risk assessment is updated accordingly so focus can again be on remaining corrective action that needs to be taken.
Safesmart’s view is that there is extensive difficulty in determining a score number for the likelihood and severity of risks as it’s a subjective process. As well as creating the issue of lower risk score hazards being forgotten, deliberation over scores can create confusion and waste time when the priority of a risk assessment is improving safety. Evidently time should be spent on this, not deliberating about the scoring of everything with a number or colour. Due to the nature of this subjectivity, discrepancy and difference in reporting by different individuals, issues are inherent in number & colour scoring of risk assessments. This means that compliance monitoring is affected, thus creating the possibility of confusion and misinformed decision as a consequence of scoring.
Conclusion
We hope reading this article has highlighted the issue of numeric risk scores and risk matrices. If you are still using a risk matrix with risk scores for your risk assessment process we urge you to reconsider and discover Safesmart’s Smartlog software. Our software is designed for fast and efficient risk assessment needed in today’s 21st century management environment that seeks to involve all in improving safety without unnecessary over complication and bureaucracy. Smartlog’s risk assessments are always live, always assigning corrective action and always helping to save lives.
Latest News
Smartlog News
Latest News
Smartlog News