How to Handle Outliers in Your KPI Values

by Stacey Barr |

Outlier KPI values need to be handled appropriately to stop them from distracting our focus or distorting our decisions.

A bird sitting as an outlier from the flock. Credit:

I recently received an email from a customer, with the only nasty criticism I’ve ever received for my work:

“Stacey I have taken tons of training over 20+ years and I must say that your training by far is the worst that I have ever taken. I would highly recommend that you spend some of your money that you have brought in and hire yourself someone who can put this material into a proper program. You are not by any means a trainer. I wouldn’t recommend this crap to my worst enemy.”

Naturally, something like this could be very upsetting. But I know it’s an outlier. And it’s the kind of outlier that is out of my control, and not useful to react to.

Outliers happen from time to time in any measure of performance.

Outliers are values that are very different to the typical distribution of values our KPI takes. They are generally out of our control and one-off events that are the product of rare or unusual causes, like these:

  • Data recording errors, like a customer rating 1 out of 10 for satisfaction when they confused 1 as excellent instead of 10
  • Data entry errors, like an HR officer typing an extra 0 when entering an employee’s wage
  • Incredibly rare or unusual events, like the effect of major flooding on emergency service response times
  • Natural extreme values of the population, like my case with the above customer

If we react to outliers, we risk changing a process (or product or service) that is actually stable and performing really well. It’s vital to tread carefully in handling outliers in our KPIs.

Ask the right questions before choosing how to handle your outlier KPI values.

Here’s a start at the kinds of questions worth asking before we do anything with our KPI outliers:

  • What makes us think it is an outlier?
  • Is the outlier a rare but legitimately possible value for our KPI?
  • Can we get more detail about the outlier, and what may have caused it?
  • Did something unusual or abnormal cause this outlier?
  • Is the outlier value incorrect?
  • Should the outlier value be replaced with corrected data or more realistic estimates?

The case with my disgruntled customer is definitely a legitimate situation. It’s true; there are people that have reacted negatively to training I’ve delivered. Often we find it’s because they were told to attend but didn’t want to. Or it challenged their existing views too much. But this has happened for only 0.49% of all participants in training I’ve personally delivered. Outliers.

There are 3 steps to handle outliers in KPI values.

Am I going to spend many weeks of my time and tens of thousands of dollars improving a training program that rates consistently at 8.5 out of 10 for overall value? Just because one customer thinks I should? Of course not. That customer’s experience is not consistent with the vast majority, so it’s not representative of the vast majority.

Rather than knee-jerk react to outliers, we can follow a simple flowchart to determine what to do with a KPI outlier, like this:

  1. Accept the outlier, if it is a possible value that our KPI can take, however rare and unusual. There’s a good chance it won’t significantly affect the interpretation of our KPI over time, nor the calculations for our XmR charts and resulting signals.If the outlier isn’t reasonable to accept, then
  2. Correct the outlier, if we can find the original and correct source or if we can create a plausible and more realistic estimate. We often did this, back in my statistical research days analyse production and financial data from the agriculture sector. We would replace outlier values with averages of the values for farms that shared very similar demographics.If the outlier value can’t be corrected, then
  3. Delete the outlier, if it’s clearly incorrect (e.g. not even a possible value) and cannot be corrected or replaced with an estimate.

It’s easy to knee-jerk react to outliers. They can be upsetting and fill us with fear of the consequences if others notice them too. So our initial tendency might be to fixate on them, wasting too much time analysing them explaining them away.

It’s best to simply identify their cause, but never let them waste the time and energy we should give to other KPIs that really aren’t performing well enough yet.

Do you knee-jerk react to outliers in your KPI values, and let them waste your time, energy and emotion?
[tweet this]


Speak Your Mind

Your email address will not be published. Required fields are marked *

  1. Mike Davidge says:

    You don’t really need me to speak up in your defence but I will anyway. My theory is that your course threatened his world view (I’m assuming it’s a him) hence the violent reaction. I’m impressed you shared, not sure I would have had the courage. I guess you wanted a segue into outliers.
    You’re right that we need to react properly to outliers. But first we need to identify them and the XmR chart does a cracking job here. Once that’s done then we need a strategy for dealing with them. And your recipe is spot on. We need a contingency plan for outliers not a process redesign as you pointed out. The latter is only an appropriate reaction for unacceptable common cause performance.

    • Stacey Barr says:

      Firstly, Mike, thanks for your encouragement. From someone as deeply committed to and experienced in our field of measurement as you are, it means a lot. It’s not comfortable sharing things like that customer’s rude feedback. But I believe in transparency and authenticity, so I have to sometimes push past comfortable to live that. And yes, the story was a perfect segue into the topic of outliers, and hopefully also permission for us all to acknowledge that outliers can feel really bad sometimes, but we still must face them as part of reality.

Upcoming KPI Training

Our next PuMP Performance Measure Blueprint Workshops include:

>> United Kingdom, Online Interactive 5, 12, 19, 26 Nov & 3 Dec 2021

>> Australia/NZ/Asia Pacific, Online Interactive, 10-11, 15-17 Nov 2021

>> North America, Online Interactive, 15-19 Nov 2021

>> Africa, Online Interactive, 14-18 Mar 2022

>> Africa, Online Interactive, 20-24 Jun 2022

>> Africa, Online Interactive, 26-30 Sep 2022

>> Africa, Online Interactive, 28 Nov - 2 Dec 2022

Register for the next PuMP Blueprint Workshop near you

Stacey's Books

Prove It! How to Create a High-Performance Culture and Measurable Success, book by Stacey Barr

Order Prove It! here.

Practical Performance Measurement: Using the PuMP Blueprint For Fast, Easy, And Engaging KPIs, book by Stacey Barr

Order Practical Performance Measurement here.

Reprinting Articles

You are welcome to use articles from the Measure Up blog, with these requirements

Connect with Stacey

Haven’t found what you’re looking for? Want more information? Fill out the form below and I’ll get in touch with you as soon as possible.

    *We respect your email privacy.
    Level 54, 111 Eagle Street
    Brisbane, Qld, 4000
    Stacey Barr Pty Ltd
    ACN: 129953635
    Director: Stacey Barr