Outlier KPI values need to be handled appropriately to stop them from distracting our focus or distorting our decisions.

I recently received an email from a customer, with the only nasty criticism I’ve ever received for my work:

“Stacey I have taken tons of training over 20+ years and I must say that your training by far is the worst that I have ever taken. I would highly recommend that you spend some of your money that you have brought in and hire yourself someone who can put this material into a proper program. You are not by any means a trainer. I wouldn’t recommend this crap to my worst enemy.”

Naturally, something like this could be very upsetting. But I know it’s an outlier. And it’s the kind of outlier that is out of my control, and not useful to react to.

Outliers happen from time to time in any measure of performance.

Outliers are values that are very different to the typical distribution of values our KPI takes. They are generally out of our control and one-off events that are the product of rare or unusual causes, like these:

• Data recording errors, like a customer rating 1 out of 10 for satisfaction when they confused 1 as excellent instead of 10
• Data entry errors, like an HR officer typing an extra 0 when entering an employee’s wage
• Incredibly rare or unusual events, like the effect of major flooding on emergency service response times
• Natural extreme values of the population, like my case with the above customer

If we react to outliers, we risk changing a process (or product or service) that is actually stable and performing really well. It’s vital to tread carefully in handling outliers in our KPIs.

Ask the right questions before choosing how to handle your outlier KPI values.

Here’s a start at the kinds of questions worth asking before we do anything with our KPI outliers:

• What makes us think it is an outlier?
• Is the outlier a rare but legitimately possible value for our KPI?
• Can we get more detail about the outlier, and what may have caused it?
• Did something unusual or abnormal cause this outlier?
• Is the outlier value incorrect?
• Should the outlier value be replaced with corrected data or more realistic estimates?

The case with my disgruntled customer is definitely a legitimate situation. It’s true; there are people that have reacted negatively to training I’ve delivered. Often we find it’s because they were told to attend but didn’t want to. Or it challenged their existing views too much. But this has happened for only 0.49% of all participants in training I’ve personally delivered. Outliers.

There are 3 steps to handle outliers in KPI values.

Am I going to spend many weeks of my time and tens of thousands of dollars improving a training program that rates consistently at 8.5 out of 10 for overall value? Just because one customer thinks I should? Of course not. That customer’s experience is not consistent with the vast majority, so it’s not representative of the vast majority.

Rather than knee-jerk react to outliers, we can follow a simple flowchart to determine what to do with a KPI outlier, like this:

1. Accept the outlier, if it is a possible value that our KPI can take, however rare and unusual. There’s a good chance it won’t significantly affect the interpretation of our KPI over time, nor the calculations for our XmR charts and resulting signals.If the outlier isn’t reasonable to accept, then
2. Correct the outlier, if we can find the original and correct source or if we can create a plausible and more realistic estimate. We often did this, back in my statistical research days analyse production and financial data from the agriculture sector. We would replace outlier values with averages of the values for farms that shared very similar demographics.If the outlier value can’t be corrected, then
3. Delete the outlier, if it’s clearly incorrect (e.g. not even a possible value) and cannot be corrected or replaced with an estimate.

It’s easy to knee-jerk react to outliers. They can be upsetting and fill us with fear of the consequences if others notice them too. So our initial tendency might be to fixate on them, wasting too much time analysing them explaining them away.

It’s best to simply identify their cause, but never let them waste the time and energy we should give to other KPIs that really aren’t performing well enough yet.

Do you knee-jerk react to outliers in your KPI values, and let them waste your time, energy and emotion?
[tweet this]