How Many KPI Data Points Do You Need To Use And Interpret Them?

by Stacey Barr

There is a statistically proven sweet spot for how many KPI data points you need before you can start using them and accurately interpreting signals from them.

handsatlaptopkeysimagesofcharts.jpg

There are two basic questions that KPIs or performance measures need to answer for us:

  • What is the current level of performance?
  • Is performance changing?

To answer these questions reliably, we need to make sure we have enough data, or enough KPI values or points, so we don’t jump to the wrong conclusions, or wait too long.

This means finding the sweet spot, just like a cricket bat or tennis racquet or golf club has. It’s the point at which we get the biggest effect from the least effort.

The number of KPI data points matters because of variation.

Variation is the reason why we can’t rely on just one or two points of data to draw conclusions from our KPIs. Dr Donald Wheeler is the world’s foremost expert on understanding variation, and he explains:

“In the mathematical world two plus two is always equal to four. But in this world, in everyday life, two plus two is only equal to four on the average. When you think about this you know that from your own experience that there is variation in everything we do and variation in everything we measure.”

Because variation means that each single KPI value is not a precise measure of reality, we can’t rely on any single KPI value. We need to look at patterns in the values, not at the points themselves.

The minimum number of KPI data points to establish a KPI baseline is five.

A baseline is the collection of points in our KPI that define our starting point, the point from which we want to improve. We need a baseline, because without it we have no starting point, or “before” level of performance, to compare future performance to.

A single point is not a baseline. That’s because of normal variability in anything we measure. Even if you had an identical diet and exercise plan, your weight would still fluctuate from day to day. One day’s weight is not a precise baseline.

We need to use enough performance measure values to calculate the baseline, but not use any more than we need. And the sweet spot is 5 measure values. True; sometimes we need more than that, if the KPI’s variation is a little chaotic. But we can’t use less than five.

The number of KPI data points for an outlier is just one.

An outlier is a single point of data that behaves very abnormally to its neighbouring points of data. It doesn’t mean performance has fundamentally changed; it means that something unusual and rare has happened. Almost always, performance goes back to normal very quickly.

We see outliers very easily in XmR charts, because they fall outside the range of normal variation. We can still see them in line charts, when the outlier ‘lies out’ far enough from the other points. But the XmR chart picks them up instantly and accurately.

We should never fixate on outliers. It’s useful to find their cause, but also to realise their cause is just a rare or exceptional occurrence and isn’t worth the effort to fix.

The minimum number of KPI data points to declare a small change is eight.

A small change is a shift in performance that moves away from average performance but stays inside the interquartile range (middle 50%) of the spread of normal variability for the measure.

These small changes are called long runs, in XmR chart theory. Because the change is small, we need a bit more convincing that it’s real. And that’s why we need at least eight consecutive measure values to declare a small change in performance.

The minimum number of KPI data points to declare a big change is three.

A big changes is a shift in performance that moves away from the average performance, and also outside of the interquartile range of the spread of normal variability for the measure.

We call these big changes short runs, in XmR chart theory. A bigger change needs fewer points than a small change to convince us. In fact, we can use as few as three consecutive points (or three out of four consecutive points) to declare a big change in performance.

The minimum number of data points to establish a new baseline is five.

When we get a real change in our KPI’s performance, it’s time to create a new baseline. Both a long run signal and a short run signal mean that performance of our KPI has changed. So if we want to see future signals, we need a fresh baseline. And we calculate it using the first five KPI data points from the start of the signal we detected.

This detailed instruction for creating XmR charts will guide you through how to set up your KPIs or measures this way. If you do this with historic data, you might be surprised to see signals you never knew were there. Or to see a correlation between specific events or initiatives and those signals.

“I can’t wait that long!!!”

Do we really have to wait until we have five KPI values before we can start using the KPI? Do we really have to wait for eight KPI values to behave differently, to be sure performance has changed? Yes. We do.

If we don’t, we’re guessing, assuming and knee-jerk reacting. We’re wasting organisational resources on action that isn’t needed, or isn’t likely going to work. We cannot escape the fact that bad decisions are made from insufficient data.

If this means that you’re freaking out because you have to wait for five years of annual survey data before your Employee Engagement measure can be used, it means you’ve chosen the wrong cadence for your KPI.

Caveat: there are always exceptions to any rule.

Chaotic performance is one case where the above rules don’t really work. Our KPI or measure needs to have stable and predictable performance, even with variation, before we can pull out signals. Chaotic variation is usually a sign that the business process being measured isn’t in control. For example, it doesn’t have standard procedures, or the procedures just aren’t followed.

And another case where the above rules are not enough is when we have more complex changes in performance. This might be signals embedded within signals (like outliers embedded in a long run) or trend patterns. And for this, we need extra signal detection rules.

My advice is that we first master the above beginner’s guide to using just enough data to draw conclusions from KPIs. Then we can move onto more complex signal rules. The basic ones will cover most situations, in any case.

Like a golf club, there is a sweet spot for how many KPI values we need to accurately hit its interpretation.
[tweet this]

DISCUSSION:

What are the rules your organisation follows for the number of KPI values to use in drawing conclusions about current performance and changes in performance? Where did the rules come from?

FacebooktwitterlinkedinmailFacebooktwitterlinkedinmail

Speak Your Mind

Your email address will not be published. Required fields are marked *

  1. Bill Wiest says:

    Regarding the short run, in researching XmR Charts, I came across this rule on p. 112 of Donald Wheeler’s book: Making Sense of Data:

    Three out of four successive values in the upper 25%, or three out of four successive values in the lower 25% of the region between the limits may be taken as a signal of the presence of an assignable cause which has a moderate but sustained effect.

    It doesn’t appear to be the case that the 3 data point short run example above satisfy the lower 25% range qualification.Can you confirm, or comment further?

    • Stacey Barr says:

      They are different rules, Bill. I have come across the one you mention, in the same book. The upper 25% of the natural process limits is equivalent to the points being on the same side of the Central Line and closer to the upper limit. Same rule, different ways of saying it.

      • Bill Wiest says:

        I know. It is just that, the points can be closer to the upper limit yet not be within the 25% range. We have a chart like that. Each of the three data points is closer to the limit but not quite within that upper 25%. How important is it that it be within this 25% range?

        • Stacey Barr says:

          Perhaps I’ve misunderstood Bill, but the band between the upper and lower natural process limit is divided into 4 parts, or 4 sections of 25%. So on the same side of the central line, there are 2 25% zones. Therefore if points are closer to the limit than to the central line, they are by definition inside the outer 25% zone. This is described on page 112 of “Making Sense of Data” by Donald Wheeler. Have I misunderstood your question?

          • Bill Wiest says:

            No. I think I misunderstood Donald Wheeler. I thought he was saying that you divide the region above the Central Line, up to the Upper Natural Process Limit into 4, and same for below down to the Lower Natural Process Limit. My bad. Thanks for clarifying!

Upcoming KPI Training


Our next PuMP Performance Measure Blueprint Workshops include:

>> Australia/NZ/Asia Pacific, Online Interactive, 18-19, 23-25 Aug 2021

>> United Kingdom, Online Interactive 10, 17, 24 Sep & 1 Oct 2021

>> North America, Online Interactive, 27 Sep - 1 Oct 2021

>> North America, Online Interactive, 15-19 Nov 2021

>> United Kingdom, Online Interactive 2-3 & 9-10 Dec 2021

Register for the next PuMP Blueprint Workshop near you

Stacey's Books


Prove It! How to Create a High-Performance Culture and Measurable Success, book by Stacey Barr

Order Prove It! here.

Practical Performance Measurement: Using the PuMP Blueprint For Fast, Easy, And Engaging KPIs, book by Stacey Barr

Order Practical Performance Measurement here.

Reprinting Articles


You are welcome to use articles from the Measure Up blog, with these requirements

Connect with Stacey


Haven’t found what you’re looking for? Want more information? Fill out the form below and I’ll get in touch with you as soon as possible.



    *We respect your email privacy.
    Level 54, 111 Eagle Street
    Brisbane, Qld, 4000
    Australia
    Stacey Barr Pty Ltd
    ACN: 129953635
    Director: Stacey Barr