Monitoring Performance is About 2 Comparisons
by Stacey Barr |Don’t default to familiar or traditional KPI monitoring methods; focus first on the two comparisons that get the most insight from our performance measures.
When we monitor performance measures, it can seem very natural to analyse the data in a variety of ways, like comparing performance between work groups or business units, or classifying the performance measure based on variables like month of the year, day of the week, work shift, geographical location or customer segment. But really what we are doing is diluting the information we are getting from our performance measures.
Generally the analysis we do is driven by what might be interesting or familiar to us. The diving straight into analysis without first thinking about the questions we’re really trying to answer.
When we look at performance measures there are really only two questions that we need to ask first and foremost:
- The first question is how does current performance compare to the past?
- The second question is how does current performance compare to where we want it to be?
These are the two questions that matter, because monitoring performance is essentially about monitoring change over time relative to an ideal change over time.
The first question, how does current performance compare to the past, helps us work out if performance has responded to the actions we’ve taken to get it to improve. The second question, how does current performance compare to where we want it to be, helps us assess whether we are doing enough of the right action to lift performance to the level we want it to be at.
So the first comparisons we’re interested in with each performance measure are:
- current performance compared to past performance (to see change over time), and
- current performance compared to desired performance (to see how far we are from targets).
Any other comparison, like comparing geographical locations or customer segments or business units, are comparisons that may be useful in understanding or explaining the way that performance is changing over time. The other comparisons are part of cause analysis, not performance monitoring.
Speak Your Mind
Join Measure Up
Sign up for the Measure Up newsletter and get free access to the "10 Secrets to KPI Success" online course and e-book.
Upcoming KPI Training
>> Africa, In-Person, Dubai UAE, 3-5 May 2023
>> Australia/NZ/Asia Pacific, In-Person, Wellington NZ, 9-11 May 2023
>> UK & Europe, Online Interactive, 22-26 May 2023
>> North America, Online Interactive, 29 May - 2 June 2023
Connect with Stacey
Haven’t found what you’re looking for? Want more information? Fill out the form below and I’ll get in touch with you as soon as possible.
167 Eagle Street,
Brisbane Qld 4000,
Australia
ACN: 129953635
Director: Stacey Barr
Hi there – thank you for the reminder – it is often hard not to find the differences interesting. We just rolled out a new implementation of a CRM. There are system faults we are monitoring for and as I have looked into these, many of them seem to be the differences between training teams and how the lesson is given …or heard in different cultures. But keeping the big picture in mind as we want to make these go away is key to working with the IT specialists.
Hope you get over your crash soon – sounded spectacular. “They” say mountain biking is the sport where you are guaranteed a crash at some point in your riding tenure. I separated a shoulder…. FYI, dictation software was great for email while my arm was in a sling…if you can find one that speaks Aussie,
Hey Tom, thanks for sharing your own crash injury :-/
I use Dragon software and I had to train it to understand my accent!!
Succinct and to the the point – and a very important point it is! Measuring results is only useful if it helps one understand why something happened in the past and informs the future!
Hi Mike, and thanks for your comment. Surprisingly not many people think of measures in this role, and use them just to judge. Hence the fear associated with measuring what really matters, hey?
Re: “how does current performance compare to where we want it to be” – this is an area where managers become difficult to manage.
Let’s imagine that 2014 ended at 2% rejects. Placing a 95% confidence interval (based on last 12 months) around that gives us 1.7 – 2.3 as a predictor for 2015 – if we do nothing to improve or degrade the system. Management has plans to improve the system and sets the target for 2015 at 1.8%. And now, the problem begins. Performance is reported monthly. If the month is above 1.8%, explanations are required – even in January, when the improvement plans haven’t had time to take hold! Normal variation is not considered. After all, 1.8 translates into a CI of 1.5 – 2.1! It’s most frustrating to have to explain (make excuses for!) why the yearend target hasn’t been met by the performance of a particular month – when we don’t even expect to be there yet!
I think the question just might be, what is current performance – this month’s value or a rolling 12 month average? I think the latter.
Comments welcomed.
Bill, I’m not a big fan of 12 month rolling averages. I think they can hide signals too easily so we lose sensitivity.
In the scenario that you describe, I would rather see the improvement target be set with some more thought – in other words, I wouldn’t set a new target that saw the full year increase apply across the whole year – as you point out, that’s just unrealistic because in the early days the improvement plans may not have had time to take effect. So in that scenario, I would set a progressive increase in the monthly targets – which would reflect the reality of what I was hoping to see.
Grant, I don’t disagree about the 12 month average hiding some signal. I think it’s good to show the monthly values. A sloped target line would be one consideration although I would like to “band” with CI lines. For me, it’s not so much the statistically correct aspect as trying to educate management to stop chasing their tail because what they’re asking for makes no statistical sense.
Bill your intentions are great and I would have the same. The easiest method I’ve seen to help people focus on patterns in variability rather than monthly values is the XmR chart.
I like it, Grant.
Bill, I think a little differently about what ‘current performance’ is. See:
https://staceybarr.com/measure-up/three-things-you-need-on-every-kpi-graph/ (to me, current performance is the Central Line in the XmR chart)
https://staceybarr.com/measure-up/targets-are-about-capability-improvement-not-chasing-numbers/
https://staceybarr.com/measure-up/are-targets-expectations-or-intentions-and-why-does-it-matter/
Stacey, I’d say that you think but very little differently. The Central Line in the XmR chart (which chart I use for this) is a rolling average of the points contained – which should be a minimum of 25 points to be statistically “valid”.
Thanks for the comments,
Bill
Bill, if you haven’t already, take a look at Donald Wheeler’s book, Understanding Variation. That’s where I get the rules for constructing XmR charts and his work in this area is the most respected in the field, as far as I can tell.