3 Essential Signals to Look for in Your KPIs

September 3, 2013 by Stacey Barr

You won’t find true signals of changes in performance by looking at month-to-month comparisons, or trendlines, or moving averages. The signals you really need to know about, the only signals that you ought to respond to, are revealed through one particular graph only.

The typical analysis methods we use for our performance measures are based on assumptions that don’t make much sense.

Month-to-month comparisons assume that there is no routine variation over time and wrongly interpret any difference as a signal.

Trendlines assume that all change is linear and gradual, and that if Excel can calculate a trend line then there must be a trend.

Moving averages assume that seasonal patterns exist, and also that change is smooth and gradual.

The only analysis technique I have ever come across that clearly filters the noise and highlights the signals in our performance measures is the XmR chart.

The XmR chart filters the noisy routine variation in our measures by showing us how much of this routine variation there is, by way of the Natural Process Limits. And coupled with the Central Line, these Natural Process Limits give us a meaningful baseline to quickly assess when performance has changed; when there is something else going on that’s not part of the routine variation.

There are three very specific signals to look for.

SIGNAL 1: Outlier or special cause

When a measure value falls outside the Natural Process Limits, it means that more than just the routine variation is at play. It’s a signal that something else has happened.

If Employee Attendance plummeted below the bottom Natural Process Limit, a likely cause could be a flu epidemic or local natural disaster that kept many more people away from work in that period.

Even though this is a signal that something out of the norm has happened, because it’s just a one-time event, we don’t react to it. We find out what caused it, but we don’t run around madly trying to fix it. That would be a massive waste of time and money because we’d essentially by trying to fix something that won’t happen again, or that is completely outside our control.

SIGNAL 2: Long run

To be convinced that a change in the level of performance has happened, we need to see seven (yes, seven) points in a row on the same side of the Central Line. The probability that a pattern like that is part of routine variation is close to zero (0.78%, to be precise). Seven points, not three or five or one.

If a measure of Invoice Accuracy showed a long run above the Central Line, it might be evidence that an initiative to simplify the pricing strategy had successfully reduced the errors in invoices.

When we see a long run signal in our measure, we certainly need to find the cause for it. Sometimes it will be a signal of improvement, and we want to confirm what caused the improvement. Other times it will be a signal that performance has deteriorated and the cause of that is very important to identify!

SIGNAL 3: Short run

You’re no doubt thinking to yourself ‘I can’t wait for seven months before I can know if I should take action!’ You can either measure more frequently to pick up signals sooner (as long as it makes sense to), or plan for bigger signals.

A bigger signal appears as a short run, of three out of four consecutive measure values closer to a Natural Process Limit than they are to the Central Line. The probability of this pattern happening also has a very close to zero probability.

A short run above the Central Line for On-time Deliveries for a trucking company would likely be due to an initiative that had a substantially large impact. It could be something like doubling the fleet size. But it also could be a new competitor in the market that poached a large percentage of their customers.

Again, with a signal like the short run, it’s really important to find the cause before responding.

XmR charts take only a little effort to create, but their usefulness is so powerful they are absolutely worth trying.

TAKE ACTION:

If you’re one of the first 5 readers to submit 20 to 30 values of your performance measure in time series, I will put your data into an XmR chart and report back on the Measure Up blog on signals the measure might contain. Yes, I’ll keep it anonymous unless you say it’s okay to identify your organisation!

Facebooktwittergoogle_pluslinkedinmailby feather

Speak Your Mind

Your email address will not be published. Required fields are marked *

  1. elshafie albasheer mohamed ali says:

    Does monthly moving averages assume seasonal variations? and how?

    • Stacey Barr says:

      Elshafie, yes moving averages are usually used to smooth out seasonal variation. For XmR charts, we don’t use moving averages when we have seasonal variation – there is another method for ‘deseasonlising’ the data so we can see signals that aren’t swamped by the seasonality.

      This is covered in detail in my online course: http://www.usingsmartcharts.com

  2. Prahlad Bhugra says:

    Stacey, just verifying my thoughts – purpose of calculating moving range is only to help in calculating the UCL & LCL limits in the X control chart. The formula is –

    A) Upper Control Limit = Average of observations + E * Average of moving range
    B) Lower Control Limit = Average of observations – E * Average of moving range
    Also, E is a constant – usually we take difference in 2 consecutive values for calculating the moving range and the value of E is 2.66

    Is my understanding correct?
    Also, if you can explain – the reason for choosing 2 consecutive values for calculating the moving range – that would be helpful – if we take 3 or 4 or 5 values for calculating the moving range – the UCL & LCL will be different & our inferences will change.

    regards
    Prahlad

    • Stacey Barr says:

      Thanks for your comment, Prahlad. Yes your calculations are correct for how we use the moving range for the UCL and LCL (although they are now referred to as the upper and lower Natural Process Limits).

      I only ever use the 2 consecutive values for the moving range for three reasons:

      1) That’s the way I learned it from Donald Wheeler.
      2) It’s simple, and I want people to use this XmR tool and get the benefits from it – if we make it more complex, that won’t happen.
      3) I am not trying to be an expert in XmR or other statistical process control tools, but want to bring them into the field of performance measurement. So I’m certainly not the expert that can answer every question about them. We have Donald Wheeler, at least, for that!

      As I mentioned in my reply to Elshafie above, I’ve kept the application of XmR charts to performance measurement as simple as I can in the Using Smart Charts course.

  3. Stacey Barr says:

    I am working on two XmR charts for readers who kindly took my challenge and sent me their measures. We have space for three more, so please don’t hesitate in sending your data.

    If you’re worried about the effort of sending the data and not being one of the first five, then just send us an email saying you want to send it, and we’ll confirm that you’re in the first five. Then you can send your data through to us.

  4. Matt says:

    Hi Stacey,

    Really enjoying your articles on smart charts, I was hoping you could answer 3 quick questions I have:

    1) Do you typically refresh the central line / natural process limits when you get any of the 3 signals?
    2) Do you refresh the central line and natural process limits the period after the signal occurs, or do you refresh on the same period the signal occurs? For example I get a signal in March, do I refresh the values in March or do I refresh in April?
    3) Have you any plans to write a book about Smart charts?

    • Stacey Barr says:

      Hi Matt, glad you’re enjoying these smart charts! I love them too, for how insightful they are. Quick answers to your questions (but please be warned: it’s not appropriate for me to teach the technical details of these XmR charts in blog comments because it’s too easy to miss important pieces of knowledge and take the answers out of context):

      1) The central line and natural process limits are recalculated after only the long run and short run signals – never with the special cause signal.

      2) When you do the recalculation, you generally use the 5 to 7 points starting from the first point in the long run or short run signal.

      3) Hadn’t thought about writing a book but I do have a very practical online course you can take that includes online demos, comprehensive notes, instructions and Excel templates. Go to http://www.usingsmartcharts.com to read more.

  5. Stacey, I’m not sure that I agree with you on the one time outlier event. For the Employee Attendance example perhaps (although if it was a flu epidemic, might the company not want to offer free flu shots?), but for quality or productivity situations, I’m not so sure.

    We use the term ‘Sentinal Events’ for outliers that are greater than x weight of defective material (all one cause) within a defined period. Should that be due to a failure to follow procedure, or the premature failure of an equipment component, I think it may well bear the “cost” of investigation and correction. If it can happen once, it can happen again.

    Admittedly, you do state that “we find out what caused it” and then go on to state that “we don’t react to it”. Finding out what caused it is a reaction and justly so. The circumstances of the one off failure may dictate a ‘do not correct’ result.

    I’m not certain that we think differently on this. It may be a difference in our perception of the “we don’t react to it” statement. I think I would have phrased it differently.

    • Stacey Barr says:

      Bill, that’s a thoughtful response. As you suspect, I do indeed see ‘finding the cause’ as something different to ‘reacting to it’. Reaction, to me at least, is when we rush to do SOMETHING, ANYTHING, to try and fix a problem before we even know it’s a problem. Action I see as different to reaction, and as something more deliberate and informed. I like to draw a distinction between the terms, because I see too much reacting and not enough cause-analysis-informed action.

  6. Hi Stacey,
    As a result of your post, I’ve introduced XmR charts to several clients and they are being well received. I’ve managed to build them in Tableau (as well as Excel) and they will become a key tools for analysing clients’ online survey data.

    Ben Jones in Communicating with Tableau has a chapter called “Variation over time:Control charts”. He cites Walter Shewhart.

    I’ve completely taken on board the definition of signals used by you and Donald Wheeler (Outlier, Long Run and Short Run). However, Ben Jones gives slightly different definitions. Can you help me to reconcile the two methods please?

    Ben’s definition of signals:
    – Outliers (data points either above the UCL or below the LCL)
    – Trends (six or more points either all ascending or all descending)
    – Shifts (nine or more points either all above or all below the average line).

    So Trends is new and there’s no equivalent of Short Run.

    Short run and Long run are meaningful to distinguish one from the other. However, when reporting, I think the term “Shifts” may be more self-evident than a “Long run”.

    Regards,
    Helen

    • Stacey Barr says:

      Helen, I wonder if Ben Jones is using XmR charts or some other type of statistical process control chart? Donald Wheeler is certainly the expert I refer to, and he has worked with Deming so really knows his stuff. The rules that Ben quotes are rules I have not seen anywhere, so I suspect they might apply to a different kind of control chart.

Connect with Stacey


Haven’t found what you’re looking for? Want more information? Fill out the form below and I’ll get in touch with you as soon as possible.



*We respect your email privacy.
PO Box 422
Samford, Qld, 4520
Australia
Stacey Barr Pty Ltd
ACN: 129953635
Director: Stacey Barr
Simple Share Buttons