Two Principles For Reporting Your Performance Measures

February 18, 2014 by Stacey Barr

We slice, we dice, we tabulate, we chart, we dashboard and then we draw conclusions. But how often have we drawn diabolically incorrect conclusions because we have violated the two basic principles of understanding our measures?

In Donald Wheeler’s book Understanding Variation, he introduces two very simple and sensible principles for understanding data:

  1. No data have meaning apart from their context.
  2. While every data set contains noise, some data sets contain signals. Therefore, before you can detect a signal within any given data set, you must first filter out the noise.

What do these principles mean, particularly in the context of reporting performance measures or KPIs?

PRINCIPLE 1: No data have meaning apart from their context.

The first principle means that you must understand where your data come from, and you must have enough data, before you can understand what a measure is telling you.

Here are some practical tips to ensure you report your measures, meaningfully, with enough context:

  • Provide a link to and a summary of the meaning of the measure: it’s calculation, data source and plain-english description of what the measure means.
  • Report more than just one or two values of your measure. In fact, do your best to report your measures in time series, including up to 3 years’ worth of historical values.
  • Put your measures into graphs so it’s faster and easier to see patterns over time. When we’re trying to improve performance, it’s the patterns over time that tell the most insightful stories of performance.

PRINCIPLE 2: Before you can detect a signal, you must first filter out the noise.

The second principle means that every time you compute a difference between two values, you cannot assume that difference means there was a change. These computations include this month to last month, this month to the same month last year, this month compared to target, and so on.

We all know how to quantify performance using averages and totals and percentages. But in order to filter out the noise in our measures, we also need to quantify the variation.

Here are some practical tips to ensure you report real signals in your measures (and not be misled by the noise):

  • For performance measures you monitor regularly over time, use XmR charts.
  • When you are reporting survey results that were based on a sample, report the confidence intervals for the survey estimates.

I’m really looking forward to data visualisation expert Stephen Few’s next book on this very topic, called Signal.

TAKE ACTION:

Take a look at your performance reports and see if you can find any measure or KPI presentation that is violating the two principles for understanding data. What risks does this expose your decision-making to?

Facebooktwittergoogle_pluslinkedinmailby feather

Speak Your Mind

Your email address will not be published. Required fields are marked *

  1. Bob Frost says:

    Stacy,

    Thanks for reminding us of these two key points. We constantly see people violating the second principle–assuming that any change between two points in the data is actually real and meaningful.

    And thanks for the alert that Stephen Few has a new book coming; I’m sure it will be as great as his others.

    Bob Frost

    • Stacey Barr says:

      Hi Bob, thanks for dropping by. You’re a man after my own heart: I agree about how many people violate the second principle.

      My motto is: patterns, not points. And XmR charts make the patterns (the real ones) so very obvious.

Reprinting Articles


You are welcome to use articles from the Measure Up blog, with these requirements

Connect with Stacey


Haven’t found what you’re looking for? Want more information? Fill out the form below and I’ll get in touch with you as soon as possible.





*We respect your email privacy.
PO Box 422
Samford, Qld, 4520
Australia
Stacey Barr Pty Ltd
ACN: 129953635
Director: Stacey Barr
Simple Share Buttons