Share:

When we measure, there are consequences. Ask these questions so you can prevent unwanted unintended consequences triggered by your KPIs.

In our world everything is interconnected, like the tides and the weather and the flowers and the bees. When something happens with one part of the system, there are flow-on consequences for other connected parts of the system.

Nothing is an exception to this. Not even such abstract things as performance measures or KPIs. When we start measuring something new, there are flow-on consequences for other parts of the organisation’s system. And many of the consequences are unintended – we didn’t choose the KPI to create them, but they happen nonetheless.

Unintended consequences can be good or bad.

Some of a KPI’s consequences will be good news. Like how Southwest Airlines’ KPI of Plane Turnaround Time has good consequences of higher profit, better customer loyalty and greater employee collaboration. Good unintended consequences can be leveraged.

But some of a KPI’s consequences will be bad news. Like how banks have measured Sales per Branch which drove employees to sell products to customers who didn’t need them or couldn’t afford them and to even create fake customers to open new accounts. Bad unintended consequences need to be managed, mitigated, or best of all, prevented.

Prevent the bad unintended consequences.

To prevent the unwanted unintended consequences of a new KPI or performance measure, here are 10 questions to prompt you to decide whether to manage, mitigate or prevent them:

1. Could measuring this cause or increase harm to anyone?
2. Could this measure insult or undermine the dignity of anyone (and thereby lose their support)?
3. If this measure improved, could that sabotage another result’s or measure’s performance?
4. Is improving this measure at odds with any of the organisation’s values?
5. Is it possible that improving this measure might create more waste?
6. Will this measure undermine collaboration?
7. Might reaching a target for this measure escalate internal competition for fixed resources?
8. Is the measure likely to encourage a quick fix that will ultimately fail?
9. Will implementing and using this measure shift the burden of cost to another part of the system?
10. Is this measure capable of triggering gaming behaviours?

What other questions would your own experience add to this list?

Any KPI can have negative unintended consequences. If you can’t mitigate them, maybe it’s not worth measuring it.
[tweet this]

1. Robert Bott says:

   September 1, 2020 at 11:02 am

   Hi Stacey, You have hit the nail squarely. An unintended consequence of QA has been the perceived need to hire “superstars”. These individuals consistently seek to add “outstanding” actions to their CV, often with little or no thought given to the impact of their “corrective actions” on the business as a whole, operations staff and operational function. Through application of your analysis as detailed, truly positive initiatives and actions can be identified and implemented safely within context rather than for personal benefit.

   Reply
   • Stacey Barr says:

     September 2, 2020 at 10:05 am

     That’s a great addition, Robert. I hadn’t thought of the flip-side of measurement as personally threatening, which you rightly point out as personal benefit.

     Reply
2. Jim Coffey says:

   September 1, 2020 at 6:49 pm

   Good points. Two thoughts:
   1. I would also ask “What will people stop doing?” It may be good to stop some things and detrimental if they stop doing others. In the SW example, what if baggage handlers stopped loading a late bag?

   While I generally agree with you comment “Any KPI can have negative unintended consequences. If you can’t mitigate them, maybe it’s not worth measuring it;” there is often tension between goals and thus measures. The key, IMHO, is to decide how to allocate resources and if the benefits of a metric outweigh the negative consequences.

   Reply
   • Stacey Barr says:

     September 2, 2020 at 10:07 am

     That balance between the benefit of a metric and its costs (including negative consequences) is a neat summary, Jim.

     Reply
3. Alberto says:

   September 2, 2020 at 1:18 am

   Hi Stacey, misaligned KPI are indeed the source of many systemic underperformances. Coudl you provide en example of “gaming behaviours”, please?

   Reply
   • Stacey Barr says:

     September 2, 2020 at 10:09 am

     Gaming is when people behave in a way that makes a measure hit its target but without performance actually improvement e.g. they change the target to something achievable, they change the metric to something they’re already good at, they manipulate the data (like leaving out bad cases) so the measure’s values look good, or they take actions that make the measure hit the target at the expense of some other performance result that matters.

     Reply
     • Alberto says:

       September 3, 2020 at 1:02 am

       Super! Thanks for your prompt reply, Stacey.

       Reply