How to Measure Government Outcomes

August 20, 2019 by Stacey Barr

Government outcomes are particularly challenging to measure meaningfully, because they are intangible, long-term, and seldom directly impacted by a single department’s or agency’s actions.

Australian Parliament House. Credit: zetter

The current common practices for choosing measures of performance are too limited to meet these challenges of creating good outcome measures for government departments and agencies.

CHALLENGE 1: Government outcomes are intangible.

Most government agencies exist to influence social stability, justice or welfare; not to produce widgets. Inherently, these social outcomes are softer and fluffier than the outcomes of non-government organisations. But this isn’t the reason that outcomes are hard to measure. If an outcome matters, we should be able to describe it clearly enough to observe, in some way, if it’s happening or not. But how most government agencies describe their outcomes is not with clear language that describes anything observable:

“The advancement of Australia’s international strategic, security and economic interests including through bilateral, regional and multilateral engagement on Australia Government foreign trade and international development policy priorities.” – Foreign Affairs and Trade

The common practice that limits our ability to meaningfully measure intangible outcomes is using vague ‘weasel words’ to articulate those outcomes. If we accepted the premise that some outcomes are too broad and intangible to measure directly, then we accept the premise that there will be no evidence of such outcomes if and when they happen. If there is no evidence of an outcome when it happens, no-one will notice if it’s happening or not. So why bother with all the effort to try and create that outcome in the first place?

To succeed at finding meaningful measures of intangible outcomes, we need to use plain language to articulate them. And we can’t start with questions like “How could we measure this?” Rather, we need a measure design process that starts with questions like “How would we recognise if this was happening or not?”

CHALLENGE 2: Government outcomes are long-term.

If it’s true that government agencies exist to influence social stability, justice and welfare, then it can’t be ignored that such things change over longer timeframes. Most government outcomes can’t be measured as frequently as sales quotas or revenue growth or tonnes of coal exported:

“Contribute to the improvement of the extent, condition and connectivity of Australia’s unique biodiversity and natural resources, consistent with national and international obligations, through protection of habitats and mitigation of threats to threatened species and ecological communities.” – Department of the Environment and Energy

The common practice that limits our ability to meaningfully measure long-term outcomes is limiting our measures to our corporate planning horizons. And this usually means that we measure only the strategic initiatives within those planning horizons. The above outcome isn’t measured directly, and only its Delivery Strategies have measures (and most are activity measures or milestones):

“Number of projects/contracts.

Project designs and contracts reflect the objectives of the Reef 2050 Plan and Reef Trust.

Project outcomes are fed into modelling and monitoring programs to inform reporting on Reef health.”

To succeed at finding meaningful measures of long-term outcomes, we can’t default to monitoring strategies and initiatives. If we do, we’ll struggle to ever learn which types of activity have the best impact on the outcome. In contrast, when we measure both the outcome and the results of our strategies and initiatives, we can test which work best, and then develop useful lead indicators to monitor more regularly to project changes in the outcome measure.

CHALLENGE 3: Government outcomes are systemically complex.

The role of government is more to influence than to directly control social outcomes. Government organisations often rely on collaboration with other public, private and nonprofit sector organisations for an outcome to be achieved. Government departments and agencies can struggle to isolate their contribution toward outcomes they can only partially influence, or ‘facilitate’:

“Facilitate jobs growth through policies and programs that promote fair, productive and safe workplaces, and facilitate the growth of small business.” – Department of Jobs and Small Business

The common practice that limits our ability to measure systemically complex outcomes is defining accountability as ‘hitting targets’. We know that the actions of a single department or agency cannot completely control if or how well a systemically complex outcome will happen. No one will accept being judged by a target they don’t have control of achieving. So, it’s no wonder that the above outcome is measured by such an operational measure as this:

“Percentage of claims processed within 16 weeks of receipt of an effective claim.”

To succeed at finding meaningful measures of systemically complex outcomes, we can’t waste time trying to isolate our own department’s or agency’s impact on those outcomes. Rather, we need to build and test a causal map of how we influence that outcome, and measure both the outcome and its drivers (causal factors) that we do control.

These challenges are the clues to how to measure government outcomes.

This white paper on How to Measure
Government Outcomes
goes into a lot more detail on dealing with these challenges, and offers a long list of techniques and resources to make the job easier.

Government outcomes can be meaningfully measured by working through the challenges of how intangible, long-term, and systemically complex those outcomes are.
[tweet this]

TAKE ACTION:

What other challenges do you face in measuring outcomes in your organisation, whether it’s government or not?

FacebooktwitterlinkedinmailFacebooktwitterlinkedinmail

Speak Your Mind

Your email address will not be published. Required fields are marked *

  1. Ian Stanbury says:

    I find this whole area a fascinating subject. I have worked in quite a few different areas of the UK public sector and seen the struggles of senior managers trying to create KPIs for societal outcomes, which has led to totally ineffectual and wasteful ‘KPIs’ and scorecards being implemented. I knew this approach was wrong somehow, but it wasn’t until I heard Professor Mark Moore (Harvard) at a conference in 2006, discussing his concept of ‘Public Value’ and explaining the ‘Public Sector Value-Chain’, that it all clicked and became clear to me.

    So, what clicked? Essentially, it was the notion that any public sector body should consider its ‘customers’, i.e. recipients of its services, as part of its whole value-chain because it’s the customers that ultimately realise the societal outcomes through their actions and behaviours – not the public sector body itself – and that might take years to occur. Therefore, public sector bodies can only measure those things it can control, i.e. people, processes and resources, which is where meaningful KPIs can be derived. The best any public sector body can do is to show a potential relationship and correlation – not cause, because other organisations might also be involved – between its activities/interventions and the longer-term measurement of changes in societal actions and behaviours.

    As an example, a UK vehicle regulatory agency was trying to claim percentage KPIs of UK government road-safety and crime targets. I pointed out that the organisation’s main area of activity and resource allocation was on vehicle inspection but the agency’s parent transport department’s own figures showed that vehicle condition accounted for less than 2% of people killed or seriously injured in road traffic accidents, whereas driver behaviour accounted for over 45% of the total. The agency had only a relatively small budget for education and prevention activity so its claims for reducing the total number of people killed or seriously injured in road traffic accidents were inversely proportionate to its activities and resources. I have other examples too.

    The same public value concept is built into the logic model that underpins the UK Government’s ‘Magenta Book’ guidance for evaluating the effectiveness of public policy. The model assumes that a public sector body takes inputs (including public money from taxes) and undertakes activities to create outputs, the outputs may lead to some interim outcomes before having the desired societal impact. Your Results Map uses the same logic.

    Sadly, in my experience, very few public sector bodies seem to understand this basic concept let alone use it. However, where I have had the opportunity to apply the concept, it has led to some ‘light bulb moments’ for executives and senior managers and helped simplify the organisational performance measurement system for the better and enabled a clearer focus on improving services that enable customers, as part of society, to realise desired government outcomes.

    Like I said, it’s a fascinating topic but can also be exceedingly frustrating!

  2. Mike Davidge says:

    Hi Stacey, your White Paper is very well written and gets to the heart of a very tricky issue – how to measure outcomes. It should be required reading for every Government department!

Upcoming KPI Training


Our next PuMP Performance Measure Blueprint Workshops include:

>> Ottawa, Canada, 22-24 Oct 2019

>> Wellington, New Zealand, 4-6 Nov 2019

>> Johannesburg, South Africa, 12-15 Nov 2019

>> London, United Kingdom, 3-4 Dec 2019

Register for the next PuMP Blueprint Workshop near you

Stacey's Books


Prove It! How to Create a High-Performance Culture and Measurable Success, book by Stacey Barr

Order Prove It! here.

Practical Performance Measurement: Using the PuMP Blueprint For Fast, Easy, And Engaging KPIs, book by Stacey Barr

Order Practical Performance Measurement here.

Reprinting Articles


You are welcome to use articles from the Measure Up blog, with these requirements

Connect with Stacey


Haven’t found what you’re looking for? Want more information? Fill out the form below and I’ll get in touch with you as soon as possible.



*We respect your email privacy.
Level 54, 111 Eagle Street
Brisbane, Qld, 4000
Australia
Stacey Barr Pty Ltd
ACN: 129953635
Director: Stacey Barr