How to Measure Government Outcomes
by Stacey Barr |Government outcomes are particularly challenging to measure meaningfully, because they are intangible, long-term, and seldom directly impacted by a single department’s or agency’s actions.
The current common practices for choosing measures of performance are too limited to meet these challenges of creating good outcome measures for government departments and agencies.
CHALLENGE 1: Government outcomes are intangible.
Most government agencies exist to influence social stability, justice or welfare; not to produce widgets. Inherently, these social outcomes are softer and fluffier than the outcomes of non-government organisations. But this isn’t the reason that outcomes are hard to measure. If an outcome matters, we should be able to describe it clearly enough to observe, in some way, if it’s happening or not. But how most government agencies describe their outcomes is not with clear language that describes anything observable:
“The advancement of Australia’s international strategic, security and economic interests including through bilateral, regional and multilateral engagement on Australia Government foreign trade and international development policy priorities.” – Foreign Affairs and Trade
The common practice that limits our ability to meaningfully measure intangible outcomes is using vague ‘weasel words’ to articulate those outcomes. If we accepted the premise that some outcomes are too broad and intangible to measure directly, then we accept the premise that there will be no evidence of such outcomes if and when they happen. If there is no evidence of an outcome when it happens, no-one will notice if it’s happening or not. So why bother with all the effort to try and create that outcome in the first place?
To succeed at finding meaningful measures of intangible outcomes, we need to use plain language to articulate them. And we can’t start with questions like “How could we measure this?” Rather, we need a measure design process that starts with questions like “How would we recognise if this was happening or not?”
CHALLENGE 2: Government outcomes are long-term.
If it’s true that government agencies exist to influence social stability, justice and welfare, then it can’t be ignored that such things change over longer timeframes. Most government outcomes can’t be measured as frequently as sales quotas or revenue growth or tonnes of coal exported:
“Contribute to the improvement of the extent, condition and connectivity of Australia’s unique biodiversity and natural resources, consistent with national and international obligations, through protection of habitats and mitigation of threats to threatened species and ecological communities.” – Department of the Environment and Energy
The common practice that limits our ability to meaningfully measure long-term outcomes is limiting our measures to our corporate planning horizons. And this usually means that we measure only the strategic initiatives within those planning horizons. The above outcome isn’t measured directly, and only its Delivery Strategies have measures (and most are activity measures or milestones):
“Number of projects/contracts.
Project designs and contracts reflect the objectives of the Reef 2050 Plan and Reef Trust.
Project outcomes are fed into modelling and monitoring programs to inform reporting on Reef health.”
To succeed at finding meaningful measures of long-term outcomes, we can’t default to monitoring strategies and initiatives. If we do, we’ll struggle to ever learn which types of activity have the best impact on the outcome. In contrast, when we measure both the outcome and the results of our strategies and initiatives, we can test which work best, and then develop useful lead indicators to monitor more regularly to project changes in the outcome measure.
CHALLENGE 3: Government outcomes are systemically complex.
The role of government is more to influence than to directly control social outcomes. Government organisations often rely on collaboration with other public, private and nonprofit sector organisations for an outcome to be achieved. Government departments and agencies can struggle to isolate their contribution toward outcomes they can only partially influence, or ‘facilitate’:
“Facilitate jobs growth through policies and programs that promote fair, productive and safe workplaces, and facilitate the growth of small business.” – Department of Jobs and Small Business
The common practice that limits our ability to measure systemically complex outcomes is defining accountability as ‘hitting targets’. We know that the actions of a single department or agency cannot completely control if or how well a systemically complex outcome will happen. No one will accept being judged by a target they don’t have control of achieving. So, it’s no wonder that the above outcome is measured by such an operational measure as this:
“Percentage of claims processed within 16 weeks of receipt of an effective claim.”
To succeed at finding meaningful measures of systemically complex outcomes, we can’t waste time trying to isolate our own department’s or agency’s impact on those outcomes. Rather, we need to build and test a causal map of how we influence that outcome, and measure both the outcome and its drivers (causal factors) that we do control.
These challenges are the clues to how to measure government outcomes.
This white paper on How to Measure Government Outcomes goes into a lot more detail on dealing with these challenges, and offers a long list of techniques and resources to make the job easier.
Government outcomes can be meaningfully measured by working through the challenges of how intangible, long-term, and systemically complex those outcomes are.
[tweet this]
WATCH OR SHARE THE VIDEO:
Speak Your Mind
Join Measure Up
Sign up for the Measure Up newsletter and get free access to the "10 Secrets to KPI Success" online course and e-book.
Upcoming KPI Training
>> Africa, In-Person, Dubai UAE, 3-5 May 2023
>> Australia/NZ/Asia Pacific, In-Person, Wellington NZ, 9-11 May 2023
>> UK & Europe, Online Interactive, 22-26 May 2023
>> North America, Online Interactive, 29 May - 2 June 2023
Connect with Stacey
Haven’t found what you’re looking for? Want more information? Fill out the form below and I’ll get in touch with you as soon as possible.
167 Eagle Street,
Brisbane Qld 4000,
Australia
ACN: 129953635
Director: Stacey Barr
I find this whole area a fascinating subject. I have worked in quite a few different areas of the UK public sector and seen the struggles of senior managers trying to create KPIs for societal outcomes, which has led to totally ineffectual and wasteful ‘KPIs’ and scorecards being implemented. I knew this approach was wrong somehow, but it wasn’t until I heard Professor Mark Moore (Harvard) at a conference in 2006, discussing his concept of ‘Public Value’ and explaining the ‘Public Sector Value-Chain’, that it all clicked and became clear to me.
So, what clicked? Essentially, it was the notion that any public sector body should consider its ‘customers’, i.e. recipients of its services, as part of its whole value-chain because it’s the customers that ultimately realise the societal outcomes through their actions and behaviours – not the public sector body itself – and that might take years to occur. Therefore, public sector bodies can only measure those things it can control, i.e. people, processes and resources, which is where meaningful KPIs can be derived. The best any public sector body can do is to show a potential relationship and correlation – not cause, because other organisations might also be involved – between its activities/interventions and the longer-term measurement of changes in societal actions and behaviours.
As an example, a UK vehicle regulatory agency was trying to claim percentage KPIs of UK government road-safety and crime targets. I pointed out that the organisation’s main area of activity and resource allocation was on vehicle inspection but the agency’s parent transport department’s own figures showed that vehicle condition accounted for less than 2% of people killed or seriously injured in road traffic accidents, whereas driver behaviour accounted for over 45% of the total. The agency had only a relatively small budget for education and prevention activity so its claims for reducing the total number of people killed or seriously injured in road traffic accidents were inversely proportionate to its activities and resources. I have other examples too.
The same public value concept is built into the logic model that underpins the UK Government’s ‘Magenta Book’ guidance for evaluating the effectiveness of public policy. The model assumes that a public sector body takes inputs (including public money from taxes) and undertakes activities to create outputs, the outputs may lead to some interim outcomes before having the desired societal impact. Your Results Map uses the same logic.
Sadly, in my experience, very few public sector bodies seem to understand this basic concept let alone use it. However, where I have had the opportunity to apply the concept, it has led to some ‘light bulb moments’ for executives and senior managers and helped simplify the organisational performance measurement system for the better and enabled a clearer focus on improving services that enable customers, as part of society, to realise desired government outcomes.
Like I said, it’s a fascinating topic but can also be exceedingly frustrating!
Ian, thanks so much for contributing your experience with this. It’s encouraging that a similar conclusion is arrived at from different angles: yours with the public value chain concept and mine with the results map logic. You really nailed the problem that separates government agencies from most other organisation types, which is how indirect their contribution can feel to such long term and broad outcomes. But there is most definitely a way, and a worthwhile purpose, to keep helping government measure its impact.
Hi Stacey, your White Paper is very well written and gets to the heart of a very tricky issue – how to measure outcomes. It should be required reading for every Government department!
Thanks Mike, if that’s what other readers believe also, I can only hope they share it! I’ve always been passionate about finding ways to make it easier for government to measure impact so impact can be improved (and proven). It would make tax payers feel like they’re making a wise investment in their future, one that gets a return and doesn’t just feel like a cost.
Govt needs to be measured in the perspective of giving benefits to people.
Agreed, and benefiting the people they govern should be at the core of their strategic outcomes. When the outcomes reflect what matters, what matters gets measured.
[…] How to Measure Government Outcomes […]