Why COVID-19 Needs Proper Performance Measurement

by Stacey Barr

Written by Trevor Hobbs, Assistant to the Town Administrator, for a municipality in Tennessee, USA. Edited by Stacey Barr.

Why aren’t existing metrics bringing world leaders to a consensus about whether the COVID-19 pandemic is improving enough, or not?

Which COVID-19 metrics are best? Credit: ake1150sb

This is a question I’m sure lots of PuMP users have asked themselves during the last few months. But with all the confusion and disagreement about what data is telling health officials and policy makers world-wide, I wonder if that question doesn’t need revisiting.

The COVID-19 pandemic, though a truly tragic and serious situation for us all, offers an interesting case study for the value and purpose of the PuMP methodology.

Several months ago, when most of the world hadn’t yet been touched by the Coronavirus, health officials and leaders focused mostly on getting prepared; making sure that we had enough face masks, hand sanitizer, and hospital beds to handle the needs of our communities. Communities stocked up on essentials so they could stay home. Businesses scrambled to get their employees set up to work from home. We did all of these things in order to survive the pandemic and hold onto as much normalcy as possible.

Now, months later, face mask orders, social distancing, limited capacity at stores and restaurants, limits on public gatherings, are all part of our new normal. Though initially considered temporary tactics to slow and prevent the spread of the Coronavirus, we are now wondering when these restrictions will no longer be necessary.

These questions offer us a great opportunity to take a strategic approach to public health amid a global pandemic.

How do we know which are the right COVID-19 metrics?

While I’m partial to it, I can’t help but think that PuMP has a lot to offer to those addressing COVID-19 and making decisions about what to do.

PuMP starts with clarifying our strategic plan, not just creating performance measures. We start by asking ourselves what success looks like: How will we know when our result has been achieved? What will we observe (see, feel, taste, smell, hear, experience)?

We need to start by defining our results, before we start discussing what we are going to DO about anything. If we don’t have a clear definition of our results, we can’t achieve them, and we might as well do nothing at all.

What are the results that matter for COVID-19?

We invite you, the reader, to share your thoughts (in the comments below this post, or email Stacey). To prime your thinking, here are some examples:

  • The spread of COVID-19 slows and stops
  • The global population has immunity to COVID-19
  • People who contract COVID-19 fully recover
  • Companies, businesses and organisations don’t go out of business due to COVID-19 restrictions
  • Everyone who lost employment due to COVID-19 finds employment again
  • [Email Stacey with additional results you think matter]

Are COVID-19 outcomes too intangible to measure?

If we accepted the premise that some outcomes are too broad and intangible to measure directly, then we accept the premise that there will be no evidence of such outcomes if and when they happen. If there is no evidence of an outcome when it happens, no-one will notice if it’s happening or not.

So why bother with all the effort to try and create that outcome in the first place?

The question then becomes, how will we know when the effects of the COVID-19 pandemic are over? And along the way, how will we know that our strategies are having the intended impact on the situation?

Which metrics are being used to monitor COVID-19?

There are many metrics being used in an attempt to monitor the direct and indirect effects of COVID-19, and these are only a few:

  • Daily New Cases
  • Daily New Deaths
  • Total Cases per Million Population
  • Total Deaths per Million Population
  • Total New Tests per Million Population
  • Transmission Rate
  • Hospital Capacity Utilisation
  • Ventilators Used
  • Unemployment Rate
  • Consumer Price Index
  • Retail Sales
  • Domestic Violence
  • [Email Stacey with additional metrics you know of]

If we know all this, why is the pandemic still wreaking havoc?

Existing COVID metrics aren’t helping.

A very real and sadly comical interview took place a few months ago between Axios correspondent Jonathan Swan and United States President Donald Trump. They discussed data reporting various information for different countries.

At approximately 12:00 minutes in, they begin discussing the COVID-19 reports. Then at approximately 13:50, they begin to discuss whether it is more appropriate to measure the number of deaths by country as a proportion of positive cases, or as a proportion of the country’s population. While those are two different performance measures that support two separate results, it did get me thinking about what results are implied if one measure is used over another, and why has population been chosen as the default basis for measurement.

With PuMP we understand that it is absolutely essential that we determine our intended results first, before we ever attempt to design a performance measure to support it. If we don’t, we may very likely spend most of our time disputing what our data is telling us.

At about 15:50 they start discussing and disagreeing about the changes they interpret from the data. Can someone please use a Smart Chart already?! I think there’s a lot of noise in the reports circulating, but none of the reports are showing whether statistically significant change has occurred.

Is the COVID-19 pandemic improving?

Trend lines are great for certain purposes, but when we want to track change over time, particularly when trying to determine whether our initiatives are having the intended impact on our results, is to use a Smart Chart. Smart Charts allow us to visualize whether central lines have shifted, identify outliers, and see when our normal range of variation has shifted.

COVID-19 Smart Chart example - https://staceybarr.com/images/COVID-19GrowthFactorXchart.jpg

Many public health authorities have established benchmarks for the kinds of trends they want to see in order to loosen, retire, or tighten restrictions. But few of them come close to capturing the essence of a performance measure. Without a clear vision for success, data can only tell us so much, and we will struggle to answer this question.

Luckily, PuMP offers us solutions to these problems.

What do we do now?

With great power comes great responsibility.

Those who are blessed with the knowledge of PuMP and the foundational principles of Performance Measurement have the responsibility to share their perspective with those who need it. Leaders and Public Health officials are working hard and doing their best to stop the spread of the Coronavirus, and they need the help of Performance Measurement specialists to construct a framework for their strategic plan. The Coronavirus may be outside of our control, but there is a lot within our circle of influence.

Part II of this article will demonstrate how PuMP can help to solve the problems we face with managing the impacts of COVID-19, by claryifying, measuring and interpreting some of those impacts.

Which COVID-19 metrics are best? We can’t answer that until we know the specific pandemic results we’re trying to improve.
[tweet this]

Speak Your Mind

Your email address will not be published. Required fields are marked *

  1. Tom Randle says:

    One thing that’s really struck me about the epidemic is how hard it is to quantify and make comparisons on.

    I’d thoroughly recommend listening to the More or Less coverage they’ve been covering a lot of the challenges: https://www.bbc.co.uk/programmes/p02nrss1/episodes/downloads

    Each country measures things differently and even a seemingly simple measure like deaths per million is complicated by different definitions. Some countries weren’t counting non-hospital deaths. At one point in the UK if you died any time after having covid, even if was being hit by a bus, you would have been registered as a covid death.

    I’ve found it bizarre that when the press present daily deaths (at least in the UK) it’s the number of new registered deaths for that day. Not the number of people who died that day. Obviously the latter isn’t possible but the former isn’t a true measure of the performance that day.

    Excess deaths per million seems to be what the experts like David Spiegelhalter
    say will be the best comparison after the fact – but it’s a slow measure to compute. Even that hides huge complexity as we care whether the deaths are directly because of Covid or knock on things like worse Cancer treatment.

    Recently some of the rise in cases has been attributed to improved testing.

    One issue covered in More or Less was the no. of tests being carried out. Because the Government put a target on it they we counting it a way that made it look as high as possible – number of tests sent out rather than carried out and counting tests rather than number of people testing it.

    I think the statistician’s view is that it’ll only be long after the fact all the impacts of Covid will be quantifiable. And even then success will be debatable based on value given to the economy versus saving lives.

    The most useful measures right now will be the ones being used by the teams carrying out specific initiatives. For governments the most useful metric of the moment is cases requiring hospital treatment – and anything that predicts that.

    • Stacey Barr says:

      You’re right, Tom, that comparisons of COVID metrics is just as convoluted at comparisons of any other metric we’ve used in the past to make judgments on (like in health and education). And I also agree that in hindsight we’ll know with more certainty what we’re currently grappling to know with measurement. But the complexity of data integrity, data timeliness, measure calculation and target setting is a complexity we’ve always had with measurement, and always will. So, I want to believe that this is important enough to find a better way. A better way to measure what’s useful, a better way to interpret those measures to inform action, and a better way to learn what gets us on top of COVID.

  2. As a facility manager, I would normally turn to PuMP to design and install an instrument to assess whether a process change brought an intended result. Otherwise, we’re flying blind.
    Circumstances with the present pandemic are complicated, with various factors outside of our control, but PuMP could be of use.
    For facility managers, COVID involves defense in depth. That is, conscientous application of various independent means together, each only somewhat effective on its own. A suitable facility operations goal is not to infect occupants. In workplaces with controlled admission, we may be able to guess when a COVID instance began in our facility, but even then, outside factors can interfere. Do we have a controlled experiment? No, but by deriving measurement and interpretation using the PuMP framework, we do obtain an ongoing measurement to conduct after critical consideration of the threat, means deployed in mitigation, and ways of addressing whether coming to our facility can be an undue hazard. This has value. Given time and information, adjustment for externals is feasible, of course, but our goal of not making matters worse might be addressable soon and worth the trouble to support top level decision making and clarifying staff thinking about actions and tradeoffs.

  3. Martien Dingemans says:

    Hi Stacey,

    Good initiative, however as a Pump professional, we should start asking the right Pump questions to the medical professionals instead:

    ‘What is the goal?’ and ‘What would you see or experience when your goal has been achieved?’

    The risk of your initiative is that we as Pump ambassadors will be defining the goal, which is not within our area of competence.

    Without the goal given by the experts, I think this initiative is very risky and can even backfire on Pump.

    Regards, Martien

    • Stacey Barr says:

      Martien, of course you’re right. The owners of the goals should always define the goals and express them measurably. It is risky that we try building a model using PuMP, not being the COVID experts. But how will those COVID experts see what’s possible, if not by a ‘strawman’ that shows the way?

  4. Greg says:

    Hi. Not interested in covid reporting its too reactive and forecasts shouldn’t be relied on.

    • Stacey Barr says:

      But it is something we need to understand; we need to know if and by how much our interventions have an effect on the impact of COVID. Surely it can’t just be ignored, when so many people are suffering and dying from it? COVID won’t be the last global issue that shuts down the world. And you’re right – the way we’re currently measuring it IS too reactive and done so poorly that it creates feelings like you have, Greg. Shouldn’t we try harder and measure smarter so we can learn how to handle such things better?

Upcoming KPI Training

>> North America, Online Interactive, 6-10 February 2023 - SOLD OUT

>> Africa, Online Interactive, 27 February - 3 March 2023

>> UK & Europe, Online Interactive, 27 February - 3 March 2023

>> North America, In-Person, Calgary AB, 7-9 March 2023

>> Australia/NZ/Asia Pacific, Online Interactive, 15-16 & 20-22 March 2023

>> Africa, In-Person, Dubai UAE, 3-5 May 2023

>> Australia/NZ/Asia Pacific, In-Person, Wellington NZ, 9-11 May 2023

>> UK & Europe, Online Interactive, 22-26 May 2023

>> North America, Online Interactive, 29 May - 2 June 2023

Register for the next PuMP Blueprint Workshop near you

Reprinting Articles

You are welcome to use articles from the Measure Up blog, with these requirements

Connect with Stacey

Haven’t found what you’re looking for? Want more information? Fill out the form below and I’ll get in touch with you as soon as possible.

    *We respect your email privacy.
    Suite 117 Level 14,
    167 Eagle Street,
    Brisbane Qld 4000,
    Stacey Barr Pty Ltd
    ACN: 129953635
    Director: Stacey Barr