When KPI Data is Too Costly, Use Sampling

by Stacey Barr |

Sampling the data for your KPI not only takes less time, energy and cost, but often it’s more accurate.

A small slice of a pie chart made up of people. Credit: https://www.istockphoto.com/au/portfolio/gf_vector

Too many great KPIs or performance measures never see the light of day because of beliefs that the data will be too hard, too costly or too time-consuming to get. Not so!

I started my career as a survey statistician and one of the powerful tools we used was sampling. I used samples to measure all kinds of things, such as:

  • the profitability of broadacre agriculture farms across Australia
  • the economic impact of motorsports on the Gold Coast
  • the accuracy of inventory records in a railway

These measures would have been impossible to implement, with any level of reliability if at all, without sampling.

Samples, believe it or not, can produce more reliable data than populations.

It’s easy to appreciate that samples are a far less expensive data collection effort than collecting data from the entire population. But, ironically, measuring with samples is often much more reliable than collecting data from entire populations.

For example, trying to visit every broadacre farmer in Australia, to survey their profitability would have cost a fortune in travel. And there wouldn’t be the luxury to spend enough quality time with each farmer. I was often invited to join them for dinner in my visits, and it helped build the rapport to gather the data in a careful and respectful way.

And with the case of measuring inventory accuracy in the railway, a sample helped them discover that their inventory records were much more accurate than their previous data collection method suggested.

When we use samples, we can afford to spend more time on designing a data collection process that preserves the integrity of the data.

The problem with sampling is the misconceptions about how to do it.

Sampling is used by lots of people who aren’t qualified statisticians, and that can be just fine. But too often their approaches include some common mistakes in using samples, such as:

  • sample sizes need to be at least 10% of your population size (not true – sample sizes depends more on the variability in our measure values and on the level of precision we want)
  • sample sizes need to be maximised to give reliable enough results (not true – we can opt for smaller sample sizes to save costs if we’re happy to live with lower precision in our measure values)
  • volunteers or hand-picked samples are reliable (not true – even though they are convenient, for quantitative measures a randomised sample will have less bias)
  • important measures shouldn’t be based on samples (not true – even though we will have sampling error in the measure values, tracking changes over time can still be very accurate)
  • a qualified survey statistician or market researcher is always needed (not true – we can learn the basics of designing simple samples)

Don’t fall for these misconceptions. Be informed and you’ll save a tonne of time and energy (and decision-making error) that would otherwise be the result of a poorly designed sample.

How to know when you can try sampling for yourself.

It’s true that for complex measurement problems, we need the expertise of a qualified survey statistician (it’s a profession: I studied at university for 4 years to become one, then worked in the field for 3 years and still had lots to learn).

For example, the railway’s inventory accuracy measure that I created a sample design for was very complex. We first sampled inventory sites, using stratification to make sure we included a representative range of the highly variable sites. Then at each site we sampled inventory line items, again using stratification to make sure we got the full spectrum of highly variable inventory types. This makes sample size calculation challenging, and you need rather complex mathematical formulae to calculate the inventory accuracy measure values. Not for novices!

However, for simple sampling, you might like to try it yourself. If you’re into DIY, here are a few guides to do sampling the right way:

It should feel like a revelation that we can use proper sampling to save heaps of time and cost, and bring to life those measures that really matter. So, when you don’t think your ideal measures are feasible, think again!

Using sampling to get KPI data is like a blood test, it can be reliable and cost-effective, but only when it’s done right. [tweet this]

FacebooktwitterlinkedinmailFacebooktwitterlinkedinmail

Speak Your Mind

Your email address will not be published. Required fields are marked *

  1. Tania Werder says:

    As a statistician this makes me smile.
    I have come across some bias for surveying as well, but luckily I’m well placed to discuss it.

  2. Mike Davidge says:

    Great article Stacey. Would your advice on sample size differ for repeated samples as opposed to a one off sample such as a survey?

    • Stacey Barr says:

      Mike, it’s been too long since I was practicing survey design, but I think that for repeated samples, especially for a KPI or performance measure that is being calculated regularly, we can handle smaller sample sizes. How much smaller I am unsure and it would certainly depend on the nature and variability of what is being measured. The cool thing about using XmR charts, is that the survey error automatically becomes part of the natural process limits, and we can still detect signals of change.

Upcoming KPI Training


Our next PuMP Performance Measure Blueprint Workshops include:

>> Australia/NZ/Asia Pacific, Online Interactive, 4-5 & 9-11 Nov 2020

>> United Kingdom, Online Interactive (NHS specific), 6, 13, 20 & 27 Nov 2020

>> North America, Online Interactive, 30 Nov - 4 Dec 2020

>> United Kingdom, Online Interactive 8-9 & 15-16 Dec 2020

Register for the next PuMP Blueprint Workshop near you

Stacey's Books


Prove It! How to Create a High-Performance Culture and Measurable Success, book by Stacey Barr

Order Prove It! here.

Practical Performance Measurement: Using the PuMP Blueprint For Fast, Easy, And Engaging KPIs, book by Stacey Barr

Order Practical Performance Measurement here.

Reprinting Articles


You are welcome to use articles from the Measure Up blog, with these requirements

Connect with Stacey


Haven’t found what you’re looking for? Want more information? Fill out the form below and I’ll get in touch with you as soon as possible.



    *We respect your email privacy.
    Level 54, 111 Eagle Street
    Brisbane, Qld, 4000
    Australia
    Stacey Barr Pty Ltd
    ACN: 129953635
    Director: Stacey Barr