When KPI Data is Too Costly, Use Sampling

by Stacey Barr |

Sampling the data for your KPI not only takes less time, energy and cost, but often it’s more accurate.

A small slice of a pie chart made up of people. Credit: https://www.istockphoto.com/au/portfolio/gf_vector

Too many great KPIs or performance measures never see the light of day because of beliefs that the data will be too hard, too costly or too time-consuming to get. Not so!

I started my career as a survey statistician and one of the powerful tools we used was sampling. I used samples to measure all kinds of things, such as:

  • the profitability of broadacre agriculture farms across Australia
  • the economic impact of motorsports on the Gold Coast
  • the accuracy of inventory records in a railway

These measures would have been impossible to implement, with any level of reliability if at all, without sampling.

Samples, believe it or not, can produce more reliable data than populations.

It’s easy to appreciate that samples are a far less expensive data collection effort than collecting data from the entire population. But, ironically, measuring with samples is often much more reliable than collecting data from entire populations.

For example, trying to visit every broadacre farmer in Australia, to survey their profitability would have cost a fortune in travel. And there wouldn’t be the luxury to spend enough quality time with each farmer. I was often invited to join them for dinner in my visits, and it helped build the rapport to gather the data in a careful and respectful way.

And with the case of measuring inventory accuracy in the railway, a sample helped them discover that their inventory records were much more accurate than their previous data collection method suggested.

When we use samples, we can afford to spend more time on designing a data collection process that preserves the integrity of the data.

The problem with sampling is the misconceptions about how to do it.

Sampling is used by lots of people who aren’t qualified statisticians, and that can be just fine. But too often their approaches include some common mistakes in using samples, such as:

  • sample sizes need to be at least 10% of your population size (not true – sample sizes depends more on the variability in our measure values and on the level of precision we want)
  • sample sizes need to be maximised to give reliable enough results (not true – we can opt for smaller sample sizes to save costs if we’re happy to live with lower precision in our measure values)
  • volunteers or hand-picked samples are reliable (not true – even though they are convenient, for quantitative measures a randomised sample will have less bias)
  • important measures shouldn’t be based on samples (not true – even though we will have sampling error in the measure values, tracking changes over time can still be very accurate)
  • a qualified survey statistician or market researcher is always needed (not true – we can learn the basics of designing simple samples)

Don’t fall for these misconceptions. Be informed and you’ll save a tonne of time and energy (and decision-making error) that would otherwise be the result of a poorly designed sample.

How to know when you can try sampling for yourself.

It’s true that for complex measurement problems, we need the expertise of a qualified survey statistician (it’s a profession: I studied at university for 4 years to become one, then worked in the field for 3 years and still had lots to learn).

For example, the railway’s inventory accuracy measure that I created a sample design for was very complex. We first sampled inventory sites, using stratification to make sure we included a representative range of the highly variable sites. Then at each site we sampled inventory line items, again using stratification to make sure we got the full spectrum of highly variable inventory types. This makes sample size calculation challenging, and you need rather complex mathematical formulae to calculate the inventory accuracy measure values. Not for novices!

However, for simple sampling, you might like to try it yourself. If you’re into DIY, here are a few guides to do sampling the right way:

It should feel like a revelation that we can use proper sampling to save heaps of time and cost, and bring to life those measures that really matter. So, when you don’t think your ideal measures are feasible, think again!

Using sampling to get KPI data is like a blood test, it can be reliable and cost-effective, but only when it’s done right. [tweet this]

Upcoming KPI Training

>> Africa, Dubai UAE, 13-15 August 2024

>> UK & Europe, Online Interactive, 30 Sep - 4 Oct 2024

>> North America, Online Interactive, 7-11 October 2024

Register for the next PuMP Blueprint Workshop near you

Reprinting Articles

You are welcome to use articles from the Measure Up blog, with these requirements

Connect with Stacey

Haven’t found what you’re looking for? Want more information? Fill out the form below and I’ll get in touch with you as soon as possible.

*We respect your email privacy.
Suite 117 Level 14,
167 Eagle Street,
Brisbane Qld 4000,
Stacey Barr Pty Ltd
ACN: 129953635
Director: Stacey Barr