Why Surveys Are NOT Real Performance Measuresby Stacey Barr |
Surveys are one of the common methods we use in business to gather data. We have customer surveys, employee surveys, corporate image surveys, and surveys just about anytime we want to collect data from people. And often you’ll see these surveys listed in the KPI column in the business plan. BIG MISTAKE! Here’s why:
Surveys are really data collection processes and not performance measures at all. And the same goes for other data sources you might see listed in the business plan’s KPI column:
- ‘Customer Survey’
- ‘Occupational Health and Safety incident reports’
- ‘SAP system’
How exactly does a data source provide evidence that a goal has been achieved? Of course the data source provides data about the goal, but which data specifically? And how exactly should those data be analysed to answer the question of whether the goal is being achieved?
The above so-called measures are means of collecting data, but they do not provide the answers to our questions about performance.
Performance measures can certainly be BASED on the data that these processes collect. But the measures need to be clearly designed and defined before the right data can be collected.
When I worked in the Queensland Government Statistician’s Office as a survey statistician, I faced over and over the problem of data being collected without a clear idea of how they would be used. Our clients would first list all of the data they wanted to collect in a survey without making it clear what they would use those data for or what business questions they needed to answer.
Seeing data collection processes listed as performance measures is a symptom of not really knowing what the organisation’s actual performance measures are. Identifying a data collection process suggests that either we have only an inkling of what kind of data might relate to our goals or that we lack discipline in being clear and specific!
Measures should be designed before data is selected, for the same reasons that a house should be designed before building materials are selected. If we start with the data, we risk ending up with the wrong measures and missing out on getting the data we really need for the right measures. Or we risk ending up arguing about exactly how to use the data to decide whether our goals are being achieved or not. Neither of these outcomes gives us a true measure of performance.
Do you have any data collection processes that assemble lots of the wrong kinds of data?
Try to write down the business questions that you most need your data collection processes to answer, and then identify the form that the answers to these questions should take. From there, you can identify the analysis that can produce these answers and the data that this analysis would require. Once you’ve done this, you will be much closer to knowing what your performance measures really are (as well as the best data to collect to measure performance).
Stacey (and company): And I think the corollary to this is the fallacy of constructing a survey and its individual questions in such a way that it provides the answers the organization seeks as opposed to actionable data/measurements.
Love it Stacey! Especially “If we start with the data, we risk ending up with the wrong measures and missing out on getting the data we really need for the right measures.” I find that many managers start with designing a chart or survey, and others start with data…and in each case we have to work back to the driving need, the thing they need to know (and why they need to know it). Then identifying the data needed to build a measure becomes an easy (and logical) step.
So true Stacey. Am currently in the middle of one of those; collected data before asking the real questions. Now the collected data cannot answer the real questions. PHEW!!!
That’s the reason someone i shared my dilemma with shared your site with me.
Thanks for the useful articles.
Favor, I’m sorry to hear of your survey dilemma. Hopefully your future surveys will be much more targeted! Thanks for sharing.
Connect with Stacey
Haven’t found what you’re looking for? Want more information? Fill out the form below and I’ll get in touch with you as soon as possible.
167 Eagle Street,
Brisbane Qld 4000,
Director: Stacey Barr
Speak Your Mind