Are you accidentally adopting these 8 ways to make your survey useless? Find out how to create successful surveys.

Surveys are one of the most popular methods for collecting data about people’s attitudes and beliefs, such as customer perception of value, corporate image and employee satisfaction. And probably because of this popularity, many surveys lack the validity they require to provide useful and usable information. A survey is not a questionnaire plus an envelope plus a stamp.

Surveys must be designed with the same kind of rigour that scientists design experiments – but don’t freak out! Surveys can be far less complex and intricate and dire than scientific experiments. It’s just that if you want to draw conclusions from your survey data, that data must be objective, and objectivity is the product of rigour.

A survey will produce objective data if it is designed with all the attributes of data integrity in mind:

• the data is relevant to the questions you are collecting it to answer
• the data gives you a representative picture of the way things are
• the data is reliable enough to be in your “ball park” of precision
• the data is readable, and therefore able to be analysed and turned into the answers to your questions
• the data is a realistic investment – the cost of collecting it is sufficiently less than the value you create from using it

The steps you take in designing a survey are paramount to the degree of integrity of the data you end up with. So, do what you can to avoid making the 8 mistakes discussed here (unless you in fact want to invest in a useless survey):

Mistake 1: Don’t bother defining your purpose and objectives.

The first way to make sure you waste the time and money you invest in your survey is to avoid at all costs defining or stating the purpose or reason for conducting the survey.

DO: Write down the primary reason you’re doing the survey, then list the exact objectives you need to meet.

Mistake 2: Leave your target population vaguely described.

Isn’t it easier just to survey everyone in whatever list you already conveniently have? Or just those that are the most obvious? Or those that you “choose” to survey? It sure is.

DO: Clearly define the characteristics of who is included in your survey population. And, just as importantly, who isn’t.

Mistake 3: Mailing the questionnaire saves you heaps.

It couldn’t be any easier or cheaper to render your survey data useless than to exploit the biasing influence that mail out surveys offer. Mail out surveys are usually posted or emailed to respondents, and they are thus very inexpensive.

DO: Adopt the best surveying techniques you can afford, to reduce bias and maximise response rates.

Mistake 4: Get everyone to contribute questions to design the questionnaire.

There are many excellent ways to ruin your survey simply in the questionnaire design. Ask too many questions. Word the questions poorly. Sequence the question illogically. Give answer options that don’t make sense.

DO: Ask respondents only the fewest set of questions that are useful (not just interesting) to meet the objectives.

Mistake 5: Send the questionnaire to whoever is convenient.

To avoid making wise decisions about sample size (it’s going to take a little bit of effort, after all), you can instead adopt one of the following dubious approaches to decide who to send the survey to: quota sample, volunteer sample, targeted selection sample.

DO: Calculate the sample size you need, then select that sample randomly to minimise bias and maximise reliability.

Mistake 6: Assume it’s obvious how to implement the survey.

The way a survey is conducted has a significant impact on the integrity of data you end up with. To flush this integrity down the toilet, you can fail to specify exactly how the sample selection, survey administration and data collation and analysis are to be performed. And just let it happen “naturally” and “conveniently”.

DO: Design and document the steps and standards for how to conduct the survey and capture the data.

Mistake 7: Set a deadline that leaves no time for testing.

If you want to save time and don’t care about the usefulness of your survey data, then don’t bother testing your survey questionnaire with a small number of your target population to fill it out “as if for real”.

DO: “Pilot test” the questions, the survey procedures, and the usefulness of the data collected in meeting the survey objectives.

Mistake 8: Treat the data as though it were perfect.

There’s more you can do to warrant your survey data useless. Yes, you can pretend it is perfect when you analyse it, then take its analysis as gospel. Firstly, don’t do anything to make it analysis-ready, like cleaning or coding the raw data. Secondly, analyse the survey data using just averages or percentages, without calculations of the error in those estimates.

DO: Calculate confidence intervals for the estimates calculated from the survey data. And report them with the survey analysis.

Doing surveys well means adopting a few simple principles.

It really isn’t too hard at all to produce a survey that produces nothing of value. In part it is because most people have really no idea that survey design and implementation is a science, and there are scientists that specialize in how to do them well. It isn’t rocket science; it’s the field of statistics. Doing surveys well means adopting a few simple principles:

1. have a clear and concise purpose and objectives
2. design the survey questions to produce the data that will answer your objectives
3. design your sample size based on your reliability requirements (not by taking 10% of the population)
4. select your sample randomly
5. define the steps in the survey implementation process
6. pilot test the survey, questionnaire, and analysis of data
7. calculate and report the margins of error of your survey estimates

 

It’s too easy to render a survey-based KPI useless by making mistakes in the survey design. If you don’t have training in survey design, get help.
[tweet this]

DOWNLOAD:

Read more about the 8 mistakes that are commonly made in survey design and learn how to
avoid them, by downloading the full white paper “Is Your Survey Useless?” here.