The intricacies of preparing a proper survey

In today’s digital world, it only takes 2 or 3 minutes to find out what parts of your product or service are frequented by your users, or what portion of your visitor base is represented by the ones that are contacting you. Accessing this type of digital data with close to %100 accuracy is very easy today, but you have to apply different research methods to find out why your visitors don’t communicate with you or why 80% of your users use only 20% of your product’s capabilities. One of the most flexible and practical research methods is undoubtedly the survey. However, the question is: How reliable are the results of your study?

There is no way to make sure that your survey results reflect one hundred percent truth. All you can do is to take into account the researcher-and-respondent-induced trends (bias) that lead to deviations in the results, in the survey preparation and dissemination phases, and to try to keep those deviations at the lowest possible level. I’ll list below the most common types of such variations, some examples, as well as tips that will help you avoid these trends as much as possible.

Response bias

You may encounter the response bias for many different reasons. It could be the survey content, or one of the target segments could be excluded due to their unavailability. There may be two main reasons for response bias, which is one of the respondent tendencies that cause the most significant distortions in the results.

One of them might be that your survey is on a sensitive matter (for some people). For instance, tax evaders will be reluctant to participate in a survey conducted by the Ministry of Finance. In this case, the results of the study will not include this group so it will not show the unbiased reality.

The survey medium may also lead to non-responsiveness. For example, illiterate people cannot respond to a written survey. Additionally, if the survey you share via e-mail is not displayed correctly on mobile phones, you will not be able to take into account the views of the people that manage their emails via mobile phones.

To minimize the response bias, make sure that your survey medium is in line with the target audience and if necessary, test it in advance. One of the most common methods used to collect data about sensitive subjects is to blend the questions about the topic with a set of questions that are entirely irrelevant.

Lack of knowledge

One of the most critical problems that can be detected is the researcher’s lack of knowledge regarding the audience and the survey subject. The survey is fruitless if the data collected does not overlap with the information required for market research. Multiple choice surveys present another set of problems if the researcher doesn’t include a common option the target audience is likely to pick.

For example, if you ask the audience who live in Turkey that “Which online shopping site do you use most often?” with options such as Amazon, eBay, Alibaba, Walmart and more (no choice originated from Turkey), you will not be able to find out which shopping sites are most popular among participants. The best way to avoid that is to conduct a preliminary survey including open-ended questions to a smaller group to get an idea of what options you should provide.

Respondent error

Two of the most common respondent-related deviations are incorrect or misleading answers. The first is that the respondent has incorrect answers due to not knowing the answer to the question. Other times, the respondent does not want to give an honest answer.

Everyone who responds to the survey may not have an answer to every question in it. For instance, a person working in a company’s customer support department may not know how many people work in the marketing department. Additionally, if you ask a person who doesn’t have a car about how many liters of gas are consumed per month, he/she might mark the option providing the smallest value. To prevent such cases from misleading the results of the survey, you must give an opportunity for the respondent to pass, such as ”I don’t know,“ ”I’m not sure” or “I have no idea.”

Nevertheless, there may be respondents that avoid picking such options out of goodwill and share the opinion of their alternate selves that would be addressed by the question. To prevent that, you should add screening questions to the survey. Considering the examples above; with questions such as “Which department do you work in?” and “Do you have a car?”, it will be possible to eliminate those who are likely to give the wrong answers to other questions.

On the other hand, the tendency of the respondent to give the correct answer is usually caused by social desirability. Respondents may choose to respond in ways that society (or researcher) will welcome, rather than responding to specific questions honestly. It is a type of deviation frequently encountered in intimate matters such as alcohol consumption, sexual orientation, and ethical or legal issues.

For example, a respondent might refrain from stating a negative opinion about disabled people to an interviewer in a wheelchair. Questions such as “Have you ever stolen?” that are at risk of causing legal consequences may not always be answered honestly. To minimize this tendency, an environment should be created to make the respondent feel safe, and questions should be formulated as impartially and in the most understandable language possible.

The survey-related deviations

In particular, in online surveys, all responses may not reflect the truth, since no interviewer is able either to help if the question is not understood correctly or to follow how respondents react to the questions. This may be due to many reasons such as how the questions are asked, the answer method such as scale, multiple choice or free text, how the options are provided, the visual structure of the survey and the question sequence.

The formulation of questions and the formulation of choices should be as neutral as possible and consistent throughout the survey. Assuming a scale set is labeled as “Good – No idea – Bad”, if another one in the same survey is labeled as “I like – No idea – I don’t like”, the neutral rating in the first sample and the emotional rating in the second sample may not be comparable, even though they are evaluated on the same 3-point scale.

If you have prepared a lengthy questionnaire, the details such as how many pages the survey consists of and how many questions are displayed on each page will affect the completion rate of the survey. Positioning difficult or complicated questions at the end of the study may cause fatigue-induced false responses. While there are five questions on the first two pages of a 5-page survey, if participants encounter 15 questions on the third page, the risk that participants leave the survey on this page increases.

Even if you have prepared your survey considering all of the reasons for possible deviations above, you should test it in a smaller environment before starting the research. Even then it’s important to have feedback from people with experience in survey/market research. Because, just as not having enough knowledge about the subject of the survey, being an expert in the subject of the questionnaire may raise the possibility that your questions and response options do not coincide with the facts.