Your organization is creating a survey to receive feedback on a particular topic or perhaps several areas of interest. The feedback you receive should help your organization make data-driven decisions that will support the achievement of organizational goals. You want to ensure the data you receive is quality and represents your audience so that you may make informed decisions. We discussed in previous blogs some do’s and don’ts to consider in a previous post, but now we will dive deeper into what is Survey Bias and how to prevent it.
What is survey bias?
“Bias” is defined as a “deviation of results or inferences from the truth, or processes leading to such a deviation.” So, Survey Bias occurs when your survey processes or results do not portray accurate information. One source reported 48 common types of biases found in questionnaires that can be sorted into three main problems: Question Design, Questionnaire Design, and Administration of Questionnaire.
1) Question Design
When writing questions on a survey, you want to ensure that the question asks exactly what it intends to ask. Participants can read or interpret a question differently, which could lead to a biased response. For example, if you ask, “Have you had issues taking time off for vacation?” the respondent could say “no”, but we do not know if they mean that they have not had any issues or if they have not asked off for vacation. You want to ensure the question is straightforward enough to generate a clear answer. So instead, you may ask, “Have you taken vacation while working here?” and then include a follow-up question, “If so, did you have any issues requesting the time off?” Using more than one question can limit confusion and reduce uncertainty.
Word Choices
Another aspect of question design to consider is your word choice. When asking questions, make sure to avoid technical jargon or vague words. For example, instead of asking, “Do you have a canine at home?” ask, “Do you have a pet dog?” Sometimes your participants will have a different first language than the survey language. Not using overly complicated words will help ensure your question is understandable. The more participants can understand and accurately answer your questions, the better quality your data will be.
Avoid Double-Barreled Questions
An additional type of question design to avoid is asking a “double-barreled” question, or a question that is two questions in one. For example, a survey question may ask the participant to share how much they agree with the following statement, “My organization provides training and networking opportunities.” The organization may only provide one of those resources, but the wording of the question does not allow the respondent to indicate whether both or only one of those resources are available. Ensuring your questions include only one question will reduce survey bias.
2) Survey Design
Survey design refers to how your survey looks to your participants. You would be surprised to know how much this can affect the quality of your data! Suppose your survey questions and responses are not spaced out adequately. In that case, participants may make errors when filling it out. For example, they may check the wrong boxes or circle the wrong answers. Ensure that your survey is spaced well. Additionally, listing responses vertically rather than horizontally has been suggested to have better outcomes for correct answers.
Order of Questions
Another area of survey design to look at is your order of questions. If you have a specific order for your questions, it could lead to question order bias. If you ask a participant, “Do you use a music streaming service?” and then ask them, “How do you feel about musicians losing money to music streamers?” you may bias how they respond to the second question. Asking a yes/no question and then having a follow-up question right after it suggests that the participant should answer it similarly to the yes/no question because participants want consistency in their answers. If you want to avoid this, randomize your questions. You can do this manually, but several survey platforms have options that will re-order your questions for you.
Survey Length
Lastly, when designing your survey, you want to avoid bias caused by a survey being too long. Most participants want to fill out a survey quickly. When a survey is long, participants may skip questions or answer questions incorrectly because they do not fully read the questions or the responses. Participants may also become frustrated and select similar answers for all questions. For example, they may choose “yes” regardless of whether that is the truthful answer, or they may decide to end the survey early.
Practice taking your survey and see how long it takes to complete. If possible, surveys should be shortened to five to eight minutes. Recent research suggests that participants are likelier to exit a survey early at the 10-minute mark (53% in 2022 vs. 23% in 2017). Keep the survey short and straightforward. You do not want people rushing through your survey, or all your work could be wasted.
3) Administration Bias
Administration bias includes how the survey is administered and who administers it. How you administer your survey can vary based on your target audience. You have already done your research and know what your target audience habits are.
How do I Administer my Survey?
Consider administering your survey online if your audience prefers to shop online. However, if they own a flip phone without internet access, you would want to pivot and offer a paper version. You want to utilize your target audience research and choose the best survey administration option for them.
Other things to consider are your audience’s language and literacy. Consider using a phone or in-person interview if your audience cannot read. Ensure you have ways to reach all of your population to collect the most survey data possible. This approach might mean you have multiple modes of delivery. In our upcoming blog, we interviewed a client who successfully used multiple administration tactics! Flexibility will ultimately help your organization in the long run by reaching more of your audience and improving your results to help your organization succeed!
Who will Administer the Survey?
When deciding how to administer your survey, you should also consider who will administer it. If you are using a person to collect survey data, how that person asks questions could affect results. You can reduce interviewer bias by instructing your interviewer to read the question exactly as it appears. You also want to ensure that the questions do not get additional interpretation from the interviewer that someone reading the survey alone would not receive. This could alter the understanding of questions. Have your interviewer be as objective as possible. This will again support your hard work in getting quality results!
What other survey biases am I missing?
While this is not an exhaustive list of survey biases, it is a great place to start. If you want more support and help, please get in touch with REC!
Sources
Choi, B. C., & Pak, A. W. (2005). A Catalog of Biases in Questionnaires. Preventing Chronic Disease, 2(1), A13. Retrieved from PubMed Central: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1323316/
Henning, J. (2023, Jan 31). The Ideal Survey Length. Retrieved from Researchscape: https://researchscape.com/blog/is-the-ideal-survey-length-20-minutes
Kowalska, K. (2023, March 14). Common Types of Survey Bias and How to Avoid Them. Retrieved from HubSpot: https://blog.hubspot.com/service/survey-bias-types
PortMA. (2018). How to Eliminate Interviewer Bias. Retrieved from PortMA: https://portma.com/resources/articles/how-to-eliminate-interviewer-bias/
Shtivelband, A. (2022). How to Beome a Data-Driven Organization. Retrieved from Research Evaluation Consulting: https://researchevaluationconsulting.com/how-to-become-a-data-driven-organization/
Qualtrics (2023). Survey bias types that researchers need to know about. Retrieved from Qualtrics.: https://www.qualtrics.com/experience-management/research/survey-bias/
Related Posts:
6 Solutions to Common Survey Mistakes