5 Examples of Biased Survey Questions and Why You Should Avoid Them - Trustmary (2022)

5 Examples of Biased Survey Questions and Why You Should Avoid Them

Biased survey questions are set up in a way that either

  1. Lead the respondent intentionally down a path to a certain answer, or
  2. Are phrased in a manner that can be confusing to them, leading to unclear responses.

Whilst it’s rarer these days to find organizations intentionally trying to skew results, it’s still common for companies to have survey bias. This is simply because they don’t have the right skills and experience to put together a good quality survey.

Furthermore, survey bias is dying out also largely because customers are more aware of unethical practices and companies will be quickly called out for it. With the current cancel culture, it’s best to avoid

Asking biased survey questions can actually be very dangerous for the company. Based on the survey answer, they’ll jump to the wrong conclusions, take incorrect actions and take away the wrong insights from the responses.

You’ll want to avoid biased survey questions at all costs, it’s vital that all feedback you get is impartial and honest.

By reading this article, you’ll learn:

  • Different types of biased survey questions
  • Their different consequences for your business (whether you’ve made the survey biased intentionally or not!)
  • 5 Different types of surveys questions to avoid
  • Examples explaining why they’re bad

1. Leading questions

Leading questions are the most obvious examples of bias to spot, as they make it very clear that there is a “correct” answer the question is leading you towards. These will always result in false information as the respondent was never given the option for an honest response to begin with.

Examples of leading questions

“How amazing was your experience with our customer service team?”

You can see this question is set up in a way that you’ve assumed already you thought the customer service team was amazing, you’ve left no room for another answer. The customer is now obliged to rate the customer service team on a scale of how amazing they were.

“What problems did you have with the launch of this new product?”

Again, this question assumes that there was something wrong in the first place and will have the customer looking for problems in their answer.

“Would you be worried if we discontinued this product line?”

This example is using emotional language to lead a customer, rather than making assumptions as the first two questions. It suggests to the respondent that they should be worried, simply because the emotional language is used.

Consequences of leading questions

  • Your survey responses will be significantly skewed
  • You won’t gain any new results or insights
  • Customers may think you’re deliberately trying to lead them into a certain response which in turn could cause them to leave a negative review or accuse you of leading surveys on social media.

What to do instead

Make sure the questions you give the customer have options, they need to be able to choose objectively. For example, instead of asking how amazing a customer experience was, you should ask:

(Video) Why Investigations Should Play a More Significant Role in Safety Management -- Ep. 16

5 Examples of Biased Survey Questions and Why You Should Avoid Them - Trustmary (1)

“On a scale of 1-10 how satisfied were you with our customer service team”

This doesn’t lead them into a response and gives the customer a chance to provide a useful and easily measurable rating.

2. Vague or ambiguous questions

This is a type of question many organizations can fall victim to without realizing it. On the surface these questions may look honest and harmless enough, but their vague nature can actually do more harm than good, confusing a customer into a poor response.

Examples of vague survey questions

“How do we compare to our competitors?

This question for example is far too broad. Maybe your customers have never actually used your competitor’s products so can’t say?

Or maybe it’s not something your customers have ever thought about before and they decide to start researching your competitors? You also haven’t given a benchmark to compare against, do you mean your product? Level of customer service? Price? You’re leaving the choice entirely in the customers hands.

“Do you think your family members would like product X”

Here the mistake is using language like “Think” which can get different reactions from different people. Other similar words would be examples like “Feel” or “Expect”. You’re asking the customer to give broad, subjective answers and respond emotionally. Your customers might also have no idea how their family members would react to the product and probably won’t be inclined to go and ask them to find out, and instead will abandon your survey.

Consequences

  • If you’re not specific enough with your question, chances are that the range of answers you get will be so wildly different to each other it’ll make any data you collect useless. Vague questions spark different thoughts for different people, you want to keep you questions focused and specific.
  • The customer won’t always interpret the question the way you expect them to, skewing your final results.

What to do instead

Be much more specific with your questions and get to the point. Instead of asking your customers if they think they might recommend something instead ask:

5 Examples of Biased Survey Questions and Why You Should Avoid Them - Trustmary (2)

“On a scale of 1-10 how likely would you be to recommend our product to others”

This takes the ambiguity away and gets them focusing on one topic, again with an easily measurable response.

3. Double barreled questions

These questions usually ask the customer to provide an opinion on two topics (usually loosely related), but only provides an opportunity for one response.

These can be another example of organizations making a common, unintentional mistake, in a desire to know more information they end up doubling up their questions.

Examples

“How satisfied are you with our customer service and aftercare?”

While at first glance these topics look related, they’re actually two very different topics altogether.

The customer may have had excellent customer service but found their aftercare package terrible. The question doesn’t allow for them to differentiate between the two and give two opinions.

”Would you like our product to be cheaper and more value for money?”

Again, as with the above example, these are really two separate topics. “Value for money” can mean different things to different people.

While some might find it synonymous with “cheap”, others will expect value to represent a high cost but a more high-quality product. Again, you’ll be skewing your answers based on people’s objective interpretation of the statement.

Consequences

You’ll confuse your customers and get skewed results. The worst case scenario would be that you end up annoying them by not giving them a chance to respond separately.

What to do instead

Never combine two questions in one. The simple solution is to split the questions into two:

“How satisfied were you with our customer service?”

“How satisfied were you with our customer aftercare?

4. Absolute Questions

These questions can bias your respondents’ choices by forcing them into an absolute categorical response when they might not have one.

They use words like

  • Never
  • Always
  • All

You’re essentially asking the customer to be 100% certain about something.

Examples of Biased Absolute Questions

“Do you always use product X for your cleaning needs?”

The problem with this is that the answer will be no.

The chances of someone using your product 100% of the time are going to be very slim and will reflect poorly on your survey results.

“Tell us why you have never purchased our product”

This question not only isolates the respondent by singling them out as not buying your product, it also comes off as aggressive and pushy. You’re leaving your customer very little room to maneuver.

Consequences

These are usually too inflexible to be used in a survey, if you don’t give the respondents the chance to opt out of a question they’ll either be forced to answer with something that doesn’t apply to them (Skewing your results again) or abandon the survey.

If the questions themselves sound aggressive thanks to the absolute language used, the customer might in turn be encouraged to respond aggressively themselves. This results in getting plenty of negative feedback.

What to do instead

Never use absolutes (and yes, we’re aware of the irony of using an absolute to get our point across!).

Use specific options instead so customers have a choice:

“What discourages you from purchasing our product?

  • The price is too high
  • The quality is not good
  • I wasn’t aware of the product
  • Other
  • I don’t want to answer

5. Acquiescence Bias Questions

These questions are usually a binary yes/no choice.

The questions are worded so that the respondent is more likely to respond positively to every question in the survey, simply clicking “yes” or “agree” to speed through the survey even if they don’t completely agree with the statement.

People are often more likely to respond positively when only two options are presented.

Examples of Biased Survey Questions

“Is our level of customer service satisfactory? – Yes/No”

It’s much more likely a respondent will find your customer service somewhere in between the two extremes of yes and no, but because they’ve probably had more good experiences than bad, they’ll simply click “yes” and move on. There’s no option to add any nuance or detail to the question.

“Our product is easy to use” – Agree/Disagree”

As above, it’s much more likely the customer was able to get your product working and didn’t experience any major problems, again, this will make them naturally hit agree. Only those customers that have had a completely terrible experience or found the product totally unusable are going to disagree.

Consequences

You’ll ultimately learn nothing of value from these types of questions.

Your entire survey will essentially be a waste of time and money. You’ll end up with an “Everything seems fine” scenario, because no-one is providing you with any genuinely valuable feedback as they’re too busy agreeing with all of your statements.

What to Do Instead

Another simple one, just don’t use yes/no or agree/disagree questions. Always offer multiple options for the customer to select from or provide an open form text box.

While we’ve given you a few tips on avoiding biased questions, if you’d like to know more about

  • writing effective survey questions,
  • or how to craft the best survey subject lines

We’ve written guidesthat cover some key dos and don’ts.

The key thing is to keep the questions focused on one specific topic, never lead your customers on and keep things simple.

Once you get positive responses, remember to respond to them!

Next Steps

First of all, read our ultimate guide to surveys that covers everything from best survey methods to survey data analysis.

Below you can see an embedded survey that’s done with Trustmary’s own survey tool and drag and drop survey-maker.

3 Techniques for Lead Generation in Digital Marketing
7 Sales Pitch Presentation Mistakes You Should Avoid
Satokausikalenteri: 20% More Sales With Video Reviews
Use Automated Emails to Measure Customer Satisfaction
How to Add Google Reviews to Website (for Dummies)

FAQs

Which is an example of a biased survey question? ›

Leading questions is the most typical example of a biased survey question. They lead the respondents towards a certain answer. The questions are phrased such that the respondents are forced to give their answers in favor of or against a subject.

What type of questions should be avoided in a questionnaire? ›

Avoid loaded questions

Loaded questions are questions written in a way that forces the respondent into an answer that doesn't accurately reflect his or her opinion or situation. This key survey mistake will throw off your survey respondents and is one of the leading contributors to respondents abandoning surveys.

How can biased surveys be avoided? ›

Here are some good tips for reducing response bias:
  1. Ask neutrally worded questions.
  2. Make sure your answer options are not leading.
  3. Make your survey anonymous.
  4. Remove your brand as this can tip off your respondents on how you wish for them to answer.
26 Aug 2022

What is a biased question in research? ›

A biased question is a question that is phrased or expressed in such a way that it influences the respondent's opinion. Such questions may provide information that leads a respondent to consider the subject in a specific way.

What are the 5 examples of bias? ›

Reduce your unconscious bias by learning more about the five largest types of bias:
  • Similarity Bias. Similarity bias means that we often prefer things that are like us over things that are different than us. ...
  • Expedience Bias. ...
  • Experience Bias. ...
  • Distance Bias. ...
  • Safety Bias.
25 Feb 2021

What are your top 5 biases? ›

The 5 Biggest Biases That Affect Decision-Making
  • Similarity Bias — We prefer what is like us over what is different. ...
  • Expedience Bias — We prefer to act quickly rather than take time. ...
  • Experience Bias — We take our perception to be the objective truth. ...
  • Distance Bias — We prefer what's closer over what's farther away.
2 Aug 2022

What is a bad survey question? ›

A bad survey question is one that prevents respondents from providing objective answers in research. These questions usually contain several biases that make it difficult for survey respondents to communicate their true thoughts, preferences, and experiences.

Why should we avoid leading questions in a survey? ›

Leading questions result in biased or false answers, as respondents are prone to simply mimic the words of the interviewer. How we word these questions may affect the user response and also may give them extra clues about the interface.

Why should we avoid open-ended questions in the survey questionnaire? ›

Most People Don't Answer Them: The greatest reason to avoid open-ended questions in your survey is because few people take the time to answer them unless absolutely necessary. Surveys are generally an inconvenience to most survey respondents, no matter the incentive.

What are the 5 tips to avoid biases in writing? ›

Tips for Reducing Bias in Your Writing
  • Avoid generalizations. Written statements should not state or imply that all or none (or always or never) assertions. ...
  • Provide evidence. Support your assertions with resources and research. ...
  • Be objective. ...
  • Describe people at appropriate levels of specificity. ...
  • Use language sensitively.
13 Oct 2022

How do you avoid bias examples? ›

Avoiding Bias
  1. Use Third Person Point of View. ...
  2. Choose Words Carefully When Making Comparisons. ...
  3. Be Specific When Writing About People. ...
  4. Use People First Language. ...
  5. Use Gender Neutral Phrases. ...
  6. Use Inclusive or Preferred Personal Pronouns. ...
  7. Check for Gender Assumptions.

How can you avoid a biased sample? ›

Use Simple Random Sampling

One of the most effective methods that can be used by researchers to avoid sampling bias is simple random sampling, in which samples are chosen strictly by chance. This provides equal odds for every member of the population to be chosen as a participant in the study at hand.

What is an example of a biased sample? ›

For example, a survey of high school students to measure teenage use of illegal drugs will be a biased sample because it does not include home-schooled students or dropouts. A sample is also biased if certain members are underrepresented or overrepresented relative to others in the population.

What are the 3 types of bias examples? ›

Confirmation bias, sampling bias, and brilliance bias are three examples that can affect our ability to critically engage with information.

What are the 8 common types of bias? ›

Here are eight common biases affecting your decision making and what you can do to master them.
  • Survivorship bias. Paying too much attention to successes, while glossing over failures. ...
  • Confirmation bias. ...
  • The IKEA effect. ...
  • Anchoring bias. ...
  • Overconfidence biases. ...
  • Planning fallacy. ...
  • Availability heuristic. ...
  • Progress bias.

What are the 7 form of bias? ›

By ignoring prejudice, racism, discrimination, exploitation, oppression, sexism, and inter-group conflict, we deny students the information they need to recognize, understand, and perhaps some day conquer societal problems.

What are the 6 types of bias? ›

We've handpicked six common types of bias and share our tips to overcome them:
  • Confirmation bias. Confirmation bias is when data is analysed and interpreted to confirm hypotheses and expectations. ...
  • The Hawthorne effect. ...
  • Implicit bias. ...
  • Expectancy bias. ...
  • Leading Language. ...
  • Recall bias.

What are the 4 types of bias? ›

4 leading types of bias in research and how to prevent them from impacting your survey
  • Asking the wrong questions. It's impossible to get the right answers if you ask the wrong questions. ...
  • Surveying the wrong people. ...
  • Using an exclusive collection method. ...
  • Misinterpreting your data results.

What are the 4 biases? ›

Remembering and running through the first letter of each of these forms of unconscious bias – Confirmation, Insider, Attribution, and Overconfidence – is one way of building greater awareness and ensuring that neither we nor our organizations fall victim to such bias ourselves.

What are the 4 types of bias in statistics? ›

There are several types of bias in statistics, including confirmation bias, selection bias, outlier bias, funding bias, omitted variable bias, and survivorship bias.

How do you know if a survey is biased? ›

Bias Survey Question Examples
  • Leading Questions. A leading question “leads” the respondent toward a “correct” answer by wording questions in a way that sways readers to one side. ...
  • Loaded Questions. ...
  • Double-Barreled Questions. ...
  • Absolute Questions. ...
  • Unclear Questions. ...
  • Multiple Answer Questions.
30 May 2022

How do you avoid leading questions in a survey? ›

7 Tips to Avoid Leading Survey Questions
  1. Don't Rephrase a Participant's Response in Your Own Words. ...
  2. Don't Suggest an Answer. ...
  3. Avoid Yes/No and Either/Or Answers. ...
  4. Remove Biased Language. ...
  5. Don't Make Assumptions. ...
  6. Don't Lead With a Biased Statement Before the Question. ...
  7. Avoid Combining Two Questions Into One.

What is a double negative survey questions? ›

What is a double-negative question? A double-negative question includes two negative words, potentially confusing or misleading the participant completely. If a participant can't understand the question, of course, their answer will be meaningless and the resulting data will be useless.

What are the 4 types of survey questions? ›

4 Classes of Survey Questions
  • Open-Ended. If you could individually interview each survey respondent, you'd probably ask a lot of open-ended questions. ...
  • Closed-Ended (Static) ...
  • Closed-Ended (Dynamic) ...
  • Task/Activity Based.
5 Sept 2018

What are leading and non leading questions? ›

A leading question is a type of question that implies or contains its own answer. It subtly prompts the respondent to answer in a particular way. They are undesirable as they can result in false or slanted information. The non-leading question allows the respondent to answer with a range of answers.

When should leading questions not be allowed? ›

In general, leading questions are not allowed during the direct examination of a witness, however, they are allowed on the cross-examination of a witness.

What are 5 open-ended questions? ›

Examples of open-ended questions include:
  • Tell me about your relationship with your supervisor.
  • How do you see your future?
  • Tell me about the children in this photograph.
  • What is the purpose of government?
  • Why did you choose that answer?

What should be avoided when asking open-ended questions? ›

Avoid questions that have the following characteristics: answers that provide facts. easy to answer questions. answers that can be given quickly and require little to no thought.

Why do we need to avoid biases in writing? ›

Bias prevents you from being objective

You need to present factual information and informed assertions that are supported with credible evidence. If you let your personal biases take over your writing, you've suddenly missed the whole point.

What type of biased language can be found in line 5? ›

What type of biased language can be found in Line 5? Racial slurs are still a common part of communication in the United States.

What are 2 ways a researcher can avoid a biased sample? ›

Here are three ways to avoid sampling bias:
  • Use Simple Random Sampling. Probably the most effective method researchers use to prevent sampling bias is through simple random sampling where samples are selected strictly by chance. ...
  • Use Stratified Random Sampling. ...
  • Avoid Asking the Wrong Questions.
23 Oct 2019

What are the two types of biased samples? ›

Types of biased samples

Self-selection: Self-selection is when a person agrees to be a part of a study. This can lead to a biased sample as someone could argue that certain people are more likely to volunteer to be a part of a study. Undercoverage: Undercoverage occurs when a study doesn't include all populations.

What are the 3 types of bias in statistics? ›

Types of statistical bias

The most common sources of bias include: Selection bias. Survivorship bias. Omitted variable bias.

What is the most common bias in research? ›

Acquiescence bias (also known as the friendliness bias, confirmation bias, or “yea-saying”) is one of the most common types of bias in research. It manifests itself when a respondent shows a tendency to agree with whatever it is that you're asking or stating.

What is a biased survey? ›

What are biased survey questions? A survey question is biased if it is phrased or formatted in a way that skews people towards a certain answer. Survey question bias also occurs if your questions are hard to understand, making it difficult for customers to answer honestly.

What are examples of survey questions? ›

These include:
  • Multiple choice questions.
  • Rating scale questions.
  • Likert scale questions.
  • Matrix questions.
  • Dropdown questions.
  • Open-ended questions.
  • Demographic questions.
  • Ranking questions.

What are an example of 4 types of surveys? ›

What are The Different Types of surveys?
  • Online surveys. Online surveys include a set of structured questions that respondents complete through the internet. ...
  • Paper surveys. This survey uses traditional paper and pencil methods to collect data. ...
  • Telephonic Surveys. ...
  • One-to-One interviews.

Top Articles

You might also like

Latest Posts

Article information

Author: Madonna Wisozk

Last Updated: 12/13/2022

Views: 5897

Rating: 4.8 / 5 (68 voted)

Reviews: 91% of readers found this page helpful

Author information

Name: Madonna Wisozk

Birthday: 2001-02-23

Address: 656 Gerhold Summit, Sidneyberg, FL 78179-2512

Phone: +6742282696652

Job: Customer Banking Liaison

Hobby: Flower arranging, Yo-yoing, Tai chi, Rowing, Macrame, Urban exploration, Knife making

Introduction: My name is Madonna Wisozk, I am a attractive, healthy, thoughtful, faithful, open, vivacious, zany person who loves writing and wants to share my knowledge and understanding with you.