Question Bias Detection

How do I know if a survey question is biased?

Work through four passes rather than reading for a feeling. Check the stem for leading and loaded wording, two ideas in one question, an assumed premise about what the respondent did or noticed, undefined quantifiers and jargon. Check the response frame for balance in both point count and label intensity, for options a respondent could legitimately double-select, for overlapping numeric ranges, and for the options that are missing. Check the whole instrument in order for priming, anchoring and contamination between questions. Then check for response-style effects such as acquiescence and social desirability. Most reviews only do the first pass, which is why most instruments field with their largest defects intact.

What is a double-barrelled question?

One that asks about two things and permits one answer, usually joined by "and" or "or": "how satisfied are you with the speed and accuracy of the service?". A respondent who is happy with one and not the other cannot answer it honestly, and the analyst cannot tell which half produced the score. It is generally unrecoverable at analysis, because no technique separates the two components after the fact. The fix is to split it, or, where length forbids, to keep the half that serves the decision and drop the other.

Why does a missing "don't know" option matter?

Because the respondents who did not know are now inside the substantive distribution and cannot be identified afterwards. Every missing non-substantive option has a direction: a missing "don't know" inflates whichever substantive answer is most available; a missing "not applicable" inflates the neutral or midpoint; a missing "none of these" on a multi-select inflates the nearest listed option; a missing "prefer not to say" on a sensitive question produces break-off or a false answer. The counterpart discipline matters too: adding "don't know" to an attitude question where everyone has a view manufactures non-response and rewards satisficing.

What is an unbalanced scale?

One where the positive and negative sides are not symmetric, in point count or in label intensity. "Excellent, very good, good, fair, poor" is the classic case: three positive points against one negative, with a mean pushed upward before anyone answers. It is partially recoverable, in that the bias applies to everyone so subgroup comparison and wave-on-wave movement can still be read, but the absolute level is uninterpretable and cannot be compared to any externally scaled measure.

How do earlier questions bias later ones?

Four mechanisms. Order contamination, where an aided list shown early destroys the ability to measure spontaneous awareness later, which is irreversible. Priming, where a block of questions establishes a frame the key question then sits inside. Carryover, where a respondent who has just committed to a position answers consistently with it whether or not that is what they think. And anchoring, where a number, a price or a range stated in one question sets the scale for the answers to the next. None of these is visible when reading question by question, which is why the instrument has to be read once end to end, in order, before anything is annotated.

Can bias be in the objectives rather than the questions?

Yes, and it is the most consequential kind, because every question can be faultless and the study still cannot return the finding that matters. "Understand why customers value the new service" cannot produce the finding that they do not. "Identify the barriers to adoption" presupposes that adoption is desirable and that the obstacle is on the customer's side. The test is to rewrite each objective symmetrically and see whether the instrument still makes sense; frequently half of it does not, and the gap is the missing finding. This cannot be fixed by rewriting a question and has to go back to whoever agreed the brief.

How should I rate the severity of the defects I find?

By consequence, not by how obviously wrong the question looks. A badly worded question nobody will report is minor. A subtly unbalanced scale on the primary measure is critical. Two things push severity up: whether the question feeds something that will be reported, and whether the defect is recoverable at analysis. A reviewer who marks everything as important gets everything ignored, so a stated rubric with a small number of critical findings is far more likely to result in changes than a long undifferentiated list.

Should a bias audit include rewrites?

Always, in full text. A flag is a complaint; a rewrite is a deliverable. Reviewers who supply replacement wording get their changes made, and reviewers who supply adjectives get a defensive meeting. Two disciplines go with it: state in one line what the rewrite does not fix, because a neutrally reworded question still sits after the block that primed it, and then audit the rewrites themselves, because corrections introduce new defects at a high enough rate that skipping that step is a known way to make an instrument worse.

How do I review an AI-generated questionnaire?

Target the places generation is weakest rather than reading it front to back for tone. Generated drafts are fluent, so stem-level defects are rarer than in human drafts. They fail most consistently on option lists (frequently invented, frequently containing every positive attribute and one token negative), on scale label symmetry, on missing non-substantive options, on acquiescence load because agree/disagree batteries are easy to generate symmetrically, and on length, because generated instruments have nothing removed. Ask for the provenance of every substantive list: an unsourced brand, feature or competitor list is a defect of unknown direction, not a neutral default.

What if the biased question is part of a tracker and cannot be changed?

Then the right output is a disclosure, not a change request. Log the defect, state its direction, and note what remains readable: an unbalanced scale that has run for six waves still supports wave-on-wave comparison, because the bias applies equally every time, even though the absolute level cannot be compared to anything external. Present the options and their costs (change and re-baseline with a parallel run, or retain and disclose) and leave the decision with the researcher, because trading measurement quality against a published commitment is a business judgement, not an audit finding.

Can an instrument ever be certified unbiased?

No, and an audit that says so has overstated its product. Every instrument makes choices that push answers in some direction, and some of those choices cannot be avoided: the sponsor is inferable, the mode has effects, the topic has a socially preferred answer. The honest output is what was checked, what was found, what was fixed and what remains, with the direction of each residual stated, so that the eventual report's limitations section can be written from the audit rather than invented at the end.