Bias Detection

How do I check whether a research study is biased?

Run the bias chain forwards, because a bias early in it changes what the later stages mean: question, frame, response, instrument, delivery, analysis, interpretation, reporting. For each mechanism you find, record the specific route by which distortion entered, which way it pushes the finding, how large it plausibly is, and whether it can be corrected or only disclosed. Then sort by direction and count. The single most informative output is usually that most entries push the same way.

What is the difference between question bias detection and a full bias audit?

Question bias detection audits the instrument before it is fielded: leading phrasing, loaded terms, double-barrelled items, unbalanced scales, order effects. A full bias audit covers the whole project and runs at any stage, including after fielding when the instrument can no longer change. When the audit reaches the instrument it hands the item-level work back and keeps the project-level judgement: which findings rest on flagged items, and does any of them carry a headline.

How do I test for non-response bias rather than just acknowledging it?

Three comparisons, in order of strength. Compare the achieved profile against known population figures on variables correlated with the outcome. Compare late responders against early ones, since late responders resemble non-responders more closely and the gradient bounds the likely bias. Compare the responder profile against the sample frame's own records where the frame holds data. If none is possible, record non-response bias as unassessed rather than absent. Response rate alone is not bias: what matters is whether non-response relates to the answer.

How do I know if subgroup findings were cherry-picked?

Get the full analysis output and count the comparisons examined against the comparisons reported. A standard banner generates thousands of comparisons, and reporting the ones that reached a threshold is a selection procedure that produces findings whether or not anything is there. The reported-to-examined ratio is the most informative number in an analysis-bias audit. If the analysis outputs are not supplied, record that analysis bias could not be assessed rather than concluding it is absent.

Should a bias audit name the client's preferred answer?

Yes, and it is the entry most often missing. The largest single source of directional pressure in commercial research is that somebody paid for the study and would prefer a particular answer. Omitting it does not make it inoperative, it makes it undocumented, and an undocumented pressure is one nobody can compensate for. Write it in neutral factual language, before reading the findings, and include the researcher's own incentives alongside the client's.

What should I look for in a report's charts?

Truncated or non-zero axes making small movements look large; scales that differ between charts a reader will compare; ordering by convenience rather than by value or a stable order; colour coding valence so the favoured option reads as good before the numbers do; dual axes manufacturing an apparent relationship; a time window that starts at a flattering point; and bases omitted where they would undercut the visual. Redraw the important ones the plain way and see what changes.

How does bias enter through quote selection?

Articulate participants are over-selected, extreme statements are more quotable than typical ones, and the quote that expresses a theme perfectly is often the outlier that expresses it too well. Compare the quotes used against the distribution of views in the source, and check specifically whether the counter-position has a quote at all. A theme with three supporting verbatims and a counter-theme rendered as a paraphrase is asymmetric even when both prevalences are correctly stated.

How do I handle bias in multi-market research?

Assume non-equivalence and prove comparability, rather than the reverse. Four mechanisms need separate treatment: construct bias, where the concept does not exist in the same form everywhere; translation bias, where back-translation confirms lexical rather than conceptual equivalence; response-style bias, where acquiescence and extreme responding differ systematically by culture; and sampling non-equivalence, where the same quota produces differently composed samples. Market differences that appear on attitudinal batteries and vanish on behavioural measures are the signature of response style, not culture.

What should a bias register actually contain?

For each entry: the specific mechanism, stated precisely enough to be argued with, not a category label; the evidence for it at a named location in this project; the direction of effect, inflates, deflates or explicitly unknown; the magnitude where it can be estimated or bounded; the claims affected; and a disposition of correct, disclose, restrict or invalidate. Then a directional summary, and a note of what could not be assessed and why. Unassessed is not the same as absent.

When is the best time to audit a project for bias?

At design stage. At that point the question can be rewritten to admit a negative answer, the frame can be extended to the group it excludes, a balanced stimulus can replace a one-sided one, and a comparison set can be pre-specified. The same findings after fielding are permanent limitations that can only be disclosed. A design-stage audit takes a fraction of the time and prevents most of what a delivery-stage audit can only document.