Screener and Quota Design

How do I write a good screening question?

Convert the criterion into something observable, answerable and stable. "Decision maker" becomes a specific act: what role the respondent had in the most recent purchase decision, selected from a list that includes "no involvement". "Regular user" becomes a frequency over a defined window set by the category's natural rhythm. "Interested in the category" becomes a behaviour only an interested person performs. Apply three tests to each: is it observable rather than internal, can the respondent answer it accurately, and would they answer the same way next week.

What order should screening questions go in?

Cheapest and most-eliminating first. Put the criterion that removes the largest share of the population at the front, provided it is quick to ask, so everything after it is asked of a much smaller group. Put anything slow, sensitive or expensive last. In a low-incidence study most people who answer the screener never enter the study, and every question asked of them is paid for and discarded. Then check the order for a second property: it must not teach. A screener that opens by naming the category has already signalled the target.

How do I stop respondents gaming a screener?

Design so that reading the question does not reveal the qualifying answer. Embed the criterion of interest in a list of plausible alternatives with no visual emphasis. Ask about behaviour rather than category membership. Avoid single yes/no eligibility questions, which have an obvious right answer and a fifty per cent guess rate; use a frequency scale or a list and set the qualifying range afterwards. Keep the invitation and introduction broad enough not to name the qualifying behaviour. Then test it: show the screener to someone and ask them to try to qualify.

How do I estimate incidence, and why does it matter so much?

Take each criterion's estimated pass rate and multiply down the chain, being explicit that criteria are rarely independent, so multiplying independent probabilities usually understates incidence while assuming full overlap overstates it. Produce a range rather than a point, name the source of every input, and mark unsourced inputs as assumptions. Then show the sensitivity, because that is what changes decisions: an incidence assumption of 8% that turns out to be 3% roughly triples the screening cost and can double the fieldwork window. Where incidence is genuinely unknown, measure it with a short screening exercise before committing the budget.

What is the difference between interlocking and non-interlocking quotas?

Non-interlocking, or marginal, quotas control each variable independently: the sample matches on age and matches on region, but nothing guarantees the joint distribution, so a sample can be perfectly correct on both margins and contain almost no young people in the smallest region. Interlocking quotas control the combination, which delivers the joint distribution and multiplies the cells: three age bands by four regions by two user types is 24 cells. Interlock only where the joint distribution matters to the analysis, and always calculate every cell's expected size at the planned sample before agreeing the frame.

Are trap questions worth using?

Sparingly, and only when the failure is unambiguous. An item a careless but honest reader could plausibly select is not a trap, it is a coin toss with consequences. Use very few, because a respondent who notices the trap now knows they are being tested and changes how they answer everything afterwards. Log every result so the false-positive rate can be estimated, and avoid terminating on a single trap unless the failure is unarguable. A fictitious brand in an awareness list works, but it contaminates the awareness measure it sits in, so it belongs in the screener rather than in a tracked question.

How do I keep professional respondents and fraudulent participants out?

Layer the controls rather than relying on one. Internal consistency, with the same fact asked in two forms at a distance. Impossible or implausible combinations of holdings, roles or behaviours. A knowledge check that a genuine member of the population answers without effort. A short open-ended verification question, which is the strongest single control for specialist audiences because it is expensive to fake and cheap to read. And behavioural signals such as implausible speed or duplicate identifiers, applied as flags rather than automatic removals. Then price the false positives: if the people wrongly removed are disproportionately slower, older or second-language respondents, the control has introduced the bias it was meant to prevent.

Should I quota on a variable, or leave it free?

Ask three things: what analysis requires it, what happens if it is left free, and what it costs to control. Variables that fail all three come out of the frame, because every quota makes fieldwork harder, slower and more expensive. Distinguish the two legitimate reasons: a representative quota matches a known population distribution and requires a named, defensible source; a design quota guarantees a minimum base on a subgroup the analysis must report, whether or not it is proportional. Confusing them produces a "representative" sample that cannot report on the small segment the study was commissioned to understand.

Do quotas fix a biased sample?

No. Quotas control who is in the sample, not how the sample is counted, and they correct nothing that is not a quota variable. A sample that is quota-correct on age and region can still be badly unrepresentative on everything else, particularly if hard-to-reach respondents have been repeatedly replaced with easy-to-reach ones from the same cell, which a quota frame is structurally unable to detect. Where the achieved sample deviates, or where the frame is marginal rather than interlocked, weighting may still be required, and that is a separate discipline.

What can I not report if I used quotas?

Anything the quota fixed. If the sample was quota-controlled to 50% users, the study cannot report the proportion of users in the population, and someone will eventually try. More broadly, the frame determines what can be compared: subgroups guaranteed a base can be compared, and everything else is whatever fieldwork happened to deliver. Write these consequences down when the frame is agreed, along with the report's population statement, which is the screening criteria written out rather than the wording from the brief.

Can I change the screener between waves of a tracker?

Only deliberately, and it is a parallel-run decision rather than an editing one. The screener is part of the trend: change a window, a role list or a frequency threshold and you change the population while every other number stays comparable-looking, which is the most expensive undetected error in tracker work. In panel-based longitudinal studies the opposite problem also applies, since eligibility itself changes over time, so add a re-qualification step at each wave rather than assuming a screening decision from two years ago still holds.