Survey Logic and Flow Review

How do I test survey logic properly?

By walking paths, not by reading the script. Build a logic inventory with a formal base condition for every question, enumerate the branch points, then construct a documented test-case matrix covering every branch in both directions, every quota cell, every terminate, the longest and shortest paths, and the paths any reported subgroup will take. Each case names the persona in terms of defining answers, the questions it should see, the questions it must not see, the expected outcome and the expected length. Then walk each case in the live instrument and record pass or fail. A reviewer who reasons about the script instead of walking it reproduces the author's assumptions exactly, which is what the review exists to check.

What are the most common survey routing errors?

Seven fault classes. Dead ends, where a respondent reaches a question with no valid answer. Orphans, where a question is unreachable and an objective goes silently unmeasured. Over-reach, where people who should have been filtered out can answer, contaminating the base invisibly. Null completions, where someone reaches the end without answering anything substantive. Loops that return respondents to a question already answered. Terminates that end the interview without recording why. And contradictory conditions, where two rules overlap and the outcome depends on the scripting tool's rule precedence.

Why does routing determine the analysis?

Because routing is the denominator. Every percentage in the eventual report was decided by a filter months before anyone built a table. If a base is defined loosely, someone under deadline will choose the denominator at analysis time, and two numbers about the same construct will appear in the same deck on different bases. Nested filters are the usual culprit: a question asked of those who considered switching, among those with a contract, among those aware of the brand, has three multiplied incidences and produces a base nobody predicted.

What is the difference between a base condition and a base description?

A base condition is machine-checkable: Q4 = 1 or 2, and Q7 is not 5. A base description is the sentence that will sit under the chart: "recent purchasers who did not use the retailer's website". Both are needed, and they must describe the same set of people. Where they disagree, that is the defect, and it is usually the description that captures what was wanted and the condition that captures what was built. A base written only as prose, such as "asked of relevant respondents", is not a base at all; it is a placeholder that survives to analysis and then becomes an argument.

How should quotas interact with survey routing?

Deliberately, and documented. Check where each quota is counted, because a quota counted several minutes into the interview terminates respondents who have already answered reportable questions, which is a data problem and an ethical one. Check what happens when a cell fills, because the remaining fieldwork then draws a systematically different sample on every variable that is not quota-controlled, and that needs recording for the analysis rather than discovering in it. And check that screen-outs, quota-fulls, quality terminates and drop-outs are separately coded, because incidence is measurable only if they are, and incidence is what the next study's feasibility and cost depend on.

How do I test piped text?

Trace every pipe to its source and test it three ways: with a blank source, with a maximum-length source, and with an unusual value. The commonest defect is a pipe whose source question sits behind a filter the piped question does not share, so on some paths it renders empty and the respondent reads "How satisfied were you with your visit to ?". Also check grammar in every case (plurals, articles, verb agreement, any gendered form), check that text piped from an "other, please specify" field cannot render something unusable, since that is free text a respondent typed, and check that no pipe introduces a brand or attribute name earlier than it should first appear.

Does randomisation solve order bias?

Only partly, and only if the order is recorded. Rotation converts a fixed, directional bias into random noise the analysis can live with, which is a real gain. But it does not remove position effects, it distributes them, and unless the order presented to each respondent is written to the data file, those effects cannot be detected or controlled afterwards. Non-substantive options such as "other" and "none of these" are always pinned, and so are items with a natural order. Watch also for randomised blocks that change the context of later questions differently for different respondents, which converts a controlled rotation into an uncontrolled context effect.

How long should a survey be, and how do I estimate it?

The average length is the least useful number in the review, because nobody experiences it. Estimate the longest path, the shortest completing path, and the length for each subgroup the study depends on, using a stated timing convention labelled as a planning heuristic rather than a measurement. Two findings recur: the longest path usually belongs to the most engaged, most category-involved respondents, who are the ones the study most needs and who are being asked to do the most work; and a path far shorter than the rest often means a filter is excluding people who should have been asked something.

What is the biggest routing risk in a tracker?

A routing change, because nobody notices it. Everyone notices a wording change and debates it. A filter widened by one option, a quota counted at a different point, or a question moved inside a randomised block changes the base or the context of a trended measure while the question text stays identical, and the resulting movement gets reported as a change in the market. Before every wave, diff the routing against the previous wave question by question and treat each difference as a potential rebasing.

Do I need to retest after fixing a routing error?

Yes, and not just the case that failed. Routing fixes break other routes at a high enough rate that partial retesting is the main mechanism by which logic errors reach the field: a fix that reroutes one group past a dead end frequently routes them past something else too. Re-run every test case that touches the changed path, freeze the instrument at sign-off, treat any later change including a wording edit that moves a question as re-triggering the affected cases, and record which version was signed off and by whom.

Can this be done after fieldwork has finished?

The method changes and the data becomes the evidence. Cross-tabulate each filtered question against its filter to find respondents who answered something they should not have reached, and count those who should have reached a question and have no record. Compare completion times by path against the estimates, and inspect randomisation positions for imbalance. The output is not a defect list, since nothing can be fixed, but a base-integrity statement naming which findings are safe, which need rebasing and which cannot be reported, plus a routing specification for the next wave.