Statistical Testing

How do I know whether a difference between two survey subgroups is real?

Run the test that matches the design, then look at three things together rather than one. The p-value tells you whether the gap is larger than sampling noise. The confidence interval on the difference tells you how large the gap plausibly is. The effect size tells you how large it is in units a reader can judge. A difference reported without all three, and without both base sizes, is not reportable.

What is the difference between statistical significance and practical significance?

They answer different questions. Statistical significance asks whether a difference could be noise. Practical significance asks whether it is big enough to matter. A large sample makes trivial differences significant and a small sample hides real ones, so significance on its own tells you as much about your sample size as about the world. Both must be reported, and the practical judgement needs a materiality threshold agreed before the analysis, not argued afterwards.

Which statistical test should I use?

It follows from two things: what the measure is, and whether the same respondents appear in both groups. Two independent groups on a proportion take a two-proportion z-test; on a mean, Welch's t-test; on ordinal or heavily skewed data, Mann-Whitney U. Three or more independent groups take a chi-square, an ANOVA or a Kruskal-Wallis, each followed by targeted comparisons. The same respondents measured twice take McNemar's test, a paired t-test or a Wilcoxon signed-rank test. Running a paired design through an independent test is the commonest selection error in tracker and pre/post work.

What is a good effect size?

There is no universal answer, and the conventional small, medium and large labels were offered as a last resort for fields with no established scale. A standardised difference of 0.2 in a mature category where nothing has moved for five years can be the largest effect anyone will find. Calibrate against the measure's own historical variation, the size of effects comparable interventions have produced, and the difference the business would actually act on.

Why do so many differences come up significant in a cross-tab?

Because a banner runs an enormous number of tests. Twelve columns produce sixty-six pairwise comparisons per row; across sixty rows that is nearly four thousand tests, of which roughly two hundred will flag at a 5% threshold even if nothing differs anywhere. Reading the flagged cells off the table and writing them up reports noise in a formal-looking way. Count the tests before running them, and either correct within a defined family or label the output as exploratory screening.

When should I apply a Bonferroni correction?

When you are screening a family of comparisons and any one of them might be reported as a finding. Bonferroni is simple and valid but conservative; Holm controls the same error rate and is uniformly more powerful, so there is little reason to prefer Bonferroni once Holm is available. Benjamini-Hochberg controls the false discovery rate and is usually the better choice for exploratory screening. None of these converts a fishing expedition into a confirmatory result. A small set of comparisons pre-specified before the data was seen is the real solution.

Does a non-significant result mean there is no difference?

No. It means the study could not distinguish the observed gap from zero, which is exactly what a real difference looks like when the sample is too small to detect it. Before writing anything about a null, work out the minimum detectable effect. For two proportions near 50%, roughly 20 percentage points at n=100 per group, 14 at n=200 and 9 at n=500. Then say what the study could and could not have seen. Establishing that two things are genuinely equivalent needs an equivalence margin set in advance, not a failed test.

Can you run significance tests on a non-probability sample?

The theoretical basis for the p-value assumes random selection from a population, and most commercial online samples are not obtained that way. Practice is genuinely split between not testing at all and testing as a rough filter for whether a gap exceeds ordinary random variation. The defensible position is to test if you choose, describe the result as a noise filter rather than inference to a population, and report no margin of error.

Why can I not test my weighted data the normal way?

Because a standard test treats weighted counts as though the weights were extra respondents, and that manufactures significance. Calculate the effective base first, then use it in every test. A design effect of 1.6 means a nominal base of 800 behaves like 500, and the difference is enough to flip results either way.

Can AI run significance testing reliably?

It can run the arithmetic consistently, which is a genuine advantage. The risks are specific: producing a plausible p-value with no calculation behind it, choosing a test from the shape of a table without knowing whether the respondents are the same people, reporting only the comparisons that came up positive, and attaching mechanical effect-size labels with no reference to the measure or the decision. Used properly, AI should be required to produce a full test log including the nulls, state the number of tests run, and refuse to test where the bases, the weighting or the sample design do not support it.