%202.png)
Analysis Plan Development
Writes down what will be tested, how, and what counts as meaningful, before any data exists. The structural defence against p-hacking.
Free. Works on Claude, ChatGPT, Gemini or any assistant that accepts a skill file.
What this skill does
The method, encoded.
Analytical flexibility is invisible and enormous. Between the arrival of a dataset and a finished chart lie dozens of defensible choices: which subgroups to cut, whether "don't know" stays in the base, how a net is composed, where a scale is dichotomised, which cases to exclude, which of forty possible comparisons to report. Made after seeing the data, each one is made in the presence of knowledge about which way it moves the answer, and none of that appears in the output.
The result is rarely deliberate manipulation. It is an honest researcher following a story the data appeared to offer, arriving at a finding that would not replicate, and reporting it with the confidence appropriate to a prediction.
This skill removes the flexibility by spending it in advance, in writing, dated. It designates one to three primary comparisons and specifies each completely. It fixes derived variables, nets, cut points and base conventions before anyone can see which version helps. It sets missing data and outlier rules that are applied blind. It faces multiplicity by counting the tests rather than adjusting the threshold afterwards. And it separates statistical significance from the question that actually matters: what size of difference would change what the organisation does.
It is equally clear that analysis discovered after seeing the data is not invalid. It is exploratory, it is often where the value is, and the only sin is reporting it as confirmatory.
Best used for
- Pre-specifying analysis before fieldwork
- Preventing p-hacking and post-hoc storytelling structurally
- Deciding what counts as a meaningful difference, not just a significant one
- Handling multiple comparisons deliberately rather than after the fact
- Fixing derived variables, nets and base conventions in advance
- Keeping exploratory findings separate from confirmatory claims
- Setting up an experiment or test with a primary outcome
- Giving a tracker a stable analysis convention
Typical inputs
What you give it.
Objectives and research questions from 01.02, Hypothesis register from 01.03 (optional but strengthening), Sample design and achievable bases per reporting cell from 01.06, The instrument, or the measures as they will be asked, The decision the study serves, and its owner, The smallest difference that would change the decision (optional), Prior wave or comparable study data for expected proportions and variances (optional), Data processing conventions in use (optional)
Typical outputs
What you get back.
Locked, dated Analysis Plan, Objectives to outputs table, Fully specified primary comparisons with tests, thresholds and required bases, Secondary comparisons with their multiplicity treatment, Derived variable, net, index and scale conventions with reliability failure rules, Subgroup register with the reason a decision turns on each, Missing data rules including the "don't know" base convention, Data quality and outlier rules applied blind to outcome, Multiplicity approach with its stated cost, Meaningfulness thresholds and the four significance-by-meaningfulness cases, Exploratory register with its governing rules, Deviation log recording whether each change preceded or followed sight of the data
Method coverage
What the skill works through.
- Why analytical flexibility is the problem
- Designating primary comparisons (three at most)
- Secondary comparisons and what makes them secondary
- Fixing derived variables, nets and cut points in advance
- Choosing subgroups for a reason, not because the variable exists
- Missing data, "don't know", and the base convention
- Outlier and data quality rules applied blind
- Multiple comparisons: the approaches and what each costs
- Meaningful versus significant: the four cases
- The exploratory register
- Locking, dating and logging deviations
Download
Free skill. One file.
Enter your email once. Every skill you download after that takes a single click.
How to install
Add the skill file and the five kernel protocols to a Claude Project, a ChatGPT Project, a Gemini Gem, or paste them at the top of any assistant conversation. Then give it your real research material, not a description of it.
Download skillQuestions
Common questions.
What is an analysis plan and why write one before fieldwork?
It is a dated document specifying what will be analysed and how: the comparisons, the tests, the thresholds, the derived variables, the base conventions, and the rules for missing data and exclusions. Writing it before the data exists means every choice is made without knowing which way it moves the answer, which is the entire mechanism.
How do I avoid p-hacking?
Spend the flexibility in advance. Designate one to three primary comparisons, count the total number of tests you plan to run, fix your derived variables and cut points, decide the base conventions, and set exclusion rules that will be applied without reference to their effect. The most effective single move is not a correction procedure: it is running fewer tests.
Is analysis I discovered after seeing the data invalid?
No. It is exploratory, and exploration is often where the value in a dataset is. The failure is reporting an exploratory result as though it had been predicted. Record every exploratory analysis including those that found nothing, report the finding in hypothesis language with what would validate it, and hand it to the next study as a hypothesis rather than to this one as a conclusion.
What is the difference between a significant difference and a meaningful one?
Significance is a property of your sample size: with a large enough sample, a trivial difference becomes reliable. Meaningfulness is a property of the business: it is the smallest difference that would change what the organisation does, and only the decision owner can set it. A study that reports the first without the second has outsourced its judgement to n.
What do I do with a difference that is not statistically significant?
Ask whether it was large enough to matter. A meaningful-sized difference that missed significance is inconclusive, not absent, and reporting it as "no difference" is the commonest quantitative error in applied research. Report the estimate, the interval, and the base that would have been needed to detect the difference that matters.
Should I correct for multiple comparisons?
Count your planned tests first, because the answer depends on the number. No correction is defensible for a few pre-specified primaries, with the cost stated: at the conventional threshold, roughly one in twenty independent tests produces a false positive by chance. Family-wise control protects against any false positive at a real cost in power. False discovery rate control suits large exploratory batteries. Whatever you choose, choose it before you see which tests survive.
Should "don't know" be in the percentage base?
Decide once, in advance, and apply it everywhere. It is a single convention that can move a headline several points, and deciding it after seeing which way it moves is the cleanest example of a contaminated choice. Where the share is material, report both versions.
How do I set the subgroups I will analyse?
By whether a decision turns on each one, not by which variables exist in the dataset. Write the reason next to each, attach the minimum base, and decide in advance what happens if the base falls short: report descriptively, combine on a stated rule, or drop the cut. Then multiply the subgroups by the measures to see the size of the multiplicity problem you have created.
Can I write an analysis plan after the data has arrived?
Yes, and it is worth doing. What you cannot do is date it as though it preceded the data. Write it now, date it now, run what it specifies, and label everything in it exploratory, disclosing how many comparisons were examined. That is a weaker position than pre-specification and a far stronger one than presenting selected findings as predictions.
The skill chain
Works well with.
Research where people already are.
Analyse it where you already work.
Yazi helps researchers conduct surveys, AI interviews and longitudinal research directly through WhatsApp.
