Research Report QA

How do I quality check a research report before it goes out?

Not by reading it once for everything. Run separate passes, each looking for one class of defect across the whole document: numbers, evidence, language, consistency, completeness, presentation. Then classify what you find by severity and issue a judgement. Attention cannot hold seven checking frames at once, which is why a single read reliably finds fluency problems and misses evidence problems. Every pass covers the executive summary, chart titles, headings, callouts and appendix, because defects concentrate in the material written last and read least.

Why should I check numbers against source rather than against the previous draft?

Because checking against the previous draft is how an error becomes established. Each version confirms the one before it, and by the fourth draft a wrong figure has been checked three times and everyone is confident in it. Open the analysis file, find the figure, and confirm the value, the base description, the base size, the filter, the weighting state and the question reference. Recompute anything that was typed by hand, which in practice means every number in a chart title, a callout, a headline or the summary.

How do I check for significance claims that were never tested?

Run it as an explicit pass. Search the document for "significant", "significantly", "notably", "markedly", "clearly", "substantially higher" and equivalents, including in headings and chart titles. For each, confirm a test was run and that the test, result and threshold are reported. Where no test was run, the wording changes to an observed comparison with both bases shown, or the claim goes. Check the reverse defect too: a tested and significant difference reported without its test understates what the study actually established.

How do I check for causal language a study cannot support?

Search for "drives", "leads to", "causes", "results in", "the impact of", "improving X will", "because" and "due to", and look hardest in chart titles and headings, where causal readings survive longest. For each, ask whether the design licenses a causal claim: a randomised experiment, a valid quasi-experimental design with its assumptions stated, or a longitudinal design with temporal order established. Where it does not, rewrite to association. Where a modelling output is conventionally called a "driver", the document should state once that these are statistical associations whose causal direction was not established.

Why does the executive summary so often overstate the body?

Because a claim loses a qualifier at each restatement, and each restatement is checked against the one before it rather than against the analysis. The fix is a specific check: compare every summary claim to the body claim it summarises, and compare both to the analysis output directly. Look for a hedge removed, a base dropped, a segment claim generalised to the whole sample, an association become a cause, and a moderate reading become a flat statement. This is the single most productive check in the whole pass.

Do I need to check every quote?

Yes, every quote, every time, word for word against transcript, with the participant identifier confirmed and the participant confirmed as belonging to the segment the quote is attributed to. Spot-checking selects against the quotes that need checking, because a tidied quote reads better than an untidied one and so is less likely to be chosen. A quote that cannot be located in a transcript is a critical defect, not a query.

What counts as a critical defect in a research report?

Anything where the document asserts something its evidence does not support, or omits something a reader needs in order to act correctly: a wrong number, an unverifiable quote, a recommendation with no traceable finding, a significance or causal claim the design does not license, a missing base where the base changes the reading, a summary claim stronger than the analysis supports, a material limitation absent from the document, or a required disclosure omitted. Severity is judged by consequence to the reader, not by how hard the fix is: a wrong number corrected in ten seconds is critical, and a terminology inconsistency across forty pages is minor.

Should QA produce comments or a decision?

A decision. A log of comments without a judgement leaves the release call to whoever is most tired at the end of the project. Fail while any critical defect is open, regardless of the deadline: delivering a document you know asserts something unsupported is a worse position than delivering late. Fail on a pattern of major defects even where each is individually tolerable, because fifteen inconsistencies indicate a systemic problem that spot fixes will not address.

Is report QA the same as a research quality review?

No, and the distinction matters. Report QA checks the document against its own evidence: does it say what its sources support, consistently, completely and defensibly. A research quality review checks the study: whether the design was appropriate, the sample adequate, the instrument valid, and the analysis the right one. A report can pass one and fail the other in both directions. Where QA reveals that a defect is methodological rather than documentary, it escalates rather than logging it as a document defect, and the report does not leave in the meantime.

What if I have no time for a full QA pass?

Do not sample randomly. Run the highest-yield subset in order: every number in the executive summary and every chart title, every recommendation traced backwards to findings, every quote, and the significance and causal language passes. That takes a fraction of the time and catches most critical defects, because critical defects concentrate in exactly those places. Then state in the log which passes were not run, so nobody reads a partial check as a full one.