Research Quality Review

How do I review the quality of a research project?

Reconstruct what was actually done before reading the report, using the artefacts in a fixed order: brief, design, instrument as fielded, sample records, fieldwork records, data handling log, analysis outputs, and the report last. Compare that reconstruction with the report's method section line by line, because the gap between planned and actual is where most defects live. Then test whether the design could have answered the question, match each claim to the sample it needs, and run the check classes across every stage.

What is the difference between a research defect and a limitation?

A limitation is a known boundary of what a design can support, disclosed honestly, with the claims kept inside it. A non-probability sample is a limitation. A defect is an error inside the design's own terms, or a claim that crosses the boundary the limitation sets. The test: write the limitation on the page next to the claim. If the claim still stands, it is a limitation to disclose. If the claim collapses, it is a defect and the required action is to change the claim, not to add a caveat.

How should I grade the severity of a research review finding?

On consequence to a named claim, not on how poor the practice was. Critical means the research cannot support a central claim and no disclosure fixes it. Major means a specific claim is unsupported as stated, or the effect of a defect is unknown and could be material. Moderate means the finding stands but an undisclosed limitation changes how it should be read. Minor means a defensible choice poorly documented. A wrong procedure that cannot have changed the answer is minor.

What is the difference between report QA and a research quality review?

Report QA checks the document: numbers consistent between summary and chart, terminology, base labels, structure, every objective addressed. A quality review checks the research: whether the design could answer the question, whether the sample supports the claims, whether the analysis was right. A report can pass QA completely and still describe a study that cannot support a word of it. Run both.

How do I review my own research objectively?

You cannot fully, and the honest move where stakes are high is to get an independent reviewer. What helps: review from the artefacts only and never from memory, because a decision that is not in a document is undocumented whatever you remember agreeing. Insert distance, by time or by writing as an external reviewer with no stake. Start with the findings you like most, since they got the least scrutiny going in. And write the three sentences a competent critic would use to discredit the study, then answer them from evidence.

When should a research design be reviewed rather than the finished study?

At design sign-off, before money is committed. At that point the question, method, sample, instrument and analysis plan are all still changeable and a finding costs a conversation. At delivery, only three things can change: the claims, the disclosures, and whether the study goes out. If a team can afford only one review, it should be the design review.

What do I do when a serious review finding is not accepted?

Escalate on a path set before the review rather than argued during it. Moderate findings resolve between reviewer and project lead. A major finding the project lead rejects goes in writing to whoever owns the deliverable, with both positions recorded. An unaccepted critical finding goes to whoever owns the research function, and the reviewer's disagreement is recorded in the project file whether or not it changes the outcome. A review whose findings can be silently overruled provides no assurance to anyone.

Can a research quality review conclude that a study is unusable?

Yes, and a review process that never returns a negative judgement is providing reassurance rather than assurance. The judgement is about fitness for the specific claims being made, so the most common useful outcome is that a study is unfit for the claim being made and fit for a narrower one. Naming the surviving claim is what gets the review acted on rather than resisted.

How do I check whether a sample supports a claim?

Take the claim, write the population it implies, write the population the achieved sample actually represents, and look at the difference. A claim about customers resting on a sample of active account holders excludes the lapsed, who are usually the most informative group. Do it claim by claim rather than for the study as a whole, and use the base of the specific claim, not the study's total.

What should I look for that is missing from a research report?

Work from the instrument and the brief rather than from the report's contents page. Every fielded question either appears in the report or is accounted for. Check for objectives with no analysis behind them, evidence streams that contradict the story and appear nowhere, subgroups analysed but not shown, and the absence of a section saying what could not be established. Missing evidence is invisible if you only assess what is present.