Research Rigour Audit

How do I review a study's methods without being influenced by what it found?

Fix the reading order. Read and audit the methods in full, complete the design, sampling, measurement, procedure and analysis layers, and only then read the results. Where full blinding is not achievable, disclose that in the audit's scope note and apply the reversal test to every finding: would you have raised it if the study had found the opposite?

What is the difference between a questionable research practice and misconduct?

Questionable practices, such as selective reporting, undisclosed analytic flexibility, hypothesising after results are known and the misuse of exclusion criteria, mostly arise from ordinary analytic freedom exercised without a protocol, in good faith, in fields where no protocol is expected. Misconduct means fabrication, falsification or plagiarism, requires intent, and is established through an institutional process with evidence standards and rights of response. An audit identifies methodological concerns; it does not make allegations.

What are the signals that an analysis was chosen after seeing the data?

An outcome in the results that never appears in the methods; a subgroup analysis carrying the headline with no prior rationale; exclusion criteria described after the analysis rather than with the sampling; an unusual analytic choice with no justification; covariates that appear in one model and not another; and a reported n inconsistent with the sampling account. Each is reported as a located observation with its innocent explanation and the material that would settle it.

How do you audit qualitative rigour properly?

On practice, not vocabulary. Ask whether the sampling strategy fits the purpose, whether the coding process and audit trail are described, whether a saturation claim carries a stated criterion, whether themes are evidenced with enough data to support the prominence claimed, whether disconfirming cases appear, and whether reflexivity states a position and traces its effect rather than merely declaring one exists.

Should an audit give the study an overall quality score?

No. A single rating collapses exactly the information the audit exists to produce and invites readers to skip the findings. Report located findings with severity assigned on consequence for the stated conclusions, record the strengths, and let the academic who commissioned the audit make any summary judgement from that.

How is this different from a research quality review of a report?

A quality review of a deliverable also asks whether the conclusions serve the decision and whether the report is fit to present. This audit is narrower and stricter: it assesses the study, takes no view on usefulness, and applies academic expectations for reproducibility and reporting completeness that a commercial deliverable is not expected to meet. Where both are needed, run them together on their respective objects.

Can I use an AI system to audit a colleague's or student's unpublished study?

Check institutional policy first. The material is usually unpublished, sometimes under review, belonging to an identified person, and may contain participant data; where it is a student's work, assessment policy applies too. Where the position is unknown, do not upload it, and confine AI assistance to structuring and testing your own reading.