Evidence Synthesis

What is the difference between evidence synthesis and a literature review?

A literature review finds, verifies and appraises sources. Synthesis combines what those sources say into a single assessment of what is known, with a strength rating attached. They are sequential rather than alternative: synthesis run on an unappraised source set produces a confident average of unknown material.

How do I combine studies that disagree?

Diagnose the disagreement before judging it. Work through six causes in order: definitional, methodological, sampling, temporal, geographic, and only then genuine. The first five account for most apparent contradictions, and each has a different consequence. Where the conflict is genuine, report both positions with their evidence and say which is better evidenced. Do not average them.

Is three studies agreeing stronger evidence than one good study?

Not necessarily, and often not. Weight depends on directness, quality and independence together, and those are not compensatory. Three small, thinly reported studies sharing a common weakness are one line of evidence with a shared flaw. Ask what the strongest source alone would support, then ask what each of the others genuinely adds.

What is the vote-counting fallacy?

Deciding a question by the majority of studies: four found an effect, two did not, so the effect is real. It ignores study size and precision, treats a non-significant result in an underpowered study as evidence of no effect, and discards direction and magnitude, which is where the information actually is. Replace the tally with a distribution of direction and magnitude alongside base and quality.

How do I know whether two studies are measuring the same thing?

Compare five things before combining: the construct definition, the operationalisation including exact item wording and scale, the referent (a brand, a category, a specific event), the population and frame, and the measurement period. Sources sharing a word routinely differ on several of these. Record the verdict as commensurable, partially commensurable (direction transfers, magnitude does not) or incommensurable.

How do I assess publication bias without statistical tests?

Reason about the mechanism. Ask who commissions research in this area, whether unfavourable results would have been released, whether a study series has a missing wave, whether a protocol lists outcomes that never appear in the results, and what languages and access limits filtered your search. An evidence base where every source points one way and every source shares an interest should be reported as one source of evidence about that interest.

When are studies too different to synthesise?

When conceptual heterogeneity is severe: different populations, different constructs, different outcomes with no defensible mapping. The correct output then is a set of separately reported context-specific findings with the incompatibility explained, not a synthesis. Deciding not to synthesise is a legitimate result and is frequently the honest one.

What is a strength-of-evidence rating and how do I assign one?

It rates the body of evidence rather than individual studies, across six domains: quantity and independence, quality, consistency, directness, precision, and risk of missing evidence. Start from what the strongest contributing evidence would support alone, downgrade for each domain that materially bites, and name the domain that capped the rating so a reader can see why it is not higher.