Meta-Analysis Across Studies

Can we answer this question from research we have already done?

Often, but only after a comparability assessment. Screen the studies against written inclusion criteria, then compare them on objective, population, base, fieldwork period, method and mode, question wording and scale, construct definition and preceding context. It is normal for a corpus of nine studies to yield three that can legitimately be compared on the measure in question, and finding that out is itself valuable.

Why do studies that look comparable turn out not to be?

Almost always because of the instrument rather than the report. A five-point scale against a seven-point, "satisfied" against "very satisfied or satisfied", "in the last year" against "ever", a base of all respondents against a base of qualifiers: any one of these makes a direct comparison invalid, and none of them is visible from a findings summary. Read the instruments, not the reports.

Is this the same as a statistical meta-analysis?

No, and the distinction matters. Formal quantitative meta-analysis pools effect sizes, weights by precision and quantifies heterogeneity, and it requires studies asking the same question of comparable populations with reported variance. Commercial, policy and organisational corpora almost never meet those conditions, because studies commissioned for different business questions at different times were never designed to be pooled. What such corpora do support is structured qualitative synthesis, which is the correct method for heterogeneous evidence rather than a weaker substitute.

How do you tell a real change over time from a difference between studies?

Work through four explanations in order, and only the last is change in the world. Methodological: wording, scale, mode, base or frame changed. Compositional: the population itself changed, so a different set of people is being measured under the same label. Noise: the difference is within what those bases would produce by chance. Real. The first three are collectively far more likely than the fourth, and a difference that coincides with an instrument change should be treated as methodological until proven otherwise.

Do more studies agreeing mean a stronger finding?

Not by itself. Three small, incidental, poorly documented studies agreeing are weaker than one well-designed study on an adequate base. Weight by quality and check independence: several studies by the same team using the same instrument corroborate far less than they appear to, and a claim repeated across six documents is often one original finding with good internal distribution.

What is the bias in an organisation's own research corpus?

It is systematic rather than random, because a company researches what it is interested in, what it has budget for and what is easy to reach. Look for topic bias, population bias (most corpora are built almost entirely on current, engaged, reachable participants and can say nothing about anyone else), question bias where one framing has been repeated until its artefact reads as a robust finding, and outcome bias where inconvenient studies were less likely to be written up.

What if the existing research cannot answer the question?

Say so. It is a frequent and legitimate verdict, and it is worth money: it stops a study being commissioned to a brief that would repeat the corpus's own blind spot, and the reason the corpus failed (population coverage, wording inconsistency, base size, ageing, or no measurement at all) is the specification for the study that would work.

What is the difference between this and a literature review?

Scope of evidence, and therefore the analytical problem. A literature review or evidence synthesis works on external published sources, where the difficulty is source quality, provenance and verification. This works on the organisation's own study corpus, where the difficulty is comparability between studies you commissioned yourself and the systematic bias in what you chose to research. Where a question needs both, run them separately and integrate rather than pooling internal and external evidence into one table.