Realist Synthesis: reviewing evidence for mechanisms, not averages

The realist premise is that programmes do not work; their mechanisms do — the responses they trigger in people’s reasoning and resources — and mechanisms fire only in conducive contexts. A mentoring scheme is a theory that contact with a credible adult changes a young person’s self-expectations; whether it does depends on who counts as credible, to whom, where.

A realist synthesis therefore begins by eliciting the candidate programme theories (from the literature’s own claims, from designers, from prior reviews) and then searches for evidence bearing on each — including evidence from adjacent interventions that share the mechanism. The review’s output is refined theory: which mechanisms fire in which contexts to produce which outcomes, stated as CMO configurations with the supporting evidence attached.

This inverts the systematic-review hierarchy: relevance to theory, not study design, governs inclusion, and appraisal asks whether each source is good enough for the specific inference it is being used to support.

The founding statement is Pawson, Greenhalgh, Harvey and Walshe’s Realist review — a new method of systematic review designed for complex policy interventions, which sets out the rationale and the review stages.

Quality and publication standards exist and are enforced by journals: the RAMESES project’s publication standards for realist syntheses (Wong and colleagues) specify what must be reported, from theory elicitation to the evidence behind each configuration.

Pawson’s book-length treatment, Evidence-Based Policy: A Realist Perspective (SAGE), remains the best account of the reasoning style — including the disciplined use of purposive, saturation-driven searching that replaces exhaustive retrieval.

• When effect estimates conflict and context is the suspect. Heterogeneous trial results are the method’s home ground: the divergence is the data.
• When the intervention is a family, not a protocol — mentoring, community health workers, pay-for-performance — where pooling averages across incommensurable variants answers nothing.
• When decision-makers need transferability guidance: not “does it work on average” but “will it work here, with our population and constraints”.
• Not when the question is a clean effect size for a standardised intervention — a conventional systematic review with meta-analysis is cheaper and stronger there.
• Not on a deadline that forbids iteration: realist searching and theory refinement are cyclical by design.

A synthesis asked why incentive schemes for smoking cessation produced wildly divergent results across dozens of studies. Theory elicitation produced three candidate mechanisms: incentives as compensation (offsetting quitting’s costs), as commitment device (external stake against relapse), and as signal (the institution cares whether I quit).

Purposive searching crossed literatures — trials, process evaluations, qualitative studies of participants, plus adjacent incentive research in weight loss and medication adherence where the mechanisms had been probed directly. Extraction into CMOs made the pattern legible: compensation-style lump sums worked while paid and decayed at cessation of payment, strongly in low-income contexts (the incentive relieved a real constraint) and negligibly in salaried populations; commitment-structured schemes (deposits, escalating schedules) showed durable effects but brutal self-selection — the smokers most likely to enrol were least in need of the device; and the signalling mechanism fired mainly in workplace schemes, where qualitative studies showed effects travelling through perceived employer regard rather than money.

The refined theory replaced “do incentives work?” with targeting guidance: which scheme architecture, for which population, expecting which decay curve — each configuration carrying its evidence, including the contradicting studies that had forced its refinement. A commissioning authority used it to redesign rather than abandon its scheme, which is the method’s definition of success.

• Systematic review with realist vocabulary. Exhaustive PRISMA searching, design-based inclusion, then CMOs bolted on at write-up — the hybrid that satisfies neither standard.
• Mechanisms that are activities. “Training was delivered” is not a mechanism; the mechanism is the change in participants’ reasoning or resources the training triggers.
• Unbounded theory sets. Testing every elicited theory guarantees a review that never converges; prioritisation is a methodological act, not a confession.
• Context as demographics. Realist context is whatever enables or blocks the mechanism — trust, discretion, resource slack — not a table of population characteristics.
• Configurations without evidence chains. RAMESES exists because early reviews asserted CMOs the reader could not trace; every configuration needs its sources on display.

Realist synthesis trades reproducibility for explanatory reach: two teams with the same question will elicit and prioritise different theories, and the method’s answer — transparency of reasoning, RAMESES reporting — bounds but does not remove that.

It is hungry for evidence the literature under-produces: mechanism-probing process and qualitative studies. Where only outcome trials exist, mechanism claims lean on inference, and honest reviews say so.

And the outputs demand more of readers than a forest plot: configurations with conditions attached resist headline simplification. The transfer to policy audiences is a writing task the method sets and does not solve.

Realist evaluation applies the same CMO logic to a single live programme rather than a literature.

A systematic literature review remains the right instrument for average-effect questions on standardised interventions.

Contribution analysis builds and tests a causal story for one programme where a realist synthesis refines theory across many studies.

Meta-synthesis integrates qualitative findings interpretively without the realist explanatory apparatus.

A realist synthesis is a coding problem over a heterogeneous corpus: dozens to hundreds of documents, each mined for evidence bearing on specific theories. Evidano applies the candidate CMOs as a codebook across the PDFs, links every extracted fragment to its passage, and lets the reviewer interrogate the corpus mechanism by mechanism — including the UN-scale precedent: a published realist evaluation used it to code 298 evaluation reports at 92% agreement with human coders, identifying 132 CMO configurations.

Theory elicitation, prioritisation, and the refinement judgements are the reviewer’s craft; the tool makes the evidence-to-configuration chain complete and auditable, which is exactly what RAMESES reporting demands.