What A Systematic Review Can And Cannot Settle

Systematic reviews are treated as the strongest form of evidence, and reviews of Ayurvedic interventions are frequently cited as though they settle a question. What they can settle depends entirely on what went into them.

What the method actually does

A systematic review specifies a question, searches for all studies meeting stated criteria, assesses their quality, and summarises what the collected evidence shows.

The value lies in the pre-specified method, which prevents an author from assembling only the studies that support a preferred conclusion.

A meta-analysis goes further and pools the numerical results, producing a single combined estimate with narrower confidence limits than any individual study.

Pooling does not improve the inputs

Combining ten small biased studies produces a precise estimate of a biased result, and the narrow confidence interval can make that estimate look more solid than it is.

Good reviews address this by rating the certainty of the evidence separately from the size of the effect, and by conducting analyses restricted to higher quality studies.

A reader should always look for that certainty rating, because a large-looking effect graded as very low certainty is a different finding from the same effect graded as moderate.

Heterogeneity is the recurring problem

Trials of a single herb may differ in preparation, extract standardisation, quantity, duration, population and outcome measure, which makes pooling them questionable.

Reviews report statistical measures of heterogeneity, and high values indicate that the studies are not estimating the same thing and that a combined figure may be meaningless.

In traditional medicine reviews, heterogeneity is often high, which is a signal to read the individual studies rather than trust the summary line.

What gets included shapes the result

Language restrictions, database choice and the handling of unpublished studies all change which trials are found, and reviews of traditional medicine are sensitive to all three.

Funnel plots and related methods can suggest that small negative studies are missing, though these tools are imperfect and require a reasonable number of studies to work.

Reviews conducted by groups with a stake in the intervention have been found across many fields to reach more favourable conclusions, which is a reason to check authorship.

Reading a review usefully

The conclusion sentence is the least informative part. The certainty rating, the heterogeneity figure, the number and size of included trials, and the risk-of-bias table carry the real content.

A review concluding that evidence is insufficient is a genuine finding and not a failure, and it is the honest outcome for many herbal questions at present.

None of this should be used to make treatment decisions alone, and a review supporting an intervention is still not a substitute for clinical advice about a particular person.