How to read a trial of a traditional medicine

A steady stream of studies on traditional medicines is published, and a subset of them are reported enthusiastically. Most of the assessment can be done with a handful of questions.

How many participants, and for how long

A large share of trials in this field are small — a few dozen participants — and short.

Small trials produce unstable results. A striking effect in twenty people per arm can easily arise by chance, and the published literature is biased toward striking results because null findings are less likely to be submitted and accepted.

Short trials cannot tell you about durability or about safety over the timescale people actually use these products.

Was it randomised, blinded and placebo-controlled

Randomisation prevents systematic differences between groups. Blinding prevents expectation from producing the result.

Blinding is genuinely difficult with herbal preparations that have distinctive taste and smell, and with procedural interventions such as massage or panchakarma, where it may be impossible. That is a real constraint rather than negligence, and it means results from unblinded trials should be weighted accordingly — particularly for subjective outcomes such as pain, wellbeing or fatigue, which are the outcomes most susceptible to expectation.

An unblinded trial with a subjective primary outcome tells you very little.

What was the comparison

Compared with nothing, most things appear to work. The informative comparisons are against placebo, or against an established treatment.

Watch for trials comparing an intervention plus standard care against standard care alone, without a placebo — any difference includes the effect of receiving additional attention.

What was measured, and was it what they said they would measure

Pre-registration of trials, with a stated primary outcome, exists to prevent outcome switching — measuring many things and reporting the ones that reached significance.

Check whether the trial was registered and whether the reported primary outcome matches the registered one. This is easier than it sounds; registries are public.

Also check whether the outcome is clinically meaningful. A change in a laboratory marker is not the same as a change in how someone feels or functions.

How large was the effect

Statistical significance means a result was unlikely under the null hypothesis. It says nothing about size.

Look for effect sizes and confidence intervals, and ask whether the difference would matter to a person. A statistically significant improvement of a few points on a hundred-point scale is a real finding and a clinically trivial one.

Where was it conducted, and who funded it

Research on traditional medicines is heavily concentrated geographically, which raises questions about independent replication.

Studies conducted exclusively in institutions with an institutional commitment to the tradition being tested are not disqualified by that fact, and they benefit from independent confirmation elsewhere.

Funding from a manufacturer of the specific formulation tested is common in this field and should be stated. It does not invalidate a result and it is a known source of bias in outcomes across medicine generally.

Was it in humans

A very large proportion of publications on traditional medicines are cell culture or animal studies.

These are legitimate research and they are not evidence that something works in people. The attrition between promising laboratory findings and effective human treatments is enormous across all of pharmacology.

Headlines derived from cell studies — a compound killing cancer cells in a dish, for instance — describe something true and almost entirely uninformative about clinical use.

What was actually tested

This is a particular problem in herbal research. Preparations vary enormously in the plant part used, the extraction method, the concentration of active constituents, and the presence of other ingredients.

A trial of a specific standardised extract does not tell you about a different product with the same plant name on the label, and the difference can be substantial.

Systematic reviews, and their limits

Reviews pooling multiple trials are generally more reliable than single studies, and they inherit the quality of what they include.

A meta-analysis of fifteen small unblinded trials is a summary of fifteen weak studies, and a good review will say so explicitly in its quality assessment. Read that section.

The useful default

Treat a single positive trial as a reason for interest rather than a reason to act. Look for independent replication, in adequately sized and blinded trials, measuring outcomes that matter, before concluding much.

That standard is applied to conventional pharmaceuticals and there is no principled reason to apply a weaker one to traditional preparations — which are, after all, being taken for the same purposes and carry their own risks.