Small Trials And Why Early Positive Results Mislead

Most trials of traditional medicines are small, and small trials behave in ways that regularly mislead readers. The statistical reasons are unglamorous and they explain a recurring pattern in this literature.

Small samples scatter widely

With few participants, the measured difference between two groups is dominated by chance variation between individuals rather than by any effect of the treatment.

Run the same small study repeatedly and the results scatter widely around the true value, producing some striking positives and some striking negatives purely by chance.

This is not a flaw in any individual study. It is an unavoidable property of measurement with few observations, and it cannot be argued away.

Which of those results get published

Positive results are more likely to be written up, submitted and accepted than null results, particularly when the study is small and the finding is novel.

The published record therefore over-represents the flattering tail of the scatter, and a reader searching for evidence on a herb encounters that tail first.

Trial registration was introduced partly to address this, by making it possible to see which studies were begun and never reported.

The winner's curse

When a small study does cross the threshold for statistical significance, the effect it reports is systematically larger than the true effect, because only large estimates cross the line.

The consequence is that the first positive result for a treatment is usually its most impressive one, and later, larger studies report something smaller.

Effects that shrink as studies get bigger are the norm rather than a sign of interference, and readers should expect the pattern rather than being surprised by it.

What multiple comparisons add

A trial measuring many outcomes has many chances to find something, and reporting the one that reached significance while omitting the rest inflates the apparent finding.

Pre-registered primary outcomes exist to prevent this, and a study that reports an outcome different from the one it registered deserves scepticism.

The same applies to subgroup findings, where an effect appearing only in one age band or one sex is far more often noise than discovery.

How to read a promising early result

Treat a small positive trial as a reason to run a larger one, not as evidence that something works, and look for whether the larger one ever appeared.

Check the sample size, the registered outcome and whether replication exists, because those three questions filter out most of what will not survive.

None of this means traditional medicines do not work. It means that the early literature about them cannot settle the question either way.