This is the first paragraph from Ben Goldacre’s recent comment piece Benefits and risks of homeopathy in The Lancet‘s November 17 edition.
Five large meta-analyses of homoeopathy trials have been done. All have had the same result: after excluding methodologically inadequate trials and accounting for publication bias, homoeopathy produced no statistically significant benefit over placebo. 1–5
(1) Kleijnen J, Knipschild P, ter Riet G. Clinical trials of homoeopathy. BMJ 1991; 302: 316–23.
(2) Boissel JP, Cucherat M, Haugh M, Gauthier E. Critical literature review on the effectiveness of homoeopathy: overview of data from homoeopathic medicine trials. Brussels, Belgium: Homoeopathic Medicine Research Group. Report to the European Commission. 1996: 195–210.
(3) Linde K, Melchart D. Randomized controlled trials of individualized homeopathy: a state-of-the-art review. J Alter Complement Med 1998; 4: 371–88.
(4) Cucherat M, Haugh MC, Gooch M, Boissel JP. Evidence of clinical efficacy of homeopathy: a meta-analysis of clinical trials. Eur J Clin Pharmacol 2000; 56: 27–33
(5) Shang A, Huwiler-Müntener K, Nartey L, et al. Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy. Lancet 2005; 366: 726–32
Note that Goldacre omits the Linde et al meta-analysis published in The Lancet in 1997 (6) from his listed studies.
Below are comments and conclusions from each of these studies. Remember, Goldacre is saying that they each support his assertion that homeopathy has no statistically significant benefit over placebo.
The 1991 study by Jos Kleijnen, Paul Knipschild and Gerben ter Riet assessed the methodological quality of 107 controlled trials in 96 published reports found after an extensive search. Trials were scored using a list of predefined criteria of good methodology, and the outcome of the trials was interpreted in relation to their quality. In 14 trials some form of classical homoeopathy was tested and in 58 trials the same single homoeopathic treatment was given to patients with comparable conventional diagnoses. Combinations of several homoeopathic treatments were tested in 26 trials; isopathy was tested in nine trials. Most trials seemed to be of very low quality, but there were many exceptions. The results showed a positive trend regardless of the quality of the trial or the variety of homoeopathy used. Overall, of the 105 trials with interpretable results, 81 trials indicated positive results whereas in 24 trials no positive effects of homoeopathy were found.Within the discussion of their findings, the authors state:
“The amount of positive evidence even among the best studies came as a surprise to us. Based on this evidence we would be ready to accept that homoeopathy can be efficacious, if only the mechanism of action were more plausible.”
And go on to say:
“The way in which the belief of people changes after the presentation of empirical evidence depends on their prior beliefs and on the quality of the evidence. Critical people who did not believe in the efficacy of homoeopathy before reading the evidence presented here probably will still not be convinced; people who were more ambivalent in advance will perhaps have a more optimistic view now, whereas people who already believed in the efficacy of homoeopathy might at this moment be almost certain that homoeopathy works.”
“At the moment the evidence of clinical trials is positive but not sufficient to draw definitive conclusions because most trials are of low methodological quality and because of the unknown role of publication bias. This indicates that there is a legitimate case for further evaluation of homoeopathy, but only by means of well performed trials.”
I don’t have a copy of this paper. This comment on it is extracted from a 2002 Italian literature review (Cornelli, Prof Umberto et al):
“These experts identified 377 clinical trials, short-listed 220, and reviewed 184. Detailed research lasting several months was conducted on the best trials, to evaluate their scientific value.The conclusions researched by the Advisory Group are unequivocal: the number of significant results cannot be attributed to chance. The analysis provided a random hypothesis value of p < 0.001.The Advisory Group remained very cautious, but expressly stated: “The null hypothesis that homeopathy has no effect can be rejected with certainty; in other words, in at least one of the studies examined the patients treated with the homeopathic remedy received benefits compared with the control patients who received the placebo”.”
Klaus Linde and Dieter Melchart’s paper on Randomized Controlled Trials of Individualized Homeopathy was a limited study designed to test a subset of homeopathic trials. Because of its small size, it’s consequently not a particularly appropriate study to cite in support of generalised conclusions about the therapy as a whole, even if it does test trials that attempt to adhere more closely to homeopathy as it’s practiced.
“Randomized or quasirandomized controlled clinical trials comparing an individualized homeopathic treatment strategy with placebo, no treatment, or another treatment were eligible. Information on patients, methods, interventions, outcomes, and results was extracted in a standardized manner and quality was assessed using a checklist and two scoring systems. Trials providing sufficient data were pooled in a quantitative meta-analysis.”
“A total of 32 trials (28 placebo-controlled, 2 comparing homeopathy and another treatment, 2 comparing both) involving a total of 1778 patients met the inclusion criteria. The methodological quality of the trials was highly variable. In the 19 placebo-controlled trials providing sufficient data for meta-analysis, individualized homeopathy was significantly more effective than placebo (pooled rate ratio 1.62, 95% confidence interval 1.17 to 2.23), but when the analysis was restricted to the methodologically best trials no significant effect was seen.”
“The results of the available randomized trials suggest that individualized homeopathy has an effect over placebo. The evidence, however, is not convincing because of methodological shortcomings and inconsistencies. Future research should focus on replication of existing promising studies. New randomized studies should be preceded by pilot studies.”
This paper was financed by the European Commission and undertaken by the Homoeopathic Medicine Research Group as in (2) above.
“The selection criteria were randomised, controlled trials in which the efficacy of homeopathic treatment was assessed relative to placebo in patients using clinical or surrogate endpoints. Prevention trials or those evaluating only biological effects were excluded. One hundred and eighteen randomised trials were identified and evaluated for inclusion. Sixteen trials, representing 17 comparisons and including a total of 2617 evaluated patients, fulfilled the inclusion criteria … The reasons for exclusion of the remaining 102 trials were primary outcome not clearly defined (92, 90%) and methodological defects (10, 10%).”
“The signficant combined P value obtained in the main analysis does not imply that the homeopathic treatments were efficacious in all the pooled comparisons. This result provides evidence that in at least one trial the homeopathic treatment was more efficacious than placebo. In other words, more trials had a positive result than would be expected due to chance alone.“
“Although we cannot exclude the possibly that the results of the meta-analysis are affected by publication bias, the results of the sensitivity analysis suggest that this is unlikely.”
“From the available evidence, it is likely that among the tested homeopathic treatments tested at least one shows an added effect relative to placebo. The meta-analysis method used does not allow any conclusion on what homeopathic treatment is effective in which diagnosis or against which symptoms. It is of no more practical value than to answer yes to the question “are homeopathic treatments effective?” without specifying which drug?, which dose or regimen? and against which disease?However, the strength of the evidence for this conclusion remains low because of the overall low quality of the trial designs and reporting and the limitations of the meta-analysis approach used.
It is clear that the strength of available evidence is insufficient to conclude that homeopathy is clinically effective; however, homeopathy can and should be assessed using the same methodology used for allopathy. More well-designed and well-run clinical trials, including many hundreds of patients, are needed before definitive conclusions can be drawn regarding the clinical efficacy of homeopathic treatments.”
The Shang et al meta-analysis published in The Lancet in August 2005 and trumpeted in a blaze of nationwide publicity as “the end of homeopathy” really deserves an entry all to itself. It attracted widespread condemnation and its methodological weaknesses, coupled with its blatant violations of transparency, have led many reviewers to state emphatically that it should never have passed peer review.
“When orthodox scientists, statisticians, molecular chemists, clinicians, and mathematicians, and rigorous, scientifically trained, academic clinical homeopaths begin corresponding in response to the publication of a paper in a learned journal, to draw attention to a serious scientific error, quite apart from its associated moral and ethical implications, and when letter after letter, quietly reasoned, and objectively critical of the original publication, is rejected by the initiating journal, it is surely time to reflect very deeply on what might be taking place and to ask “why?””
Jobst, K A. Homeopathy, Hahnemann, and The Lancet 250 Years On: A Case of the Emperor’s New Clothes? Journal of Alternative & Complementary Medicine. Volume 11, Number 5, 2005, pp. 751–754
Of the total 220 matched trials (110 for each therapy), the authors identified 21 “high-quality” homeopathic studies, and 9 “high-quality” conventional studies. No comparative analysis of this subset of trials was presented. The authors then proceeded to further select the small subset of purportedly larger and higher methodological quality trials (8 homeopathy trials and 6 conventional medicine trials) from which the paper’s conclusion that homeopathy is no better than placebo is drawn, but failed to describe the weighting of their selection attributes (size and methodological quality). They didn’t explain how they chose the particular cut off point that they used to select the final 14 trials, neither did they identify which trials these were, neither did they provide any information on which to assess whether those trails were still matched. Initial requests by other researchers to identify these trials were refused repeatedly and data finally permitting their identification was not made available until several months after.
Most telling of all perhaps is the following from the Discussion section of the paper:
“We assumed that the effects observed in placebo-controlled trials of homoeopathy could be explained by a combination of methodological deficiencies and biased reporting. Conversely, we postulated that the same biases could not explain the effects observed in comparable placebo-controlled trials of conventional medicine.”
This a priori assumption means that essentially the study can do nothing but produce a null result. Having filtered for methodological deficiency, the assumption is that any positive effect remaining must be bias. By those standards, the only course of action left is to remove all the trials with any positive effect and leave the ones that show no benefit.
The authors do at least admit that it they’d chosen a different set of 8 homeopathy trials, the results would have been quite different:
“For example, for the eight trials of homoeopathic remedies in acute infections of the upper respiratory tract that were included in our sample, the pooled effect indicated a substantial beneficial effect (odds ratio 0·36 [95% CI 0·26–0·50]) and there was neither convincing evidence of funnel-plot asymmetry nor evidence that the effect differed between the trial classified as of higher reported quality and the remaining trials. Such sensitivity analyses might suggest that there is robust evidence that the treatment under investigation works. However, the biases that are prevalent in these publications, as shown by our study, might promote the conclusion that the results cannot be trusted.”
The study’s designer, Matthias Egger, is well known for his anti-homeopathy stance. His assumption that any positive effect from homeopathy, after allowing for methodological deficiency, is due to publication bias merely highlights the extent of his own bias.
Linde K, Clausius N, Ramirez G, Melchart D, Eitel F, Hedges LV, Jonas WB. Are the clinical effects of homeopathy placebo effects? A meta-analysis of placebo-controlled trials. The Lancet. 1997 Sep 20;350(9081):834-43.This is the meta-analysis which Goldacre omitted to mention.
“Double-blind and/or randomised placebo-controlled trials of clinical conditions were considered. Our review of 186 trials identified 119 that met the inclusion criteria. 89 had adequate data for meta-analysis, and two sets of trial were used to assess reproducibility.”
“The combined odds ratio for the 89 studies entered into the main meta-analysis was 2·45 (95% CI 2·05, 2·93) in favour of homoeopathy. The odds ratio for the 26 good-quality studies was 1·66 (1·33, 2·08), and that corrected for publication bias was 1·78 (1·03, 3·10).”
“We believe that a serious effort to research homoeopathy is clearly warranted despite its implausibility.”
“The results of our meta-analysis are not compatible with the hypothesis that the clinical effects of homoeopathy are completely due to placebo. However, we found insufficient evidence from these studies that homoeopathy is clearly efficacious for any single clinical condition.”
The evidence from these studies is not conclusive either way. The general consensus is that there seems to be an effect greater than placebo, but it’s not very strong in trials of higher quality and that more studies, and more studies of different kinds testing the therapy in different ways, are needed. This is a long long way from supporting Goldacre’s null hypothesis.
For more comment, see this annex to The European Committee for Homeopathy’s press release on The Lancet’s November 17 issue.