In meta-analysis, when the difference in results between studies is greater than would be expected by chance, one needs to investigate whether the observed variation in results across studies is associated with clinical and/or methodological differences between studies. This article reviews methods used in meta-analysis for exploring heterogeneity, including statistical tests for homogeneity, methods for visually displaying results of primary studies, methods for reducing heterogeneity, methods for investigating sources of heterogeneity, and identification of moderator variables or effect modifiers. The investigation of sources of heterogeneity in meta-analysis is by nature exploratory, and therefore its results should always be interpreted with caution. However, careful investigation of heterogeneity may provide an important second level of evidence that can be useful in suggesting direction of future research. Sometimes, it may provide clinically important results by indicating who might benefit more or less from a treatment or how an intervention should be applied.
Get full access to this article
View all access options for this article.
References
1.
Anello, C.
, & Fleiss, J. L. (1995). Exploratory or analytic meta-analysis: Should we distinguish between them?Journal of Clinical Epidemiology, 48, 109–116.
2.
Atallah, A. N.
, Hofmeyr, G. J., & Duley, L. (2000). Calcium supplementation during pregnancy for preventing hypertensive disorders and related problems [CD-ROM]. From: The Cochrane Library, (3). Oxford, UK: Update Software.
3.
Ausejo, M.
, Saenz, A., Pham, B., Kellner, J. D., Johnson, D. W., & Moher, D. (1999). The effectiveness of glucocorticoids in treating croup: Meta-analysis. British Medical Journal, 319, 595–600.
4.
Bailar, J. C.
(1997). The promise and problems of meta-analysis. New England Journal of Medicine, 337, 559–560.
5.
Bailey, K. R.
(1987). Inter-study differences: How should they influence the interpretation and analysis of results?Statistics in Medicine, 6, 351–358.
6.
Begg, C. B.
, & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101.
7.
Berkey, C. S.
, Hoaglin, D. C., Mosteller, F., & Colditz, G. A. (1995). A random-effects regression model for meta-analysis. Statistics in Medicine, 14, 395–411.
8.
Berlin, J. A.
, & Antman, E. M. (1994). Advantage and limitations of meta-analytic regressions of clinical trials data. Online Journal of Current Clinical Trials. Document number 134.
9.
Berlin, J. A.
, Laird, N. M., Sacks, H. S., & Chalmers, T. C. (1989). A comparison of statistical methods for combining event rates from clinical trials. Statistics in Medicine, 8, 141–151.
10.
Brand, R.
, & Kragt, H. (1992). Importance of trends in the interpretation of an overall odds ratio in the meta-analysis of clinical trials. Statistics in Medicine, 11, 2077–2082.
11.
Breslow, N. E.
, & Day, N. E. (1980). Classical methods of analysis of grouped data. In N. E. Breslow & N. E. Day (Eds.), Statistical methods in cancer research: Volume 1—The analysis of case-control studies (pp. 122–148). Lyon, France: International Agency for Research on Cancer.
12.
Cappelleri, J. C.
, Ioannidis, J.P.A., Schmid, C. H., de Ferranti, S. D., Aubert, M., Chalmers, T. C., & Lau, J. (1996). Large trials vs. meta-analysis of smaller trials. How do their results compare?Journal of American Medical Association, 276, 1332–1338.
13.
Clarke, M.
, Stewart, L., Pignon, J. P., & Bijnens, L. (1998). Individual patient data meta-analyses in cancer. British Journal of Cancer, 77, 2036–2044.
14.
Colditz, G. A.
, Brewer, T. F., Berkey, C. S., Wilson, M. E., Burdick, E., Fineberg, H. V., & Mosteller, F. (1994). Efficacy of BCG vaccine in the prevention of tuberculosis. Meta-analysis of the published literature. Journal of the American Medical Association, 271, 698–702.
15.
Davey-Smith, G.
, Egger, M., & Phillips, A. N. (1997). Meta-analysis: Beyond the grand mean?British Medical Journal, 315, 1610–1614.
16.
Davey-Smith, G.
, Song, F., & Sheldon, T. A. (1993). Cholesterol lowering and mortality: The importance of considering initial level of risk. British Medical Journal, 306, 1367–1373.
17.
DerSimonian, R.
, & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177–188.
18.
DerSimonian, R.
, & Levine, R. J. (1999). Resolving discrepancies between a meta-analysis and a subsequent large controlled trial. Journal of the American Medical Association, 282, 664–670.
19.
Duval, S.
, & Tweedie, R. (2000). Trim and fill: A simple funnel plot based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 50, 455–463.
20.
Egger, M.
, Davey-Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634.
21.
Engels, E. A.
, Schmid, C. H., Terrin, N., Olkin, I., & Lau, J. (2000). Heterogeneity and statistical significance in meta-analysis: An empirical study of 125 meta-analyses. Statistics in Medicine, 19, 1707–1728.
22.
Galbraith, R. F.
(1988). A note on graphical presentation of estimated odds ratios from several clinical trials. Statistics in Medicine, 7, 889–894.
23.
Gill, D.
, & Hatcher, S. (2000). Antidepressants for the depressed physically ill [CD-ROM]. From: The Cochrane Library (3). Oxford, UK: Update Software.
24.
Gotzsche, P. C.
, Podenphant, J., Olesen, M., & Halberg, P. (1992). Meta-analysis of second-line antirheumatic drugs: Sample size bias and uncertain benefit. Journal of Clinical Epidemiology, 45, 587–594.
25.
Greenland, S.
(1994). Invited commentary: A critical look at some popular meta-analytic methods. American Journal of Epidemiology, 140, 290–296.
26.
Hall, J. A.
, Rosenthal, R., Tickle-degnen, L., & Mosteller, F. (1994). Hypotheses and problems in research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 17–28). New York: Russell Sage.
27.
Hardy, R. J.
, & Thompson, S. G. (1998). Detecting and describing heterogeneity in meta-analysis. Statistics in Medicine, 17, 841–856.
28.
Helfenstein, U.
, & Steiner, M. (1993). Fluoride varnishes (duraphat): A meta-analysis. Community Dentistry and Oral Epidemiology, 22, 1–5.
29.
L’Abbe, K. A.
, Detsky, A. S., & O’Rourke, K. (1987). Meta-analysis in clinical research. Annals of Internal Medicine, 107, 224–233.
30.
Lau, J.
, Ioannidis, J. P., & Schmid, C. H. (1998). Summing up evidence: One answer is not always enough. Lancet, 351, 123–127.
31.
Lau, J.
, Schmid, C. H., & Chalmers, T. C. (1995). Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. Journal of Clinical Epidemiology, 48, 45–57.
32.
LeLorier, J.
, Gregoire, G., Benhaddad, A., Lapierre, J., & Derderian, F. (1997). Discrepancies between meta-analyses and subsequent large randomized, controlled trials. New England Journal of Medicine, 337, 536–542.
33.
Light, R. J.
(1987). Accumulating evidence from independent studies: What we can win and what we can lose. Statistics in Medicine, 6, 221–228.
34.
Light, R. J.
, & Pillemer, D. B. (1984). Summing up: The science of reviewing research. Cambridge, MA: Harvard University Press.
35.
Limburg, H. J.
(2000). Calcium antagonists for acute ischemic stroke [CD-ROM]. From: The Cochrane Library (3). Oxford, UK: Update Software.
36.
Lipsey, M. W.
(1994). Identifying potentially interesting variables and analysis opportunities. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 111–123). New York: Russell Sage.
37.
McIntosh, M. W.
(1996). The population risk as an explanatory variable in research synthesis of clinical trials. Statistics in Medicine, 15, 1713–1728.
38.
Morgenstern, H.
(1982). Uses of ecologic analysis in epidemiologic research. American Journal of Public Health, 72, 1336–1344.
39.
Olkin, I.
(1999). Diagnostic statistical procedures in medical meta-analyses. Statistics in Medicine, 18, 2331–2341.
40.
Petticrew, M.
, Gilbody, S., & Sheldon, T. (1999). Evidence does not support link: Relation between hostility and coronary heart disease. British Medical Journal, 319, 917-917.
41.
Pignon, J. P.
, Arriagada, R., Ihde, D. C., Johnson, D. H., Perry, M. C., Souhami, R. L., Brodin, O., Joss, R. A., Kies, M. S., Lebeau, B., Onoshi, T., Osterlind, K., Tattersall, M.H.N., & Wagner, H. (1992). A meta-analysis of thoracic radiotherapy for small-cell lung cancer. New England Journal of Medicine, 327, 1618–1624.
42.
Pittler, M. H.
, & Ernst, E. (1999). Artemether for severe malaria: A meta-analysis of randomized clinical trials. Clinical Infectious Diseases, 28, 597–601.
43.
Poole, C.
, & Greenland, S. (1999). Random-effects meta-analyses are not always conservative. American Journal of Epidemiology, 150, 469–475.
44.
Rothwell, P. M.
, Slattery, J., & Warlow, C. P. (1996). A systematic review of the risks of stroke and death due to endarterectomy for symptomatic carotid stenosis. Stroke, 27, 260–265.
45.
Schmid, C.
, Lau, J., McIntosh, M., & Cappelleri, J. (1998). An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. Statistics in Medicine, 17, 1923–1942.
46.
Schmid, C. H.
(1999). Exploring heterogeneity in randomized trials via meta-analysis. Drug Information Journal, 33, 211–224.
47.
Senn, S.
(1994). Letters to the editor. Importance of trends in the interpretation of an overall odds ratio in the meta-analysis of clinical trials. Statistics in Medicine, 13, 293–296.
48.
Sharp, S. J.
, Thompson, S. G., & Altman, D. G. (1996). The relation between treatment benefit and underlying risk in meta-analysis. British Medical Journal, 313, 735–738.
49.
Shukla, V. K.
, Otten, N., Dube, C., & Moher, D. (2000). Use of cisapride in patients with non-ulcer dyspepsia: A meta-analysis of randomized trials. Ottawa: Canadian Coordinating Office for Health Technology Assessment.
50.
Song, F.
(1999). Exploring heterogeneity in meta-analysis: Is the L’Abbé plot useful?Journal of Clinical Epidemiology, 52, 725–730.
51.
Song, F.
, Eastwood, A., Gilbody, S., Duley, L., & Sutton, A. (2000). Publication and related biases [Monograph]. Health Technology Assessment, 4(10).
52.
Spector, T. D.
, & Thompson, S. G. (1991). The potential and limitations of meta-analysis. Journal of Epidemiology and Community Health, 45, 80–92.
53.
Sutton, A. J.
, Abrams, K. R., Jones, D. R., Sheldon, T. A., & Song, F. (2000). Methods for meta-analysis in medical research. New York: John Wiley.
54.
Takkouche, B.
, Cadarso-Suarez, C., & Spiegelman, D. (1999). Evaluation of old and new tests of heterogeneity in epidemiologic meta-analysis. American Journal of Epidemiology, 150, 206–215.
55.
Tang, J. L.
, & Liu, J. L. (2000). Misleading funnel plot for detection of bias in meta-analysis. Journal of Clinical Epidemiology, 53, 477–484.
56.
Thompson, S.
, Smith, T., & Sharp, S. (1997). Investigating underlying risk as a source of heterogeneity in meta-analysis. Statistics in Medicine, 16, 2741–2758.
57.
Thompson, S. G.
(1994). Why sources of heterogeneity in meta-analysis should be investigated. British Medical Journal, 309, 1351–1355.
58.
van-Houwelingen, H.
, & Senn, S. (1999). Letters to the editor: Investigating underlying risk as a source of heterogeneity in meta-analysis. Statistics in Medicine, 18, 110–115.
59.
Villar, J.
, Carroli, G., & Belizan, J. M. (1995). Predictive ability of meta-analyses of randomized controlled. Lancet, 345, 772–776.
60.
Walter, S. D.
(1997). Variation in baseline risk as an explanation of heterogeneity in meta-analysis. Statistics in Medicine, 16, 2883–2900.
61.
Zalcberg, J. R.
, Siderov, J., & Simes, J. (1996). The role of 5-fluorouracil dose in the adjuvant therapy of colorectal cancer. Annals of Oncology, 7, 41–46.
