To systematically review methods for systematic review/meta-analysis in order to identify the different methodological and statistical methods that have been proposed. A summary of the main findings is presented here, with emphasis given to health services research topics.
Methods:
A thorough systematic search for methodological papers was carried out using a variety of methods, including the use of electronic databases. Approximately 1000 potentially relevant references were identified, a number of them from education, psychology and sociology.
Results:
After briefly reviewing the procedural methods required to carry out a review, and the basic statistical methods used to combine study estimates, less established methods are discussed. These include methods for dealing with publication bias, meta-regression, meta-analysis of individual patient data, the synthesis of nonrandomized evidence alone and in combination with randomized studies, Bayesian modelling and economic evaluation through meta-analysis. Recommendations for meta-analytical practice are given; these are either distilled from previous guidelines, or constructed where there appears to be a broad consensus across the literature.
Conclusions:
It is hoped that this review will provide a consistent and comprehensive, but concise, description of the methods available for synthesizing evidence, that it will promote better quality reviews of the results of health services research and identify specific areas which require methodological development.
Get full access to this article
View all access options for this article.
References
1.
BucherHCSchmidtJG.Does routine ultrasound scanning improve outcome in pregnancy? Meta-analysis of various outcome measures. BMJ1993; 307: 13–17
2.
GrimshawJMRussellIT.Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet1993; 342: 1317–1322
3.
BalasEABorenSABrownGDEwigmanBGMitchellJAPerkoffGT.Effect of physician profiling on utilization. Meta-analysis of randomized clinical trials. Journal of General Internal Medicine1996; 11: 584–590
4.
DeeksJGlanvilleJSheldonT.Undertaking systematic reviews of research on effectiveness: CRD guidelines for those carrying out or commissioning reviews. Report no. 4. York: CRD, 1996
BlairABurgJForanJGibbHGreenlandSMorrisRGuidelines for application of meta-analysis in environmental epidemiology. ISLI Risk Science Institute. Regulatory Toxicology and Pharmacology1995; 22: 189–197
7.
CookDJSackettDLSpitzerWO.Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on Meta-Analysis. Journal of Clinical Epidemiology1995; 48: 167–171
8.
SuttonAJAbramsKRJonesDRSheldonTASongF.Systematic reviews of trials and other studies. Health Technology Assessment1999; 2 (19)
CookDJReeveBKGuyattGHHeylandDKGriffithLEBuckinghamLStress-ulcer prophylaxis in critically ill patients — resolving discordant meta-analyses. Journal of the American Medical Association1996; 275: 308–314
11.
MoherDJadadARNicholGPenmanMTugwellPWalshS.Assessing the quality of randomized controlled trials — an annotated bibliography of scales and checklists. Controlled Clinical Trials1995; 12: 62–73
12.
MorrisRD.Meta-analysis in cancer epidemiology. Environmental Health Perspectives1994; 102 (suppl 8): 61–66
13.
HedgesLVOlkinI.Statistical methods for meta-analysis. London: Academic Press, 1985
14.
EmersonJD.Combining estimates of the odds ratio: the state of the art. Statistical Methods for Medical Research1994; 3: 157–178
15.
GreenlandSSalvanA.Bias in the one-step method for pooling study results. Statistics in Medicine1990; 9: 247–252
16.
DerSimonianRLairdN.Meta-analysis in clinical trials. Controlled Clinical Trials1986; 7: 177–188
17.
BiggerstaffBJTweedieRL.Incorporating variability in estimates of heterogeneity in the random effects model in meta-analysis. Statistics in Medicine1997; 16: 753–768
18.
PetoR.Why do we need systematic overviews of randomized trials?Statistics in Medicine1987; 6: 233–240
19.
ThompsonSG.Controversies in meta-analysis: the case of the trials of serum cholesterol reduction. Statistical Methods in Medical Research1993; 2: 173–192
20.
HedgesLV.Fixed effects models. In: CooperHHedgesLV, eds. The Handbook of Research Synthesis. New York: Russell Sage Foundation, 1994: 285–300
21.
Random effects models. In: CooperHHedgesLV, eds. The Handbook of Research Synthesis. New York: Russell Sage Foundation, 1994: 301–322
22.
SmithGDSongFSheldonTASongFJ.Cholesterol lowering and mortality: the importance of considering initial level of risk. BMJ1993; 306: 1367–1373
23.
SharpSJThompsonSGAltmanDG.The relation between treatment benefit and underlying risk in meta-analysis. BMJ1996; 313: 735–738
24.
SowdenASheldonTA.Does volume really affect outcome? Lessons from the evidence. Journal of Health Services Research and Policy1998; 3: 187–190
25.
BerryDAStanglDK.Bayesian biostatistics. New York: Marcel Dekker, 1996
26.
SmithTCSpiegelhalterDJThomasA.Bayesian approaches to random-effects meta-analysis: a comparative study. Statistics in Medicine1995; 14: 2685–2699
27.
National Research Council. Combining information: statistical issues and opportunities for research. Washington, DC: National Academy Press, 1992
28.
TweedieRLScottDJBiggerstaffBJMengersenKL.Bayesian meta-analysis, with application to studies of ETS and lung cancer. Lung Cancer1996; 14: S171–194
29.
KadaneJBWolfsonLJ.Experiences in elicitation. Journal of the Royal Statistical Society Series D1998; 47: 3–19
30.
AbramsKRSansoB.Approximate Bayesian inference in random effects meta-analysis. Statistics in Medicine1998; 17: 201–218
31.
GivensGHSmithDDTweedieRL.Publication bias in meta-analysis: a Bayesian data-augmentation approach to account for issues exemplified in the passive smoking debate. Statistical Science1997; 12: 221–250
32.
HigginsJPTWhiteheadA.Inclusion of both patient level and study-level covariates in a meta-analysis. Controlled Clinical Trials1997; 18: 84S
33.
SmithTCAbramsKRJonesDR.Using hierarchical models in generalized synthesis of evidence: an example based on studies of breast cancer screening. Department of Epidemiology and Public Health technical report 95–02. Leicester: University of Leicester, 1995
34.
GrilliRMinozziSTinazziALabiancaRSheldonTALiberatiA.Do specialists do it better? The impact of specialisation on the process and outcomes of care for cancer patients. Annals of Oncology1998; 9: 365–374
35.
SpitzerWO.Meta-meta-analysis: unanswered questions about aggregating data. Journal of Clinical Epidemiology1991; 44: 103–107
36.
BlackN.Why we need observational studies to evaluate the effectiveness of health care. BMJ1996; 312: 1215–1218
37.
EddyDMHasselbladVMcGivneyWHendeeW.The value of mammography screening in women under the age of 50 years. Journal of the American Medical Association1988; 259: 1512–1519
38.
EddyDMHasselbladVShachterR.Meta-analysis by the Confidence Profile Method. San Diego: Academic Press, 1992
39.
General Accounting Office. Cross design synthesis: a new strategy for medical effectiveness research. Washington, DC: General Accounting Office, 1992
40.
Lancet. Cross design synthesis: a new strategy for studying medical outcomes?Lancet1992; 340: 944–946
41.
JeffersonTMugfordMGrayADeMicheliV.An exercise in the feasibility of carrying out secondary economic analysis. Health Economics1996; 5: 155–165
42.
RigbyKSilagyCCrockettA.Health economic reviews: are they compiled systematically?International Journal of Health Technology Assessment1996; 12: 450–459
43.
JeffersonTDeMicheliV.Economic-evaluation of influenza vaccination and economic modeling — can results be pooled?PharmacoEconomics1996; 6: S67–72
44.
MoherDFortinPJadadARJuniPKlassenTLe LorierJCompleteness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. Lancet1996; 347: 363–366
45.
Publication bias. In: CooperHHedgesLV, eds. The Handbook of Research Synthesis. New York: Russell Sage Foundation, 1994: 399–410
46.
EggerMSmithGDSchneiderMMinderC.Bias in meta-analysis detected by a simple, graphical test. BMJ1997; 315: 629–634
47.
DearKBGBeggCB.An approach for assessing publication bias prior to performing a meta-analysis. Statistical Science1992; 7: 237–245
48.
SimesRJ.Publication bias: the case for an international registry of clinical trials. Journal of Clinical Oncology1986; 4: 1529–1541
49.
LittleRJARubinDB.Statistical analysis with missing data. New York: Wiley, 1987
50.
StewartLAClarkeMJ.Practical methodology of meta-analyses (overviews) using updated individual patient data. Cochrane Working Group. Statistics in Medicine1995; 14: 2057–2079
