Abstract
Systematic reviews and meta-analysis are techniques for synthesizing primary empirical studies to produce a summary of effects. To facilitate this goal, the Campbell Collaboration (C2) supports reviews within the disciplines of crime and justice, education, international development, and social welfare. At the annual Campbell Colloquium, experts on systematic review and meta-analysis provide introductory and advanced methodological training to researchers and practitioners. The C2 has filmed the training sessions and offers access to these videos, free of charge, on their website. The purpose of this article is to provide some background on the C2 and to introduce the presentations available on their website.
Keywords
Systematic reviews and meta-analysis have become important techniques in evidence-based policy decisions. Given the large number of studies on social issues and interventions, and the divergent results from these studies, systematic reviews and meta-analyses can help researchers to organize and synthesize existing evidence leading to a deeper understanding of a research area. The widespread use of systematic reviews and meta-analysis has also led to greater interest in understanding how to conduct and evaluate high-quality reviews of the literature. The importance of these methods for evidence-based policy make it imperative that researchers and policy makers understand how systematic reviews and meta-analyses are carried out, not only to participate in their production but also to be able to assess carefully the findings of reviews. The Campbell Collaboration (C2) has conducted a number of training sessions on aspects of systematic reviewing and meta-analysis. The videos from these training sessions are available for viewing, free of charge, from C2 website at http://www.campbellcollaboration.org/resources/training.php. Below, we provide an introduction to C2 and a guide to the videos available to researchers interested in learning how to conduct a systematic review and meta-analysis.
The Campbell Collaboration
The C2 is an international, interdisciplinary organization dedicated to the production, maintenance, and dissemination of high-quality reviews of interventions primarily in the social sciences (Schuerman et al., 2002). Largely a voluntary organization, the C2 provides editorial and methodological support to reviewers across its four substantive coordinating groups: crime and justice, education, international development, and social welfare. The coordinating groups of these disciplines are responsible for ensuring high substantive quality and methodological rigor. Both substantive and methodological experts peer review the protocol and completed review prior to publication in the C2 library. 1
C2 provides a number of methodological resources to reviewers. On the Resource Center 2 webpage, researchers will find a series of links to meta-analysis resources as well as a popular web-based effect size calculator developed by David B. Wilson that is a companion to the introductory book on systematic review by Lipsey and Wilson (2001). In addition to documents that cover editorial policies of C2, the Resource Center is home to a series of videos given at its annual Colloquium in August 2011 at George Mason University. The videos were taped over 2 days of introductory and advanced training with leading meta-analysis methodologists. There are two sets of videos, one introducing techniques for systematic reviews and meta-analysis and a second covering more advanced topics in meta-analysis. Below, we describe each video and provide a brief discussion of the purpose of each presentation. In Tables 1 and 2, we provide video timestamps to direct viewers to important issues.
Introductory Video Highlights.
Advanced Video Highlights.
Introductory Series
The introductory series provides the viewer with detailed instructions to initiate a systematic review, conduct a dedicated search, code retrieved literature, and calculate and synthesize quantitative measures of effects (i.e., effect sizes). The series is designed to introduce novice reviewers to the methods and assumes a basic understanding of research design and statistical analysis. Interested researchers will also find more detail in the introductory texts by Lipsey and Wilson (2001), and Borenstein, Hedges, Higgins, and Rothstein (2009).
Problem formulation
This video covers the initial step of identifying a research problem suitable for a systematic review. Valentine (2011), discusses the issues that researchers need to consider before conducting a systematic review and the importance of formulating the research question for the conduct of a quality review.
Literature searching
A critical stage in a systematic review is the literature search. The findings from a systematic review depend on obtaining the most complete set of studies conducted in a given area. Hammerstrøm (2011), provides a general description of the search process, including how to incorporate search filters, where and how to report search findings, and the types of bibliographic databases available. Interested researchers can also examine a C2 guide to informational retrieval written by Hammerstrøm, Wade, and Jørgensen (2010) available in the C2 Resource Center.
Coding
After obtaining a set of relevant studies, the next stage in a systematic review is to evaluate the literature by coding important characteristics of the studies. In this presentation, Wilson (2011), provides a number of strategies for coding studies with an emphasis on the importance of coding reliability for a quality research synthesis. The presentation starts with an overview of the screening process and then provides the audience with detailed descriptions of the types of codes relevant to reviews. A brief tutorial on study quality is also provided. Finally, the presentation concludes with an overview of Filemaker Pro, a database-driven software package that enables coders to utilize a standardized coding form. This presentation is suitable for introductory-level participants as well as more advanced participants who are in the process of conducting a review.
Effect size calculation and basic meta-analysis
When studies provide quantitative results, the technique of meta-analysis can be used to synthesize the findings across studies and to examine correlates of variability in study results. The first step in a meta-analysis is to code an effect size from each study. Wilson (2011a), introduces the most common metrics for effect size: the standardized mean difference, correlation, and odds ratio. In addition, Wilson gives an overview of the basics of meta-analysis, such as how to compute a weighted mean effect size and its variance and how to examine heterogeneity across studies.
Advanced Series
The advanced series is intended to introduce conceptually intricate ideas in the meta-analytic literature as well as provide exposure to recent methodological developments. Unlike the introductory series, viewers are expected to have a working knowledge of meta-analysis in addition to basic statistical understandings.
Calculating effect sizes, advanced
It is often the case that primary studies fail to provide basic summary statistics to compute an effect size. There are, however, many methods available to estimate an effect size from the values of statistical tests given in a study. Wilson (2011b), provides formulas for computing an effect size from a number of different statistical tests. The video begins with an overview of the standardized mean difference and provide a number of strategies for computing this effect size from statistics commonly reported in the primary literature. Effect sizes for categorical data, such as the odds ratio, are presented at the end. Finally, a description of the effect size calculator and its capabilities are also discussed.
Fixed versus random effects models
When conducting a meta-analysis, researchers must make a decision between two types of models, fixed effets versus random effects. Much confusion exists in the literature about the choice between these two models. Pigott (2011a), provides an overview of these models as well as their statistical foundations. Starting with the fixed effect model, the presentation provides detailed descriptions of how to estimate the fixed effect weighted mean and relevant statistical tests. The discussion also includes how the random effects model differs from the fixed effect model and the various methods of estimating between-study variance. The presentation concludes by presenting an empirical example comparing fixed- and random-effects outcomes. Although this presentation is in the advanced series, it is applicable to viewers who have a working understanding of meta-analysis. More detail about the differences among fixed and random effects models can be found in Borenstein, Hedges, Higgins, and Rothstein (2010).
Cluster adjustments in computing effect sizes
One problem faced by researchers using meta-analysis is the computation of an effect size from a cluster randomized experimental trial. Computing effect sizes that reflect the nested structure of the data is more complex than standard methods of estimating effects. Pigott (2011b), introduces several estimates of effect sizes from cluster randomized trials. The form of these estimates depends on the nested structure of the data. For instance, the clusters may be balanced or unbalanced, the standard deviations may be presented for individual participants or across clusters, or two or more levels of nesting may be present. Each scenario requires the analyst to determine the appropriate summary statistics to calculate an effect size. More statistical detail about effect sizes for clustered trials can be found in Hedges (2007).
Using robust standard errors for dependent effect sizes
A common problem faced by researchers using meta-analysis is the treatment of multiple outcomes reported in a single study. Effect sizes computed on the same sample of participants are not independent. One common strategy used by researchers is to synthesize the results of the studies by outcome so that each study provides only a single effect size for any analysis. More recently, an alternative strategy has been developed that synthesizes these dependent effect sizes across studies using robust standard errors. Elizabeth Tipton and Emily Tanner-Smith (Tipton & Tanner-Smith, 2011), introduce the method of robust standard errors through a series of informative examples. The lecture begins with an explanation of how robust variance estimators ameliorate the problem of dependent effect sizes. Statistical guidelines and relevant formulas, along with an empirical example, are also provided. The second half of the lecture demonstrates the use of a STATA (StataCorp, 2011) package written specifically for robust standard errors in meta-analysis. More detail about this method is available in Hedges, Tipton, and Johnson (2010).
Summary
The C2 supports, maintains, and disseminates systematic reviews and meta-analyses on social interventions in order to promote the use of evidence in important policy decisions. Part of C2’s mission is to provide training to researchers interested in using robust methods for systematic review and meta-analysis. We encourage all interested researchers to visit the C2 website and attend our annual Colloquium 3 to be held on May 21–23, 2013 in Chicago.
Footnotes
Authors’ Note
This article was invited and accepted by the editor.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.