Abstract
Research syntheses provide one means of managing the proliferation of research knowledge by integrating learnings across primary research studies. What it means to appropriately synthesize research, however, remains a matter of debate: Syntheses can assume a variety of forms, each with important implications for the shape knowledge takes and the interests it serves. To help shed light on these differences and their stakes, this chapter provides a critical comparative review of six research synthesis infrastructures, entities that support research syntheses through investments they make in synthesis production and/or publication—enabling (and constraining) the ways knowledge takes shape. Identifying our critical cases through purposive selection, we examined research synthesis infrastructure variations with respect to four different kinds of investments they make: in the genres of synthesis they support, in their promotion of synthesis quality, in sponsoring stakeholder engagement, and in creating the conditions for collective work. We draw on this comparison to suggest some of the potential changes and challenges in store for education researchers in future years.
With bodies of research continually expanding, it is arguably impossible for education scholars to “know” every study relevant to an area of interest. Investments in methodologies for integrating learnings from existing research are increasingly crucial for developing and managing knowledge. Research syntheses integrate learnings from “primary” empirical research studies to develop theory and to enable more robust conclusions about the phenomena studied. Formal synthesis methodologies only partly account for how syntheses are conducted and reported—much less how they shape knowledge and how we engage with it. Methodologies are embedded in larger sociotechnical infrastructures that enable and constrain how we can collectively learn from existing research. Taking stock of research syntheses requires us to engage the knowledge infrastructures they participate in: the ecologies of policies, practices, norms, resources, social structures, technologies, and methodologies through which knowledge is produced. In this chapter, we examine research synthesis infrastructures (RSIs): knowledge infrastructures that subtend and shape the synthesis of research—in turn, enabling and constraining our engagements with(in) it. 1
Science studies scholars who research infrastructures define them as forms of “substrate: something upon which something else ‘runs’ or ‘operates’” (Star & Ruhleder, 1996, p. 112)—for example, the electric grid, a municipal sewage system, or the Internet. Infrastructure refers to the prior work (be it building, organization, agreement on standards, and so forth) that supports and enables the activity we are really engaged in doing. More particularly, infrastructure refers to those systems, technologies, organizations, and built artifacts that do not need to be reconsidered at the start of a new venture [emphasis added]. (Slota & Bowker, 2017, p. 529)
By extension, knowledge infrastructures can be understood as “robust networks of people, artifacts, and institutions that generate, share, and maintain specific knowledge about the human and natural worlds [italics removed]” (Edwards, 2010, p. 17; see also Borgman, 2015; Edwards et al., 2013). For well over a century, education scholarship has been supported by knowledge infrastructures that include libraries, publishers, professional organizations, and the Academy itself—with books and journal articles as its primary “knowledge objects” (Bowker, 2017, p. 391). In recent decades, research syntheses have emerged as increasingly important knowledge objects for empirical scholarship: “Synthesis extends the analysis of the characteristics of individual research studies . . . to investigate what they mean as a collective body of knowledge,” meaning that a research synthesis “generates new knowledge and understanding in response to the review research question” (Thomas, O’Mara-Eves, et al., 2017, p. 182). This work has been described as providing education research an “architecture” (Murphy et al., 2017, p. 3)—an intellectual foundation on which present practices and future developments can depend.
Thanks to the Review of Educational Research, journals dedicated to reviewing education research have been around for almost a century, but the modern rise of explicit research synthesis methodology can be traced back to the 1970s, when education researchers Feldman (1971) and Light and Smith (1971) highlighted the potential of methodologically explicit integrations of research, and when—in his 1976 AERA (American Educational Research Association) presidential address—Glass advanced a new synthesis methodology in the form of meta-analysis, which pools effect sizes across quantitative studies (Glass, 1976; see also Chalmers et al., 2002). Notably, while not invoking the word “infrastructure,” Glass followed up this address with a 1977 Review of Research in Education article that noted how technical issues of research integration were underpinned by a host of nontechnical factors and forces—among them, access to primary research (e.g., through libraries or databases), disciplinary incentives for undertaking and publishing syntheses, and communal norms and standards for conducting and reporting research (Glass, 1977, pp. 353–354).
Research syntheses have become complexly (and controversially) enmeshed in the disciplinary ecology of education research, with synthesis methodologies and infrastructures emerging (and evolving) in response to political pressures or disciplinary commitments—and with researchers regularly debating about synthesis work in education (e.g., Ginsburg & Smith, 2016; Gough & Thomas, 2016; Green & Skukauskaitė, 2008; Slavin, 2008; Suri & Clarke, 2009; Zawacki-Richter et al., 2020). Following Edwards et al. (2013), an infrastructural understanding of synthesis work requires us to investigate how knowledge infrastructures configure the “shape and possibility of knowledge” (p. 15) and to consider whose interests are advanced and whose are constrained, ignored, or hindered by different infrastructural configurations (pp. 14–15). Considerations like these take on particular urgency in the present digital age, when the ubiquity of networked, mobile, and cloud computing promise to disrupt the “control zone” (Lagoze, 2010) within which scholarly results are evaluated, disseminated, and curated (see Borgman, 2015). Big data and automation are augmenting the ways research syntheses can be—and are being—produced and used (see Gough, Thomas, & Oliver, 2019; Snilstveit et al., 2018; White, 2019). As Bowker (2017) notes, “The great potential of the current moment is to re-imagine the engines of knowledge production in ways which produce new forms of alignment” (p. 397). The risk is that these changes will occur without critical reflection on their consequences in time to shift their trajectories. How might we take advantage of these opportunities to enhance our collective learning from research?
We build toward answers to these overarching questions with the following (interrelated) analytical questions about infrastructural investments RSIs make, consistently asked of six RSI cases we review:
What are the primary genres or types of synthesis RSIs invest in?
How do RSIs invest in promoting or controlling synthesis quality?
In what ways do RSIs invest in outreach to and engagement with stakeholders?
Through what structures and tools do RSIs invest in collective work?
Answering these questions empowers us to critically grapple with ways current RSI configurations shape knowledge in education, and (paraphrasing Bowker) to re-imagine the engines of research synthesis production moving forward.
Methodology and RSI Case Overviews
We examine six RSI cases with instructively different features relevant to our analytical questions. To be considered as a potential case, RSIs had to be active as of 2005, noncommercial (with exceptions made for commercial publishing partnerships), and primarily configured to support the production and dissemination of research syntheses. To select cases, we drew on prior knowledge of prominent organizations sponsoring research syntheses and conducted a more extensive online search, informed by insights from the secondary literature reviewing such organizations (e.g., Green & Skukauskaitė, 2008; Means et al., 2015; Slavin, 2008), from prominent synthesis methodology textbooks (e.g., Booth et al., 2016; Cooper et al., 2019; Gough et al., 2017b), and from the Systematic Review Toolbox—a website that curates synthesis-supporting tools (http://systematicreviewtools.com/). Seeking to illustrate instructive variations in RSI configurations (not an exhaustive overview of the field), we privileged RSIs with longer histories, more explicit guidance, broader recognition, and/or instructively different features. We included not only education-centric RSIs but also health- and social science-centric RSIs, because such RSIs have served as models that education-centric RSIs have regularly been compared with; they also offered alternative configurations from which we could learn. Below, we provide brief overviews of (and reasons for selecting) RSI cases.
Three selected RSI have historically privileged systematic reviews of intervention effectiveness (and, where possible, meta-analysis)—a synthesis genre often colloquially called “what works” reviews—although the latter two RSIs now support reviews addressing other kinds of questions:
What Works Clearinghouse (WWC), included for its significance in U.S. education. An outgrowth of the 2002 Education Sciences Reform Act, WWC seeks to provide “education decisionmakers” with “access to the best evidence about the effectiveness of education practices, products, programs, and policies . . . by identifying existing research on education interventions, assessing the quality of this research, and summarizing and disseminating the evidence from studies that meet WWC Standards” (What Works Clearinghouse, 2020a, p. 1).
Cochrane (formerly the Cochrane Collaboration), an international RSI selected for its major historical significance, including as a model against which other RSIs are often compared (e.g., Slavin, 2008). Founded in 1993, Cochrane seeks “to promote evidence-informed health decision making by producing high quality, relevant, accessible systematic reviews and other synthesized research evidence” (Chandler et al., 2019, p. 1).
The Campbell Collaboration (Campbell), an international RSI founded in 2000 (and closely related to Cochrane), selected because it supports decision-making regarding “the effectiveness of programs, policies, and practices (and, in some instances, closely related topics) in the areas of crime and justice, education, international development, and social welfare” (Campbell Collaboration, 2019, p. 6). 2 Campbell’s methodological guidance is now largely coordinated with Cochrane’s, but we treat them separately below wherever their infrastructural elements significantly distinguish them.
We selected one RSI, less well known in the United States, because it supports synthesis methodologies that challenge/expand those historically privileged by WWC, Cochrane, and Campbell:
The Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre), founded in 1993 and based in the University College London’s Social Science Research Unit, selected for the diversity of synthesis genres and methodologies they support. EPPI-Centre focuses on “complexity and mixed methods reviews to understand research relevant to decision making as well as methods for how that research is used in practice” (EPPI-Centre, n.d., para. 2), supporting “reviews driven by any research questions and including any research methods and types of data” (Gough et al., 2017a, p. 12). 3
Finally, we included two venerable RSIs because their infrastructural elements depart in significant ways from other selected RSIs:
Review of Education Research (RER), chosen as one prominent example of the many journals focused on research syntheses. “Since its inception in 1931,” editors Murphy et al. (2017) write, “RER has stood as a prestigious journal within the field of education, publishing critically important reviews of research focused on educational constructs, processes, and outcomes” (pp. 3–4).
The National Academies of Sciences, Engineering, and Medicine (National Academies), a private nonprofit that began in 1863 as the National Academy of Sciences, incorporated to “investigate, examine, experiment, and report upon any subject of science or art” (quoted in Feuer & Maranto, 2010, p. 260). National Academies was chosen as a prominent example of the many learned societies that synthesize and publicize scientific knowledge.
Importantly, our selection of RSIs—especially RER and National Academies—is broader than is typical for comparative reviews of research synthesis-supporting entities (but see Slavin, 2008, p. 5; Zawacki-Richter et al., 2020). Synthesis methodology scholarship now constitutes an active field, complete with journals (e.g., Research Synthesis Methods, Systematic Reviews). Although RER and National Academies significantly depart from this tradition (and from more “conventional” RSIs associated with it), they nevertheless produce research syntheses and expand the range of answers to our critical questions. Table 1 provides an overview of the six RSIs.
Overview of Research Synthesis Infrastructure (RSI) Cases
Note. Bolded text is bold in the original.
The materials cited as “primary guidance documents” represent—as of January 8, 2020—the most recent versions associated with the primary synthesis genres each RSI supports; they do not represent the full diversity of guidance materials that these RSIs develop and/or publish. Additionally, Cochrane, Campbell, EPPI-Centre, and RER publish (or link to) methodology articles not listed here; and while Campbell currently adapts and/or links to primary guidance documents produced by Cochrane, the table lists only those documents produced by the RSI in question itself.
This column lists only the primary digital tools and automations for supporting synthesis production that were publicized online by the RSIs themselves at the time we are writing this chapter. Additionally, while not captured in this table, Campbell and Cochrane link to the Systematic Review Toolbox (http://systematicreviewtools.com/), which maintains an extensive database of synthesis-supporting tools.
These RSIs routinely evolve, in ways small and large. Our analysis focuses on each RSI’s public documentation (e.g., handbooks, standards, software, policy statements) appearing/referenced on their websites as of January 8, 2020; we also considered external literature providing relevant context and/or commentary, though except where otherwise noted, data regarding RSIs were drawn from their websites (see Table 1). While the issues we raise are (we argue) enduring and forward-looking, the RSI comparisons illustrating these issues reflect a snapshot in time. Infrastructure studies sometimes involve network analyses and ethnographic research (e.g., Star & Ruhleder, 1996), including considerations of “infrastructuring”—that is, infrastructure as a function of complex, ongoing processes and design-related activities (e.g., Karasti, 2014; Pipek & Wulf, 2009)—but such research was beyond this review’s goals and scope. 4 In analyzing available documentation, we drew on comparative case study research—especially structured, focused comparison (George & Bennett, 2005)—seeking information relevant to our critical questions about genre, quality, stakeholders, and collective work. For each RSI, we asked a senior scholar who is/was in a leadership role within the RSI to review an earlier draft for accuracy and fairness.
Findings
We contrast how RSIs invest in and shape knowledge by considering
which primary genres of research synthesis they support,
how they promote and control quality,
how they engage stakeholders, and
how they support collective work in knowledge production.
These kinds of investments, examined separately below, are intricately interrelated—a circumstance to which we turn in the Discussion and Conclusion.
Genre
The primary synthesis genres RSIs sponsor represent a priori positions regarding the types of research that should be synthesized and the forms such syntheses should take. Investments in genre can have institutional ramifications for the kinds of research that are funded and rewarded—and thus for the kinds of knowledge that get generated. Apart from National Academies and RER, the examined RSIs identified systematic reviews as the primary syntheses they support. Generally, a systematic review can be described as “a review of existing research using explicit, accountable rigorous research methods” (Gough et al., 2017a, p. 4)—but across RSIs, what counts as “systematic” can vary significantly (see also Gough & Thomas, 2017). For Cochrane, for instance, A systematic review attempts to collate all the empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made. . . . (Lasserson et al., 2019, p. 4)
This emphasis on identifying all studies that meet eligibility criteria (i.e., comprehensive or “exhaustive” searching) is more controversial. EPPI-Centre staff caution that “for an ‘exhaustive’ approach to be manageable, the scope of the review needs to be limited” (G. Brunton et al., 2017, p. 98), with emphasis placed on crafting “a ‘reasonable’ search strategy” that documents choices made and efforts “to minimise . . . biases” (p. 100). As an alternative, EPPI-Centre offers purposive searching: “identifying studies that contain new conceptualisations of the phenomena of interest” until the point “of saturation,” where no new concepts of interest are contributed (G. Brunton et al., 2017, p. 101)—an approach Cochrane also acknowledges for synthesizing qualitative evidence (Noyes et al., 2019, p. 531).
Three RSIs focus largely on systematic reviews of the effects of interventions on designated outcomes (Cochrane, Campbell, WWC)—often colloquially called “what works” reviews (see Gough & Thomas, 2017, p. 53). While intervention reviews are the genre these RSIs privilege (and, thus, our focus here), they each produce other review genres. In addition to reviews of intervention effectiveness, Cochrane lists the following as major review genres it supports: reviews of diagnostic test accuracy, reviews of prognosis, systematic overviews of reviews, and reviews of methodology (Chandler et al., 2019, pp. 4–6). 5 Campbell also sponsors systematic overviews of reviews, methods research papers, and evidence and gap maps that “systematically identify and report the range of research activity in broad topic areas or policy domains” (Saran & White, 2018, p. 5)—with Campbell allowing reviews “on topics . . . closely related to interventions,” such as cost-effectiveness, predictive validity, and other associational studies (Campbell Collaboration, 2019, p. 7). The RSI further allows for what it calls “Campbell Innovations”—that is, “novel evidence synthesis types that have not yet been published by the Campbell Collaboration,” which receive a “specialized methods peer review” (Campbell Collaboration, 2019, p. 6). Cochrane, Campbell, and WWC also publish documents that are prospective to/derivative of systematic reviews, including review protocols (WWC, Cochrane, Campbell) and reviews of individual studies (WWC). (Except RER, all examined RSIs also routinely produce materials translating knowledge for nonspecialist stakeholders; see “Stakeholders.”) Notably, alongside its intervention effectiveness reports, WWC publishes educator’s practice guides—written with practitioners in mind (see “Stakeholders”)—which synthesize “rigorous research” (as evaluated against WWC’s standards) to provide “recommendations for a coherent approach to a multifaceted problem” (Institute of Education Sciences [IES], 2018, p. 43).
For systematic reviews of interventions, WWC, Campbell, and Cochrane promote comprehensive searching—though WWC adopts a priori restrictions on this process, allowing only for “publicly available” English-language studies (WWC, 2020a, p. 7). These RSIs embrace a methodological hierarchy privileging randomized controlled trials (RCTs): experimental studies randomly assigning participants to intervention and control groups. Because they produce unbiased effect size estimates for studied outcomes, RCTs are treated—within these RSIs—as providing the best evidence for causal conclusions regarding intervention effectiveness. Quasi-experimental (nonrandomized) studies are treated as potentially worthy of inclusion in intervention reviews, provided they meet specified minimal internal validity standards. 6 These RSIs recommend meta-analysis (wherever it is possible) as their preferred synthesis method, pooling effect sizes to produce an overall effect size estimate of a (type of) intervention (Campbell Collaboration, 2019; Higgins, Thomas, et al., 2019; WWC, 2020a).
For Campbell and Cochrane (but not WWC), studies from other methodological traditions—including qualitative research—are permitted to contextualize quantitative synthesis findings and to address questions supplementary to intervention reviews. Campbell notes that such research “can help paint a richer picture of the intervention, its effects, how or why it produces those effects (or not), and other such features that provide texture and explanatory context to a review” (Campbell Collaboration, 2019, p. 11). Cochrane identifies “synthesis of qualitative evidence” as beneficial to understanding interventions by “increasing understanding of a phenomenon of interest,” of the relationship between interventions and the “broader environment” they take place in, of the ways “health conditions and interventions” are perceived and experienced, and of the context-contingent “complexity” of the “impacts and effects” of “interventions and implementation . . . on different subgroups of people” (Noyes et al., 2019, pp. 525–526). Describing “designs for synthesizing qualitative evidence with evidence of the effects of interventions,” Cochrane identifies two forms this integration can take: Sequential reviews, which integrate qualitative evidence synthesis findings with pre-existing intervention review findings; and convergent mixed-methods reviews, when “no pre-existing intervention review exists” and both the (trial-based) intervention review and qualitative evidence synthesis must be conducted and integrated (Noyes et al., 2019, p. 526).
EPPI-Centre contrasts its work with that of Cochrane and Campbell (and, by implication, WWC), privileging different methodologies within its broader definition of “systematic reviews.” Encouraging a theory-driven methodological flexibility, EPPI-Centre primarily supports systematic reviews—and related secondary synthesis genres (e.g., maps and overviews of reviews)—that are “driven by any research questions and including any research methods and types of data” (Gough et al., 2017a, p. 12), stressing that synthesis work should be driven by “a ‘fit for purpose’ approach” (Gough & Thomas, 2017, p. 52), where synthesis methods are calibrated to context-contingent objectives. (Additionally, EPPI-Centre generates publications on synthesis methodology and reviews/primary research studies of evidence use.) EPPI-Centre distinguishes between “aggregative” and “configurative” orientations toward synthesis work—first introduced by Sandelowski et al. (2006)—a key distinction carried throughout their guidance manual An Introduction to Systematic Reviews: “The diversity of methods . . . used to bring together (‘synthesise’) study findings lie on a continuum between approaches that aim to aggregate or ‘add up’ findings from multiple, similar studies; and those that aim to configure or ‘organise’ findings” (Gough et al., 2017a, p. 7). Synthesis approaches with a more aggregative orientation tend to test hypotheses and make inferential claims based on statistical information (as with meta-analysis); approaches with a more configurative orientation tend to generate and explore theory, making inferential claims based on the theory developed (Gough & Thomas, 2017, pp. 62–65). Configurative synthesis approaches highlighted by EPPI-Centre, such as the following, include those developed by its staff and those developed elsewhere:
Thematic synthesis, which “emphasises the development of theory from a starting point of open questions and few secure initial concepts,” and “makes no inferential claims based on statistics, but aims to enlighten decision making through the creation of new theory” (Thomas, O’Mara-Eves, et al., 2017, p. 190; see also Thomas & Harden, 2008)
Realist synthesis, which “aims to uncover the (usually) hidden mechanisms which cause interventions to result in specific outcomes in specified contexts” (Thomas, O’Mara-Eves, et al., 2017, p. 200; see also Pawson, 2006)
Meta-narrative review, which helps manage plural (and perhaps competing) perspectives concerning “complex and contested topic areas which have been studied by different disciplines and traditions,” such that these syntheses “unpack the shared set of concepts, theories and methods within a research tradition and chart how the research conducted within the tradition unfolds and changes over time, highlighting key discoveries and insights” (Thomas, O’Mara-Eves, et al., 2017, p. 202; see also Greenhalgh et al., 2005)
Meta-ethnography, which provides “a way of thinking about drawing conclusions across multiple ethnographic studies” and other sources of qualitative evidence (Thomas, O’Mara-Eves, et al., 2017, p. 195), with meta-ethnographers “systematically comparing conceptual data from primary qualitative studies to identify and develop new overarching concepts, theories, and models,” instead of “aggregat[ing] findings” (France et al., 2019, p. 2; see also Noblit & Hare, 1988)
The latter three approaches have standards supporting their enactment or reporting developed by the RAMESES (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) Projects (Wong et al., 2014) and eMERGe (Meta-Ethnography Reporting Guidance) Project (France et al., 2019), respectively. RAMESES scholars Wong et al. (2014) argue that synthesis genres like these can “supplement, extend and in some circumstances replace Cochrane-style systematic reviews” (p. 1).
In contrast to the RSIs above, RER includes systematic reviews among the genres it supports but invests in the broader category of integrative reviews, which pull together the existing work on an educational topic and work to understand trends in that body of scholarship. In such a review, the author describes how the issue is conceptualized within the literature, how research methods and theories have shaped the outcomes of scholarship, and what the strengths and weaknesses of the literature are. (Review of Educational Research, n.d.-a, para. 2)
Beyond empirical reviews, RER supports historical, theoretical, and methodological reviews as well, investing in an inclusive understanding of what counts as data, “including empirical articles, secondary databases, and historical and political archives” (Murphy et al., 2017, p. 4).
For its part, National Academies invests in expert consensus study reports and workshop summaries, undertaken in response to queries posed by sponsors, such as the U.S. federal government and state agencies. An expert consensus report (our focus here) is simply “a report produced by a committee of experts convened by the National Academies . . . to study a specific scientific or technological issue of national importance” (National Academies, 2018, para 1). In its public documentation, National Academies offers no a priori methodological restrictions for these genres, beyond expecting that report findings be “supported by the scientific evidence and the arguments presented” (National Academies, n.d.-b, “Stage 4” section)—rendering the methodological scope of its work more implicit (and perhaps less narrowly defined) than is typical of the RSIs we examined. 7
Quality
Each examined RSI enacts a conception of synthesis quality, offering explicit recommendations (or requirements) for quality judgments. While no two RSIs make identical investments, each could be understood as primarily adopting one of two general approaches to quality we characterize as (a) procedural control or (b) conceptual guidance and professional judgment.
Primarily Procedural Control
For systematic reviews of interventions, WWC, Cochrane, and Campbell maintain highly detailed, centralized control over the process of synthesis development and reporting, providing easily documented and audited quality assurances. Prioritizing review topics chosen (with stakeholder input) by the Institute of Education Services (IES; WWC, 2020a, p. A-2), WWC exerts procedural control at several review stages: A topic-specific review protocol (containing required features) is made before reviews are conducted, specifying how relevant research should be searched for and appraised; relevant literature is identified and screened in accordance with protocol requirements; studies included in the synthesis are individually assessed using standards established in WWC guidance documents; and synthesis findings are publicized in one or more privileged formats (WWC, 2020a, pp. 1–2). WWC authors are screened for conflicts of interest, those reviewing individual studies must first be formally trained and certified by the RSI, and review team members are chosen on the basis of prior WWC-certification and (in the case of team leaders) perceived methodological and content expertise (WWC, 2020a, pp. C-1–C-5). WWC reports receive internal review by the IES and a blind external “peer review by researchers who are knowledgeable about WWC standards and are not staff with the WWC contractor that prepared the draft publication” (p. C-3)—in addition to “a final review by IES staff to ensure that any issues have been addressed appropriately,” after authors have had an opportunity to address reviewer comments (p. C-4). 8
Cochrane and Campbell coordinate intervention review production through documents and tools that procedurally regulate synthesis work, including the organizations’ primary guidance texts (Higgins, Thomas, et al., 2019; Campbell Collaboration, 2019), checklists for additional/special considerations (e.g., equity), and their respective guidelines for review conduct and reporting: the MECIR (Methodological Expectations of Cochrane Intervention Reviews) and the MECCIR (Methodological Expectations of Campbell Collaboration Intervention Reviews), each of which contains close to 200 distinct standards—a level of procedural detail unmatched by any other examined RSI. 9 These RSIs employ required software for protocol and review submission—Review Manager (RevMan)—ensuring a standardized reporting format with limited flexibility (see “Collective Work” section). When reviewing the quality of studies to be included in syntheses, Cochrane expects (and Campbell encourages, where applicable) that authors use the procedurally oriented GRADE (Grading of Recommendations, Assessment, Development and Evaluation) approach—an evidence quality evaluation framework with associated software (GRADEpro GDT [Guideline Development Tool]). For qualitative evidence synthesis, Cochrane promotes use of GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative research), a related tool for appraising “relevance, methodological limitations, adequacy and coherence . . . to formulate an overall assessment of confidence in the synthesized qualitative finding” (Noyes et al., 2019, p. 537). Thus, as Campbell puts it, “Systematic reviews are developed through a process that helps ensure that they are accurate, methodologically sound, comprehensive, and unbiased” (Campbell Collaboration, 2019, p. 6). And, following Cochrane, the systematic review process seeks “to minimize bias by using explicit, systematic methods documented in advance with a protocol” (Chandler et al., 2019, p. 1)—deviations from which must be justified in the full report.
Cochrane and Campbell also invest in centralized editorial oversight, such that systematic reviews undergo quality evaluation at three stages: when titles (and thereby research questions/topics) for specific reviews are registered, when protocols for those reviews are submitted, and when reports adhering to those protocols are completed (Campbell Collaboration, 2019; Higgins, Thomas, et al., 2019). 10 For Cochrane and Campbell, quality evaluation requires peer review by one or more topic specialists and at least one methodologist, overseen by editorial groups that determine (at each stage) whether and how review production should continue. Cochrane typically provides for named peer review, where reviewers and authors know one another’s names; Campbell, by contrast, invests in single-blind peer review, preserving reviewer anonymity. And because, under certain circumstances, Cochrane and Campbell allow copublication of their reviews elsewhere, these reports may undergo peer review more than once. 11
Primarily Conceptual Guidance and Professional Judgment
Other RSIs offer principles for synthesis conduct while supporting local, context-contingent uptake of those principles (EPPI-Centre)—or else invest primarily in the professional judgment of authors and reviewers (RER, National Academies). Supporting reviews that span aggregation-oriented and configuration-oriented genres (see “Genre” section), EPPI-Centre makes clear “there is not an EPPI-Centre method for undertaking reviews; rather there are principles that guide our work” (Gough et al., 2017b, p. 14)—with web-based EPPI-Reviewer software providing (paying subscribers) a flexible resource supporting development of multiple synthesis genres (see “Collective Work” section). Producing many of its systematic reviews at the request of organizational partners (with reviews coauthored, or otherwise supported by, EPPI-Centre staff), EPPI-Centre has “adapted and adjusted [its] review methods in order to answer the questions asked by [its] stakeholders in meaningful and useful ways” (Gough & Thomas, 2017, p. 52). General principles of methodological quality, tailored for appropriateness to the questions asked, are a function of three primary dimensions: “a review’s method (Dimension 1), the nature of its included studies (Dimension 2) and the evidence produced (Dimension 3),” which—taken together—provide “a framework for appraising a review in terms of whether it provides enough evidence, good enough evidence and relevant enough evidence to make evidence claims” (Liabo et al., 2017, p. 253). The reviews EPPI-Centre publishes “have been extensively peer reviewed before being published online, apart from one or two, and the absence of peer review is highlighted where this happens (usually for ‘maps’ of research, rather than syntheses),” with this review process occasionally overseen by organizations with which EPPI-Centre has partnered: For example, EPPI-Centre’s reviews for the U.K. Department of Health are “always independently peer reviewed by up to five reviewers in a process that is managed by NIHR [National Institute for Health Research]” (James Thomas, personal communication, January 7, 2020).
Quality control for National Academies is largely a matter of managing the quality of the authoring committee, with explicit procedures for nomination (and opportunity for public comment on prospective membership) to ensure that the committee embodies a range of expertise and balance of perspectives, unburdened by conflicts of interest (National Academies, 2003). Prior to committee selection for a study, a draft “statement of task, work plan, and budget” is internally reviewed by National Academies (National Academies, n.d.-b, “Stage 1” section). Once constituted, committees are not held to strict a priori reporting requirements and are expected to turn to a variety of sources—among them, “reviews of the scientific literature” (National Academies, n.d.-b, “Stage 3” section). Indeed, while National Academies is chartered to take up “request[s] for scientific advice” and convenes expert committees for this purpose, “the specifics of the study question can be refined and improved by the committee members after they are appointed” (Feuer & Maranto, 2010, p. 267). This emphasis on expertise and “flexibility” is shaped by a belief that “complex questions are better defined through deliberation rather than by a priori certitudes”—but investments of this kind are not without critical “tradeoffs,” regularly discussed within National Academies (Feuer & Maranto, 2010, p. 267). Committee members are permitted to submit brief dissenting opinions as appendices to the consensus report. National Academies reports are submitted to expert reviewers (anonymous, prior to the review’s publication), attentive to whether the “report addresses its approved study charge and does not go beyond it,” whether report “findings are supported by the scientific evidence and arguments presented,” whether “the exposition and organization are effective,” and whether “the report is impartial and objective” (National Academies, n.d.-b, “Stage 4” section). Here, the fact that the authoring committee can augment the study question(s) it was charged to investigate can “have its downsides: reviewers asked to judge the extent to which a committee adhered to its ‘charge’ may find it difficult to respond without more specificity” (Feuer & Maranto, 2010, p. 267). The report committee’s response to the reviewers’ comments is then audited by “independent report review ‘monitors’” prior to the report’s publication (The National Academies, n.d.-b, “Stage 4” section).
RER provides an example of a traditional “peer review” evaluation, with quality control almost exclusively invested in editors’ and peer reviewers’ judgments. To focus its double-blind peer review process, RER offers a general set of peer review guidelines addressing quality of the literature reviewed, quality of the analysis, significance of the topic, impact of the article (i.e., usefulness for stakeholders), advancement of the field, style (cautioning against “unexplained jargon and parochialism”), balance and fairness, and purpose (with respect to relevance to education; Review of Educational Research, n.d.-a, paras. 8–16). Specific journal editors refine and establish additional priorities for article quality, with recent priorities being not only methodological explicitness and transparency but also “that authors articulate potential avenues for addressing the critical question or problem they interrogated based on the findings they have mapped from the literature” (Murphy et al., 2017, p. 5; see also Murphy et al., 2020). In its online guidance (Review of Educational Research, n.d.-b), RER recommends that authors consult AERA’s Standards for Reporting on Research in AERA Publications (which, we note, are intended for primary empirical studies), 12 Guidelines for Reviewers (which focus primarily on ethical concerns), and—for “systematic reviews”—the PRISMA Statement: Guidelines for reporting meta-analyses and systematic reviews of intervention effectiveness (Moher et al., 2009), referenced also by Cochrane, Campbell, and EPPI-Centre, among others (see, e.g., Campbell Collaboration, 2019; Gough et al., 2017a; Higgins, Thomas, et al., 2019). RER does not mandate where and how reviewers are to make use of these diverse guidelines. 13
Stakeholders
Rees and Oliver (2017) remind us that the stakeholders for systematic reviews can be any groups affected by or contributing to that review (p. 20), and Suri (2014) notes, “Stakeholders with an interest in educational research . . . include policy-makers, administrators, teachers, funding agencies, researchers, students, parents and the wider community” (p. 2). While all examined RSIs aim to inform/improve policy and/or practice, each configures its investments in stakeholder engagement differently, according to how it (a) structures access to knowledge objects, (b) circumscribes whether/how stakeholders participate in knowledge production, and (c) sponsors “knowledge translation” work: ways to re-mediate (i.e., translate or package) knowledge objects and promote their consumption and uptake.
Access
Stakeholders’ (differential) access to RSIs’ knowledge objects is a fundamental equity concern with which all RSIs are grappling. All RSIs maintain some sort of searchable database of syntheses (see “Collective Work” section). Campbell and WWC are notable for making all their syntheses, review protocols, and guidance handbooks open access; National Academies also ensures nearly all its reports are available for free download. EPPI-Centre provides free PDF versions of the systematic reviews and primary research reports it participates in producing, in addition to some of the publications on methodology its staff has (co)authored. Cochrane makes its influential guidance documents freely available online but places new syntheses behind a paywall for 12 months, unless authors (or funders) pay for immediate open access. Importantly, the 12-month paywall period currently applies only to review versions published after (or on) February 1, 2013; syntheses published earlier remain behind a paywall (see also Heywood et al., 2018). That said, Cochrane makes several exceptions to its default access restrictions. For instance, some countries pay Cochrane to offer open access to their citizens, and the RSI provides for free access for some countries on the basis of income (see Heywood et al., 2018). Of the RSIs we examined, RER currently invests the least in open access, with paywalls for most of the syntheses they publish.
Enfranchising Stakeholders in Knowledge Production
When RSIs actively engage stakeholder/user communities in synthesis production, this involvement tends to take the form of consultation, solicited at one or more stages of the synthesis process. All but RER facilitate some form of stakeholder engagement in the review production process—though for some RSIs, stakeholder enfranchisement is limited. EPPI-Centre, for one, recommends stakeholder participation to support several aspects of synthesis work—such as aiding in evidence collection, synthesis oversight, promoting synthesis relevance, and the sharing of syntheses (Rees & Oliver, 2017)—paying particular attention to stakeholder use of synthesis-based evidence (see Gough et al., 2017a).
Of the RSIs examined, Cochrane and Campbell arguably make the most intensive investments in stakeholder enfranchisement. “Cochrane author teams are encouraged to seek and incorporate the views of users, including consumers, clinicians and those from varying regions and settings to develop protocols and reviews” (Cumpston & Chandler, 2019, p. 5). Cochrane reminds its authors that stakeholder engagement can take many forms in addition to synthesis coauthorship: Other “methods for working with consumers and other stakeholders include surveys, workshops, focus groups and involvement in advisory groups”—with the choice of method “typically . . . based on resource availability” (Lasserson et al., 2019, p. 6). A Campbell Policy Brief notes that stakeholder contributions have included “identifying and prioritising review topics, defining review questions and important outcomes, conducting reviews, [enhancing scientific quality through discussion about intervention integrity . . . and transferability], editing review protocols and reports, and disseminating and implementing review findings in practice” (Konnerup & Sowden, 2008, p. 1). Cochrane even maintains multiple web-mediated platforms for stakeholder participation, including the citizen science Cochrane Crowd platform, through which stakeholders can volunteer to identify and characterize/summarize health care–related research (specifically, RCTs), and a related TaskExchange platform, through which stakeholders can volunteer to complete a variety of tasks, such as assessing risk of bias and translating knowledge objects into languages other than English (Cochrane, 2019).
WWC, which contracts out production of its intervention effectiveness reports, offers more limited roles for stakeholders in knowledge production: It consults with policymakers and education association members and provides occasional opportunities for public feedback, including on potential review topics and proposed modifications to WWC’s guidance. It also enfranchises stakeholders to make recommendations regarding potential intervention report topics or studies to review; WWC further provides a public-facing version of its guide for reviewing evidence (to spread popular awareness of the RSI’s evaluative processes). The actual review and selection of studies for WWC’s database is conducted by official reviewers who have completed a certification process where “typically, individuals with the requisite skills will have an earned doctorate in social science . . . or equivalent” (WWC, n.d.-b, “Becoming a WWC-Certified Reviewer” section).
National Academies circumscribes the roles stakeholders can play in order to “maintai[n] a safe space for deliberations by defending its exemption from federal rules that would otherwise require all meetings to be held in public” (Feuer & Maranto, 2010, p. 270)—a move Feuer and Maranto (2010) argue “can protect scientific integrity” (p. 271). While National Academies does not formalize many opportunities for stakeholder participation in knowledge production, “all documents provided by outsiders to . . . [report-authoring] committees are now kept in a so-called ‘public access file,’ to which the public is welcome, and efforts are made to enable open and public sessions during many phases of committee life” (p. 272). Moreover, authoring committees can themselves be composed of figures other than higher education–affiliated academics/researchers (including K–12 educators, consultants, and nonprofit representatives).
Knowledge Translation
Knowledge translation work supports new and broader publics in consuming and using RSI-produced knowledge, which is often initially authored with specialist audiences in mind. All RSIs, excepting RER, routinely publish free policy briefs, evidence bulletins, plain language/executive summaries, and/or snapshots distilling their longer syntheses for stakeholders. In WWC’s case, translation efforts like these extend to recommendation-providing free “practice guides” tailored to support educator uptake of “what works,” where at least two practitioners serve on the authoring committee (see WWC, 2020a, pp. C-2–C3)—a peer-reviewed genre that relies on and reinforces the evidence hierarchy endorsed within the RSI’s intervention reports (see “Genre” section). National Academies also provides open access to a range of knowledge translation documents to accompany their consensus reports, including not only highlights, summaries, and press releases but also report-specific videos, webinars, commissioned reports, and podcasts. It also maintains some general resources (aligned to or informed by National Academies reports), such as teaching materials, interactive infographics, and a question-and-answer platform. Making particularly heavy investments in knowledge translation, Cochrane produces a diversity of free guidance and training resources (e.g., standards for plain language summaries) and has cultivated government partnerships to “increase policy makers’ awareness of the value of systematic reviews” and build their “capacity for using research in policy work” (Brennan et al., 2016, p. 26; see also Cochrane, 2019). Cochrane partners with Wikipedia to publicize high quality medical evidence (https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Medicine/Cochrane); the RSI further produces free podcasts and a synthesis-centric journal club, as well as question-and-answer formatted summaries based on synthesis evidence (though the full texts of these summaries seem not to be open access by default). Importantly, Cochrane, Campbell, and National Academies invest in translating some synthesis-related materials into multiple languages.
Relatedly, in addition to maintaining a centralized website, each RSI provides at least one venue for connecting with stakeholders (e.g., for stakeholders to receive updates or provide feedback). All maintain some kind of mailing list, newsletter, forum, or listserv, and list email addresses or offer online feedback forms; all maintain official social media presences (or have parent organizations that do so); a few maintain synthesis-related blogs with comment functionality (EPPI-Centre) or without it (Campbell, Cochrane, and WWC through its parent, IES), and Cochrane’s users can leave comments on reviews, protocols, and editorials published in its library. All but RER and National Academies provide for free synthesis production-supporting online training materials, instructional videos, and/or webinars (for more on professional development, see “Collective Work” section). These forms of online outreach establish opt-in relationships with stakeholders, while tending to position RSI-supported knowledge objects as entities stakeholders can consume or respond to, rather than shape.
Collective Work
Given the increasing proliferation of research, attention to how RSIs can facilitate, incentivize, and capitalize on collective work is becoming increasingly crucial. While much of the work entailed in producing individual research syntheses is researcher- or team-specific, RSIs support and sponsor this work by investing in structures to facilitate collective work across time, geography, and research teams (see Gough, Thomas, & Oliver, 2019)—enabling what Bowker (2017) calls a “division of cognitive labor” (p. 398). Here we focus on their investments in creating the social, conceptual, and technical conditions for collective work by providing for (a) coordination across synthesis projects, (b) databases, (c) metadata, (d) professional development, and (e) digital tools and automations to support synthesis work. While these investments enable a shared, common knowledge base that a diversity of researchers can contribute to, draw from, or build on, they also limit the scope of knowledge an RSI can support.
Coordination
By orchestrating which topics are reviewed and by whom, Cochrane and Campbell discourage synthesis duplication so that researchers’ efforts are distributed to a greater number of interventions (see Campbell Collaboration, 2019; Cumpston & Chandler, 2019). 14 Campbell, for example, provides reviewers who register a topic with “priority rights to the topic of the systematic review; no other review team will be approved by Campbell for a review on that specific topic as long as the team is making progress toward completing the review” (Campbell Collaboration, 2019, p. 13)—and those reviewers have exclusive rights to update their review within 5 years, “unless the original team explicitly waives their claim on that opportunity” (p. 16). Cochrane registers its review protocols with the international PROSPERO systematic review database (https://www.crd.york.ac.uk/PROSPERO/), a move intended to “reduc[e] duplication of effort, research waste, and promot[e] accountability” (Lasserson et al., 2019, p. 9). Practices that seek to distribute synthesis topics in ways that minimize duplication appear to imply that another qualified author team would arrive at the same conclusions—an assumption that is, perhaps, easier to justify with meta-analysis than with other synthesis genres. 15
Entities intended to coordinate efforts across RSIs are beginning to take shape. Notably, Evidence Syntheses International (https://evidencesynthesis.org/) “provide[s] a global hub where evidence synthesis organizations meet to build and share capacities, resources and guidance, and enhance and advocate for the synthesis and use of research evidence . . . in all areas of human enterprise” (Gough, Davies, et al., 2019, p. 1). Participating RSIs include Cochrane, Campbell, and EPPI-Centre, advancing a multimethodological orientation to systematic reviews. The STEPP (Statistics for Evidence-Based Policy and Practice) Center at Northwestern University (https://stepp.center/) has similar goals in the context of education and applied social science, although their work seems exclusively oriented to quantitative reviews. STEPP’s directors have convened meetings with representatives from several organizations, including WWC and Campbell, “to think about how to get clearinghouses to be better linked with users” (Elizabeth Tipton, personal communication, December 4, 2019). In pursuit of their aims, these entities variously cite and/or link to a diversity of synthesis-supporting resources, including methodological guidance materials, software, and professional development opportunities.
Databases
In addition to databases of syntheses, protocols, and guidance/training materials variously maintained by all RSIs examined (see “Stakeholders” section), three RSIs maintain databases or extractable datasets of primary evidence (e.g., studies, trials) on which researchers can draw to produce new syntheses or audit existing ones. EPPI-Centre provides nine free topic-specific databases (e.g., Bibliomap), partly built from the citations of studies its staff have drawn on when (co)authoring syntheses. WWC offers an open repository of data from individual studies that WWC-certified reviewers have evaluated for quality (in accordance with WWC’s RCT-privileging evidence hierarchy). And Cochrane allows subscribers to access its Central Register of Controlled Trials (CENTRAL)—an extensive bibliographic record of RCTs and quasi-experimental studies, identified through a mixture of handsearching (partially via Cochrane Crowd-sourcing) and, more recently, through machine learning–aided searches (see Cochrane, 2019; Marshall et al., 2018). 16
Metadata
Decisions about how RSIs classify and organize knowledge in their databases relate to decisions around metadata, which are “most simply defined as ‘data about data,’ [and] are a means to name things, to represent data, and to represent relationships” (Borgman, 2015, p. 66). Investments in metadata shape how patterns (and gaps) in studies and syntheses within topic areas can be made visible, including via evidence and gap maps (see Saran & White, 2018). Well-designed metadata are increasingly important, as the work of synthesizing research increasingly relies on algorithmic tools to search for, select, extract data from, statistically integrate, and generate reports about synthesis-relevant materials. Setting aside the bibliographic metadata (e.g., authors, publication date) they associate with publications, RSIs tend to differ in the primary metadata they invest in to support navigation of (and/or data extraction from) their databases: general keywords or topic area/subject matter focus (all); document genre/type, such as protocol or systematic review (Campbell, Cochrane, EPPI-Centre, WWC); and formalized study and/or synthesis features, such as population or (type of) intervention examined (Cochrane, EPPI-Centre, WWC).
For example, Cochrane’s library offers detailed search functionality using MeSH (Medical Subject Headings) and PICO (Population, Intervention, Comparison, Outcome) classification frameworks, in addition to more general topic- and genre-based search functionality. For its part, WWC makes the documents in its synthesis library searchable by topic, synthesis characteristics, and genre, while enabling its primary dataset of individual study evidence to be filtered by formalized features of those studies (e.g., intervention, quality rating). Investments in centralized control over synthesis production (see “Quality” section) may help ensure that metadata are standardized in ways that smooth away some of the dangers of “data friction” (Edwards et al., 2011, p. 669), wherein knowledge objects lack interoperability and are thus ill-formatted for synthesis. A “consistent classification schema”—such as Cochrane’s PICO framework—could also enable data to be easily shared between organizations, promoting “more efficient systematic review workflows,” such that “duplication of effort was minimised” (Gough, Thomas, & Oliver, 2019, p. 8). “Robust hypotheses require information in standardized formats,” Edwards et al. (2013) remind us, adding by way of example, that “the spread of a particular disease around the world cannot be tracked unless everyone is calling it the same thing” (p. 8). Of course, whether and how a phenomenon is (required to be) named has material consequences that can privilege some interests over others (see Bowker & Star, 1999).
Professional Development
WWC, Campbell, Cochrane, and EPPI-Centre each provide formal support for researchers learning to engage in systematic reviews (beyond the formal documentation described in the “Quality” section). For Cochrane and Campbell, professional learning opportunities are available online (e.g., short courses, interactive learning modules, videos) and in a range of face-to-face events. WWC offers training and certification in various aspects of the review process (see “Stakeholders” section) and publishes a list of certified reviewers and organizations, which interested parties may contact for help with (non-WWC sponsored) reviews; it also makes available free online training modules and videos. In addition to offering free instructional videos as well as free software (MetaLight) that provides training in meta-analysis, EPPI-Centre provides a range of professional development services—though these activities typically require external funding. These capacity-building services include direct support and consultations for clients undertaking systematic reviews, as well as EPPI-Centre researcher–led workshops, seminars, college courses, and even two synthesis-related masters of science degrees offered through University College London (which houses EPPI-Centre). Additionally, as noted by one of its current associate editors, RER regularly engages with researchers and other stakeholders in “meet the editor” events at the annual conference of its parent entity, AERA (Alicia C. Dowd, personal communication, December 14, 2019)—with access seemingly dependent on conference attendance fees.
Digital Tools and Automations
EPPI-Centre, Cochrane, and Campbell each make available one or more digital tools that automate, facilitate, or otherwise algorithmically augment some aspect of the systematic review process, including synthesis production. 17 These infrastructural investments shape (and often help standardize) the ways in which syntheses can be produced or consumed, while also augmenting the scale at which data(bases) can be mined, mapped, and synthesized (see, e.g., J. Brunton et al., 2017; Snilstveit et al., 2018). Arguably the most flexible software tool is made available by EPPI-Centre (for a subscription fee, after a trial period): the web-based EPPI-Reviewer, a platform for supporting qualitative and quantitative synthesis work through functionalities for “reference management, study classification and data extraction, synthesis, and general review management” (J. Brunton et al., 2017, p. 170). Included among these functionalities are tools to consistently code and classify studies, tools to generate descriptive maps of evidence (e.g., frequency and cross-tabulation visualizations), and machine learning features to automate some parts of the review process.
Both Campbell and Cochrane provide (or link to) a free suite of software tools for their reviewers, aiding them in complying with the myriad procedural requirements that synthesis production in those infrastructures entails. These tools include an information management system (Archie), a review production platform (RevMan), and a data extraction and screening tool (Covidence)—among other tools the RSIs provide access or link to (e.g., EPPI-Reviewer, the Systematic Review Toolbox; see Cumpston & Chandler, 2019). For instance, Cochrane further links to study summarization and evaluation software (GRADEpro GDT); Campbell provides authors with a stand-alone web-based effect size calculator.
Existing technical developments in synthesis automation like these have paved a path for the emergence of living systematic reviews, so-called because they are “continually updated, incorporating relevant new evidence as it becomes available” (Elliott et al., 2017, p. 24). Because living reviews represent “an approach to review updating, not a formal review methodology” per se (Thomas, Noel-Storr, et al., 2017, p. 32), their livingness can take a number of forms, and be supported by a variety of means—including by machine-based automations (e.g., text-mining and machine-learning algorithms), human-based distributed labor and cognition (e.g., crowdsourcing and task sharing), or some combination of the two (see Elliott et al., 2017; also Marshall & Wallace, 2019; Tsafnat et al., 2014). To date, EPPI-Centre and Cochrane—in addition to the Epistemonikos Foundation (https://www.epistemonikos.cl/), partnered with Cochrane—are among the entities that have begun actively exploring (or, in Cochrane’s case, piloting and publishing) living systematic reviews (see, e.g., Cochrane, 2019; Gough et al., 2017b). Though the emergence of living reviews may represent an important change in the RSI status quo, these innovations have—thus far—largely taken advantage of the digital environment to do a more systematic, rapid, and up-to-date job at what RSIs are already doing: producing well-warranted syntheses in accordance with the assumptions they rely on and the aims they endorse. Under current conditions, “the nature of the final synthesis product may not be all that different to what is currently known” (Gough, Thomas, & Oliver, 2019, p. 9).
Discussion: Critical Questions for RSIS in Education
Today, the idea that syntheses and the evidence underlying them should be methodologically explicit, accountable, and well-warranted (as is expected of primary research) is not much a matter of debate. The questions confronting us are about what kinds of syntheses and synthesis methodologies are valued, about how infrastructural choices affect the shape and possibility of knowledge, and about who should have access to participate in the production and use of that knowledge. Emerging digital innovations may be creating the conditions for new answers to these questions (see Borgman, 2015). Bowker (2017) teaches us that knowledge infrastructures (e.g., RSIs) change in response to the information ecologies in which they participate: Less centrally planned than emergent, a given “knowledge enterprise . . . will ‘learn’ from the network of infrastructures it is integrated into” (p. 397). Below, we review some of the complex choices our RSIs comparisons highlight and consider how such infrastructures might “learn.”
One tension RSIs must resolve is that between the need for common standards to support collective work and the need for flexibility to enable (local) adaptations (Star & Ruhleder, 1996)—a tension that has been further dimensionalized as between global and local, and between technical and social, ways of addressing problems (see, e.g., Borgman, 2015, pp. 35–39; Bowker, 2017, pp. 392–393). RSI comparisons reveal a fundamental contrast in how this tension has been resolved. One set of RSIs focuses on detailed, centralized standards and production technologies that procedurally regulate synthesis conduct and reporting (Cochrane, Campbell, WWC); another set relies more on professional judgment in synthesis production and evaluation (National Academies, RER), or else provides more general frameworks and flexible tools for the development of locally meaningful syntheses (EPPI-Centre). These choices have complex affordances (and limitations) not only for collective work but also for the ability of RSIs to evolve to better meet (or reconsider) their goals in a changing world.
Through investments in centralized, procedurally detailed standards for systematic reviews of interventions (reflecting an RCT-privileging evidence hierarchy), Cochrane and Campbell have been able to catalyze large communities of reviewers and sponsor large numbers of (consistently formatted) reviews—reviews that can be efficiently evaluated, tagged with useful metadata, readily translated/summarized for various stakeholder communities, and efficiently composed (or updated) with the aid of algorithmic tools. To what extent is it the focus on a particular methodology—and this particular methodology—that enables this kind of progress?
While Cochrane and Campbell are working to expand the types of evidence they allow (including qualitative studies), their syntheses of such evidence remains largely positioned—for now at least—as supplementary to studies of the effectiveness of interventions. This historical choice to privilege a particular type of research question and methodological tradition has constrained what stakeholders can learn from approaches that draw on different methodologies and prioritize different questions. In efforts at promoting evidence-based practice, these limitations can go underacknowledged. Consider, for instance, the description on WWC’s homepage: The What Works Clearinghouse . . . reviews the existing research on different programs, products, practices, and policies in education. Our goal is to provide educators with the information they need to make evidence-based decisions. We focus on the results from high-quality research to answer the question ‘What works in education?’” (WWC, n.d.-a, para. 1)
Evaluated in light of WWC’s evidence hierarchy, this statement can be read as implying that some methodologies (including qualitative ones) are not relevant to WWC’s goals—and that these methodologies may disqualify research from being considered “high quality.” To the extent that characterizations like this are codified in policy and law, as WWC’s evidence hierarchy has been in the Every Student Succeeds Act, their reach risks disenfranchising alternative perspectives from which stakeholders might learn.
Direct challenges to a priori evidence hierarchies can be found in the methodologically pluralist stance taken by EPPI-Centre, in the range of critical synthesis genres supported by RER, and in the methodologically flexible consensus reports of National Academies. This press for methodological inclusion is also reflected in the growing attention Cochrane and Campbell pay to qualitative research. The expanded chapter on qualitative research in the recent Cochrane Handbook (Higgins, Thomas, et al., 2019), for example, moves toward procedural guidance for synthesizing qualitative evidence—pointing to named frameworks for this work (e.g., GRADE-CERQual, eMERGe) and calling on synthesists to adapt, adopt, or develop alternative guidance where Cochrane’s guidance is insufficient (Noyes et al., 2019, pp. 538–539). We also note both that “Campbell launched a working group on stand-alone qualitative evidence synthesis in 2018” (Campbell Collaboration, 2019, p. 11) and that the RSI will consider publishing new/innovative synthesis genres—provided they are “accompanied by a plan and timeline to develop methodological expectation checklists (matching MECCIR, where relevant)” and also “include a discussion of the strengths and limitations of the methods used, and how they could be improved” (Campbell Collaboration, 2019, p. 7). To what extent is detailed procedural control necessary to support high-quality reviews?
Evidence that well-warranted, well-respected reviews can be produced—in volume—under a more flexible model can be found in National Academies’ consensus reports, which emphasize careful selection of expert authors, given considerable authority to determine what evidence they synthesize (and how). For the most part, RER has similarly resolved infrastructural tensions in favor of the social and local—though the practices warranting its reviews have fewer layers and fewer reviewers than for National Academies. To what extent does this (local, social, flexible) approach require an organization of the stature (and funding resources) of National Academies to warrant the quality of its reports?
RER—with its critical, integrative reviews—creates an important space for syntheses that bring external theory to bear in critiquing a body of empirical research and the methodologies/philosophies on which it draws. While such critical synthesis genres can be methodologically explicit and accountable, enabling rigorous peer review, they raise substantial challenges to the notion that syntheses should seek to yield an objective, replicable set of conclusions on which stakeholders can rely: The conclusions of such syntheses depend on the critical perspectives from which the body of research is reviewed. Thus, stakeholders are instead invited to critically consider the implications of different approaches to research. 18
It is also worth remembering that shifts toward the local and the social are not without trade-offs. If we are to support collective work in learning from the expanding proliferation of education research and responding to the consequential problems and opportunities research is needed to address, the resolution cannot be abandonment of common standards that enable and sustain shared resources. As Friedman et al. (2017) note, “Without infrastructure, each learning cycle is figuratively a ‘tub on its own bottom’ requiring its own concepts, methods, tools, and support systems” (p. 19). To the extent education researchers seek to promote evidentiary and methodological pluralism over existing hierarchies while also supporting collective work, it seems fair to say that RSIs have much more “learning” still to do.
Furthermore, while methodological transparency is a principle all examined RSIs promote, transparency in quality evaluation seems to have attracted less attention or interest. No RSI we examined conducts open quality reviews of the synthesis reports they support; none, in other words, publishes the full record of the formal evaluations their reports accrue on their way to publication. Relatedly, while some RSIs (notably, Cochrane) are expanding opportunities for stakeholders to comment on knowledge objects, no RSI yet provides for the public, crowdsourced evaluation of syntheses by aggregated/accumulated ratings (beyond collecting and publicizing altmetrics, such as the number of times a synthesis has been downloaded or tweeted about). These choices regarding quality evaluation speak to issues of authority over how knowledge is shaped—and by whom. “Systematic reviewing is a tool of democracy,” Oakley (2017) argues, stressing that methodological transparency is a matter of politics and power: “Too much reliance is placed on our believing what researchers say they found. In a democratic world, we would rather be in a position to decide for ourselves” (p. xiv). If Oakley (2017) is right, it is arguably also true that in a democratic world, RSIs should position their publics to more richly and deeply participate in synthesis production and evaluation. Such a shift depends, of course, on expanding access for the full range of stakeholders—an issue most examined RSIs are investing in or grappling with (as noted above). How can the synthesis of research, viewed as “a public good” rather than “a commodity” (Heywood et al., 2018, p. 131), be made even more publicly accessible—and in what ways can stakeholder publics be further enfranchised in the development of those goods?
There is perhaps particular urgency for us “to re-imagine the engines of knowledge production” (Bowker, 2017, p. 397) at the present moment. Increasingly, RSIs are developing tools to augment or automate synthesis production—developments that may further stabilize (even rigidify) the ways in which syntheses are produced and reported. Edwards et al. (2013) caution that “programmatic efforts to improve science and other knowledge infrastructures have frequently prioritized investments in technical systems over research on how to effectuate equally crucial cultural, social, and organizational transformations” (p. 13). How do we establish common ground (including sufficient commonality in concepts, classifications, and procedures) to enable collective work, without losing the ability to answer important questions that cannot be addressed within a priori evidence hierarchies? As the evidence RSIs synthesize becomes more heterogeneous, what new possibilities for (inter)disciplinary collaboration—between science, social science, and the humanities—can or should be explored (see Teston, 2017, pp. 130–133)? Efforts to support dialogue and resource sharing across RSIs (e.g., Evidence Synthesis International, STEPP) provide promising opportunities for collective learning, but such metacollaboratives are not without risks. It is important we ask: To what extent is the goal to privilege commonality and work toward a synthesis in perspectives—or to privilege diversity and seek practices that sustain dialogue and collaboration across different perspectives (see Moss & Haertel, 2016)? What is excluded from consideration? Who gets to be at the table when such decisions are made? Finding a productive and fair balance may be crucial to enabling “learning” in the face of an unknown future.
These are major challenges—and opportunities—all RSIs face.
Conclusion: The Challenges (and Opportunities) Ahead
The future of knowledge in education will assume different shapes, depending on the infrastructural investments made—in emerging tools, methodologies, genres, and social arrangements. The challenge we have presented to our readers is thus both critical and generative: illuminating the consequences of existing infrastructural configurations (whether designed or inherited) while also imagining how things might be otherwise. We believe this sort of critical reflection and proactive attention to infrastructural design is urgent. As Bowker (2017) notes, infrastructural shifts of the magnitude that the internet enables are “rare in human history” (p. 394). During this period of “emergence” (p. 393), it is possible to engage in “play and experimentation” supporting intentional design (p. 401)—yet “once the die is cast, the infrastructural choice seems inevitable” (p. 393).
Infrastructures never provide a neutral background that serves everyone’s interests equally. They are value-laden, privileging some interests and disenfranchising others. “The study and practice of knowledge infrastructures therefore require new languages of distributive justice that can map change to consequence in more ethical and effective ways” (Edwards et al., 2013, p. 14); considering these consequences at the time of design is crucial, because well-conceptualized infrastructural changes can “not only provide new maps to known territories—they reshape the geography itself” (Edwards et al., 2013, p. 14; see also Bowker, 2017; Slota & Bowker, 2017). We write at a time when changes in fundamental infrastructures supporting research are both inevitable and tractable. Let’s take advantage of it.
Footnotes
Acknowledgements
We are tremendously grateful to colleagues who reviewed earlier drafts for the accuracy and fairness of our representations of each RSI and to those who offered substantive comments on our arguments, including Alicia Dowd, John Easton, Michael Feuer, David Gough, Jennifer Lawlor, Jane Noyes, James Thomas, Elizabeth Tipton, Vivian Welch, and Geoffrey Wong. We take full responsibility for the many omissions and for any errors remaining. This material is based, in part, upon work supported by the Spencer Foundation (under Grant No. 201900070). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Spencer Foundation.