Abstract
As an editorial team whose training and professional socialization have occurred within different research communities, we have had many discussions about the markers of high-quality research that we value in the literature reviews published in the Review of Educational Research (RER). Apparent in our discussions, debates, and sometimes disagreements is that while our broader definitions of high-quality research align, there remains some variability of perspectives and understandings among members of our editorial team. This variation is only multiplied throughout the peer review process as reviewers and members of the RER editorial board become involved. We are aware, of course, that such variation is likely to arise any time a group of diversely oriented scholars enter into conversation; indeed, this is among the goals of peer review. At the same time, we believe that the endeavor of editorial review and publication requires us to interrogate and illuminate this variation. We consider transparency to be a key principle of this endeavor—an explicit stating of what we consider a high-quality review of research, for others to consider and critique. This standard of transparency is one we have long expected of authors publishing in RER. That is, authors submitting to RER are asked to be transparent in identifying the focal questions driving their reviews, the parameters they establish in querying the relevant literatures, and the standards they apply in arbitrating studies for inclusion. As editors, we seek to hold ourselves to a similarly high standard.
To this goal of bolstering transparency in the editorial process, from peer review to editorial decision-making, we have initiated a series of Methodological Guidance Papers. These papers are intended to serve an educative role in making transparent the process of conducting a high-quality review and the criteria applied in evaluating such reviews. Our intention is for these papers to offer supportive, rather than strictly prescriptive, guidance to potential authors and reviewers of RER. Moreover, we aim to offer guidance that clarifies the notions of quality that we, as editors, envision for reviews that are systematic, meta-analytic, conceptual, thematic, or narrative in nature. Our goal is to support authors who undertake literature reviews and craft manuscripts for publication in RER.
Given the high value we place on the articles comprising the Methodological Guidance Papers, we made several decisions to ensure quality papers. We detail these here in alignment with our overall goal of fostering transparency. First, we decided to draw upon the expertise of our editorial board members for the initial round of articles. This decision was based, in part, on the fact that many board members have published reviews in RER and other venues, reviewed manuscripts of high quality during their tenure on the board, and demonstrated expertise in one or more of the review approaches common among submissions to RER. In September 2018, we issued a call to all RER board members for proposals to write a methodological guidance paper, resulting in a robust pool of proposal submissions.
Each proposal submitted for publication in the Methodological Guidance Paper series was juried by our editorial team, and each paper chosen for the series underwent peer review by us and by other editorial board members, serving as reviewers, at least twice before publication. We felt it important for the rigor of the review process for the Methodological Guidance Papers to meet or exceed that applied to regularly submitted manuscripts. Having undergone this rigorous process of review, two papers have been completed and appear in this issue (February 2020). “The Art and Science of Quality Systematic Reviews” by Patricia Alexander (2019) and “High-Quality Meta-Analysis in a Systematic Review” by Therese Pigott and Joshua Polanin (2019). Both of these papers will remain freely available online. We anticipate adding at least one additional paper to the series.
Following selection, Methodological Guidance Paper authors engaged in collaborative conversations with the editorial team to implement our vision for the series. To date, these conversations have served as a forum for establishing the overarching aims of the series and the necessary and salient components for all contributions. For instance, we asked contributors to include references to quality literature review syntheses that exemplified the specific points they introduced. This was done to ensure that prospective RER authors would have model articles to refer to when developing quality reviews of their own. The editorial team also decided to select two to three exemplary reviews mentioned in each of the Guidance Papers and make them freely accessible through links embedded in each of the Guidance Papers.
Articles in the Methodological Guidance Paper series draw on existing methodological literature on conducting reviews of empirical research, highlighting aspects of that literature in ways that inform readers about key decision points, competing views, and prevailing issues. Specifically, authors contributing to this series have been asked to stress essential decisions that researchers undertaking quality systematic, meta-analytic, thematic, or other types of literature reviews are likely to confront, and how those decisions might be approached. These decisions encompass both broad, epistemic concerns and narrower, procedural ones.
For instance, establishing meaningful criteria for including or excluding articles in the literature synthesis is an epistemic concern that authors of systematic reviews face and one that must tie clearly back to the author’s purpose in conducting the review. For authors of meta-analytic reviews, all articles identified through the search process may be classified according to a set of quality metrics that those authors construct (e.g., only quasi- or experimental articles). In contrast, authors of thematic reviews of empirical works interpreting qualitative data, while speaking more generally about their approach to article selection, may need to offer a more detailed accounting of researcher positionality and participant engagement. Authors of systematic reviews may be aided in addressing procedural concerns, and reduce risk of bias in reporting, by availing themselves of standardized conventions for registering review procedures and protocols via open-access repositories.
To enhance transparency of the criteria applied to evaluating the methods of the literature reviews and syntheses regularly submitted to RER, the Methodological Guidance Papers also explicitly articulate the features and characteristics of high-quality and rigorous reviews of the literature conducted from different methodological lenses or approaches (e.g., narrative synthesis or meta-analysis). For example, one expectation for quality reviews is that they are conducted in a logical and consistent manner (i.e., systematically) and that those methods are explicated in the manuscript. Similarly, the critical question guiding the review should be made apparent to the reader, along with a convincing rationale for its importance to educational research, policy, or practice. Finally, where warranted, articles in the Methodological Guidance Paper series include “how-to” tips, pointers, or recommendations that may prove useful to those undertaking such reviews.
To date, the completed articles that initiate this important and innovative series have surpassed our expectations in every way. We consider the publication of these articles and the commitment to transparency that they underscore to be a strong foundation upon which to base our second term as editors of RER. Our second term will also be enhanced by the addition of a new associate editor, Alexandra List, to our editorial team. As a newly expanded editorial team, we are confident that these articles and the series as a whole will provide valuable guidance to authors seeking to cross the threshold from submitting a manuscript to publishing a research synthesis in RER.