Abstract
Keywords
As editors of leading academic outlets for research in experiential, environmental, adventure, and outdoor education and recreation, our responsibilities include examining how changes in the broader landscape of education and social science research might affect the journals in our charge. These include the Journal of Outdoor Recreation, Education, and Leadership (JOREL), the Journal of Outdoor and Environmental Education (JOEE), the Journal of Adventure Education and Outdoor Learning (JAEOL), and the Journal of Experiential Education (JEE). Recently, the American Educational Research Association (AERA), the Society for Research on Child Development (SRCD), and the American Psychological Association (APA) all produced statements directing how research should be reported in their respective outlets. In our view, these statements merit attention as the kinds of research the AERA, SRCD, and APA publish overlaps substantively with research published in the journals we manage. New standards in these areas are therefore relevant to our collective project as stewards of knowledge in our fields.
The purpose of this essay is to introduce authors, reviewers, and readers to the recent statements issued by the above associations, particularly concerning reports of qualitative and quantitative research. Although such reports are not pertinent to every type of article published in our journals (e.g., conceptual articles), sharing aspects of these statements with authors and reviewers contributing to our journals will, we believe, help improve the quality of scholarship in our fields and thereby increase the impact, significance, and reach of our contributors’ research. Our aim is not to legislate particular research topics or methods, but rather to highlight salient elements of these statements so they can inform the manner in which future research is reported, reviewed, and approached by readers in the journals we manage.
In the following sections, we summarize recent guidelines for published research issued by the AERA, SRCD, and APA and further explain our rationale for adopting some of their recommendations. We then highlight the most pertinent elements of each statement with regard to qualitative and quantitative research, respectively. (Reporting on mixed-methods research is beyond the scope of this essay. See Levitt et al., 2018, for guidance.) We conclude by summarizing key points and advising readers where to seek journal-specific information.
Summary of Recently Published Guidelines by the AERA, SRCD, and APA
Our knowledge of the process by which the AERA, SRCD, and APA developed their respective recommendations is limited to the descriptions provided in their original statements. Rather than recapitulating those descriptions here (they are available in the originals; see AERA, 2006; Appelbaum et al., 2018; Levitt et al., 2018; SRCD, 2020), it seems more useful for us to describe our motivations for adopting these guidelines and for approaching this editorial essay collectively. First, we all serve overlapping communities of inquiry and receive manuscripts on related topics. We therefore thought it would be more efficient to compose one document that provides greater continuity and clarity of expectations for scholars across leading outlets in our field(s). Second, although the guidelines referenced in this essay emanate from associations based largely in the United States, they are aligned with best practices that are currently advocated worldwide (e.g., Wyse et al., 2018). We therefore thought it beneficial to compose a document jointly to ensure any proposed guidelines were not overly restrictive or too narrowly tailored. Our priority remains encouraging diverse submissions while also appreciating the distinct emphases of our respective journals. Third, although the lifespans and publishing histories of the JOREL, JOEE, JAEOL, and JEE differ, they all are moving from peripheral to full participants in the Web of Science and wish to continue progress in this respect because this increases their appeal to a broader array of authors and audiences (we elaborate on this point in the final section of this article). These developments, though positive, nonetheless introduce certain editorial considerations. Externally, it is important to continually demonstrate relevance and rigor in published research. Internally, editorial teams, authors, and reviewers need support adapting to heightened expectations. A freely available, joint statement seemed the best way to initiate this support. Finally, we feel it is important for readers to know that this project was endorsed by our Editorial Boards and the essay reviewed by two peers whose identities were blind to us. 1
Below, we focus on salient parts of each of the AERA, SRCD, and APA statements in terms of their relevance to reporting research on various aspects of educational, recreational, environmental, and adventure experiences in the JOREL, JOEE, JAEOL, and JEE. Our evaluation of these statements reflects our experiences with the hundreds of manuscripts we have received as editors. Note, the associations’ full statements are freely available online, so readers are also encouraged to read those closely and assess for themselves which elements are most applicable to their specific projects. (See reference section for web addresses for the full statements.)
General Guidelines
AERA statement
The AERA’s statement, although released in 2006, is still maintained as the AERA’s definitive standard for reporting educational research (see https://www.aera.net/Publications/Standards-for-Research-Conduct). It also provides the most general guidance. The statement’s foundational concern is that reports should be warranted and transparent (p. 33). Warrantability refers to the adequacy of evidence in justifying results and conclusions, and transparency concerns the explicitness of the logic connecting all parts of the reported study. The AERA statement breaks these two foundational criteria into eight elements comprising a research report: (1) problem formulation, (2) design and logic, (3) sources of evidence, (4) measurement and classification, (5) analysis and interpretation, (6) generalization, (7) ethics and reporting, and (8) title, abstract, and headings. Manuscripts submitted to the JOREL, JOEE, JAEOL, and JEE are sometimes underdeveloped in one or more of these respects, which undermines the warrantability of the claims and/or the transparency of the report. Here, we summarize two important points from the AERA statement that cut across research approaches—issues concerning evidence and generalization—integrating other issues into later sections of this article.
AERA Item 3, sources of evidence (p. 35)
Selecting sources of evidence depends on the purpose of the research, the chosen theoretical framework (which governs methodology), and the analytic aims (e.g., cross-sectional, comparative, richly descriptive). Sufficient detail should be provided about the sample population and any intervention/program to warrant or delimit claims. If the study hinges on a program or intervention, design features that are theoretically relevant to the analysis should be specified in detail, and their relationship to any items of interest should be explained such that the relationship could be investigated in another location or a later study. For single case studies, authors should explain what phenomenon their case represents, describe how the specific sample was selected to represent the focal phenomenon (beyond convenience), and justify the data they opted to collect.
AERA Item 6, generalization (p. 39)
Research reported in the JOREL, JOEE, JAEOL, or JEE is not ordinarily aimed at statistical generalization; however, it often seeks to generate insights that may extend to other contexts, methods, settings, populations, or practices. Moreover, regardless of the original author’s intent, subsequent researchers tend to cite prior work as if its findings apply outside the original research context. Therefore, two points deserve summary here. It will be easiest to quote the AERA statement directly: 6.1. Whether generalization is intended by the author or not, it is crucial to make clear the specifics of the participants, contexts, activities, data collections, and manipulations involved in the study. This includes all of the specifics that are relevant either to the logic by which the study should apply to the generalizations or to permit readers to draw the necessary comparisons to their own contexts of interest. (p. 39)
The AERA statement (2006) further explains that authors should clearly specify the contexts to which inferences might reasonably extend. For example, this could include a similar age or grade level, demographic composition, institutional setting, program purpose, and so forth. In our assessment, knowledge has also advanced to the point where it is no longer sufficient to speak in general terms about an intervention, for example, “an adventure-based teambuilding program,” “wilderness therapy,” “a service-learning project,” or “a school outdoor education program.” Instead, specific, study-relevant aspects of practice should be highlighted to aid readers in making inferences about other settings or for implementing or testing an intervention elsewhere. A good example can be found in the article Increasing and generalizing self-efficacy: The effects of adventure recreation on the academic efficacy of early adolescents (Widmer et al., 2014).
SRCD sociocultural policy
In January 2020, the SRCD adopted a new policy across all its journal platforms. The policy represents an effort to appropriately contextualize findings from research involving human subjects and to avoid problems of replicability and generalizability that stem from the overrepresentation of participants from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) populations (see Henrich et al., 2010). Participant characteristics that might influence (or potentially explain) processes and outcomes of interest are sometimes not sufficiently considered in manuscripts submitted to the journals we manage. This lack of transparency weakens the warrantability of empirical findings. Therefore, we consider the SRCD policy to be a best practice in research reporting. Their policy is as follows: As developmental science becomes more global, and the role of context in human development becomes more evident, it is necessary that SRCD publications provide, in addition to age, an indication of the unique characteristics of the sample and the “socioeconomic and cultural place” from which their findings originate. Accordingly, it is now required that manuscripts to be published in SRCD journals specify clearly in the appropriate section(s) (e.g., Method, Discussion) and in an abbreviated form in the Abstract: (1) the dates of data collection (if applicable); (2) the theoretically relevant characteristics of the particular sample studied, for example, but not limited to: race/ethnicity, socioeconomic status, language, sexual orientation, gender identity (inclusive of non-binary options), religion, generation, family characteristics; and (3) the place(s) from which that sample was drawn, including country, region, city, neighborhood, school, etc. and all other context variables that are relevant to the focus of the publication, except when it violates expectations of privacy and confidentiality by an institutional review board or the setting itself. Additionally, selection and recruitment procedures should be clearly specified in the Method section. (SRCD, 2020, para. 1)
As our journals welcome international submissions across a range of settings and populations, it is increasingly important that authors describe the sociocultural characteristics of their participants, justify the inclusion or exclusion of these characteristics in their analyses, and explain how these factors are pertinent to the research questions, hypotheses, results, and limitations. These explanations should be substantiated by citing relevant literature. Finally, whenever describing research participants' sociocultural characteristics, authors should follow bias-free language guidelines (see e.g.,: https://apastyle.apa.org/style-grammar-guidelines/bias-free-language).
Reporting Qualitative and Quantitative Research
In 2018, the APA released separate statements summarizing the work of two committees charged with providing recommendations for reporting qualitative and quantitative research (Appelbaum et al., 2018; Levitt et al., 2018). Below, we summarize select points from each of these articles that are pertinent to issues we see in the JOREL, JOEE, JAEOL, and JEE, beginning with qualitative research. Again, readers are encouraged to consult the original statements for specific guidelines that might enhance their presentations. Here, we limit our discussion to the points that are most relevant to the concerns we frequently notice as editors.
Reporting Qualitative Research
Reporting style and manuscript structure
Like the committee charged with developing the APA’s recommendations, we do not believe authors should devote space to justifying the use of qualitative methods as such. Authors should justify their methods according to their research purposes, focal phenomena, and tradition of inquiry. By implication, the traditional five-part reporting structure (Introduction, Literature Review, Methods, Findings, Discussion and Conclusion) will serve some but not all qualitative reports. Authors should follow that structure only if it serves their presentation. The structure of a qualitative report will most likely be determined by the research aims, nature of the data, and/or tradition of inquiry. For example, it may include a narrative style of reporting, in which the research endeavor is presented as a story. These reports may be organized thematically or chronologically. They may be presented in a reflexive first-person style, detailing the ways in which researchers arrived at questions, methods, findings, and considerations for the field . . . qualitative researchers often combine Results and Discussion sections, as they may see both as intertwined and therefore not possible to separate a given finding from its interpreted meaning within the broader frame of the analysis. Also, they may use headings that reflect the values in their tradition (such as “Findings” instead of “Results”) and omit ones that do not. (Levitt et al., 2018, pp. 28–29)
Reports of grounded theory, for example, may locate the bulk of a literature review in the latter part of a manuscript, because that is consistent with the methodology (Corbin & Strauss, 2008). These decisions are not haphazard but flow from the study’s methodology and decisions about how best to communicate the study’s logic. Authors should therefore be faithful to the tradition of inquiry guiding their study rather than following a formulaic structure, while also providing a rationale if they depart from the conventions of their chosen tradition. Regardless of these structural choices, authors should still follow a journal’s formatting requirements regarding section heading style and placement.
Rhetorical features
Reports of qualitative research can involve other rhetorical features that might be unfamiliar to some reviewers and readers. These include but are not limited to:
The use of first-person narration typical of the kind of self-reflexivity required when detailing analytic procedures. In quantitative reports, this can be perceived as bias, whereas in qualitative reports it is used to demonstrate transparency and enhance trustworthiness.
The evolution of research questions throughout a project due to the iterative process often used to collect and analyze data.
The alteration of methods (e.g., modifying interview questions, changing observational foci) due to the evolution of one’s research questions, thematic development, or recruitment of new participants (see Levitt et al., 2018, p. 29).
Appearance of these rhetorical features should demonstrate to reviewers and readers that an author is striving for transparency, not that they are biased by their involvement. Authors should avoid solipsism, however, and stay focused on the report as a means of communicating the salient points of a research study. (Autoethnography and autobiography have unique stylistic considerations. See, for example, Sparkes, 2020.) The requirement of transparency places a heavy burden on authors to both reveal and explain the logic of their inquiry as it proceeded, including how their own perspective and judgments shaped its evolution.
Representation of context
A main strength of qualitative research is its ability to represent context, or the “situatedness” of the researcher’s phenomenon of interest (Levitt et al., 2018). Context is represented in at least three ways: (1) “the context of the investigators,” which involves “researchers’ relationship to the study topic, with their participants, and to related ideological commitments”; (2) the “context within which a phenomenon or study topic is being construed” (e.g., a particular cultural setting, historical period, or geographic/environmental space); and (3) the “contexts of [the researcher’s] data sources,” especially as they help to understand participants’ responses or experiences (p. 29, emphases in original). Even if the focus of the research is not on context per se, researchers should describe how the contexts listed above might influence the analysis, themes, and conclusions, as this is a critical aspect of qualitative reporting.
Terminology
Different qualitative traditions use distinctive terminology to communicate their philosophical and methodological commitments. As far as possible, authors should strive for fidelity with the conventions of their chosen tradition of inquiry. This situates a given study in a particular discipline, helps readers interpret its findings, and expands research in outdoor, experiential, environmental, and adventure fields in new directions. Where the meaning of terms is not obvious, such as when technical precision is required, authors should provide definitions to aid understanding and interpretation. At the same time, authors should avoid excessively using jargon that could confuse or obfuscate. Inviting a colleague who is not familiar with the research to read a draft prior to submission is often useful in this respect.
Levitt et al. (2018) also provide some broadly accepted terms authors may opt to use when describing their projects. For example, authors may elect to use:
“approach to inquiry to refer to the philosophical assumptions that describe researchers’ understanding of the research traditions or strategies . . . For instance, they could indicate whether their approaches to inquiry are descriptive, interpretive, feminist, psychoanalytic, postpositivist, critical, postmodern, or constructivist” (p. 32, emphasis in original);
“data-collection strategies” to refer “to the many ways qualitative researchers gather data. These can include activities such as conducting archival research, focus groups, interviews, ethnographic observation, fieldwork, media searches, and reflexive note-taking” (p. 32, emphasis in original);
“data-analytic strategies” to refer “to the procedures used to analyze the data (e.g., constant comparison, eidetic reduction, the generation of themes)” (p. 32, emphasis in original);
“research design” to refer to “the combination of approaches to inquiry, data-collection strategies, and data-analytic strategies selected for use in a given study” (p. 32, emphasis in original). This might involve using a set of conventions governed by a specific tradition, such as grounded theory, or it might involve combining from different traditions to suit a particular study.
Methodological integrity
Even if authors decide not to use the headings above, the categories they reference should be considered required elements of qualitative research reports. Their inclusion is important for establishing a study’s methodological integrity, which reflects how well the literature review is conducted to situate a study’s aims, approaches to inquiry are selected to address those aims, methods and procedures are used in an investigation to meet those aims, and the articulation of implications are grounded in the methods used and the findings produced. (Levitt et al., 2018, p. 33)
In other words, the style and structure of a manuscript should help the reader understand the logic underlying an inquiry and grasp the coherence of this logic across all aspects of a report. Methodological integrity is thus central to the validity of claims made in a report of qualitative research—to its warrantability. When elements of a report are missing or not aligned with each other, it can undermine the trustworthiness of the report and weaken its claims (p. 33). (For an extended discussion, see Levitt et al., 2016.)
Manuscript structure
The APA statement (Levitt et al., 2018) provides a chart outlining the above features in greater detail (see pp. 35–37). Although their chart surpasses the level of specificity we wish to include here, we will summarize and elaborate on some of its recommendations that are sometimes missing in qualitative reports submitted to the JOREL, JOEE, JAEOL, and JEE. Please note that a manuscript’s structural features may vary from the headings used below and may also be applied in different ways. For example, some authors may adopt headings that enable them to discuss the significance of their findings as they are presented rather than separating “Findings” and “Discussion” sections.
Title. Manuscript titles should help readers understand the focus of a study. It can also be helpful to reference the research tradition or approach to inquiry used by the author, or the research outcome (e.g., The positive effects of outdoor spaces on young people’s mental health during a pandemic).
Abstract and Keywords. In addition to summarizing the categories (a) to (d) listed in the preceding section, the abstract should put the study in context, establish its importance, and highlight major findings or claims. In addition, keywords should be selected that (a) are distinct from the title, (b) communicate the essence of the study, and (c) increase the likelihood of the article turning up in an internet search. Authors should consider the kinds of audiences they hope will discover their article and consider questions such as: What terms would those people use in their literature searches? In what other disciplinary contexts do you want your article to be discovered? Authors should select keywords that maximize visibility in desired contexts/audiences.
Introduction. Authors should outline the study’s purpose and explain how the chosen methodology contributes to that purpose. Levitt et al. (2018) offer as possibilities “theory building, explanatory, developing understanding, social action, description, highlighting social practices” (p. 35). These details help to evaluate the warrantability of the claims made in the report. AERA (2006) Item 1 Problem Formulation (p. 34) suggests that introductions to research reports should describe the problem or central issue, establish context for understanding it, and emphasize why it is important to address. Often manuscripts lack these crucial elements or expect the reader to infer them. Authors should not conclude the introduction before furnishing readers with a clear understanding of these points because they establish grounds for the rest of the report.
Review of extant literature. Although not specifically mentioned in the APA statement, it is imperative to include some kind of literature review addressing prior research. This not only situates the current report in relevant fields of knowledge, it outlines key concepts and relationships that inform the present analyses and findings, and discussion of these findings. AERA (2006) Item 1 Problem Formulation (p. 34) also advises on this issue: authors should review the relevant literature, including (a) an up-to-date review of pertinent discussions from the target journal and (b) examples from the wider literature on the focal phenomenon. This approach will help authors demonstrate the relevance of their research within and outside the outdoor, experiential, environmental, and adventure fields. (Good sources of guidance on conducting literature reviews include Boote & Beile, 2005; Randolph, 2009.)
Conceptual framework and research questions. Authors should specify the conceptual framework used to conduct the research, noting that the theory informing the research design may differ from the theory used to design a program. These elements are often conflated, inadequately differentiated, or their relationship insufficiently explained, making study results and conclusions hard to interpret and limiting the advancement of theory and methodology in our fields. In addition, we sometimes receive manuscripts that do not clearly articulate the research questions or focuses driving a study. This can lead to unfocused, low-level analyses along with findings and discussion sections that are neither conceptually illuminating nor practically instructive. Clear, theoretically informed research questions or focuses are critical parts of all empirical reports, including presentations of qualitative research.
Method: Participants and data sources. Qualitative research often involves fewer participants than quantitative studies. Although researchers typically should not feel compelled to justify their sample size, it is important to observe (and state) the limits of their analyses based on their sample size and characteristics. In general, smaller sample sizes lend themselves to very rich description or fine-grained analyses rather than thematic saturation, which requires greater breadth and diversity within the sample. Sample sizes should be justified not by an arbitrary or inappropriate standard (e.g., “statistical power”) but rather by the objectives of the study, the tradition of inquiry employed, the analytic methods used, and the claims advanced by the author. Authors of reports with questionable sample sizes may wish to qualify their claims by using phrases like “provisional,” “exploratory,” and “findings suggest,” and by use of the past tense, as in “participants in this study said/reported . . . ” rather than “participants in outdoor programs experience . . . ” Regardless of sample size, sufficient information should be provided about the research participants to help readers determine whether broader inferences can be drawn from the study (see also the SRCD statement above).
Findings/results. Journal word limits constrain the amount of qualitative data that can be included in a research report. However, sufficient examples should be included to substantiate the major findings and claims. Tabular summaries are sometimes appropriate but are usually not sufficient by themselves, and researcher impressions without evidence from fieldnotes or other supporting data cannot be accepted. Data excerpts should be selected that most clearly illustrate the themes or findings. For brevity, authors may choose to omit data segments determined to be superfluous; however, additional data could be uploaded in a supplementary file if the journal permits (concerns about confidentiality might outweigh the desire for transparency with respect to qualitative data; authors should use their judgment). Data excerpts should be attributed to specific respondents, events, settings, and time points to inform readers of relevant context. To support the plausibility of their interpretation of the data, researchers should consider presenting possible alternative interpretations and offer reasons for rejecting them. If diagrams, illustrations, or photographs are important to include (e.g., in studies involving photo-elicitation), the images should help clarify the method, inform the analysis, or advance the reader’s understanding of the focal phenomenon. If images are not essential for grasping the study’s logic, helping establish context, or substantiating the author’s claims, they should be excluded.
Discussion. Authors should synthesize their findings, specify the inferences that can be drawn from their analyses (bearing in mind there might be several inferences, and they could be contradictory), and argue for their contribution. This typically requires situating the claims in the context of prior research or other relevant literature. The discussion should also make clear what is known as a result of the research that was not known before. In doing so, authors should address questions such as: what conceptual, methodological, or practical issue does the research help address? How do these specific findings help readers to better understand a phenomenon and/or advance knowledge in this area? Acknowledging the study’s limitations is also necessary, either in a separate “Limitations” section or alongside other points.
Conclusion. This section should re-articulate the issues raised in the Introduction, summarize the key points or outcomes of the research, and emphasize its impact and significance in the appropriate context. Application of the research to an international audience is also important for publication in this suite of journals.
Reporting Quantitative Research
The APA statement for reporting quantitative research in psychology (Appelbaum et al., 2018) is very comprehensive. In the following section, we focus mainly on selected items from Table 1: Journal Article Reporting Standards (JARS), which provides information recommended for inclusion in manuscripts that report new data collections regardless of research design (pp. 6–7). We highlight points that are often missed in manuscripts submitted to our journals. As most of the quantitative articles we receive are effect studies of varying durations, we also draw on Table 4: Reporting Standards for Longitudinal Studies (p. 14).
Manuscript structure. Unlike reports of qualitative research, reports of quantitative research should typically follow the five-part reporting structure of Introduction, Literature Review, Material and Methods, Results, and Discussion/Conclusion.
Title. In the title, authors should mention the phenomena under investigation and the relationships between key constructs or variables. Furthermore, the research populations should be identified in the title whenever possible (see SCRD statement above).
Abstract and Keywords. The abstract should begin with a clear statement of the problem under investigation and present the main hypotheses followed by a description of the research population, “specifying their pertinent characteristics for this study” (p. 6). A more thorough and detailed description should follow in the body of the manuscript. The method should be described including the research design, sample size, materials used, outcome measures, and data collection procedures. The main findings should be reported in plain language and should be substantiated by effect sizes and confidence intervals and/or statistical significance levels for frequentist analyses, or Bayes factors and/or credible intervals for Bayesian analyses. The abstract should conclude by reporting implications or applications of the findings. Authors should select keywords that are likely to be used by other scholars searching for relevant studies (see expanded discussion on keywords in qualitative section above). For quantitative studies, a keyword that indexes the methods used in the study is often helpful to include (authors of qualitative reports might also consider this suggestion).
Introduction. Authors should state the importance of the problem, including its theoretical and/or practical implications, and provide a summary statement capturing main themes from the up-to-date scholarship. Thereafter, specific hypotheses, aims, and objectives should be stated.
The Literature Review, or review of previous research, should provide a comprehensive summary and discussion of current scholarship both within the target journal and from other relevant sources. Authors should demonstrate familiarity with recent studies concerning the basic phenomena underlying the focus of their research in other disciplinary journals to align, justify, and characterize their scholarship with reference to recent primary research.
Materials and Methods. This section must enable the reader to retrace all steps in the research process and gauge the findings on the basis of the information provided. Please be aware that not all readers will be familiar with the highly technical language sometimes used to report quantitative research, so basic explanations of terms will enable more readers to access the article. If relevant, authors should report inclusion and exclusion criteria for the participants, including any restrictions based on demographic or logistical considerations. It is important to describe the procedures for the selection of participants as well as the settings and locations where the data were gathered, and details of the sampling plan, including power and precision calculations, if the sample is in fact randomized. It might also be important to identify the dates of data collections. For longitudinal designs, the sample characteristics are especially important and should be reported for each measurement occasion, including reasons for any attrition. As modeling of change over time in statistical analyses can be done in various ways, authors should argue for their choice of analytical strategy based on the specific features of the data and the research questions, i.e., if time is conceptualized categorically (“pre,” “post,” “post-plus”) or as real-time intervals. Especially when results of more than two time points are presented graphically with line charts, we encourage authors to consider how time is reflected in the study.
A central focus of the methods section should be the description and definition of all primary and secondary measures and covariates, including measures collected but not included in the report, if this is critical to understanding the design. If applicable, methods used to enhance the quality of the measurements (e.g., training and reliability of data collectors/experimenters) should be described, and information on validated instruments provided. Researchers must also include reliability estimates for scales used in their research, based on their own data rather than past reports. (Estimates may be reported in either the methods or results sections.) In any case, it must be demonstrated that the measurement model actually worked in the sample. In some studies, this might include establishing a model with factor analytical procedures to verify that the measured change can be attributed to the participant and not the instrument.
Changes to any items or variables (and the supporting rationale) must be reported. When using psychometric scales, this calibration can be difficult to accomplish, especially with small sample sizes. The APA statement (Appelbaum et al., 2018) requires test–retest reliability in longitudinal studies (p. 7), which builds on the concept that rank-ordering of the scores is stable over repeated measures. However, the reliability of observed scores at a given time point does not necessarily translate to their reliability at other times. In cases where growth or decrease of a score is modeled (which is the case in most longitudinal intervention studies we have seen in the JOREL, JOEE, JAEOL, and JEE), we encourage authors to (a) test for partial and/or approximate measurement invariance (MI) of the factor structure of the score and (b) establish measurement models that allow for comparisons over time and between groups (e.g., between classes or gender). We consider it more appropriate to report any difficulties in MI rather than simply assuming stability of the instruments and to critically evaluate the credibility of the results in the discussion section. The different models analyzed in this MI process may be most effectively reported as supplementary material in an online repository, either provided by the journal or on public online services. In such cases, where the stability of instruments over time and/or across groups can only be assumed, warrantability may be aided by arguing for the validity of the instruments from a theoretical perspective.
Depending on the design of the study, specific additional information may be required that cannot be elucidated here. Most importantly, the analytic rationales and strategies for inferential statistics need to be described for primary hypotheses, secondary hypotheses, and exploratory hypotheses.
Results. The total number of subjects (“n”s) in each group at each stage of the study should be reported. In more complex studies, a flowchart diagram might be appropriate for reporting this. With respect to the statistics and data analysis, diagnostic tests that have been run on the data are required, including an analysis of missing data and the chosen strategy to deal with them, descriptive statistics with checking assumptions of normality, and strategies to deal with violations of the latter.
For all inferential statistics, authors should provide the results of all tests conducted, including the exact p-value if null hypothesis statistical testing (NHST) methods were employed. Effect size estimates and confidence intervals should also be included. For regression analyses, correlation tables for all variables in the models are often helpful.
For complex data analyses (e.g., structural equation models, generalized hierarchical models, and factor or other multivariate analyses), the results of model comparisons and fit indices should be reported, together with any estimation problems (e.g., failure to converge or other analytic anomalies).
Discussion. In this section, authors need to make a clear statement for or against all the hypotheses tested in the analysis and discuss the results in relation to similar or different findings in the literature. The results should be interpreted with reference to sources of potential bias, threats to internal and external validity, and the adequacy of sample sizes and sampling validity. Authors should then critically discuss the generalizability of their findings (external validity), taking into account the target population (sampling validity; see SRCD section above) and other contextual issues (e.g., setting, measurement, time, ecological validity). Finally, implications for future research, theory, and/or practical and policy recommendations should be stated.
Conclusion. The conclusion section in a quantitative article is no different from the qualitative research paradigm. Please refer to the section above.
Summary and Conclusion
The purpose of this essay was to review salient points from recently published statements by leading associations and to discuss their applicability to research reports submitted to the JOREL, JOEE, JAEOL, and JEE. Our aim was not to prescribe specific scholarly agendas but rather to provide guidelines for reporting research in the journals we manage. Our effort was occasioned by the 2018 publication of the APA guidelines, the SRCD’s adoption of a sociocultural policy for their journals in 2020, and similar reports issued internationally (e.g., by the British Educational Research Association; see Wyse et al., 2018). As these guidelines are already circulating in academic communities that intersect with ours, it seemed appropriate to examine their implications for authors seeking to publish in the JOREL, JOEE, JAEOL, and JEE, and also for reviewers and consumers of research in these outlets.
Our presentation was selective as our purpose was to identify points most appropriate to our journal contexts. We again recommend that readers consult the original statements for more specific guidelines, which we believe merit consideration over and above what we were able to include here. We also do not wish to close without referencing some of the broader factors that shape how research is reported and journal priorities are established. Although a thoroughgoing discussion of these topics is beyond the scope of the present essay, it seems important to acknowledge, first, the crucial role methodology plays in determining the conduct of research and how it ultimately gets reported. Because the AERA, SRCD, and APA statements focus on research reporting, they understate this basic aspect of the research enterprise. We want to emphasize its importance here and refer readers to sources that explore the topic in greater depth. Of the many fine sources available, two might be of particular interest to researchers in the outdoor, experiential, environmental, and adventure fields. These include Theory and philosophy in education research: Methodological dialogues (Quay et al., 2018) and Research methods in outdoor studies (Humberstone & Prince, 2019).
Second, a significant feature in the current world of academic publishing is the concept of citation metrics. In brief, a citation metric is a numeric value that estimates a journal’s impact, significance, and reach. This value is computed using a number of factors, but a common method involves dividing the number of citations of published articles over a given time period (e.g., two years) by the total number of articles published in that same period. Hence, a higher value ostensibly indicates greater impact and reach of that journal. Other extraneous factors can influence this calculation, for instance, the size of a given field and the way the journal is indexed by its publisher. In our cases, the JAEOL has been produced by a major academic publisher for 15 years, which may position it more prominently than its peers in some evaluative systems such as SCOPUS (see www-scopus-com.web.bisu.edu.cn).
Citation metrics matter to journals because they not only signal a journal’s prestige, they are used in consequential matters like institutional subscriptions and faculty promotion. Citation metrics can influence a library’s purchasing decisions and guide scholars’ choices about where to submit their research. They therefore shape what a journal receives, and thus publishes, along with its fiscal health. Space prohibits us from examining this somewhat controversial topic in any greater depth, but readers can consult Allin et al. (2020) for further discussion. (For more on the most prominent citation metric, Clarivate Analytics’ Impact Factor, see Garfield, n.d. For a critical view on citation metrics, see the San Francisco Declaration of Research Assessment, or DORA, at https://sfdora.org/read/.)
Finally, readers may wonder how the guidelines included in this essay affect the submission processes or publication policies of the JOREL, JOEE, JAEOL, or JEE. To be clear, we are neither proposing nor adopting any blanket policies across all of these journals, and we remain committed to their diversity and autonomy as independent outlets. Readers are advised to consult the submission guidelines pages on each journal’s website for specific guidance stemming from the recommendations outlined in earlier sections. There authors can find further direction on, for example, reporting ethical approval in reports involving human subjects or uploading supplementary files including datasets, instruments, or additional analyses. Journal websites are listed after the references below. Readers are of course also encouraged to contact the respective authors with further questions about the recommendations provided here, which we hope will aid in refining the reporting, review, and consumption of research in outdoor, experiential, environmental, and adventure education or recreation fields.
Footnotes
Acknowledgements
The authors wish to thank Dr. Pat Maher for managing this essay, the two anonymous reviewers for their comments, and Julia Slater and Gaurav Kumar at SAGE Publications for producing this article as a freely available resource.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.