Reviewing Manuscripts Reporting Findings From Single-Case Research Design Studies

Abstract

This article, part of the Learning Disability Quarterly special series dedicated to single-case research design (SCRD), summarizes the review process and outlines how to provide high-quality SCRD study manuscript reviews. We offer recommendations for the entirety of the review process, including accepting the role of reviewer, evaluating quality of SCRD studies, and, finally, adhering to reporting standards with focused attention on the applied nature of the engaging work that encompasses research in learning disabilities. In addition, we discuss (a) participant and setting description, (b) implementation of particular designs, (c) measurement procedures for dependent variable(s), (d) providing evidence of reliability, (e) graphic display of time-series data, (f) descriptions of results, and (g) acknowledgment of design limitations. We describe the challenges facing SCRD researchers and reviewers engaging in learning disabilities research within applied settings, identify fatal flaws that authors cannot address, and point out omissions that researchers can address through the revision process. Implications for practice, resources, and future directions are shared.

Keywords

single-case research design peer review learning disabilities

Science is first of all a set of attitudes.

B. F. Skinner (1953, p. 12).

Peer review of scholarly work is a key mechanism in the advancement of science and an essential service activity for members of the research community (Hancock et al., 2019). Peer review of manuscripts that report findings from single-case research design (SCRD) requires specialized knowledge, including an understanding of the unique characteristics of single-case methodology, applied contexts, and common limitations and challenges faced by single-case researchers. Although learning disabilities (LD) scholars working in the special education field are likely engaged in SCRD coursework during their graduate programs, which provides basic background and understanding of the methodology, some scholars may not use SCRDs to answer their own research questions or may not be proficient in or aware of methodological updates and best practice. When called upon to peer review manuscripts reporting findings from SCRD, scholars with conceptual understanding but minimal practical knowledge and experience may lack sufficient preparation for rigorous and constructive manuscript reviews.

Manuscript review is a specialized skill that requires both understanding of research designs and their application. Although randomized controlled trials (RCTs) are considered the gold standard in intervention research, reviewers must appreciate the contributions of other designs and the conditions that might necessitate them. SCRD is considered to be the gold standard for research in applied behavior analysis (ABA) and may be an appropriate choice for an intervention study in the LD field given the researcher’s purpose, goals, and resources (Tate et al., 2014). Researchers may elect to use SCRD for intervention research to explore the effects of two or more different interventions, settings, or individuals, to test the effects of different combinations of intervention package components, or to test the effects of a modified intervention for a small group of non-responders. Researchers may also be interested in repeatedly observing specific academic and social-emotional behaviors that may develop over the course of an intervention; SCRD provides a practical and efficient means to do so. Furthermore, it may be important and of interest to consider in situ variables that require direct observation (e.g., opportunities to respond, time on task).

The goal of this article is to provide strategies for reviewing manuscripts that report results of SCRD in the LD field. Researchers in the LD field may not have practical background experiences with ABA methods because the nature of service provision for students with LD is largely in inclusive environments (U.S. Department of Education, 2020). Therefore, there is a need to provide additional guidance. LD researchers who serve as reviewers may have content expertise rather than SCRD methodological expertise. This article will assist in providing a more comprehensive review of manuscripts and ensure that published SCRD research meets minimum standards of quality. Figure 1 provides an overview of the review process, beginning with manuscript submission.

Figure 1.

Editorial roles, responsibilities, and processes.

Reviewing Components of a SCRD Manuscript

SCRD Manuscript Title and Abstract

Several design and reporting standards are available to support SCRD manuscript review. For example, Tate and colleagues (2016) developed the Single-Case Reporting Guideline in Behavioral Interventions (SCRIBE) which summarizes reporting standards developed through a Delphi process. The American Psychological Association (APA) also developed guidelines for reporting, Journal Article Reporting Standards (JARS), which include N-of-1 studies (i.e., one person). The SCRIBE checklist notes that manuscript titles should clearly indicate that the study uses SCRD. The abstract should be accurate, nonevaluative, coherent, and readable (American Psychological Association [APA], 2020) and include research questions, population, design (including the independent and dependent variables), results, and conclusions (Tate et al., 2016). A reviewer would expect that the authors clearly convey that participants qualified for services under the LD category and the study employed SCRD. Reviewer feedback regarding the quality of the abstract guides the authors in their revision and would not be considered a substantive change.

Introduction of a SCRD Manuscript

The introduction provides a rationale for a study. In SCRD specifically, one must ensure that the research is conceptually systematic and there is literature upon which to propose its implementation (Baer et al., 1968). This is a foundational tenant of the design. Roberts and colleagues (2004) note that the review process is iterative, and reviewers can best evaluate whether the introduction has provided a strong rationale after conducting a thorough review of the rest of the manuscript.

Roberts et al. (2004) recommend that reviewers revisit their evaluation of the introduction after assessing the entire manuscript. Reviewers can start by simply reading the entire manuscript from beginning to end. Reviewers should then read the introduction and ensure that a clear rationale and framework for the study exists. Next, read the method, results, and discussion sections. Before concluding the review, revisit the introduction and attend to its connection to the methods and discussion (Roberts et al., 2004). It is important to evaluate whether the introduction includes details about previous research that support the authors’ current method and procedures. Ensure that the discussion clearly shows how the previous research included in the introduction is replicated or extended. Reading the discussion followed by rereading the introduction makes it easier to identify inconsistencies. This allows the reviewer to fully understand the authors’ method, results, conclusion, and the connection to the introduction by providing a complete, coherent story of the research process. The reviewer can more effectively evaluate how the introduction provides a framework for the rest of the story. Specific for students with LD and students at risk for LD, one might consider references to specific populations and how the authors compare to their current participants for replication or extension of findings.

When reviewing the introduction, reviewers should consider appropriateness and rigor of SCRD for the purpose of the study. The primary purpose of the introduction in any research paper is to provide a succinct review of the literature and a strong rationale (i.e., justification) for the study (Hill, 2016; Roberts et al., 2004; Tandon, 2014). To address this goal, authors often choose to begin with a brief description of a problem to illustrate the importance of the chosen topic, provide context, and set the stage for the rest of the introduction. Following the brief opening, the introduction should summarize relevant research on the topic as it relates to students with LD or at risk for LD. It should describe the conceptual or theoretical framework for the study and clearly identify the gap(s) in the research for individuals with LD or at risk for LD that the study addresses. The introduction should close with a final, clear statement about the aims, purpose, goals of the study, and the research question(s). The introduction may also include hypotheses. See Figure 2 for a peer review checklist.

Figure 2.

Questions to assist with peer review.

In addition to considering general guidelines for reviewing the introduction of any manuscript, reviewers assessing the introduction of a SCRD study in the field of LD should consider the purpose and justification authors provide for using SCRD. In LD-focused research journals (e.g., Learning Disability Quarterly, Journal of Learning Disabilities, Learning Disabilities Research & Practice), SCRDs are most often associated with studies designed to test academic interventions for students with or at risk of LD. Peltier and colleagues (2020) completed a 10-year review of prominent LD journals and found that 0.63% to 9.02% of published articles reported findings from SCRD studies (Learning Disabilities: A Contemporary Journal = 11 SCRDs of 122 documents—9.02%; Learning Disability Quarterly = 17 SCRDs of 213 documents—7.98%; Learning Disabilities Research and Practice = 16 SCRDs of 195 documents—8.205%; Journal of Learning Disabilities = 3 SCRDs of 473 documents—0.63%). Recent examples of SCRDs published in LD journals include a writing fluency intervention study (Datchuk & Rodgers, 2019), a fractions intervention study (Ennis & Losinski, 2019b), and a paraphrasing intervention targeting improvement in word problem solving (Kong & Swanson, 2019).

Rationales for using SCRD include ethics (i.e., small studies should be conducted prior to scaling), financial (i.e., funding sources are limited), nature of the sample (e.g., students attending a rural school who have comorbid learning and behavioral disabilities which impact mathematics), and feasibility (prior to randomized control trials). When reviewing an introduction, consider how authors can enhance this component of a manuscript. Specific and clear suggestions will best guide the authors in making changes. Remember that the journal has page limits when suggesting major additions; reviewers can suggest portions that might not need to be prioritized. The feedback provided by reviewers allows the authors to shorten those portions to make room for suggested revisions. The alternative leaves authors to delete and abbreviate in a way that may detract from the manuscript. When rating the manuscript’s suitability for eventual publication, remember that the introduction is malleable and that it could be published with changes, even if significant changes are warranted.

Method Section of a SCRD Manuscript

SCRD study methods are unique. Researchers working in applied settings use SCRDs because of the need for practicality and flexibility for both the researcher and participants. Furthermore, variations can be made to the type of design used to meet the participants’ needs (Cooper et al., 2020). Reviewer and editorial feedback on the method should use common SCRD language, provide constructive feedback, and ensure that the method section is clear for readers. Intervention researchers rely on a clearly outlined method to conduct replication studies. The variable eligibility definitions and procedures used to identify students at-risk of being identified as LD are issues that underscore the importance of clarity. For example, simply stating that a student participated in “Tier three” is not sufficient for replication because students with the same designation in different schools, states, or regions (e.g., urban vs. rural) may have very different needs, available supports, and prerequisite experiences. Therefore, it is imperative that reviews evaluate method clarity.

Reviewing the method section entails evaluating the extent the study uses approaches consistent with those used in high-quality, evidence-based research for students with LD (Hill, 2016). Although the special education field has yet to adopt a single set of criteria to evaluate study quality, the research community relies on two main sources for guidance: The What Works Clearinghouse Standards Handbook (WWC, 2020) and the Council for Exceptional Children Standards for Evidence-Based Practices in Special Education (CEC, 2014). A panel of research experts developed the initial WWC standards, which draw heavily on earlier work (i.e., Gersten et al., 2005; Horner et al., 2005) and have refined standards as the methods has progressed. The CEC Evidence-Based Practice Workgroup used the expertise of the members of the workgroup, the WWC standards, previous research (i.e., Gersten et al., 2005; Horner et al., 2005), and feedback from special education researchers to develop the CEC 2014 standards. The WWC (2020) standards provide specific criteria researchers can use to evaluate SCRD studies in regard to data availability, treatment of the independent variable, interrater agreement, residual treatment effects, and demonstration of study effects. Both Maggin and colleagues (2021) and Kratochwill and colleagues (2021) identify areas of WWC standards for further field discussion. Specific areas include (a) limitations of relying solely on design-comparable effect-size measures, (b) importance of the use of visual inspection techniques, and (c) review of the feasibility of using the standards in applied settings. Furthermore, Kratochwill et al. (2021) emphasize the need for inclusion of content pertaining to diversity, equity, and inclusion. The CEC (2014) standards outline study quality indicators on study context and setting, participants, intervention agent, description of the intervention, intervention implementation fidelity, internal validity, and the appropriateness of outcome measures and data analysis. Reviewers should be knowledgeable of field standards and balance standards with the applied methods that researchers are reporting.

The previously mentioned guidelines provide the foundation for SCRD study evaluation, and it is important to note that researchers may provide justifiable reasons for not meeting the indicators. Encouraging an explanation for methodological choices and alignment with standards, rather than relying solely on a comparison, of a method to standards, is often beneficial when reviewing a manuscript. For example, WWC denotes five baseline data points are required to fully meet the criteria. However, a researcher may opt to have fewer baseline data points for several reasons, such as the student earned zeros on all probes, exhibited challenging behavior that placed the student at-risk for harm, or faced other constraints associated with working in applied settings (Harris et al., 2019). Another common constraint in applied settings for students with LD or at risk for LD includes scheduling and allotted time for academic intervention. For example, intervention time may only be a portion of a time block on limited days of the week. This lack of time may impact intervention implementation.

Systematic reviews of research quality indicate that most published intervention studies in LD do not meet all of the quality indicators listed (see Ennis & Losinski, 2019a; Royer et al., 2019; Sanders et al., 2019); yet the studies offer significant contributions to the field. The lack of alignment with research quality standards may be due to a lack of reporting areas of a study or issues in applied settings. Methodological issues, like the number of data points in baseline, cannot be changed based on reviewer feedback. When reviewing methodological suitability, consider the extent to which standards are not met and identify them specifically (e.g., one baseline data point vs. two or violation for just one participant or for all) so the authors can note it in the limitations section.

In addition to using the WWC and CEC guidelines for designing SCRD studies, researchers can also draw upon articles that focus on issues surrounding specific SCRD methodological challenges for additional guidance. Various sources provide useful information for reviewers regarding design selection (Bailey & Burch, 2017; Cooper et al., 2020), data analysis techniques (Manolov & Moeyaert, 2017), visual analysis of single-case data (J. D. Lane & Gast, 2014; Radley & Dart, 2019), single-case effect sizes (Maggin et al., 2019; Parker et al., 2011; Pustejovsky & Swan, 2018), and social validity (Snodgrass et al., 2018).

Reporting standards should also be considered. Authors should clearly identify study design (i.e., reversal, multiple-baseline, alternating-treatment, changing criterion, or combination) along with the rationale for choosing the design that aligns with the research questions provided. Furthermore, authors should note whether randomization procedures were used, including which parts of the study were randomized (Tate et al., 2016). Randomization supports investigator efforts to address the potential for bias. For example, students with or at-risk of LD are a diverse group of students who have many strengths and needs. A researcher may choose the student with more preferable characteristics to begin intervention first in a multiple baseline study (e.g., the least absences, the greatest recorded time on task). These characteristics might result in more positive intervention effects. Randomly assigning participants decreases this risk.

The method section should be written explicitly. A clearly written methods section supports replication efforts and allows future researchers to use the procedures provided in the manuscript. When considering manuscript suitability, consider the need for greater detail or a stronger rationale for decision-making that can be addressed by the authors in the revision process.

Setting

The setting description needs to provide enough detail for replication, translation from research to practice, and inclusion in meta-analyses. The description includes geographic location (e.g., southwestern United States), type of setting (e.g., public school, private school, clinic, community), and any unique characteristics, including location (e.g., rural, suburban, urban). For example, simply stating that the study took place in an elementary school does not provide enough detail. A more specific setting description example is as follows: the study took place in a rural K–5 elementary neighborhood school in a resource classroom for third-grade students with LD, where 25% of the school received free and reduced lunch. In addition, failure to conduct studies in the settings where interventions will likely to be implemented is a threat to validity (Egel et al., 2019). For example, if an intervention addresses organization using an agenda to plan for long-term school-based projects, the logical setting would be school rather than in the community (face validity).

Applied settings sometimes present confounding variables and data collection challenges because it is difficult to control adverse events (e.g., fire drills, changes in schedules, staffing; Bailey & Burch, 2017). However, interventions for students with LD will eventually be implemented by practitioners under these conditions. Nonetheless, it is important that researchers disclose this information as there is potential to present confounding variables (Egel et al., 2019). One way researchers might address confounding issues in applied research is to implement in controlled settings (i.e., one-on-one intervention tutoring). Consider the appropriateness of these conditions in the developmental stages of an intervention package. When evaluating a manuscript, remember that the setting cannot be changed, but the authors’ purpose, rationale, and acknowledgment of limitations can be modified for clarity given the reviewers’ recommendations.

Participants

Thorough descriptions should be provided for all participants. For example, demographic information may include race, ethnicity, age, education level, employment status (if applicable). Rosenberg and colleagues (1994) provided guidance to describe participants with LD and expanding on demographic information to include cognitive ability and achievement. Given states’ differing eligibility requirements for LD, it is helpful for authors to include those so readers have context regarding the participants’ disability and the area of LD that qualifies participants. In addition, it is helpful to define at risk for LD, with information about students’ intervention level and how the school determines student placement within the system. Intervention studies in primary (K–2) settings will likely include more students at risk for LD. When evaluating suitability based on the LD journal’s focus area, consider the contribution to the field of LD. Early intervention is critical even though definitions of at risk for LD are not always clear. As with all intervention studies, participant inclusion and exclusion criteria should also be provided.

Materials

Materials in SCRDs frequently include researcher-created intervention resources provided to the participants for the duration of the study. For example, a researcher conducting a remote coaching study to increase teacher-delivered behavior-specific praise for students with disabilities in their classroom will provide the participant with a researcher-created online module to watch along with the electronics to conduct the study (i.e., Swivl, iPod, and Bluetooth earpiece; Randolph et al., 2021). This practice works in applied settings; however, potential threats to validity, accuracy, and reliability need to be disclosed and addressed. Explicit description of the materials assists readers in understanding the intervention as well as in replicating the study. Generalization of SCRD research requires replication, and reviewers should evaluate the description of materials for ease of understanding and clarity.

Procedures

Procedures for obtaining necessary internal and external review board approvals as well as consent and assent procedures must be clearly stated (Tate et al., 2016). Next, experimental procedures including a step-by-step description and the researcher or team’s actions during each phase of the study, including criteria for determining a functional relation between independent and dependent variables, should be included. Procedures should be outlined in enough detail and operationally defined to support replication and translation from research to practice. For example, rather than stating that coaching was used with teachers of students with LD, the researcher should provide details on what phrases were used during coaching, how often coaching comments were provided to participants, any indicators used to increase or decrease coaching, and length of coaching phases. Generalization of SCRD results depends on replication by multiple researchers and teams in multiple locations (Horner et al., 2012). Service delivery for students with LD varies widely (i.e., inclusion, co-teaching) and with student need, researchers need clearly outlined guidelines and implementation procedures to systematically mimic and replicate the conditions under which the original study was implemented.

Interobserver agreement and treatment fidelity

Although any research likely uses these procedures, they are of particular importance to establishing experimental control in SCRD. Researchers must ensure the reader that no other variable influenced the dependent variable. Therefore, adherence to implementation and reliable data collection are critical. Interobserver agreement (IOA) is the comparison of the degree to which two independent observers report values of the same observed event (Cooper et al., 2020). IOA should be conducted for at least 20% of sessions across all phases of a study, and agreement should least 80% or higher (Kratochwill et al., 2013). Often in LD research, expected IOA is higher due to use of permanent products (e.g., curriculum-based assessments). Although many researchers choose to use a basic total agreements formula, there are limitations that need to be disclosed, or an alternative method of IOA completed. Consideration of IOA procedures, strengths, and alignment with the data collected is important. See Table 1 for a list of common IOA procedures to assist with reviews and feedback.

Table 1.

Options for Evaluating Interobserver Agreement.

Method	Procedure	Advantages/disadvantages
Total count	1. Sum total counted for primary observer 2. Sum total counted for secondary observer 3. (smaller value/larger value) × 100	Easiest Cannot analyze data to find specific behavior related to the disagreements
Partial agreement within intervals	1. (smaller value/larger value) × 100 within each interval 2. Average across intervals	Extent of agreement unknown
Exact agreement within intervals	1. Code as “non-agreement” or “agreement” for each interval 2. Sum total number of “agreement” codes across intervals 3. (agreements/total number of intervals) × 100	Provides greater specificity about reliability of observations
Scored interval	1. Disregard all intervals with both observers coding as non-occurrence of behavior 2. Sum total number of intervals with an “agreement” coded by both observers 3. (total number of agreements/total number of intervals retained) × 100	Focus is on observers’ ability to identify the occurrence of the behavior
Unscored interval	1. Disregard all intervals with both observers coding as occurrence of behavior 2. Sum total number of intervals with an “agreement” coded by both observers 3. (total number of agreements/total number of intervals retained) × 100	Focus is on observers’ ability to identify the absence of the behavior
Total duration	1. Sum total duration for primary observer 2. Sum total duration by secondary observer 3. (smaller duration/larger duration) × 100	Easiest Cannot analyze data to find occurrence disagreements
Mean duration	1. (smaller duration/larger duration) × 100 within each occurrence 2. Average across occurrences	Can analyze data to find occurrence disagreements

Treatment fidelity (i.e., procedural fidelity) is the degree to which the researcher (i.e., teacher or instructor) implemented assessment procedures and the intervention as intended (Smith et al., 2007). Treatment fidelity is typically measured at the same time as IOA using a checklist with the steps for implementation (K. L. Lane et al., 2004). Reviewers may look for notation of live fidelity observations or recording review. A more rigorous procedure for treatment fidelity data collection is the use of two observers who consistently compare their IOA ratings. If data collection occurs though direct observation rather than through a permanent product, treatment fidelity may occur during IOA sessions.

When reviewing an SCRD manuscript, reviewers should check to ensure that fidelity checks occurred during all phases of the study (e.g., at least 20% of each phase, not just a percentage of total sessions). This applies to all types of designs. For example, data should be collected across baseline and intervention phases in a withdrawal design and across all participants or behaviors in a multiple baseline design. Inattention to a particular phase would raise doubt as to the consistency of procedures or conditions and be a threat to internal validity. Reviewers should also look for additional ways where the researchers ensured treatment fidelity, such as describing how they trained the instructor or prepared manuals (Smith et al., 2007). When evaluating SCRD research, experimental control is paramount because the determination of a functional relation between the independent and dependent variables can only be demonstrated if all other variables are controlled and the visually presented data are reliable. Failure in these areas is a significant flaw that authors may not be able to revise during the review process.

Research design and analysis

Researchers should identify the SCRD used in the study and provide a rationale for use (Johnston et al., 2019). In addition, researchers should describe implementation and include operational definitions as appropriate (Kennedy, 2005). For example, if intervention phases begin when there are five stable baseline data points, the researcher defined stability. Reviewers may look for additional procedures related to phase change. For example, researchers using multiple baseline or multiple probe designs include the amount of change (e.g., 50% improvement over baseline) that should be observed in the first setting, behavior, or student before implementation begins with the second setting, behavior, or participant. The manuscript should include detailed analyses and a rationale for use of each metric. Unique to SCRD, as opposed to other types of design, the detailed descriptions are needed to show experimental control and inform other researchers for replication for generalization purposes. See Figure 2 for questions to support method review.

Results Section of a SCRD Manuscript

Reviewing the results section is different for SCRD when compared to group designs because of the detail necessary. The SCRD results section walks the reader through the visual (i.e., graphic) display of the data shown in graphs. All aspects of the data points and data paths within phases (i.e., baseline, intervention, maintenance) are described. Student data are not aggregated to describe results; each student’s individual performance is clear. The results section should parallel the method, where it is reported with the understanding that intervention treatment effects are influenced by the type of intervention, interventionist, and the setting. The intent is to determine whether a functional relation between the independent and dependent variables is present. At a minimum, researchers should report (a) level—mean or median within a phase; (b) trend—slope or line of best fit; (c) immediacy—change in data between phases; (d) consistency—extent to which data are similar across corresponding phases; (e) overlap—portion of data from one phase that is included in the range of data in a subsequent phase; and (f) variability—differences among data typically measured by distance from the trend line (Krasny-Pacini & Evans, 2018).

When reporting study outcomes for students with LD, particularly when conducting academic interventions, it is important to consider that students may not exhibit linear paths to skill acquisition, which may influence outcomes. For example, when reporting level it may be more accurate to use median rather than mean, as it accounts for highs and lows (Cooper et al., 2020). Traditional treatment effects may not appear evident due to the nuances of LD or intervention characteristics. Immediacy of effect may not be evident when looking at the last data point in baseline compared to the first data point collected during intervention because some students, particularly students with LD, demonstrate slow incremental progress. The same type of progress may also lead to overlap of data between baseline and intervention phases. Another factor is the intervention is designed for incremental progress. For example, reading fluency or speed in identifying sums/products for single-digit mathematics problems are skills that generally develop incrementally and build upon one prior skill. The associated interventions may begin with emphasis on foundational skills that are prerequisites to observed behavior.

Reviewers should consider the outcomes and the significance of outcomes rather than narrowly viewing immediacy of effect or overlap when making effectiveness determinations. Immediacy of effect and overlap are ways of determining efficiency. It is desirable to have efficient and effective interventions, but reviewers should also consider that there are other factors to consider in evaluating effectiveness. For example, not all interventions result in immediate effects, but they are effective with results that are enduring or generalized to other situations. For instance, interventions with focusing primarily on repeated practice may lead to more immediate fluency gains (e.g., reading decoding, spelling, single-digit operations in mathematics) when compared to multi-component interventions culminating in repeated practice. Those other components might lead to critical gains in other domains of the content area and lead to fluency, but not immediately (see Flores, 2010; Hott et al., 2021; Peltier et al., 2020).

Reviewers should note the organization of the results section. Data regarding each research question should be clear and organized according to phases so the reader can observe the visual (i.e., graphic) display (or follow the data table as an alternative to a line graph) while reading the narrative description. An organized results section allows the reader to conclude that researchers demonstrated effects at the appropriate points in time, thus determining a functional relation.

One of the hallmarks of SCRD is the use of visual inspection techniques, with a current debate in SCRD being use of effect size calculations when determining an intervention’s effectiveness. Authors may choose to report both results of visual inspection and treatment effects, or may report one of these. Most importantly, authors must provide a rationale and thorough description of the method used along with any a priori methodological changes. Graphs should be easily interpreted with the following general characteristics: (a) zero on the ordinate raised from the abscissa; (b) free of extra horizontal grid lines; (c) data paths should not cross over phase change lines; and (d) if more than one behavior, setting, or intervention is represented on the same graph with different symbols, a clear legend should be present (Cooper et al., 2020).

When reporting outcomes, it is important to report deviations from the method and the reason for those deviations (e.g., snow day, lawn mowing outside during assessment/intervention). If an author omits a visual inspection measure, this is not likely to cause rejection, rather warrants reviewer feedback requesting the authors to further clarify and revise. Evidence-based practices are gold standards in intervention; some SCRD researchers are using measures of treatment effect either in lieu of or to complement visual inspection techniques. As with any procedure, it is important to provide a rationale, an operationalized definition, and meet assumptions. A summary of common measures of treatment effect, advantages, disadvantages, and recommendations for additional readings is provided as a supplementary document. A copy can be obtained by contacting the first author. See Figure 2 for questions to support study results review.

Discussion Section of a SCRD Manuscript

The discussion section of a manuscript should clearly explain the new knowledge generated based on the result of the study. Given that the results section is limited to reporting data, the discussion section should interpret the results and contributions to the literature. In this section, authors should include a clear statement about the presence or absence of a functional relation. A functional relation is present when the researchers show experimental control over the dependent variable; the data show that the dependent variable changed as a result of manipulation of the independent variable at three different points in time (Kennedy, 2005). The discussion sections include the authors’ main findings and expand on the meaning behind those findings with attention to how their results contribute (or do not) to the larger body of related research. Reviewers should look for connections to the research questions with accompanying explanations. The discussion section should also cover implications of the findings and conclusions for research and practice. Conclusions should be well-justified and should clearly and logically flow from the methods and results. Furthermore, authors should take care not to overstate conclusions or their implications. Any discrepant results should be discussed in consideration of the empirical and theoretical framework for the study and, if warranted, authors should discuss alternative interpretations of their results. Finally, authors should acknowledge study limitations and weaknesses.

In reviewing the discussion section of an SCRD manuscript, reviewers evaluate the extent to which the authors connect the results to research in the introduction, noting similarities and differences. Remember that SCRD depends on systematic replication for generalization, and the results of one study are generalized only to the participants included in the study (Kazdin, 2013). Implications should be limited to statements of promise and potential rather than making sweeping recommendations. When reviewing, it is beneficial to keep in mind that discussion sections can be updated, clarified, and improved with specific and clear feedback. See Figure 2 for questions to support review of the discussion.

Conclusion and Implications for Practice

High-quality peer review is an essential step in the process of disseminating research outcomes. This article provides potential SCRD manuscript reviewers with foundational knowledge related to the design as researchers in the LD field (Gerwing et al., 2020; Maggin et al., 2021). This article also aimed to provide SCRD manuscript review guidance and to provide a basis for future discussion. Diversity of thought and differing perspectives are critical features of the peer review process. Reviewer perspectives are important and contribute to science and manuscript improvement. To ensure a quality contribution to the LD literature, reviewers should have a basic understanding of SCRD design features to provide a thoughtful review (Tennant & Ross-Hellauer, 2020). When reviewing a manuscript disseminating findings from a study, design standards are a basis or starting point. Considering the authors’ purpose, rationale for their method, and implementation settings is also important.

There may be a strong case made for meeting minimum standards while still contributing to the LD field. One might ask authors to provide an explanation if they do not provide one, or expand the explanation if they need to add more clarity. However, the standards exist to maintain quality research that leads to identifying and supporting evidence-based practices (Maggin et al., 2021). Decisions about suitability should outweigh the number of indicators met and to what extent. One variance with others mostly met completely, with few indicators met at a minimum is different from a study in which most indicators are not met with few met at a minimum. Another contributing factor to balance is significance to the field; perhaps the research is a groundbreaking idea in the stage of pilot testing. In addition, one also might consider the research setting and context. For example, studies conducted during the COVID-19 pandemic likely included new and novel confounding variables (e.g., new technology, remote learning, alternate response modalities). However, if instructional modalities in the field of LD evolve because of the pandemic or the field faces other similar challenges in the future, the study under review may contribute significantly with a need for further replication and refinement.

When reviewing manuscripts, keep in mind the hallmarks of SCRD. Authors must demonstrate a functional relation with proof of experimental control. In a case where a functional relation is lacking, feedback may be supportive of the promising work, but experimental control is needed (Cooper et al., 2020). Authors can inform the field with clarity that their approach was not experimental due to various constraints. In a case lacking proof of experimental control, consider whether the authors failed to provide adequate detail or whether data may exist, and further analysis is warranted (e.g., all lessons recorded, but 20% of all phases were not observed). In some cases, meeting the standard may be possible with substantial revision. Peer review improves research and reviewers should be judicious in their decisions to reject and open to continuous improvement.

Reviewing manuscripts is a critical service to the LD field and all research fields. Careful and constructive feedback supports researchers in their development and refinement of interventions for students with LD. The design and reporting standards provide frameworks, and the reviewer evaluates the quality of the manuscript components. There may be some variability in quality level across components, so the reviewer should consider the rationale and issues related to applied settings that might have necessitated deviation from the standards. When deviation occurs, consider whether issues can be fixed with further revision by providing clear and detailed feedback detailing the course of action needed. If a manuscript is not appropriate for the journal or includes a fatal flaw, be supportive and constructive (Belcher, n.d.). Suggest means for improvement or other directions that will assist the authors when revising the manuscript for another outlet and/or continuing the line of research. The goal of peer review is continuous improvement and contribution to the field that will ultimately benefit students with LD. Supportive reviews help researchers, especially early career researchers, persist in and refine their work.

Supplemental Material

sj-docx-1-ldq-10.1177_07319487221089616 – Supplemental material for Reviewing Manuscripts Reporting Findings From Single-Case Research Design Studies

Supplemental material, sj-docx-1-ldq-10.1177_07319487221089616 for Reviewing Manuscripts Reporting Findings From Single-Case Research Design Studies by Brittany L. Hott, Margaret M. Flores, Stephanie Morano, Kathleen M. Randolph and Corey Peltier in Learning Disability Quarterly

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Brittany L. Hott

Margaret M. Flores

Kathleen M. Randolph

Corey Peltier

Supplemental Material

Supplemental material is available on the Learning Disability Quarterly website with the online version of the article.

References

American Psychological Association. (2020). Reporting standards for N-of-1 studies. https://apastyle.apa.org/jars/quant-table-5.pdf

Bailey

J. S.

Burch

M. R.

(2017). Research methods in applied behavior analysis. Routledge.

Baer

D. M.

Wolf

M. M.

Risley

T. R.

(1968). Some current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 1(1), 91–97.

Belcher

(n.d.). Steps in writing good peer reviews. Writing Advice for Academics. https://wendybelcher.com/writing-advice/manage-peer-reviewed-journal/

Cooper

J. O.

Heron

T. E.

Heward

W. L.

(2020). Applied behavior analysis (3rd ed.). Pearson Education.

Council for Exceptional Children. (2014). Standards for evidence-based practice in special education. https://cecpioneers.exceptionalchildren.org/sites/default/files/2021-04/EBP_FINAL.pdf

Datchuk

S. M.

Rodgers

D. B.

(2019). Text writing within simple sentences: A writing fluency intervention for students with high-incidence disabilities. Learning Disabilities Research & Practice, 34(1), 23–34. https://doi.org/10.1111/ldrp.12185

Egel

A. L.

Barthold

C. H.

Kouo

J. L.

Maajeeny

F. S.

(2019). Single-subject design and analysis. In Hancock

G. R.

Stapleton

L. M.

Mueller

R. O

(Eds.), The reviewers guide to quantitative methods in the social sciences (2nd ed., pp. 417–433). Routledge.

Ennis

R. P.

Losinski

(2019a). Interventions to improve fraction skills for students with disabilities: A meta-analysis. Exceptional Children, 85(3), 367–386. https://doi.org/10.1177/0014402918817504

10.

Ennis

R. P.

Losinski

(2019b). SRSD fractions: Helping students at risk for disabilities add/subtract fractions with unlike denominators. Journal of Learning Disabilities, 52(5), 399–412. https://doi.org/10.1177/0022219419859509

11.

Flores

M. M.

(2010). Using the concrete-representational-abstract sequence to teach subtraction with regrouping to students at risk for failure. Remedial and Special Education, 31(3), 195–207. https://doi.org/10.1177/0741932508327467

12.

Gersten

Fuchs

L. S.

Compton

Coyne

Greenwood

Innocenti

M. S.

(2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71(2), 149–164.

13.

Gerwing

T. G.

Gerwing

A. M. A.

Avery-Gomm

Choi

C. Y.

Clements

J. C.

Rash

J. A.

(2020). Quantifying professionalism in peer review. Research Integrity and Peer Review, 5(1), 1–8. https://doi.org/10.1186/s41073-020-00096-x

14.

Hancock

G. R.

Stapleton

L. M.

Mueller

R. O.

(2019). The reviewers guide to quantitative methods in the social sciences (2nd ed.). Routledge.

15.

Harris

K. R.

Stevenson

N. A.

Kauffman

J. M.

(2019). CEC division for research position statement. https://cecdr.org/cec-dr-publications/position-papers.

16.

Hill

J. A.

(2016). How to review a manuscript. Journal of Electrocardiology, 49(2), 109–111. https://doi.org/10.1016/j.jelectrocard.2016.01.001. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4775410/pdf/nihms757065.pdf

17.

Horner

R. H.

Carr

E. G.

Halle

McGee

Odom

Wolery

(2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71(2), 165–179.

18.

Horner

R. H.

Swaminathan

Sugai

Smolkowski

(2012). Considerations for the systematic analysis and use of single-case design research. Education and Treatment of Children, 35(2), 269–290. https://doi.org/10.1353/etc.2012.0011

19.

Hott

B. L.

Peltier

Heiniger

Palacios

Chen

(2021). Using Schema-Based Instruction to improve the mathematical problem solving skills of a rural student with EBD. Learning Disabilities, 19(2), 127–142.

20.

Johnston

J. M.

Pennypacker

H. S.

Green

(2019). Strategies and tactics of behavioral research and practice. Routledge.

21.

Kazdin

A. E.

(2013). Behavior modification in applied settings(7th ed.). Waveland Press.

22.

Kennedy

C. H.

(2005). Single case designs for education research. Pearson.

23.

Kong

J. E.

Swanson

H. L.

(2019). The effects of a paraphrasing intervention on word problem-solving accuracy of English learners at risk of mathematic disabilities. Learning Disability Quarterly, 42(2), 92–104. https://doi.org/10.1177/0731948721994843

24.

Krasny-Pacini

Evans

(2018). Single-case experimental designs to assess intervention effectiveness in rehabilitation: A practical guide. Annals of Physical and Rehabilitation Medicine, 61(3), 164–179. https://doi.org/10.1016/j.rehab.2017.12.002

25.

Kratochwill

T. R.

Hitchcock

J. H.

Horner

R. H.

Levin

J. R.

Odom

S. L.

Rindskopf

D. M.

Shadish

W. R.

(2013). Single-case intervention research design standards. Remedial and Special Education, 34(1), 26–38.

26.

Kratochwill

T. R.

Horner

R. H.

Levin

J. R.

Machalicek

Ferron

Johnson

(2021). Single-case design standards: An update and proposed upgrades. Journal of School Psychology, 89, 91–105.

27.

Lane

J. D.

Gast

D. L.

(2014). Visual analysis in single-case experimental design studies: Brief review and guidelines. Neuropsychological Rehabilitation, 24(3–4), 445–463. https://doi.org/10.1080/09602011.2013.815636

28.

Lane

K. L.

Bocian

K. M.

McMillan

D. L.

Gresham

F. M.

(2004). Treatment integrity: An essential but forgotten component of school-based interventions. Preventing School Failure, 48(1), 36–43.

29.

Maggin

D. M.

Barton

Reichow

Lane

Shogren

K. A

. (2021). Commentary on the what works clearinghouse standards and procedures handbook (v. 4.1) for the review of single-case research. Remedial and Special Education. Advanced online publication. https://journals-sagepub-com-s.web.bisu.edu.cn/doi/abs/10.1177/07419325211051317

30.

Maggin

D. M.

Cook

B. G.

Cook

(2019). Making sense of single-case design effect sizes. Learning Disabilities Research & Practice, 34(3), 124–132. https://doi.org/10.1111/ldrp.12204

31.

Manolov

Moeyaert

(2017). Recommendations for choosing single-case data analytical techniques. Behavior Therapy, 48(1), 97–114. https://doi.org/10.1016/j.beth.2016.04.008

32.

Parker

R. I.

Vannest

K. J.

Davis

J. L.

(2011). Effect size in single-case research: A review of nine nonoverlap techniques. Behavior Modification, 35(4), 303–322. https://doi.org/10.1177/0145445511399147

33.

Peltier

Sinclair

T. E.

Pulos

J. M.

Suk

(2020). Effects of schema-based instruction on immediate, generalized, and combined structured word problems. The Journal of Special Education, 54(2), 101–112. https://doi.org/10.1177/0022466919883397

34.

Pustejovsky

J. E.

Swan

D. M.

(2018). Single-case effect size calculator (Version 0.5) web application. https://jepusto.shinyapps.io/SCD-effect-sizes

35.

Radley

K. C.

Dart

E. H.

(2019). Graphing data and visual analysis. In Radley

K. C.

Dart

E. H.

(Eds.), Handbook of behavioral interventions in schools: Multi-tiered systems of support (pp. 71–88). Oxford University Press.

36.

Randolph

K. M.

Chubb

C. S.

Hott

B. L.

Cruz-Torres

(2021). iCoaching behavior-specific praise in a rural classroom. Rural Special Education Quarterly, 40(1), 4–13. https://doi.org/10.1177/8756870520982302

37.

Roberts

L. W.

Coverdale

Edenharder

Louie

(2004). How to review a manuscript: a “down-to-earth” approach. Academic Psychiatry, 28(2), 81–87.

38.

Rosenberg

M. S.

Bott

Majsterek

Chiang

Simmons

Gartland

Wesson

Graham

Smith-Myles

Miller

Swanson

H. L.

Bender

Rivera

Wilson

(1994). Minimum standards for the description of participants in learning disabilities research. Remedial and Special Education, 15(1), 56–59.

39.

Royer

D. J.

Lane

K. L.

Dunlap

K. D.

Ennis

R. P.

(2019). A systematic review of teacher-delivered behavior-specific praise on K–12 student performance. Remedial and Special Education, 40(2), 112–128. https://doi.org/10.1177/0741932517751054

40.

Sanders

Losinski

Parks Ennis

White

Teagarden

Lane

(2019). A meta-analysis of self-regulated strategy development reading interventions to improve the reading comprehension of students with disabilities. Reading & Writing Quarterly, 35(4), 339–353. https://doi.org/10.1080/10573569.2018.1545616

41.

Skinner

B. F.

(1953). Science and human behavior. Macmillan.

42.

Smith

S. W.

Daunic

A. P.

Taylor

G. G.

(2007). Treatment fidelity in applied educational research: Expanding the adoption and application of measures to endure evidence-based practice. Education and Treatment of Children, 30(4), 121–134.

43.

Snodgrass

M. R.

Chung

M. Y.

Meadan

Halle

J. W.

(2018). Social validity in single-case research: A systematic literature review of prevalence and application. Research in Developmental Disabilities, 74, 160–173. https://doi.org/10.1016/j.ridd.2018.01.007

44.

Tandon

(2014). How to review a scientific paper. Asian Journal of Psychiatry, 11, 124–127.

45.

Tate

R. L.

Perdices

McDonald

Togher

Rosenkoetter

(2014). The design, conduct and report of single-case research: Resources to improve the quality of the neurorehabilitation literature. Neuropsychological Rehabilitation, 24(3–4), 315–331. https://doi.org/10.1080/09602011.2013.875043

46.

Tate

R. L.

Perdices

Rosenkoetter

Shadish

Vohra

Barlow

D. H.

Horner

Kazdin

Kratochwill

McDonald

Sampson

Shamseer

Togher

Albin

Backman

Douglas

Evans

J. J.

Gast

Manolov

. . . Wilson

(2016). The single-case reporting guideline in behavioral interventions (SCRIBE) 2016 statement. Remedial and Special Education, 37(6), 370–380.

47.

Tennant

J. P.

Ross-Hellauer

(2020). The limitations to our understanding of peer review. Research Integrity and Peer Review, 5(1), 1–14. https://doi.org/10.1186/s41073-020-00092-1

48.

U.S. Department of Education. (2020). 41st annual report to Congress on the implementation of the Individuals with Disabilities Education Act. https://www2.ed.gov/about/reports/annual/osep/index.html

49.

What Works Clearinghouse. (2020). What works clearinghouse standards handbook, version 4.1. US Department of Education, Institute of Education Sciences. National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/wwc/handbooks

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB