Abstract
An ABAB reversal design was utilized to examine the effects of an electronic daily behavior report card (eDBRC) on non-compliant, off-task, and disruptive classroom behaviors of a 16-year-old with autism spectrum disorder and a 17-year-old with an intellectual disability. The intervention was implemented by two preservice teachers (interventionists) in their final student teaching requirement. In addition to visual analysis procedures, effect sizes (i.e., Tau-U) were calculated. The research questions were: (1) What are the effects of eDBRCs on participants’ disruptive and challenging classroom behaviors, (2) To what extent do preservice teachers implement eDBRCs with fidelity, and (3) Are eDBRCs a socially valid intervention? Upon completion of our study the eDBRC was effective in reducing non-compliant, off-task, and disruptive behaviors. We found that preservice special educators can implement a complex behavior intervention to support the outcomes of their learners during the student teaching experience. Additionally, we suggest eDBRCs are an acceptable intervention for students with autism spectrum disorder and intellectual disabilities.
Teachers report that student disruption, noncompliance, and disengagement are among the most consistent challenging behaviors they need to address on a daily basis (Alter et al., 2013; Gage et al., 2018; Meister & Melnick, 2003); resulting in feelings of under preparedness to manage behaviors that disrupt classroom instruction (Coggshall et al., 2012; Melnick & Meister, 2008). Further highlighting teachers’ need for knowledge and support, Moore et al. (2017) notes via teacher self-reports, that teachers need additional training on classroom management strategies for students with problematic behaviors. Teachers are particularly concerned with managing problematic student behaviors (e.g., talking out, work completion, following classroom rules). One of the reasons for this may be due to preparation programs focusing on ideal classroom environments where students were assumed to be generally well behaved and not what they have observed in authentic classrooms (Meister & Jenks, 2000). Although effective classroom management practices have been identified, a significant gap exists between the effective classroom management research base and teacher preparation (Freeman et al., 2014).
Wesley and Vocke (1992) provide one example of the lack of teacher preparation in classroom management. The authors studied 111 preparation programs and found only 36.9% of the programs had a course specifically focusing on classroom management. Further, two recent reports (Greenberg et al., 2014; Oliver & Reschly, 2010) criticized teacher preparation programs for their perceived lack of comprehensive coverage of research-based strategies to prevent and reduce problem classroom behaviors indicating little change over time. Specifically, only 27% of special education preparation programs sampled by Oliver and Reschly (2010) contained a specific classroom management course. The issues in teacher preparation programs surrounding behavior management go on to impact teachers in the field. Moore et al. (2017) surveyed 160 Pre-K through fifth grade elementary teachers to explore teachers’ knowledge about and implementation of behavior and classroom management strategies. Their results suggest that the majority of teachers in the sample reported to have no or limited knowledge or to be only somewhat knowledgeable for teaching replacement behaviors and for designing, implementing, and evaluating behavior interventions
Research on Daily Behavior Report Cards
Daily behavior report cards are designed to: (a) specify a behavior, (b) rate that behavior at least daily, and (c) share that information with someone other than the rater (Chafouleas et al., 2002). Most often in schools, the classroom teacher serves as the rater, given his or her continual presence throughout the day (Riley-Tillman et al., 2007). Research indicates DBRCs can have positive impacts on academic and social outcomes for students with disabilities (Atkeson & Forehand, 1979; Barth, 1979; Burke & Vannest, 2008; Chafouleas et al., 2002; Smith et al., 1983; Taylor & Hill, 2017; Vannest et al., 2010). Vannest and colleagues (2010) conducted a meta-analysis of single-case research analyzing 17 single-case research studies that examined DBRCs, from 1970 to 2007, involving 107 participants. Using improved rate difference (IRD) as the single case effective size measure, the study reports a wide range of effectiveness of DBRCs (range = −0.14–.97). IRD is interpreted as follows: 50 or lower: small and questionable effect; between .50 and .60: moderate effect; and .70 or higher: large or very large effect (Parker et al., 2009). Mean IRD effect size across studies was 0.61.
Building on the work of Vannest et al. (2010), Riden et al., (2018) conducted an examination and analysis of single-case and group design research literature that focused on the effectiveness of DBRCs from 2007 to 2017. Their review included 390 participants identified as at-risk, eligible for 504 services, or as having a disability (e.g., ADHD, specific learning disability, emotional behavioral disorder). Using Tau-U as the single case effect size measure, the study reports a range of effectiveness for DBRCs (range = .51–.81). Tau-U scores can be interpreted using the following criteria: .65 or lower: weak or small effect; between .66 and .92: medium to high effect; and .93 to 1: large or strong effect (Parker & Vannest, 2012; Rakap, 2015). Using Hedges’ g as the group design effect size measure, the study reports a broad range of effectiveness (range = 0.03–0.72). Interpretation of Hedges’ g effect sizes is as follows: .50 or lower: small effect; between .50 and .80: medium effect; .80 to 1: large effect (Cohen, 1988). Ultimately, within the research studies examined by the two reviews (Riden et al., 2018; Vannest et al., 2010), findings suggest that DBRCs are a viable tool for teachers to use for classroom/behavior management. Beyond evaluating the efficacy of DBRCs for students and classroom teachers, recent research examines the use of DBRCs for novice and pre-service teachers specifically.
Riden et al., (2020) examined the effectiveness of a DBRC implemented by a preservice special education teacher to reduce problematic and disruptive behavior of a high school female with autism spectrum disorder (ASD). A DBRC package consisting of (a) operational definitions of target behaviors; (b) the creation of a simple number rating system; (c) daily monitoring of target behaviors; (d) daily feedback provided to students on her performance of target behaviors and goals; and (e) communicating daily performance to the student’s mother was taught to and implemented by the preservice teacher. Results indicate improvement across three target behaviors (i.e., talks out, looks at other, picks finger). Visual analysis indicated a functional relation between intervention and dependent variables. Data from a DBRC treatment integrity checklist shows that the preservice teacher implemented intervention with 100% fidelity. Aggregated Tau-U results suggest a strong effect [ES = .98, p < .001,] in reducing target behaviors of the student. Based on the previous literature regarding DBRCs, Authors identified and incorporated essential components of DBRCs that increase their potential success on changing target behaviors in the desired direction.
Essential Components of DBRCs
There are five considerations when constructing DBRCs: (a) operationally defining target behavior or constellation of behaviors; (b) rating of behaviors using simple numbers or symbols on a behavior scale; (c) daily monitoring of behaviors; (d) providing feedback to students on their behavior(s); and (e) communicating performance of DBRCs between the student’s teacher and home (Chafouleas et al., 2002; Chafouleas et al., 2007; Riley-Tillman et al., 2007). When operationally defining target behaviors teacher must avoid ambiguous definitions, instead definitions should be specific that leads to enhanced clarity (Mires & Lee, 2017). After behaviors of concern have been identified, teachers should identify positive replacement behavior using the fair-pair method to identify positive behaviors that serve the same function as the inappropriate behavior (Kaplan & Drainville, 1991). After identifying replacement behaviors teachers should create the DBRC with parental input, complete the DBRC daily and send it home with the student for parent review, collect the DBRC the next morning to gain parental insights, collect data on student behavior progress, and use that information to make data based decisions about fading the support (Mires & Lee, 2017).
An additional consideration when designing a DBRC is to focus on principles of ABA. Specifically, when developing a DBRC, teachers should hypothesize the function of behavior by administering assessments that aide in identifying function, develop the intervention with the Premack principle in mind, include reinforcement strategies, identify preferred items that maybe used as reinforcers, and systematically fade supports to prevent reliance on the DBRC. Some additional components teacher and researchers should consider when designing a DBRC whether traditional or electronic the focus on five principles of ABA including hypothesizing function of behavior, the Premack principle, reinforcement strategies, identifying preferred items, and systematically fading supports that can aide in the effectiveness of this intervention. Designing the eDBRC or traditional DBRC with empirically sound ABA principles contributed to the effectiveness and support the efficacy of this intervention.
Technology and Daily Behavior Report Cards
Technology is now ubiquitous in and out of the classroom and is corroborated through the use of non-evidenced base strategies like ClassDojo and LiveSchool. As technology has advanced, so has the use of technology in the classroom. For behavioral/classroom management in schools, most of the strides in technology use have occurred in past 20–25 years (see Anshari et al., 2017). Research that examines technology and behavior management has included the use of pager/vibratory monitors (Markelz & Riden, 2019) and bug-in-ear technology (Scheeler et al., 2012) amongst other technological advancements. Most recently, the development of mobile device application (app) or web-based behavior/classroom management systems (e.g., ClassDojo).
When searching the literature for these systems there is no shortage of instructional how-to’s. Robacker et al. (2016) go into great detail using ClassDojo as a token economy and the implementation procedures, the response cost aspects of the program, as well as providing a vignette about Ms. Zimmerman, a teacher implementing ClassDojo. Along similar lines there are articles that describe the plethora of other technologies teachers can choose from. Cumming (2013) provides a list of 50 possible mobile devices/apps that can be used to implement interventions. This information is only a few results out of thousands of search results connected to systems used to manage behavior. Yet, empirical evidence of their effectiveness is incredibly limited unlike several traditional approaches to supporting students.
Electronic Daily Behavior Report Cards
To date, only a few empirical studies combine technology and DBRC implementation exists. Williams et al. (2012) examined emailed DBRCs with and without performance feedback for elementary-aged students displaying disruptive behaviors and found that an e-mailed DBRC decreased disruptive behaviors. Yeo and colleagues (2018) examined the use of online DBRCs for students with ADHD who engaged in off-task behaviors in Singapore. In their study the online DBRC intervention was modified to involve an additional school mentor who supported the parents in monitoring and guiding the students. Results indicated that the online DBRC intervention had been effective in decreasing off-task behavior in three students with ADHD.
Purpose of Present Study and Research Questions
The purpose of the current study was to examine a novel approach to manage challenging and disruptive behaviors in the classroom. As mentioned above only one study has been conducted using technology to implement a DBRC. Specifically, this study seeks to use free and accessible technology via Google to implement an electronic daily behavior report card (eDBRC) to foster positive student behavior implemented by preservice teachers. The research questions are as follows: What are the effects of eDBRCs on participants’ disruptive and challenging classroom behaviors? To what extent do preservice teachers implement eDBRCs with fidelity? Are eDBRCs a socially valid intervention?
Method
Participants, Interventionists, and Settings
Two high school students and two preservice special education teachers participated in this study (see Table 1). Henceforth preservice teachers will be referred to as interventionists. One white female high school student with ASD (Sarah) and one white male student with multiple disabilities (ID, Mike) and their guardians/parents participated. The teachers were entering their final student teaching experience at a large public university in the northeast. One teacher was a 38 year old white female (Hannah) and the other teacher was a white 22 year old male (Nelson). Inclusion criteria for the teachers included working with students with disabilities who were displaying disruptive or challenging behavior, placement in a K–12 academic setting, had never implemented a DBRC of any kind, and had access to technology (e.g., laptop, tablet). The teachers and their cooperating teachers were asked to refer a student with disabilities who displayed high frequencies of off-task behavior.
Participant Demographic Information.
Note. ASD = Autism Spectrum Disorder; B.S. = Bachelors of Science; ID = Intellectual Disability; N/E = No Experience
The study took place in a public high school serving students in 8th–12th grade (total high school population = 2,301) in the northeast. The intervention for Sarah was implemented in an autism support classroom consisting of seven students. Instruction during this time was delivered with Hannah sitting at a table with three peers facing the teacher. Intervention for Mike was implemented during English language arts (ELA) and Art class consisting of nine students each including Mike. Instruction during this time was delivered with Mike sitting at a table with two peers facing the teacher during ELA (small groups) and with eight peers (whole group) facing the teacher during Art.
Dependent Variables and Data Collection
Two behaviors were targeted for intervention for each student participant. For Sarah the behaviors were talking out and off-task behavior. Behaviors for Mike were non-compliance and off-task behavior. Each target behavior was operationalized prior to data collection. A frequency count was used to collect data on each target behavior. To collect frequency data a tally mark was placed on the data sheet whenever the students engaged in the behaviors. Operational definitions of target behaviors for Sarah were as follows: talking out was counted as any occurrence of an uninvited verbal statement during whole group instruction, independent work, or silent reading. Verbal statements included academic or non-academic verbal statements (e.g., shouting out answers). Talking out was not counted if they were a result of stereotypy, interactions with peers during group work, or asking her paraprofessional a question or for help. Off-task was counted as engaging with personal possessions (e.g., necklace, headphones), out of seat behavior, picking at self (e.g., picking at lips, biting fingernails). Operational definitions of target behaviors for Mike were as follows: non-compliance (i.e., any occurrence of an uninvited verbal statement during whole group instruction, independent work, or silent reading but not if it was a result of stereotypy, interactions with peers during group work, or asking the paraprofessional a question or for help) and off task (i.e., engagement in any tasks other than the assigned task or ongoing activity but does not include asking the teacher or paraprofessional for help).
Interobserver agreement
We conducted mean count per interval interobserver agreement (IOA) on 40% of baseline phases on frequency count data collected by Hannah. The first author sat in an inconspicuous spot in the classroom and independently collected data on the target behavior during the 60-min intervention period. At the end of each IOA observation period the first author compared IOA data to Hannah’s data. Mean count per interval IOA data for baseline were 89.75% across all behaviors (range = 87.5% to 92%). Mean count per interval IOA was also calculated for a minimum of 20% for each intervention, withdrawal, fading, and maintenance phases. Mean count per interval IOA data for intervention, withdrawal, fading, and maintenance phases were 89.2% across all behaviors (range = 72% to 100%). The first author conducted mean count per interval interobserver agreement (IOA) on 29% of baseline phases on frequency count data collected by the Nelson. The first author sat in an inconspicuous spot in the classroom and independently collected data on the target behavior during the 60-min intervention period. At the end of each IOA observation period the first author compared IOA data to the Nelson’s data. Mean count per interval IOA data for baseline were 80.15% across all behaviors (range = 64% to 90%). Mean count per interval IOA was also calculated for a minimum of 20% for each intervention, withdrawal, fading, and maintenance phases. Mean count per interval IOA data for intervention, withdrawal, fading, and maintenance phases were 89% across all behaviors (range = 81% to 96%). According to Kratochwill et al. (2013) a summary of IOA for a variable must be based on at least 20% of data points within each condition with acceptable values of IOA at ≥ 80%. We calculated mean count per interval IOA using the following formula describe in Cooper et al. (2020, p. 113)
eDBRC Trainings
Interventionist training
The interventionists were provided training to introduce them to the study and train them on the data collection and intervention procedures. Training session were developed based on previous DBRC studies (See Riden et al., 2020). One day of training was provided prior to baseline data collection and included: a) the rationale for eDBRC as an intervention, b) the essential components of eDBRC, c) implementation procedures, d) operational definitions, e) opportunities to practice baseline data collection procedures, and f) time to answer any questions and lasted one hour. The second day of training consisted of instruction on a) how to conduct the functional assessment screening tool (FAST), b) how to conduct a multiple stimulus without replacement (MSWO) preference assessment, c) intervention training, d) intervention data collection procedures and practice and e) scheduling of weekly intervention meetings and lasted one hour. The trainings we conducted separately for each interventionist.
Participant training
A participant training session was conducted after baseline data were collected. The training included rationale for support, how support will be delivered, what the eDBRC will look like, and instruction on replacement behavior. Sarah’s replacement behaviors included: raising a hand to ask or answer a question instead of talking out and engaging in academic activities as opposed to engaging in off-task behaviors, while Mike’s was complying with a task demand the first time a request was made and engaging in academic activities. Once the replacement behaviors were introduced and modeled for the participants, the participants practiced the replacement behaviors until the behaviors were performed to at least 80% mastery.
Parent training
As part of the preservice teachers training on the eDBRC a section of time was dedicated to training the parents on their role in the intervention. Parent training was conducted on the use of the eDBRC. The pre service teachers met with the parent/guardian in the autism support classrooms and gave a brief presentation explaining the target behaviors, introducing the eDBRC, and defining roles in the intervention. Time was allotted at the end of the presentation to answer any questions of the parents/guardians.
Experimental Design
An ABAB reversal design was used to measure functional relations between the independent variable (i.e., eDBRC) and dependent variables (i.e., target behaviors; Cooper et al., 2020). The experimental conditions included baseline (i.e., business as usual [no eDBRC]), eDBRC intervention, return to baseline (back to business as usual [no eDBRC]), return to intervention, fading, and maintenance. The business as usual condition involved no manipulation of rules, routines, and expectations across settings. In all settings classroom rules were in place and posted in clear view for all students to see. Within all classes surface management techniques such as proximity and redirection were used as well as the use of praise statements. Additionally, negative and corrective statements were delivered often.
Pre-baseline
Prior to baseline the first author conducted preliminary behavior screenings using both indirect and direct methods, to determine the appropriateness of a behavior change program. Behavior was then defined according to Bicard et al. (2012) who state behaviors should be defined in an observable manner, describes the behavior in measurable terms and positive terms, and is clear, concise, and complete. Interviews were conducted with the interventionists and the classroom special education teachers familiar with the participants. Based on answers to interview questions the author conducted three one-hour direct observation sessions. Continuous measurements were used to collect antecedent-behavior-consequence data so all instances of the behaviors of interest were detected during the observation sessions (Johnston et al., 2019). Based on the results of the direct and indirect measurements the interventionists hypothesized the functions of the target behaviors. Next, replacement behaviors were selected based on the hypothesized function of the target behaviors.
Independent Variable
Electronic daily behavior report card
An eDBRC package, adapted from Riden et al. (2020) was created and consisted of operational definitions of target behavior, simple number rating system, daily monitoring of behaviors, feedback provided to participants on their behavior, and communication of performance between the teachers and home. The eDBRC was created using Google Forms and daily summary sheet using Google Sheets. The eDBRC was completed by the teachers each day. The daily summary sheet was used to communicate participant progress to the guardians/parents and included the following (See Figure 1):
Electronic daily behavior report card.
two yes/no questions for the teachers to select from a drop down menu (i.e., today we did a “check-in” with yesterday’s card, today we did a “check-out with today’s card) a space for the participant’s ID number and the date sections for each target behavior sections to provide a rating for each behavior using a drop down menu (i.e., yes, no) two sections at the bottom of the DBRC for teacher and parent comments
Intervention sessions were divided into six 10-min intervals during 90-min periods. At the beginning of each observation session the teachers reviewed the behaviors necessary to earn points for the class period, reminded the participants about reinforcers that could be earned contingent on performance, and delivered all instruction on behavior expectations in a positive manner. Throughout the eDBRC period, the interventionist collected behavior data, reviewed performance at the end of each interval with the participants, assigned points, reviewed points not earned in a non-threatening tone, and described how the participants could earn points next time. Behavior-specific praise was provided to the participants during each interval. At the conclusion of observations, the interventionists reviewed overall performance using the eDBRC. After which, the participants were granted access to reinforcers contingent on their performance (i.e., Sarah earned time with a preferred teacher in the class to work on a puzzle book and Mike earned time working with a preferred teacher to create artwork and notes for his mother). Lastly, the interventionists provided feedback to the guardians/parents using the eDBRC Summary Sheet and electronically shared it with guardians/parents for review at home and delivery of reinforcement contingent on participants performance at school. See Figure 2 for a procedural checklist for implementation.
Treatment integrity checklist.
Treatment integrity
The first author conducted treatment integrity checks on 20% of intervention and maintenance phases. A 16-item treatment integrity checklist was used to assess the extent to which the intervention was correctly implemented. The checklist was separated into three distinct sections: (1) upon arrival, (2) throughout the DBRC period, and (3) end of DBRC review period. Additionally, a second observer (doctoral candidate in special education) who was naïve to the study used the DBRC treatment integrity checklist to calculate treatment integrity during 20% of the treatment integrity observations. IOA for implementation fidelity was 100%.
Fading
Using participant data to determine responsiveness and abruptly ending an intervention often leads to a participant reverting to preintervention levels of behavior (Estrapala et al., 2018). Removing components or dosage of an intervention while monitoring participant behavior allows educators to analyze the effects of intervention components and increase the likelihood that positive behavioral change will maintain over time (Rusch & Kazdin, 1981). Fading began with each participant when the data were at low levels and had relative stability. After intervention was implemented with check-ins for each 10-min interval (total of 6 each day), intervention check-ins were faded to one check-in before the session, one check-in at the 30-min mark (after the third 10-min interval) and one check-in at the conclusion of the session. The intervention was faded further after three days of steady responding to one check-in before the start of the session and one check-in at the conclusion of the session. Finally, intervention was withdrawn when behavior maintained stability at low levels, and behaviors were moved into maintenance.
Maintenance
Maintenance data were collected on each target behavior after the fading procedures to see if the participant engaged in the desired behavior after intervention was removed. The interventionists collected maintenance data for 3 days over the span of two weeks.
Data Analysis
Data analysis consisted of visual analysis of the data and Tau-U effect size to examine the effectiveness of the eDBRC intervention. In visual analysis, level, trend, and variability in the data are examined when evaluating experimental control (Cooper et al., 2020) across all behavior to determine a functional relation between independent and dependent variables. Median for each condition across target behaviors were used to determine level changes. When discussing level we used the terms deteriorating and improving. Level changes were considered to be improving if the behavior reduced from baseline levels and were considered to be deteriorating if the levels of behavior increased.
The split middle method was used to determine a therapeutic or non-therapeutic trend (Lane & Gast, 2014). A therapeutic trend refers to a decreasing trend in the data. Conversely, a non-therapeutic trend refers to an increasing trend in the data. A stability envelope criterion of 80% of data must be within ±30% of the median to be considered stable. The distance from the median is calculated by drawing horizontal lines above and below the median line using the distance from median total to complete your envelope. If 80% or more of the data fall within the bound of the stability envelope the data is considered to be stable. Stability was determined for each phase from baseline through maintenance.
Statistical analysis was calculated to examine effectiveness of eDBRCs on target behavior within and between phases. Tau-U was calculated to show percentage of non-overlap between phases or percentage of data showing improvement between phases (Parker et al., 2011) for baseline data and phase one data (to detect immediate effect) and baseline data and total intervention (overall effectiveness) data for all target behavior separately. An online calculator was used to calculate Tau-U effect sizes. (Vannest et al., 2016). The Tau-U formula was
eDBRC Social Validity
The teachers, participants, and parents completed an adapted version of The Usage Rating Profile-Intervention (Chafouleas et al., 2009) social validity survey to access the acceptability of the eDBRC intervention. Each social validity survey used a Likert-type scale of one through six (i.e., 1 = strongly agree, 2 = agree, 3 = somewhat agree, 4 = somewhat disagree, 5 = disagree, and 6 = strongly disagree). The social validity survey were delivered in sealed envelopes after the completion of the study and asked to be returned to school the following day.
Results
Effects of eDBRC on Participant Behaviors
Visual analysis for Sarah
There was an immediate and decrease in level of off-task behavior from baseline to the first intervention phase for Sarah. During baseline, data indicate off-task behaviors were occurring at a high level (Median = 38). A marked decrease in level occurred when the intervention was introduced (Median = 7). During the second baseline phase there was an increase in level (Baseline2 Median = 16) and decrease in level when the intervention was reintroduced (Intervention2 Median = 2). During fading phase one, there was a slight increase in level (Median = 5). Fading phase two resulted in a slight decrease in level (Median = 3). Once off-task behavior was moved to maintenance phase a slight decrease in level (Median = 2).
Sarah’s data indicate an immediate change in level from the first baseline to intervention phases for talking out behavior. During baseline, data indicate talking out behaviors were occurring at a high level (Median = 25). A noted change in level occurred when intervention was introduced (Median = 8). During the return to baseline and intervention phases there was an increase in level (Baseline2 Median = 15) and decrease in level (Intervention2 Median = 7). During fading phase one, there was a further decrease in level (Median = 3). Fading phase two resulted in a slight increase in level (Median = 6). When talking out behavior was moved to maintenance a slight decrease in level was observed (Median = 2).
Trend analysis of off-task behavior data for Sarah resulted in a non-therapeutic trend during baseline, a therapeutic trend when intervention was introduced, a non-therapeutic trend when intervention was removed, a therapeutic trend when intervention was reinstated, and a therapeutic trend during fading phases one and two. After this behavior was moved to maintenance a therapeutic trend was observed in the data. Trend analysis of talking out behavior for Sarah resulted in a non-therapeutic trend during baseline, a therapeutic trend when intervention was introduced, a non-therapeutic trend when intervention was removed, a therapeutic trend when intervention was reinstated, and a non-therapeutic trend during fading phases one and two. After moving behavior to maintenance a therapeutic trend in the data was observed.
An analysis of baseline data of off-task behavior for Sarah indicates a stable baseline. Data during intervention phase one and withdrawal were stable. When intervention phase two was implemented the data were not stable. During fading phases one and two, the data returned and maintained stability. Visual analysis of maintenance data suggests minimal variability result in a result of not stable. An analysis of baseline data of talking out behavior for Sarah indicates a stable baseline. Data during intervention phase one, withdrawal, and intervention phase two were stable. Data were not stable during phase one of fading but returned to stability during the second fading phase. Visual analysis of maintenance data suggests stability in the data for the talking out behavior.
Visual Analysis for Mike
There was an immediate change in level from initial baseline to initial intervention for off-task behavior for Mike. During baseline, data indicate off-task behaviors were occurring at a high level (Median = 34). A change in level when intervention was introduced (Median = 10). When intervention was removed an increase in level (Baseline2 Median = 26) and a drop in level when intervention was reintroduced was observed (Intervention2 Median = 11). During fading phase one there was a slight increase in level (Median = 12). Fading phase two resulted in a decrease in level (Median = 5). When off-task behavior was moved to maintenance there was a slight increase in level (Median = 6).
For Mike there was an immediate change in level from baseline to intervention phase one for non-compliant behavior. During baseline, data indicate non-compliant behaviors were occurring at a high level (Median = 23). A change in level when intervention was introduced (Median = 10). When intervention was removed there was an increase in level (Baseline2 Median = 16) and saw a drop in level when intervention was reintroduced (Intervention2 Median = 13). During fading phase one there was a decrease in level (Median = 9). Fading phase two resulted in a decrease in level (Median = 6). When non-compliant behavior was moved to maintenance a decrease in level was observed (Median = 4).
Trend analysis of off-task behavior data for Mike resulted in a non-therapeutic trend during baseline, a therapeutic trend when intervention was introduced, a non-therapeutic trend when intervention was removed, a therapeutic trend when intervention was reinstated, a therapeutic trend during fading phase one, and non-therapeutic trend during fading phase two, and a therapeutic trend during maintenance. Trend analysis of non-compliant behavior data for Mike resulted in a non-therapeutic trend during baseline, a therapeutic trend when intervention was introduced, a non-therapeutic trend when intervention was removed, a therapeutic trend when intervention was reinstated, a therapeutic trend during fading phase one, and non-therapeutic trend during fading phase two, and a therapeutic trend during maintenance.
An analysis of baseline data of off-task behavior for Mike indicates a stable baseline. Data during intervention phase one, withdrawal, and intervention phase two were stable. Data were not stable during phase one of fading but returned to stability during the second fading phase. Visual analysis of maintenance data suggests stability in the data for the off-task behavior. An analysis of baseline data of non-compliant behavior for Mike indicates a stable baseline. Data during intervention phase one, withdrawal, and intervention phase two were stable. Data were not stable during phase one of fading but returned to stability during the second fading phase. Visual analysis of maintenance data suggests stability for the non-compliant behavior.
Effect Size Analysis
Tau-U scores were calculated for Sarah and Mike to determine the effect of the eDBRC intervention on reducing the frequency of target behaviors. Tau-U was calculated per participant per behavior, and as aggregated across participants. Tau-U results for Sarah are as follows: off-task [ES = 1.00, p < .001, CI (95%) = .47–1.00]; talking out [ES = .88, p < .001, CI (95%) = .35 – 1.00]. Aggregated Tau-U for Sarah suggests a strong effect on behavior [ES = .94, p < .001, CI (95%) = .41–1.00]. Tau-U results for Mike are as follows: off-task [ES = 1.00, p < .001, CI (95%) = .49–1.00]; non-compliance [ES = .87, p < .001, CI (95%) = .362–1.00]. Aggregated Tau-U for Mike suggests a strong effect on behavior [ES = .94, p < .001, CI (95%) = .42–1.00]. Overall, there was a calculated strong effect size across behaviors for both participants [ES = .94, p < .001, CI (95%) = .42–1.00]. A Forest Plot is included to visually represent Tau-U results (see Table 2).
Results of Statistical Analysis and Forest Plot.
Note. CI = Confidence Interval; ES = Effect Size; Tau-U effect sizes 0.65 or lower = small effect; between 0.66 and 0.92 = medium to high effect; and 0.93 to 1.00 = very high effect
Interventionists eDBRC Implementation Fidelity
Hannah implemented the intervention with 98.4% fidelity (range = 92%–100%) during each fidelity probe. Reliability was 100%. Nelson implemented the intervention with 95% fidelity (range = 86%–100%) during each fidelity probe. Additionally, treatment integrity reliability was conducted on 20% of treatment integrity sessions for each interventionist. Reliability was 100%. Results of treatment integrity data analysis suggest interventionists can implement an eDBRC intervention with high levels of fidelity.
Social Validity of eDBRC
On the six-question social validity survey, Hannah and Nelson scored all of the questions as Strongly Agree or Agree (M = 1.33) lending strength to the social validity of the eDBRC intervention from a teacher’s perspective as did the participants (M = 1.58). Likewise, the Sarah’s guardian scored all of the questions as Agree or Somewhat Agree (M = 2.16) lending strength to the social validity of the eDBRC intervention from a parent or guardian perspective. Mike’ parents did not return the survey.
Discussion
Results suggest improvement across behaviors for all participants. Visual analysis indicates an overall positive effect with some limitations (see Figure 3). Data from the eDBRC treatment integrity checklist shows interventionists implemented intervention with high levels of fidelity. Treatment integrity results indicate that interventionists can implement an eDBRC with high levels of fidelity that results in positive behavior changes for high school students with ASD and ID. Social validity data indicate that the eDBRC intervention is acceptable to teachers, participants, and parent(s)/guardian(s) participants.
ABAB reversal design.
There were three primary purposes of the current study: 1) to assess the effectiveness of eDBRCs in reducing challenging and/or disruptive classroom behavior of students with disabilities; 2) to examine the effectiveness of an eDBRC when implemented by preservice special education teachers; and 3) to examine the social validity of the eDBRC as rated by, teachers, participants, and parents. A positive effect between the eDBRC and problem behaviors across both participants was found. Both teachers consistently implemented eDBRC with acceptable levels of fidelity. These results support previous researcher findings on DBRCs in reducing challenging behavior and promoting positive behaviors (Lebel et al., 2013; Williams et al., 2012). Encouraging social validity ratings by interventionists and student participants as well as the guardian of one participant suggest that eDBRCs are an acceptable intervention for all parties involved. The eDBRC was designed to support new teachers in providing targeted behavior supports to students with challenging and/or disruptive behaviors. Moreover, the author built the eDBRC to examine the efficacy of using technology to aid in behavior management to reduce undesirable classroom behaviors. Outcomes of this study support novice special education teachers using eDBRCs effectively to manage classroom behavior.
Extending DBRC Research
The current study extends the research on DBRCs in several ways. As described above there are a limited number of studies that have looked at using technology in conjunction with DBRCs. Williams et al., 2012 utilized email paired with traditional paper DBRCs hypothesizing that that DBRC e-mailed to parents would lead to a decrease in observed disruptive classroom behavior and improved teacher ratings of student performance. In another study Yeo and colleagues (2018) incorporated a DBRC with an online platform to reduce off-task behaviors with students with ADHD. This study takes DBRC a step further and investigates a DBRC that can be created, adapted, and implemented by educators looking to support their learners at no cost.
Traditionally, DBRC studies have yielded positive effects (e.g., LeBel et al., 2013; Owens et al., 2012; Sanetti et al., 2016) for students with a range of disabilities. The social validity of DBRC have been reported as mostly favorable (e.g., Riden et al., 2020; Williams et al., 2012) which has been documented in this study as well. Perhaps the most promising findings of this study is the high treatment integrity of the eDBRC. Previous studies have reported the efficiency of implementing a DBRC study (see Vannest et al. 2010) which again was acknowledged in this study.
The historical record of DBRC implementation indicates that DBRC can be used for the duration of an entire school year (i.e., Vujnovic et al., 2013) or in a more truncated time span (Cowart, 1999), spanning age groups from preschool (i.e., LeBel et al., 2013) to high school (Riden et al., 2020), across several disabilities categories (see Riden et al., 2018; Vannest et al., 2010, and can be used to measure multiple behaviors at the same time (e.g., Burke et al., 2009). The current study extends these results using an electronic delivery method.
Implementation Fidelity
Implementing evidence-based practices is becoming both a goal and standard across medicine, psychology, and education (Horner et al., 2017). Fidelity of treatment, or integrity of implementation for interventions, means interventions are used according to predetermined criteria detailing activities, materials, and behaviors so the intervention has the desired effect on participants’ behaviors (Smith et al., 2007). Fidelity means the intervention is used true to the way it was originally designed (King-Sears et al., 2018). Stahmer et al. (2015) state students are more likely to benefit from interventions used as intended, although researchers find many evidence-based practices are not implemented as originally planned.
As such, the authors examined the treatment integrity of eDBRCs as implemented by the interventionists. The teachers had minimal experience working with student with disabilities in a classroom setting and had zero experience as a full time special education teacher managing a classroom and the challenges that accompany it. Due to the lack of experience in the field, the author developed trainings for the participants that would support implementation. Implementing interventions as intended strengthens the effects of intervention (Fiske, 2008) and is supported in the research literature (e.g., Arkoosh et al., 2007; Vollmer et al., 1999) and may be one reason for the effectiveness of the independent variable in this study.
Social Validity of eDBRC
High levels of social validity from the teachers, parent/guardian, and participants lend support to a highly socially valid intervention that has been shown to be effective in reducing challenging classroom behaviors. The relative simplicity in using the Google platform to implement an eDBRC and the high ratings on social validity surveys adds to the research literature in using technology to create an effective eBMP. Yet, it is important to note that the author created and prepared all intervention materials for the interventionists. Therefore, the interventionists only had to log in to their Google Drive and open the shared folder containing all materials, separated into week, to run the intervention. This may have led to the high social validity ratings from the interventionists.
Limitations
There are at least six limitations to this study that must be considered and may inform future researchers use of eDBRCs to reduce disruptive and/or challenging classroom behaviors. First, variability in the data was observed toward the end of intervention, particularly during fading phases. Due to the reduction in inappropriate participant behaviors, our data envelope used to assess stability was thin leading to a non-stable result using visual analytic techniques. A second limitation was the lack of parental involvement in the eDBRC intervention. Sarah was moved to a group home for one week during intervention, during this time no eDBRC were completed by the guardian. Additionally, eDBRCs were completed sporadically during intervention by the parent of the second participant. Researchers must consider developing contingencies for the parent(s)/guardian(s) to increase their involvement in implementing eDBRCs. Next, broadly speaking, frequency is best suited for behaviors with a clear beginning and end (e.g., frequency of words or frequency of kicks). The behaviors in the present investigation (talking out, off task, and non-compliance) do not have clear beginnings and endings. However, collecting frequency data during classroom instruction as opposed to duration or latency is less cumbersome for novice and experiences teachers alike. Next, two phase changes were made without establishing sufficient stability within phases. It is plausible that both participants’ performance could have happened without the phase change. Finally, there were instances of a non-therapeutic trend in the fading phase when a therapeutic trend was anticipated. It is theorized that this was due to the systematic removal of supports during fading phases. However, the frequency of inappropriate behaviors never exceeded prior intervention phases. Future researchers should investigate strategies for fading eDBRC to ensure behavior maintains at low levels as supports are removed.
DBRCs (in this instance eDBRC) are often used to reduce challenging and disruptive behavior or increase desired behaviors. A variable that should be investigated further is barriers to returning information to school via relevant stakeholders (e.g., parents, guardians). The guardians were instructed to review, complete, and return the summary sheet each day. The return rate was 50% for Sarah (8/16 opportunities) and 6.25% for Mike (2/16 opportunities). It is important to note that Sarah was placed in a group home for 5 days during intervention phases, which may have influenced, the rate of return. Although the return rates are not ideal and home-school communication is important, eDBRC was still successful for reducing Mike’s challenging classroom behavior.
There are several implications from this study that may inform future research and practice. There are several how to’s for DBRCs in traditional pen and paper format. During this study, the eDBRC was developed using a host of Google tools and was completely housed using the same platform. Using Google allowed for free access to technology to create an individualized eDBRC that has been empirically shown to reduce challenging and disruptive classroom behaviors. This is significant as the deluge of companies developing and marketing technologies to schools that focus on class wide strategies and to date have not been investigated at the individual student level. Therefore, introducing novice teachers to behavior modification strategies, in this case eDBRCs, can have significant impacts on student behavior as well as reducing stressors impacting teachers on a daily basis.
One approach for classroom management has been to utilize technology to monitor student behavior. These types of technologies have the potential to benefit teachers, students, and families in several ways. If technology is used effectively, they may have the ability to reduce problematic student behavior and, at the same time, increase desired behavior, reduce teacher stress and burnout, and facilitate positive home-school communication. However, there is still scant evidence in the literature that show evidence-based practices such as daily behavior report cards result in the same positive effects when implementing using various forms of technology. The field of behavior management and teacher preparation can benefit from research surrounding high quality training, technology to manage and monitor student behavior, and developing contingencies that evoke desirable classroom behaviors.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
