A Technology-Based Self-Monitoring Intervention for Secondary Students with High-Incidence Disabilities

Abstract

Secondary students with high incidence disabilities who also display disruptive behaviors struggle to be successful in general education settings. As a result, general education teachers are looking for ways to utilize technology to provide them with opportunities to implement evidence-based interventions in their classrooms. In this study, teachers used MoBeGo, an iPad application, in a single-case withdrawal design (ABAB), to implement self-monitoring in high school general education classrooms with four students who received special education services for a high incidence disability. The results of this study indicate that teachers could implement MoBeGo with fidelity to improve students’ academic engagement and appropriate behavior. Additionally, both the teachers and students rated MoBeGo as a socially valid intervention. Implications for practice and future research are discussed.

Keywords

secondary age/grade level inclusive education educational perspecitves high incidence disabilities exceptionality tablets/ipad technology perspectives

High school students labeled with high incidence disabilities (HIDs; e.g., specific learning disability [SLD], emotional and behavioral disorders [EBD], attention-deficit/hyperactivity disorder [ADHD]) who display disruptive behavior are at an increased risk to experience decreased behavioral and academic outcomes (Griggs et al., 2016; Sutherland et al., 2008). For example, students labeled with ADHD are likely to display hyperactivity or impulsiveness in the classroom that increases their risk of experiencing deficits in reading, math, or other academic areas (Arnold et al., 2020). Researchers have also found that students labeled with EBD experience some of the poorest academic outcomes and greatest behavioral challenges among their peers with and without disabilities (Maggin et al., 2016). As a result, these students are more likely to encounter behavioral and academic challenges that lead to poor post-secondary outcomes (e.g., lower-income, contact with the correctional system; Vaughn et al., 2011). To improve outcomes for students with HIDs, educators are increasingly expected to collect and analyze individualized student data to implement proactive behavior management interventions (Simonsen et al., 2011). However, both veteran and novice teachers report that they often feel inadequately prepared to provide evidence-based behavioral interventions in their classroom, and cite high-frequency, low-intensity disruptive behavior (e.g., speaking out of turn, not following directions, getting out of seat without permission), such as those typically displayed by students with HIDs, as among the biggest challenges they face (Busacca et al., 2015; Wehby & Kern, 2014). One way to address these challenges, is to provide teachers with evidence-based interventions that utilize technology to increase ease of implementation and intervention fidelity (Bruhn et al., 2017).

Electronic Self-Monitoring Interventions

Self-monitoring is an evidence-based intervention (Maggin et al., 2016) that has demonstrated the ability to reduce students’ problem behaviors, increase appropriate behavior, and increase academic engagement (Busacca et al., 2015). Typically, self-monitoring involves a student collecting their own data on a task (e.g., academic engagement) instead of having the teacher collect the data. Self-monitoring has its roots in cognitive behavioral therapy (Meichenbaum, 1977) and in self-determination research developed in the late 1960s when researchers realized that participants collecting their own data improved their results beyond the expectation of the original intervention (Bruhn et al., 2015; Carter et al., 2011; Hallahan et al., 1979). When using self-monitoring interventions, students reflect on how their behavior affected their results, which in turn allows the students to reflect on their behavior and take control of their actions. Traditionally, self-monitoring interventions have been implemented with paper and pencil and involved defining a target behavior as well as having a student collect real-time data on their behavior in class. This usually involves having a student tally or mark each time a behavior occurred, or rating their behavior at regular intervals. Additional components have also been used to target individual student needs, such as goal setting, graphing, or check-ins to discuss student outcomes (Riden et al., 2020). However, including additional components in a paper and pencil format is time consuming, which may make it less likely for educators to use self-monitoring in their classrooms.

Due to the time constraints associated with collecting individualized data, and then graphing and analyzing those data, researchers have utilized technology to develop applications that can digitally include multiple self-monitoring components (e.g., goal setting, graphing). For example, Wills and Mason (2014) conducted a pilot study of I-Connect, a self-monitoring application, and found that the student outcomes of on-task and disruptive behavior improved using the application and received generally positive acceptability feedback from the students. Since then, the I-Connect self-monitoring application has been used by educators to improve outcomes for K–12 students. Similarly, Bruhn and colleagues (2015), have developed and used SCORE IT, an electronic self-monitoring intervention, in K–12 schools to improve student-specific classroom-based behavioral outcomes for a diverse student population. The SCORE IT application includes features that automatically provide bar graphs for the self-monitoring data (e.g., on-task behavior) and provides a goal line (Bruhn et al., 2015). Recently, Bruhn and colleagues (2019) have developed an updated self-monitoring application called MoBeGo, which builds upon SCORE IT with automated data-based recommendations for goal-setting, as well as enhanced graphing, and monitoring functions.

Study Purpose

Self-monitoring interventions have been utilized by teachers to improve students classroom behavior; however, researchers have only recently begun to examine how electronic self-monitoring applications can be utilized in high school classrooms for students with HIDs. To further examine how these applications can be used to improve outcomes for high school students with HIDs, additional research is needed. Therefore, the purpose of this study was to assess the efficacy of the MoBeGo self-monitoring intervention to improve academic engagement and appropriate behaviors for high school students with HIDs who also displayed elevated levels of externalizing behaviors in classroom settings. The following research questions guided this study:

Is there a functional relation between the MoBeGo self-monitoring application and academic engagement and appropriate behavior for high school students with HIDs?

To what extent can high school teachers and students with HIDs implement the MoBeGo self-monitoring application with fidelity?

To what extent do high school teachers and students with HIDs who use the MoBeGo self-monitoring application find the intervention to be socially valid?

Method

Participants and Setting

Teachers

To be eligible for this study, the high school teachers had to teach students with HIDs who also displayed problem behaviors in a general education classroom. Recruitment emails were sent to teachers who had publically available (e.g., school website) contact information. Two general education replied to the email and participated in this study. Ms. Lynn was a White female with a master’s degree in secondary education and 12 years’ experience teaching high school English. Ms. Wilkinson was a White female with a master’s degree in secondary education with 9 years’ experience teaching middle and high school math.

Students

The two teachers were asked to nominate students who met the inclusion criteria: 1) had an individualized education program (IEP) for a HID, and 2) displayed elevated levels of disruptive behavior in the classroom. For the purpose of this study, a HID included specific learning disability, emotional disturbance, EBD, or ADHD under the category of other health impairment (Gage et al., 2012; Trainor et al., 2016). The teachers spoke with the parent/guardians of the students nominated and obtained a signed consent form before the study began. A multistep process was used to confirm that the students displayed elevated levels of disruptive behaviors in the classroom. First, the teachers completed the Strength and Difficulties Questionnaire (SDQ; Goodman, 1997), and each student had to receive a score of high or very high to be eligible. A score of high or very high on the SDQ indicates that a student may have, or is likely to have, significant emotional, behavioral, or social problems Bruhn et al., 2015; Goodman, 1997). Additionally, two pre-baseline observations were conducted to assess the students’ percent of time academically engaged and displaying appropriate behaviors. These observations occurred in the classroom in which the study was conducted, for the 50-min duration of the course, using the same data collection procedures and forms as during the study. Student participants were excluded if they (a) did not have an IEP for a high incidence disability, (b) did not receive a high or very high score on the SDQ, or (c) did not display elevated levels (i.e., more than 50% of the time) of disruptive behavior during the pre-baseline direct observation. A total of four students met the inclusion criteria, agreed to participate, and obtained parent consent.

Korey

Korey was a 17-year-old White male with freshman-level credits. His IEP was for ADHD which he received 6 months prior to this study. He stated that he was a class-clown, frequently talked to his peers during academic instruction, and had poor attendance. Despite the increased services provided as a result of his IEP, he accrued more than 103 absences in the school year this study occurred. Korey scored in the very high range on the SDQ and during the pre-study screening observations was academically engaged 24% and 21% of the class period and displayed appropriate behavior 32% and 36% of the class periods.

Deandre

Deandre was a 15-year-old Black freshmen male who began receiving IEP services for ED earlier in the school year in which this study occurred. Deandre scored in the very high range on the SDQ. During the two pre-study screening observations Deandre was academically engaged 0% and 13% of the class period, and he displayed appropriate behavior 0% and 20% of the class period. Deandre was expelled from school during the second intervention phase.

Ezekiel

Ezekiel was a 16-year-old Black male who had earned freshman credits and received IEP services for ADHD and stated that he had a history of behavioral problems in school, but he got his work done. Ezekiel scored in the very high range on the SDQ, and during the two pre-study screening observations Ezekiel was academically engaged 7% and 10% of the class period, and he displayed appropriate behavior 6% and 18% of the class period.

Noel

Noel was a 15-year-old Black male freshman who received special education services for SLD. Noel scored in the high range on the SDQ and during the pre-study screening observations Noel was academically engaged 24% and 21% of the class period and displayed appropriate behavior 32% and 36% of the class period.

Setting

This study was conducted in two different general education classrooms in a public high school in a mid-size city in the Midwest that serves ninth to 12th grade students with and without disabilities. Both classrooms had one general education teacher with no additional supports. The high school had a total enrollment of roughly 2,000 students, of whom 60% were from low-income families, nearly 20% were receiving special education services, and nearly 70% of students received free or reduce-priced lunches. Ms. Lynn’s ninth and 10th grade English class had 22 students, six of whom had IEPs. The student’s in Ms. Lynn’s class participated in this study at the same time. The other classroom, Ms. Wilkinson’s 9th grade math class had 33 students, five of whom had IEPs, only one of which participated in this study. No other students in either classroom used an iPad to engage in learning during this study and no changes to the setting occurred during this study.

Dependent Variable

Data were collected on two dependent variables: (a) academic engagement, defined as any time the student was working on the assigned task, actively attending to the teacher (i.e., looking at the teacher or material that was being presented), or was appropriately engaged in a teacher-directed conversation. Examples of academic engagement included independently working on the assigned task, independently reading the assigned material, participating in an academic discussion, task-appropriate talking, actively participating in group work, or hand-raising; and non-examples included when a student was not working on the assigned task, a student was not engaging in teacher-directed conversations, the student was working on materials that are not for the current class, or the student was using a cellphone (Bruhn Vogelgesang, Fernando & Lugo, 2016; Chafouleas, 2011; Vogelgesang et al., 2016); and (b) appropriate behavior, defined as verbal or nonverbal exchanges between the student and a teacher or peer that were compliant with the teacher’s requests that did not disturb the learning environment. Examples of appropriate behavior included social interactions that were academically focused, keeping hands to oneself, using appropriate voice levels (i.e., not yelling), remaining silent while the teacher or peers were speaking, staying seated, and properly using classroom equipment (Lewis et al., 1998); non-examples included talking with a peer outside one’s assigned group, disturbing the learning of others (e.g., making noise by tapping or hitting desk, talking out of turn, touching another student), improperly using classroom equipment, using a cellphone, and arguing with teacher or peers (Briesch & Chafouleas, 2009; Chafouleas, 2011).

Independent Variable

The MoBeGo self-monitoring intervention was the independent variable in this study (Bruhn et al., 2019). MoBeGo is a non-commercially available iPad application in which the teachers and students share an iPad and they rate pre-determined or pre-selected positive behaviors (e.g., on-task, respectful) behavior in regular intervals (e.g., every 5 min), and was used for the duration of the 50-min class periods. MoBeGo includes a timer and visual cue to keep track of each interval. At the end of each interval, the teacher and student rated each behavior on a 0–4 scale (0 = never, 1 = rarely, 2 = sometimes, 3 = often, 4 = always). The teachers recorded behavior first, and then the students. After the student entered his scores, the MoBeGo software displayed both scores so that the teacher and student could see each other’s rating. Separate line graphs of student and teacher ratings updated in real time with each rating entry. After the teacher entered ratings for five sessions (i.e. class periods), MoBeGo created a student goal 10% above the 5-day average. The teacher and student also met at the end of each class to briefly review the data and discussed if the student met their goal. During each of the phases described below, the first author was in charge of collecting, monitoring, and distributing the iPads to prevent implementation outside of the study. MoBeGo was being developed as part of an Institute for Educational Science Development and Innovation grant. As such, the version used in this study was a developer version of an early prototype that had different functions than the final user version tested in a later randomized control trial. Therefore, the researcher made a modification to MoBeGo by using 3 days, instead of the 5 days in the developer version, to make decisions related to changing phases.

Data Collection

Momentary time sampling was used to collect data on the dependent variables, which consisted of observing each student every 15 s and recording if they were or were not displaying academic engagement and appropriate behavior. Each observation session lasted the length of the 50-min class period. After each observation, the total number of occurrences the student displayed each dependent variable was added, and then its sum divided each by the total number of intervals observed and multiplied by 100. This resulted in a percentage of intervals that each dependent variable occurred (Kratochwill et al., 2010). The primary data collector for this study was the first author of this paper. A second data collector was trained to collect interobserver agreement (IOA) by the first author using two steps. First, they met to discuss the definitions of the dependent variables, discuss examples and non-examples, and review the data collection forms. Second, the two observers practiced by observing students without an IEP in one of the classrooms in which the study occurred, by using the same data collection forms and momentary time sampling procedures. A total of three practice observation sessions occurred, with a resulting Cohen’s κ agreement over the observes pre-established goal of .8 (strong agreement) for each of the practice observations (Cook et al., 2014; McHugh, 2012). Inter-observer agreement was conducted across all phases of the study and for each participant. Before each observational session, both observers read the definitions and examples of the behaviors together and discussed the participants and any challenges associated with observing each participant. Each observer then started their own phone-based time at the exact same time to indicate when data should be collected.

MoBeGo Training

The teachers and students were provided with training to help them gain a better understanding of self-monitoring interventions, how to use the MoBeGo application, and how to apply the definitions and examples of academic engagement and appropriate behavior to their classroom. The teachers and students were trained during three lunch periods. First, the definitions, examples, and non-examples of academic engagement and appropriate behavior were presented and then discussed. Next, the teacher and student provided additional examples that they thought applied to their classroom. Second, the SDQ results and baseline data were reviewed, followed by instructions on how to use the MoBeGo application in class. During this session, the teachers and students defined the student-specific behaviors they would self-monitor for each student participant, and the interval they would collect data (e.g., every 5 min). Third, the teacher and student practiced using MoBeGo by using the app and discussing how they would rate behaviors, until they reached at least 80% agreement.

Research Design

An ABAB withdrawal design was used with five conditions: baseline, MoBeGo, withdrawal, MoBeGo, and maintenance. In the baseline and withdrawal conditions, the teachers used their existing classroom management strategies, and MoBeGo was used during the intervention conditions. A maintenance phase, in which the teachers and students continued to implement MoBeGo in the same format, occurred 3 weeks after the final intervention condition. The teachers and students continued to use MoBeG Each condition length was determined by data variability and continued until the data were stable (Baer et al., 1968). A stability envelope was used to determine stability and defined as three consecutive data points within a plus or minus 15% envelope of the mean score (Dooley, 2018; Ledford & Gast, 2018), with students eligible to transition to the next condition if data were within the stability envelope.

Social validity

This study assessed social validity at the conclusion of the final intervention phase to measure the usability and efficacy of MoBeGo. Teachers completed both the User Profile Rating—Intervention Revised (URP-IR; Chafouleas et al., 2011) and the Intervention Rating Profile-15 (IRP-15; Witt & Elliott, 1985). The students completed the Children’s Usage Profile Rating (CURP; Briesch & Chafouleas, 2009).

Treatment fidelity

A fidelity monitoring Likert rating scale, modified from Vogelgesang and colleagues (2016), was completed after each observation by the primary observer. A 1- to 3-point scale, with one equaling less than half the time, two most of the time, and three almost or almost always, was used to rate (a) MoBeGo was used every 5 min by the teacher, (b) MoBeGo was used every 5 min by the student, and (c) if all components of the MoBeGo were properly used (for a total of 9 points). A 1-point rating (one was yes and no was zero) was used to determine if the student and teacher met after class, resulting in a total of 10 points per session. The total rating score was then multiplied by 100 to determine the percent of implementation fidelity.

Visual analysis and effect sizes

Visual analyses were conducted for condition changes and at the conclusion of the study to assess if there was a functional relation. Analyses for condition changes included: (a) variability, (b) level, (c) trend. Analysis for functional relation included (a) overlap and (b) immediacy of effect. The relative level change between adjacent conditions was calculated by a) identifying the median value of the second half of a phase and the first half of the next adjacent conditions, b) subtracting the smallest value from the largest value, and c) determining if there was improvement or deterioration (Gast & Ledford, 2014). Three effect sizes were calculated to provide detailed examination of the magnitude of the intervention. Between condition PND was calculated by a) determining the range of data point values from the first condition, b) counting the number of data points in the second condition, c) counting the data points in the second condition that are outside the range of the first condition, and d) dividing the number of data points that fall outside of the range of the first condition by the total number of data points in the second condition and multiplying by 100 (Scruggs & Mastropieri, 1998). To further determine the magnitude of the intervention, both Tau-U (Parker et al., 2011) and Log Response Ratio (Pustejovsky, 2015) were calculated using an online calculator (Pustejovsky & Swan, 2018) to determine effect sizes.

Results

The purpose of this study was to determine (a) if there was a functional relation between the MoBeGo self-monitoring application and students’ academic engagement and appropriate behavior, (b) if students and teachers could implement the intervention with fidelity, and (c) if the teachers and students viewed the intervention as socially valid. Per a visual analysis of each student’s data, there was a functional relation between the electronic MoBeGo application and academic engagement and appropriate behavior for the three participants who completed the study. The fourth student was suspended before a sufficient number of demonstrations of MoBeGo could be observed to determine a functional relation (Ledford & Gast, 2018). The effect size analyses (see Table 1) indicates there was no PND between conditions for any participant, Tau-U indicated a strong effect size for all dependent variables, and LRR indicated the MoBeGo intervention had a positive effect on dependent variables across all participants.

Table 1.

Effect Sizes.

Outcome	Participant	PND	Tau-u	LRR
Academic Engagement	Korey	100%	.78	1.83
	Ezekiel	100%	.99	1.17
	Deandre	100%	.99	1.21
	Noel	100%	1.19	1.31
Appropriate Behavior	Korey	100%	1.0	1.80
	Ezekiel	100%	1.18	1.21
	Deandre	100%	1.18	.87
	Noel	100%	1.08	1.12

Note. PND = Percentage of non-overlapping data, LRR = Log response ratio.

Research Question One

Korey

Korey displayed an abrupt relative level change each time that he transitioned conditions, with a mean between condition changes of 55% (range = 43.5%–64%) for academic engagement and 54% (range = 50.2%–57.3%) for appropriate behavior with 0% overall (see Figure 1). It was determined that there was a functional relation between the MoBeGo interventions and academic engagement and appropriate behavior for Korey. He missed a total of six sessions due to absences. Additionally, he moved and changed schools after the end of the reintroduction condition; therefore, a maintenance condition did not occur.

Figure 1.

Korey’s percent of intervals displaying academic engagement and appropriate behavior. A = baseline phase, B = intervention phase.

Ezekiel

Ezekiel displayed an immediate relative change in between each condition, with a mean between condition change of 58% (range = 56.3%–59.1%) for academic engagement and 58% (range = 56.9%–59.1%) for appropriate behavior with 0% overlap (see Figure 2). It was determined that there was a functional relation between the MoBeGo interventions and academic engagement and appropriate behavior for Ezekiel. He was absent during two sessions and was transferred to an alternative school immediately upon completion of his second MoBeGo condition; therefore, a maintenance condition did not occur.

Figure 2.

Ezekiel’s percent of intervals displaying academic engagement and appropriate behavior. A = baseline phase, B = intervention phase.

Noel

Noel displayed a relative change in between each condition, with a mean between condition change of 55% (range = 38.2%–72.3%) for academic engagement and 42% (range = 34.7%–48.4%) for appropriate behavior with 0% overlap (see Figure 3). It was determined that there was a functional relation between the MoBeGo interventions and academic engagement and appropriate behavior for Noel. Noel did not miss any sessions and was still enrolled in school for the maintenance condition.

Figure 3.

Noel’s percent of intervals displaying academic engagement and appropriate behavior. A = baseline phase, B = intervention phase, M = maintenance Phase.

Deandre

Deandre displayed an abrupt and immediate relative change between conditions, with a mean between condition change of 54% (range = 51.7%–54.8%) for academic engagement and 54% (range = 48.3%–59%) for appropriate behavior with 0% overlap (see Figure 4). A functional relation could not be determined for Deandre due to missing data in the final intervention condition due to suspensions and then expulsion.

Figure 4.

Deandre’s percent of intervals displaying academic engagement and appropriate behavior. A = baseline phase, B = intervention phase.

Research Question Two

Implementation fidelity data were collected on 100% of all observations. Ms. Lynn and her students implemented the intervention with 97.5% fidelity (range = 90%–100%). Ms. Wilkinson and her student (Noel) implemented the intervention with 96.1% fidelity (range = 90%–100%). All of the occasions in which the teachers and students did not have 100% fidelity were due to teachers being unable to enter data at 5-min intervals.

Research Question Three

Per the CURP, Korey and Ezekiel rated MoBeGo with a high acceptability (3.9 and .85 respectively) while Noel rated MoBeGo 2.95, indicating moderate acceptability across all three subscales (personal desirability, feasibility, and understanding). Ms. Lynn rated the pre-intervention IRP 83 (m = 5.53) and post-intervention score 76 (m = 5.06) for Korey, she rated the pre-intervention score 78 (m = 5.2) and post-intervention score 77 (m = 5.1) for Deandre, and she rated the pre-intervention score 81 (m = 5.40) and the post-intervention 77 (m – 5.13) for Elijah. Ms. Wilkinson rated the pre- and post-intervention IRP-15 as 90 (m = 6). Additionally, for the URP-I, which is rated on a one to six Likert scale. Ms. Lynn gave an overall MoBeGo rating average of 4.65 and Ms. Wilkinson gave an overall rating average of 6.0.

Interobserver Agreement

Interobserver agreement was calculated using Cohen’s κ, which accounts for agreement between two observers that occurs due to chance (Cohen, 1968). κ’s ranged from a low of .81 to a high of .96, with a mean of .88 and a standard deviation of .05 across all IOA sessions. κ is scored from 0 to 1, with .4 or below indicating poor agreement, .4 to .6 indicating fair agreement, .6 to .75 indicating good agreement, and .75 or higher indicating excellent agreement (Watkins & Pacheco, 2000).

Discussion

Researchers have found that technology help teachers increase their ability to implement behavioral management interventions in their classroom (Bruhn et al., 2016). However, as technology continues to progress and educators develop new applications to enhance the feasibility of implementing evidence-based interventions, there is a need to assess the ability of these new applications in classroom settings. This study expanded upon previous research on self-monitoring by (a) using the electronic MoBeGo self-monitoring application, which incorporated several self-monitoring components (i.e., goal setting, graphing); and (b) focusing on high school students with high incidence disabilities who also displayed elevated levels of externalizing behavior. The findings from this study indicate that MoBeGo can be implemented with fidelity to improve student engagement and appropriate behavior and received positive social validity feedback from teachers and students. However, there are numerous confounding variables that may have impacted the outcome and fidelity. Additionally, teacher and student feedback on implementation may provide opportunities to further improve the use of technology to deliver self-monitoring interventions in classrooms.

Research Question One

The first research question asked whether there was a functional relation between the electronic MoBeGo self-monitoring application and academic engagement and appropriate behavior for high school students with HIDs. Each of the student participants in Ms. Lynn’s class improved his academic engagement and appropriate behavior while using MoBeGo. When Ms. Lynn was not rushed and had the time to engage the student participants, she would quickly engage in a brief conversation with them to explain why she had given them a particular rating. For example, if she rated Korey a two on academic engagement, she would remind him to put his phone away, so he was not distracted and could receive a higher rating at the next opportunity. Additionally, the three students in Ms. Lynn’s class reported that they enjoyed trying to select the same rating as Ms. Lynn and were excited to see if they reached their goal for the day. This aligns with previous reviews of using goal-setting as part of multi-component interventions to improve classroom behavior (Bruhn et al., 2016). Therefore, having teachers enter data and goal setting may have improved outcomes for the students in Ms. Lynn’s class beyond the impact of self-monitoring alone.

Ms. Wilkinson’s freshman math class, which had 33 students on the roster, which often changed throughout study, providing numerous challenges to Ms. Wilkinson’s ability to deliver effective behavioral management interventions to her students with high incidence behavioral disabilities. The only behavioral management strategy that was observed in Ms. Wilkinson’s class involved her writing students’ names on the board when she caught them with a phone, which means they had to serve a lunch detention. This punitive behavioral management strategy appeared to be ineffective, which is in line with recent research that indicates positive behavioral interventions are more effective than punitive strategies (e.g., detention, office referrals; Bradshaw et al., 2015; Houchens et al., 2017). Ms. Wilkinson stated that she felt too busy to use proactive behavior management strategies during class, but she stated that she could use MoBeGo because the application was user friendly and did not require much additional work by her. Additionally, when Ms. Wilkinson started using the intervention, she checked Noel’s academic progress each time he entered his ratings and discovered that Noel typically stopped working if he did not know how to work through a question. Ms. Wilkson spoke with Noel and instructed him to start seeking assistance (e.g., raising his hand) when he needed help instead of talking to his peers. Although Ms. Wilkinson was not trained to teach a functional replacement behavior as part of this study. However, she naturally assessed her perceived reason for the behavior and provided a solution, which appeared to help Noel stay engaged, and the additional academic support allowed him to work for longer periods of time. Providing immediate academic and behavioral feedback is a self-monitoring component that Bruhn and colleagues (2015) identified as one aspect that may improve outcomes for students with behavior problems due to the teacher having more frequent and immediate contact with a struggling student. As a result of using MoBeGo, Ms. Wilkinson was able to identify Noel’s academic needs before disruptive behaviors occurred, and therefore Noel improved his rates of academic engagement and appropriate behavior.

Research Question Two

The second research question asked if teachers and students could implement the MoBeGo intervention with fidelity. Ms. Lynn implemented MoBeGo with complete fidelity 85% of the time. Both of the sessions during which she did not adhere to the 5-min intervals were due to her providing whole-class direct instruction and being unable to walk over to the students to enter ratings. Ms. Lynn began each of her classes with whole-class instruction for a quick review, and then the students typically transitioned to using technology for small group or individual work. However, during two of the 20 class periods, she provided whole-class direct instruction for an extended period of time (e.g., 30 min). As a result, she was unable to interrupt her instruction and had to lengthen the duration of the self-monitoring to longer than 5-min intervals. Ms. Lynn noted that it would have been easier for her to implement the intervention with fidelity had she only used MoBeGo when she was engaged in instructional activities that involved her walking around the room as opposed to whole group direct instruction. In the two previous studies using electronic self-monitoring in high school settings, Dooley (2018) implemented the intervention in 15–20 min segments, and Vogelgesang and colleagues (2016) implemented for the length of the class period, and both had high implementation fidelity, which indicates that teachers can implement electronic self-monitoring for varying durations (i.e., 20 min or an entire class period) with fidelity.

Although Ms. Wilkinson had provided perfect feedback on her social validity measures related to workload and ease of use, she was unable to enter ratings every 5 min during 30% of the sessions because she was occupied with other classroom activities. Her class regularly had over 30 students in attendance, which meant that during individual or group work, she was often busy providing one-on-one support and therefore, could not enter a rating. At the conclusion of the study, Ms. Wilkinson noted that she would like to continue using the intervention with the target student and another student in the same class, but she would use longer intervals (e.g., 10 min). Additionally, Ms. Wilkinson also stated that the intervention was difficult to implement during certain activities, such as when she was teaching a new math skill to the entire class. She stated that it was easier for her to implement with fidelity when she was already walking around the class and did not have to interrupt direct instruction to enter ratings.

Research Question Three

The third research question asked if high school teachers and students with HIDs rated MoBeGo as a socially valid intervention. Examining social validity is an important step to closing the research to practice gap, because it examines if MoBeGo was perceived as effective and if they would continue to use the intervention in their classroom beyond the duration of the study (Ledford & Gast, 2018). Previous research on self-monitoring interventions has indicated high social validity according to both teachers and students; however, there is much less research on how using digital application impact social validity. The social validity survey results from this study indicated that both teachers regarded MoBeGo as an acceptable invention and that they would be willing to use the intervention in the future. An analysis of Ms. Lynn’s subscales indicated that she had two concerns related to the intervention. She reported that in the future, she would need an iPad to implement the intervention, which her school did not provide. Ms. Lynn also indicated that the intervention was not consistent with her previous classroom management strategies and that she would require additional training on how to more effectively use the intervention to improve outcomes for other students, which is consistent with previous research indicating that teachers require adequate and ongoing training when incorporating new behavioral interventions in their classroom (Wehby & Kern, 2014). Ms. Wilkinson provided the highest scores possible on every question in both social validity scales. During conversations that occurred during the study, Ms. Wilkinson said that she thought MoBeGo was effective and worthwhile because it reminded her to check on Noel every 5 min, which gave her a chance to make sure that he was working and to answer any questions he had, which in turn kept him engaged. After the study, she also noted that she wanted to use the intervention with other students who were not part of the study. As with previous research on electronic self-monitoring in high school settings (e.g., Dooley, 2018; Vogelgesang et al., 2016), Ms. Lynn and Ms. Wilkinson indicated that they perceived MoBeGo to be socially valid and effective in high school settings. The social validity results from this study further contribute to the evidence base for teacher’s acceptability to use electronic-self monitoring in high school classrooms.

The students’ scores on the social validity measure also indicated that they thought MoBeGo was effective and that they would like to continue to use it in the future. Although all of the students gave the MoBeGo an acceptability rating of high or moderately high, it is worth noting that each of the students lowest scores were related to intrusiveness because the teacher was required to walk to the students and hand them the iPad every 5 min. This was also noted by the students when they indicated that they would like to turn off the audible noise that reminded the teacher each 5 min. Noel rated the intervention with the lowest social validity scores of any of the student participants, and analysis of his social validity scores indicated that Noel thought the intervention was intrusive because it focused too much attention on him. However, Noel and the other participants rated the interventions acceptability as highly or moderately high and indicated that they would like to continue using MoBeGo in the future, despite the extra attention.

Limitations

Several limitations should be considered when interpreting the results of this study. One limitation is that neither of the teachers were implementing behavioral management strategies in the baseline conditions. Second, due to the study occurring late in the school year, a minimum number of observations were collected to complete the study. Third, no fading conditions were utilized in this study, which, coupled with three students not being available for maintenance, reduced the ability to assess the efficacy of MoBeGo beyond the duration of the study. Fourth, all of the student participants in this study were male.

Similar to findings from previous research that teachers have struggled to deliver evidence-based interventions in classroom settings (Barrett et al., 2013), both of the teachers in this study reported that they did not implement any evidence-based behavioral interventions in their classroom in the past; therefore, it should be noted that due to the lack of an active intervention in the baseline condition, introducing any novel evidence-based intervention was likely to improve behavior over the business as usual condition (Shadish et al., 2002). The introduction of a novel intervention without an active control intervention might have resulted in an improvement if any new intervention (e.g., the Good Behavior Game, positive peer reporting) had been introduced (Shaddish et al., 2002). Additionally, providing class-wide positive behavioral interventions may reduce the need for individualized interventions, such as MoBeGo.

Implications for Research and Practice

Researchers have documented the effectiveness and social validity of traditional paper and pencil self-monitoring interventions in classrooms (Busacca et al., 2015), but as electronic self-monitoring applications become more common, there is a need for research on how electronic self-monitoring applications, such as MoBeGo, can best be utilized to improve student outcomes beyond traditional paper and pencil self-monitoring because MoBeGo automatically graphs data and sets goals. Additionally, is still a dearth of research on evidence-based behavioral interventions for students with EBD, especially high school students, which indicates a need for several areas of additional research. For example, high school students with EBD are likely to have coercive interactions with teachers (Sutherland et al., 2008), which occurred in this study when the student did not agree with the ratings that the teacher provided. Since teacher-student interactions occur in regular intervals (e.g., 5-min intervals) during self-monitoring interventions, additional research can investigate ways to provide teachers with training to mitigate coercive interactions and to avoid drawing unwanted attention to students using MoBeGo.

As noted, teachers report that implementing classroom-based behavioral interventions is one of the biggest challenges they face (Busacca et al., 2015; Wehby & Kern, 2014). This study also added to the growing literature base of teachers needed training on delivering universal positive behavioral interventions in their classroom. The findings from this study indicated that teachers found that MoBeGo was an effective and acceptable classroom-based behavioral intervention for high school students with high incidence disabilities who displayed high-frequency, low-intensity behaviors, and as the use of mobile devices (e.g., iPad, tablet) becomes more widespread in the classroom, teachers can use the devices as part of behavioral interventions. The students also stated that they enjoyed using the intervention and found it to be effective, but they did not always like the attention that it brought during class. Therefore, teachers should try to use the intervention in a format that is not disruptive to the class and should regularly check with the students to make sure that they are not uncomfortable using the intervention. This can be accomplished through thorough and ongoing professional development on implementing technology-based behavioral supports in the classroom so they are no more disruptive than traditional paper and pencil methods.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Skip Kumm

References

Arnold

L. E.

Hodgkins

Kahle

Madhoo

Kewley

(2020). Long-term outcomes of ADHD: Academic achievement and performance. Journal of Attention Disorders, 24(1), 73–85. https://doi.org/10.1177/1087054714566076

Baer

D. M.

Wolf

M. M.

Risley

T. R.

(1968). Some current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 1, 91–97. https://doi.org/10.1901/jaba.1968.1-91

Barrett

Eber

Weist

M. D.

(2013). Advancing education effectiveness: Interconnecting school mental health and school wide positive behavioral support. OSEP Center on Positive Behavioral Interventions and Supports. https://www.pbis.org/common/cms/files/Current%20Topics/Final-Monograph.pdf

Bradshaw

C. P.

Pas

E. T.

Debnam

K. J.

Johnson

S. L.

(2015). A focus on implementation of positive behavioral interventions and supports (PBIS) in high schools: Associations with bullying and other indicators of school disorder. School Psychology Review, 44, 480–498. https://doi.org/10.17105/spr-15-0105.1

Briesch

A. M.

Chafouleas

S. M.

(2009). Review and analysis of literature on self-management interventions to promote appropriate classroom behaviors (1988–2008). School Psychology Quarterly, 24, 106–118. https://doi.org/10.1037/a0016159

Bruhn

A. L.

Hirsch

S. E.

Kittleman

(2019, February). Technology trends across the tiers. Presentation at Association for Positive Behavior Support Conference.

Bruhn

A. L.

Vogelgesang

Fernando

Lugo

(2016). Using data to individualize a multicomponent, technology-based self-monitoring intervention. Journal of Special Education Technology, 31, 64–76. https://doi.org/10.1177/0162643416650024

Bruhn

A. L.

Vogelgesang

Schabilion

Waller

Fernando

(2015). “I don’t like being good!” Changing behavior with technology-based self-monitoring. Journal of Special Education Technology, 30(3), 133–144.

Bruhn

A. L.

Waller

Hasselbring

(2016). Tweets, texts, and tablets: The emergence of technology-based self-monitoring. Intervention in School and Clinic, 51, 157–162. https://doi.org/10.1177/1053451215585803

10.

Bruhn

A. L.

Woods-Groves

Fernando

Choi

Troughton

(2017). Evaluating technology-based self-monitoring as a tier 2 intervention across middle school settings. Behavioral Disorders, 42(3), 119–131. https://doi.org/10.1177/0198742917691534

11.

Busacca

M. L.

Anderson

Moore

D. W.

(2015). Self-management for primary school students demonstrating problem behavior in regular classrooms: Evidence review of single-case design research. Journal of Behavioral Education, 24, 373–401. https://doi.org/10.1007/s10864-015-9230-3

12.

Carter

E. W.

Lane

K. L.

Crnobori

Bruhn

A. L.

Oakes

W. P.

(2011). Self-determination interventions for students with and at risk for emotional and behavioral disorders: Mapping the knowledge base. Behavioral Disorders, 36, 100–116. https://doi.org/

13.

Chafouleas

S. M.

(2011). Direct behavior rating: A review of the issues and research in its development. Education & Treatment of Children, 34, 575–591. https://doi.org/10.1353/etc.2011.0034

14.

Chafouleas

S. M.

Briesch

A. M.

Neugebauer

S. R.

Riley-Tillman

T. C.

(2011). Usage rating profile—Intervention (revised). University of Connecticut.

15.

Cohen

(1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213–220. https://doi.org/10.1037/h0026256

16.

Cook

Buysse

Klingner

Landrum

McWilliam

Tankersley

Test

(2014). Council for Exceptional Children: Standards for evidence-based practices in special education. Exceptional Children, 80, 504–511. https://doi.org/10.1177/0741932514557271

17.

Dooley

(2018). Improving the social behavior of high school students with behavioral challenges through self-management technology [Doctoral dissertation] . https://opencommons.uconn.edu/dissertations/1657/

18.

Gage

N. A.

Lierheimer

K. S.

Goran

L. G.

(2012). Characteristics of students with high-incidence disabilities broadly defined. Journal of Disability Policy Studies, 23, 168–178. https://doi.org/10.1177/1044207311425385

19.

Gast

D. L.

Ledford

J. R.

(2014). Single-case research methodology: Applications in special education and behavioral sciences (2nd ed.). Routledge/Taylor & Francis Group.

20.

Goodman

(1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581–586. https://doi.org/10.1111/j.1469-7610.1997.tb01545.x

21.

Griggs

M. S.

Mikami

A. Y.

Rimm-Kaufman

S. E.

(2016). Classroom quality and student behavior trajectories in elementary school. Psychology in the Schools, 53, 690–704. https://doi.org/10.1002/pits.21941

22.

Hallahan

D. P.

Lloyd

Kosiewicz

M. M.

Kauffman

J. M.

Graves

A. W.

(1979). Self-monitoring of attention as a treatment for a learning disabled boy’s off-task behavior. Learning Disability Quarterly, 2(3), 24–32. https://doi.org/10.2307/1511021

23.

Houchens

G. W.

Zhang

Davis

Niu

Chon

K. H.

Miller

(2017). The impact of positive behavior interventions and supports on teachers’ perceptions of teaching conditions and student achievement. Journal of Positive Behavior Interventions, 19, 168–179. https://doi.org/10.1177/1098300717696938

24.

Kratochwill

T. R.

Hitchcock

Horner

R. H.

Levin

J. R.

Odom

S. L.

Rindskopf

D. M.

Shadish

W. R.

(2010). Single-case designs technical documentation. http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf.

25.

Ledford

J. R.

Gast

D. L.

(2018). Single-case research methodology: Applications in special education and behavioral sciences. Routledge.

26.

Lewis

T. J.

Sugai

Colvin

(1998). Reducing problem behavior through a school-wide system of effective behavioral support: Investigation of a school-wide social skills training program and contextual interventions. School Psychology Review, 27, 446–454.

27.

Maggin

D. M.

Wehby

J. H.

Farmer

T. W.

Brooks

D. S.

(2016). Intensive interventions for students with emotional and behavioral disorders: Issues, theory, and future directions. Journal of Emotional and Behavioral Disorders, 24(3), 127–137. https://doi-org.libdata.lib.ua.edu/10.1177/1063426616661498

28.

McHugh

M. L.

(2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282.

29.

Meichenbaum

(1977). Cognitive behaviour modification. Cognitive Behaviour Therapy, 6, 185–192. https://doi.org/10.1080/16506073.1977.9626708

30.

Parker

R. I.

Vannest

K. J.

Davis

J. L.

Sauber

S. B.

(2011). Combining nonoverlap and trend for single-case research: Tau-U. Behavior Therapy, 42, 284–299. https://doi.org/10.1016/j.beth.2010.08.006

31.

Pustejovsky

J. E.

(2015). Measurement-comparable effect sizes for single-case studies of free-operant behavior. Psychological Methods, 20, 342–359. https://doi.org/10.1037/met0000019

32.

Pustejovsky

J. E.

Swan

D. M.

(2018). Single-case effect size calculator (Version 0.5) web application. https://jepusto.shinyapps.io/SCD-effect-sizes/

33.

Riden

B. S.

Taylor

J. C.

Ruiz

Lee

D. L.

Scheeler

M. C.

(2020). Using a daily report card to reduce off-task behaviors for a student with autism spectrum disorder. Journal of Behavioral Education. https://doi-org.libdata.lib.ua.edu/10.1007/s10864-020-09382-6

34.

Scruggs

T. E.

Mastropieri

M. A.

(1998). Summarizing single-subject research: Issues and applications. Behavior Modification, 22, 221–242. https://doi.org/10.1177/001440290507100203

35.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.

36.

Simonsen

Myers

Briere

D. E.

III (2011). Comparing a behavioral check-in/check-out (CICO) intervention to standard practice in an urban middle school setting using an experimental group design. Journal of Positive Behavior Interventions, 13, 31–48. https://doi.org/10.1177/1098300709359026

37.

Sutherland

K. S.

Lewis-Palmer

Stichter

Morgan

P. L.

(2008). Examining the influence of teacher behavior and classroom context on the behavioral and academic outcomes for students with emotional or behavioral disorders. The Journal of Special Education, 41, 223–233. https://doi.org/10.1177/0022466907310372

38.

Trainor

A. A.

Morningstar

M. E.

Murray

(2016). Characteristics of transition planning and services for students with high-incidence disabilities. Learning Disability Quarterly, 39, 113–124. https://doi.org/10.1177/0731948715607348

39.

Vaughn

M. G.

Wexler

Beaver

K. M.

Perron

B. E.

Roberts

(2011). Psychiatric correlates of behavioral indicators of school disengagement in the United States. Psychiatric Quarterly, 82, 191–206. https://doi.org/10.1007/s11126-010-9160-0

40.

Vogelgesang

K. L.

Bruhn

A. L.

Coghill-Behrends

W. L.

Kern

A. M.

Troughton

L. C.

(2016). A single-subject study of a technology-based self-monitoring intervention. Journal of Behavioral Education, 25, 478–497. https://doi.org/10.1007/s10864-016-9253-4

41.

Watkins

M. W.

Pacheco

(2000). Interobserver agreement in behavioral research: Importance and calculation. Journal of Behavioral Education, 10(4), 205–212.

42.

Wehby

J. H.

Kern

(2014). Intensive behavior intervention: What is it, what is its evidence base, and why do we need to implement now? Teaching Exceptional Children, 46, 38–44. https://doi.org/10.1177/0040059914523956

43.

Wills

H. P.

Mason

B. A.

(2014). Implementation of a self-monitoring application to improve on-task behavior: A high-school pilot study. Journal of Behavioral Education, 23(4), 421–434. https://doi.org/10.1007/s10864-014-9204-x

44.

Witt

J. C.

Elliott

S. N.

(1985). Acceptability of classroom intervention strategies. In Kratochwill

T. R.

(Ed.), Advances in school psychology (Vol. 4, pp. 251–288). Erlbaum.