Abstract
Students who are not academically engaged spend less time mastering material, are less likely to be successful in school, and are more likely to be disruptive. The purpose of the current brief report was to investigate the effects of a technology-based self-monitoring intervention on elementary students’ academic engagement during independent work time. The intervention, CellF-Monitor, is an iPad application that allows individuals to self-rate their on-task behavior. In this multiple-baseline, single-case-design study, four nominated students used the CellF-Monitor during independent work time in their regular education classrooms. Systematic direct observations, self-ratings, and teachers’ Direct Behavior Ratings of academically engaged and on-task behaviors were collected to measure students’ behavioral changes from baseline to the intervention and reinforcement phases. Visual analyses illustrated positive effects of the CellF-Monitor on academic engagement and on-task behavior, and findings were augmented by effect size estimates.
Academic engagement (AE), operationalized as both active and passive on-task behaviors, has been recognized as an essential contributor to school achievement (Greenwood, Terry, Marquis, & Walker, 1994; Shapiro, 2004). Low AE has not only been shown to negatively influence a student’s ability to learn but also has the potential to negatively affect other students in the classroom, and the teacher. While a myriad of interventions for increasing AE have been studied (e.g., teacher-directed contingency interventions, opportunities-to-respond, or function-based interventions), instructor led supports have shown increased teacher stress and student dependence (Briesch & Chafouleas, 2009b). Therefore, student-managed interventions (sometimes referred to as self-management interventions) offer additional advantages by shifting the burden of implementation away from teachers and onto students. One common approach utilized within self-management interventions involves increasing self-monitoring (SM) skills, or the ability to reflect and rate one’s own behaviors.
SM interventions typically consist of a cue (e.g., audio tone) that directs the participant to measure, through self-reflection, and record, through self-rating, their behaviors. More than 40 years of research suggest SM interventions have been beneficial for a variety of outcomes across subject areas, ability levels, and differing ages (e.g., Briesch & Chafouleas, 2009b). In addition, including contingent reinforcement has been shown to amplify the effects of SM (e.g., Graham-Day, Gardner, & Hsin, 2010). This abundance of support, however, does not eliminate the need for future research. In particular, SM interventions are constantly evolving due to the integration of technology, creating a need for continuous research.
Several devices (e.g., the MotivAider, repeat timers) have been used in the SM literature to cue students to reflect and rate their behavior with positive results indicated for students identified with autism spectrum disorder and emotional and behavioral disorder (EBD; Blood, Johnson, Ridenour, Simmons, & Crouch, 2011; Legge, DeBar, & Alber-Morgan, 2010). However, technology allows for the combination of prompting and recording responses within the same device, streamlining the process. Studies have been conducted using a Palm Pilot with a student identified with EBD (Gulchak, 2008), android-based tablets with students identified with other health impairments (OHIs) and specific learning disabilities (SLDs; iConnect; Wills & Mason, 2014), and cell phones with students identified with OHIs and SLDs (CellF-Monitor; Bedesem, 2012), all of which increased on-task behavior. One emerging use of technology involves applications on tablet devices in school settings. Unfortunately, applications are more likely to be chosen for high ratings and low costs than the research support (Falloon, 2013). This fact, coupled with the fast turnover rate of new technology, has led to an even wider research-to-practice gap, making research on tablet-based interventions essential. One such application, CellF-Monitor, has been adapted as an iOS application (available on iPhones, iPods, or iPads) that prompts and records student responses to on-task behavior, and was utilized in the present investigation.
Current Study
Research has supported the use of technology-based SM interventions; results indicated increased on-task behavior, decreased disruptive behavior, and high ratings of social validity from both teachers and students when technology-based SM interventions were implemented on their own (i.e., without any external reinforcement; Bedesem, 2012; Gulchak, 2008; Wills & Mason, 2014). However, studies with technology-based SM have focused solely on students with disabilities despite SM being beneficial across ability levels (Briesch & Chafouleas, 2009b). This study aimed to answer the following questions for students without exceptionality using the iOS CellF-Monitor application:
Method
Participants and Setting
The present study was conducted in 2016 in an elementary charter school in an urban area within the mid-western United States. The student population of the school comprised 81% Black, 18% Hispanic, and less than 1% White students, 95% of whom qualified for free or reduced lunch. Participants, described using pseudonyms, included three regular education teachers and four students. All procedures were followed in accordance with a university-approved human subjects review board.
Student participants included two males (Noah and Sam) and two females (Samira and Gia) between the ages of 8 and 10 years who were identified through teacher nomination for having lower rates of on-task behaviors during independent work time. Each student was observed to have increased rates of off-task behaviors, which included taking a long time to get started on tasks, looking around the room, and fidgeting with other materials. Teachers’ initial Direct Behavior Rating–Single Item Scale (DBR-SIS) indicated that nominated students were academically engaged 30% (Gia and Sam) and 50% (Samira and Noah) of independent work time. Thus, initial goals of 60% AE for Gia and Sam, and 70% for Samira and Noah were set by the teacher as guided by the researcher during an identification interview. Three of the participants were African American, and one student (Gia) identified as Hispanic. Two students completed the intervention during their English Language Arts (ELA), while Gia completed the intervention during math, and Sam completed the intervention during independent writing time.
Measures
DBR-SIS
DBR-SIS includes three single-item scales that quantify duration of academically engaged, respectful, and disruptive behaviors during a target observation period. The ratings from the AE scale were used as the primary outcome for this study. DBR-SIS ratings have demonstrated moderate agreement with systematic direct observation (SDO) scores (Riley-Tillman, Chafouleas, Sassu, Chanese, & Glazer, 2008), sensitivity to detecting behavior change over short periods of time (Chafouleas, Sanetti, Kilgus, & Maggin, 2012), and reliable teacher ratings as compared with researchers who have higher training and fewer responsibilities during the observation period (Chafouleas et al., 2010).
Behavioral Observation of Students in Schools (BOSS
The BOSS is a type of SDO that determines the exact rate of engaged and off-task behaviors through momentary and partial-interval time sampling, and is sensitive to change (Shapiro, 2004; Volpe, DiPerna, Hintze, & Shapiro, 2005). For the purpose of the present study, active and passive AE were combined to align to the AE scale of the DBR-SIS. Interobserver agreement (IOA) was calculated across 37% of observations, distributed across conditions, and scheduled based on the research team availability. Point-by-point agreement was used to calculate IOA with a mean of 98% across all phases and participants with a range of 95% to 99% across all co-observed sessions.
Intervention
iPad minis were used to access the CellF-Monitor application, an iOS-based SM intervention. A prompting interval (i.e., 1 minute between reminders) and length of each session were selected for each student. During the intervention, the students were prompted with a buzzing sound while the screen showed the question “are you on task?” and students responded by pressing “yes” or “no.” Students were not exposed to graphs of their progress as the application did not yet have that feature at the time of data collection.
Design
An ABC concurrent multiple-baseline design across participants was used to assess the effectiveness of the CellF-Monitor intervention. Phases included baseline, training and intervention, and reinforcement, with the onset of the training phase staggered across participants. Decisions regarding the number of cases (i.e., at least three replications) and the length of each phase (i.e., a minimum of five data points) aligned to criteria for multiple-baseline designs set by What Works Clearinghouse, with the exception of Samira who only had three baseline points (Kratochwill et al., 2010).
Procedures
Recruitment and teacher training
Three of the six eligible third- through fifth-grade teachers expressed interest in participating, signed consent forms, and completed the student nomination interview. The teachers completed the 45-min long online training module for the DBR-SIS, which consisted of an overview of the measure, a model and frame-of-reference training, and multiple opportunities to practice and receive immediate corrective feedback (Chafouleas, Riley-Tillman, Jaffery, Miller, & Harrison, 2015). Subsequently, teachers rated the nominated student’s AE to ensure ratings below a seven. Once eligibility was confirmed, parental consent and student assent was granted, and teachers completed student and teacher demographic forms.
Baseline
Teachers began completing the DBR-SIS every day during the targeted independent work time, and the researchers completed at least two 20-min systematic direct observations using the BOSS. Baseline was continued until stability was established from the primary dependent variable (i.e., a minimum of three DBR-SIS AE scores), and clear effects were illustrated from the previous student’s introduction to the intervention.
Student training phase
Students transitioned into a training phase for the CellF-Monitoring intervention in a randomized treatment order, as determined by an online randomization tool. Students received a brief 10-min training from the primary researcher showing them how to start a session, and how to use the application. Guided access was placed on each iPad so students could only access the CellF-Monitor. Students began using the application and compared responses to their teacher’s ratings of AE at the end of each day’s designated independent work time. The training phase continued until three consecutive days of teacher and student ratings that fell within 20% of one another, based on criteria set by Chafouleas et al. (2015).
Intervention phase
The intervention phase began once students successfully demonstrated an understanding of how to accurately use the CellF-Monitor and lasted a minimum of two weeks. During this phase, students continued to use the CellF-Monitor, teachers continued to complete DBR-SIS after each targeted independent work time, and at least one systematic direct observation was completed per week.
Reinforcement phase
During the last two weeks of the intervention, all participating students began a reinforcement phase. Goals were set by teachers based on each student’s demonstration of on-task behavior during the intervention phase. An AE goal of 90%, determined by the DBR-SIS, was set for Gia, Noah, and Sam, while Samira’s goal was 80%. If the student met their goal, he or she was able to earn a reward of their choice each day (i.e., a prize from the class store, play a game on the iPad).
Data Analysis
The primary method of interpreting the effects of the CellF-Monitor was visual inspection of the percentage of time students were academically engaged according to the DBR-SIS for each of the students across phases (Kratochwill et al., 2010). In addition, nonoverlap of all pairs (NAP) and Kendall’s tau for nonoverlap between groups (Taunovlap) effect sizes (ESs) were used to quantify the effectiveness of this intervention to corroborate results from the visual analysis. Strengths of NAP include accuracy and comparability to visual analysis; however, it lacks precision as compared with some other metrics (Parker & Vannest, 2009). Taunovlap, on the other hand, accounts for tied data between phases by removing any negative changes from the positive changes between phases (Parker, Vannest, & Davis, 2011). ESs were calculated based on the DBR-SIS data and further verified based on SDO data.
Results
Visual analysis of all four students’ academically engaged data indicated a positive change in their behavior during their respective independent work times (see Figure 1). Samira and Sam showed an immediate increase (25% and 40%, respectively) in their academic engaged time, as indicated by their teachers’ completion of the DBR-SIS, on introduction to the intervention. While Gia and Noah did not demonstrate immediate effects, their data did indicate an increasing trend of on-task behavior while using the CellF-Monitor. On introduction to the reinforcement, Samira demonstrated an immediate increase of AE, as indicated by her teacher’s DBR-SIS ratings. Gia, Noah, and Sam, on the other hand, demonstrated an increasing trend in AE during the reinforcement phase. Overall, data were not variable with the highest variability observed during Gia’s training phase and Sam’s intervention phase.
Teacher reported academic engagement and systematic direct observation reported on-task behavior for participating students across phases.
Visual analyses were augmented with tests of statistical significance (p < .001) and ES estimates; the overall ESs based on the DBR-SIS were 93% and 86% as calculated by NAP and Taunovlap, respectively. The effects based on SDO ratings validated these findings with ESs of 95% and 90% as calculated by NAP and Taunovlap, respectively (see Table 1). ESs varied across participants, with Samira demonstrating the largest effect, as determined by the fewest overlap of all pairs, based on her teachers DBR-SIS rating (NAP = 100%; Taunovlap = 100%). Interestingly, ESs were larger for Gia, Noah, and Sam compared with Samira based on the fewest overlapping pairs of SDO data.
Effect Sizes for Direct Behavior Ratings and Systematic Direct Observations Across Phase Changes.
Note. NAP = nonoverlap of all pairs; DBR-SIS = Direct Behavior Rating–Single Item Scale; SDO = systematic direct observation; ES = effect sizes.
moderate effects. ** strong effects
Visual analysis of the reinforcement phase indicated overall increases of AE (p < .001). Visual analyses were also augmented with ES estimates—On average, the ESs between intervention and the reinforcement phase as indicated by the DBR-SIS were 70% and 40% for the NAP and Taunovlap calculations, respectively. The effects based on SDO ratings corroborated these findings with ESs of 77% and 55% for respective NAP and Taunovlap calculations (see Table 1). Thus, the overall effect of including reinforcement fell within the moderate effect range for both the DBR-SIS and SDO data (Parker & Vannest, 2009).
Social Validity
Social validity was measured from the perspective of both the participating teachers and students. Teachers completed the Usage Rating Profile–Intervention Revision (URP-IR), and the students completed the Children’s Usage Rating Profile (CURP) to evaluate the social validity of the CellF-Monitor. The URP-IR assesses acceptability, understanding, feasibility, family–school collaboration, system climate, and system support as a six-factor structure (Cronbach’s α = .67–.95; Briesch, Chafouleas, Neugebauer, & Riley-Tillman, 2013). The CURP measures students’ perceptions of the intervention including feasibility, understanding, and desirability (Cronbach’s α = .75–.92; Briesch & Chafouleas, 2009a).
URP-IR
Overall, participating teachers agreed that the CellF-Monitor was acceptable, feasible, understandable, and aligned to system climate. The highest scores were obtained on the Acceptability subscale (M = 4.96), with responses falling between the slightly agree to strongly agree range, indicating participating teachers agreed that the CellF-Monitor was an acceptable intervention. In addition, teachers agreed that the intervention was feasible (M = 4.89), they understood the strategies of the intervention (M = 4.89), and it was consistent with the climate of their school (M = 4.73). Home School Collaboration (M = 3.11) and System Support (M = 3.33) subscales received the lowest scores; teachers did not view home-school collaboration or additional supports as essential components of this intervention.
CURP
The highest scores from participating students were obtained on the Understanding subscale (M = 3.71), with responses falling between the kind of disagree to totally agree range. This indicated that participating students understood the components of the CellF-Monitoring intervention. In addition, students agreed that the intervention was desirable (M = 3.00). The Feasibility subscale (M = 2.00) received the lowest scores; students did not view the CellF-Monitor as a feasible intervention. In summary, students agreed that they understood the intervention and found using the CellF-Monitor application personally desirable; however, there were concerns reported by students regarding feasibility.
Dosage and Treatment Integrity
Participants used the iPad during their designated independent work time 84% of eligible periods (Samira = 92%, Gia = 71%, Noah = 79%, Sam = 95%), falling within the 70% to 90% threshold recommended by Gresham (2009). Of the missed implementation opportunities of the CellF-Monitoring intervention, 37% were explained by school-wide testing, 26% were due to class field trips, 26% were due to teacher absences, and 11% were due to student absences. It is important to note that the intervention was implemented in all occasions in which both the teacher and student were present for the designated independent work time.
Discussion
Overall, visual analyses suggested that the use of the CellF-Monitor application increased AE during independent work time, as indicated by each students’ immediacy of effect, level, trend, and low incidents of overlap after the introduction of the intervention, meeting the standards of Evidence set by What Works Clearinghouse (Kratochwill et al., 2010). The most variability in AE was observed during Gia’s training phase and Sam’s intervention phase. One explanation for Gia’s variable training phase is that the introduction of the CellF-Monitor occurred just before school-wide testing began, limiting the number of days the application could be used. Regarding Sam’s variable intervention phase, his lowest rating occurred directly after a 10-day break. It is possible that time away from the school affected Sam’s behavior more than other students.
Across cases, the staggered onset of the CellF-Monitor intervention illustrated changes in behavior across phases rather than time; in other words, baseline levels remained consistent as other students were introduced to the intervention, providing evidence for the functional relationship between using the CellF-Monitor and increasing AE. Visual analyses were strengthened with ES estimates; the overall effect of using the CellF-Monitor fell within the strong effect range for both the DBR-SIS and SDO data according to criteria set by Parker and Vannest (2009).
ESs varied across participants; while Samira demonstrated the fewest overlapping pairs based on her teacher’s DBR-SIS rating (NAP = 100%; Taunovlap = 100%), ESs were larger for Gia, Noah, and Sam based on SDO data. This variability could be explained by a number of factors. First, teachers completed the DBR-SIS after every designated independent work time for the participating students on a daily basis. SDOs were not completed daily and only represented segments of the independent work time (e.g., 20-minute observations representing 30 minutes of independent work). As illustrated in Figure 1, SDO ratings of on-task behavior were well aligned to teacher’s DBR-SIS ratings. Thus, it was not a difference in ratings but rather a function of the days an SDO occurred compared with the daily DBR-SIS ratings. It is possible that behavior was influenced by the presence of an outside observer, indicating a larger effect for some students.
Aligned to findings from Graham-Day et al. (2010), visual analysis and ES estimates of the reinforcement phase indicated increased AE compared with the intervention phase (p < .001). Thus, the ES estimates seem to support the addition of contingent reinforcement when using the CellF-Monitor application for elementary students. However, unlike the introduction of the CellF-Monitor intervention, significant gains were only present for half of the participating students with the introduction of contingent reinforcement (see Table 1). Thus, the benefits of using contingent reinforcement with the CellF-Monitor appears to vary across students; including contingent reinforcement is not a necessary component to improving AE using the CellF-Monitor but did enhance overall effects.
Limitations and Future Directions
Aligning with previous findings, the present study yielded promising findings regarding the use of the CellF-Monitor application to increase AE and on-task behavior during independent work time for elementary students without exceptionality. However, this study has a number of limitations that must be considered. First, the study contained a relatively small sample of only four students. Although this meets the standards set by What Works Clearinghouse, further research is necessary to replicate this study using a broader sample. A second limitation was the inconsistency in how frequently the application was used across participants; whereas all students demonstrated increased AE, there was not enough control to determine how dosage affected AE in the present study. Future research should consider the influence of how often the CellF-Monitor should be used for optimal effects. In addition, the present study did not measure work completion, achievement, or any other proxy for educational outcomes. Thus, future research should determine whether increasing AE in students during independent work time through the use of the CellF-Monitor improves academic outcomes.
Although this study found promising results for the use of the iOS-based SM intervention, CellF-Monitor, it is not well understood what added gains the technology provides in comparison with nontechnology-based SM interventions. Further research is necessary to understand whether technology-based interventions have additional benefits. Finally, neither generalization nor maintenance data were collected due to the school year ending. Future research is needed to determine the lasting effects of using the CellF-Monitor and generalizability across settings.
Conclusion
Results from this study inform SM practices in a number of ways. This was the first study to use the iOS application, CellF-Monitor, with elementary-aged students. Moreover, this was the only technology-based SM study with nondisabled participants to date. The present study also included a reinforcement phase—Previous studies that utilized technology-based SM interventions did not include any type of reinforcement. While the CellF-Monitor on its own was found to be effective in increasing on-task behavior and AE, the addition of contingent reinforcement enhanced these effects. It is important to note that while the overall effects were higher when contingent reinforcement was used in combination with the intervention, benefits were not seen across all participants. Therefore, although unnecessary, reinforcement could be used to further increase the effects of using the CellF-Monitor to increase on-task behavior and AE for certain individuals. Consistent with previous findings, the present study found that the use of this technology was not a distraction to other students within the classroom (Blood et al., 2011; Gulchak, 2008). Both participating students and teachers agreed that the CellF-Monitor on the iPad minis were not problematic for nonparticipating students. Thus, there appears to be promise for using the CellF-Monitor to support AE during independent work time based on findings from the present study; CellF-Monitor effectively increased AE and on-task behavior for four elementary students, without exceptionality, across subject areas.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.