Check-In/Check Out: Electronic Adaptation and Individual Progress Monitoring

Abstract

Check-in/Check-Out (CICO) has a long line of research evidence demonstrating its effectiveness in increasing prosocial behavior. The current paper demonstrated an electronic application of CICO utilizing Google Sheets® with teacher feedback. Google Sheets® offers an inexpensive, collaborative, and remote method for tracking behaviors. In the first study, 2,322 teacher ratings (from 38 teachers) were compared between traditional paper CICO forms or electronic Google Sheets®. Results found that teacher ratings were significantly more complete with the use of the electronic forms. In the second study, an electronic CICO form was used for progress monitoring and performance feedback with a middle school student. Through the form we were able to successfully track our participant’s behavior change in response to CICO with the combination of feedback and a differential reinforcement intervention. Social validity showed that overall teacher ratings were high with respect to ease of use, usefulness, cost-effectiveness, and convenience of the electronic Google Sheets®.

Keywords

check-in check-out applied behavior analysis methodologies Google

Behavioral interventions often require prolonged and consistent progress monitoring data. Such data is pertinent for evaluating the responsiveness of implemented strategies and attainment of individualized behavioral objectives (Burke & Vannest, 2008). Methods chosen for data collection often depend on (a) the form of behavior (e.g., topography) and (b) the feasibility of documentation on the part of the observer. For example, a classroom teacher who is charged with managing a class of 30 students may opt for utilizing daily ratings for aggression, as opposed to a 1:1 paraeducator who may be more likely to use frequency recording. For social-behavioral interventions, progress monitoring has not been as robust as academic interventions and there is a need for efficiency and specificity of behavior with the frequent data collection needed to monitor change (Bruhn et al., 2020). While the gold standard in behavioral data collection is direct observation data, this has often been reported as too time-consuming and cumbersome for teachers (e.g., Burke & Vannest, 2008; Gresham & Lopez, 1996). However, when conducting direct behavior observations, there is still great variability in the data collected considering the frequency of a behavior as well as inter-rater reliability and operational definitions for recording an occurrence or non-occurrence (Adamson & Wachsmuth, 2014; Bruhn et al., 2020). In general, a chosen method of data collection should sufficiently reflect the scope of the behavioral concern at hand to increase the frequency of which data is able to be meaningfully collected.

Technology in Education

Technology has revolutionized over the past decade how researchers and practitioners are addressing behavioral concerns. For example, Burke and Vannest (2008) presented an electronic, web-based system to allow for flexible behavioral progress monitoring. Benefits of such a system include greater ease in sharing and collaboration among stakeholders and prompt feedback regarding progress. Various authors (e.g., Carpenter et al., 2015; Dulin et al., 2014; Lui et al., 2017; McConnell et al., 2017, 2018) across a range of health-related occupations have demonstrated how mobile applications could be applied toward collecting and analyzing behavioral data. For example, Google Fit® and Samsung Health® are apps designed to assist consumers in monitoring their own fitness-related behaviors (Johnson et al., 2016; Polese et al., 2019).

Technological trends have also been observed in the field of education and psychology. These have included the use of mobile applications (e.g., Gulchak, 2008; Lui et al., 2017), text messages for self-monitoring of behavior (e.g., Bicard et al., 2012), and the incorporation of student response systems (e.g., Szwed & Bouck, 2013) in the implementation of behavioral interventions. Bruhn and colleagues (2015) examined the utility of an iPad application named SCORE IT with two middle school students. SCORE IT allowed the participants to self-monitor their behaviors and compare their scores with their teacher’s ratings through automatic graphs. Teachers then provided feedback to students, as well as reinforcers, contingent on behavioral goals being met. The authors found a decrease in disruptive behavior and increased academic engagement with the intervention components of self-monitoring and performance feedback.

Performance Feedback

Performance feedback is well researched in implementation science and school-based consultation practice, including evidence of effectiveness from a meta-analysis demonstrating improvements and maintenance in treatment (Noell et al., 2014). Most performance feedback literature has emphasized the use of this between a teacher and consultant to improve treatment integrity (e.g., Noell et al., 1997); however, this method has also demonstrated effectiveness with individual students. For individual student feedback, behavior specific praise is one form of student-level performance feedback, which may also include providing the student with a visual of their progress to support self-monitoring (Reinke et al., 2008). Further, praise for students is most effective when it is immediate, descriptive, and specific (Sutherland et al., 2001). When performance feedback is used directly with students, prior research has shown this application helps students understand their behaviors relative to behavior expectations (e.g., Stevenson & Mussalow, 2019) and can be incorporated in combination with other intervention procedures. Unfortunately, teachers often do not use praise consistently or frequently enough (Landrum et al., 2003).

Teacher Ratings and Check-In/Check-Out

The incorporation of teacher ratings as a behavioral intervention component also has a long line of research evidence. Examples have included the use of targeted positive behavior intervention and supports implemented with individual students such as daily behavior report cards (Iznardo et al., 2017) and Check-In/Check-Out (CICO; Crone et al., 2010). CICO is a popular intervention which has demonstrated effectiveness in the reduction of problem behavior and an increase of academic engaged time (Miller et al., 2015). Typically, the intervention is implemented daily and begins with a check-in meeting between the student and their assigned mentor at the start of each school day. The mentor provides a review of the student’s progress, behavioral expectations, and goals. As the student progresses throughout the school day, teachers rate the student’s performance on a progress monitoring sheet. Check-out, which typically occurs at the end of each school day, includes a review of the student’s performance ratings with their mentor, as well as the provision of feedback and incentives in response to their performance. Various authors have conducted systematic reviews on CICO throughout the past several years demonstrating the overall effectiveness of the intervention for both monitoring student progress and increasing student knowledge of behavioral expectations such as for escape-maintained behavior and internalizing behavior in, primarily, elementary school settings (e.g., Drevon et al., 2019; Laging et al., 2018; Wolfe et al., 2016). Further, in a recent meta-analysis by Drevon and colleagues (2019) the authors examined 32 studies on CICO across 59 dependent variables and 180 participants. Results showed that overall CICO improved student behavior by more than one standard deviation in comparison to control or baseline conditions. Findings were similar when problem behavior and academic engagement were analyzed, with effect size outcomes of 1.16 and 1.53 for each dependent variable, respectively. However, the effectiveness can also be attributed to the higher percentage of studies (68%) that included assessments (FBA’s) prior to implementing CICO when working with individual students (Drevon et al., 2019).

While prior meta-analyses have shown success for a majority of participants, other work has shown roughly 22% of participants are considered non-responders, citing student absences and non-instructional time as concerns that affect the usability of data (Swoszowski et al., 2013). Additionally, CICO studies often refer to modified procedures for behaviors maintained by escape suggesting a possible interaction effect between procedure and the type of problem behavior (Drevon et al., 2019). Prior research has shown a number of adaptations to the CICO procedure with the use of a peer mentor, addition of social skills training, or even homework planner checks (Majeika et al., 2020). As a demonstrated evidence-based intervention established by What Works Clearinghouse (WWC, 2017) for attention-maintained behavior, CICO may benefit from a technological adaptation that enhances usability and ease of implementation. Additionally, a technological adaptation may help address the previously cited concerns of missing or incomplete data. However, prior studies have not examined the difference between CICO progress monitoring with a technological adaptation versus the standard paper-pencil format.

Thus, the following three research questions were examined using two-studies. The first study involved a comparison between paper and electronic CICO forms to answer the following research question: 1) To what extent did the use of an electronic adaptation of CICO support improvements in frequency and completeness of data collection compared to the traditional paper-pencil CICO data collection procedure? Next, in study 2, the electronic CICO form was implemented with a middle school student and its effectiveness for monitoring a differential reinforcement of other behaviors (DRO) intervention was examined. Study 2 addressed the following research questions: 2) To what extent did the addition of an electronic CICO form facilitate student progress monitoring during a DRO intervention to improve student prosocial attempts for requesting attention? And 3) To what extent did teachers indicate a preference for the CICO measure in an electronic versus paper-pencil format for daily behavior ratings? In the first study, we conducted a comparison between paper and electronic CICO forms by examining percent of completion and ratings of effectiveness between both forms. In the second study, we examined an application of the electronic CICO form with a middle school student to monitor intervention effectiveness of a DRO intervention.

Methods: Study 1

Twelve middle school students (Grade 6–8) and 38 teachers from an urban school district in Southern California participated in the first study (see Table 1). Referrals for intervention support came from administrators and general education teachers based upon student behavioral need (see operational definition below). Teachers and staff were familiar with the use of the paper-pencil CICO forms as the staff had all used CICO for at least 1 year or more. The paper CICO forms required 15 ratings per day including the beginning and end of the day and each class period with room for ratings by five academic content-area teachers. All forms required students to state behavioral goals in the three school-wide PBIS target areas of respect, responsibility, and safety. Students would be rated in these areas by each teacher. The students would hand deliver the paper forms to their teachers as they moved through their classes each day. Once a week, students would then meet with their counselor or case carrier, who would collect the forms and review their progress. Parents were not part of the data-collection procedures and their review of the CICO forms were not recorded as part of the current study.

Table 1.
Demographics Study 1.

Students N = 12

Gender (% Male) 66.67 (N = 8)

Race

% Hispanic/Latino 66.67 (N = 8)

% African American 8.33 (N = 1)

% White 16.67 (N = 2)

% Other/Mixed 8.33 (N = 1)

Grade

% Sixth Grade 16.77 (N = 2)

% Seventh Grade 41.67 (N = 5)

% Eighth Grade 41.67 (N = 5)

Special Education (%) 58.34 (N = 7)

% Autism 8.33 (N = 1)

% Emotional Disturbance 16.67 (N = 2)

% Other Health Impairment 16.67 (N = 2)

% Specific Learning Disability 16.67 (N = 2)

Teachers/Staff N = 38

Highest Degree (% MA+) 52.63 (N = 20)

Years of Educational Service (% 10+) 36.84 (N = 14)

Independent and Dependent Variables

Study 1 was a descriptive study to examine the outcomes and to compare the use of the two CICO forms. Student behaviors being monitored with a CICO procedure were based on individual referral concerns for increasing prosocial behaviors such as appropriately requesting or reducing challenging behaviors including aggression or safety-related concerns. The independent variable included the format of the CICO form (electronic or paper).

Google Sheets ® was the electronic format used for the electronic data collection in this study. Six primary teachers were trained on the procedures for using the electronic forms. The remaining staff (approximately 32 teachers) were second-hand trained (i.e., trainer-of-trainers model). This was a free online spreadsheet designed for the input and analysis of data, and with a variety of templates and editable properties, Google Sheets® allows users to tailor spreadsheets to a wide range of purposes. For this study, Google Sheets® was customized to create an electronic adaptation of a standard CICO form. Each participant teacher using the electronic form was assigned a workbook in which CICO data was input each day and accessible through teacher work emails only with records kept by student initials only to minimize student identity in the event of a data breach. The electronic CICO forms required 12 ratings per day with room for ratings for six teachers (incorporating the non-academic Physical Education teacher). The case carrier would call or email the teachers on a daily basis to remind them to complete the electronic CICO sheet if the student did not self-initiate since the form was not held with the student as with the paper-pencil form. Each workbook contained a “Cover Sheet,” in which users could input the student identifying information, reinforcers, and teacher ratings. Each teacher would log into the sheet on a daily basis to input their ratings of the student’s behaviors, as shown in row 16 of Figure 1. Once submitted, the data would transfer to a separate sheet where the workbook could calculate and chart the student’s progress. This was completed through a simple javascript add-on application created by the first author and programmed into the sheet. Calculations involved summing the total ratings for each form type and dividing them by the total possible ratings, based on separate scales for an equivalent comparison in the results. Figure 1 provides a hypothetical visual illustration of a workbook.

Figure 1.
Visual depiction of Google Sheets CICO progress monitoring tool. The first tab is the current “Cover Sheet,” and the second tab (Graph) provides a visual depiction of the charted data.

Teacher ratings of target behaviors were provided according to Likert scales (0–3 ratings). Dependent variables examined included the rate of teacher completion for each form, as well as highest ratings given for each version (e.g., score of 3 indicating appropriate behavior observed on a 0–3 rating on a Likert scale). Highest total ratings (e.g., students met the behavior goal) across forms allowed us to gain a better understanding of (a) teacher rating patterns and (b) reported intervention effectiveness. A combined total of 2,322 teacher ratings (across multiple observation days) were examined from the paper (N = 629 individual ratings) and electronic forms (N = 1,693 individual ratings) with a large difference revealed in higher completion rates with the electronic forms.

Procedure

The Institutional Review Board of the participating University approved the study procedures as well as the partner school district for the use of behavior specialist data to evaluate intervention procedures (across Study 1 and 2). Student participants were referred based on problem behaviors and/or failing grades in general education classes. The number of student participants who had CICO paper and electronic forms were proportionally split in half for student data collection; six participants received paper CICO forms throughout the progress monitoring phase and the other six received electronic forms. Thus, teacher/staff participants had the opportunity to complete both paper and electronic forms across the different student participants as teachers overlapped across student participants. Five of the six general education students in this sample had a grade of “D” or “F” in two or more classes. Convenience sampling was used for recruitment of student participants based on referrals for intervention services. Five of the six students who received CICO with paper forms were not eligible for special education services at the time of the study and the six students who received CICO electronic forms were eligible for special education services.

Daily CICO paper and electronic ratings were gathered and entered into an electronic database by a graduate student assistant. Each form was coded and analyzed by the first author examining (a) the number of total ratings, (b) the number of blank entries, (c) number of absences for each student, and (d) the total number of highest ratings received (e.g., student fully met their behavior goal). The number of blank entries were then corrected by the number of student absences so as to provide a more accurate depiction of teacher completion. The proportion of completed ratings were determined by dividing the total number of teacher ratings given by the total possible number of ratings. The proportion of highest rating (i.e., student meeting behavior expectations) was calculated by dividing the number of ratings received at the highest value by the total number of ratings.

Results

To answer research question one, the total number of forms completed was compared across paper and electronic CICO measures. Across both forms, teacher completion was the highest for the CICO electronic forms, as teachers completed 80% of CICO ratings. In contrast, form completion was only 49% of possible CICO ratings by paper forms. When a χ² analysis was conducted to examine if there were significant group differences between completed forms and the type of form used (electronic or paper) results indicated a significant difference, χ² = 50.12, p < .001. Further, 90% of all ratings given on the paper forms were the highest rating possible (e.g., a score of “3” on a Likert scale 0–3) with little variability in lower ratings given across teachers. In contrast, only 55% of all ratings given on the electronic forms were the highest rating, indicating a difference in teacher ratings, between paper and electronic forms for students meeting their behavior goals.

Discussion

Based on comparison data between paper and electronic forms, the electronic forms resulted in a significantly higher completion rate by teachers than paper ones. Electronic forms required teachers responsible for student check-out to formally submit ratings each day by pressing a button. It was noted by such teachers that pressing this button called their attention to the completion of the data, and would therefore prompt them to notify their colleagues (e.g., other teachers) if their ratings were missing. We would typically expect less extreme ratings when teachers are given a larger scale (e.g., 3 out of 0–3 on a Likert scale compared with a 2 on a 0–2 Likert scale), but we actually saw a wider range of ratings on the 0–3 scale (Barnett et al., 2005).

Standard CICO Procedures

Interestingly, although paper forms were completed with much lower frequency, teachers provided the highest ratings (e.g., students meeting the behavioral expectations) at a much greater rate than electronic forms. It therefore appears that teachers are more willing to provide a rating on a paper form when a student does exceptionally well. This could be due to teachers ensuring that students receive positive recognition (e.g., social praise) to reinforce their performance as well as due to avoiding providing negative feedback. As Downs and colleagues (2019) found, students with behavior problems often are more engaged and less disruptive when they receive higher rates of praise and lower rates of reprimands. Thus, this avoidance of reprimanding or providing negative written feedback (i.e., CICO), could be a result of the format of feedback between paper and electronic forms. Unlike the electronic forms which teachers tracked on their computers and students could “view” on their shared Google Sheets®, paper forms are traditionally kept by the student throughout their school day and require manual signatures. Students must present the forms to teachers as a prompt with this method and with the electronic method, teachers were prompted by the case carrier. With the paper ratings, teachers were also required to rate the student in front of them as they rotate through each class period regardless if the feedback was positive or negative. Low ratings may therefore evoke undesirable behaviors on the part of the student or a withdrawal from turning in their forms. Some teachers are poorly trained in how to provide instructional changes or on providing performance feedback and need additional management strategies, which may include providing both positive and corrective feedback (Downs et al., 2019; Landrum et al., 2003). In contrast, low ratings on the electronic forms could be provided immediately after a student leaves their period, therefore allowing teachers to avoid undesirable behaviors from the student and allow students to view their feedback at a preferred time (e.g., after class or end of day). However, when used appropriately, performance feedback that includes a clear description of behaviors and resulting consequences should be provided to help students understand behavioral expectations (Stevenson & Mussalow, 2019).

Methods: Study 2

The goal of Study 2 was used to evaluate the effectiveness of the electronic CICO form as in progress monitoring and performance feedback for a middle school student. The primary aim was to show how the form could both monitor the student’s response to CICO with staff feedback in combination with a DRO reinforcement schedule. We evaluated the effectiveness of these components using an ABCA reversal design with the ability to describe, predict, and test the prediction of the behavior through the reversal of baseline (Harding et al., 2001; Kazdin, 2011) to answer the following questions: 1) To what extent did the addition of an electronic CICO form facilitate student progress monitoring during DRO intervention to improve student prosocial attempts for requesting attention?; and 2) To what extent did teachers indicate a preference for the CICO measure in an electronic versus paper-pencil format for daily behavior ratings?

Participant

The participant was an eighth grade male student who attended a public urban middle school. “Steven” was a 13 year old Hispanic-Caucasian male who qualified for special education services under the disability category of Emotional Disturbance (ED). The student was referred for additional support for his problem behavior and after conducting a FBA, a behavior intervention plan was implemented using the following CICO and DRO intervention procedures. Steven exhibited attention-seeking behaviors such as inappropriately requesting attention through vocal outbursts (e.g., audible vocalizations emitted during class without teacher permission at a volume high enough to interfere with instruction) in his classes. Classroom teachers across four class periods rated the number of times the student inappropriately gained a peer or adult’s attention in their class after being given the full observational definition of the behavior. Teacher participants were the same as in Study 1 and were familiar with the use of the electronic CICO forms being used. The number of behavioral occurrences in each class period were rated along a Likert-scale ranging from 0–3 (i.e., with 0 representing 8 or more behavioral occurrences and 3 representing no occurrences of the target behavior). Ratings provided across all four classes were averaged together each school day. Due to the end-of-semester schedule change and a change to the student’s schedule during the study, a total of 15 teachers provided ratings throughout the intervention period.

Procedure

Setting

All of the participant’s classrooms included approximately 30 students arranged in either rows or groups. The classrooms included desks, chairs, and a whiteboard.

CICO as DBR (baseline)

In order to measure the dependent variable, or changes in student attention-seeking behavior, an ABCA design was used to test the intervention with the new CICO form and a DRO intervention procedure. The baseline phase primarily included progress monitoring data through the Google Sheets® adapted CICO form. Each teacher logged into the sheet on a daily basis for their marking period and rated the student’s behaviors. The student would receive a verbal review of his behavior at end of the day regarding his goals and rated progress during the check-out meeting with the last teacher. Such feedback included a verbal review and overall progress toward his goal. No incentives were provided during baseline nor was the student involved in a review of his detailed CICO progress. Therefore, the electronic form served as an independent variable and method to solely collect direct behavior ratings (DBR; e.g. Chafouleas et al., 2012) during the baseline phase.

Prompting + CICO (Phase 1)

During phase one of the intervention, Steven was assigned a 1:1 paraeducator. The paraeducator implemented both the CICO check-in portion and provided the student prompts for appropriate requesting throughout each of his periods. The student would check-in with his paraeducator at the beginning of each day. The paraeducator would review the student’s goals and expectations for the school day. During class time, the paraeducator and teachers would prompt Steven to use appropriate attention-seeking behavior when “precursor behaviors” were observed. Precursor behaviors were those which generally preceded the occurrence of inappropriate attention-seeking behaviors such as vocal outbursts. Precursor behaviors included the student becoming fidgety, getting out of his seat, and engaging in more negative verbal language. Prompts included verbal statements from teachers and the paraeducator such as “Remember to ask,” “Use appropriate language,” etc. At the end of each day, the student would check-out with his homeroom teacher who would review the student’s scores, provide verbal and visual performance feedback that included a review and summary of the CICO ratings given in the electronic form as well as a preferred incentive if he met his daily goal (e.g., 80% accuracy or score of 3 in 80% of observable occasions).

Prompting + CICO + DRO (Phase 2)

Similar to the previous phase, Steven continued to receive support from the 1:1 paraeducator. The prompting support from both the paraeducator and the classroom teacher continued as outlined in Phase 1. However, the student’s behaviors were also placed on a DRO schedule in which the paraeducator assigned a point once every 2 minutes (i.e., by timer on a clock) for the absence of inappropriate behaviors. This was done to motivate the student to use alternative prosocial behaviors (e.g., appropriate requesting). Points were provided electronically in a Google Sheet® spreadsheet that was continuously displayed in front of the student on a Chromebook throughout each period. Points were “cashed in” twice a day for larger incentives with the CICO performance feedback times at homeroom and lunch.

Return to baseline

Conditions were similar to those described during the baseline phase. Each teacher logged into the sheet on a daily basis for their marking period and rated the student’s behaviors. The student would receive feedback at the end of the day regarding his rated progress during the check-out meeting with the last teacher. Feedback included a verbal review and overall progress toward his goal, no additional incentives or reinforcement were provided during this phase.

As noted above, the experimental design included two baseline and two intervention phases. The decision to move from the first baseline to the first and second intervention phases was due to lack of stable progress on the part of the participant’s behavior. Robust responding in the second intervention phase removed the need to introduce further intervention.

Interobserver agreement (IOA) with teacher ratings were collected for 26% of recorded school days by a second independent observer trained by the behavioral specialist with exact reliability across 20% of observations and review of behavioral occurrences and non-occurrences. IOA was calculated for each daily period by dividing the smaller rating by larger rating and multiplying by 100%. Overall, IOA between the teachers and the second observer was 88%.

Measures

Social validity

A measure of social validity was created for feedback from the teacher participants who used the electronic CICO form across the intervention period with this student. All teacher participants had used the paper CICO forms in the past with other students and were instructed to compare the use of the electronic form to the traditional paper data collection. The social validity measure contained five items with a Likert scale response from 0 to 4 (strongly disagree to strongly agree).

Results

Results of the study are illustrated in Figures 2 and 3. Figure 2 provides an average of daily ratings of appropriate requesting behaviors that the student received in class from the teacher electronic CICO forms. Figure 3 provides an average of total ratings on the electronic CICO forms for each phase. Overall, the student received low ratings of requesting attention during the initial baseline phase in which he only received their daily check in and check out, or DBR (M = 1.56). This increased substantially during the Prompting + CICO (M = 2.46) phase with an overall effect as calculated by percent of all non-overlapping data (PAND; Parker et al., 2007 = 77.5%) suggesting this phase was moderately effective (Scruggs & Mastropieri, 1998). During the second phase, prompting + CICO + DRO (M = 2.66) resulted in a greater effect (PAND = 95.7%). Ratings then demonstrated a decreasing trend during the return to baseline phase as a result of intervention removal. The student’s overall ratings during both baselines averaged 1.85.

Figure 2.
Displayed progress monitoring data for Steven across school days.

Figure 3.
Phase averages for teacher ratings for appropriately requesting attention.

Results of the teacher ratings of social validity were used to answer research question two. Based on the teacher participants who used the new form for Steven throughout the day, 100% of respondents indicated the form was “easy to use.” With regard to item 2, “The sheet provides useful information about behavior,” 75% of respondents either agreed or strongly agreed. Responses for item 3, “The sheet is more convenient to track than paper versions,” were similar in that 75% of respondents either agreed or strongly agreed. Responses for item 4, “The sheet is cost-effective,” included 75% of respondents either agreeing or strongly agreeing with the statement. Finally item 5 responses, “Student(s) respond better to feedback from the sheet than paper versions,” were split, with 25% of respondents strongly agreeing, 50% neutral, and 25% disagreeing. Overall, teachers found the electronic form useful and easy to use but were unsure if it was more helpful in providing feedback than the paper-pencil version.

Discussion

These two studies examined a school-based application of CICO using a Google Sheets® adaptation across multiple students in comparison to paper-pencil ratings and the individual use to provide performance feedback in combination with prompting and DRO procedures. Google Sheets® provided a simple and cost-free method of tracking behavior that allows for increased frequency for student feedback throughout a school-day.

Adaptation to CICO Procedures

When examining the use of the electronic CICO forms for progress monitoring as an intervention through daily ratings made by teachers, we were able to track a student’s behavioral progress and examine the efficacy of the school-based intervention. As a recent evaluation showed, 71% of the studies made adaptations to the standard protocol of CICO procedures based on student-level variables (e.g., for praising or incentives) as well as how and when to monitor data (Majeika et al., 2020). In this adaptation, we aimed to maintain as close to a standard CICO protocol as possible and to consider making progress monitoring simpler for teacher participants. The Google Sheet® furthermore provided a very quick and easy display of progress monitoring data (see Figure 4) to use for prompting with the student throughout the day or week.

Figure 4.
Hypothetical visual depiction of Google Sheets CICO progress monitoring chart. The sample-charted data shows the x-axis with the dates of each rating, and the y-axis with the percent of total possible ratings.

With use of prompting, CICO, and differential reinforcement, the student participant in Study 2 was able to demonstrate appropriate requesting for attention with the frequency of this behavior improving during the CICO phases. The use of performance feedback including visual and verbal prompting aided the student in drawing his attention to the behavior expectations as well as bring his awareness to the goals and progress he was, or was not, making (Stevenson & Mussalow, 2019). The student in Study 2 had the Google Sheet® available to him throughout the day on his Chromebook and in which he was able to view and monitor his progress continuously. Further, as shown in the return to baseline, just having the behavior monitored by an adult without feedback, may not be sufficient alone for a student to understand how and when to modify their behavior to meet the behavioral expectations. The combination of behavior intervention procedures was found to be even more beneficial than just one procedure alone and demonstrated that the CICO forms with electronic access could allow the paraeducator use for student prompting and feedback with the summary of overall performance. A second observer collected data on the student and ratings were compared to the paper CICO forms which demonstrated consistent IOA reliability. To note, such high IOA is an indication of convenience that the sheet affords in progress monitoring and increasing accurate data tracking when teachers can record the students’ behavior and designate appropriate feedback meetings with the student. While treatment fidelity does not guarantee an intervention is effective, it does ensure correct decisions are made about the student’s responsiveness to intervention (e.g., Bruhn et al., 2018). Given the high IOA data with the electronic forms and social validity ratings, it appears encouraging that this adaptation is both beneficial for the teacher and students. Thus, the hypothesis that the electronic forms were a tool for improving data collection was supported by the teachers’ social validity ratings as teachers strongly agreed with the statement that the sheet was “easy to use” and “more convenient to track behavior than paper versions.” Interestingly, only 25% of the teachers felt that students responded better to feedback with the electronic form.

Limitations and Future Directions

Overall, the use of an electronic adaptation for CICO rating forms was an efficient method for data collection as well as for use with for progress monitoring. While there are other electronic CICO tracking forms available for download, the use of this adaption was primarily for demonstrating the effectiveness of an inexpensive and collaborative method for progress monitoring and data collection in which multiple teachers could access and provide feedback throughout the day and week. It is important to note that since Google Sheets® is a cloud-based program, student confidentiality could still be violated if forms are not secured and it is recommended to obtain consent for using this as a means for data collection. A limitation of the sheet includes developers needing background knowledge and training in how to navigate through Google Sheets® as well as quality check of training in a trainer-of-trainers model for all staff use. Although users such as the teachers in this study had very limited knowledge of Google Sheets®, they were able to easily access and rate the participant’s behavior as well as access the form to provide the student feedback. To note, a background in Microsoft Excel® could be sufficient as Google Sheets® utilizes comparable formulas and functions; however, the increased access to efficient data collection systems is beneficial to a wider audience.

Future research should consider randomization of teachers in use of paper versus electronic forms to compare the two data-collection methods across users and required teacher training and support for transitioning to a new progress monitoring tool if teachers are primarily comfortable with paper-pencil methods for data collection. While social validity was not measured in Study 1, the frequency of ratings and completion of the electronic forms being higher than the paper forms suggests a more efficient and useful way for teachers to provide these daily behavior ratings. Further, for consistency, the paper and electronic forms should have been reviewed with students at the end-of-period but with students’ independent access to their electronic form, this was not monitored as closely for each participant in this study. This is important to consider as a school-based provider for consultation efforts to support the teacher and ensure intervention treatment fidelity. Ongoing consultation support is always recommended when modifying an intervention procedure and to increase treatment fidelity. In Study 2, while the student did have continuous access to his performance throughout the day on his tablet, the review of feedback with verbal prompting was retained until end-of-day. However, the behavior change was documented through a reversal in behavior when the intervention was withdrawn. While we did not tease out the specific elements of the intervention in a component analysis, this can be an avenue for future research. Future research should consider the time and level of support needed to assist teachers in accurate data collection when implementing a new system.

In Study 2, it was noted that the return to baseline was an example of the use of the CICO form in the most basic application but that use of the paraeducator was not counterbalanced in this reversal design and does pose a confound to the effects of the aide. Further, while Study 2 did not include a measure of treatment fidelity, the authors findings are limited to the assumption that each phase’s procedures were implemented with integrity. Due to school closures, further phases of the intervention were unable to be completed but future studies will examine a robust look between the intervention phases.

This study is promising in making an adaptation to data collection procedures that can fit evolving teacher needs (e.g., ease of access and reduced use of paper-materials). Additionally, with substitute teachers and rotating staff, the use of electronic access to a measurement system can increase access for all parties involved when most schools have shared electronic materials. Further, additional research could be conducted on student buy-in and perception of the behavior support regarding paper and electronic versions of CICO sheets. It was noted by teachers in this study that ‘Steven’ would have been very reluctant to carry a paper CICO sheet to each of his periods and have face-to-face check-ins at the end of each period. This would have therefore made it very difficult to track progress if the student was resistant. With the changing demands of technology in educational settings, this method of progress monitoring was a convenient adaptation to create a cost-effective solution that affords users the same functionality as professional applications without in-depth programming. In addition, the progress monitoring procedures with Steven allowed for more specific feedback as forms were completed more consistently. With forms completed more consistently across periods and days, the decision-making process for intervention changes can be evaluated for greater efficiency. Thus, this is an area that future research should consider when designing progress monitoring tools to increase social validity and utility for intervention decision making.

Students	N = 12
Gender (% Male)	66.67 (N = 8)
Race
% Hispanic/Latino	66.67 (N = 8)
% African American	8.33 (N = 1)
% White	16.67 (N = 2)
% Other/Mixed	8.33 (N = 1)
Grade
% Sixth Grade	16.77 (N = 2)
% Seventh Grade	41.67 (N = 5)
% Eighth Grade	41.67 (N = 5)
Special Education (%)	58.34 (N = 7)
% Autism	8.33 (N = 1)
% Emotional Disturbance	16.67 (N = 2)
% Other Health Impairment	16.67 (N = 2)
% Specific Learning Disability	16.67 (N = 2)
Teachers/Staff	N = 38
Highest Degree (% MA+)	52.63 (N = 20)
Years of Educational Service (% 10+)	36.84 (N = 14)

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Leigh Ann Tipton-Fisler

References

Adamson

R. M.

Wachsmuth

S. T.

(2014). A review of direct observation research within the past decade in the field of emotional and behavioral disorders. Behavioral Disorders, 39(4), 181–189. https://doi.org/10.1177%2F019874291303900403

Barnett

A. G.

van der Pols

J. C.

Dobson

A. J.

(2005). Regression to the mean: What it is and how to deal with it. International Journal of Epidemiology, 34(1), 215–220. https://doi.org/10.1093/ije/dyh299

Bicard

D. F.

Lott

Mills

Bicard

Baylot-Casey

(2012). Effects of text messaged self-monitoring on class attendance and punctuality of at-risk college student athletes. Journal of Applied Behavior Analysis, 45(1), 205–210. https://dx-doi-org.web.bisu.edu.cn/10.1901/jaba.2012.45-205

Bruhn

A. L.

McDaniel

S. C.

Rila

Estrapala

(2018). A step-by-step guide to tier 2 behavioral progress monitoring. Beyond Behavior, 27(1), 15–27. https://dx-doi-org.web.bisu.edu.cn/10.1177/1074295618756984

Bruhn

A. L.

Vogelgesang

Schabilion

Waller

Fernando

(2015). “I don’t like being good!” Changing behavior with technology-based self-monitoring. Journal of Special Education Technology, 30(3), 133–144. https://dx-doi-org.web.bisu.edu.cn/10.1177/0162643415618911

Bruhn

A. L.

Wehby

J. H.

Hasselbring

T. S.

(2020). Data-based decision making for social behavior: Setting a research agenda. Journal of Positive Behavior Interventions, 22(2), 116–126. https://doi.org/10.1177%2F1098300719876098

Burke

M. D.

Vannest

K. J.

(2008). Behavioral-progress monitoring using the electronic daily behavioral report card (e-DBRC) system. Preventing School Failure: Alternative Education for Children and Youth, 52(3), 51–60. https://doi.org/10.3200/PSFL.52.3.51-60

Carpenter

V. L.

Hertzberg

J. S.

Kirby

A. C.

Calhoun

P. S.

Moore

S. D.

Dennis

M. F.

Dennis

P. A.

Dedert

E. A.

Hair

L. P.

Beckham

J. C.

(2015). Multi-component smoking cessation treatment including mobile contingency management for smoking cessation in homeless veteran smokers. The Journal of Clinical Psychiatry, 76(7), 959–964. https://doi.org/10.4088/JCP.14m09053

Chafouleas

S. M.

Sanetti

L. M. H.

Jaffery

Fallon

(2012). Research to practice: An evaluation of a class-wide intervention package involving self-management and a group contingency on behavior of middle school students. Journal of Behavioral Education, 21, 34–57. https://dx-doi-org.web.bisu.edu.cn/10.1007/s10864-011-9135-8

10.

Crone

D. A.

Hawken

L. S.

Horner

R. H.

(2010). Responding to problem behavior in schools: The behavior education program (2nd ed.). Guilford Press.

11.

Downs

K. R.

Caldarella

Larsen

R. A. A.

Charlton

C. T.

Wills

H. P.

Kamps

D. M.

Wehby

J. H.

(2019). Teacher praise and reprimands: The differential response of students at risk of emotional and behavioral disorders. Journal of Positive Behavior Interventions, 21(3), 135–147. https://dx-doi-org.web.bisu.edu.cn/10.1177/1098300718800824

12.

Drevon

D. D.

Hixson

M. D.

Wyse

R. D.

Rigney

A. M.

(2019). A meta-analytic review of the evidence for check-in check-out. Psychology in the Schools, 56(3), 393–412. https://dx-doi-org.web.bisu.edu.cn/10.1002/pits.22195

13.

Dulin

P. L.

Gonzalez

V. M.

Campbell

(2014). Results of a pilot test of a self-administered smartphone-based treatment system for alcohol use disorders: Usability and early outcomes. Substance Abuse, 35(2), 168–175. https://dx-doi-org.web.bisu.edu.cn/10.1080/08897077.2013.821437

14.

Gresham

F. M.

Lopez

M. F.

(1996). Social validation: A unifying concept for school-based consultation research and practice. School Psychology Quarterly, 11(3), 204–227. https://dx-doi-org.web.bisu.edu.cn/10.1037/h0088930

15.

Gulchak

D. J.

(2008). Using a mobile handheld computer to teach a student with an emotional and behavioral disorder to self-monitor attention. Education and Treatment of Children, 31(4), 567–581. https://dx-doi-org.web.bisu.edu.cn/10.1353/etc.0.0028

16.

Harding

J. W.

Wacker

D. P.

Berg

W. K.

Barretto

Winborn

Gardner

(2001). Analysis of response class hierarchies with attention-maintained problem behaviors. Journal of Applied Behavior Analysis, 34(1), 61–64. https://doi.org/10.1901/jaba.2001.34-61

17.

Iznardo

Rogers

M. A.

Volpe

R. J.

Labelle

P. R.

Robaey

(2017). The effectiveness of daily behavior report cards for children with ADHD: A meta-analysis. Journal of Attention Disorders, 24(12), 1623–1636. https://dx-doi-org.web.bisu.edu.cn/10.1177/10870547177346

18.

Johnson

Turek

Dornfeld

Drews

Hansen

(2016). Validity of the Samsung phone S health application for assessing steps and energy expenditure during walking and running: Does phone placement matter? Digital Health, 2, 1–8. https://dx-doi-org.web.bisu.edu.cn/10.1177/2055207616652747

19.

Kazdin

A. E.

(2011). Single-case research designs: Methods for clinical and applied settings (2nd ed.). Oxford University Press.

20.

Laging

Buerger

Hoekstra

Childs

(2018). Systematic review of the check-in/check-out components and the effectiveness for students with challenging behaviors. The Online Journal of New Horizons in Education-October, 8(4). https://www.tojned.net

21.

Landrum

T. J.

Tankersley

Kauffman

J. M.

(2003). What is special about special education for students with emotional and behavioral disorders? Journal of Special Education, 37(3), 148–156.

22.

Lui

J. H.

Marcus

D. K.

Barry

C. T.

(2017). Evidence-based apps? A review of mental health mobile applications in a psychotherapy context. Professional Psychology: Research and Practice, 48(3), 199–210. https://dx-doi-org.web.bisu.edu.cn/10.1037/pro0000122

23.

Majeika

C. E.

van Camp

A. M.

Wehby

J. H.

Kern

Commisso

C. E.

Gaier

(2020). An evaluation of adaptations made to check-in check out. Journal of Positive Behavior Interventions, 22(1), 25–37. https://dx-doi-org.web.bisu.edu.cn/10.1177/1098300719860131

24.

McConnell

M. V.

Shcherbina

Pavlovic

Homburger

J. R.

Goldfeder

R. L.

Waggot

Cho

M. K.

Rosenberger

M. E.

Haskell

W. L.

Myers

Champagne

M. A.

Mignot

Landray

Tarassenko

Harrington

R. A.

Yeung

A. C.

Ashley

E. A.

(2017). Feasibility of obtaining measures of lifestyle from a smartphone app: The MyHeart counts cardiovascular health study. JAMA Cardiology, 2(1), 67–76. https://dx-doi-org.web.bisu.edu.cn/10.1001/jamacardio.2016.4395

25.

McConnell

M. V.

Turakhia

M. P.

Harrington

R. A.

King

A. C.

Ashley

E. A.

(2018). Mobile health advances in physical activity, fitness, and atrial fibrillation: Moving hearts. Journal of the American College of Cardiology, 71(23), 2691–2701. https://dx-doi-org.web.bisu.edu.cn/10.1016/j.jacc.2018.04.030

26.

Miller

L. M.

Dufrene

B. A.

Sterling

H. E.

Olmi

D. J.

Bachmayer

(2015). The effects of check-in/check-out on problem behavior and academic engagement in elementary school students. Journal of Positive Behavior Interventions, 17(1), 28–38. https://dx-doi-org.web.bisu.edu.cn/10.1177/1098300713517141

27.

Noell

G. H.

Gansle

K. A.

Mevers

J. L.

Knox

R. M.

Mintz

J. C.

Dahir

(2014). Improving treatment plan implementation in schools: A meta-analysis of single subject designs. Journal of Behavioral Education, 23, 168–191. https://dx-doi-org.web.bisu.edu.cn/10.007/s10864-013-9177-1

28.

Noell

G. H.

Witt

J. C.

Gilbertson

D. N.

Ranier

D. D.

Freeland

J. T.

(1997). Increasing teacher intervention implementation in general education settings through consultation and performance feedback. School Psychology Quarterly, 12, 77–88. https://dx-doi-org.web.bisu.edu.cn/10.1037/h0088949

29.

Parker

R. I.

Hagan-Burke

Vannest

K. J.

(2007). Percent of all nonoverlapping data PAND: An alternative to PND. Journal of Special Education, 40(4), 194–204. https://psycnet.apa.org/doi/10.1177/00224669070400040101

30.

Polese

J. C.

Faria

G. S. E.

Ribeiro-Samora

G. A.

Lima

L. P.

de Morais Faria

C. D. C.

Scianni

A. A.

Teixeira-Salmela

L. F.

(2019). Google fit smartphone application or Gt3X Actigraph: Which is better for detecting the stepping activity of individuals with stroke? A validity study. Journal of Bodywork and Movement Therapies, 23(3), 461–465. https://dx-doi-org.web.bisu.edu.cn/10.1016/j.jbmt.2019.01.011

31.

Reinke

W. M.

Lewis-Palmer

Merrell

(2008). The classroom check-up: A classwide teacher consultation model for increasing praise and decreasing disruptive behavior. School Psychology Review, 37(3), 315–332. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2603055/

32.

Scruggs

T. E.

Mastropieri

M. A.

(1998). Summarizing single-subject research: Issues and applications. Behavior Modification, 22(3), 221–242. https://dx-doi-org.web.bisu.edu.cn/10.1177/01454455980223001

33.

Stevenson

N. A.

Mussalow

P. R.

(2019). The effects of planning, goal setting, and performance feedback on avoidance behaviors for struggling readers. Journal of Positive Behavior Interventions, 21(3), 171–180. https://dx-doi-org.web.bisu.edu.cn/10.1177/1098300718804566

34.

Sutherland

K. A.

Copeland

Wehby

J. H.

(2001). Catch them while you can: Monitoring and increasing the use of effective praise. Beyond Behavior, 11(1), 46–49. http://www.ecstresources.com/uploads/6/1/9/1/6191025/_monitoring_and_increasing_the_use_of_effective_praise.pdf

35.

Swoszowski

N. C.

McDaniel

S. C.

Jolivette

Melius

(2013). The effects of a tier II check-in/check-out including adaptation for non-responders on the off-task behavior of elementary students in a residential setting. Education and Treatment of Children, 36(3), 63–79. https://dx-doi-org.web.bisu.edu.cn/10.1353/etc.2013.0024

36.

Szwed

Bouck

E. C.

(2013). Clicking away: Repurposing student response systems to lessen off-task behavior. Journal of Special Education Technology, 28(2), 1–12. https://doi.org/10.1177%2F016264341302800201

37.

What Works Clearinghouse. (2017). What works clearinghouse: Standards handbook, Version 4.0. Department of Education. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_standards_handbook_V4.pdf

38.

Wolfe

Pyle

Charlton

C. T.

Sabey

C. V.

Lund

E. M.

Ross

S. W.

(2016). A systematic review of the empirical support for check-in check-out. Journal of Positive Behavior Interventions, 18(2), 74–88. https://dx-doi-org.web.bisu.edu.cn/10.1177/1098300715595957