Using a Changing-Criterion Design to Evaluate the Effects of Check-In/Check-Out With Goal Modification

Abstract

Check-in/check-out (CICO) is a Tier 2 behavioral intervention that has demonstrated effectiveness for students with challenging behavior in a variety of educational settings. Existing research has focused primarily on testing the intervention’s effectiveness and the role of behavioral function in moderating response to intervention. Only a handful of studies have included examinations of different procedures for fading CICO to promote maintained behavioral change. These have included decreasing the frequency of teacher feedback, using a mystery motivator to thin the schedule of reinforcement, and increasing goal contingencies over time. In this study, two middle school females with conduct problems were identified for participation in CICO with goal modification. During intervention, the first modification entailed setting the CICO card goal for each participant based on averages obtained on the CICO card during baseline rather than the commonly used 80% goal. Then, predetermined decision rules about how to modify subsequent goals were executed according to student progress. Effects on student behavior were evaluated using a changing-criterion with withdrawal design, and scores on the CICO card improved and classroom problem behavior decreased for both participants. Limitations and recommendations for future research are discussed.

Keywords

for children with or at risk for emotional/behavioral disorders intervention(s)positive behavior support(s)challenging behavior(s)adolescent age group

As educators are called to serve the complex academic, behavioral, and social needs of students, many schools have adopted Positive Behavioral Interventions and Support (PBIS) models. Rather than typical “wait-to-fail” models of the past in which students were referred for special education evaluations and interventions following multiple disciplinary infractions, PBIS models are used to identify and support students prior to problems escalating to severe levels requiring resource-intensive supports. Traditionally, PBIS includes a three-tiered system including primary (Tier 1 or universal), secondary (Tier 2 or targeted), and tertiary (Tier 3 or specialized) levels of support (Hawken, Vincent, & Schumann, 2008). The supports offered at each level increase in intensity and individualization based on student need. Central to PBIS is the use of multiple data sources to drive decisions about placement, resource allocation, and student responsiveness (Marchant et al., 2009).

At the primary level of prevention, all students are taught behavioral expectations, have opportunities to practice, and are provided instructive feedback and reinforcement for meeting expectations (Lewis, Jones, Horner, & Sugai, 2010). Students who are nonresponsive to primary prevention efforts, as determined by data such as the number of office discipline referrals (ODRs), may go on to receive Tier 2 or Tier 3 interventions (Hawken et al., 2008). Supplemental to the primary plan, Tier 2 interventions are implemented similarly across groups of students who display comparable problems and thus are apt to benefit from the same intervention (Anderson & Borgmeier, 2010). Generally, this is approximately 15% of the school population. Some key features of Tier 2 include (a) using evidence-based interventions that are implemented with fidelity, (b) aligning these supports with core behavioral instruction in Tier 1, (c) ongoing progress monitoring, and (d) using data to guide intervention planning decisions such as programming for maintenance and adapting intervention according to student response (Mitchell, Bruhn, & Lewis, in press). To improve effectiveness and efficiency at Tier 2, it is important to identify specific information regarding behavioral intensity and deficit, and provide students Tier 2 interventions that confidently reflect their needs.

Two recent reviews of Tier 2 interventions conducted within multitiered prevention models revealed a variety of effective targeted supports for students with academic, behavioral, or social skills deficits (Bruhn, Lane, & Hirsch, 2014; Mitchell, Stormont, & Gage, 2011). Several different interventions were examined, including social skills instruction (e.g., Gresham, Van, & Cook, 2006), literacy training (e.g., Wills, Kamps, Abbott, Bannister, & Kaufman, 2010), writing instruction (e.g., Lane et al., 2011), self-management (e.g., Ness, Sohlberg, & Albin, 2011), and check-in/check-out (CICO) procedures (e.g., Fairbanks, Sugai, Guardino, & Lathrop, 2007) among others. The interventions most frequently occurring in the literature were variations of CICO (Bruhn et al., 2014; Mitchell et al., 2011).

CICO

CICO is grounded in basic behavioral principles such as clearly defined expectations, contingent reinforcement, increased prompting, frequent feedback, and positive social interactions (Hawken & Johnston, 2007). A student participating in CICO begins and ends each day by checking in with the CICO coordinator. The adult coordinator is responsible for multiple intervention components (e.g., checking the student in and out, setting behavioral goals, tracking data, providing feedback) and is often selected based on schedule flexibility and rapport with students (Hawken, 2006). When students check-in in the morning, the coordinator ensures students have all materials necessary for participating in school activities that day, including the CICO card (Hawken, 2006). The CICO card is designed to allow teachers to rate students on adherence to the three to five school-wide expectations established as part of a school’s Tier 1 plan. If the school is not implementing school-wide PBIS, then the expectations may be selected and agreed upon by the school’s student support team. Teachers complete the progress report by rating students on a Likert-type scale (e.g., 0 = not at all, 1 = some of the time, 2 = all of the time) and provide constructive feedback concerning student behavior at the end of a specified class or time period. Feedback may be in the form of praise (e.g., “Great job meeting your goal today!”) or correction (e.g., “You need to work a little harder on being responsible in class. That means you need to get to work immediately when you are given an assignment. You can do it!”). At the end of the day, the student returns to meet with the CICO coordinator to check-out. During check-out, the coordinator calculates points earned from teacher ratings for the day and delivers reinforcement (e.g., praise, tangibles) for meeting behavioral goals (e.g., 80% of points; Hawken, 2006). Finally, students are provided their CICO card to take home. The expectation is that students will share the report with their parents, who will sign it and have the student return it to school the following day. The CICO coordinator is in charge of tracking student data, which can be used to guide intervention decisions (Hawken & Johnston, 2007). Generally, teams analyze graphed data of daily points (or percentage of points) earned to determine whether students (a) can be faded from CICO support, (b) should continue participating in the intervention as designed, (c) need modifications to the original CICO plan, or (d) need more intense, tertiary support (Hawken & Johnston, 2007).

CICO has been evaluated using a variety of methodologies including single-subject, descriptive, quasi-experimental, and experimental designs. Across the literature base, CICO has been attributed to increases in academic engagement (Hawken & Horner, 2003) and points earned on the CICO card (Cheney et al., 2009), as well as decreases in ODR rates (McIntosh, Campbell, Carter, & Dickey, 2009) and problem behaviors (Campbell & Anderson, 2008). Yet, questions remain about for whom CICO is most effective, how to set behavioral goals for the CICO card, and how to adapt CICO to promote maintenance of improved behavior.

First, in terms of determining who could benefit from CICO, research has focused mostly on the role of function. There is conflicting evidence suggesting CICO may be more effective for students with attention-motivated behavior than escape-maintained behavior (Hawken, O’Neill, & MacLeod, 2011; March & Horner, 2002; McIntosh et al., 2009). To determine function, a functional behavior assessment (FBA), which is a time-consuming and resource-intensive process that generally involves teacher and student interviews, direct observation, and records reviews, must be conducted by someone with behavioral expertise (Hawken et al., 2011). Although McIntosh and colleagues (2009) conducted a brief assessment using the Functional Assessment Checklist for Teachers and Staff (FACTS; March & Horner, 2002) to determine the function of behavior and subsequently select Tier 2 interventions matched to function, replications are needed to ascertain the generality of this approach.

In addition to function potentially moderating response to CICO, it is possible the type of emotional and behavioral problems a student displays could influence responsiveness. For example, a student demonstrating emotional problems could respond differently to CICO than a student with conduct problems or hyperactivity. One tool that is free, easy to access online, relatively simple to complete, and provides information about a student’s area of emotional or behavioral risk is the Strengths and Difficulties Questionnaire (SDQ; Goodman, 1997). The SDQ is a 25-item, validated behavioral rating scale that can be completed by teachers, students, and parents. Scores on the SDQ indicate whether a student demonstrates risk for emotional difficulties, peer problems, conduct problems, hyperactivity/inattention, prosocial behavior, and total behavioral difficulties. Because of the efficiency with which the SDQ can be completed as well as the information derived from it, it may be a practical tool to match students to intervention (McDaniel, Bruhn, & Mitchell, 2015). One could reasonably think students at elevated risk for conduct problems as determined by the SDQ may respond well to CICO based on the fact that previous research studies of CICO have included many participants with disruptive, defiant, and noncompliant behaviors (Campbell & Anderson, 2008; Hawken & Horner, 2003; Todd, Campbell, Meyer, & Horner, 2008), which according to the SDQ are indicators of conduct problems.

A second understudied issue with CICO involves determining how to set the behavioral goal for percentage points on the CICO card. Conventional wisdom has been that the goal should start at 80% of possible CICO card points (Campbell & Anderson, 2011; Hawken et al., 2011). However, it is possible 80% may be too high initially for some students. Specifically, Crone, Hawken, and Horner (2010) suggested that the CICO coordinator modifies the CICO card goal if there is indication that the student will not meet the standard 80% criteria. Researchers have yet to examine using baseline scores on the CICO card to determine the initial goal. In addition, once the student reaches the goal, decisions need to be made about how to systematically fade or adapt intervention to maintain positive behavior.

Only a few studies have evaluated the effectiveness of adapting CICO over time, and all of them differed in their approach. In a study by Campbell and Anderson (2011) utilizing a withdrawal design followed by a component analysis, teacher feedback to students occurred 3 times per day at the beginning of the study and was eventually faded to students receiving no feedback during the component analysis phases. With gradual fading of feedback, students reduced their disruptive behaviors and improved their academic engagement, and their behavior change maintained over time.

Using an ABAB design, Miller, Dufrene, Sterling, Olmi, and Bachmayer (2014) implemented a fading procedure during the second B condition. Once students demonstrated a stable or decreasing trend in problem behavior, they introduced a “mystery motivator.” Specifically, the students received a slip of paper after meeting the daily goal that indicated whether or not a reward was available, which reduced reinforcement from a fixed to a variable schedule. Then, if a student responded well to this condition, teacher feedback was eliminated and students self-monitored by completing the CICO card themselves. Results were mixed as two of three students maintained behavior during fading to the mystery motivator, and only one of three moved to the self-monitoring condition, which did not prove to be effective in maintaining behavior.

Besides fading teacher feedback and reinforcement, another way to modify CICO over time is to gradually increase the CICO card goal required for the student to earn reinforcement. In a changing-criterion design study by Lane, Capizzi, Fisher, and Ennis (2012), point goals on the CICO card were raised incrementally for four students after they consistently met their goals. For example, one participant’s goals were changed from 60% to 70% to 75% to 80% to 85%, and then the intervention was discontinued. These procedures proved somewhat effective as evidenced by modest functional relations for three of four students; however, students did not maintain improvements in behavior when intervention stopped. In addition, because baseline data were not collected, it is difficult to determine how the initial daily point goal was established.

Purpose

The base of literature supporting the use of CICO in its traditional format and subsequent modified versions warrants continued extensions regarding how best to responsively adapt CICO to promote improvements in and maintenance of positive behavior. The purpose of modifying the goal-setting component of CICO was to maximize student outcomes by tailoring the intervention based on student baseline data and subsequent response or nonresponse to intervention. The primary research questions were as follows:

Research Question 1: For students identified with conduct problems, is there a functional relation between CICO with goal modifications and scores on the CICO card?

Research Question 2: Are changes in CICO card scores reflected in fewer problem behaviors in the classroom?

Method

Setting

This study was conducted in a middle school (Grades 6–8) that had approximately 349 students and 25 teachers. Of those students, 76.8% qualified for free and reduced lunch, an elevated percentage from the state average of 52.7%. Two years prior to the study, the school hired a new principal and adopted PBIS school-wide expectations; however, a ticket-based reinforcement system was marginally used. To assess Tier 1 fidelity, the School-Wide Evaluation Tool (SET; Sugai, Lewis-Palmer, Todd, & Horner, 2001) was conducted during the study with a result of 54% fidelity of school-wide PBIS implementation. The school-wide expectations were PROWL: Punctual, Respectful, Organized, Winning Attitude, and Learning. At the time of the study, no Tier 2 strategies or interventions were in place for students with challenging behavior. Furthermore, no identification process for students requiring Tier 2 supports was in place prior to the study.

Participants

A team of seventh-grade teachers and an administrator nominated seventh-grade students for the study if they had a history of problem behavior and had at least two ODRs prior to the study. Two ODRs were used as the criterion because that has been the generally accepted minimum number indicating the need for support beyond the Tier 1 plan (Horner, Sugai, Todd, & Lewis-Palmer, 2005). Four seventh-grade students who had at least two ODRs were nominated. Next, the four students’ homeroom teachers completed the SDQ for their respective student (see previous description of SDQ). Only students scoring in the “borderline” or “abnormal” range in the conduct problems domain were selected to participate in the intervention study of CICO. Of the four students meeting initial inclusion criteria, two scored in the abnormal range for conduct only, one scored in the abnormal range on four different domains as well as total difficulties, and one scored in the normal range in all domains but borderline on total difficulties. The two students scoring abnormal for conduct problems only were selected to participate. The student with the most significant problems according to the SDQ was referred for a more intensive intervention, whereas the student with a borderline score in total difficulties only was referred for a less intensive intervention.

Three school staff members served as CICO coordinators. Of these, two were teachers and one was the school counselor. The teachers were male and female, respectively, and the counselor was a female. The male teacher had taught history for 6 years. The female teacher had taught math for 3 years. The counselor had 27 years of experience. All three held undergraduate degrees. The two teachers were Caucasian and the school counselor was African American. The two teachers served as the primary CICO coordinators, and the counselor was trained as third coordinator in case one of the teachers was unavailable or absent to maintain consistency of treatment delivery. All three adult participants were trained in a 2-hr training session conducted by the research team during the baseline phase. In this training, the research team first described the purpose of CICO as a Tier 2 intervention. Second, the research team reviewed the critical components of CICO procedures, using the CICO card as the material to highlight how CICO components are integrated into the CICO card permanent product. A review sheet including the primary CICO steps was provided and left with the staff members. Next, the research team reviewed how to calculate CICO card percentages, how to fill out all sections of the CICO card, and how goal-setting procedures would be conducted. Then, the research team reviewed the complete process of CICO with CICO card in relation to the school schedule and exactly what time and location all of the components (e.g., check-in, teacher ratings, check-out) would occur. In this discussion, the research team explained explicitly the study design and described how phase changes would be determined and how phase changes would affect coordinator duties. The research team answered questions as they arose throughout the 2-hr session, and the final activity of the training session was to run through an example as a group of how CICO with CICO card procedures would be conducted.

Kaiya

The first student, Kaiya, was a 13-year-old African American female. She had not been identified with a disability or evaluated for special education. Her teachers reported she had an inconsistent home life with an unstable and transient mother. School personnel indicated Kaiya may experience mental health issues and, several times to no avail, suggested to her family she be evaluated formally. Kaiya had a history of behavioral issues at her middle school and averaged one ODR per month. Her infractions were classified as “Class 2” and “Class 3” offenses meaning that they were serious and resulted in in-school suspensions, out-of-school suspensions, and family conferences. In the current school year, she missed 17 days of school due to out-of-school suspensions. Her target behaviors, as identified by her teachers, included social problems, disrespect, and confronting authority. Specifically, Kaiya had a history of arguing with female peers, an inability to build lasting relationships with female peers due to high rates of teasing and disrespectful language toward them. Kaiya also demonstrated disrespectful and confrontational behavior toward adults at school including using inappropriate language toward teachers, inappropriately arguing with teachers and administrators, and refusing to comply with adult directions.

Tina

Tina was a 13-year-old African American female. She did not receive special education services, but had a history of disruptive behavior at school. Tina lived with her grandmother and, according to her teachers, had strained relationships with her siblings and care providers primarily due to a lack of respect for authority and rules. She earned an average of 0.5 ODRs per month, and all infractions were considered “Class 2” resulting in 4 days of in-school suspension and 1 day out-of-school suspension. Tina’s target behaviors, as identified by her teachers, included poor social skills, disruption, impulsivity, and physical aggression. Tina’s challenging social behavior emerged with both male and female peers and involved lying to teachers about peer behavior and demonstrating inappropriate language with peers. Tina displayed verbal disruptions in classroom instruction and when receiving adult feedback. She was impulsive in her reactions to peer and adult conflict and displayed physical aggression such as pushing and kicking students in conflict situations.

Measures

Dependent variables

Two dependent variables were used to measure intervention effectiveness. The primary dependent variable used to make phase change decisions (increased goal criteria) was the percentage of points earned on the CICO card. The CICO card is based on teachers’ daily direct observation of students’ behavior (Fairbanks et al., 2007). The standard CICO card format was individualized with the PROWL school-wide expectations. Students were rated 5 times each day by five different teachers on a Likert-type scale of 0 to 2 for each of the five expectations, resulting in a total of 50 possible points earned per day. On some days (usually due to scheduling or teacher-related demands), students may not have been rated for a class period or on a specific expectation. In these cases, the total possible was adjusted to match the number of class periods and expectations that were rated. Also, in some cases, teachers chose to circle more than one number, indicating that the student’s performance was between two Likert-type scale ratings (e.g., if “1” and “2” were circled the rating was considered “1.5”). Although teachers were given a rubric of how to score student behaviors (see Table 1), they were not aware of the students’ CICO card percentage goals. CICO card percentages were calculated by dividing the total points earned by the points possible and multiplying that quotient by 100.

Table 1.

PROWL Rubric for DPR.

Score on DPR
Expectations	0	1	2
Punctual	More than 2 min late	1–2 min late	Not late
Respectful	More than 2 instances of disrespect	Some disrespect	Respects all peers and adults
Organized	Missing all work and/or materials	Missing some work and/or materials	Has work and materials
Winning attitude	Negative, lacking effort	Somewhat positive and persistent	Positive and persistent
Learning	Did not make improvement	Made some improvement	Made improvements in learning and behavior

Note. PROWL = Punctual, Respectful, Organized, Winning Attitude, and Learning; DPR = Daily Progress Report.

While the CICO card served as the primary dependent variable, researchers used direct observation of problem behavior as a secondary dependent measure collected periodically, but not continuously. All direct observations of behavior occurred during participants’ third period class; however, Kaiya and Tina were not in the same class. In each phase, the students were observed a minimum of 1 time by researchers (with the exception of the 40% goal phase for Kaiya, see Figure 1). The direct observation procedure involved a 30-min observation in which researchers used partial interval recording at 15-s intervals. For a total of 120 intervals, researchers indicated “yes” if the problem behavior occurred at any time during the 15-s interval. Problem behavior was operationally defined as any behavior not related to the academic task such as the student being off-task or disruptive to peers or the teacher. Examples included talking out, being disrespectful, refusing to complete the assigned task, leaving assigned area, and arguing. To calculate the percentage of problem behavior, the total number of intervals in which problem behavior occurred were totaled and divided by 120. To ensure reliability of the direct observation, two observers collected data simultaneously, but independently on both participants for a total of 29% of all direct observations distributed across conditions. Interobserver agreement (IOA) was calculated using the point-by-point method and averaged across all observations (Cooper, Heron, & Heward, 2007).

Figure 1.

Kaiya’s behavior.

Conditions

Baseline

The purpose of the baseline phase in this study was twofold: (a) to establish a standard of problem behavior prior to introducing the intervention and (b) to determine appropriate initial CICO card goals for each student in the intervention phase. During baseline, both dependent variables (i.e., CICO card and percentage of problem behavior) were recorded. Teachers were instructed to use the CICO card to rate Kaiya’s and Tina’s behavior unbeknownst to the students. They were not allowed to provide students feedback regarding their ratings. Also, there was no morning check-in or afternoon check-out for students. Kaiya and Tina were not receiving other behavioral interventions or supports. Procedures when intervention was withdrawn were the same as baseline conditions, so that students did not experience any CICO components.

Intervention

The independent variable in this study was CICO with a modified goal-setting procedure. Traditionally, CICO has students start with a goal of 80% of earned points on the CICO card (Campbell & Anderson, 2011; Hawken et al., 2011). Whereas 80% is a common criterion for goal mastery, in the current study we examined the use of modified mastery criteria that was individualized to student baseline data and response or nonresponse throughout intervention. The goal for the first intervention phase was based on the percentage of points earned on three CICO cards in the baseline phase. Coincidently, the two CICO coordinators rated the participants similarly during baseline, which resulted in the same initial goal for both participants. The initial modified goal was determined through averaging the baseline phase CICO card percentages, which was 40%, and the initial goal was equal to this to allow students to experience early initial success reaching their goals. The predetermined modified goal-setting procedure for the remaining intervention called for an average of phase CICO card percentage points, plus an additional 10%, which was either rounded up or down through collaboration between researchers and CICO coordinators and the participants. For instance, when the first intervention phase average was 68.7%, the goal was increased to 75% through discussion with consideration to this goal increase being the first within the intervention. In discussion of goal modification, the researchers, CICO coordinators, and participants focused primarily on setting attainable goals that would promote increased achievement, reinforce improved outcomes, and decrease the likelihood for frustration from not reaching unattainable goals. Prior to intervention, a decision rule was established stating that if a student did not reach her goal at least 1 time in the first week of having that goal, a new goal 10% less than the existing goal would be assigned. However, this decision rule was not necessary as students responded within 1 week for each new goal criterion determined by the researchers, CICO coordinators, and participants.

In this intervention, the students and CICO coordinators followed the traditional components of check-in procedures (e.g., morning meeting, give CICO card, review daily goal) and check-out procedures (e.g., afternoon meeting, review CICO card, record scores and percentage, sign CICO card) with the exception that the CICO cards were not sent home for parental signature and return due to a history of lack of communication and responsiveness with the school. The CICO coordinators were confident that the CICO cards would either not make it home or not be returned to school; thus, this component was eliminated. Also, to ensure teachers completing the CICO card did not alter their ratings based on the students’ goals, the teachers were unaware of the students’ CICO card goals. When students and their coordinator met in the afternoon to review their daily score, percentage, and goal, they were provided with verbal feedback including positive reinforcement in the form of specific praise when the goal was met. Behavior-specific praise was used as the primary positive reinforcement strategy for meeting goal criteria because the classroom teachers did not want to use tangible reinforcers, and participants were already earning PROWL tickets for displaying expected behaviors in the classroom as a part of the school-wide PBIS framework. When the daily goal was not met, the CICO coordinators provided corrective feedback regarding specific ratings throughout the day that were low.

Treatment integrity

To ensure treatment integrity to CICO (minus the family component), researchers conducted a 2-hr training session for the three CICO coordinators. Training procedures followed the CICO manual (Everett, Sugai, Fallon, Simonsen, & O’Keeffe, 2011). Training components included (a) description of the intervention procedures, (b) examples and nonexamples of correct CICO interactions, and (c) role-playing through all intervention procedures. Coordinators were given handouts of guidelines, procedures, and expectations for communicating with researchers. The two students also participated in a brief training session regarding CICO procedures and the CICO card at the conclusion of the baseline phase. In this session conducted by the CICO coordinators, students were given descriptive information outlining all of the components of check-in and check-out with the coordinator. In addition, a detailed description and model of the CICO cards were provided.

Researchers conducted direct observations of CICO procedures on 21.4% of all CICO interactions in the intervention phases using a checklist adapted from the CICO training manual (Everett et al., 2011). Components expected during the interactions were rated on a 2-point Likert-type scale (2 = in place, 1 = partial, 0 = not in place). The total number earned was divided by the total number possible (i.e., 8 for check-in, 10 for check-out) to calculate the percentage of treatment integrity to the CICO procedures discussed in the training session. The mean treatment integrity was 98% across sessions observed. In addition, the CICO cards were inspected for completion as a form of permanent product confirmation of CICO card procedures. Ninety-nine percent of all CICO cards were completed fully for both check-in and check-out procedures and teacher ratings.

Experimental Design and Analysis

To determine whether there was a functional relation between CICO with goal modifications and points earned on the CICO card, a changing-criterion with withdrawal design was used. The addition of the withdrawal condition was used to strengthen the rigor of the design, and thus, the conclusions that could be drawn (Kennedy, 2005). Although direct observation data were collected, these data served as a secondary dependent measure and were not used to guide phase change decisions. Rather, these decisions were guided by CICO card data. First, baseline data were used to establish the initial criterion level, which was 40% of possible points on the CICO card. The team consisting of the CICO coordinator and the primary investigator selected 40% as the goal based on the average for both participants of the baseline CICO card percentages for students to experience early initial success. After students demonstrated a stable pattern of behavior or an increasing trend in behavior, the criterion for the next condition was raised based on the mean during the previous condition and a phase change occurred. The criterion set for each subsequent condition was determined by the team who had agreed it should be at or near 10% higher than the mean in the previous phase to remain attainable, and phase changes were made based on student performance as judged by stability, level, and trends in the CICO card data. As previously described, the first goal increased from 40% to 75%, because during the 40% condition, both girls averaged near 65%, and so approximately 10% was added to the average to establish the next goal of 75%. In the next modification, both girls averaged a little above 75%, so the goal was increased to 85%. After three separate goal modifications (40%, 75%, 85%), CICO was withdrawn such that this phase was equivalent to baseline. That is, participants were still rated on the CICO card, but had no contact with the CICO card, as teachers completed these ratings unbeknownst to Kaiya and Tina. All other components (e.g., check in with a mentor) were completely withdrawn. When CICO card data demonstrated a reversal in behavior, CICO was reinstated at the last goal level (85%). In addition to visually analyzing the data, means, standard deviations, and ranges were calculated for each phase. Pearson correlations (r) between CICO card scores and percent of problem behavior also were calculated.

Results

Results for each phase are presented for each individual participant (see Figures 1 and 2 for graphed data, and Table 2 for means, standard deviations, and ranges obtained on the CICO Card).

Figure 2.

Tina’s behavior.

Table 2.

Descriptive Statistics for Percentage Earned on DPR.

	Kaiya			Tina
Condition	M (%)	SD	Range	M (%)	SD	Range
Baseline	40.33	6.66	33–46	40.33	6.66	33–46
40% goal	68.67	6.42	64–76	62.33	6.80	57–70
75% goal	78.00	6.73	76–86	79.33	1.15	78–80
85% goal	82.50	12.15	72–100	86.20	1.79	84–88
Withdrawal	71.67	9.50	62–81	64.33	4.73	59–68
85% goal	99.67	0.58	99–100	97.33	4.62	92–100

Note. DPR = Daily Progress Report.

Kaiya

During baseline conditions in which teachers completed the CICO card unbeknownst to the participants, Kaiya earned an average of 40.33% of CICO card points. Direct observation indicated she was involved in problem behavior about 33% of intervals, on average. Next, CICO was implemented with Kaiya’s CICO card goal set at 40%. She far exceeded that goal as she averaged 68.67%, and her data showed a clear increasing trend. After 3 consecutive days above 40%, the goal was increased to 75% according to the predetermined decision rules. Kaiya exceeded this goal 3 out of 4 days and averaged 78% on the CICO card. During the only day of direct observation in this phase, she exhibited no problem behavior (i.e., 0%). The goal was then increased to 85%. Kaiya hovered slightly below that, but then showed an increasing trend and reached the ceiling of 100% on the CICO card, at which time the intervention was withdrawn. Problem behavior occurred in only 1% of intervals. During withdrawal, there was an instant change in level and a decreasing trend with her average CICO card score dropping to 71.67%, which was above baseline levels, but not as high as intervention levels. Her problem behavior increased to 19%. Intervention was reinstated at the 85% criterion, and she far exceeded that by averaging 99.67% on the CICO card and exhibiting high, stable scores. Her problem behavior returned to 0%. IOA for directly observed problem behavior was 100%. The correlation between CICO card and problem behavior was r(4) = − .94, p = .00.

Tina

Tina’s baseline CICO card averaged 40.33%, and her problem behavior averaged 35.5% of intervals. Like Kaiya, her first intervention goal was set at 40%, and she responded positively as her data indicated an immediate change in level. Her CICO card average was 62.33%, and her problem behavior average was 32% of intervals. The team then raised the goal to 75%, and again, there was an immediate change in level with Tina averaging 79.33% across 3 days of intervention. Direct observation indicated Tina was engaging in problem behavior just slightly less than in the previous condition (i.e., 29% of intervals). When the goal was raised to 85%, again Tina’s behavior improved consistently (as evidenced by stability) with scores on the CICO card averaging 86.2% and her problem behavior gradually decreasing to an average of 16.5%. The intervention was withdrawn, and Tina’s scores on the CICO card immediately declined as evidenced by the change in level. Her average CICO card during withdrawal was 64.33%, and problem behavior increased to 32%. Finally, the intervention was reinstated at the 85% goal. This resulted in an instant and robust level change with an average of 97.33% on the CICO card. Her problem behavior decreased to 24%. Like Kaiya, IOA for problem behavior was 100%. The correlation between CICO card and problem behavior was r(6) = −.68, p = .06.

Discussion

Students with conduct problems often display disruptive and defiant behaviors, which can not only limit their academic engagement in class but also disturb the overall classroom environment (Kauffman & Landrum, 2009). To help these students be successful in schools, evidence-based interventions implemented with integrity are needed. Particularly within PBIS models, if these students respond well to targeted, Tier 2 supports, then costly and intensive Tier 3 interventions can be reserved for students with the most severe issues. Many schools implementing PBIS often rely on CICO as the “go to” intervention for students who are nonresponsive to the primary plan. Yet, there are still questions about whether or not CICO is effective for all students needing targeted supports. One issue in the CICO literature is how to adapt the intervention according to students’ response. Various techniques for adapting CICO over time to increase the likelihood of maintained improvements in behavior have been examined, including using a mystery motivator to reduce the frequency of reinforcement (Miller et al., 2014), reducing teacher feedback (Campbell & Anderson, 2011), and gradually increasing behavioral goals on the CICO card (Lane et al., 2012). To extend this area of inquiry, we followed a somewhat similar process as Lane and colleagues with respect to increasing CICO card goals over time in a changing-criterion design. Different from that study, however, was the use of a baseline phase to establish the first CICO card goal as well as a withdrawal phase to strengthen the design (Kennedy, 2005).

Findings

The purpose of this study was to determine whether there was a functional relation between CICO with goal modifications and CICO card scores for two seventh-grade female students with conduct problems. The modification to CICO required determining appropriate goal levels based on CICO card scores in baseline and in subsequent phases. This was a simple adaptation to the traditional 80% goal criterion. The purpose of setting a goal was to increase motivation while requiring students to attend to their behavior to progress toward the expected outcome. In this study, the changing goal was adapted to student response and increased incrementally based on predetermined decision rules created and implemented by the team of the CICO coordinator and researcher. In addition, a decision rule was put in place for students who were nonresponsive to the determined goal, although this decision rule was not needed given the positive response both participants displayed. Results showed that both participants improved CICO card scores from baseline to the first intervention phase with a 40% goal and each subsequent phase with higher goals. Both participants’ behavior regressed during withdrawal and improved substantially when CICO was reintroduced. Whereas these multiple demonstrations of effect indicated a functional relation was established between CICO with goal modifications and CICO card scores for Tina, Kaiya’s results were not quite as lucid. Specifically, Kaiya’s CICO card scores were more variable during each phase change and not at or above the goal criterion for all sessions as they were for Tina. However, there was still a functional relation between the overall CICO intervention and her CICO card data. To confirm these results, decreases in problem behavior were observed directly in the classroom for both students. The correlation between CICO card points and directly observed problem behavior for Kaiya was highly and negatively significant at p < .01, whereas the correlation between Tina’s CICO card and problem behavior was moderately and negatively significant at p < .10. Furthermore, because the intervention was implemented with fidelity and procedures associated with the CICO card (e.g., teacher ratings, modified goal setting) were followed, this study revealed that CICO card scores may be a feasible alternative to direct observation for tracking student behavior and making data-based decisions about student responsiveness for students in CICO. This is an important finding given the time and expertise required for direct observation, which is an accurate yet cumbersome way to monitor behavioral progress. Although we attempted to assess the teachers’ and coordinators’ perceptions about the intervention and CICO card use, we did not receive the electronically distributed forms back and, thus, are unable to report social validity of findings. We speculate the lack of completed and returned forms was due to the forms being distributed at the end of the school year.

Second, to determine who was eligible for CICO, teachers completed the SDQ, and students scoring in the abnormal range for conduct problems were selected. It is important to note that both of these participants with documented histories of problem behavior, conduct specifically, as noted by the SDQ displayed pronounced positive response to CICO with the modified goal component. Modifying goals within CICO may have further contributed to the positive behavior change, as goal setting is an effective strategy to help students, particularly with conduct problems, to self-manage their behavior. This application addresses the issue of using a “one size fits all” approach to Tier 2 intervention (Stormont & Reinke, 2013). Instead of assigning all students who require Tier 2 behavioral interventions one evidence-based intervention such as CICO, Tier 2 interventions should target specific behavioral deficits (Newcomer, Freeman, & Barrett, 2013). A comprehensive Tier 2 framework should involve assessing students for deficit-specific information, matching behavioral deficits to appropriate Tier 2 interventions with corresponding intervention intensity and dosage, monitoring progress, and modifying based on responsiveness (McDaniel et al., 2015).

Limitations and Future Directions

The results of this study indicated positive behavioral change for both participants, yet results should be interpreted with the following limitations and future directions in mind. First, the school was implementing Tier 1 at only 54%. CICO is heavily tied to universal instruction and, specifically, school-wide expectations. Although both participants improved behavior during CICO, it is possible that had the students had better and more frequent exposure to the Tier 1 plan, such as being taught expectations and reinforced for meeting them, CICO may not have been necessary.

Second, the CICO card was used as the primary dependent variable and may not be as sensitive to behavioral change as direct observation. Although direct observation may more accurately depict behavior, it was used only as a secondary dependent measure. Results showed a parallel, yet inverse, pattern between CICO card and direct observation scores of behavior, indicating that, perhaps, the CICO card accurately reflected classroom behavior. In other words, the negative correlations indicated that as CICO card levels increased, classroom problem behavior decreased. However, in this study, direct observation was only conducted 1 to 2 times each phase for just 30 min of the school day and was not included in the 40% goal phase for Kaiya. Furthermore, although Tina’s problem behavior improved, it still occurred during 24% of intervals. Future research should examine the utility of CICO cards for informing phase changes, determining response to intervention, and programming for maintenance. Also, as in this study, some research has documented a moderate to high correlation between direct observation and behavior ratings (Riley-Tillman, Chafouleas, Sassu, Chanese, & Glazer, 2008). Further research is needed to explore the statistical relation between the CICO card and direct observation particularly in light of the small number of direct observations included in this study.

Third, the coordinators and researchers predetermined decision rules for changing goals based on student performance; yet when Kaiya was in the 85% goal phase, she did not achieve 85% more than 1 time. However, at the end of this phase, she reached the ceiling of 100%. This was followed by withdrawal of intervention and her behavior declining and then subsequent reinstatement of the intervention with her behavior increasing to at or around the ceiling. Further research may involve determining how long it is necessary for a student to stay at a certain goal level prior to increasing or decreasing the goal, as well as collecting maintenance data once a student has demonstrated a high and stable level of performance. Unfortunately, because the school year ended, a maintenance phase was not possible. Along those same lines, the magnitude of intervention effects for Kaiya is tempered by the fact that it is unclear whether the specific component of changing the CICO card goal resulted in improved behavior. It is possible that Kaiya would have improved her behavior during CICO without modifying the CICO card goal. However, research has shown that setting goals and monitoring progress toward those goals are an effective way to change behavior.

Fourth, in addition to extending the research of modifying goal components within CICO, additional avenues exist for CICO research. For instance, the current study is related to modifying intervention over time by increasing CICO card goals, which changed the contingencies for reinforcement. In this case, as the students demonstrated an acceptable level of positive behavior, gradually higher CICO card scores were required for students to receive reinforcement (i.e., praise). Essentially, this served to thin the schedule of reinforcement by requiring consistently high levels of positive behavior. Previous research has examined the use of fading teacher feedback and reinforcement, utilizing a “Mystery Motivator,” and increasing daily CICO goals. Future research should continue to examine not only how but also when to adapt support to promote maintenance and generalization. This requires knowledge and testing of appropriate decision rules applied to the intervention process.

Footnotes

Action Editor: Lee Kern

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Anderson

C. M.

Borgmeier

(2010). Tier II interventions within the framework of school-wide positive behavior support: Essential features for design, implementation, and maintenance. Behavior Analysis in Practice, 3(1), 33–45.

Bruhn

A. L.

Lane

K. L.

Hirsch

S. E.

(2014). A review of Tier 2 interventions conducted within multi-tiered models of prevention evidencing a primary behavioral plan. Journal of Emotional and Behavioral Disorders, 22, 171–189.

Campbell

Anderson

C. M.

(2008). Enhancing effects of check-in/check-out with function-based support. Behavioral Disorders, 33, 233–245.

Campbell

Anderson

C. M.

(2011). Check-in/check-out: A systematic evaluation and component analysis. Journal of Applied Behavior Analysis, 44, 315–326.

Cheney

Stage

Hawken

Lynass

Mielenz

Waugh

(2009). A two-year outcome study of the check, connect, and expect intervention for students at-risk of severe behavior problems. Journal of Emotional and Behavioral Disorders, 17, 226–243.

Cooper

J. O.

Heron

T. E.

Heward

W. L.

(2007). Applied behavior analysis (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall.

Crone

Hawken

Horner

(2010). The Guilford Practical Intervention in the Schools Series. Responding to problem behavior in schools, second edition: The behavior education program. New York, NY: Guilford Press.

Everett

Sugai

Fallon

Simonsen

O’Keeffe

(2011). School-wide Tier II interventions: Check-in check-out getting started workbook. Storrs, CT: OSEP Center on Positive Behavioral Interventions and Supports Center for Behavioral Education and Research.

Fairbanks

Sugai

Guardino

Lathrop

(2007). Response to intervention: Examining classroom behavior support in second grade. Exceptional Children, 73, 288–310.

10.

Goodman

(1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry and Allied Disciplines, 38, 581–586.

11.

Gresham

F. M.

Van

M. B.

Cook

C. R.

(2006). Social skills training for teaching replacement behaviors: Remediating acquisition deficits in at-risk students. Behavioral Disorders, 31, 363–377.

12.

Hawken

L. S.

(2006). School psychologists as leaders in the implementation of a targeted intervention: The behavior education program (BEP). School Psychology Quarterly, 21, 91–111.

13.

Hawken

L. S.

Horner

R. H.

(2003). Evaluation of a targeted intervention within a schoolwide system of behavior support. Journal of Behavioral Education, 12, 225–240.

14.

Hawken

L. S.

Johnston

S. S.

(2007). Preventing severe problem behavior in young children: The behavior education program. Journal of Early and Intensive Behavior Intervention, 4, 599–613.

15.

Hawken

L. S.

O’Neill

R. E.

MacLeod

K. S.

(2011). An investigation of the impact of function of problem behavior on effectiveness of the behavior education program (BEP). Education and Treatment of Children, 34, 551–574.

16.

Hawken

L. S.

Vincent

C. G.

Schumann

(2008). Response to intervention for social behavior: Challenges and opportunities. Journal of Emotional and Behavioral Disorders, 16, 213–225.

17.

Horner

R. H.

Sugai

Todd

A. W.

Lewis-Palmer

(2005). School-wide positive behavior support. In Bambara

Kern

(Eds.), Individualized supports for students with behaviors: Designing positive behavior plans (pp. 359–390). New York, NY: Guilford Press.

18.

Kauffman

J. M.

Landrum

(2009). Characteristics of emotional and behavioral disorders of children and youth (8th ed.). Columbus, OH: Merrill.

19.

Kennedy

C. H.

(2005). Single-case designs for educational research. Boston, MA: Allyn & Bacon.

20.

Lane

K. L.

Capizzi

A. M.

Fisher

M. H.

Ennis

R. P.

(2012). Secondary prevention efforts at the middle school level: An application of the behavior education program. Education and Treatment of Children, 35, 51–90.

21.

Lane

K. L.

Harris

K. R.

Graham

Driscoll

Sandmel

Morphy

. . . Schatschneider

(2011). Self-regulated strategy development at Tier 2 for second-grade students with writing and behavioral difficulties: A randomized controlled trial. Journal of Research on Educational Effectiveness, 4, 322–353. doi:10.1080/19345747.2011558987

22.

Lewis

T. J.

Jones

S. E.

Horner

R. H.

Sugai

(2010). School-wide positive behavior support and students with emotional/behavioral disorders: Implications for prevention, identification and intervention. Exceptionality, 18, 82–93.

23.

March

R. E.

Horner

R. H.

(2002). Feasibility and contributions of functional behavioral assessment in schools. Journal of Emotional and Behavioral Disorders, 10, 158–170.

24.

Marchant

Anderson

D. H.

Caldarella

Fisher

Young

B. J.

Young

K. R.

(2009). Schoolwide screening and programs of positive behavior support: Informing universal interventions. Preventing School Failure, 52, 131–444.

25.

McDaniel

S. C.

Bruhn

A. L.

Mitchell

(2015). Tier 2: A framework for identification and intervention. Beyond Behavior, 24(2), 10–17.

26.

McIntosh

Campbell

A. L.

Carter

D. R.

Dickey

C. R.

(2009). Differential effects of a tier two behavior intervention based on function of problem behavior. Journal of Positive Behavior Interventions, 11, 82–93.

27.

Miller

L. M.

Dufrene

B. A.

Sterling

H. E.

Olmi

D. J.

Bachmayer

(2014). The effects of check-in/check-out on problem behavior and academic engagement in elementary school students. Journal of Positive Behavior Interventions, 17, 28-38.

28.

Mitchell

B. S.

Bruhn

A. L.

Lewis

T. J.

(in press). Essential features of Tier 2 & 3 school-wide positive behavioral supports. In Jimerson

S. R.

Burns

M. K.

VanDerHeyden

A. M.

(Eds.), Handbook of response to intervention: The science and practice of assessment and intervention (2nd ed.). New York, NY: Springer.

29.

Mitchell

B. S.

Stormont

Gage

N. A.

(2011). Tier two interventions implemented within the context of a tiered prevention framework. Behavioral Disorders, 36, 241–261.

30.

Ness

B. M.

Sohlberg

M. M.

Albin

R. W.

(2011). Evaluation of a second-tier classroom-based assignment completion strategy for middle school students in a resource context. Remedial and Special Education, 32, 406–416.

31.

Newcomer

L. L.

Freeman

Barrett

(2013). Essential systems for sustainable implementation of Tier 2 supports. Journal of Applied School Psychology, 29, 126–147.

32.

Riley-Tillman

T. C.

Chafouleas

S. M.

Sassu

K. A.

Chanese

J. A. M.

Glazer

A. D.

(2008). Examining the agreement of direct behavior ratings and systematic direct observation for on-task and disruptive behavior. Journal of Positive Behavior Interventions, 10, 136–143. doi:10.1177/1098300707312542

33.

Stormont

Reinke

W. M.

(2013). Implementing Tier 2 social behavioral interventions: Current issues, challenges, and promising approaches. Journal of Applied School Psychology, 29, 121–125.

34.

Sugai

Lewis-Palmer

Todd

Horner

R. H.

(2001). School-wide evaluation tool. Eugene: University of Oregon Press.

35.

Todd

A. W.

Campbell

A. L.

Meyer

G. G.

Horner

R. H.

(2008). The effects of a targeted intervention to reduce problem behaviors: Elementary school implementation of check in-check out. Journal of Positive Behavior Interventions, 10, 46–55.

36.

Wills

Kamps

Abbott

Bannister

Kaufman

(2010). Classroom observations and effects of reading interventions for students at risk for emotional and behavioral disorders. Behavioral Disorders, 35, 103–119.