Abstract
Maladaptive adolescent behavior patterns often create escalating conflict with adults and peers, leading to poor long-term social trajectories. To address this, school-based behavior management often consists of contingent reinforcement for appropriate behavior, behavior reduction procedures, and placement in self-contained or alternative settings. Yet, these commonplace practices may not foster the self-regulation processes necessary to override the habitual and negative response sequences that prohibit independent and sustained positive social functioning. As such, we developed I Control, a curriculum to teach middle school students with significant behavior problems how to engage in appropriate social self-regulation. Pre–post pilot data analyses using Mplus from 152 students in 14 schools/17 classrooms indicated that students taught I Control evidenced more positive scores than controls on teacher-reported contextualized executive function, externalizing behavior problems, and general problem behavior, and student-reported emotional control, social problem solving, and externalizing and internalizing behavior problems. Also, students who were taught the curriculum had greater curricular knowledge than control students. These positive findings indicate that I Control warrants more extensive investigation.
Keywords
Students who receive special education services for emotional and behavioral disorders (EBD) are generally less engaged in school activities than typical peers and as a result, struggle to succeed both academically and behaviorally. According to Lane, Wehby, and Barton-Arwood (2005), these students cause interruptions in their classrooms and schools and have more discipline-related referrals. Students with EBD drop out of school at twice the rate of peers without disabilities (55%) and spend large amounts of time suspended from school (Bradley, Doolittle, & Bartolotta, 2008). The result of their maladaptive behavior negatively affects their own academic outcomes, as well as those of their schoolmates (Nelson, Benner, Lane, & Smith, 2004; Trout, Nordness, Pierce, & Epstein, 2003). Long-term outcomes are also disheartening, as students with EBD have higher rates of post-school unemployment (Zigmond, 2006) with approximately 75% having contact with the criminal justice system during their lifetime (Newman, Wagner, Cameto, & Knokey, 2009; Newman et al., 2011).
Over the years, researchers have established that students with EBD typically lack the skills necessary for positive social and emotional functioning (e.g., Kauffman & Landrum, 2013; Kavale, Mathur, & Mostert, 2004). They often have difficulty understanding complex social situations and are less able to interact effectively (Gresham, Van, & Cook, 2006), maintain drive and motivation (Kauffman & Landrum, 2013), and recognize and/or manage their emotional reactions to stressful and anger-provoking situations (Zeman, Cassano, Perry-Parrish, & Stegall, 2006). In addition, compared with typical peers, students with significant behavior problems are less able to use working memory (temporary storage and manipulation of information) to interpret social cues accurately and are thus more likely to attribute hostile intentions to a peer’s ambiguous actions (Crick & Dodge, 1994; Dodge & Godwin, 2013). They are also less able to access schemata effective for navigating difficult situations, tend to make inaccurate inferences about what others might be thinking, focus on hostile or negative cues, and generate fewer prosocial solutions than typical peers. Even when they are able to access solutions, deficits in the ability to be flexible in their thinking and shift attention (cognitive shift) and to stop automatic responses (inhibitory control) can cause students to act impulsively or perseverate on antisocial responses, resulting in poor solution selection (Ellis, Weiss, & Lochman, 2009; Fairchild et al., 2009). Recently, researchers are discovering that these deficits are linked to the poor development of self-regulatory functioning and neurocognitive mechanisms collectively known as executive functioning (e.g., Blair & Raver, 2015; Bridgett, Oddi, Laake, Murdock, & Bachmann, 2013).
Self-Regulation (SR) and Executive Function (EF)
According to Polsgrove and Smith (2004), the ultimate instructional goal for students at risk for or who exhibit EBD is to learn the requisite skills necessary to regulate their own behavior. These self-regulatory skills involve the ability to control thoughts, emotions, and actions intentionally (Blair & Diamond, 2008). When students have greater self-regulatory control, they can more effectively interpret, manage, and monitor their behavior and emotions in social situations and set and achieve personal goals. These processes become increasingly important as students approach adolescence, rely less on adult supervision, and are subjected to pressure from peers (Lerner & Steinberg, 2004). Moreover, students with significant behavior problems who gain self-regulatory skills can increase their choice-making abilities, self-determination, and self-esteem (Cobb, Lehmann, Newman-Gonchar, & Morgen, 2009; Wehmeyer, Agran, & Hughes, 2000).
By almost any measure, the ability to self-regulate behavior is critical to developing and maintaining positive social relations. There is growing consensus in neuroscience that self-regulatory processes are dependent in part on the development of EF (e.g., Heatherton & Wagner, 2011; Hofmann, Schmeichel, & Baddeley, 2012), which is defined as the active manipulation, sequencing, and monitoring of information for the purpose of producing goal-oriented actions during novel situations (e.g., Diamond, 2013; Miyake et al., 2000; Pennington & Ozonoff, 1996). These processes influence SR in domains such as goal setting (including goal commitment and goal progress assessment), social problem solving (SPS), and emotion regulation (Carver & Scheier, 1998; Zelazo & Cunningham, 2007) and play an important role in the healthy integration of cognition and emotions (Blair & Diamond, 2008). Individuals with proficient EF are able to set goals competently, plan activities, and monitor their performance, and thus to self-regulate successfully (Rueda, Posner, & Rothbart, 2005; Zelazo, Carlson, & Kesek, 2008). More precisely, when individuals regulate their own emotions and behavior by engaging in deliberate, goal-directed problem solving, they are effectively recruiting EF-related processes. Conversely, EF deficiencies can affect a person’s self-regulatory functioning and contribute to social-emotional and behavioral difficulties (Hughes, 2011; Olson, Sameroff, Kerr, Lopez, & Wellman, 2005). As such, SR skills that contribute to effective goal setting, emotion regulation, and SPS play an important role in social-emotional functioning. Thus, what is known about SR can guide school-based intervention efforts to assuage the effects of oppositional, socially inappropriate, and/or antisocial behavior.
Over the years, researchers have developed universal social-emotional programming such as Promoting Alternative Thinking Strategies (PATHS; Conduct Problems Prevention Research Group, 2010; Riggs, Greenberg, Kusché, & Pentz, 2006) and Tools for Getting Along (Daunic et al., 2012; Smith et al., 2014), along with the more targeted Coping Power program (e.g., Lochman et al., 2009; Lochman & Wells, 2002), to improve students’ self-regulatory skills. Yet few, if any, current evidenced-based SR intervention efforts provide substantive, comprehensive, and intensive instruction necessary for students who exhibit the most significant social-emotional and behavioral needs. Thus, there is a pressing need for intensive, evidenced-based SR programming for students with significant behavior problems. We agree with Blair and Diamond (2008) who argue that SR interventions should be an important research focus.
In this article, we explore preliminary outcomes from a pilot study of I Control, an intervention designed to foster SR and improve social-emotional outcomes for middle school students who exhibit significant emotional and behavioral problems. I Control is a yearlong program with four units of instruction in SR (i.e., introduction to EF, goal setting, emotion regulation, SPS) and computerized games to directly train EF skills. Delivered twice a week, I Control provides direct instruction of SR-related skills, teacher modeling, and guided and independent practice through activities that promote generalization of learned skills. Our research hypotheses were that I Control would reduce emotional and behavioral difficulties by improving (a) teacher-rated general behavioral problems, EF-related skills, and social skills; (b) student self-ratings of general behavior problems, along with goal setting, emotion regulation, and SPS; and (c) curriculum-related knowledge about goal setting, emotion regulation, and SPS as a check on treatment fidelity.
Method
Setting and Participants
Recruitment
We contacted special education coordinators in multiple school districts across North Florida to assess potential interest in pilot study participation, targeting middle schools with self-contained EBD classes and alternative settings for students with significant behavioral problems (i.e., separate schools/centers, alternative schools). Project personnel met with principals and special education teachers who expressed interest to inform them about the project and confirm their participation. All recruitment and study procedures met university Institutional Review Board standards.
Random assignment and sample characteristics
Seventeen teachers located in four school districts in North Florida agreed to participate in the study. All classrooms were self-contained and focused on remediation for students with emotional and/or behavioral challenges. Of the 15 participating schools, seven (47%) were designated as Title 1 schools, and four (30%) were alternative schools serving students with significant behavioral problems. We met directly with each to explain our research design and when random assignment to the I Control or control condition (business as usual) would occur. Teachers were then randomly assigned to treatment or control, and those assigned to the control condition were offered the curriculum and training after the conclusion of the study. The resulting sample consisted of 167 student participants in 17 classrooms (nine intervention, eight control). Of the 167 students, we obtained consent from 152 (91% total; 87% treatment, 96% control). Demographic data are shown in Table 1.
Student Gender, Race, Grade Level, and Free/Reduced Lunch Status for Two Conditions.
Note. Percentages are within parentheses. Tx = Treatment; Ctrl = Control; M = Male; F = Female.
For grade level eighth to ninth, four high school-aged students in two settings were in combined middle/high school classrooms, thus we included their data in the analysis.
Intervention
The I Control program contains two components: the teacher-delivered curriculum and the computer-based Brain Training Lab (BTL). During curriculum development, we created lessons within four units of instruction starting with a basic introduction in Unit 1, which includes the overall goal and components of the I Control program and attendant information about EF and how it connects to effective engagement in SR. Subsequent units focus on goal setting, emotion regulation, and SPS, which are specific skill-building processes known to influence SR (see Carver & Scheier, 1998; Zelazo & Cunningham, 2007) and play an important role in the healthy integration of cognition and emotions. To guide the overall direction of the I Control intervention, we used available theoretical and empirical evidence about the link between poor SR and deficits in EF (Morgan & Lilienfeld, 2000; Raine, 2002; Séguin, Boulerice, Harden, Tremblay, & Pihl, 1999; Séguin & Zelazo, 2005). Building on the theoretical work of Miyake et al. (2000) and others (e.g., Hofmann et al., 2012; Pennington & Ozonoff, 1996; Zelazo & Cunningham, 2007), we developed Unit 1 to teach students how EF skills are related to the ability to self-regulate. To develop Units 2 to 4, we used the theoretical and empirical literature about goal setting and SR (e.g., Duckworth, Grant, Loew, Oettingen, & Gollwitzer, 2011; Gollwitzer & Sheeran, 2006; Pintrich, 2000; Zimmerman, 2012), regulating strong emotions mentally and behaviorally during challenging situations (e.g., Gross, 1998, 2002; Gross & Thompson, 2014; Koole, Van Dillen, & Sheppes, 2011), and the multistep cognitive processes necessary for solving social conflicts for positive overall social functioning (e.g., Crick & Dodge 1994; D’Zurilla, Nezu, & Maydeu-Olivares, 2004; Smith & Daunic, 2006).
As an intensive intervention to meet the needs of students with significant emotional and behavioral problems in self-contained and alternative settings, the curriculum’s 46 scripted lessons (eight–12 lessons per unit) are designed to last approximately 30 min each. I Control is designed for self-contained whole group instruction delivered at the rate of two lessons per week for a total of approximately 80 to 90 min of weekly curriculum instruction including BTL practice. In Unit 1 (Introduction), we provide instruction on the EF processes of working memory, cognitive shift, and inhibitory control. For example, as an enjoyable activity to demonstrate basic EF processes at work, students participate in a Stroop (1935) task in which they are shown a series of color words, each in a color that may or may not match the word. Participants are first instructed to say the word and, in a second trial, to say the color the word is written in. This task is designed to show students how successful completion requires that they control their impulses, shift to a new set of rules, and use working memory to update their thinking and actions based on the new rules. In another Unit 1 activity, students are shown a series of slides with colored shapes, some with colored borders and some without. Students are asked to circle the name of the shape when the slide has a border and the name of the color when there is no border. Similar activities included throughout Unit 1 provide students with practice opportunities using EF-based skills in engaging, performance-based tasks.
In Unit 2, I Control My Goals, students learn how to set value-based goals, use reality checking, and use strategies that strengthen goal commitment and pursuit. Developmentally appropriate scenarios (e.g., using realistic goal setting strategies to make a sports team, creating goals appropriate to maintain friendships) are used throughout the unit to facilitate learning, and graphic organizers provide visual support and a means for progress monitoring. As students work through the unit, they develop a personal goal and use a goal planning worksheet to identify a value-based goal and possible resources and barriers, visualize and detail a plan for goal completion, and monitor personal progress toward their goal. They have several opportunities to practice creating if-then plans to approach attaining their goal and avoiding possible temptations proactively. Thus, students are provided ongoing individualized practice to apply goal-setting skills in a personally meaningful way.
I Control’s third unit (I Control My Emotions) is focused on emotion regulation, where students learn how to identify their feelings, describe feelings in terms of common emotions, and recognize emotion intensity and triggers. A thermometer provides a visual support to help students recognize the intensity of an experienced emotion, and this skill is later applied in an emotion intensity game. Throughout Unit 3, students use journals to record an intense or difficult emotion experienced recently and its possible cause and function. Knowledge about emotions is followed by instruction in strategies for controlling arousal responses. Students learn specific techniques to alter a situation (e.g., avoiding a situation known to be emotionally challenging), change their focus (e.g., shifting thoughts to something pleasant), reframe their thinking (e.g., interpreting the situation in a positive way), or modify their actions (e.g., using deep breathing) to regulate their emotions more effectively. They are taught to select a best-fitting strategy based on an emotional state and are given opportunities to practice newly acquired skills by working with peers, journaling, and conducting role-plays that help consolidate new knowledge.
Unit 4, I Control My Problem Solving, focuses on the skills necessary to work through everyday social problems effectively. The problem-solving sequence incorporates clearly defined, sequential steps using the “CNTRL” acronym (C = Check for a problem; N = Name the problem and the goal; T = Think of solutions; R = Respond with a plan; and L = Look at how you did). Students are provided with opportunities to hone their skills using scenarios typically experienced by adolescents, such as rumors spread by peers, working with classmates on group assignments, or conflicts with friends. In this last unit of I Control, students can integrate their understanding of EF, goal setting, and emotion regulation in the social problem-solving process. To provide a cumulative review and further practice for generalization, the curriculum concludes with six booster lessons that require students to apply self-regulatory skills learned throughout I Control in novel problem-solving role-plays, games, whole and small group activities, and homework.
Across all I Control units, lessons include direct instruction, teacher modeling, guided practice through role-plays, journaling, interactive partner- and team-based activities, and independent practice to support skill development and generalization. During curriculum development, we monitored and evaluated lesson implementation by teachers using lesson-specific observation tools and obtained teacher feedback that helped us adjust components to enhance treatment fidelity, social validity, and I Control’s potential for improving target students’ SR and social competence.
BTL
As an adjunct to the I Control lessons, the BTL is an age-appropriate set of computerized games designed to train and improve EF skills. To develop the BTL, we used theoretical and empirical literature that supports the effectiveness of direct training of EF skills through modification of neural networks associated with SR (Klingberg, 2010; Klingberg, Forssberg, & Westerberg, 2002; Olesen, Westerberg, & Klingberg, 2004; Posner, Rothbart, & Rueda, 2008). We created the BTL by using the framework from the Psychology Experiment Building Language, a free psychology software for creating experiments and measuring various psychological constructs. We selected and adapted games that used specific EF-related skills so that students could spend 20 to 30 min weekly engaged in direct training of working memory, cognitive shift, and inhibitory control. For example, a game to develop better inhibitory control instructs students to hit the space bar for every letter presented in a series except when they see an X. A student self-monitoring system provides students and teachers the opportunity to monitor and record progress on such computerized tasks. As students exhibit skills, they can “level up” to more complex games.
Implementation Procedures
After informing school personnel about the results of random assignment, we provided professional development for teachers in the treatment group. The initial 2-day, 12-hr workshop, held during the second month of the school year, provided teachers with theoretical and practical understanding of SR and its relation to EF skills, and training in I Control implementation procedures. Both treatment and control teachers administered assessments within 2 weeks after training and prior to beginning I Control lessons, allowing adequate time for teachers in both conditions to become familiar with their students.
We instructed I Control teachers to begin teaching lessons 2 times per week immediately following pretest data collection and to add the BTL sessions as soon as possible. We held follow-up meetings at each I Control school to re-orient teachers to curricular goals, answer questions, and solicit feedback, and we conducted a follow-up professional development session in January as teachers were nearing the end of the first two units of instruction. This 1-day refresher workshop provided teachers the opportunity to share implementation successes and challenges and allowed the project team to respond to concerns, retrain specific procedures that were problematic for teachers, and introduce the last two curricular units. Treatment group teachers were instructed to complete posttest measures after teaching the final lesson (in May), and control group teachers were asked to complete assessments within the same time frame.
Assessment of Treatment Efficacy
SR-related behaviors
The Behavior Rating Inventory of Executive Function Teacher Form (BRIEF; Gioia, Isquith, Guy, & Kenworthy, 2000) is a standardized instrument consisting of 86 items that comprise eight clinical scales. Respondents use a 3-point Likert-type scale to indicate never, sometimes, or often for each item. The Behavior Regulation Index (BRI) is comprised of Inhibit, Shift, and Emotional Control scales; the Metacognitive Index (MI) is comprised of the Initiate, Working Memory, Plan/Organize, Organization of Materials, and Monitor scales. The BRIEF has demonstrated adequate reliability and construct validity (see Gioia et al., 2000). Full sample Cronbach’s alphas for the eight individual scales in our study at posttest ranged from .89 (Organize) to .94 (Emotion Control); the BRI alpha was .97 and the MI alpha was .94.
We developed the I Control Goal Setting (ICGS) Questionnaire by using items from two measures well aligned with project goals: the Adolescent Self-Regulatory Inventory (ASRI; Moilanen, 2007) and the Self-Regulation Questionnaire (SRQ; Brown, Miller, & Lawendowski, 1999). We compiled all ASRI and SRQ items specific to goal setting, eliminated overlap, and selected age-appropriate items that aligned with the I Control theoretical orientation. The resulting ICGS questionnaire consists of 19 Likert-type scale items (eight from the ASRI and 11 from the SRQ). Of the eight ASRI items, five reflect long-term SR (e.g., It’s hard for me to get started on big projects that require planning in advance), and two reflect short-term SR (e.g., In class, I can concentrate on my work even if my friends are talking). One item was not identified previously as reflecting either long or short-term SR. The 11 items from the SRQ are from the receiving, searching, planning, implementing, and assessing subscales. The full sample Cronbach’s alpha for the total ICGS scale at posttest was .72.
The Emotion Regulation Index for Children and Adolescents (ERICA; MacDermott, Gullone, Allen, King, & Tonge, 2010) consists of 14 items that comprise three clinical scales. Respondents use a Likert-type scale to indicate strongly disagree, disagree, half and half, agree, or strongly agree for each item. Items assess the ability to manage emotions and behavior toward the achievement of intrapersonal or interpersonal goals and comprise three factors: Emotional Control (inappropriate emotional displays), Emotional Awareness (emotional self-awareness, modulation), and Situational Responsiveness (empathy, situationally appropriate affective displays; MacDermott et al., 2010). The ERICA has demonstrated adequate internal consistency and reliability. Cronbach’s alphas obtained from the current sample at posttest were .75 for Emotional Control, .62 for Emotional Self-Awareness, and .51 for Situational Responsiveness.
The Emotion Regulation Questionnaire for Children and Adolescents (ERQ-CA; Gullone & Taffe, 2011) is an adapted version of the ERQ for adults (Gross & John, 2003). The ERQ-CA has demonstrated sound internal consistency, stability, and construct and convergent validity (Gullone & Taffe, 2011) and consists of 10 self-report items that measure Cognitive Reappraisal and Expressive Suppression (e.g., When I’m worried about something, I make myself think about it in a way that helps me feel better; I control my feelings by not showing them). Respondents use a Likert-type scale to indicate strongly disagree, disagree, half and half, agree, or strongly agree for each item. Full sample Cronbach’s alphas for the individual scales at posttest were .66 for Cognitive Reappraisal and .61 for Expressive Suppression.
The Social Problem-Solving Inventory for Adolescents–Short Version (SPSI-A; Frauenknecht & Black, 2003) includes 30 Likert-type scale self-report items that comprise three scales: the Automatic Process scale (e.g., To solve a problem, I do what has worked for me in the past), the Problem Orientation scale (e.g., I put off solving a problem for as long as I can), and the Problem-Solving Skills scale (e.g., Before I solve a problem, I gather as many facts about the problem as I can). Item responses range from 0 (not at all true of me) to 5 (extremely true of me). The SPSI-A has shown strong internal consistency across all scales and subscales, regardless of test version used. Cronbach’s alphas obtained from the current sample at posttest were .71 for Automatic Process, .74 for Problem Orientation, and .90 for Problem Solving.
General problem behaviors and social interactions
The Achenbach System of Empirically Based Assessment–Child Behavior Checklist–Teacher Report and Youth Self-Report Forms (ASEBA/CBCL-TRF and YSR; Achenbach, 2009; Achenbach & Rescorla, 2001) are standardized instruments consisting of 112 items each that comprise eight subscales (Withdrawn/Depressed, Somatic Complaints, Social Problems, Thought Problems, Attention Problems, Rule-Breaking Behavior, Aggressive Behavior) and measure internalizing and externalizing behaviors. Respondents (teachers or students) use a Likert-type scale to indicate not true (as far as you know), somewhat or sometimes true, or very true or often true for each item. The TRF and YSR have demonstrated adequate reliability and construct validity (Achenbach & Rescorla, 2001). Full sample Cronbach’s alpha estimates in our pilot study at posttest were .96 for Externalizing and .91 for Internalizing on the TRF, and .91 for Externalizing and .91 for Internalizing on the YSR.
The Social Skills Improvement System (SSiS; Gresham & Elliott, 2008) is a standardized teacher report measure that has demonstrated high reliability and validity and assesses three domains: Social Skills, Problem Behaviors, and Academic Competence. For this study, we used the first two domains. Respondents use a Likert-type scale to indicate never, sometimes, often, or almost always for each item. The Social Skills domain is comprised of seven subscales: Communication, Cooperation, Assertion, Responsibility, Empathy, Engagement, and Self-Control. The Problem Behaviors domain consists of five subscales: Externalizing, Bullying, Hyperactivity/Inattention, Internalizing, and Autism Spectrum. Full sample Cronbach’s alphas for the two domains were .95 for Social Skills and .93 for Problem Behaviors.
Assessment of Curriculum-Based Knowledge and Treatment Fidelity
To assess students’ knowledge of curricular concepts as a check on “treatment received,” we developed the I Control Knowledge Questionnaire (ICKQ) that consists of 36 multiple-choice items (A–D format) that address primary lesson objectives from all four units. The ICKQ contains eight items testing Unit 1 knowledge, with nine, nine, and 10 items testing Units 2 to 4, respectively. Full sample Cronbach’s alphas for the four units at posttest were .65, .70, .68, and .76.
We observed 102 (34%) of 297 total lessons taught across all I Control teachers to assess treatment fidelity. We coordinated with treatment teachers to determine lesson observation times, and all teachers were provided video recording devices, resulting in videotapes of most lessons. Research team members used lesson-specific fidelity checklists to assess adherence to lesson components and level of student engagement (e.g., listening attentively, eyes on teacher, answering questions, participating in activities) evaluated on a 5-point scale (1 = very little to no engagement and 5 = most or all students were very engaged). Observers did not offer teachers formal or systematic instruction. Pairs of observers measured treatment fidelity on 20% of lessons. We used percent of rater agreement averaged across lessons/steps to determine overall inter-rater agreement for adherence to curriculum components. We estimated inter-rater reliability on student engagement using percent of adjacent agreement (percent of rater scores within one point of each other). The adjacent agreement approach is a commonly used method in educational research when using Likert-type scales (Jonsson & Svingby, 2007).
Statistical Analysis
We used Mplus 7.3 to conduct all inferential analyses. We considered two approaches to analyzing the data because students were clustered within schools/teachers. First, we used a two-level analysis, wherein clustering within teachers is accounted for by estimating teacher-level and student-level residual variances. Second, we used the complex option in Mplus 7.3, wherein clustering is accounted for by correcting the standard errors for clustering within teacher (Asparouhov & Muthén, 2006; Rabe-Hesketh & Skrondal, 2006). We chose the complex option with teacher as a cluster because a two-level model resulted in non-convergence for some variables.
For each variable, we estimated two models using pretest as the covariate and posttest as the dependent variable. In the first model, the slopes relating the dependent variable to the covariate were allowed to vary across I Control and control groups, and significance of the slope difference was tested to examine the Covariate × Treatment (C × T) interaction. When the interaction was significant, we conducted simple effects tests at three values of the pretest: the grand covariate mean (GCM) and the GCM plus and minus 1 standard deviation. These three values are termed low, moderate, and high in the results section. If the interaction was non-significant, the slopes in the second model were restricted to be equal, and we tested the difference between adjusted dependent variable means to examine the main effect of treatment.
We used non-directional tests for interactions and directional tests for main and simple effects. Reported p values correspond to non-directional tests of the C × T interaction and directional tests of main and simple effects. We used α = .05 for all hypothesis tests. There were no variables for which (a) the adjusted mean was better for the control group, (b) the two-tailed test of the main effect was significant, (c) the two-tailed test of the simple effect was significant, or (d) the simple effect indicated a better outcome for the control group. We calculated effect sizes (ES) by the difference between I Control and control group conditional means (in models including the C × T interaction) or adjusted means (in models excluding the C × T interaction) divided by the standard deviation for the control group on the dependent variable. For instruments on which high scores are desirable, a positive value of ES indicates better performance for I Control participants, and for instruments on which low scores are desirable, a negative value of ES indicates better performance for I Control participants.
For each variable, there were a few cases with a missing covariate score and an observed dependent variable score. Therefore, we formulated models in Mplus so that the likelihood was defined as conditional on treatment and included both the covariate and the dependent variable (see Muthén & Muthén, 1998–2013, p. 7).
Results
Comparison of Cases With Missing Dependent Variable Scores
The number of cases with missing teacher report measures at post ranged from 36 to 38 for the I Control group and was 16 for the control group. The number of cases with missing student-report measures at post ranged from 16 to 47 for the I Control group and 19 to 22 for the control group. Means on the covariate for these participants were compared with means on the covariate for participants with no missing data. We conducted the comparison separately for participants assigned to I Control versus the control group. Mean differences were non-significant for both groups on all variables.
Treatment Efficacy
For the ICGS and ERQ-CA, all C × T interactions and main effects were non-significant. We did, however, find one or more significant tests for the BRIEF, ERICA, SPSI-A, TRF, YSR, and SSiS. Test statistics and p values for treatment and C × T interactions for each variable are shown in Table 2; ES are also presented for main effects. We explain interaction effects associated with variables in each outcome category in the following sections.
Results of Mplus Analyses for I Control Effects on Outcome Variables.
Note. BRI = Behavior Regulation Index; MI = Metacognitive Index; ICGS = I Control Goal Setting Questionnaire; ERICA = Emotion Regulation Index for Children and Adolescents; ERQ-CA = Emotion Regulation Questionnaire for Children and Adolescents; SPSI-A = Social Problem-Solving Inventory for Adolescents; TRF = Achenbach Child Behavior Checklist, Teacher Report Form; YSR = Achenbach Child Behavior Checklist, Youth Self-Report; SSiS = Social Skills Improvement System.
p < .05, one-tailed. **p < .05, two-tailed.
SR-related behaviors
C × T interaction effects for the BRI and associated Shift and Emotional Control subscales of the BRIEF indicated that the effect of I Control was larger for each variable when participants had higher (worse) covariate (pretest/baseline) scores, with I Control associated with lower (better) outcome scores. For BRI, regression lines indicated that simple effect tests were significant when the covariate score was moderate (z = −1.68, p = .046, ES = −.38) or high (z = −3.284, p = .000, ES = −.76), indicating relatively moderate or high risk at baseline. This pattern was also found for Shift, with a significant simple effect at moderate (z = −2.19, p = .015, ES = −.50) and high (z = −4.24, p = .000, ES = −.82) values of the covariate (risk at baseline). For the Emotional Control subscale, the simple effect test was significant only when the covariate score was high (z = −2.54, p = .006, ES = −.58).
For the Problem Orientation scale of the SPSI-A, regression lines indicated that at the GCM, there was little difference between conditional means for I Control and control participants (ES = −.13). The effect of I Control was larger, however, for participants with lower (more positive) covariate scores, with the simple effect test significant at the low value (z = −2.07, p = .020, ES = −.55).
General problem behaviors and social interactions
For Externalizing Problems measured by the TRF, the effect of I Control was larger for participants with higher (worse) covariate scores, and the simple effect was significant at the high covariate score (z = −3.17 p = .001, ES = −.63). Similarly, on the YSR Externalizing and Internalizing subscales, plots of the regression lines indicated a larger effect of I Control for participants with higher (worse) scores on the covariate. For Externalizing, the simple effect test was significant when the covariate score was high (z = −2.58, p = .005, ES = −.70) but not when the covariate score was equal to the GCM (ES = −.17) or was low (ES = .37). For Internalizing, the simple effect tests were significant when the covariate was moderate (z = −1.92, p = .028, ES = −.22) or high (z = −2.90, p = .002, ES = −.57).
For Problem Behavior measured by the SSiS, there was little difference between conditional means for I Control and control participants (ES = −.08) at the GCM; the treatment effect was larger for participants with higher (worse) covariate scores and significant at the high value of the covariate only (z = −1.89, p = .030, ES = −.50). There were no significant findings for the Social Skills domain.
Treatment Fidelity
Main effect tests for the ICKQ were significant for Unit 1 (z = 2.98, p = .002, ES = .72), Unit 2 (z = 1.80, p = .036, ES = .41), and Unit 3 (z = 2.92, p = .002, ES = .55). For Unit 4 knowledge, the test of the C × T interaction was significant (z = −2.58, p = .010), with the effect of I Control being larger when participants had lower covariate (pretest) scores. Simple effect tests were significant at low (z = 3.33, p = .001, ES = .68) and moderate (z = 1.93, p = .027, ES = .32) covariate values.
The average fidelity of curriculum implementation (i.e., percent of total lesson components observed based on observation checklists) across teachers was 81% (SD = 11.6%, range = 62.3%–97.2%), indicating that the majority of teachers followed the steps as outlined in the curriculum. The mean student engagement across observed lessons was 3.6 (SD = 0.43, range = 3.50–4.32) on a 5-point scale, with 1 = very little to no engagement and 5 = most or all students were very engaged; thus, observers found the majority of students to be moderately engaged during lesson implementation. The average inter-observer agreement was 94.6% (SD = 5.3%, range = 80%–100%) on curriculum adherence and 100% on student engagement, with a single outlier of 75%. Trained observers, therefore, used the treatment fidelity checklists with adequate reliability. (Note: Lessons ranged from 15 to 39 min, with an average time to complete lessons of 26.5 min.)
Discussion
Our aim in this pilot study was to conduct a preliminary investigation of I Control, designed as an intensive SR intervention for students with significant behavioral issues. First, we wanted to see whether I Control improved teacher-rated general behavioral problems, EF-related behavior, and social skills. Second, we wanted to examine whether I Control improved students’ self-rated behaviors related to general behavior problems, goal setting, emotion regulation, and SPS at posttest and were more positive than those of comparable students in the control condition. Finally, we wanted to determine whether general student knowledge about EF, goal setting, emotion regulation, and SPS were better for students who received I Control compared with students who did not.
As this was a pilot study of a newly developed curriculum, we also wanted to field test the feasibility and adequacy of I Control protocols, such as sampling frame and recruitment, professional development, fidelity of implementation, research instruments, data analysis techniques, and the extent of resources needed to conduct a future, more adequately powered study (Leon, Davis, & Kraemer, 2011). Through a rigorous iterative process, we developed and implemented the four-unit I Control curriculum, and this preliminary investigation provides initial findings that I Control may be a viable intervention for adolescents with significant behavioral problems.
SR-Related Behaviors
EF
Our findings indicate that I Control had a positive impact on teacher reports of some student EF-related skills. The greatest ES for behavior regulation were noted for students with the higher initial deficits on the BRI and on the shift and emotional control subscales. In addition, all treatment students, regardless of pretest score, were better able to initiate (ability to begin a task or activity and independently generate ideas, responses, or problem-solving strategies) than control students, as measured by the MI of the BRIEF. These differences could be due to the explicit instructional focus on EF-related knowledge and skills in I Control. A cornerstone of the curriculum is the intentional targeting of EF-based skills by (a) a unit of instruction on knowledge about EF (i.e., working memory, cognitive shift, inhibitory control) and (b) activities that draw on EF abilities interspersed throughout the other three units. Students were taught explicitly about EF and provided multiple opportunities to practice and improve EF-related skills. Given the link between deficits in EF and poor SR (see, for example, Morgan & Lilienfeld, 2000; Raine, 2002; Séguin & Zelazo, 2005), improvements in a variety of teacher-reported EF-related skills are encouraging.
Goal setting
Although we found improvements in students’ goal-setting knowledge, we did not find evidence that I Control affected students’ self-reported goal-setting abilities. There may have been insufficient time for the gains in knowledge to transfer to goal-setting skill development. According to Gollwitzer (1990) and Gollwitzer and Sheeran (2006), goal setting and goal pursuit consist of a number of action phases that take place over time, from goal commitment (forming strong goal intentions), moving to initiating goal-directed behaviors (when, where, how to implement) and successfully engaging goal-directed actions (responding flexibly to threats to progress), and, finally, to evaluating goal achievement (degree of attainment and quality). Thus, while knowledge of goal setting is foundational at all phases of the model, students who were taught I Control may have required more instructional time to acquire goal-setting skills. Furthermore, while the goal-setting measure developed for this study demonstrated good internal reliability, additional psychometric analyses are warranted before placing confidence in the results obtained.
Emotional control
We found that all treatment students, regardless of pretest score, who were taught I Control reported better emotional control than students in the control condition on the ERICA; thus, they were better able to modulate inappropriate displays of emotion. Teachers reported similar observations for the students with the highest deficits at pretest as evidenced by the emotional control subscale of the BRIEF. Results are most likely due to I Control’s focus on emotion regulation, a key instructional unit, devoted to learning and applying strategies that help students control their emotions and, ultimately, improve their behavior. Our findings align with those resulting from other curricula that include instruction in emotion regulation (Daunic et al., 2012; Lochman & Wells, 2002).
SPS
Students who were taught I Control with better SPS skills at pretest reported a greater tendency to approach a problem positively than comparable peers in the control condition as measured by the SPSI-A. As some teachers implemented I Control at a slower pace than others, resulting in differential completion rates for this final unit, the lack of consistent exposure to SPS knowledge and skill practice may explain the lack of improvement for students with poorer pretest SPS skills. These students may have needed more instruction to alter preexisting schemas and beliefs (Crick & Dodge, 1994) that affect related skill application.
General Problem Behaviors and Social Interactions
We found that after students were taught I Control, those with more prominent externalizing behavior at pretest differed significantly from controls on both teacher- and student-reported measures. The externalizing domain for these measures included behaviors such as student rule-breaking and aggressive behavior, with larger ES for those with poorer pretest scores on these behaviors. The fact that I Control appears to have the greatest effect for students most in need of intervention is promising, given the poor outcomes associated with chronic externalizing behaviors (Bradley et al., 2008).
Interestingly, students who self-reported moderately greater internalizing behavior problems at pretest differed from controls at posttest but not on parallel teacher reports. Given that internalizing behavior is not as easily observed as externalizing, teachers may not have noticed these differences within the classroom. Students may be better able to identify the covert processes connected with internalizing symptoms and thus better able to recognize behaviors that resulted from I Control. This finding is particularly promising given the trend to underidentify these students in education systems, which has resulted in a lack of services (Gresham & Kern, 2004). I Control appears to benefit students who exhibit problematic internalizing behaviors as well as those with significant externalizing behaviors.
We also found that treatment effects on Problem Behavior, as measured by the SSiS, were only for students who had the most significant risk at pretest. Although this subscale includes behaviors such as bullying, hyperactivity/inattention, and behavior relative to autism spectrum, it also measures externalizing and internalizing behavior that correspond with outcomes found on the TRF and YSR. Thus, the outcomes across these measures are consistent. There were no significant findings for the SSiS Social Skills domain, perhaps because I Control does not teach specific social skills but focuses on underlying self-regulatory processes.
Knowledge of Curricular Domains
Findings from the ICKQ revealed main effects for Units 1 to 3. Students who acquired foundational knowledge of these self-regulatory domains should be better positioned to develop, and then engage in, related skills within the classroom and generalize them to other settings (Bandura, 1977; Mayer & Van Acker, 2009). In turn, when students with significant behavior problems successfully enact learned self-regulatory skills, the probability of their receiving subsequent reinforcement for appropriate behavior is enhanced. The lack of a main treatment effect for Unit 4 knowledge may reflect the fact that not all students received the complete Unit 4 on SPS instruction due to time constraints. Nevertheless, students with less pretest knowledge who were taught I Control had higher posttest knowledge than comparable control group students.
Limitations and Future Research
Although we found preliminary evidence that I Control can improve social-emotional and behavioral outcomes for adolescents who exhibit behavioral problems within school settings, we acknowledge limitations of the current pilot study that warrant a cautious interpretation of findings. First, we experienced a larger than expected rate of student and teacher attrition for a variety of reasons. For example, one teacher was reassigned to an administrative position, some students returned to their home school or transferred to other classrooms, some students moved out of district, and at least one student was adjudicated. Furthermore, this attrition occurred at a higher rate in the treatment condition than in the control condition. We had no reason to suspect that attrited students differed systematically from those who completed the study; nevertheless, differential attrition is problematic. We acknowledge that this warrants interpreting study findings with caution, but we did address the high attrition rate statistically by using full information maximum likelihood (FIML), which considers all cases with pretest information. This may be an especially useful method for studies of at risk populations who are more likely to be in situations that necessitate reassignments and mobility.
Second, the number of classrooms and students (sample size) proposed for the current study was appropriate for pilot implementation associated with intervention development, but it could well have resulted in relatively low power to detect differences due to treatment. Although we found statistically significant differences between conditions, a future randomized controlled field trial with a sample large enough to provide ample power to conduct multiple comparisons at the school, classroom, and student levels is warranted. A larger sample would also diminish the likelihood of differential rates of attrition.
Furthermore, as part of an innovative development project, our field test illuminated additional limitations associated with length of lessons and rate of delivery across teachers in classroom environments with varying resources and schedules. Although we followed an intent-to-treat model (see Lachin, 2000), the information gained from this investigation (e.g., amount of time involved in lesson delivery across the school year) informed our curricular modifications and scheduling expectations that will guide future investigations.
Finally, when teachers involved in intervention delivery are also respondents on outcome measures, expectancy effects can influence findings and thus must be recognized. As such, we also used student self-report measures, and the results from these assessments are generally consistent with those from teacher-reported outcomes. Moreover, teacher reports are contextualized in authentic settings and result from observations that occur over a period of time, making them a valuable source of data, despite possible bias (see Daunic et al., 2012). Future research, however, should incorporate measures completed by teachers, student participants, and additional informants, such as school counselors and parents, when possible. Future studies could also incorporate direct behavior ratings using a hybrid methodology that entails brief ratings of target behaviors after direct observation during a specified period, thus incorporating the strengths of both rating scales and systematic direct observations (e.g., Chafouleas, Riley-Tillman, & Christ, 2009).
Also worthy of future investigation would be the covariate (pretest) by treatment interaction effects we found on many of the measures, most indicating that students with the highest baseline risk benefited more from the intervention. It is important to remember that student risk was idiosyncratic to measure; that is, the same students may not have evidenced high baseline risk consistently across characteristics measured. It could be that students whose scores indicated relatively less baseline risk on a given measure had little room to improve and therefore did not differ significantly from control students with a similar baseline risk level. It is also possible that students who exhibit a particular pattern of risk across measures may benefit more from I Control. How student risk profiles affect intervention efficacy would be an interesting topic for future studies.
In sum, despite several methodological limitations and the consequent use of caution in drawing conclusions about the efficacy of I Control from this preliminary study, the fact that we obtained statistically significant positive findings related to general student behavior, emotion regulation, and EF is promising. Our findings argue for further explorations of this intervention’s potential to improve outcomes for adolescents with significant behavior problems. We are hopeful that interventions that teach explicit self-regulatory strategies, such as I Control, will continue to be the subject of rigorous evaluations that potentially benefit students most in need of effective curricular programming.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by a 4-year grant from the U.S. Department of Education, Institute of Education Sciences (#R324A110182) submitted to CFDA 84.324A, National Center for Special Education Research.