A Randomized Effectiveness Trial of a Behavioral Teacher Program Targeting ADHD Symptoms

Abstract

Objective: This study investigated the effectiveness of the Positivity & Rules Program (PR program), a behavioral teacher program targeting ADHD symptoms in the classroom involving both student-focused and classroom-focused programs. Method: Primary school children with ADHD symptoms (N = 114) were randomly assigned to the PR program (n = 58) or control group (n = 56). Teacher and parent ratings were used to assess behavioral, social, and emotional functioning at baseline, during and after the intervention. Intervention effects were assessed using intention-to-treat multilevel analyses. Results: Teachers reported positive effects on ADHD symptoms and social skills (.01 < f² > .36). Effects did not generalize to the home setting. Conclusion: The PR program holds promise for improving classroom behavior in children with ADHD symptoms and might prevent escalation of problem behavior

Keywords

contingency management behavioral treatment school teacher ADHD

ADHD is one of the most common childhood psychiatric disorders, affecting 5% of all children (Polanczyk, de Lima, Horta, Biederman, & Rohde, 2007), and is characterized by excessive inattention and/or hyperactivity/impulsivity (American Psychiatric Association, 2013). Children with ADHD often experience behavioral and social problems at home and at school, and comorbid disorders including oppositional defiant disorder (ODD), conduct disorder (CD), and anxiety disorders (Daley & Birchwood, 2010; Hong et al., 2014). They also show an increased risk for adverse life events such as occupational and relational problems, substance abuse, and delinquency (Molina et al., 2007; Shaw et al., 2012). In addition, more than 10% of school-aged children display ADHD symptoms without meeting full diagnostic criteria, while experiencing similar problems and increased risks compared with children diagnosed with ADHD (Faraone et al., 2007; Hong et al., 2014). At school, academic problems of children with ADHD symptoms (e.g., poor reading and numeracy skills) often coincide with behavioral problems such as off-task behavior, disturbing behavior (e.g., not awaiting turns), and teacher disobedience, especially in case of comorbid conduct problems (Daley & Birchwood, 2010; Diamantopoulou, Rydell, Thorell, & Bohlin, 2007; Wilens et al., 2002). Social problems, such as having few friends and difficulties sustaining friendships, are also common in children with ADHD symptoms, irrespective of meeting full diagnostic criteria (Hong et al., 2014; Hoza, 2007). Through recurrent negative feedback from teacher or peers, increasing emphasis on the child’s negative behavior is likely to occur, further propelling the child’s problem behavior (Hoza, 2007). Cleary, early identification and treatment of ADHD symptoms is essential to reduce the risk of problem behavior escalation (Kutcher et al., 2004). In this study, the effectiveness of a classroom-based behavioral program targeting children with ADHD symptoms is investigated.

Psychostimulant medication is currently used as first-line treatment of ADHD and has been shown to powerfully reduce ADHD core symptoms while also improving oppositional behavior, aggression, and social interactions (Kutcher et al., 2004). Adverse side effects (i.e., headache, sleeping problems, and loss of appetite) are common though, and 20% to 30% of children do not respond favorably to stimulants (Kutcher et al., 2004; MTA Cooperative Group, 1999). Furthermore, long-term effects of stimulants have not yet been established (Parker, Wales, Chalhoub, & Harpin, 2013; Van der Oord, Prins, Oosterlaan, & Emmelkamp, 2008). Behavioral interventions, heavily relying on principles of contingency management, are found to enhance ADHD symptoms, conduct problems, and social skills in children displaying ADHD symptoms, including both children with and without a diagnosis of ADHD (Charach et al., 2013; Daley et al., 2014; Fabiano et al., 2009; Wilson & Lipsey, 2007). Indeed, behavioral interventions are advised as first-line treatment for children with mild to moderate ADHD (Atkinson & Hollis, 2010). Therefore, behavioral teacher programs appear most suitable to address the problems that children with ADHD symptoms encounter at school.

Several studies have found that behavioral teacher programs effectively reduce classroom problem behavior including ADHD symptoms, oppositional behavior, and conduct problems (Bradshaw, Mitchell, & Leaf, 2010; Evans, Owens, & Bunford, 2014; Miranda, Presentación, & Soriano, 2002; Pelham & Fabiano, 2008; Pelham et al., 2000; Webster-Stratton, Jamila Reid, & Stoolmiller, 2008). Many of these programs, however, involve intensive and time-consuming training of teachers, which could compromise long-term sustainability as funding is often not available for such programs after research is terminated. The Positivity & Rules Program (PR program), a behavioral teacher program targeting ADHD symptoms, relies on a manual not requiring any additional teacher training and could possibly benefit many children while being sustainable at little cost (Durlak & DuPre, 2008). This program is based on common behavioral and classroom management strategies that are part of most behavioral classroom programs, which have proven to be effective as stand-alone program (see, for example, the review of Pelham & Fabiano, 2008). Low-cost programs such as the PR program are particularly important for children not meeting full diagnostic ADHD criteria, whose treatments are often not covered by private or public insurance.

The current randomized controlled trial tested the efficacy of the PR program on ADHD symptoms, conduct problems, social and emotional functioning compared to no-treatment or care as usual. Effects of treatment were measured using teacher and parent ratings assessed before, during, and after the intervention. If proven effective, this behavioral teacher program might be used at large scale to address behavior problems in children displaying ADHD symptoms in the classroom.

Method

Participants

Participants were 114 children between 6 and 13 years of age attending regular primary school and displaying elevated levels of ADHD symptoms in the classroom. A total of 58 children (from 44 classrooms of 30 schools; 91% male) were randomly allocated to the PR program (intervention group), and 56 children (from 43 classrooms of 34 schools; 77% male) were allocated to the control group. Teachers of children in the control group did not use the PR program, but these children were allowed to receive care as usual (22% eventually received some form of care, such as advice from a school counselor or parent training). Randomization occurred at school level to prevent information drift from teachers using the PR program to teachers of children in the control group.

Inclusion criteria were (a) elevated levels of ADHD symptoms (>90th percentile) as reported by the child’s teacher on the Hyperactivity/Impulsivity and/or Inattention scale of the Disruptive Behavior Disorders Rating Scale (DBDRS; Oosterlaan et al., 2008; Pelham, Gnagy, Greenslade, & Milich, 1992) and (b) at least three clinical and three subthreshold ADHD symptoms as assessed by Teacher Telephone Interview (TTI; Holmes et al., 2004), a semi-structured interview based on Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; DSM-IV-TR; American Psychiatric Association, 2000). Exclusion criteria were (a) treatment for ADHD (including medication) at study entry or in the preceding 6 months, (b) a neurological or severe physical condition interfering with daily functioning, (c) IQ < 80 estimated using a short version of the third edition of the Wechsler Intelligence Scale for Children (WISC-III, including Block Design and Vocabulary; Sattler, 1992), or (d) participant being enrolled in a daily contingency management program or another teacher program addressing behavior or social problems at study entry or in the preceding month. No more than two children per classroom and five classrooms per school were allowed to participate to limit teacher burden and to increase heterogeneity of teacher and school settings involved (Scherbaum & Ferreter, 2009). Figure 1 displays the flowchart of participants.

Figure 1.

Consolidated Standards of Reporting Trials (CONSORT) flow diagram of participants during enrollment, allocation, follow-up, and analysis.

PR Program (Druk in de Klas)

The PR program consists of a behavioral teacher program addressing ADHD symptoms in the classroom through a teacher manual not requiring additional expert training. The program involves evidence-based elements of behavioral classroom programs (e.g., the Summer Treatment Program; MTA Cooperative Group, 1999) such as psycho-education for the teacher, classroom behavior management strategies, and contingency management (e.g. a reward and time-out system; Pelham & Fabiano, 2008). During the 18-week program, students with ADHD symptoms and their classmates are administered a universal program encompassing elements such as physical adjustments within the classroom (e.g., table set-up, creating a time-out corner), positively formulated classroom rules, effective teacher instructions and teacher strategies to reinforce appropriate behavior (such as a universal reward system for the entire classroom). Although several techniques will be familiar to most teachers, the manual instructs teachers how to systematically and adequately implement all intervention elements by providing detailed practical instructions on implementation. Six weeks after the start of the program, an individual program for the student with ADHD symptoms may be added to the universal program if the child’s ADHD symptoms have not been reduced sufficiently. This is determined by calculating the proportion of behavior change by dividing the student’s sum score of all “below average” and “far below average” items of the Strengths and Weaknesses of ADHD-symptoms and Normal-behavior scale (SWAN, Teacher version; Swanson et al., 2006) obtained 6 weeks after start of the program by the sum score of all “below average” and “far below average” items obtained at baseline, using a cutoff score of 0.75 (scores ≤ 0.75 indicate sufficient behavior change).

The individual program involves a Daily Report Card (DRC) in which teacher and student with ADHD symptoms set and evaluate mutually agreed goals (e.g., “stay seated during individual work” or “raise your hand before asking questions”) to alleviate student’s classroom problems. Rewards are provided by the teacher (i.e., through activities or privileges) at the end of each school day within the classroom. The individual program consists of three intensity levels, differing in the number of times per day goals are evaluated (once a day in Level 1 and 3 times a day in Levels 2 and 3), and rewards are provided (once a day in Levels 1 and 2, and 2 times in Level 3). Every 3 weeks, the student’s behavior is evaluated using the SWAN (teacher version), after which the proportional behavior change (see previous paragraph) is used to determine whether the intensity level needs to be adjusted. Practical examples (e.g., of suitable rewards and positive classroom rules), work sheets (e.g., DRC), flow diagrams (e.g., of the time-out system), and a calendar (weekly schedule of all program elements indicating when to implement each element) facilitate implementation of the program. Although teachers were supposed to use the manual without assistance, they could consult the researcher if necessary, which was done by 15% of the teachers.

Outcome Measures

Program efficacy was assessed using teacher and parent ratings of behavioral, social, and emotional functioning. Teachers were aware of treatment allocation as they were responsible for implementing the PR program in their classroom. Parents were aware of treatment allocation because hiding allocation status would have been practically impossible (due to visibility of program elements within the classroom).

ADHD symptoms were assessed using the SWAN (Swanson et al., 2006). This questionnaire contains 18 items measuring the presence and severity of ADHD symptoms on a continuum, using a 7-point Likert scale ranging from −3 (far above average) to +3 (far below average; Lubke et al., 2007; Young, Levy, Martin, & Hay, 2009). In current study, internal consistency for the teacher and parent SWAN was high (Cronbach’s α = .91 and .87, respectively).

Psychosocial problems and prosocial behavior were assessed using the Strengths and Difficulties Questionnaire (SDQ; Goedhart, Treffers, & Widenfelt, 2003; Goodman, 1997). This questionnaire consists of 25 items on a 3-point Likert scale (0 = not true, 1 = somewhat true, and 2 = certainly true), divided into five scales: ADHD, Conduct Problems, Emotional Symptoms, Peer Problems, and Prosocial Behavior. Adequate psychometric properties have been reported for the Dutch SDQ (Van Widenfelt, Goedhart, Treffers, & Goodman, 2003).

To assess the child’s social functioning, the Social Skills Rating Scale (SSRS) was used (Gresham & Elliott, 1990). The teacher version consists of 30 items divided over three scales (Collaboration, Assertiveness and Self-Control), and the parent version contains 10 extra items related to responsibility for property or work and the ability to communicate with adults, using the total scores of the teacher and parent version as dependent variables. The child’s social functioning was rated on a 3-point Likert scale (0 = never, 1 = sometimes, and 2 = often). Adequate internal consistency and high predictive validity have been reported for the Dutch SSRS (κ = 0.77; Van der Oord et al., 2005).

A teacher questionnaire (available on request) was administered at the end of each week to provide an indication of implementation fidelity. This checklist contains 13 items on a 3-point Likert scale, requiring teachers in the intervention group to indicate whether and to what extent they used each of the intervention elements during that week (0 = not used or inadequate use, 1 = adequate use, and 2 = good use). The average item score was calculated based on all weekly checklists. Acceptable internal consistency was found for this measure (Cronbach’s α = .67).

Procedure

Teachers and parents were recruited through educational consultant associations, the national parent association for children with developmental problems, and the study’s website. Teachers and parents showing interest in participating in the study received an information letter explaining the research aim and responsibilities of all parties involved. In case teachers were interested in participating, they enlisted one or two children displaying ADHD symptoms in their classroom. Written consent was obtained from teachers, parents, and children older than 11 years, after which potential participants were screened for eligibility. ML, who had not been in contact with any participants, was responsible for the subsequent computer-generated randomization. Although teachers in the intervention condition used the universal program in the entire classroom, the effectiveness of the PR program was only investigated for the participating children displaying ADHD symptoms. Outcome variables were assessed 1 week prior to the intervention (T_o), 6 weeks after start of the intervention (T₁), and after 18 weeks at the end of the intervention (T₂). Teachers received financial compensation for their participation in the study (control group: €50; intervention group: €125). The study was carried out in the Netherlands between September 2011 and July 2014. Procedures were approved by the medical ethical committee of the Vrije Universiteit Amsterdam.

Statistical Analyses

The software Optimal Design (Raudenbush et al., 2011) was used for sample size estimation. For a repeated-measures multilevel analysis with a maximum of two participants per classroom, a sample size of 116 was calculated to be sufficient, assuming a moderate effect size (d = 0.50) and using an alpha of .05, a power of 80%, and a intraclass correlation of .10.

Multilevel analyses were conducted using the Statistical Package for the Social Sciences (SPSS) to evaluate effects of the PR program, while using Statistical Analysis System (SAS) for effect size calculation (Selya, Rose, Dierker, Hedeker, & Mermelstein, 2012; Statistics, 2012). All participants were included in the intention-to-treat multilevel analyses. Four hierarchical levels were distinguished: observations at Level 1, nested within students (Level 2), nested in classrooms (Level 3), and nested in schools (Level 4; Heck, Thomas, & Tabata, 2013). Group was inserted as dichotomous variable, using control group as reference group, and Time was inserted as covariate, which was expressed in number of weeks (0, 6, and 18 weeks for T₀, T₁, and T₂, respectively). The group by time interaction effect was inserted in the model to investigate whether behavior in the intervention group improved more over time compared with the control group. Random time slopes at student-, classroom- or school-level were included if significantly improving model fit using the Likelihood Ratio Test. The Akaike’s Information Criteria (AIC) was used to determine which random time slope (student-, classroom- or school-level, respectively) resulted in the best fit (Heck et al., 2013). For the final models with significant group by time interactions, it was assessed whether effects remained significant when controlling for baseline levels of the dependent variables. Effect sizes were calculated using Cohen’s f² with values of .02, .15, and .35 as thresholds for small, medium, and large effects, respectively (Selya et al., 2012). Alpha level was set at .05.

Results

Table 1 displays demographic characteristics of the intervention and control group. Possible group differences were tested using t tests for continuous variables and chi-square or Fisher’s exact tests for categorical variables. No group differences were observed on the screening variables (p values > .26), with two exceptions: (a) there were somewhat more boys in the intervention group (n = 53) compared with the control group (n = 43; χ²(1) = 4.56, p = .03); and (b) the intervention group displayed more symptoms of hyperactivity/impulsivity as measured by the TTI than controls (t(112) = −2.32, p = .02), although scores on the Hyperactivity/Impulsivity scale of the DBDRS did not differ between groups. For all outcome measures yielding significant group by time interactions, we investigated whether the baseline gender difference could have driven the group by time interactions by including gender into the model (by inserting the main effect, two-way interactions with group and time, and three-way interaction between group, time, and gender). The three- and two-way interactions were included if model fit significantly improved, or else only gender was included to control for baseline gender differences.

Table 1.

Demographic and Clinical Characteristics of Participants (N = 114) in Intervention and Control Group.

	Intervention group (n = 58)	Control group (n = 56)
Demographic characteristics
Age (years)	8.48 (1.85)	8.25 (1.97)
Gender (% male)*	91% (n = 53)	77% (n = 43)
IQ	104.02 (11.34)	100.21 (10.41)
SES^a	3.37 (0.67)	3.24 (0.95)
Race (% Caucasian)	86% (n = 50)	82% (n = 46)
ADHD diagnosis	10% (n = 6)	9% (n = 5)
Other psychiatric diagnosis	2% (n = 1; CD)	2% (n = 1; PDD-NOS)
Parent DBDRS
Inattention	11.71 (5.23)	10.77 (5.57)
Hyperactivity/Impulsivity	11.79 (5.42)	10.47 (5.49)
ODD	5.88 (3.84)	4.93 (4.00)
CD	0.86 (1.31)	0.90 (1.40)
Teacher DBDRS
Inattention	14.63 (5.26)	14.90 (5.83)
Hyperactivity/Impulsivity	15.67 (5.35)	15.03 (6.23)
ODD	6.77 (4.75)	5.95 (4.75)
CD	1.43 (1.65)	1.57 (1.90)
TTI
Inattention	12.50 (6.12)	12.45 (5.34)
Hyperactivity/Impulsivity*	15.62 (6.03)	12.88 (5.78)
Combined	28.12 (8.97)	25.33 (8.34)

Note. M and SDs are depicted unless stated otherwise. SES = socioeconomic status; PDD-NOS = pervasive developmental disorder—not otherwise specified; DBDRS = Disruptive Behavior Disorder Rating Scale; ODD = oppositional defiant disorder; CD = conduct disorder; TTI = Teacher Telephone Interview.

SES was measured by parental educational level (average of both parents) through an adapted version of the Dutch classification system (1 = primary education, 2 = secondary vocational education, 3 = secondary general education, 4 = undergraduate school, 5 = graduate school; Verhage, 1983).

p < .05.

Implementation Fidelity

Teachers reported to have used all elements of the universal program for at least 85% of the time during the 18 intervention weeks (M = 0.88%; SD = 0.11), with the single exception of the element “provide the child with three compliments after a reprimand is given” which was used on average 61% of the time (SD = 0.37). Most teachers (81%) reported adequate or good implementation of the universal reward system during the entire intervention, with the remaining 19% reporting inadequate implementation in 1 or 2 weeks during the course of the entire 18 weeks. Teachers reported to have used all elements of the DRC adequately most of the 18 weeks (M = 0.78% of the time; SD = 0.23).

Intervention Effects

Results on all teacher- and parent-rated outcomes are depicted in Table 2. Only group by time interactions are discussed because of our focus on the program’s effectiveness, with significant interactions being displayed in Figure 2. Four-level models were used, but reduced to three- or two-level models when the intercept variance at a specific level was zero.

Table 2.

Multilevel Results on Teacher- and Parent-Rated Behavioral, Social, and Emotional Functioning.

	SWAN	SDQ ADHD	SDQ conduct	SDQ emotion	SDQ peer	SDQ prosocial	SSRS social
Teacher ratings (coefficient and SE)
Fixed effects
Constant	−1.378 (0.111)	7.473 (0.272)	2.134 (0.265)	1.982 (0.233)	2.236 (0.266)	6.122 (0.301)	32.225 (1.327)
Group (Time = 0)^a	0.028 (0.156)	0.690 (0.381)	0.707 (0.372)	−0.023 (0.327)	0.540 (0.373)	−0.715 (0.423)	−2.309 (1.863)
Time	0.009 (0.005)	0.004 (0.018)	0.009 (0.012)	0.016 (0.012)	−0.003 (0.011)	0.021 (0.015)	0.041 (0.063)
Group × Time	0.030 (0.008)***	−0.074 (0.025)**	−0.048 (0.017)**^b	−0.034 (0.017)	−0.028 (0.016)	0.039 (0.022)	0.269 (0.089)**
Random effects
$σ_{u 0}^{2}$ (school)	0.079 (0.070)	0.194 (0.369)	—	—	—	0.079 (0.674)	8.852 (8.584)
$σ_{u 01}$ (school)	—	−0.004 (0.016)	—	—	—	—	—
_{$σ_{u 1}^{2}$} (school)	—	0.003 (0.001)	—	—	—	—	—
$σ_{u 0}^{2}$ (classroom)	0.144 (0.109)	0.996 (0.580)	1.438 (0.499)	0.289 (0.494)	0.629 (0.655)	0.871 (0.959)	17.862 (14.285)
$σ_{u 01}^{2}$ (classroom)	−0.005 (0.003)	—	—	—	—	—	−0.716 (0.408)
_{$σ_{u 1}^{2}$} (classroom)	0.0006 (0.0002)	—	—	—	—	—	0.059 (0.028)
_{$σ_{u 0}^{2}$} (student)	0.238 (0.075)	1.005 (0.428)	1.052 (0.393)	1.645 (0.542)	2.256 (0.685)	2.084 (0.730)	37.370 (10.980)
_{$σ_{e}^{2}$}	0.098 (0.013)	1.779 (0.213)	1.248 (0.121)	1.335 (0.130)	1.097 (0.106)	2.135 (0.207)	21.814 (2.647)
Parent ratings (coefficient and SE)
Fixed effects
Constant	−0.665 (0.084)	6.365 (0.292)	1.804 (0.224)	2.336 (0.267)	1.764 (0.231)	8.189 (0.206)	54.267 (1.358)
Group (time = 0)^a	0.012 (0.119)	0.733 (0.411)	0.559 (0.314)	0.060 (0.377)	0.084 (0.332)	0.192 (0.290)	−1.876 (1.910)
Time	0.001 (0.003)	−0.024 (0.013)	−0.019 (0.010)	−0.014 (0.013)	−0.016 (0.011)	−0.003 (0.010)	0.043 (0.053)
Group × Time	0.002 (0.005)	−0.011 (0.019)	−0.006 (0.014)	0.006 (0.019)	0.008 (0.015)	−0.022 (0.014)	0.138 (0.075)
Random effects
_{$σ_{u 0}^{2}$} (school)	—	—	—	—	—	—	0.614 (8.483)
$σ_{u 01}$ (school)	—	—	—	—	—	—	—
_{$σ_{u 1}^{2}$} (school)	—	—	—	—	—	—	—
_{$σ_{u 0}^{2}$} (classroom)	0.044 (0.059)	0.732 (0.637)	—	—	—	—	—
$σ_{u 01}^{2}$ (classroom)	—	0.031 (0.017)	—	—	—	—	—
$σ_{u 1}^{2}$ (classroom)	—	0.001 (0.001)	—	—	—	—	—
$σ_{u 0}^{2}$ (student)	0.257 (0.068)	2.744 (0.716)	2.010 (0.325)	2.815 (0.453)	2.345 (0.363)	1.721 (0.272)	81.696 (14.774)
_{$σ_{e}^{2}$}	0.093 (0.009)	1.238 (0.155)	0.856 (0.085)	1.463 (0.146)	0.956 (0.095)	0.819 (0.081)	24.017 (2.390)

Note. SWAN = Strengths and Weaknesses of ADHD-symptoms and Normal-behavior scale (total score); SDQ = Strengths and Difficulties Questionnaire; Emotion = Emotional Problems; Peer = Peer problems; SSRS = Social Skills Rating Scale.

The fixed group effect represents group differences at baseline. The control group was used as reference group.

This effect was no longer significant when controlling for conduct problems at baseline.

p < .05. **p < .01. ***p < .001.

Figure 2.

Significant improvements over time for the intervention group compared with control group demonstrating positive effects of the PR program.

ADHD symptoms

As shown in Figure 2, teachers in the intervention group reported significant reductions of ADHD symptoms during the course of the intervention compared with teachers in the control group, both according to the SWAN (p < .001) and the SDQ ADHD (p = .004; see Table 2 for coefficients). The effect size was large for the SWAN (f² = .37), but small for the SDQ ADHD (f² = .01). The additional analysis investigating the possible moderating effect of gender on the program’s effectiveness revealed no significant better model fit when including the three-way gender by group by time interaction or the two-way interactions (all G² values (1) ≤ 0.32, p values ≥ .564), with similar improvements in favor of the intervention group when controlling for the main effect of gender (p < .001 for the SWAN, and p = .006 for the SDQ ADHD). Both intervention effects (on the SWAN and SDQ ADHD) remained significant when controlling for baseline levels of ADHD symptoms (p = .001 and p = .023, respectively). No significant reductions in favor of the intervention or control group were found for parent-rated ADHD symptoms as assessed by the SWAN and the SDQ ADHD (p = .720 and p = .566, respectively; see Table 2 for coefficients).

Conduct problems

A significant reduction of teacher-reported conduct problems was found in the intervention group compared with the control group as measured by SDQ Conduct Problems (p = .004, f² = .04; see Table 2 and Figure 2). Including gender as three- or two-way interaction did not improve model fit (all G² values (1) ≤ 0.43, all p values ≥ .512) and similar improvements in teacher-reported conduct problems were found for children in the intervention group compared with the control group when controlling for the main effect of gender (p = .006). However, when controlling for conduct problems at baseline, the reduction of conduct problems in the intervention compared with control group was no longer significant (p = .277). Parents did not report a significant reduction of SDQ Conduct Problems in favor of one of the two groups (p = .692; see Table 2).

Social and emotional functioning

Teachers reported a significant improvement of social skills of children in the intervention group compared with children in the control group (SSRS; p = .003, f² = .04; see Table 2 and Figure 2). The gender by group by time interaction and the two-way interactions with group or time did not significantly improve model fit (all G² values (1) ≤ 1.33, all p values ≥ .252) and similar improvements in teacher-reported social skills were found in the intervention group compared with the control group when controlling for gender (p = .002). The positive intervention effect on social skills remained significant when controlling for social skills at baseline (p = .040). No significant improvements in favor of the intervention group were found for teacher-reported Prosocial Behavior (p = .074), Emotional Problems (p = .052), or Peer Problems (p = .076) as assessed by the SDQ (see Table 2 for coefficients). No statistically significant improvements were found for parent-rated SSRS-Social Skills (p = .068), SDQ Prosocial Behavior (p = .118), SDQ Peer Problems (p = .581), and SDQ Emotional Problems (p = .973; see Table 2 for coefficients).

Discussion

The purpose of this study was to assess the efficacy of the PR Program on teacher- and parent-reported behavioral, social, and emotional functioning of children displaying ADHD symptoms at school. Results revealed positive effects on teacher-reported ADHD symptoms and conduct problems, and some positive effects on teacher-rated social skills. Improvements on ADHD symptoms were consistently found for both ADHD scales, although a large effect was found on the SWAN compared with a small effect on the SDQ ADHD. This discrepancy might be explained by higher sensitivity of the SWAN than the SDQ to detect changes in behavior as the SWAN assesses ADHD symptoms on a continuum ranging from severe problems in attentional functioning to extremely well-developed attentional skills (Young et al., 2009). Although results indicated that children in the intervention group displayed a larger decrement in conduct problems than children in the control group, this effect was no longer significant when controlling for baseline differences. The fact that teachers also reported some positive effects on social skills within the rather short intervention period of 18 weeks is most interesting, given that the primary focus of the intervention was on ADHD symptoms. Results on social-emotional functioning were somewhat inconsistent though, with positive effects on the teacher-rated SSRS but not on the SDQ scales assessing social functioning (Prosocial Behavior, Peer Problems, and Emotional Problems). Although additional analyses revealed significant effects for all three SSRS subscales (not reported in the Results), effects were larger for the Self-Control scale (f² = .04) than for the Cooperation and Assertion scales (f² = .01 and f² = .02, respectively). This seems consistent with our positive effects on ADHD symptoms as the Self-Control scale involves the capability to control temper in conflict situations (Gresham & Elliott, 1990), which is common in children with ADHD symptoms who often react impulsively (Daley & Birchwood, 2010). Overall, current results indicate that teachers mainly notice improvements in ADHD symptoms after using the PR program, with some improvements in social functioning.

Results are largely consistent with literature on behavioral teacher programs, revealing improvements in ADHD symptoms and social skills, but also in oppositional behavior (Miranda et al., 2002; Östberg & Rydell, 2012; Pelham et al., 2000). As in current study, effects are generally small (Daley et al., 2014; Sonuga-Barke et al., 2013). Common elements of those effective programs that were also part of the PR program include psycho-education, the use of classroom management strategies, and the use of a reward and time-out system. Treatment options recommended for children displaying ADHD symptoms not meeting full diagnostic criteria are less elaborate than for children diagnosed with ADHD (Subcommittee on Attention-Deficit, 2011). The development of behavioral teacher programs such as the PR program targeting children with ADHD symptoms may contribute to prevent exacerbation of problem behavior and possibly even contributes to a reduction of the number of children ultimately receiving a diagnosis of ADHD, although future research should further investigate this hypothesis.

Despite positive effects observed by teachers, all parent-rated outcomes were non-significant, indicating that no generalization of effects took place to the home setting. This is consistent with earlier work showing that interventions often fail to generalize to non-treatment settings (Abikoff, 2009), and can be explained by our exclusive focus on classroom behavior problems. To reduce problem behavior at home, multimodal programs involving additional parent trainings would be most appropriate (Drugli & Larsson, 2006; Östberg & Rydell, 2012).

Sustainability over time is often reported as a challenge difficult to achieve under “real-world” conditions after research is terminated, while being crucial for program effectivity (Durlak & DuPre, 2008; Han & Weiss, 2005; Wilson, Lipsey, & Derzon, 2003). Although future research is necessary to investigate sustainability of the PR program, this intervention could be of great value thanks to several advantages. The low costs involved in the PR program and the wide target group increase the likelihood of gaining (financial) support from key stakeholders at multiple levels (federal, district, and school level), a contextual factor important for long-term sustainability (Coffey & Horner, 2012). Besides that, thanks to the different intensity levels (universal program, and the individual program with three intensity levels) and the practical implementation instructions, the PR program can easily be adapted to changing circumstances (e.g., an increase of behavior problems or changing reward preferences as the child grows older), which is also necessary for a program to be sustainable (Han & Weiss, 2005). Moreover, the limited amount of time required to implement the program is likely to increase teachers’ willingness to implement the program (Han & Weiss, 2005). Whereas literature indicates that intensive training and consultation is required for adequate implementation (Coffey & Horner, 2012; Durlak & DuPre, 2008), current results suggest that the PR program was implemented with high fidelity despite the lack of teacher training. Program implementation was assessed through self-report instead of observations, but only 15% of the teachers consulted the helpdesk, which shows that teachers felt capable of implementing the program without additional support. This could due to teachers being acquainted to some extent with part of the techniques beforehand, and due to the practical examples and work sheets in the manual. Moreover, most teachers (98%) reported that they intended to use the program in the future, indicating a high satisfaction rate and perceived effectiveness of the PR program among teachers.

Current findings should be interpreted in light of some limitations. First, our teacher-rated positive effects are possibly inflated due to our unblinded assessments (i.e., teacher ratings) and by teachers’ involvement in treatment delivery, as teachers invested in the program’s success or perhaps strongly believed in its efficacy (Jadad & Enkin, 2008; Sonuga-Barke et al., 2013). Positive effects of behavioral ADHD programs have been found to disappear when using less-proximate raters not involved in treatment delivery, although effects on conduct problems do persist (Daley et al., 2014; Sonuga-Barke et al., 2013). More objective instruments (e.g., classroom observations) are thus necessary to complement current results. Nonetheless, teachers’ perceptions on behavior improvements will likely resort to more positive teacher–child interactions, an important predictor of school functioning (Hoza, 2007). Second, despite random allocation to intervention or control group, there were somewhat more boys in the intervention group than in the control group. Additional analyses, however, revealed that gender did not moderate the positive effects of the PR program and that the beneficial effects of the PR program remained intact when controlling for gender. Third, treatment fidelity was assessed through teacher self-report rather than an independent classroom observation. Although observations are regarded as more objective and should therefore be preferred (Durlak & DuPre, 2008), practical constraints precluded implementing observations in the current study. Nonetheless, program’s beneficial effects and high teacher satisfaction support the high treatment fidelity of our program. Last, future research is necessary to assess whether the program’s treatment fidelity can be confirmed through observations and to assess long-term effectiveness.

In conclusion, results of this study show that the PR program improves teacher-rated ADHD symptoms, and also provides some beneficial effects on social skills. Implementation fidelity and satisfaction among teachers was high. The program’s positive effects await further support by more objective measures in future research, for example, using raters not involved in treatment delivery. Nonetheless, our behavioral teacher program relying on a manual not requiring expert training holds promise for sustainable large-scale implementation and may help preventing escalation of problem behavior in children with ADHD symptoms.

Footnotes

Acknowledgements

Authors thank families and teachers participating in this project and Dr. Weeda for his statistical advice. Furthermore, we would like to thank Sabrina Schuck, PhD; James Swanson, PhD; Tim Wigal, PhD; Andrew Greiner; Dr. Lisa Burrows-MacLean, PhD; and Arga Paternotte for their advice and help during the development of the Positivity & Rules Program.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by a number of charity organizations including Children’s Welfare Fund (Grant 5930790), Nederlandse Stichting voor het Gehandicapte Kind (Grant 20110028), Weeshuis der Doopsgezinden, and Stichting Zonnige Jeugd.

Author Biographies

Betty Veenman is a PhD student at the Child Study Group of the Clinical Neuropsychology section. Her research concentrates on assessing the efficacy of the Positivity & Rules program targeting ADHD symptoms within the classroom of primary school teachers.

Marjolein Luman is Assistant Professor at the Child Study Group, section of Clinical Neuropsychology. Her research focuses on neurocognitive functions in developmental disorders such as ADHD and oppositional defiant disorder (ODD).

Jan Hoeksma is Associate Professor, and works at the section of Clinical Neuropsychology. His research focuses on dynamic models and the dynamics of emotions in particular.

Karlijn Pieterse is a Pediatric Neuropsychologist and expert in the field of parent and child coaching

Jaap Oosterlaan is Professor in Pediatric Neuropsychology and Chair of the Child Study Group of the Clinical Neuropsychology section. His research concentrates on neurocognitive deficits in diverse childhood conditions, with one line of research focusing on childhood disruptive behavior disorders and ADHD in particular, and the other line of research targeting medical conditions affecting the central nervous system, such as premature birth.

References

Abikoff

(2009). ADHD psychosocial treatments: Generalization reconsidered. Journal of Attention Disorders, 13, 207-210.

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text rev.). Washington, DC: Author.

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.

Atkinson

Hollis

(2010). NICE guideline: Attention deficit hyperactivity disorder. Archives of Disease in Childhood-Education & Practice Edition), 95, 24-27.

Bradshaw

C. P.

Mitchell

M. M.

Leaf

P. J.

(2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12, 133-148.

Charach

Carson

Fox

Ali

M. U.

Beckett

Lim

C. G.

(2013). Interventions for preschool children at high risk for ADHD: A comparative effectiveness review. Pediatrics, 131(5), e1584-e1604.

Coffey

J. H.

Horner

R. H.

(2012). The sustainability of schoolwide positive behavior interventions and supports. Exceptional Children, 78, 407-422.

Daley

Birchwood

(2010). ADHD and academic performance: Why does ADHD impact on academic performance and what can be done to support ADHD children in the classroom? Child: Care, Health and Development, 36, 455-464.

Daley

van der Oord

Ferrin

Danckaerts

Doepfner

Cortese

, . . . European ADHD Guidelines Group. (2014). Behavioral interventions in attention-deficit/hyperactivity disorder: A meta-analysis of randomized controlled trials across multiple outcome domains. Journal of the American Academy of Child & Adolescent Psychiatry, 53, 835-847.e1-5.

10.

Diamantopoulou

Rydell

A. M.

Thorell

L. B.

Bohlin

(2007). Impact of executive functioning and symptoms of attention deficit hyperactivity disorder on children’s peer relations and school performance. Developmental Neuropsychology, 32(1), 521-542.

11.

Drugli

M. B.

Larsson

(2006). Children aged 4-8 years treated with parent training and child therapy because of conduct problems: Generalisation effects to day-care and school settings. European Child & Adolescent Psychiatry, 15, 392-399.

12.

Durlak

J. A.

DuPre

E. P.

(2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327-350.

13.

Evans

S. W.

Owens

J. S.

Bunford

(2014). Evidence-based psychosocial treatments for children and adolescents with attention-deficit/hyperactivity disorder. Journal of Clinical Child & Adolescent Psychology, 43, 527-551.

14.

Fabiano

G. A.

Pelham

W. E.

Coles

E. K.

Gnagy

E. M.

Chronis-Tuscano

O’Connor

B. C.

(2009). A meta-analysis of behavioral treatments for attention-deficit/hyperactivity disorder. Clinical Psychology Review, 29, 129-140.

15.

Faraone

S. V.

Wilens

T. E.

Petty

Antshel

Spencer

Biederman

(2007). Substance use among ADHD adults: Implications of late onset and subthreshold diagnoses. American Journal on Addictions, 16(Supp1.), 24-34.

16.

Goedhart

Treffers

Widenfelt

(2003). Vragen naar psychische problemen bij kinderen en adolescenten: De Strengths and Difficulties Questionnaire [Questions regarding psychological problems in children and adolescents: The Strengths and Difficulties Questionnaire]. Maandblad Geestelijke Volksgezondheid, 58, 1018-1035.

17.

Goodman

(1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581-586.

18.

Gresham

F. M.

Elliott

S. N.

(1990). Social skills rating system: Manual. Circle Pines, MN: American Guidance Service,

19.

Han

S. S.

Weiss

(2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33, 665-679.

20.

Heck

R. H.

Thomas

S. L.

Tabata

L. N.

(2013). Multilevel and longitudinal modeling with IBM SPSS. New York, NY: Routledge.

21.

Holmes

Lawson

Langley

Fitzpatrick

Trumper

Pay

. . . Thapar

(2004). The child attention-deficit hyperactivity disorder teacher telephone interview (CHATTI): Reliability and validity. The British Journal of Psychiatry, 184, 74-78.

22.

Hong

S. B.

Dwyer

Kim

J. W.

Park

E. J.

Shin

M. S.

Kim

B. N.

. . . Cho

S. C.

(2014). Subthreshold attention-deficit/hyperactivity disorder is associated with functional impairments across domains: A comprehensive analysis in a large-scale community study. European Child & Adolescent Psychiatry, 23, 627-636.

23.

Hoza

(2007). Peer functioning in children with ADHD. Journal of Pediatric Psychology, 32, 655-663.

24.

Jadad

A. R.

Enkin

M. W.

(2008). Bias in randomized controlled trials. In Randomized Controlled Trials: Questions, Answers, and Musings, Second Edition (pp. 29-47). Oxford, England: Blackwell Publishing Ltd.

25.

Kutcher

Aman

Brooks

S. J.

Buitelaar

van Daalen

Fegert

Tyano

(2004). International consensus statement on attention-deficit/hyperactivity disorder (ADHD) and disruptive behaviour disorders (DBDs): Clinical implications and treatment practice suggestions. European Neuropsychopharmacology, 14, 11-28.

26.

Lubke

G. H.

Muthen

Moilanen

I. K.

McGough

J. J.

Loo

S. K.

Swanson

J. M.

. . . Smalley

S. L.

(2007). Subtypes versus severity differences in attention-deficit/hyperactivity disorder in the Northern Finnish Birth Cohort. Journal of the American Academy of Child & Adolescent Psychiatry, 46, 1584-1593.

27.

Miranda

Presentación

M. J.

Soriano

(2002). Effectiveness of a school-based multicomponent program for the treatment of children with ADHD. Journal of Learning Disabilities, 35, 547-563.

28.

Molina

B. S.

Flory

Hinshaw

S. P.

Greiner

A. R.

Arnold

L. E.

Swanson

J. M.

Wigal

(2007). Delinquent behavior and emerging substance use in the MTA at 36 months: Prevalence, course, and treatment effects. Journal of the American Academy of Child & Adolescent Psychiatry, 46, -1027-1039.

29.

MTA Cooperative Group. (1999). A 14-month randomized clinical trial of treatment strategies for attention-deficit/hyperactivity disorder. Archives of General Psychiatry, 56, 1073-1086.

30.

Oosterlaan

Baeyens

Scheres

Antrop

Roeyers

Sergeant

(2008). Vragenlijst voor gedragsproblemen bij kinderen 6-16 jaar (Handleiding) [Questionnaire on behavioral problems in children aged 6-16]. Amsterdam, The Netherlands: Harcourt.

31.

Östberg

Rydell

A. M.

(2012). An efficacy study of a combined parent and teacher management training programme for children with ADHD. Nordic Journal of Psychiatry, 66, 123-130.

32.

Parker

Wales

Chalhoub

Harpin

(2013). The long-term outcomes of interventions for the management of attention-deficit hyperactivity disorder in children and adolescents: A systematic review of randomized controlled trials. Psychology Research and Behavior Management, 6, 87-99.

33.

Pelham

W. E.

Fabiano

G. A.

(2008). Evidence-based psychosocial treatments for attention-deficit/hyperactivity disorder. Journal of Clinical Child & Adolescent Psychology, 37, 184-214.

34.

Pelham

W. E.

Gnagy

E. M.

Greenslade

K. E.

Milich

(1992). Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorders. Journal of the American Academy of Child & Adolescent Psychiatry, 31, 210-218.

35.

Pelham

W. E.

Gnagy

E. M.

Greiner

A. R.

Hoza

Hinshaw

S. P.

Swanson

J. M.

. . . McBurnett

(2000). Behavioral versus behavioral and pharmacological treatment in ADHD children attending a summer treatment program. Journal of Abnormal Child Psychology, 28, 507-525.

36.

Polanczyk

de Lima

M. S.

Horta

B. L.

Biederman

Rohde

L. A.

(2007). The worldwide prevalence of ADHD: A systematic review and metaregression analysis. American Journal of Psychiatry, 164, 942-948.

37.

Raudenbush

S. W.

Spybrook

Congdon

Liu

X. F.

Martinez

Bloom

(2011). Optimal design software for multi-level and longitudinal research (Version 3.01)[Software]. Available from www. wtgrantfoundatio n. org.

38.

Sattler

J. M.

(1992). Assessment of children: WISC-III and WPPSI-R supplement. San Diego, CA: Author.

39.

Scherbaum

C. A.

Ferreter

J. M.

(2009). Estimating statistical power and required sample sizes for organizational research using multilevel modeling. Organizational Research Methods, 12, 347-367.

40.

Selya

A. S.

Rose

J. S.

Dierker

L. C.

Hedeker

Mermelstein

R. J.

(2012). A practical guide to calculating Cohen’s f², a measure of local effect size, from PROC MIXED. Frontiers in Psychology, 3, 111.

41.

Shaw

Hodgkins

Caci

Young

Kahle

Woods

A. G.

Arnold

L. E.

(2012). A systematic review and analysis of long-term outcomes in attention deficit hyperactivity disorder: Effects of treatment and non-treatment. BMC Medicine, 10(1), 99-113.

42.

Sonuga-Barke

E. J.

Brandeis

Cortese

Daley

Ferrin

European ADHD Guidelines Group. (2013). Nonpharmacological interventions for ADHD: Systematic review and meta-analyses of randomized controlled trials of dietary and psychological treatments. American Journal of Psychiatry, 170, 275-289.

43.

IBM Corp. (2011). IBM SPSS statistics for Windows. Armonk,NY: IBM Corp.

44.

Subcommittee on Attention-Deficit. (2011). ADHD: Clinical practice guideline for the diagnosis, evaluation, and treatment of attention-deficit/hyperactivity disorder in children and adolescents. Pediatrics, 128, 1007-1022.

45.

Swanson

Schuck

Mann

Carlson

Hartman

Sergeant

. . . Wigal

(2012). Categorical and dimensional definitions and evaluations of symptoms of ADHD: History of the SNAP and the SWAN Rating Scales. International Journal of Educational and Psychological Assessment, 10(1), 51-70.

46.

Van der Oord

Prins

P. J.

Oosterlaan

Emmelkamp

P. M

. (2008). Efficacy of methylphenidate, psychosocial treatments and their combination in school-aged children with ADHD: A meta-analysis. Clinical Psychology Review, 28, 783-800.

47.

Van der Oord

Van der Meulen

Prins

P. J.

Oosterlaan

Buitelaar

Emmelkamp

P. M

. (2005). A psychometric evaluation of the social skills rating system in children with attention deficit hyperactivity disorder. Behaviour Research and Therapy, 43, 733-746.

48.

Van Widenfelt

B. M.

Goedhart

A. W.

Treffers

P. D.

Goodman

. (2003). Dutch version of the Strengths and Difficulties Questionnaire (SDQ). European Child & Adolescent Psychiatry, 12, 281-289.

49.

Verhage

(1983). Het coderen van het opleidingsniveau voor researchdoeleinden [Educational classification system for research purposes: revised version]. Internal publication Groningen, the Netherlands: Academic Hospital Groningen, State University Groningen.

50.

Webster-Stratton

Jamila Reid

Stoolmiller

(2008). Preventing conduct problems and improving school readiness: Evaluation of the incredible years teacher and child training programs in high-risk schools. Journal of Child Psychology and Psychiatry, 49, 471-488.

51.

Wilens

T. E.

Biederman

Brown

Tanguay

Monuteaux

M. C.

Blake

Spencer

T. J.

(2002). Psychiatric comorbidity and functioning in clinically referred preschool children and school-age youths with ADHD. Journal of the American Academy of Child & Adolescent Psychiatry, 41, 262-268.

52.

Wilson

S. J.

Lipsey

M. W.

(2007). School-based interventions for aggressive and disruptive behavior: Update of a meta-analysis. American Journal of Preventive Medicine, 33, S130-S143.

53.

Wilson

S. J.

Lipsey

M. W.

Derzon

J. H.

(2003). The effects of school-based intervention programs on aggressive behavior: A meta-analysis. Journal of Consulting and Clinical Psychology, 71, 136-149.

54.

Young

D. J.

Levy

Martin

N. C.

Hay

D. A.

(2009). Attention deficit hyperactivity disorder: A Rasch analysis of the SWAN Rating Scale. Child Psychiatry & Human Development, 40, 543-559.