Informant Discrepancies in the Assessment of Attention-Deficit/Hyperactivity Disorder

Abstract

An attention-deficit/hyperactivity disorder (ADHD) diagnosis requires symptoms to be present across two or more settings, thus requiring information from multiple informants. Research consistently shows low to moderate agreement between parents and teachers; however, the mechanisms underlying these discrepancies remain unclear. This study examined (a) agreement between parents and teachers, (b) effects of using different combination rules in assigning diagnoses, and (c) the role of contextual influences and/or personal biases in informants’ reports. Fifty-five children, their parents, and teachers participated. Parent and teacher ratings on the Attention-Deficit/Hyperactivity Disorder Rating Scale–Fourth edition (ADHD-RS-IV) and clinician ratings on the Behavioral Observation of Students in Schools (BOSS) were obtained. Results indicated moderate agreement among parent and teacher ratings on the ADHD-RS. Diagnostically, the rule for combining information from multiple informants dramatically altered the ADHD classification assigned to the child. With regard to rater differences, the clinician-rated school observation gave some support for the notion that ratings are person rather than context specific.

Keywords

ADHD psychopathologies and symptomatologies diagnostic classification models measurement elementary school participants

Attention-deficit/hyperactivity disorder (ADHD) is a chronic condition which begins in childhood and is characterized by pervasive and developmentally inappropriate levels of hyperactivity, impulsivity, and/or inattention. Symptoms must occur across two or more settings and lead to significant impairment in functioning (Diagnostic and Statistical Manual of Mental Disorders [5th ed.; DSM-5; American Psychiatric Association, 2013]). To establish the pervasiveness of symptomatology and impairment, clinicians need to obtain information regarding the child’s behavior across multiple contexts (Frazier & Youngstrom, 2006), typically via parents and teachers (Barkley, 1998; Pelham, Fabiano, & Massetti, 2005). Informants should agree to some extent on the presence and severity of symptoms and associated impairment. However, this is often not the case, and as such the “cross-situationality” specification of the diagnostic criteria has become one of the most problematic and controversial issues in diagnosing ADHD (Healey, Miller, Castelli, Marks, & Halperin, 2008).

In their seminal paper, Achenbach, McConaughy, and Howell (1987) first documented disparity in across-informant reporting with their meta-analyses, showing that among 119 studies, parent, child, teacher, support worker, and clinician ratings of a child’s symptomatology differed greatly. Agreement rates showed an average effect size of only 0.2, falling in the low to moderate range of agreement (Achenbach et al., 1987). Numerous studies have since also demonstrated low to moderate rates of agreement across multiple informants rating symptoms of child psychopathology in general (De Los Reyes & Kazdin, 2005; Youngstrom, Loeber, & Stouthamer-Loeber, 2000), and for an ADHD diagnosis of any subtype (Malhi, Singhi, & Sidhu, 2008, kappa statistic [κ] = .11; Mitsis, McKay, Schulz, Newcorn, & Halperin, 2000, κ = .20; Sollie, Larsson, & Mørch, 2012, κ = .24 for mother–teacher agreement and κ = .43 for father–teacher agreement).

Several studies have examined different rules for combining information across raters. These have included the most lenient “OR” rule (total number of symptoms reported regardless of informant, for example, a parent could report six symptoms within a subtype and a teacher none; or a parent three and a teacher three within a subtype) which does not require cross-situationality of symptoms; the “AND” rule (i.e., parents and teachers must rate six or more symptoms each, but these do not need to be the same symptoms); and the “AND duplicated” rule (i.e., both informants are required to rate a specific symptom as present for it to be endorsed). Wolraich et al. (2004) documented the percentage of children meeting criteria as 10% using the AND duplicated rule and 35% using the OR rule. Specifically, the disparity across the different combinational rules was more pronounced for inattentive symptoms (17% met criteria for inattentive ADHD using the AND rule and 47% using the OR rule). Moreover, Valo and Tannock (2010) showed that, in a clinically referred sample, 95% of children met criteria for ADHD using the most lenient OR rule which does not require cross-situational presence of symptoms and 42% using the AND rule. More prominent was the effect on subtype distribution, where 83% of children were classified as combined type using the OR rule, compared with 19% using the AND rule (Valo & Tannock, 2010). Interestingly, Malhi et al. (2008) reported that increasing the stringency of criteria for combining multiple informants’ reports increased rates of children in the hyperactive/impulsive subtype (5.8%-19.3%) and inattentive subtype (15.0%-28.6%). However, those meeting criteria for combined subtype decreased dramatically (62.8%-5.9%) with the increased stringency.

In the absence of formal guidelines on how to combine information across informants, clinicians are left to decide what to do with this information (Valo & Tannock, 2010). If ratings differ significantly, it can often be impossible to know which informant is correct, or whether they are both correct. Two factors that could account for the lack of agreement between informants’ ratings are source bias and contextual effects. Source bias involves a systematic bias in the responses of one or more informants (De Los Reyes & Kazdin, 2005), whereas contextual effects are indicative of how children’s behavior generally varies across environments (Dumenci, Achenbach, & Windle, 2011).

Given the known discrepancies between raters, clinicians may consider conducting their own observations of the child (Barkley, 2003). Naturalistic observations are recommended for children being assessed for ADHD as these children often behave differently in one-on-one settings. A classic example being the clinician’s office, where the child receives the full attention of an adult, often in a very rewarding manner (Merrell, 2000). A commonly used tool is the Behavioral Observation of Students in Schools (BOSS; Shapiro, 2004), which measures a range of academic engagement and non-engagement behaviors in the classroom environment. The BOSS has received considerable empirical support in assessing disruptive behavior disorders, particularly ADHD (Kofler, Rapport, & Alderson, 2008; Volpe, DiPerna, Hintze, & Shapiro, 2005).

Given the ongoing debate and confusion around how clinicians are to handle conflicting informant reports when making a diagnosis of ADHD, this study examined the reporting of ADHD symptoms by parents, teachers, and a clinician-rated school observation. First, parent and teacher ratings were correlated to examine levels of agreement within our sample. Then, parent and teacher ratings alone were reported to indicate diagnostic levels using one rater only. Next, ratings were combined using three combinational rules (OR rule, AND rule, AND duplicated rule) to highlight the effects that these rules had on diagnostic rates within this sample. Finally, the novel aspect of this study involved the inclusion of a clinician-rated school observation used to explore whether source or contextual effects best explain the discrepancies in parent and teacher ratings of ADHD symptomatology.

Method

Participants

Fifty-five children aged between 6 and 12 years (44 males and 11 females with a mean age of 104.33 months, SD = 23.67), and their families and teachers participated in the present study. The children were rated by at least one rater (i.e., parent and/or teacher) as displaying six or more symptoms of ADHD. They were recruited from two sources: (a) a preexisting database within the Department of Psychology at the University of Otago, New Zealand (NZ), which provided a record of children with ADHD and their families who had participated in previous research, and had consented to being contacted about future research; and (b) children who were referred from the Southern District Health Board’s Paediatric Outpatients and Child and Family Mental Health Services.

As concluded by a diagnostic assessment using the Kiddie Schedule for Affective Disorders and Schizophrenia (K-SADS) semi-structured interview with parents (Kaufman et al., 1997), combined with parent and teacher ratings, 44 of the 55 participating children (80%) met formal criteria for ADHD and 11 children (20%) displayed significant symptoms but did not meet full criteria due to lack of cross-situationality or significant impairment. Of those who met full criteria, 19 were classified as ADHD-predominantly inattentive subtype (34.5%), five as ADHD-predominantly hyperactive/impulsive subtype (9.1%), 17 as ADHD-combined subtype (30.9%), and three as ADHD-not otherwise specified (5.5%). Furthermore, 15 children (27.3%) were taking medication for behavior management (Ritalin, Rubifen, Concerta, and Methamphetamine).

Of the 44 children who met formal criteria for ADHD, 41 families consented to a school observation, the data for which were used in the final set of analyses comparing parent, teacher, and school observation ratings.

Information regarding parents’ age, ethnicity, education (highest qualification), and current income was obtained. Demographic data were missing for one mother (1.8%) and six fathers (10.9%). Mothers of the children were aged 27 to 59 years (M = 39.29, SD = 7.16). Of these, 78.2% were New Zealand European, 7.3% were New Zealand European/Māori, 5.5% were Māori, 3.6% were British, and 1.8% from each of Australian and Chinese ethnicities. The mean highest qualification for mothers was “NZ seventh form certificate,” with a range of “some high school” to “postgraduate degree.” The average current income for mothers was between NZ$15,001 and NZ$20,000 annually, with a range of NZ$1-NZ$5,000 to NZ$70,000-NZ$100,000 annually. Fathers of children were aged 28 to 63 years (M = 42.02, SD = 7.45). Of these, 74.5% were New Zealand European, 7.3% British, 3.6% Australian, and 1.8% from each of New Zealand European/Māori and Pasifika ethnicities. The mean highest qualification level for fathers was “NZ seventh form certificate” with a range of “some high school” to “postgraduate degree.” The mean current income for fathers was NZ$35,001 to NZ$40,000 annually, with a range of NZ$1-$5,000 to NZ$100,001 or more.

Measures

Attention-Deficit/Hyperactivity Disorder Rating Scale–Fourth edition (ADHD-RS-IV)

The ADHD-RS-IV (DuPaul, Power, Anastopoulos, & Reid, 1998) is a measure of ADHD symptomatology derived from the Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; DSM-IV-TR; American Psychiatric Association, 2000) diagnostic criteria, which can be completed for home and school environments. Eighteen symptoms of ADHD are rated by frequency on a 4-point Likert-type scale (0 = never or rarely, 1 = sometimes, 2 = often, and 3 = very often). The overall score out of 54 is indicative of symptom severity, with higher scores corresponding to greater severity. The score can be divided into hyperactive/impulsive and inattentive symptoms (each out of 27), such that scores within each subtype can be obtained. The ADHD-RS-IV has demonstrated strong test–retest reliability (.78-.86 for home and .88-.90 for school) and internal consistency (.86-.92; Collett, Ohan, & Meyers, 2003). The ADHD-RS was completed by parents (52 maternal ratings, one mother and father rated, one father rated) and teachers.

Behavioral Observation of Students in Schools (BOSS)

The BOSS (Shapiro, 2004) is a clinician-rated systematic direct observation system designed specifically for use in a classroom setting. It utilizes a momentary time sampling procedure to code academic engagement or non-engagement over a defined time period. Academic engagement behaviors are defined as those in which the individual is on task (Shapiro, 2004), and can be further delineated as the categories of active or passive engagement behaviors. Academic non-engagement is characterized by off-task behaviors (Shapiro, 2004). Three categories of off-task behaviors were coded as follows: off-task motor, which represented motor activity not associated with the academic task; off-task verbal, which represented any audible verbalizations not associated with the academic task; and off-task passive, which represented any behavior that was passive but persisted for more than 3 s (e.g., staring out the window, looking around the room). The BOSS uses a partial interval method, where the target child’s behavior was rated every 15 s on all of the five categories of behavior. Raw scores of behaviors in each category are converted into percentages of the time observed. A high score on the non-engagement domain is indicative of more severe off-task behavior. The inter-observer reliability of the BOSS was found to be very high, with agreement rates ranging between .90 and 1.0, which was especially important for the present study as there were multiple observers. Specific to ADHD, DuPaul et al. (2004) provided support for the discriminant validity of the BOSS, with effect sizes between 0.53 and 1.28 shown in discriminating children with ADHD from their typically developing peers.

For this study, acceptable inter-observer agreement was required prior to conducting the school observations. To establish the level of inter-observer agreement, unweighted kappa statistics (Cohen, 1960) were calculated over a 30-min observation (120 intervals, six categories) between the primary researcher and the two secondary researchers. Kappa values of .90 and .84 were reported. Interpreted in accordance with Viera and Garrett (2005), these values fall within the range of almost perfect agreement.

Procedure

Once the formal consent forms were obtained, parents were requested to fill out a set of parent rating scales, as per the instructions on the top of each rating scale. If a child had been prescribed medication, parents were asked to rate their child off medication. Once completed, the families were asked to return the rating scales in the envelope provided. The parent information pack also included the ADHD-RS-IV school form and an information sheet for teachers with instructions for completing the rating scales, and guidelines for rating the child off medication. Parents were asked to pass the forms on to a teacher who knew their child well. Teachers were requested to complete the forms and return them in the envelope provided.

An appointment was then scheduled, which lasted 3 hr, where the child completed a range of tasks (these were part of the larger study and not used in the present study). At the same time, the parent was interviewed on the relevant sections of the K-SADS (the school adaptation and social relations form and ADHD screener and supplement) and on additional sections that were relevant to the larger study (Oppositional Defiant Disorder and Conduct Disorder screeners and supplements). After the interview, the parent completed a number of additional questionnaires (also not included in the present study). Following their appointment, parents received a NZ$40 petrol voucher, and children received a NZ$10 toy or movie voucher in recognition of the time spent and travel costs involved in participating in the study.

School observation

If consent for a school observation was granted, a different researcher, who had not had any contact with the family, nor been involved with the above-mentioned assessment appointment, contacted the child’s school. This was so that the child would not know the person nor be aware that their behavior was being observed. The child’s teacher was contacted to request permission to visit the classroom and complete a 30-min school observation during an academic subject, at a time that best suited the teacher. Each teacher was informed that the child was unaware that he or she was being observed, and to protect the privacy of the child, the teacher was asked to introduce the researcher as a student who was there to observe the class.

Data Analysis

Based on the previous literature (Malhi et al., 2008; Mitsis et al., 2000; Narad et al., 2015; Sollie et al., 2012) reporting a median correlation of .33 between parent and teacher ratings of ADHD symptomatology, prior to the study’s commencement, a power analysis was conducted to gauge the necessary sample size required to find a similar or larger correlation. This showed that given alpha = .05 and power = 0.80, a sample size of 55 participants would be required to find a similar magnitude of effect between the variables. The library “pwr” in R was used to conduct the power calculations (pwr: Basic Functions for Power Analysis. R package version 1.1-3. https://CRAN.R-project.org/package=pwr).

Pearson product–moment correlations were used to establish the relations between parent and teacher ratings, and the relations between both informants’ reports and the clinician-rated school observation data. Kappa statistics were used to determine the magnitude of agreement between informants.

Results

Parent and Teacher Agreement When Rating the Symptoms of ADHD

The relations and level of agreement between parents and teachers were analyzed (a) for total number of symptoms endorsed, (b) for total number of hyperactive/impulsive and inattentive symptoms endorsed, and (c) on an individual symptom level, using Pearson product–moment correlations and kappa statistics. Kappa statistics were interpreted as representing less than chance for values <0, slight agreement for values .01 to .20, fair agreement for values .21 to .40, moderate agreement for values .41 to .60, substantial agreement for values .61 to .80, and almost perfect agreement for values .81 to .99 (Viera & Garrett, 2005).

There was a significant positive correlation between the total number of symptoms endorsed by parents and teachers (r = .251, p < .05). The kappa statistic of .292 was indicative of fair agreement between parent and teacher ratings. Similarly, for hyperactive/impulsive symptoms, there was a significant positive correlation between parent and teacher ratings (r = .435, p < .01). The kappa statistic of .356 was indicative of fair agreement. In contrast, the correlation between parent-rated and teacher-rated inattention was not significant (r = .099, p = .236). The kappa statistic of .096 fell within the range of slight agreement.

As depicted in Table 1 for hyperactive/impulsive symptoms and Table 2 for inattentive symptoms, significant positive correlations were found for six hyperactive/impulsive symptoms (3. runs/climbs, 4. difficulties playing quietly, 5. on the go, 6. talks excessively, 7. blurts out, and 8. difficulties taking turns) and two inattentive symptoms (1. careless mistakes and 6. avoids tasks). Parent–teacher agreement fell within the fair agreement range for three hyperactive/impulsive symptoms, slight agreement range for five symptoms, and one symptom fell within the less than chance range. Agreement for all inattentive symptoms fell within the slight agreement or less than chance ranges.

Table 1.

Correlations Among Parent and Teacher Ratings of Hyperactive-Impulsive Symptoms on the ADHD-RS-IV.

Parent ratings	Teacher ratings
Parent ratings	1	2	3	4	5	6	7	8	9
1. Fidgets	.046
2. Leaves seat		.563**
3. Runs/climbs			.421**
4. Play quietly				.314**
5. On the go					.379**
6. Talks excessively						.480**
7. Blurts out							.289*
8. Taking turns								.318**
9. Interrupts									.171

Note. ADHD-RS-IV = Attention-Deficit/Hyperactivity Disorder Rating Scale–Fourth edition.

Correlation is significant at the .05 level (one-tailed). **Correlation is significant at the .01 level (one-tailed).

Table 2.

Correlations Among Parent and Teacher Ratings of Inattentive Symptoms on the ADHD-RS-IV.

Parent ratings	Teacher ratings
Parent ratings	1	2	3	4	5	6	7	8	9
1. Careless mistakes	.413**
2. Sustained attention		.166
3. Doesn’t listen			.037
4. Instructions				.062
5. Organizing					.010
6. Avoids tasks						.292*
7. Loses things							.165
8. Distracted								−.008
9. Forgetful									−.031

Note. ADHD-RS-IV = Attention-Deficit/Hyperactivity Disorder Rating Scale–Fourth edition.

Correlation is significant at the .05 level (one-tailed). **Correlation is significant at the .01 level (one-tailed).

Distribution of Subtype Classifications Across Informants

To illustrate the implications of multiple informant reporting, diagnostic classifications using parent and/or teacher reports on the ADHD-RS were assigned in a number of stages (see Table 3). First, parent and teacher ratings were considered in isolation. Then, to test methods of establishing cross-situationality, diagnostic classifications were assigned using three different rules for combining parent and teacher ratings: (a) OR rule, ≥six symptoms regardless of informant; (b) AND rule, ≥six symptoms by both informants (i.e., both parents and teachers are required to rate the child as displaying six or more symptoms); and (c) AND duplicated rule, ≥six duplicated symptoms. (i.e., parent and teacher must both rate the same symptom for it to be endorsed).

Table 3.

Distribution of Diagnostic Classifications Based on Parent, Teacher, and Different Combinational Rules for Ratings of ADHD Symptoms.

ADHD diagnosis	Parent (≥6 parent)	Teacher (≥6 teacher)	OR rule (6 ≥ total, from either informant)	AND rule (6 ≥ from both)	AND duplicated rule (6 ≥ from both)
Hyperactive/impulsive subtype	3	4	3	4	4
Inattentive subtype	10	12	16	8	7
Combined subtype	26	15	28	11	8
Total	39	31	47	23	19

Note. ADHD = attention-deficit/hyperactivity disorder.

Of the 55 participating children, three were rated by parents as hyperactive/impulsive and four by teachers. When combining informant ratings, three children were rated by parents and teachers as collectively displaying six or more hyperactive/impulsive symptoms (OR rule), four as symptomatic by both informants (AND rule), and four children when informants were required to endorse the same symptoms (AND duplicated rule).

Ten of the 55 participants were rated by parents as inattentive, and 12 were rated by teachers as inattentive. When combining informant ratings, 16 children were rated by parents and teachers as collectively displaying six or more inattentive symptoms (OR rule), eight as symptomatic by both informants (AND rule), and seven children when informants were required to endorse the same symptoms (AND duplicated rule).

For the combined subtype, characterized by six or more symptoms in both domains, 26 children were rated by parents and 15 children were rated by teachers. When combining informant ratings, 28 children were rated by parents and teachers as collectively displaying six or more hyperactive/impulsive and inattentive symptoms (OR rule), 11 as symptomatic by both informants (AND rule), and eight children when duplicated symptoms were required (AND duplicated rule).

Parent, Teacher, and Clinician Agreement, Using Observations of Children’s Behavior in the Classroom

The following analyses were conducted with the subsample (n = 41) of children who received a diagnosis of ADHD using the K-SADS summary score and consented to a school observation being conducted. To investigate the possible mechanisms underlying differences in parent and teacher symptom ratings, Pearson product–moment correlations were calculated between parent and teacher ratings on the ADHD-RS-IV and clinician-rated school observations using the BOSS.

Consistent with the results of the full sample, a significant positive correlation was found between parent and teacher ADHD-RS-IV total symptom ratings within this subsample (r = .366, p < .001). There was a significant positive correlation between teacher-rated total symptoms and BOSS-total off-task behavior (r = .264, p < .05) but not between parent-rated total symptoms and BOSS-total off-task behavior (i.e., motor, verbal, and passive off-task behavior).

There were significant positive correlations between parent and teacher ADHD-RS-IV ratings of hyperactive/impulsive symptoms (r = .571, p < .001). The BOSS-hyperactive/impulsive score (i.e., motor and verbal off-task behavior) was also significantly correlated with both parent (r = .309, p < .05) and teacher (r = .353, p < .05) ratings of hyperactivity/impulsivity.

The opposite was the case for inattentive symptoms where there were no significant correlations between parent, teacher, and the clinician-rated BOSS-inattentive scores (i.e., passive off-task behavior), indicating high variability among the raters for this symptom domain.

Discussion

The present study sought to broaden the understanding of informant discrepancies in ratings of children’s symptoms of ADHD.

Parent and Teacher Agreement When Rating the Symptoms of ADHD

The first aim was to replicate and extend previous research regarding the relations between parent and teacher ratings of ADHD symptoms. There was a significant positive correlation between parent and teacher total and hyperactive/impulsive scores. As hypothesized, the magnitude of the relationship fell within the moderate range, and the kappa statistic was indicative of fair agreement. While parent–teacher agreement was slightly higher than some previous research (Malhi et al., 2008; Mitsis et al., 2000), it was consistent with the more recent work of Sollie et al. (2012). In contrast, agreement for inattentive symptoms was poorer in this study than has been documented by previous research (Efstratopoulou, Simons, & Janssen, 2012; Gomez, 2007; Malhi et al., 2008; Mitsis et al., 2000; Sayal & Goodman, 2009; Van der Oord, Prins, Oosterlaan, & Emmelkamp, 2006) and fell within slight to less than chance agreement. Multiple informants appear to show higher agreement when rating hyperactive/impulsive rather than inattentive symptoms. This could be explained by the nature of the symptoms being reported. Hyperactive/impulsive symptoms are by nature overt and disruptive, and are consequently easily observable (McConaughy et al., 2010). In contrast, inattentive symptoms may be less easily observed (McConaughy et al., 2010).

Parent–teacher agreement was also assessed for specific symptoms. The results suggested that the magnitude of agreement varied across the specific symptoms; however, there was an overall trend toward increased concordance for hyperactive/impulsive symptoms, relative to inattentive symptoms. Across all symptoms, the level of agreement ranged from non-significant to moderate, with moderate agreement found for only several of the hyperactive/impulsive symptoms. Again, these findings are consistent with previous research, both for the broader category of child externalizing behaviors (Youngstrom et al., 2000) and for ADHD specifically (Antrop, Roeyers, Van Oost, & Buysse, 2000; Gomez, 2007; Nijis et al., 2004).

Pattern of Parent-Rated and Teacher-Rated Symptoms of ADHD

The second aim was to describe reporting of ADHD symptoms by parents and teachers, by comparing the frequency of symptoms reported and the distribution of diagnostic classifications across informants. Significant differences were found for total symptoms of ADHD, hyperactive/impulsive symptoms, and for 10 of the 18 individual symptoms. Across all observed differences, parents rated children as more symptomatic than teachers. This finding is consistent with recent research (Gomez, 2007; Malhi et al., 2008, Murray et al., 2007; Rettew et al., 2011; Sollie et al., 2012). Regarding the distribution of diagnostic classifications, the frequencies of children meeting criteria varied markedly across the different informants and combinational rules. Overall, the proportion of children meeting symptom criteria was highest for the combined subtype, followed by inattentive subtype, and finally for hyperactive/impulsive subtype. This pattern is evidenced in previous research (Biederman et al., 1997; Lahey, Pelham, Loney, Lee, & Willcutt, 2005). In addition, more children met the symptom threshold for ADHD based on parent report. At a subtype level, parents rated significantly more children as combined subtype than teachers. In contrast, more children met criteria for the inattentive or hyperactive/impulsive subtypes based on teacher report. These findings are similar to those found by Sollie et al. (2012) and Youngstrom et al. (2000) where, in clinically referred samples, children were more likely rated as combined subtype by parents and inattentive subtype by teachers.

A substantial proportion of children were rated as symptomatic by one informant but not the other. This is also consistent with previous research (Rettew et al., 2011; Sayal & Goodman, 2009; Valo & Tannock, 2010). One reason for this may be that some children display home-specific or school-specific behavioral problems (Rettew et al., 2011). Alternatively, this may be indicative of lack of agreement between raters. Even within the home setting, Langberg et al. (2010) found only moderate concordance between mothers’ and fathers’ ratings of their children.

In investigating methods of establishing cross-situationality, three combinational rules (OR rule, AND rule, AND duplicated rule) were used to collate parent and teacher ratings, with considerable effects on the distribution of diagnostic classifications. In line with the work of Valo and Tannock (2010), fewer children fell into each diagnostic category as the stringency of the combinational rule increased.

Parent, Teacher, and Clinician Agreement Using Observations of Children’s Behavior in the Classroom

The final aim was to examine whether informant or contextual influences could account for the observed discrepancies. Moderate agreement was found between parent and teacher ratings for hyperactivity/impulsivity. Teacher ratings correlated positively with total off-task behavior and off-task verbal and motor (i.e., hyperactive and impulsive symptoms), but not off-task passive (i.e., inattention) behavior rated during the school observation. Parent ratings were only significantly correlated with off-task verbal and motor behavior (i.e., hyperactivity/impulsivity) on the BOSS. Given that hyperactivity symptom ratings among parent, teacher, and clinician were all similarly and moderately correlated, and inattentive symptoms were not correlated at all; these results are suggestive of personal influences in ratings, where each person brings his or her own views and biases into the ratings of the child’s behavior. One issue that may drive this is the lack of clarity over what each anchor point means on rating scales (e.g., one person may interpret “often” meaning daily whereas another may consider “often” to be twice a week). Future studies could examine whether assigning frequencies to each anchor point assists in the concordance between raters using the same scale (e.g., rarely is once a month, sometimes is once a week, often is 3 times a week, and very often is daily).

Study Limitations

A number of limitations need to be considered when interpreting the results of the present study. First, despite attempts to minimize method variance (e.g., use of validated measures, consistency of instruments used for parent and teacher ratings to achieve similar item coverage, detailed instructions for rating scale completion, and standardized administration of the BOSS), this cannot be ruled out as a possible explanation for the findings. In particular, some issues that remain include that children were observed across different academic subjects (e.g., maths, writing, reading) in different composition classrooms, and may have differed in their level of interest in the task. The conclusions that can be drawn from the present study are also limited by clinician-rated observations only being completed in the classroom setting. Future research may observe the child in both the home and school settings, possibly in the playground at school. Third, the generalizability of the results may also be limited by the nature of the sample which included children from a broad age range (6-12 years) and the majority were New Zealand European families, and thus the sample lacked cultural diversity.

Conclusion

The present study sought to investigate parent–teacher agreement and the mechanisms engendering discrepancies between multiple informants. The results indicated, first, that parent–teacher agreement was slightly higher than that of previous studies; however, it remained within the moderate range. Second, the level of agreement differs across the domains of symptoms being rated. Hyperactive/impulsive symptoms were rated more concordantly by multiple informants than inattentive symptoms. This trend was observed across the core ADHD domains and individual symptoms. Finally, investigation of potential mechanisms for the discrepancies between parents and teachers in rating ADHD symptoms was unclear but leaning toward rater perceptions being related to the observed discrepancies across parent and teacher ratings. This study highlights the ongoing need for researchers to address this issue and for clinicians to think carefully about how informant discrepancies can affect their diagnostic decisions, and the impact that this can have on a child’s holistic development over time.

Footnotes

Authors’ Note

The present study was reviewed and accepted by the Lower South Regional Ethics Committee and the University of Otago Human Ethics Board.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The study was funded by the Deaprtment of Psychology, Univeristy if Otago as part of a larger PhD and masters thesis programme for psotgradute students.

References

Achenbach

T. M.

McConaughy

S. H.

Howell

C. T.

(1987). Child/adolescent behavioural and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213-232.

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text rev.). Washington, DC: Author.

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.

Antrop

Roeyers

Van Oost

Buysse

(2000). Stimulation seeking and hyperactivity in children with ADHD. Journal of Child Psychology and Psychiatry, 41, 225-231.

Barkley

R. A.

(1998). Attention-deficit hyperactivity disorder: A handbook for diagnosis and treatment (2nd ed.). New York, NY: Guilford Press.

Barkley

R. A.

(2003). Issues in the diagnosis of attention-deficit/hyperactivity disorder in children. Brain & Development, 25, 77-83.

Biederman

Faraone

S. V.

Weber

Russel

R. L.

Rater

Park

K. S.

(1997). Correspondence between DSM-III-R and DSM-IV attention-deficit/hyperactivity disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 36, 1682-1687.

Cohen

(1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.

Collett

B. R.

Ohan

J. L.

Meyers

K. M.

(2003). Ten-year review of rating scales: Scales assessing attention-deficit/hyperactivity disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 42, 1015-1037.

10.

De Los Reyes

Kazdin

A. E.

(2005). Informant discrepancies in the assessment of child psychopathology: A critical review, theoretical framework, and recommendations for further study. Psychological Bulletin, 131, 483-509.

11.

Dumenci

Achenbach

T. M.

Windle

(2011). Measuring context-specific and cross-contextual components of hierarchical constructs. Journal of Psychopathology and Behavioural Assessment, 33, 3-10.

12.

DuPaul

G. J.

Power

T. J.

Anastopoulos

A. D.

Reid

(1998). ADHD Rating Scale-IV: Checklist, norms, and clinical interpretation. New York, NY: Guilford Press.

13.

DuPaul

G. J.

Volpe

R. J.

Jitendra

A. K.

Lutz

J. G.

Lorah

K. S.

Grubner

(2004). Elementary school students with Attention-Deficit/Hyperactivity Disorder: Predictors of academic achievement. Journal of School Psychology, 42, 285-301.

14.

Efstratopoulou

Simons

Janssen

(2012). Concordance among physical educators’, teachers’, and parents’ perceptions of attention problems in children. Journal of Attention Disorders, 17, 437-443.

15.

Frazier

T. W.

Youngstrom

E. A.

(2006). Evidence based assessment of attention-deficit/hyperactivity disorder: Using multiple sources of information. Journal of the American Academy of Child & Adolescent Psychiatry, 45, 614-620.

16.

Gomez

(2007). Australian parent and teacher ratings of the DSM-IV ADHD symptoms: Differential symptom functioning and parent-teacher agreement and differences. Journal of Attention Disorders, 11, 17-27.

17.

Healey

D. M.

Miller

C. J.

Castelli

K. L.

Marks

D. J.

Halperin

J. M.

(2008). The impact of impairment criteria on the rates of ADHD diagnoses in preschoolers. Journal of Abnormal Child Psychology, 36, 771-778.

18.

Kaufman

Birmaher

Brent

Rao

Flynn

Moreci

. . . Ryan

(1997). Schedule for affective disorders and schizophrenia for school-age children–Present and lifetime version (K-SADS-PL): Initial reliability and validity data. Journal of the American Academy of Child & Adolescent Psychiatry, 37, 980-988.

19.

Kofler

M. J.

Rapport

M. D.

Alderson

M. R.

(2008). Quantifying ADHD classroom inattentiveness, its moderators, and variability: A meta-analytic review. Journal of Child Psychology and Psychiatry, 49, 59-69.

20.

Lahey

B. B.

Pelham

W. E.

Loney

Lee

S. S.

Willcutt

(2005). Instability of the DSM-IV subtypes of ADHD from preschool through elementary school. Archives of General Psychiatry, 62, 896-902.

21.

Langberg

J. M.

Epstein

J. N.

Simon

J. O.

Loren

R. E. A.

Arnold

E. L.

Hechtman

. . . Wigal

(2010). Parent agreement on ratings of children’s attention deficit/hyperactivity disorder and broadband externalising behaviours. Journal of Emotional and Behavioural Disorders, 18, 41-50.

22.

Malhi

Singhi

Sidhu

(2008). Impact of parent and teacher concordance on diagnosing attention deficit hyperactivity disorder and its sub-types. Indian Journal of Pediatrics, 75, 223-228.

23.

McConaughy

S. H.

Harder

V. S.

Antshel

K. M.

Gordon

Eiraldi

Dumenci

(2010). Incremental validity of test session and classroom observations in a multimethod assessment of attention deficit/hyperactivity disorder. Journal of Clinical Child & Adolescent Psychology, 39, 650-666.

24.

Merrell

K. W.

(2000). Informant reports: Theory and research in using child behaviour rating scales in school settings. In Shapiro

E. S.

Kratochwill

T. R.

(Eds.), Behavioural assessment in schools: Theory, research and clinical foundations (2nd ed., pp. 233-256). New York, NY: Guilford Press.

25.

Mitsis

E. M.

McKay

K. E.

Schulz

K. P.

Newcorn

J. H.

Halperin

J. M.

(2000). Parent–teacher concordance for DSM-IV attention-deficit/hyperactivity disorder in a clinic-referred sample. Journal of the American Academy of Child & Adolescent Psychiatry, 39, 308-313.

26.

Murray

D. W.

Kollins

S. H.

Hardy

K. K.

Abikoff

H. B.

Swanson

J. M.

Cunningham

. . . Chuang

S. Z.

(2007). Parent versus teacher ratings of attention-deficit/hyperactivity disorder symptoms in the preschoolers with attention-deficit/hyperactivity disorder treatment study (PATS). Journal of Child and Adolescent Psychopharmacology, 17, 605-619.

27.

Narad

M. E.

Garner

A. A.

Peugh

J. L.

Tamm

Antonini

T. N.

Kingery

K. M.

. . . Epstein

(2015). Parent–teacher agreement on ADHD symptoms across development. Psychological Assessment, 27, 239-248.

28.

Nijis

P. F. A.

Ferdinand

R. F.

De Bruin

E. I.

Dekker

M. C. J.

Van Duijn

C. M.

Verhulst

F. C.

(2004). Attention-deficit/hyperactivity disorder (ADHD): Parents’ judgment about school, teachers’ judgment about home. European Child & Adolescent Psychiatry, 13, 315-320.

29.

Pelham

W. E.

Fabiano

G. A.

Massetti

G. M.

(2005). Evidence-based assessment of attention deficit hyperactivity disorder in children and adolescents. Journal of Clinical Child & Adolescent Psychology, 34, 449-476.

30.

Rettew

D. C.

van Oort

F. V. A.

Verhulst

F. C.

Buitelaar

J. K.

Ormell

Hartman

C. A.

Hudziak

J. J.

(2011). When parent and teacher ratings don’t agree: The tracking adolescents’ individual lives survey (TRAILS). Journal of Child and Adolescent Psychopharmacology, 21, 389-397.

31.

Sayal

Goodman

(2009). Do parental reports of child hyperkinetic disorder symptoms at school predict teacher ratings? European Child & Adolescent Psychiatry, 18, 336-344.

32.

Shapiro

E. S.

(2004). Academic skills problems workbook (rev.). New York, NY: Guilford Press.

33.

Sollie

Larsson

Mørch

W. T.

(2012). Comparison of mother, father, and teacher reports of ADHD core symptoms in a sample of child psychiatric outpatients. Journal of Attention Disorders, 17, 699-710.

34.

Valo

Tannock

(2010). Diagnostic instability of DSM-IV ADHD subtypes: Effects of informant source, instrumentation, and methods for combining symptom reports. Journal of Clinical Child & Adolescent Psychology, 39, 749-760.

35.

Van der Oord

Prins

P. J. M.

Oosterlaan

Emmelkamp

P. M. G.

(2006). The association between parenting stress, depressed mood and informant agreement in ADHD and ODD. Behaviour Research and Therapy, 44, 1585-1595.

36.

Viera

A. J.

Garrett

J. M.

(2005). Understanding interobserver agreement: The kappa statistic. Family Medicine, 37, 360-363.

37.

Volpe

R. J.

DiPerna

J. C.

Hintze

J. M.

Shapiro

E. S.

(2005). Observing students in classroom settings: A review of seven coding schemes. School Psychology Review, 34, 454-474.

38.

Wolraich

M. L.

Lambert

W. E.

Bickman

Simmons

Doffing

M. A.

Worley

K. A.

(2004). Assessing the impact of parent and teacher agreement on diagnosing attention-deficit hyperactivity disorder. Journal of Developmental & Behavioural Pediatrics, 25, 41-47.

39.

Youngstrom

Loeber

Stouthamer-Loeber

(2000). Patterns and correlates of agreement between parent, teacher, and male adolescent ratings of externalising and internalising problems. Journal of Consulting and Clinical Psychology, 68, 1038-1050.