Profiles of Problem Behavior in Children With Varying Language Ability

Behavior Measurement

To date, the literature that examines the overlapping symptomology of language and behavior problems has used rating scales or special education status to index problem behavior. Although these methods are established and widely used, they may not be conducive to capturing specific types of behaviors that teachers may consider problematic in classrooms. That is, although rating scales allow for a more general assessment of behaviors across contexts, they may be less sensitive to behaviors where precision is important (e.g., frequency counts, intervals) such as frequency of aggressive behavior and duration engaged in academic instruction. These behaviors require more intensive, observational methods for measurement. Comparing the relation between these two types of frequently used behavior measures and language may provide insight into (a) what types of behaviors is language most related to and (b) the direction for future research and subsequent intervention.

Current Study

In the current study, we aim to identify and evaluate patterns of behavior in children (N = 300) who exhibit problem behavior in classrooms based on their language ability. Identifying differences in classroom behavior based on language ability is an important step to identifying targets for intervention or remediation that will best support the social and academic adjustment of this population of students. To do this, we identify subgroups of children based on cutoff scores of language abilities and examine patterns in teacher-rated problem behavior and directly observed classroom behavior. Through profile analysis, we can (a) elucidate patterns of both rated and observed behavior and (b) compare relative patterns between these language skill–based subgroups of children. Most studies have only reported rating scales of problem behavior, and are typically small-N samples. We aim to extend the current literature by providing an analysis of a large, multisite sample of students with or at risk of EBD, and by exploring and comparing how different levels of language relate to teacher-rated and observed behavior in the same sample of students.

Although this study is exploratory in nature, we do expect to see differential patterns in behavior based on language skill–based subgroups of children. Because low language skills are highly prevalent in children with EBD (Hollo et al., 2014), and there is a significant association between low language and problem behavior in the general school-age population (Chow & Wehby, 2016), we expect the behavioral profiles of children with or at risk of EBD to differ based on language ability.

Method

Participants

The data analyzed in the current study were collected as a part of a multiyear efficacy trial assessing the effects of a comprehensive teacher-level behavioral management intervention with an academic tutoring component. Data in this article were collected across two cohorts (N = 300) during their preliminary assessment wave. Students were recruited from both general and special education classrooms (K-4) from participating schools across three school districts. The current sample is a subset of a larger intervention study that screened in 422 original participants. Here, we utilize cases who had complete pretest data on all relevant variables. See Table 1 for full participant demographics.

OPEN IN VIEWER

Table 1.

Demographic Characteristics of Included Sample (N = 300).

Sample	Grade	Sex	Ethnicity	Label	Lunch	ELL
SPED (63%)	9% K 11% first 14% second 19% third 10% fourth	55% M 14% F	53% Black 10% White 4% Hispanic 1% Other	32% emotional disturbance 20% developmental delay 12% learning disability 11% speech/language impairment 11% ADD/ADHD 5% intellectual disability 4% other health impairment 5% other	54% free 6% reduced 9% N/A	<1%
At risk (37%)	8% K 10% first 10% second 4% third 5% fourth	21% M 10% F	23% Black 4% White 2% Hispanic 3% Other		26% free2% reduced3% N/A	<1%
Total (100%)	17% K21% first24% second23% third15% fourth	76% M24% F	76% Black14% White6% Hispanic4% Other		80% free8% reduced12% N/A	<2%

Note. ELL = students receiving English language learning services; SPED = students receiving special education services; ADD = attention deficit disorder; ADHD = attention deficit hyperactivity disorder.

Students were selected for inclusion into the sample via two methods. First, to identify at-risk students, the first two stages of the Systematic Screening for Behavior Disorders (SSBD; Walker & Severson, 1992) were used. Following consent procedures, general education teachers rank ordered consented students in their respective classrooms based on SSBD-defined characteristics for externalizing disorders. The top three ranked students in each classroom moved to Stage 2 of the SSBD. At this stage, teachers completed three brief rating scales—the Adaptive Behavior (AB) scale, the Maladaptive Behavior (MB) scale, and the critical events index (CEI). The AB scale is a 12-item scale that describes a student’s current functioning with items such as “Follows classroom rules” and “Exhibits positive social peer interactions.” The MB scale is an 11-item scale that describes a student’s maladaptive behavior with items such as “Creates a disturbance during classroom activities” and “Refuses to participate with other children at recess.” The CEI rates low-frequency, high-intensity events such as theft, setting fires, and property damage. Student who exceeded SSBD norms on the AB scale and MB scale or exceeded more than five critical events on the CEI were included in the study. Of the 300 participants in this study, 112 students in general education classrooms were identified as at risk based on the SSBD.

For the second approach, students receiving special education services in self-contained special education classrooms for students with behavioral difficulties were selected for participation, and, thus, 188 were included in the study because of their status as special education students. Given there is some evidence of differential eligibility patterns across schools and districts (e.g., Siperstein, Wiley, & Forness, 2011; Wiley, Siperstein, Bountress, Forness, & Brigham, 2008), we analyzed this as a single aggregated sample of at-risk and identified students that allowed us to leverage the variability in language and behavior scores and measures that were important characteristics for the current study’s purpose.

Data Analysis

Preliminary analyses and variable transformations

Table 2 includes the means, standard deviations, and correlations for the sample of 300 students of language and behavior measures; all analyses in the present study were conducted in Stata 12 (StataCorp, 2011). Direct observation–dependent variables were transformed into within-sample z scores (M = 0.00, SD = 1.00). We plotted the bivariate relations between language and each behavior construct–dependent measure. No outliers were identified; thus, all observations were included in further analyses. Then, we examined the distributions of each dependent variable to determine whether transformations were necessary. For frequency of aggression and negative talk, we used log-based transformations to help account for the expected moderate skew.

OPEN IN VIEWER

Table 2.

Sample Means, Standard Deviations, and Correlations (N = 300).

Variable	M	SD	1	2	3	4	5	6	7	8	9	10	11	12
1. EXTtotal	68.65	8.57	—
2. INTtotal	61.79	10.28	.32	—
3. Language cluster a	88.73	13.15	−.07	.06	—
4. Affect b	62.64	8.34	.42	.83	.04	—
5. Anxiety b	60.39	7.56	.26	.76	−.01	.56	—
6. Attention b	65.88	9.76	.67	.13	−.16	.28	.23	—
7. Oppositional b	65.78	7.60	.89	.26	−.01	.38	.23	.58	—
8. Conduct b	67.99	11.16	.89	.20	−.08	.30	.12	.62	.71	—
9. Engagement c	0.00	1.00	−.29	−.02	.23	−.10	−.13	−.29	−.28	−.26	—
10. Active responding c	0.00	1.00	−.06	−.005	−.07	−.08	−.02	−.07	−.10	−.07	.23	—
11. Negative talk d	0.00	1.00	.25	.06	−.10	.07	.11	.16	.29	.15	−.49	.001	—
12. Aggression d	0.00	1.00	.11	.06	−.16	.03	.09	.02	.08	.08	−.49	−.02	.67	—

Note. EXT = total externalizing t score; INT = total internalizing t score (Teacher Report Form; Achenbach & Rescorla, 2001).

a

Group formation. ^bSubscales of teacher ratings of behavior. ^cDirect observation proportion. ^dDirect observation frequency.

Language group formation

We used the Oral Language Cluster from the WJ-III to group cases into three groups. The low language group is defined as having a language cluster composite score greater than 1 SD below the mean (<85). Moderate language is defined as a composite score between 1 SD below the mean and the mean (85–100). High language, in the context of this sample, is defined as a composite score above the mean (>100). We selected these cutoffs because, due to the nature of the current sample and past literature, students with or at risk of problem behavior likely have lower average language scores than a normative sample. For example, in a comprehensive meta-analysis, the mean comprehensive language score for students with EBD across 22 studies was 76.33, which was significantly below average (Hollo et al., 2014).

Sociodemographic comparisons

To determine whether groups based on language ability were different, we used chi-square difference tests to examine whether groups were different by gender, special education status, free or reduced lunch, grade, and ethnicity. Ethnicity was the only sociodemographic variable that was significantly different between language groups (p = .04). Due to this significant difference, ethnicity was included as a covariate in all analyses.

Site- and school-level comparisons

First, we examined intraclass correlations (ICC) to determine whether multilevel modeling was necessary to account for clustering effects. With clustering effects, we would violate the independence assumption of ANOVA, potentially resulting in spurious significance (Raudenbush, Bryk, & Congdon, 2002). Site (i.e., district) accounted for less than 1%, and school contributed less than 5% unique variance across dependent measures. Using Raudenbush and Liu’s (2000) standards that identify small, medium, and large as 5%, 10%, and 15%, we did not account for clustering and proceeded to the overall analysis.

Overall analyses

Because the effect of site and school contributed little unique variance in the sample, we conducted profile analyses using repeated measures ANOVA. Language was the between-subjects factor, and teacher ratings of behavior and directly observed behavior were the within-subject factors. After examining level and shape effects, we conducted univariate follow-up contrasts to explore how profiles differed by group.

Results

We aimed to examine profiles of teacher-rated and observed classroom behavior based on student language ability. With teacher-rated behavior subscales of problem behavior as the dependent variables, we found no significant differences between language groups, F(2, 327) = 2.05, p = .130; however, when averaged across groups, mean scores of teacher-rated behavior was significantly different, F(4, 324) = 28.44, p < .001. This difference suggests that language does not differentiate between ratings of separate subscales, but that teachers rate students’ behavior differently on average based on language group (see Figure 1).

OPEN IN VIEWER

Figure 1.

Subscale t scores of the TRF based on language scores.

Next, we examined profiles of students based on language on direct observation outcomes. Here, we aimed to determine whether patterns of direct observation data differed from patterns of teacher-rated behavior (first analysis; see Figure 1) in groups formed based on language ability. The first dependent variables we examined were positive classroom behaviors (engagement and active responding). When aggregated across language groups, there was no significant main effect, F(1, 327) = 0.036, p = .849. However, this analysis yielded a significant shape effect, F(2, 327) = 5.19, p = .006, suggesting a Language × Positive Behavior interaction (see Figure 2). That is, group mean scores depend on language, and there is a significant difference between groups on positive classroom behavior. Students in the high language group are associated with the highest mean within-sample z score, whereas the low language group is associated with the lowest z score. Students with the lowest language were engaged in class for the least amount of time during observation sessions. Alpha-adjusted contrasts on individual-dependent variables reveal that the low language group is significantly different from both medium (p = .04) and high groups (p = .009) on engagement.

OPEN IN VIEWER

Figure 2.

Profiles of positive and negative classroom behaviors; within-sample z scores are on the y axis.

For negative classroom behavior, groups based on language ability differ significantly when negative behaviors are averaged, F(2, 327) = 3.79, p = .024. The effect for shape was not statistically significant, F(4, 654) = 2.087, p = .081; however, visual interpretation of Figure 2 suggests potential practical differences. Students with low language ability are associated with more negative behavior; specific to the high language group, their mean level of negative talk is highest across groups.

Discussion

The aim of this article was to examine the behavioral profiles of students with or at risk of EBD with varying language ability and to explore differences between profiles of teacher-rated and directly observed behavior. Because students with behavior problems have been associated with significant linguistic deficits, it is important to understand how language is associated with different types and measures of classroom behavior. The current findings suggest that language ability may underpin behavioral phenomena that students exhibit in classrooms, as well as how students experience their classroom environments. Noteworthy differences between teacher ratings and observed behavior imply a mismatch between how behaviors and language impact teacher perceptions and how behaviors manifest themselves in relation to researcher observations.

Profiles of observed classroom behaviors based on language groups provide some interesting points for discussion. First, with positive behavior–dependent variables, students with high oral language were engaged for the highest proportion of pooled observations, and students with low language were engaged for the least amount of time. In this study, language group membership predicted rates of general engagement during academic instruction, but did not predict response to opportunities to respond. It is likely that language skills are needed to self-regulate and actively participate in the classroom, which manifests itself in higher rates of general engagement. Conversely, because opportunities to respond are isolated events that are teacher initiated, it may be less likely that children need higher language skills to attend to a teacher-delivered prompt.

Relative to negative classroom behaviors, the low language group is associated with higher frequency of aggressive behavior. Both moderate and high language groups are below the mean of the full sample. However, for frequency of negative talk, high language group membership is associated with the highest level of this behavior. Although minimal language ability is needed to engage in negative talk, these data suggest the possibility that the topography of problem behavior for students with higher language may more frequently be verbal than for students with lower language skills. Students who have more developed language skills may resort to verbal aggression when engaging in problem behaviors to access attention or escape from academic tasks. In addition, students with higher language skills may appear to be less problematic to teachers as they appear more engaged and be viewed as less aggressive.

In addition, comparing the relative association between oral language and measures of teacher-rated behavior and directly observed behavior is an important step in moving forward with decisions and subsequent interpretation of the measurement of behavior. Although behavior observation has been long considered a superior method for measuring behavior (Johnston & Pennypacker, 2010; Walker, 2015), observational coding systems are subject to methodological and measurement problems (Yoder & Symons, 2010). Relative to teacher ratings, there are inconsistencies in the literature. In a longitudinal analysis of six large-scale data sets, adult-rated problem behavior did not emerge a significant predictor of later achievement (Duncan et al., 2007). Conversely, some studies have reported that teacher ratings predict achievement outcomes relatively well. This may be, in part, due to the definitions of “outcomes.” For example, Lane, Menzies, Kalberg, and Oakes (2012) used grade point average and course failure as their achievement outcome. The current study uses subtests from the WJ-III. Teacher ratings may be a better predictor of outcomes that are tied more closely to teachers (e.g., grades) than standardized assessments. In a study comparing language assessments of children with ED, Chow and Hollo (2017) reported meaningful discrepancies between teacher-rated language ability and standardized assessments of language. Teachers were much more likely to underestimate than overestimate the presence of language risk in students with ED. Along these same lines and when thinking about measuring behavior, teacher ratings may not be tapping the same underlying construct as direct observation of behavior.

The current study provides descriptive evidence that oral language may be more strongly associated with observed classroom behavior than teacher ratings. It is possible that teacher ratings of behavior and direct observation may not measure the same construct, because the analysis with teacher-rated behavior as the within-subjects factor did not produce significant level or shape effects, whereas the direct observation within-subjects factor analyses did. Also, these findings provide additional evidence that teachers may be unaware or untrained to notice the language deficits that are associated with many children who have identified behavioral difficulties and do not account for these differences in their instruction. Although there appeared to be no significant difference in the percentages of active engagement by students (i.e., opportunities to respond responded to by students), it was not coded whether these instructional prompts were linguistically different based on student language ability. If not, such a finding would be additional evidence of an instructional mismatch between these groups of students.

When comparing teacher ratings to direct observation of behavior, it is important to consider the substantive difference. Teacher ratings are typically checklists that are completed outside of an active instructional context, whereas direct observations are usually conducted by a trained observer in the classroom, often when the teacher is engaged in instruction. Thus, these measures provide very little basis on which to make qualitative comparisons, and, findings from the present study suggest that teacher ratings and direct observation of behavior may potentially be measuring different constructs of behavior. However, one potential interpretation is that rating scales are more reflective of a teacher’s overall impression of student behavior while observations are capturing levels of behavior at a more molecular level. In addition, it is unclear how intensity of behavior affects both ratings and observations. For example, low-frequency/high-intensity behavior events may skew teacher ratings toward a more negative rating. Future measurement studies should use multiple measures of both observed and teacher-rated student behavior to explore issues relative to agreement and construct validity.

Taken together, however, our data reveal that what teachers perceive as problematic behavior or report on as important negative behaviors may not align with how researchers define behavior problems in classrooms. Observational coding systems are complex, precise, and often are employed for brief, systematic periods of time within the context of a research study or an intervention program. Teacher ratings, however, likely draw from a broader perspective a student’s behavior, and may potentially be biased depending on the behavioral characteristics of peers and their classroom composition. That is, how a teacher rates a student’s behavior may depend on other student’s level of problematic behavior, as well as a teacher’s personality, perspective, and past teaching experiences. Conversely, as designed, direct observation of behavior provides a discrete characterization of the behaviors present during observation.