Abstract
This study examined the extent to which clinical and demographic characteristics predicted outcome for children with autism spectrum disorder. Participants included 152 students with autism spectrum disorder in 53 kindergarten-through-second-grade autism support classrooms in a large urban public school district. Associations between child characteristics (including age, language ability, autism severity, social skills, adaptive behavior, co-occurring psychological symptoms, and restrictive and repetitive behavior) and outcome, as measured by changes in cognitive ability following one academic year of an intervention standardized across the sample were evaluated using linear regression with random effects for classroom. While several scales and subscales had statistically significant bivariate associations with outcome, in adjusted analysis, only age and the presence of symptoms associated with social anxiety, such as social avoidance and social fearfulness, as measured through the Child Symptom Inventory-4, were associated with differences in outcome. The findings regarding the role of social anxiety are new and have important implications for treatment. Disentangling the construct of social anxiety to differentiate between social fearfulness and social motivation has important implications for shifting the focus of early treatment for children with autism spectrum disorder.
Autism spectrum disorder (ASD) is a complex neurobiological disorder that manifests in early childhood and usually lasts throughout the lifespan. The Centers for Disease Control (Center for Disease Control and Prevention, 2014) estimates that 1 in 68 children in the United States meets criteria for ASD. ASD is characterized by impairments in social reciprocity and communication, and restrictive and repetitive behavior (APA, 2013); however, significant heterogeneity in clinical presentation has been observed. That is, the autism spectrum is associated with many different levels of language, social and cognitive impairment, developmental ages, aberrant behavior, and restrictive and repetitive behaviors (Waterhouse et al., 1996).
The heterogeneity found among individuals with ASD is matched by their heterogeneity in response to intervention. While interventions based on the principles and techniques of applied behavior analysis have been repeatedly cited as having strong evidence for effectiveness in treating individuals with ASD (National Autism Center, 2009; National Research Council, 2001; Weitlauf et al., 2014), significant differences in outcomes for different children have been noted (Schreibman and Anderson, 2011). In most studies of behavioral treatments for children with ASD, some children make significant progress, while others make minimal to no progress (Eikeseth et al., 2002; Lovaas, 1987; Smith et al., 2000). There are few data on what child characteristics are associated with treatment response (Eapen et al., 2013; Schreibman, 2000), with research yielding mixed findings. Data regarding the individual profiles of children who may benefit from particular treatment approaches would potentially allow clinicians to more effectively match children to particular treatment approaches based on individual profiles, enabling clinicians to deliver the most effective interventions for a particular child earlier (Vivanti et al., 2014).
The existing research regarding individual child characteristics associated with differences in outcome has identified mixed and sometimes conflicting results. Higher baseline IQ scores (Ben-Itzchak et al., 2014; Ben-Itzchak and Zachor, 2007; Eldevik et al., 2012; Goldstein, 2002; Magiati et al., 2011) and language ability have been repeatedly associated with better treatment outcomes (Magiati et al., 2011; Remington et al., 2007). Yet, other studies have not found this association (Vivanti et al., 2013; Zachor et al., 2007). Several studies also have identified higher levels of adaptive skills as predicting improved outcomes (Goldstein, 2002; Sallows and Graupner, 2005). However, the use of measures of broad child characteristics such as IQ, language, and adaptive skills may not provide adequate information about the specific underlying child characteristics that relate to differences in response to treatment (Vivanti et al., 2014). Studies that examined more specific child variables have found that greater joint attention, imitation, play, social approach behaviors, and non-verbal communication skills in early childhood are associated with better outcomes following intervention (Charman et al., 2003; Ingersoll et al., 2001; Sigman and Ruskin, 1999; Toth et al., 2006). The few studies that include demographic variables have found no significant differences in outcome associated with sex or ethnicity. Younger children, however, show greater improvement than older children in response to intervention (Baker-Ericzen et al., 2007; Mandell et al., 2013).
Previous research that evaluated child characteristics associated with improved outcomes provided important preliminary findings but was constrained by several important limitations. The majority of these studies involved children referred to tertiary care settings, such as outpatient or university-based clinics with treatment provided by trained specialists, rather than children selected from community-based settings; thus, these findings may not readily generalize to the larger population of children with ASD. Perhaps a more important limitation is that in most cases, intervention was not standardized in the sample. Intervention type, intensity, and quality may have varied among children, potentially confounding the observed findings. Furthermore, many of the studies had small sample sizes, so they were underpowered to study predictors of outcome. Lastly, previous studies did not evaluate the presence of co-occurring psychological symptoms, and their potential effects on outcomes for children with autism, despite the prevalence of such symptoms in children with autism spectrum disorders (Simonoff et al., 2008; Van Steensel et al., 2001).
The present study extends this body of research by examining the extent to which clinical and demographic characteristics predicted outcome for children with ASD. The data collected for this evaluation were a subset of a larger comparative effectiveness trial (Year 1) and subsequent implementation study (Years 2 and 3) conducted in partnership with the School District of Philadelphia. The parent study provided us with a large community sample, a consistent intervention across the sample, and longitudinal data over the course of an academic year for each child. The current study examined differences in outcome following 1 year of a behaviorally based intervention as a function of children’s cognition, challenging behavior, language ability, autism severity, social skills, adaptive behavior, co-occurring presence of psychological symptoms, and restrictive and repetitive behavior.
Method
Participants and setting
Participants came from a randomized field trial (Mandell et al., 2013) of the Strategies for Teaching based on Autism Research (STAR) program (Arick et al., 2004). STAR is a manual-based comprehensive treatment program for children with autism. It consists of three behaviorally based methods of instruction: discrete trial training, pivotal response training, and teaching within functional routines, and includes content areas targeting language, academic, and social skills, as well as activities necessary to complete daily school routines. Regular staff training, consultation, and fidelity monitoring were provided as part of the parent study (Mandell et al., 2013). In the first year of the parent study, half of the participating classrooms were randomized to STAR. In the second and third years of the parent study, all classrooms received training and consultation in STAR. Participants in the present study all were students in the intervention group of the parent study who were enrolled in STAR classrooms in one of those 3 years. As part of the parent study, teachers were provided with intensive training and ongoing consultation throughout the study on how to implement STAR; fidelity was measured monthly and was satisfactory (see Mandell et al., 2013).
Participants were enrolled in 1 of 53 kindergarten-through-second grade autism support classrooms. The sample included 152 students. Mean student age was 6.0 years (SD = 0.9, range = 5–8 years). The participants were ethnically diverse; 53% were African American, 30% Caucasian, 10% Hispanic, 5% Asian, and 2% were of other ethnicities. Consistent with the epidemiology of autism, 87% were male. Three quarters were eligible for free or reduced-price lunches. All participants had an educational classification of autism provided by a licensed practitioner; diagnosis was confirmed using the Autism Diagnostic Observation Schedule (ADOS: Lord et al., 2012) by a research-reliable clinician.
Measures
Baseline measures
Adaptive Behavior Assessment System
Adaptive behavior was measured using the parent version of the Adaptive Behavior Assessment System (ABAS-II) at the start of the academic year (Harrison and Oakland, 2003). The ABAS-II uses a behavior-rating format to assess adaptive behavior and related skills for individuals, birth through 89 years of age. The scores derived from the ABAS-II describe an individual’s general adaptive behavior and functioning in 10 related adaptive skill areas: communication, community use, functional academics, school/home living, health and safety, leisure, self-care, self-direction, social, and work (for older adolescents and adults). Reliability for the ABAS-II is high; estimates of internal consistency and test–retest reliability are above 0.90. Average reliability coefficients for the 10 skill areas across all ages range from 0.85 to 0.97. Inter-rater reliability and cross-form consistency also are high (0.91–0.99; Harrison and Oakland, 2003). Factor analytic, concurrent validity, and clinical studies provide strong support for its validity. The ABAS-II subscale-scaled scores were used to evaluate baseline levels of independence with adaptive skills.
ADOS
Autism severity was measured using the ADOS (Lord et al., 1999). The ADOS is a semi-structured, standardized observational measure of social interaction, communication skills, and play or imaginative use of materials for the assessment of ASD. The ADOS has strong psychometric properties, with high sensitivity and specificity reported for each of its four modules (Lord et al., 1999). To measure severity, ADOS scores were converted into the ADOS severity algorithm, a validated measure that allows for comparison of autism severity across the modules of the ADOS (Gotham et al., 2009). Evaluators trained to research reliability administered the ADOS at the start of the academic year.
Child Symptom Inventory– 4
Co-occurring psychiatric symptoms were measured using the parent report version of the Child Symptom Inventory– 4 (CSI-4) at the start of the academic year (Gadow and Sprafkin, 2002). The CSI-4 is based on diagnostic criteria from the Diagnostic and Statistical Manual of Mental Disorders (DSM), shows high predictive and concurrent validity, and is used across a wide variety of settings. The test–retest reliability ranges from 0.46 to 0.88 and internal consistency for the CSI-4 ranges from 0.74 to 0.94 across categories in previous studies (Sprafkin et al., 2002). The CSI-4 provides information regarding the presence of symptoms associated with specific psychiatric symptoms typically diagnosed in children ages 5 through 12. A symptom severity raw score for each disorder is developed from individual items, which are scored from 0 (never) to 3 (very often). Raw scores were converted into T-scores, as indicated in the manual, and were used to evaluate the presence of symptoms associated with each of these disorders in the participant sample.
Pervasive Developmental Disorders Behavior Inventory
The teacher-report version of the Pervasive Developmental Disorders Behavior Inventory (PDDBI) was used to assess social, language, and communication skills, and restrictive and repetitive behavior associated with autism (Cohen and Sudhalter, 2005). The PDDBI is a 124-item rating scale that assesses problem behaviors, social skills, language skills, and learning or memory skills in children who have been diagnosed with ASD (formerly pervasive developmental disorders). It can be used with children between the ages of 1.6 and 12.5 years. The PDDBI has high reliability and validity, test–retest reliability for the teacher ratings ranged from 0.65 to 0.99 over an average 2-week interval and inter-rater reliability among teachers ranged from 0.85 to 0.92 (Cohen, 2003).
Outcome measure
The Differential Ability Scales, 2nd Edition
Cognitive ability was measured by evaluators, trained to research reliability, using the Differential Ability Scales, 2nd Edition (DAS-II) at baseline and after one academic year of intervention (Elliott, 1990). The DAS-II was designed to assess a wide range of cognitive abilities in children aged 2 years 6 months through 17 years 11 months and has been used with children with autism in many previous studies (Anderson et al., 2007; Thurm et al., 2007). The DAS-II is an appropriate assessment tool for evaluating the cognitive abilities of children with autism because it relies less on expressive language ability than other cognitive assessments and its primary purpose is as a tool for identifying and understanding the strengths and weaknesses in individuals rather than establishing a general IQ score. Similar to traditional tests of IQ, the DAS is normed to have an average of 100 and a standard deviation of 15 in the general population. Consistent with previous intervention studies for children with autism, a measure of overall cognitive ability as indicated by the General Conceptual Ability (GCA) in the DAS was used to assess outcomes in this study. Child outcomes are commonly reported as change in overall cognitive ability or IQ in autism intervention studies, especially outcome studies of behavioral interventions similar to the one used in our study (e.g. Eikeseth et al., 2002; Lovaas, 1987; Smith et al., 2000).
Data analyses
Differences in outcomes were estimated using linear regression with random effects for classroom. Each classroom included one lead teacher who delivered the instruction; therefore, the classroom was the smallest possible unit of clustering in this model. Random effects were used to account for potential clustering of outcomes among students in the same classroom, which could potentially result from unobserved differences in teacher practices. Each subscale for each measure was included as an independent variable in the analysis. Therefore, the independent variables included baseline measures of language and communication (PDDBI and ABAS-II), adaptive skills (ABAS-II), challenging behavior (CSI-4), autism severity (ADOS), social skills (PDDBI), and age, as well as measures of symptoms associated with common co-occurring psychiatric difficulties, (CSI-4) such as attention deficit hyperactivity disorder (ADHD), anxiety, social phobia, depression symptoms, and conduct problems. Collinearity among the independent variables was examined and indicated no correlations that would interfere with interpretation of the regression results, with variance inflation factors less than 4 for all of the factors (Allison, 1999). Standardized scores were used in the analyses to allow for meaningful comparisons across measures consistent with previous outcome studies for children with autism (Sallows and Graupner, 2005; Szatmari et al., 2003). Consistent with established models of regression analyses (Hosmer et al., 2013), variables with a bivariate association with outcome significant at p < 0.2 were included in a multiple regression model.
Results
Descriptive characteristics for each of the child variables measured at baseline are presented in Table 1. The mean change in DAS GCA score was 6.3 points, with a standard deviation of 11.2 and a range of −43 to 43 across the sample. Child characteristics are presented as standardized scores to allow for comparison across measures. Table 2 presents the results of the regression analyses predicting change in overall cognitive ability. In unadjusted analyses, several measures of adaptive behavior from the ABAS-II, including functional academics, health and safety, self-direction, social skills, and the overall adaptive composite predicted changes in DAS scores at p < 0.05. Social anxiety symptoms as measured by the CSI-4 also predicted changes in DAS scores. In the adjusted analysis, in which only variables with a bivariate statistical significance of p < 0.2 were included, social anxiety symptoms and student age significantly predicted student outcome at p < 0.05. Each point increase on the CSI-4 scale for social phobia was associated with a 0.15 point average decrease in DAS change score. Each year increase in age was associated with a 2.35 point average decrease in DAS change score.
Descriptive characteristics of the sample.
DAS: The Differential Ability Scales; GCA: General Conceptual Ability; ODD: oppositional defiant disorder; ABAS: Adaptive Behavior Assessment System; PDDBI: Pervasive Developmental Disorders Behavior Inventory; CSI: Child Symptom Inventory; ADHD: attention deficit hyperactivity disorder; ADOS: Autism Diagnostic Observation Schedule.
Scaled scores.
Standard scores.
T-scores.
Linear regression depicting the association between change in DAS and child characteristics.
DAS: The Differential Ability Scales; GCA: General Conceptual Ability; ABAS: Adaptive Behavior Assessment System; PDDBI: Pervasive Developmental Disorders Behavior Inventory; CSI: Child Symptom Inventory; ADHD: attention deficit hyperactivity disorder; ADOS: Autism Diagnostic Observation Schedule.
Discussion
The results of this study indicate that increased social anxiety symptoms, as measured by the CSI-4, and increased age were associated with poorer outcome following one academic year of intervention. The finding that children with higher levels of social anxiety symptoms had poorer outcomes is novel and warrants careful examination. Social anxiety symptoms were evaluated using a tool designed to identify psychiatric symptoms consistent with the diagnostic criteria for disorders as outlined in the Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV; American Psychiatric Association, 1994). According to these criteria, social phobia is characterized by an intense and persistent fear of social situations with unfamiliar people. To meet criteria, children must demonstrate the capability of age-appropriate social relationships with familiar people, and the anxiety must occur in interactions with peers, not just unfamiliar adults (Gadow and Sprafkin, 2002). Social anxiety is diagnostically distinct from more generalized anxiety, in that the anxiety is specific to social interactions. A growing body of research finds that anxiety commonly co-occurs with ASD (De Bruin et al., 2007; Kerns and Kendall, 2012; Simonoff et al., 2008). A meta-analysis found that almost 40% of youth with ASD met criteria for at least one DSM-IV anxiety disorder, with 17% meeting criteria for social phobia (van Steensel et al., 2011). Other research has also found elevated levels of social anxiety symptoms for children with autism (Gadow et al., 2004, 2005; Settipani et al., 2012).
More recent literature suggests a challenge in identifying social anxiety in youth with ASD due to the inherent social difficulties associated with ASD (Kerns and Kendall, 2012; Settipani et al., 2012). It is difficult to determine whether the presence of symptoms associated with social anxiety in individuals with ASD are manifestations of anxiety related to social situations or a lack of social motivation, in large part because of the absence of assessment tools that adequately differentiate between them in children with ASD. The CSI-4 has been used to measure social anxiety symptoms in youth with ASD in previous studies (Gadow et al., 2004, 2005), and can provide important information regarding the presence of elevated symptoms of social anxiety in youth with ASD, but the social phobia scale of the CSI-4 includes only four questions:
Tries to avoid contact with strangers; abnormally shy;
Is excessively shy with peers;
Is generally warm and outgoing with family members and familiar adults;
When put in an uncomfortable social situation, child cries, freezes, or withdraws from interacting.
Closer consideration of the items included in this scale highlight the challenge of disentangling whether, in children with ASD, the scale captures behaviors consistent with social anxiety or a lack of social motivation. Additional regression analyses using these four items individually indicated that none was significantly associated with outcome, adding further support to the notion that the scale measures a multi-layered construct. If what is measured affects treatment outcomes, then identifying the relevant construct—anxiety or motivation—has direct implications for treatment, with distinct treatment approaches indicated for each.
If the construct is social anxiety, then employing treatments proven to be effective in reducing social anxiety in other populations, such as cognitive behavioral therapy (CBT), should be considered. Preliminary research supports the efficacy of CBT in reducing anxiety symptoms for individuals with autism and at least average levels of cognitive ability (Danial and Wood, 2013; Moree and Davis, 2010), with recent research indicating the efficacy of CBT in improving the symptoms of social anxiety consistent with social phobia in youth with autism (Koning et al., 2013). Additionally, a recent review of the treatment of fears and phobias among children with ASD found that some single-subject studies have successfully used components of CBT such as exposure, modeling, and reinforcement to reduce phobic responses in youth with ASD who have more limited verbal and cognitive skills (Lydon et al., 2014).
If the construct is social motivation, then interventions should increase opportunities for and improve social interaction as a fundamental element of autism intervention, rather than address anxiety per se. Many comprehensive treatment packages for children with ASD address functional academics, behavior, and routines, but fewer meaningfully address social engagement as a primary area of intervention (Arick et al., 2004; McEachin et al., 1993; Mesibov et al., 2005). A previous study that evaluated outcomes for toddlers with autism in an inclusive preschool program found that higher levels of social avoidance were associated with poorer outcomes (Ingersoll et al., 2001); however, it was unclear whether the social avoidance was due to social phobia or low social motivation.
The finding that younger ages were associated with differences in outcome is consistent with previous research (Baker-Ericzen et al., 2007; Granpeesheh et al., 2009; Mandell et al., 2013). It is widely recognized that intervening early leads to improved outcomes for children with autism (National Research Council, 2001). The results of this study support this consensus, with an important caveat. Children in this study were of school age. The developmental window during which interventions are effective may not shut when children reach school, but rather gradually close over a longer period of time, suggesting the potential for increasing cognitive abilities (rather than only adaptive behaviors) of children with ASD during elementary school.
Previous research found an association between challenging behavior, language ability, social skills, and adaptive behavior with differences in outcome (Ben-Itzchak and Zachor, 2007; Luyster et al., 2007; Mawhood et al., 2000; Remington et al., 2007). While some of these associations were apparent in the unadjusted analyses in this study, these associations did not sustain in adjusted analyses. These differences may be due to variations in the length of time pre- and post-intervention between studies. We measured differences in outcome following one academic year of intervention; this is a shorter time frame than most of the studies which found an association between these variables and outcome. Additionally, the current study used one intervention that was consistent across all children, a distinct difference from many of the previous studies which reported aggregated differences across treatments. It is possible that using one intervention model that is heavily focused on language, behavior, and social skills ameliorated any differences in outcome that would have been associated with baseline levels of these skills. Furthermore, the sample in the current study was a community-based sample, rather than a clinic-based sample, and is likely more reflective of the general population of children with autism who are educated within public schools. Finally, other studies may not have adjusted for important, potentially confounding variables that were included in this study.
Consistent with previous research, autism severity or a greater number of autism symptoms was not associated with differences in outcome (Fein et al., 1999; Lord et al., 2006; Sutera et al., 2007; Szatmari et al., 2003). Although this study examined improved outcomes as indicated by changes in cognitive ability, not changes in autism symptoms, the finding that baseline levels of autism severity were not associated with outcome is promising, and indicates that children with more severe impairments are just as likely to benefit from intervention as less severely impaired children.
Several limitations were present in this study. First, parent report on the CSI-4 was the only measure of social anxiety and other co-morbid conditions used in the study. Although the CSI-4 has previously been used as a primary measure in assessing co-occurring psychiatric symptoms in children with autism (Gadow et al., 2004, 2005), adding a measure that more thoroughly evaluates the presence of psychiatric symptoms would strengthen the validity of the findings. A second limitation relates to the association between social anxiety, anxiety, and more general social deficits found among individuals with autism. There is considerable overlap among the symptoms associated with each of these conditions, making it difficult to disentangle the relative association of each with outcomes. However, in our analyses, we included measures of anxiety and social skills individually, neither of which was associated with outcome. This finding adds support to the idea that the social disengagement and social distress as measured by the CSI is a distinct construct and is associated with outcome. A third limitation relates to the substantial variability observed in changes in DAS GCA scores across the sample, with some students demonstrating extremely large changes pre- and post-intervention. It is likely that these extremely large changes in DAS-II scores captured issues related to improvements in behavior or test-taking ability, or increases in challenging or other behavior that interfered with the test, rather than a true change in cognitive ability of 20–40 points over one academic year. To account for the effects of these potential outliers, a sensitivity analysis was conducted in which outliers, as defined by change scores greater than 30 points, were excluded from the analysis. No difference was observed in the association of the independent variables with student outcome.
Despite these limitations, the results of this study have important implications for treating children with ASD. This study is the first to identify an association between increased social anxiety symptoms and poorer outcomes for children with autism. Scale scores for social phobia ranged from 43 to 108, meaning that the predicted difference in change in DAS between those at the low end of the scale and those at the high end was almost 10 points, a clinically significant amount, especially given that the average increase in DAS score for the whole sample was 6.3 points.
Impairment in social interaction is a core deficit and diagnostic indicator of autism. Regardless of whether the measure of social anxiety used in this study truly measures anxiety or motivation in our sample, the results suggest that addressing social impairment may be a necessary precursor or adjunct to improving response to intervention. Including treatment to address either social anxiety or social motivation would constitute a shift in focus for most widely used school-based interventions for children with autism. Yet addressing social impairment early may have profound implications for maximizing outcomes for these children.
Footnotes
Funding
This work was supported by the following grants from the National Institute of Health: 1R01MH083717 and the Institute of Education Sciences: R324A080195 to Mandell.