Examining the Validity of the Columbia Impairment Scale for Assessing Level of Functioning in Youth with Autism Spectrum Disorder

Abstract

Background:

Youth with autism spectrum disorder (ASD) exhibit impairment in numerous areas of functioning, most notably in the areas related to social interactions, communication, and behavior at school and at home. Understanding the severity of the impairment in each of the domains associated with areas of functioning is imperative when evaluating the efficacy of an intervention, whether it be medical, therapeutic, or both.

Objective:

This study sought to examine the convergent and discriminant validity of the Columbia Impairment Scale (CIS) for youth with ASD, and their parents.

Methods:

A sample of 77 adolescents with ASD and their parents completed the CIS and various other measures that examined mood, anxiety, and behavior.

Results:

Although there was some evidence of convergent validity for the parent-report CIS, there was inadequate discriminant validity. The child-report version of the CIS yielded generally poor validity indices.

Conclusions:

There appear to be important limitations when using this measure for youth with ASD.

Introduction

Impairment at a clinically significant level in one or more areas of functioning is often a criterion that must be met in order to receive a mental health diagnosis (Lewandowski et al. 2009), as is the case with autism spectrum disorder (ASD). According to the Diagnostic and Statistical Manual of Mental Disorders, 5th ed. (DSM-V) (American Psychiatric Association [APA], 2013), symptom presentation must cause impairment in notable areas of functioning such as social and occupational, or in the case of youth, school. In treating ASD, it is helpful to understand what types of functioning are most impaired (i.e., social, language, behavioral) and what impact this has on the different areas of the patient's life (i.e., home, school, interpersonal).

This article examines the use of the Columbia Impairment Scale (CIS) (Bird et al. 1993) as a means of assessing functional impairment in youth with ASD. The measure consists of four subscales: Social/work functioning, use of leisure time, psychopathology, and interpersonal relationships, the combination of which provides a representation of global functioning. Previous research has found functional impairment in individuals with ASD to correlate with the number and severity of symptoms, as well as comorbid psychiatric disorders such as anxiety, which are commonly associated with this population (Kaat et al. 2013). In recent years, there has been an influx of treatment-based research focused on alleviating the symptoms of ASD, and, therefore, a measure of treatment outcome and change in global functioning as a response to treatment is needed (Rapee et al. 2012). As there has not yet been an examination of a measure designed to assess functional impairment in this population, it is important to determine whether there is evidence of convergent and divergent validity for the CIS when used with youth with ASD.

Methods

Participants

Participants were 77 youth with an ASD and a concurrent anxiety disorder between the ages of 11 and 15 years (mean=12.27 years, SD=1.13).The majority of participants were male (79.2%; n=61). According to parent report, the sample consisted of 72.3% white, 9.2% Latino, 6.2% Asian American, 3.1% African American, and 9.2% other/mixed participants.

Procedure

Data were collected as part of the intake evaluation of a multisite study examining the efficacy of cognitive behavioral therapy (CBT) to treat anxiety in adolescents with high-functioning ASD (see Wood et al. 2014, for description of the CBT study). The institutional review boards for human subject research at data collection sites approved the procedures, which included parent consent and child assent. Participants were recruited from medical center-based autism clinics, parent support groups, and regional centers. The Autism Diagnostic Observation Schedule (ADOS) (Lord et al. 2002) and the Autism Diagnostic Interview-Revised (ADI-R) (Lord et al. 1994) were used to confirm a diagnosis of ASD. All participants met criteria for a principal anxiety disorder, as assessed using the Anxiety Disorders Interview Schedule-Child/Parent (ADIS-C/P) (Silverman and Albano 1996). The Wechsler Intelligence Scale for Children (WISC) was used to assess the participants' full scale intelligence quotient (IQ). Youth were excluded if they had: 1) Received other interventions for anxiety; 2) started an antidepressant or antipsychotic medication, or changed medication within 6–12 weeks prior to commencing treatment; 3) exhibited clinically significant suicidality within the prior 6 months; or 4) met Diagnostic and Statistical Manual of Mental Disorders, 4th ed. (DSM-IV) criteria for bipolar disorder, schizophrenia, schizoaffective disorder, or substance abuse (APA 1994).

Validity measures

Columbia Impairment Scale-Parent and Child (CIS-P/C)

The CIS-P/C is a parent/child-report measures for youth 9–17 years of age. The CIS-P/C is composed of 13 questions scored on a scale of 0 (“no problem”) to 4 (“a very big problem”) (Bird et al. 1993).The measure assesses four areas of functioning: Social/work (two items; e.g. “Behavior at school/on the job?”), use of leisure time (two items; e.g., “Getting involved in activities like sports or hobbies?”), psychopathology (four items, e.g. “Feeling unhappy or sad?”), and interpersonal relationships (five items, “Getting along with his/her mother?”). The CIS has demonstrated high interrater reliability and concurrent and discriminant validity in typical populations (Bird et al. 1993).

Pediatric Anxiety Rating Scale (PARS)

The PARS (Research Units on Pediatric Psychopharmacology Anxiety Study Group 2002) is a clinician-rated scale assessing anxiety symptoms and the associated severity and impairment in youth over the past week. The PARS consists of 50 questions and seven severity/impairment items. The PARS has excellent psychometric properties.

Revised Child Anxiety and Depression Scale (RCADS)

The RCADS (Chorpita et al. 2005) is a self-report measure of symptoms of most anxiety disorders and depressive disorders for youth ages 6–18 years. The RCADS has demonstrated sound psychometrics (Chorpita et al. 2005). It is composed of 47 items corresponding to six subscales (social phobia, panic disorder, major depression, separation anxiety, generalized anxiety, and obsessive-compulsive disorder).

Achenbach System of Empirically Based Assessment (ASEBA)

The ASEBA school-aged measure (Achenbach 2001) is a psychometrically sound, 118 item scale that assesses specific behavioral and emotional problems from the parent's perspective, for youth 6–18 years. The ASEBA provides measures of internalizing and externalizing behaviors.

Multidimensional Anxiety Scale for Children–Parent Report (MASC-P)

The MASC-P (March 1997) is a standardized 39 item parent-report index of anxiety in youth, with excellent psychometric properties (Wood et al. 2002) and treatment sensitivity in ASD populations (Wood et al. 2009).

Results

Descriptive statistics

Descriptive statistics for the CIS-P are as follows: Total impairment score mean=22.22, SD=8.84; interpersonal relations subscale mean=6.72, SD=3.81; psychopathology subscale mean=8.08, SD=3.13; social subscale mean=3.46, SD=2.20; and leisure subscale mean=4.04, SD=1.88. Chronbach's α for the CIS-P were 0.84 for the total impairment score, 0.64 for the interpersonal relations subscale, 0.62 for the psychopathology subscale, and 0.73 for the social subscale. Descriptive statistics for the CIS-C are as follows: Total impairment score mean=14.52, SD=9.03; interpersonal relations subscale mean=5.06, SD=3.65; psychopathology subscale mean=5.35, SD=3.56; social subscale mean=2.00, SD=1.94; and leisure subscale mean=2.02, SD=1.92. Chronbach's α for the CIS-C were 0.82 for the total impairment score, 0.60 for the interpersonal relations subscale, 0.65 for the psychopathology subscale, and 0.58 for the social subscale. The leisure subscale was dropped for the CIS-P/C because both Chronbach's α were<0.6 which demonstrated a lack of internal consistency. Correlation analyses were run to examine convergent and discriminant validity.

Convergent validity analyses

The CIS-P total impairment score was significantly correlated with the CIS-C total score and the PARS severity of interference with family relationships scale, but not the PARS severity of interference with peer and adult relationships scales (see Table 1). CIS-P total impairment was significantly correlated with the ASEBA social subscale, but not with the competence subscale, which is made up of the social, activities, and school subscales. Of the CIS-P subscales, only the interpersonal relations and psychopathology subscales significantly correlated with the PARS severity of interference with family relationships; no significant correlations were found with ASEBA subscales. The CIS-C did not significantly correlate with the PARS or ASEBA scales.

Table 1.

Convergent Validity: Correlations Among CIS-P Total Scores and Scores from the CIS-C, PARS, and ASEBA

	CIS-P: Total Impairment Score
CIS-C: Total Impairment Score	0.35^**
PARS: Severity interference with family relationships	0.39^**
PARS: Severity interference with peer and adult relationships	−0.03
ASEBA: Social	0.53^**
ASEBA: Competence	0.12

^** p<0.01.

CIS-P, Columbia Impairment Scale, Parent; CIS-C, Columbia Impairment Scale, Child; PARS, Pediatric Anxiety Rating Scale; ASEBA, Achenbach System of Empirically Based Assessment.

Discriminant validity analyses

To evaluate discriminant validity, correlations between the CIS-P total impairment score and unrelated scales on the PARS, RCADS, and ASEBA were examined (see Table 2). The CIS-P total was not significantly correlated with anxiety severity as measured by the PARS clinician anxiety symptom totals, child anxiety symptom totals, overall severity of anxiety feelings, or overall severity of physical symptoms of anxiety; MASC-P total score; or RCADS total internalizing scale. However, the CIS-P total impairment score was significantly correlated with PARS overall number of anxiety symptoms. The CIS-P total impairment score was also significantly correlated with the ASEBA internalizing, externalizing, and attention subscales. Because the significant correlations between the CIS-P and these relatively unrelated constructs (e.g., externalizing) were generally as strong or stronger than the significant correlations in the convergent validity analyses, this pattern of findings provided little evidence of discriminant validity for the CIS-P. These analyses were not attempted with the CIS-C, given its poor convergent validity indices.

Table 2.

Discriminant Validity: Correlations Among CIS-P Total Score and PARS, RCADS, ASEBA, and MASC-P

	CIS-P: Total Impairment Score
PARS: Anxiety symptoms total-clinician	0.17
PARS: Anxiety symptoms total-child	0.01
PARS: Overall number of anxiety symptoms	0.33^*
PARS: Overall severity of anxiety feelings	0.02
PARS: Overall severity of physical symptoms of anxiety	−0.02
RCADS: Total internalizing score	0.09
ASEBA: Internalizing scale	0.41^**
ASEBA: Externalizing scale	0.71^**
ASEBA: Attention scale	0.39^**
MASC-P: Total anxiety score	0.13

^*p<0.05; ^** p<0.01.

CIS-P, Columbia Impairment Scale, Parent; PARS, Pediatric Anxiety Rating Scale; RCADS, Revised Child Anxiety and Depression Scale; ASEBA, Achenbach System of Empirically Based Assessment; MASC-P, Multidimensional Anxiety Scale for Children–Parent Report.

Discussion

The CIS-P converged with two other indices of impairment when used with adolescents with high functioning ASD. When compared with the PARS interference items, the CIS-P was associated with the PARS item that focused on the extent to which youth experienced impairment in their relationships with family as a result of anxiety, and in performance at home, but not with the item that focused on the youths' impairment, as a result of anxiety, in relationships or performance outside of the home. When compared with the ASEBA, the CIS-P was associated with the social subscale, which could be because both measures tapped into the same construct in regard to general problems with social functioning, irrespective of anxiety. The CIS-P did not correlate with the ASEBA competence scale (composed of social, activities, and school subscales). In terms of CIS-P subscales, only the interpersonal relations and psychopathology subscales correlated with the two PARS items that examined the impact of the youths' impairment on the family and on their relationships with adults outside of the home. This would be expected, given that the interference items from the PARS focus on the disorder's impact on interpersonal relations, aligning with the CIS-P interpersonal subscale, and that the PARS itself is a clinical measure of a specific psychological disorder (anxiety), which aligns with the CIS-P psychopathology subscale.

Several measures were used to evaluate discriminant validity; however, few yielded results indicative of differentiating the measures. Namely, on indices that seem unrelated, such as internalizing, externalizing, and attention, the ASEBA and CIS-P correlated with one another. The same was true of the PARS symptom count, which also correlated with the CIS-P, when it was hypothesized that no relationship would be found. These findings point to the potential overlap of these measures unintentionally tapping into the same constructs. In taking the nature of ASD into consideration, it is likely that a higher number of anxiety symptoms or increased severity in one of the ASEBA subscales could indicate greater functional impairment, suggesting the CIS-P is unable to differentiate these symptoms from overall functional impairment in ASD. The high convergence scores also suggest this to be true, as there was strong evidence of the similarities of these measures, implicating an inability to tap into variation within this population.

These findings should be interpreted while considering the weaknesses of the study such as the small sample size, the anxiety-specific subsample of youth with ASD, and the potential that not enough measures of functioning or impairment were included for comparison. Therefore, directions for future research could include additional studies, with larger sample sizes, that focus on validating or creating measures of impairment for use with youth with ASD. The lack of convergence with and divergence from other measures demonstrated by the CIS-C also needs to be further examined. This could be because of the lack of insight often observed in this population, resulting in decreased accuracy when reporting on themselves. Further exploration into the aspects of functioning that are most impaired in and specific to ASD would be useful for better understanding this complex disorder.

Clinical Significance

Overall, these findings underscore the need for caution when using the CIS with youth with ASD. Probably because of the heterogeneous manifestations of the symptoms and impairment associated with ASD, the CIS may fail to capture meaningful variability of impairment, per se, in ASD. Conservatively, these results suggest that without further research to the contrary, and with the limited overlap of the CIS with measures evaluating similar constructs and the lack of differentiation from measures of distinct constructs, the CIS may not be an appropriate measure to assess impairment in youth with ASD.

Footnotes

Disclosures

No competing financial interests exist.

References

Achenbach

: Manual for the ASEBA School-Age Forms and Profiles. Burlington: University of Vermont; 2001.

American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders, 4th ed. Washington, DC: American Psychiatric Association; 1994.

American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders, 5th ed. Washington, DC: American Psychiatric Association; 2013.

Bird

, Shaffer

, Fisher

, Gould

: The Columbia Impairment Scale (CIS): Pilot findings on a measure of global impairment for children and adolescents. Int J Methods Psychiatr Res, 3:167–176, 1993.

Chorpita

, Moffitt

, Gray

: Psychometric properties of the Revised Child Anxiety and Depression Scale in a clinical sample. Behav Res Ther, 43:309–322, 2005.

Kaat

, Gadow

, Lecavalier

: Psychiatric symptom impairment in children with autism spectrum disorders. J Abnorm Child Psychol, 41:959–969, 2013.

Lewandowski

, Lovett

, Gordon

: Measurement of symptom severity and impairment. In: Assessing impairment: From Theory to Practice, edited by Goldstein

and Naglieri

. New York: Springer, 5–14, 2009.

Lord

, Rutter

, DiLavore

, Risi

: Autism Diagnostic Observation Schedule: ADOS. Los Angeles: Western Psychological Services; 2002.

Lord

, Rutter

, Le Couteur

: Autism Diagnostic Interview-Revised: A revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. J Autism Dev Disord, 24:659–685, 1994.

10.

March

: Manual for the Multidimensional Anxiety Scale for Children (MASC). Toronto: Multi-Health Systems; 1997.

11.

Rapee

, Bőgels

, van der Sluis

, Craske

, Ollendick

: Annual research review: Conceptualizing functional impairment in children and adolescents. J Child Psychol Psychiatry, 53:454–468, 2012.

12.

Research Units on Pediatric Psychopharmacology Anxiety Study Group: The Pediatric Anxiety Rating Scale (PARS): Development and psychometric properties. J Am Acad Child Adolesc Psychiatry, 41:1061–1069, 2002.

13.

Silverman

, Albano AM Anxiety Disorders Interview Schedule (ADIS-IV) Child and Parent Interview Schedules: 5 of Each, Vol. 2. New York: Oxford University Press; 1996.

14.

Wood

, Drahota

, Sze

, Har

, Chiu

, Langer

: Cognitive behavioral therapy for anxiety in children with autism spectrum disorders: A randomized, controlled trial. J Child Psychol Psychiatry, 50:224–234, 2009.

15.

Wood

, Ehrenreich-May

, Alessandri

, Fujii

, Renno

, Laugeson

, Piacentini

, De Nadi

, Arnold

, Lewin

, Murphy

, Storch

. Cognitive behavioral therapy for early adolescents with autism spectrum disorder and clinical anxiety: A randomized controlled trial. Behav Ther, 2014 [Epub ahead of print].

16.

Wood

, Piacentini

, Bergman

, McCracken

, Barrios

: Concurrent validity of the anxiety disorders section of the anxiety disorders interview schedule for DSM-IV: Child and parent versions. J Clin Child Adolesc Psychol, 31:335–342, 2002.