The added value of the combined use of the Autism Diagnostic Interview–Revised and the Autism Diagnostic Observation Schedule: Diagnostic validity in a clinical Swedish sample of toddlers and young preschoolers

Abstract

The diagnostic validity of the new research algorithms of the Autism Diagnostic Interview–Revised and the revised algorithms of the Autism Diagnostic Observation Schedule was examined in a clinical sample of children aged 18–47 months. Validity was determined for each instrument separately and their combination against a clinical consensus diagnosis. A total of N = 268 children (n = 171 with autism spectrum disorder) were assessed. The new Autism Diagnostic Interview–Revised algorithms (research cutoff) gave excellent specificities (91%−96%) but low sensitivities (44%−52%). Applying adjusted cutoffs (lower than recommended based on receiver operating characteristics) yielded a better balance between sensitivity (77%−82%) and specificity (60%−62%). Findings for the Autism Diagnostic Observation Schedule were consistent with previous studies showing high sensitivity (94%−100%) and alongside lower specificity (52%−76%) when using the autism spectrum cutoff, but better balanced sensitivity (81%−94%) and specificity (81%−83%) when using the autism cutoff. A combination of both the Autism Diagnostic Interview–Revised (with adjusted cutoff) and the Autism Diagnostic Observation Schedule (autism spectrum cutoff) yielded balanced sensitivity (77%−80%) and specificity (87%−90%). Results favor a combined usage of the Autism Diagnostic Interview–Revised and Autism Diagnostic Observation Schedule in young children with unclear developmental problems, including suspicion of autism spectrum disorder. Evaluated separately, the Autism Diagnostic Observation Schedule (cutoff for autism) provides a better diagnostic accuracy than the Autism Diagnostic Interview–Revised.

Keywords

Asperger syndrome assessment early detection pervasive developmental disorder psychometrics

Introduction

In clinical practice and research, it has become increasingly important to diagnose toddlers and young preschoolers with autism spectrum disorder (ASD) before the age of 48 months (Bölte et al., 2013; Elsabbagh et al., 2012; Zwaigenbaum et al., 2005). Parental concerns regarding their child’s development often appear already during the second year of life (Baghdadli et al., 2003; Chawarska et al., 2007; De Giacomo and Fombonne, 1998; Hess and Landa, 2012), and a formal diagnosis is often a prerequisite for access to clinical service, such as early intervention, which may improve long-term outcomes for individuals with ASD (Dawson, 2008; Reichow, 2012; Rogers et al., 2012; Fein et al., 2013; Yirmiya, 2010).

A multitude of research from North America and the United Kingdom has provided evidence endorsing the possibility of early and reliably diagnosing ASD (Charman et al., 2005; Gonzalez et al., 1993; Lord et al., 2006; Piven et al., 1996; Sigman et al., 1999; Turner and Stone, 2007; Venter et al., 1992), particularly for experienced clinicians using standardized instruments (Corsello et al., 2013; Lord et al., 2006; Risi et al., 2006). Diagnostic and Statistical Manual of Mental Disorders (5th ed., DSM-5; American Psychiatric Association (APA), 2013) acknowledges the value of using behavioral standardized diagnostic instruments in the assessment of ASD by recommending questionnaires, caregiver interviews and observation measures to improve reliability of ASD diagnoses over time and between clinicians. Currently, the best evaluated standardized diagnostic instruments for evaluating suspected ASD are the Autism Diagnostic Interview–Revised (ADI-R; Rutter et al., 2003) and the Autism Diagnostic Observation Schedule (ADOS; Lord et al., 1999, 2012). The ADI-R is a comprehensive, structured caregiver interview that operationalizes the Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; DSM-IV-TR)/International Statistical Classification of Diseases and Related Health Problems 10th Revision (ICD-10) criteria for autism through an assessment of descriptions of an individual’s behavior, elicited by an experienced interviewer and a set of standardized questions. The ADOS is a structured observation protocol to collect a sample of an individual’s social-communication and stereotypic, restricted behavior patterns (Lord et al., 1999, 2012).

The main aim of this study was to examine the diagnostic validity of the ADI-R and the ADOS in an independent clinical sample consisting of children below 48 months of age. Until recently, neither instrument had been specifically validated for use with children below age 48 months, and few studies applied the tools’ revised algorithms. Most research have been conducted in US lab settings, and especially with regard to the ADOS, results have not been presented for all ASDs combined versus nonspectrum (NS), but often rather separately for autistic disorder or pervasive developmental disorder not otherwise specified (PDD-NOS) versus NS.

More precisely, the existing body of literature on the ADI-R and ADOS in young children with ASD can be summarized as follows: Previous studies indicate that the ADI-R might be either overinclusive (Lord et al., 1993; Risi et al., 2006) or underinclusive (Ventola et al., 2006; Wiggins and Robins, 2008) of autism in children with a nonverbal mental age below 24 months. Thus, the usage of the standard ADI-R in young and very low functioning individuals has not been encouraged in the past. Several attempts have been made to adapt the ADI-R for younger ages. For instance, studies used a “toddler” version of the ADI-R (Chawarska et al., 2007; Lord et al., 2004; Richler et al., 2007), which essentially represent an extended form of the ADI-R without diagnostic algorithm. Recently, Kim and Lord (2012b) developed and evaluated a new set of ADI-R research algorithms for toddlers and young preschoolers aged 12–47 months with mental ages of 10 months and above. Three novel algorithms depending on the child’s age and expressive language showed promising diagnostic validity: children aged 12–20 months and those aged 21–47 months without speech (NV), children aged 21–47 months using single words (SW), and children aged 21–47 months with developed phrase speech (PH). The validity of these algorithms were confirmed by Kim et al. (2013) in two independent research samples. Sensitivities for ASD versus NS disorders using the clinical cutoff were 85%−90% in the NV group, 94%−97% in the SW group, and 80%−89% in the PH group, while specificities ranged from 64% to 94% in the NV group, from 58% to 83% in the SW group, and from 70% to 94% in the PH group. When excluding children with ASDs other than autism, sensitivities against NS increased. Compared to the standard diagnostic algorithm, the new ADI-R research algorithms for young children yielded better balanced sensitivities and specificities, that is, increased specificity especially for the NV group and decreased sensitivity in some cases (Kim and Lord, 2012b). Kim et al. (2013) concluded that the new research algorithms, more closely reflecting the conception of ASD in DSM-5, seemed to better correspond to clinical judgment, at least in the United States.

Despite a rich literature corroborating the diagnostic validity of the ADOS, it is known that chronological age and developmental level may substantially influence ADOS scores (Gotham et al., 2007). In particular, the ADOS module 1 was overly inclusive in children with nonverbal mental ages < 15 months (Gotham et al., 2007). Thus, more recently, the new ADOS toddler module (Luyster et al., 2009) was developed. The toddler module is part of an updated version of the ADOS, the ADOS-2 (Lord et al., 2012), which also includes new algorithms for modules 1–3, which are improved to better fit the child’s expressive language level and age and also comprise items from the repetitive and restricted behavior domain (Gotham et al., 2007). Aside from two large US research samples (Gotham et al., 2007, 2008), the validity of the new algorithms has been examined in two Dutch research samples (De Bildt et al., 2009; Oosterling et al., 2010). In Gotham et al. (2007, 2008), sensitivities for module 1 (no words and some words) and module 2 (<5 years of age algorithm) ranged from 86% to 98%, whereas specificities were 80% to 100% using the autism cutoff for autism only versus NS. The diagnostic validity for non-ASD versus NS was generally lower. The largest gain of accuracy was found in module 1 (no words) for children with nonverbal mental age below 15 months. Here, the specificity increased from 19% to 50% for autism versus NS. Thus, the problem of overinclusivity of the ADOS for very young and developmentally delayed children was reduced but remained a concern. In the Dutch samples, specificities and particularly sensitivities were generally lower than in the US studies (De Bildt et al., 2009; Oosterling et al., 2010). Oosterling et al. (2010) reported sensitivities (61%−93%) and specificities (70%−86%) for ASD versus NS for module 1 (no words and some words) and module 2.

The revised algorithms have also been used in clinical settings. Gray et al. (2008), in a clinical Australian sample, obtained sensitivities (89%−98%) and specificities (82%−86%) for module 1 (no words and some words) comparable to those found in US research samples using the autism cutoff for the comparison autism versus non-ASD plus NS. ASD versus NS using the autism spectrum cutoff yielded somewhat lower sensitivities (78%−92%) and specificities (86%−92%). In a US clinical sample, using the autism spectrum cutoff, Molloy et al. (2011) reported sensitivities for ASD versus NS of 76%−98% for module 1 (no words and some words) and module 2; specificities were 29%−60%. When the autism cutoff was applied, sensitivities were 63%−83% and specificities 65%−81%. In summary, the evidence generally endorses the usefulness of the new ADOS algorithms. Nevertheless, as the research described above examined children of a large age range (13–144 months) without an explicit separate analysis of young children, the diagnostic accuracy of the ADOS in children aged less than 48 months is unknown.

The ADI-R and the ADOS represent considerably different but equally important strategies and sources of data collection: interviewer guided verbal parent-report, including history taking (ADI-R), and expert rating of a prompted social interaction sample (ADOS). A moderate to substantial agreement of the ADOS with clinical diagnosis (κ > .59), but only a slight to moderate agreement between the ADI-R and clinical diagnosis (κ = .15–.46) has been reported (Gray et al., 2008; Ventola et al., 2006). The convergence of the ADI-R and ADOS has shown mixed results. Ventola et al. (2006) found that their agreement of the diagnostic classifications of the ADI-R and the ADOS was below chance (κ = −.07), while Gray et al. (2008) reported a fair agreement (κ = .35). Le Couteur et al. (2008) found significant correlations between different domain totals of the ADI-R and the ADOS (r = .51–.71). In young children aged below 36 months, Risi et al. (2006) found a correlation of r = .60 between ADI-R and ADOS domain.

Generally, a combined rather than a separate usage of the ADI-R and ADOS is recommended and has sometimes being labeled the “gold standard” of diagnosing ASD. Risi et al. (2006) reported well-balanced sensitivities and specificities (~80%) in most cases for both autism versus NS and ASD versus NS in children below 36 months of age, when diagnosis was based on combined ADI-R and ADOS use. In Le Couteur et al. (2008), the categorical agreement of the combination of the ADI-R and the ADOS with diagnosis was 67% for autism and 16% for ASD.

Studies investigating the value of the ADI-R/ADOS combined and the single use of the instruments have applied the standard algorithms of the ADI-R and the ADOS. They often presented results separately for children with different forms of ASD (not the whole autism spectrum group versus NS) and were in the majority conducted in research settings. Only one study examined the diagnostic validity of combined ADI-R/ADOS usage and their new algorithms in children below 48 months of age. In a sample of N = 604 children aged 12–47 months, recruited from both research projects and clinical practice, Kim and Lord (2012a) found well-balanced sensitivity and specificity (≥80%) for all ASD versus NS comparisons (ADI-R clinical and ADOS autism spectrum cutoff). The separate usage of the ADI-R and the ADOS showed lower and less well-balanced sensitivity and specificity compared to their combined usage.

As mentioned initially, and described in this overview, despite rich evidence generally supporting the diagnostic validity of the ADI-R and the ADOS, there is an apparent scarcity of studies on their accuracy using the new algorithms in young children with suspicion of ASD, particularly in purely clinical settings outside of the United States. In addition, only one study has yet examined the combined use of the instruments in this age group.

Methods

Participants

The current study comprised N = 268 (76% boys) children below 48 months of age assessed between 2006 and 2012 at the Neuropsychiatric Resource Team Southeast, Division of Child and Adolescent Psychiatry, Stockholm County Council. The unit is a multidisciplinary diagnostic specialist clinic and part of the public health care system. It serves preschool children aged below 48 months of age (~60 children/year). Children are either referred to the unit via other departments of the health care system or directly by their caregivers. The study was approved by the Regional Board of Ethical Vetting, Stockholm.

The mean age of the participating children was 37.9 months (standard deviation (SD) = 7.2 months, range: 18–47 months). Following assessment, 171 children were given a diagnosis of ASD (autism: 103, PDD-NOS: 68). Ninety-seven children were classified as a NS: 67 children received a neurodevelopmental diagnosis other than ASD (intellectual disability: 9, attention deficit hyperactivity disorder (ADHD): 16, language disorder: 42), and 30 children no psychiatric diagnosis. The majority of the children without psychiatric diagnosis exhibited different kinds of special needs due to developmental delays and adaptive and behavioral problems that were too subtle or vague for qualify for a diagnosis, and therefore, they were included in the NS group in all analyses. Gender distribution did not differ between the diagnostic groups (χ²(1) = 2.64, p = .10). The maternal country of origin was used as a proxy for ethnicity (it is illegal to register race or ethnicity in Sweden): 59% of the children were of Swedish origin, 11% of other European, and 30% of non-European origin (on average in Stockholm County, around 80% of the children are of Swedish origin). Ethnicity was not associated with diagnosis (χ²(2) = 2.25, p = .32). Sample characteristics by ADI-R algorithms and ADOS modules are presented in Tables 1 and 2.

Table 1.

Sample description by ADI-R developmental cells.

	12–20/NV21–47				SW21–47				PH21–47
	ASD	NS	F	p	ASD	NS	F	p	ASD	NS	F	P
Gender (male, female)	41, 18	11, 2			46, 10	20, 12			39, 8	34, 13
Age in months (SD)	32.4 (7.0)	33.4 (9.6)	.20	.655	38.6 (4.8)	38.9 (7.3)	.04	.837	41.9 (4.2)	41.9 (5.2)	.00	.983
Mean NVIQ (SD)	67.0 (19.0)	79.2 (24.3)	3.57	.064	73.3 (14.0)	84.8 (17.0)	10.98	.001	86.8 (10.9)	94.3 (13.4)	7.53	.007
Mean VABC (SD)	67.8 (7.8)	81.1 (7.6)	31.08	<.001	74.8 (8.7)	82.09 (8.2)	14.66	<.001	77.2 (8.1)	84.2 (9.9)	13.95	<.001
ADI-R SA/SC	9.7 (4.2)	4.6 (4.0)	15.46	<.001	8.6 (4.3)	5.1 (3.8)	15.00	<.001	7.9 (4.7)	4.3 (3.7)	16.74	<.001
ADI-R RRB	3.0 (2.2)	1.8 (1.7)	3.80	.055	3.0 (2.5)	1.5 (1.4)	9.26	.003	2.8 (2.5)	1.9 (2.4)	3.59	.061
ADI-R IGP/RPI	8.8 (2.8)	4.5 (3.2)	24.70	<.001	5.3 (2.7)	3.4 (2.2)	15.72	<.001	3.5 (2.0)	2.0 (1.8)	14.57	<.001
ADI-R total	12.7 (5.6)	6.4 (5.1)	13.83	<.001	11.6 (6.1)	6.6 (4.2)	16.69	<.001	14.2 (7.4)	8.2 (6.2)	18.27	<.001
ADOS SA	16.9 (3.0)	8.8 (4.9)	54.70	<.001	13.0 (3.9)	6.1 (4.6)	54.87	<.001	12.8 (3.2)	5.5 (3.3)	120.30	<.001
ADOS RRB	3.7 (2.0)	1.4 (1.4)	14.52	<.001	2.5 (1.8)	1.0 (1.1)	18.50	<.001	2.4 (1.8)	0.7 (0.8)	33.39	<.001

NVIQ: nonverbal IQ; VABC: Vineland-II Adaptive Behavior Composite; ADI-R SA/SC: Autism Diagnostic Interview–Revised Social Affect/Social Communication; RRB: Restricted and Repetitive Behavior; IGP/RPI: Imitation, Gestures, and Play/Reciprocal Peer Interaction; ADOS: Autism Diagnostic Observation Schedule; SA: Social Affect; RRB: Restricted and Repetitive Behavior; 12–20/NV21–47: all children 12–20 months + nonverbal children 21–47 months; SW21–47: children 21–47 months with single words; PH21–47: children 21–47 months with phrase speech; SD: standard deviation.

In the 12–20/NV21–47 group, three children were less 21 months old and NVIQ was available for 43/59 children; SW21–47: NVIQ was available for 49/56 children; PH21–47: NVIQ available for 37/47 children with ASD and VABC for 46/47 children.

Table 2.

Sample description by ADOS module and algorithm.

DX	ADOS module 1 no words						ADOS module 1 some words							ADOS module 2, younger than 5 years
	N	Mean	SD	Range	F	p	N	Mean	SD	Range	F	p	N	Mean	SD	Range	F	p
ASD
Gender (male, female)	58, 14						50, 14						25, 7
Age	72	33.6	7.2	21–47	1.3	.25	64	38.3	5.7	20–47	.3	.566	32	42.6	3.6	34–47	.3	.603
VIQ	39	42.6	27.6	6–94	20.1	< .001	41	60.4	21.6	10–94	8.5	.005	29	85.6	10.0	65–100	1.7	.199
NVIQ	52	68.1	17.6	23–94	9.1	< .001	52	76.2	14.5	45–109	6.0	.016	28	88.2	10.0	65–112	6.2	.015
VABC	72	68.7	7.3	51–83	42.6	< .001	63	74.8	9.4	53–100	5.0	.028	32	78.6	7.4	66–95	13.9	<.001
ADOS SA	72	16.6	2.8	9–20	66.7	< .001	64	12.7	3.8	5–20	67.7	<.001	32	12.5	3.2	6–19	87.7	<.001
ADOS RRB	72	3.4	2.1	0–8	13.7	< .001	64	2.4	1.5	0–5	29.7	<.001	32	2.5	1.9	0–8	26.0	<.001
ADOS SA + RRB	72	20.1	3.9	10–27	76.5	< .001	64	15.1	4.3	6–24	81.9	<.001	32	15.0	3.8	7–22	109.0	<.001
NS
Gender	18, 3						15, 14						29, 11
Age	21	35.7	7.9	20–45			29	39.0	6.5	25–47			40	43.1	4.4	29–47
VIQ	20	72.4	15.1	46–102			28	73.9	14.0	40–100			40	90.5	17.6	44–139
NVIQ	21	81.1	14.2	65–110			29	84.5	15.0	39–114			40	95.5	12.9	71–132
VABC	21	80.7	7.8	65–97			29	79.3	8.2	66–102			40	86.0	9.2	65–102
ADOS SA	21	9.8	4.9	4–17			29	5.7	3.8	1–15			40	5.2	3.4	0–14
ADOS RRB	21	1.5	1.3	0–4			29	0.8	1.0	0–4			40	0.8	0.8	0–3
ADOS SA + RRB	21	10.9	5.2	4–20			29	6.4	4.2	1–19			40	5.9	3.8	0–15

ASD: autism spectrum disorder; NS: nonspectrum; VIQ: verbal IQ; NVIQ: nonverbal IQ; VABC: Vineland-II Adaptive Behavior Composite; ADOS: Autism Diagnostic Observation Schedule; SA: Social Affect; RRB: Restricted and Repetitive Behavior; ADOS SA + RRB: total score; nine children in module 1 no words (NW) had a nonverbal mental age (NVMA) of 15 months or below.

Measures

ADI-R

The Swedish version of the ADI-R using the new algorithms (Kim and Lord, 2012b) was administered. The new research algorithms reflect the conceptualization of ASD in DSM-5 and are consistent with the ADOS-2 algorithms. Items compose the domains of Social Affect (SA)/Social Communication (SC), Restricted and Repetitive Behavior (RRB), and Imitation, Gestures, and Play (IGP)/Reciprocal Peer Interaction (RPI). Different item combinations are used in the algorithms to generate a single total score in each ADI-R developmental group. For the NV and SW groups, the SA and RRB domains, but not the items of the IGP domain, are combined to generate cutoff scores (NV: 13 items and SW: 16 items), while for the PH group, the total scores of the SC, RRB and RPI domains (20 items) are combined. There is a clinical lower cutoff, optimizing sensitivity, and a higher research cutoff, which optimizes specificity, as well as ranges of concern (little-to-no, mild-to-to moderate, and moderate-to-severe), comparable to the scoring and interpretation on the ADOS toddler module (Lord et al., 2012; Luyster et al., 2009). According to the new ADI-R research algorithm developmental group nomenclature (Kim and Lord, 2012b) (see introduction), 72 participants fell into the NV group (three of them were aged < 21 months), 88 into the SW group, and 94 into the PH group. In the present sample, in the NV group, verbal IQ (VIQ) correlated with the different ADI-R domain totals (r = −.45 to −.50, p < .005) and age correlated with both the total score and the SA domain score (r = −.34, p ≤ .004). No other participant characteristics (age, VIQ, and nonverbal IQ (NVIQ)) and domain totals correlated significantly or r > .40.

ADOS

The Swedish version of the ADOS (Lord et al., 1999) was used applying the revised algorithms included in the ADOS-2 (Gotham et al., 2007; Lord et al., 2012). Depending on the child’s expressive language level and/or age, the observer chooses a specific set of activities operationalized through the modules in order to minimize the influence of expressive language and developmental level. Module 1 is conceived for children that have not developed any fluent phrase speech. Herein, depending on the expressive language level, two different algorithms are used for scoring: one if the child has no words (module 1 no words), the other if the child has some words (module 1 some words). For the youngest children without phrase speech, between 12 and 30 months, the new toddler module is now available (Luyster et al., 2009). Module 2 addresses children who use phrases flexibly but not yet in a complex way. Again, there are two ways of scoring depending on the child’s age: below 5 years or 5 years and older. Module 3 is designed for children with fluent speech used in a complex way and is scored using one algorithm. In the revised ADOS algorithms, items from the Social Interaction and Communication domains have been restructured to form the SA domain. The RRB domain, not part of the original algorithm scoring, has been added. Even though consisting of different combinations, all revised algorithms have the same number of items being summed to a total score to increase their comparability. On all algorithms, the scores of 14 items from the SA and the RRB domains are combined. Like in the standard algorithms, the new algorithms provide cutoffs for autism spectrum and autism, except for the toddler module that has three different “ranges of concern” instead of a diagnostic classification. Five children were given the toddler module, 93 were given module 1 no words algorithm, 93 module 1 some words algorithm, 72 were given module 2 (<5 years algorithm), and two module 3. In the current sample, correlations between age, VIQ and NVIQ and ADOS domain totals were r ≤ −.40 in most cases. In module 1 no words, VIQ and NVIQ and the ADOS total score was r = −.47 (p < .001) and in module 1 some words, VIQ and the ADOS total and SA domain totals was r = −.41 (p < .001) in both cases. The correlation between age and domain totals was r ≤ −.21 in all modules.

Intellectual and adaptive functioning

Merrill-Palmer-R (Roid and Sampers, 2005), Wechsler Preschool and Primary Scales of Intelligence–Third Edition (Wechsler, 2004), or Mullen Scales of Early Learning (Mullen, 1995) were used to measure NVIQ and VIQ (see Tables 1 and 2). They were calculated by averaging the age equivalents of the nonverbal subtests (Visual Reception and Fine Motor (Mullen) and Cognitive and Fine Motor (Merrill-Palmer-R)), and the verbal subtests to obtain mental age, which was divided by chronological age and multiplied by 100. ASD children had lower NVIQ than NS children, except for the NV ASD group. To measure adaptive function, the Vineland Adaptive Behavior Scales-II (Sparrow et al., 2005) was used, a structured caregiver interview that assesses age appropriate self-sufficiency skills. We applied the summary score of the Vineland-II, the Adaptive Behavior Composite, to describe functional level of the participants (Table 1). Levels of adaptive functioning in ASD were generally lower than in NS.

Procedure

Participants were referred to the Neuropsychiatric Resource Team Southeast due to unclear developmental concerns, for instance, language delay or global developmental delay, interaction difficulties and internalizing or externalizing behavior problems. Subsequently, children underwent a developmental assessment routine by a multidisciplinary team consisting of child psychiatrists, psychologists, and social workers. First, history taking was carried out by a child psychiatrist interviewing at least one caregiver, followed by a psychologist testing the child’s cognitive abilities and administering the Vineland-II. Parents not having enough knowledge in Swedish language were interviewed (history taking, ADI-R, Vineland-II) assisted by a certified interpreter. In addition, a psychologist, social worker, or child psychiatrist spent half a day on observing the child’s in his or her (natural) preschool environment and interviewing the staff. The ADOS was administered by a psychologist and/or a social worker, one administering and the other only passively observing. Both scored the examination independently and reached consensus scores after discussion. The ADI-R was administered by any of the team members but most often by one of the child psychiatrists not familiar with the child and not taking part in the diagnostic clinical consensus discussion. Clinicians administering the ADI-R or ADOS were either official clinically trained on the instruments or a certified trainer. The first and last authors are certified ADI-R and ADOS trainers. All available information, results and observational data from the assessments were discussed by the team having seen the child and his or her parents to generate a clinical consensus diagnosis according to DSM-IV-TR.

Analyses

Diagnostic validity was determined by calculating sensitivity and specificity with 95% confidence intervals (Wilson score method; Newcombe, 1998) as well as classification accuracy (% correctly diagnosed) and positive likelihood ratios (LR+) for different combinations of single and combined use of the ADI-R and the ADOS compared to clinical consensus diagnosis. In order to enable comparison with previous studies, the sample was divided into developmental groups according to the new ADI-R algorithms (NV, SW, and PH) when analyzing ADI-R data ADI-R/ADOS combined, and according to the module 1 (no words and some words) and module 2 when analyzing the ADOS data alone (no separate analyses for the toddler module (n = 5) and module 3 (n = 2)). Receiver operating characteristics (ROC) statistics with area under the curve (AUC) were computed for all developmental groups of the ADI-R and modules of the ADOS to examine the discriminative properties of the instruments. Pearson’s correlation between the domain and the total scores of the ADI-R and the ADOS were calculated to assess the agreement between the two instruments, and correlation coefficients differences were analyzed with Fischer’s Z transformation (Cohen and Cohen, 1983). Kappa statistics (κ) were used to determine categorical agreement between the ADI-R, ADOS, and clinical consensus diagnosis. Characteristics of correctly and misclassified children as well as the differences of total scores on the ADI-R and the ADOS between groups were examined with Bonferroni post hoc tests and effect sizes were calculated with Cohen’s d.

Results

ADI-R

The domain totals in all developmental groups differed statistically significantly between the children with ASD and the children with NS except for the RRB domain in the NV group (F(1, 71) = 3.80, p = .055) and PH group (F(1, 92) = 3.59, p = .061) (Table 1). Using the clinical cutoff of the new algorithms, sensitivities ranged from 53% to 70% and specificities from 69% to 81%, while the research cutoff yielded sensitivities from 44% to 52% and specificities >90%. The amount of correctly diagnosed cases ranged between 60% and 70% for both sets of algorithms. LR+ were 2.2–2.8 for the clinical cutoff, and 4.6–10.5 for the research cutoff. The AUC was NV: .79, SW: .75, and PH: .74 (all p < .001). Inspecting the ROC curves revealed that lowering cutoff points in the current sample compared with the ones proposed by Kim and Lord (2012b) improved the balance between sensitivity and specificity and the classification accuracy in all groups (see Table 3). A more useful clinical cutoff, that is, higher sensitivity and accuracy, for this sample in the NV group was 8 (instead of 11 in Kim and Lord, 2012b), 7 (instead of 8) in the SW group, and 9 (instead of 13) in the PH group. The sensitivities then increased to around 80% and the accuracies to >70% in two of the three groups but did not affect the LR+s in a positive way (1.9–2.2).

Table 3.

Validity of ADI-R, ADOS, and ADI-R/ADOS combined for all conditions tested, with the sample divided by developmental cells of the ADI-R.

	Numbers of					Sensitivity 95% CI	Specificity 95% CI	ASD vs NS	Accuracy (%)
	N	TP	TN	FP	FN	ASD	NS	LR+	Accuracy (%)
12–20/NV21–47
ADI-R (CLI) and ADOS (AS)	66	32	10	0	24	57% (44–69)	100% (72–100)	–	64
ADI-R (RES) and ADOS (AS)	66	29	10	0	27	52% (39–64)	100% (72–100)	–	59
ADI-R (CLI) and ADOS (AUT)	66	31	10	0	25	55% (42–68)	100% (72–100)	–	62
ADI-R (RES) and ADOS (AUT)	66	28	10	0	11	50% (37–63)	100% (72–100)	–	58
ADI-R (ADJUSTED) and ADOS (AS)	66	45	9	1	11	80% (68–89)	90% (60–98)	8.0 (1.2–51.8)	82
ADI-R (ADJUSTED) and ADOS (AUT)	66	43	9	1	13	77% (64–86)	90% (60–98)	7.7 (1.2–49.6)	79
ADI-R (CLI) alone	72	34	10	3	25	58% (45–69)	77% (50–92)	2.5 (0.9–6.9)	61
ADI-R (RES) alone	72	31	12	1	28	52% (40–65)	92% (67–99)	6.8 (1.0–45.6)	60
ADI-R (ADJUSTED) alone	72	48	8	5	11	81% (70–89)	61% (35–82)	2.1 (1.0–4.2)	78
ADOS (AS) alone	66	55	6	4	1	98% (90–100)	60% (31–83)	2.5 (1.2–5.2)	92
ADOS (AUT) alone	66	53	9	1	3	95% (85–98)	90% (60–98)	9.5 (1.5–60.8)	94
SW21–47
ADI-R (CLI) and ADOS (AS)	88	36	28	4	20	64% (51–76)	88% (72–95)	5.1 (2.0–13.1)	73
ADI-R (RES) and ADOS (AS)	88	24	32	1	31	44% (31–57)	97% (85–99)	14.4 (2.0–101.5)	64
ADI-R (CLI) and ADOS (AUT)	88	27	29	3	29	48% (36–61)	91% (76–97)	5.1 (1.7–15.6)	64
ADI-R (RES) and ADOS (AUT)	88	19	31	1	37	34% (23–47)	97% (84–99)	10.9 (1.5–77.3)	57
ADI-R (ADJUSTED) and ADOS (AS)	88	43	28	4	13	77% (64–86)	88% (72–95)	6.1 (2.4–15.5)	81
ADI-R (ADJUSTED) and ADOS (AUT)	88	33	28	4	23	59% (46–71)	88% (72–95)	4.7 (1.8–12.1)	69
ADI-R (CLI) alone	88	39	22	10	17	70% (57–80)	69% (51–82)	2.2 (1.3–3.8)	69
ADI-R (RES) alone	88	24	29	3	32	44% (31–56)	91% (76–97)	4.6 (1.5–14.0)	60%
ADI-R (ADJUSTED) alone	88	46	20	12	10	82% (70–90)	62% (45–77)	2.2 (1.4–3.5)	75
ADOS (AS) alone	88	52	23	9	4	93% (83–97)	72% (55–84)	3.3 (1.9–5.8)	85
ADOS (AUT) alone	88	41	26	6	15	73% (60–83)	81% (65–91)	3.9 (1.8–8.2)	76%
PH21–47
ADI-R (CLI) and ADOS (AS)	93	25	43	3	22	53% (39–67)	94% (82–98)	8.2 (2.6–25.2)	73
ADI-R (RES) and ADOS (AS)	93	21	46	0	26	45% (31–59)	100% (93–100)	–	72
ADI-R (CLI) and ADOS (AUT)	93	24	44	2	24	51% (37–65)	96% (85–99)	11.7 (2.9–46.9)	73
ADI-R (RES) and ADOS (AUT)	93	20	46	0	27	43% (30–57)	100% (92–100)	–	71
ADI-R (ADJUSTED) and ADOS (AS)	93	36	40	6	11	77% (63–86)	87% (74–94)	5.8 (2.7–12.6)	82
ADI-R (ADJUSTED) and ADOS (AUT)	93	32	42	4	15	68% (54–80)	91% (80–97)	7.8 (3.0–20.4)	80
ADI-R (CLI) alone	94	25	38	9	21	53% (39–67)	81% (67–90)	2.8 (1.5–5.3)	67
ADI-R (RES) alone	94	21	45	2	26	45% (31–59)	96% (86–99)	10.5 (2.6–42.3)	70
ADI-R (ADJUSTED) alone	94	36	28	16	0	77% (63–86)	60% (45–72)	1.9 (1.3–2.8)	68
ADOS (AS) alone	93	47	30	16	0	100% (92–100)	65% (51–77)	2.9 (1.9–4.3)	83
ADOS (AUT) alone	93	41	38	8	6	87% (75–94)	83% (69–91)	5.0 (2.6–9.5)	85
All children
ADI-R (CLI) and ADOS (AS)	247	93	81	7	66	58% (51–66)	92% (84–94)	7.4 (3.6–15.2)	70
ADI-R (RES) and ADOS (AS)	247	74	87	1	85	46% (39–54)	99% (94–100	41.0 (5.8–290.0)	65
ADI-R (CLI) and ADOS (AUT)	247	82	83	5	77	52% (44–59)	94% (87–98)	9.1 (3.8–21.6)	67
ADI-R (RES) and ADOS (AUT)	247	67	87	1	92	42% (35–50)	99% (94–100)	37.1 (5.2–262.5)	63
ADI-R (ADJUSTED) and ADOS (AS)	247	124	77	11	35	78% (71–84)	88% (79–93)	6.2 (3.6–10.9)	81
ADI-R (ADJUSTED) and ADOS (AUT)	247	108	79	9	51	68% (60–75)	90% (82–94)	6.6 (3.5–12.4)	76
ADI-R (CLI) alone	254	98	70	22	64	60% (53–68)	76% (66–84)	2.5 (1.7–3.7)	66
ADI-R (RES) alone	254	76	86	6	86	47% (39–55)	94% (88–97)	7.2 (3.3–15.9)	64
ADI-R (ADJUSTED) alone	254	130	56	36	32	80% (73–86)	61% (51–70)	1.9 (1.4–2.6)	73
ADOS (AS) alone	260	163	60	32	5	97% (93–99)	65% (55–74)	2.8 (2.1–3.7)	86
ADOS (AUT) alone	260	142	75	17	26	84% (78–89)	82% (72–88)	4.6 (3.0–7.0)	83

12–20/NV21–47: all children 12–20 months and nonverbal children 21–47 months; SW21–47: children 21–47 months with single words; PH21–47: children 21–47 months with phrase speech; ADI-R: Autism Diagnostic Interview–Revised: ADOS Autism Diagnostic Observation Schedule; CLI: clinical cutoff, RES: research cutoff; ADJUSTED: own adjusted cutoff; AS: autism spectrum cutoff; AUT: autism cutoff; CI: confidence interval; TP: true positives; TN: true negatives; FP: false positives; FN: false negatives; ASD: autism spectrum disorder; NS: nonspectrum.

Accuracy % correctly classified. The cutoffs of the 12–20/NV21–47 group are 13 (RES), 11 (CLI), and 8 (ADJUSTED); of the SW21–47 group are 13 (RES), 8 (CLI), and 7 (ADJUSTED); and of the PH21–47 are 16 (RES), 13 (CLI), and 9 (ADJUSTED).

Kappas for the agreement between different ADI-R cutoffs and clinical consensus diagnosis were κ = .21 (p = .024) for the clinical cutoff, κ = .25 (p = .003) for the research cutoff, and κ = .36 (p = .001) for the adjusted cutoff in the NV group. The agreement in the SW group was κ = .37 (p < .001) (clinical), κ = .28 (p = .001) (research), and κ = .45 (p < .001) (adjusted). In the PH group corresponding kappas were κ = .34 (p = .001) (clinical), κ = .40 (p < .001) (research), and κ = .36 (p < .001) (adjusted).

When looking at the children who were misclassified by the ADI-R, some differences emerged regarding age and Vineland-II scores between true positives (TP) and false negatives (FN) as well as between false positives (FP) and true negatives (TN) on the ADI-R (Table 4). FP (NS children falsely classified as ASD) were younger than TN (NS children correctly classified as NS) on the ADI-R (F(3, 250) = 8.22, post hoc: FN > TP, p < .001, d = .90), while FN (children with ASD falsely classified as NS) had higher Vineland-II scores than the TP (children with ASD correctly classified) (F(3, 249) = 34.27, post hoc: TN > FP, p < .001, d = .83).

Table 4.

Characteristics of misclassified children.

	N	All children
		ADI-R PPV = .82; ADOS PPV = .84; ADI-R ADJUST PPV = .78		NPV = .52; NPV = .92; NPV = .64
		Age	NVIQ	VABC
ADI-R TP	98	36.8 (7.2)	73.9 (17.7)	70.3 (9.2)
ADI-R FN	64	38.0 (6.2)	76.8 (15.8)	77.1 (7.2)
ADI-R FP	22	34.6 (8.8)	89.0 (19.6)	81.8 (9.3)
ADI-R TN	70	41.2 (5.8)	88.8 (16.8)	83.4 (9.0)
ADOS TP	163	37.1 (6.9)	75.4 (16.9)	72.6 (9.0)
ADOS FN	5	36.6 (8.5)	78.2 (13.3)	81.2 (4.3)
ADOS FP	32	39.0 (6.8)	87.7 (15.4)	83.7 (7.6)
ADOS TN	60	40.8 (6.6)	90.0 (16.0)	82.2 (9.8)
ADI-R adjusted TP	130	37.0 (6.8)	74.9 (17.6)	71.7 (9.0)
ADI-R adjusted FN	32	38.7 (6.8)	75.9 (14.2)	78.2 (7.6)
ADI-R adjusted FP	36	36.9 (8.5)	88.9 (19.5)	80.8 (9.5)
ADI-R adjusted TN	56	41.4 (5.7)	88.8 (16.0)	84.4 (8.5)

PPV: positive predictive value; NPV: negative predictive value; ADI-R: Autism Diagnostic Interview–Revised; ADI-R adjusted: own adjusted cutoff; ADOS: Autism Diagnostic Observation Schedule; TP: true positives; FN: false negatives; FP: false positives; TN: true negatives.

Sample size was too limited to permit an analysis of data for the developmental cells separately.

ADOS

The ASD group had higher domain totals (SA and RRB together) than the NS group in all modules: module 1 no words: F(1, 91) = 76.5, p < .001; module 1 some words: F(1, 91) = 81.9, p < .001; and module 2: F(1, 70) = 109.0, p < .001 (see Table 2). When using the autism spectrum cutoff, sensitivities for module 1 (no words and some words) and module 2 ranged between 94% and 100%, while specificities ranged between 52% and 76%. Using the autism cutoff yielded more balanced sensitivities and specificities in all modules, that is, lower sensitivities (81%−94%), alongside higher specificities (81%−83%). The rate of correct classifications was in the same range for both sets of algorithms, 80%−88% for the autism spectrum cutoff and 82%−88% for the autism cutoff. The LR+s ranged between 2.1 and 3.9 for the autism spectrum cutoff and between 4.4 and 5.4 for the autism cutoff (Table 5).

Table 5.

Diagnostic validity of the ADOS by ADOS algorithms.

	N	TP	TN	FP	FN	Sensitivity 95% CI	Specificity 95% CI	ASD vs NS	Accuracy
						ASD	NS	Positive likelihood ratio	% correctly classified
ADOS module 1 no words (AS)	93	71	11	10	1	99% (92–100)	52% (32–72)	2.1 (1.3–3.2)	88
ADOS module 1 no words (autism)	93	60	17	4	12	83% (73–90)	81% (60–92)	4.4 (1.8–10.6)	83
ADOS module 1 some words (AS)	93	60	22	7	4	94% (85–98)	76% (58–88)	3.9 (2.0–7.4)	88
ADOS module 1 some words (autism)	93	52	24	5	12	81% (70–89)	83% (65–92)	4.7 (2.1–10.6)	82
ADOS module 2 (AS)	72	32	26	14	0	100% (89–100)	65% (50–78)	2.9 (1.9–4.4)	80
ADOS module 2 (autism)	72	30	33	7	2	94% (80–98)	82% (68–91)	5.4 (2.7–10.6)	88
ADOS all modules (AS)	260	163	60	32	5	97% (93–99)	65% (55–74)	2.8 (2.1–3.7)	87
ADOS all modules (autism)	260	142	75	17	26	84% (78–89)	82% (72–88)	4.6 (3.0–7.0)	83

ADOS: Autism Diagnostic Observation Schedule; AS: autism spectrum cutoff; AUT: autism cutoff; CI: confidence interval; ASD: autism spectrum disorder; NS: nonspectrum; TP: true positives (children with ASD classified as ASD); TN: true negatives (children without ASD classified as NS); FP: false positives (children without ASD misclassified as ASD); FN: false negatives (children with ASD misclassified as NS).

The AUCs of the different modules were .91 for module 1 no words, .92 module 1 some words, and .95 for module 2 (all p < .001). ROC curve analyses showed that the published cutoffs of the revised algorithms suited the present sample. Agreement between the different modules and clinical diagnosis were for module 1 no words κ = .60 (autism spectrum cutoff) and κ = .57 (autism cutoff), and for module 1 some words κ = .72 (autism spectrum) and κ = .60 (autism). For module 2, agreement was κ = .62 (autism spectrum) and κ = .75 (autism) (all p < .001). No differences emerged regarding age and Vineland-II scores between TP and FN or between FP and TN on the ADOS (Table 4).

ADI-R and ADOS combined

In all developmental groups, the different combinations of ADI-R and the ADOS yielded higher specificities (88%−100%) than sensitivities (34%−64%) with correct classifications ranging from 57% to 73%. The LR+s ranged between 5.1 and 14.4 (for some analyses, LR+ could not be computed due to a specificity of 100%). Using the lower adjusted cutoff for the ADI-R and the autism spectrum cutoff for the ADOS gave better balanced sensitivities (77%−80%) and specificities (87%−90%), an increased amount of correct classification (81%−82%), and LR+s of 5.8–8.0 (see Table 3 for details).

ADI-R/ADOS agreement

Agreement between different combinations of the ADI-R and the ADOS comparing all participants yielded a κ = .23 (p < .001) for the ADI-R clinical and research cutoffs and ADOS autism spectrum cutoff. Using the ADOS autism cutoff and the ADI-R clinical and research cutoff gave a κ = .29 and κ = .30, respectively (both p < .001). The agreement between the adjusted ADI-R cutoff combined and the ADOS autism spectrum and autism cutoffs were κ = .31 and .34 (both p < .001) for all participants. Looking at each developmental group of the ADI-R combined with the ADOS autism spectrum and autism cutoffs, findings were more variable and ranged from κ = .15 to κ = .36 (all p ≤ .046). The correlations between the ADI-R and the ADOS total scores were r = .53 (NV, p < .001), r = .31 (SW, p = .004), and r = .42 (PH, p < .001). The correlations between the SA domain of both instruments showed the same pattern: NV r = .50 (p < .001), SW r = .28 (p = .009), and PH r = .45 (p < .001), while the RRB domains correlated weakly for the SW and PH groups: r = .19 (p = .076 and p = .062) but stronger for the NV group (r = .40, p = .001).

Discussion

Few studies have examined the psychometric properties of combined ADI-R and ADOS use, when applying their new diagnostic algorithms, in young children in purely clinical settings. Our findings indicate added value to the accuracy of ASD diagnoses through the combined use of the ADI-R and the ADOS in toddlers and young preschoolers with unclear developmental concerns. Diagnostic validity of combined ADI-R and ADOS use was superior to single instrument information. ADOS usage alone achieved an almost equally high diagnostic validity for the ADOS autism cutoff. Both instruments separately showed, in some case after cutoffs adjustment, diagnostic validity comparable to the results from US research settings.

We found the ADI-R alone to have a pattern of substantially lower sensitivities than specificities, especially when using the research cutoff, where sensitivities rarely reached 50%, while specificities exceeded 90%. In two previous studies that examined the ADI-R new research algorithms, higher and better balanced sensitivities and specificities were found (Kim et al., 2013; Kim and Lord, 2012b). Compared to the present study, their samples were not purely clinical. The lack of balance between sensitivity and specificity using the recommended cutoffs in our sample was to a certain degree remediated by the adjusted cutoffs, which also basically set aside the differences with the US samples. Lowering the cutoffs increased sensitivities to ~80%, decreased specificities to ~60%, and a larger part of the children were accurately classified. Nevertheless, LRs+ decreased, were lower than in previous studies, and none of the AUCs exceeded a fair level. The sample characteristics in this study differed to a certain extent from previous US research (Kim et al., 2013; Kim and Lord, 2012b). While age, NVIQ, and how the participant characteristics were associated with domain totals resembled the US samples rather closely; all domain totals were lower and especially those of the RRB domain. Finally, there were some differences in participant characteristics of large effect size (d ≥ .83) between correctly classified and misclassified children in this study. Consistent with Kim and Lord (2012a), the children with ASD misclassified as NS had higher Vineland-II composite scores, that is, were more adaptively able than the children with ASD correctly classified. The latter indicates a negative relationship between adaptive skills and autistic symptomatology as measured by the ADI-R. Contrary to Kim and Lord (2012a), children with NS misclassified as ASD were younger than the correctly classified children with NS. This suggests that even the new research algorithms of the ADI-R might be sensible to age effects, leading to some overinclusivity in young children. Thus, in sum, in our clinical sample of toddlers and young children with serious developmental concerns, the diagnostic validity of ADI-R for ASD showed limited accuracy and did not classify children as efficiently as in the US samples.

With regard to the evaluation of the ADOS in the current study, our sample and analyses differed from previous research. Particularly, no earlier study has presented data separately for each ADOS module using the new algorithms in a sample restricted to children aged 47 months and younger classified in an ASD versus a NS group. In most other studies, older children often up to the age 12 years were included, the diagnostic classification were conducted for autism versus NS groups and ASD (except autism) versus NS groups, respectively, or when the same age range were studied, data were presented for the different modules lumped together (i.e. Kim and Lord, 2012a). Besides, our sample was a purely clinical one, whereas most other studies reported data from research samples. Looking at NVIQ, it also seems that the participants in this study were more high functioning than those in the two Gotham et al. (2007, 2008) studies, the Gray et al. (2008), and the De Bildt et al. (2009) studies but lower functioning than in Oosterling et al. (2010), but all these studies reported data for children with autism and ASD apart. The association between participant characteristics and domain totals resembled that of Gotham et al. (2007), but the correlations were higher than in De Bildt et al. (2009). Therefore, the comparison with other studies where children were older and results were presented for children with autism and other ASDs separately is somewhat compromised. This notwithstanding, and taking the variation of the reported domain totals into account, the conclusion is that our ASD sample did not deviate from the reviewed studies in any substantial way, and especially not when considering the diagnostic validity of the ADOS.

For the diagnostic validity of the ADOS autism spectrum cutoff in our sample, we found excellent sensitivities alongside lower specificities being consistent with prior studies by Kim and Lord (2012a), Molloy et al. (2011) (except for module 2), and Risi et al. (2006). On the contrary, Oosterling et al. (2010) found lower sensitivities and higher specificities for module 1 (some words) and module 2. The autism cutoff showed more balanced sensitivities and specificities (≥81%), being considerably lower than those found by Gotham et al. (2007, 2008). However, these values were obtained when the autism cutoff was used in children with core autism (not ASDs) versus NS. When considering the 95% confidence intervals (when such are reported), many differences were attenuated or disappeared between this and previous studies. AUCs in our sample were excellent; kappas for autism spectrum and autism cutoffs versus clinical diagnosis were in the moderate to substantial range, while LR+s were modest.

In our cohort, the ADOS (κ = .57–.75) was superior to the ADI-R (κ = .27–.45) in terms of diagnostic accuracy, but the best diagnostic validity was generated by the combination of the ADI-R, and the ADOS, using the adjusted ADI-R cutoff and the ADOS autism spectrum cutoff. The latter generated sensitivities and specificities that were basically in the same range as in Kim and Lord (2012a) and Risi et al. (2006: using the standard algorithms with children below 36 months of age), except for a somewhat lower sensitivity of the SW group in our study. The agreement of the ADI-R and the ADOS was limited in terms of correlation between domain totals. However, aside from the NV group (Z = 2.90, p = .004), they did not differ significantly from those found by Kim and Lord (2012a) (Z = 1.39, p = .16 for SW and Z = 1.61, p = .11 for PH). Older studies using the standard algorithms of both instruments, Risi et al. (2006) and Le Couteur et al. (2008) reported substantial correlations between domain totals across instruments. Ventola et al. (2006), on the contrary, found a weak agreement measured by kappa statistics between the ADI-R and the ADOS classifications, probably due to a reduced capacity of the ADI-R alone to classify children correctly, like in our study. However, such findings should not be misinterpreted in terms of an underestimation of the added contribution of both instruments to the diagnostic decision making, as have been carefully pointed out by others (Kim and Lord, 2012a; Risi et al., 2006).

The considerably lower levels of the ADI-R totals in the current study compared to the previous three samples in Kim and colleagues (Kim et al., 2013; Kim and Lord, 2012b) need to be addressed. They were not associated with any differences in child characteristics, and scores on the ADOS in our sample did not reveal lower levels compared to the US samples. Thus, we speculate that the differences are linked to the parent/caregiver characteristics, to the fact that we examined a purely clinical sample, as well as cultural factors. According to our experience, parents of toddlers and young preschoolers who are referred to clinical services for the first time due to a suspicion of a neurodevelopmental disorder are initially often either reluctant to describe their child’s behavior in terms of abnormality or generally unaware that their child’s behavior might be developmentally altered. This is endorsed by governmental educational policies in Sweden being predominantly socially inclusive, and exclusion from regular schools is in the majority viewed as a societal failure (almost all children in our study attended regular preschools). Moreover, societal awareness or sensitivity to ASD symptoms and traits might still be lower in many countries compared to the United States. For instance, other cross-cultural studies on parent-report-based tools of autistic behaviors have found lower scores in the general population and ASD cases (see, for example, Bölte, 2012; Bölte et al., 2008). All of these latter aspects might account for the lower ADI-R scores in this study. Indeed, a multicountry European replication study of the new research algorithms of the ADI-R also found lower levels of domain totals than reported in the US studies (De Bildt et al., in press). Finally, a purely statistical note of caution concerning the scoring differences between the current and prior ADI-R studies. Except for the RPI domain of the PH developmental cell, there was an overlap in confidence interval with at least one of the samples for the SA/SC domain in all developmental cells but not for the RRB domain. Hence, at least some of the differences discussed above might be in the area of chance rather than truly significant.

This study has several limitations. First, currently, the ADI-R and the ADOS are considered the most efficient tools (“gold standard”) for standardized collection of information to assist clinical consensus diagnosis in clinical settings. Thus, in this study, using data from a specialized clinical unit, and comparable to most other diagnostic validation studies (Gotham et al., 2007, 2009; Kim et al., 2013; Kim and Lord, 2012a, 2012b; Le Couteur et al., 2008; Risi et al., 2006), the clinical consensus diagnosis was not fully independent of the results of the ADI-R and the ADOS. Nevertheless, the diagnostic procedure leading to the consensus diagnosis comprised comprehensive information from a variety of sources, enriching, and balancing results of the ADI-R and the ADOS, especially naturalistic observation of the child in its preschool. Second, in this study, 30 children of the NS group had no DSM-IV-TR diagnosis. It is well known that including typically developing children is likely to overestimate diagnostic validity (see, for example, Kim and Lord, 2012b: where specificities increased when introducing typically developing children in the comparison). However, the children without diagnosis in this sample were not typically developing in a narrow sense. They were clinically referred and assessed children owing to developmental concerns. Most of them exhibited different kinds of developmental problems even though these were not sufficiently severe to justify a diagnosis. As a matter of fact, the specificities did actually decrease slightly when excluding the NS children without diagnosis from the analysis.

Implications for clinical practice and research

Our results indicate that a combination of the ADI-R and the ADOS should be viewed the first choice to assist the diagnostic assessment of ASD in toddlers and young preschoolers in clinical practice. Earlier research shows that the same is equally true in research contexts (Kim and Lord, 2012a; Risi et al., 2006). Aside from the scientific advantages of combined ADI-R/ADOS use demonstrated by our and other studies, it can be argued that applying these two ways of collecting information have additional pedagogical effects for the parents who might serve the child’s future well-being. That is, being exposed to a comprehensive set of concrete questions about the child’s development and behavior plus watching it interacting with an expert in standardized setting, followed by feedback on the child’s strengths and weaknesses based on these assessments is often experienced as a key and first step to better understand the expression of ASD in their child. If for whatever reason a combined ADI-R/ADOS use is not possible, using the ADOS only (autism cutoff) can be viewed as the second choice to assist the diagnostic decision making, with seemingly equivalent psychometric properties, at least in clinical settings.

Kim and Lord (2012b) presented different cutoffs for the novel research algorithms of the ADI-R with different psychometric properties and discussed a rational for the usage of them. We completely agree with their reasoning: in rare clinical and virtually all research settings where FP need to be avoided, the higher research cutoff is appropriate, as it maximizes specificity. In most clinical and rare research settings where access to support and services is prioritized, the lower clinical cutoff or even our adjusted, even lower, cutoff could be applied. Theoretically, the same applies for the ADOS in young children, where the higher autism cutoff might be utilized as a counterpart to the research cutoff of the ADI-R, and the autism spectrum cutoff as the counterpart to the ADI-R’s clinical cutoff.

Footnotes

Acknowledgements

We sincerely thank Per-Olof Björck, head of the southeastern part of the Division of Child and Adolescent Psychiatry in Stockholm, for facilitating the realization of this study.

Funding

Sven Bölte was supported by the Swedish Research Council. This study was supported by the Swedish Research Council and the Swedish Research Council in partnership with FAS, FORMAS and VINNOVA (cross-disciplinary research program concerning children’s and youth’s mental health), Riksbankens Jubileumsfond and Jerringfonden.

References

American Psychiatric Association (APA) (2013) Diagnostic and Statistical Manual of Mental Disorders (DSM-5). 5th ed. Arlington, VA: APA.

Baghdadli

Picot

Pascal

(2003) Relationship between age of recognition of first disturbances and severity in young children with autism. European Child & Adolescent Psychiatry 12: 122–127.

Bölte

(2012) Brief report: the Social Responsiveness Scale for Adults (SRS-A): initial results in a German cohort. Journal of Autism and Developmental Disorders 42: 1998–1999.

Bölte

Marschik

Falck-Ytter

(2013) Infants at risk for autism: a European perspective on current status, challenges and opportunities. European Child & Adolescent Psychiatry 22: 341–348.

Bölte

Poustka

Constantino

(2008) Assessing autistic traits: cross-cultural validation of the social responsiveness scale (SRS). Autism Research 1: 354–363.

Charman

Taylor

Drew

(2005) Outcome at 7 years of children diagnosed with autism at age 2: predictive validity of assessments conducted at 2 and 3 years of age and pattern of symptom change over time. Journal of Child Psychology and Psychiatry 46: 500–513.

Chawarska

Paul

Klin

(2007) Parental recognition of developmental problems in toddlers with autism spectrum disorders. Journal of Autism and Developmental Disorders 37: 62–72.

Cohen

(1983) Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences. Hillsdale, NJ: Erlbaum.

Corsello

Akshoomoff

Stahmer

(2013) Diagnosis of autism spectrum disorders in 2-year-olds: a study of community practice. Journal of Child Psychology and Psychiatry 54: 178–185.

10.

Dawson

(2008) Early behavioral intervention, brain plasticity, and the prevention of autism spectrum disorder. Development and Psychopathology 20: 775–803.

11.

De Bildt

Charman

Zander

(in press) Autism Diagnostic Interview-Revised (ADI-R) algorithms for toddlers and young preschoolers: replication in a non-US sample of 910 children.

12.

De Bildt

Sytema

van Lang

(2009) Evaluation of the ADOS revised algorithm: the applicability in 558 Dutch children and adolescents. Journal of Autism and Developmental Disorders 39: 1350–1358.

13.

De Giacomo

Fombonne

(1998) Parental recognition of developmental abnormalities in autism. European Child & Adolescent Psychiatry 7: 131–136.

14.

Elsabbagh

Mercure

Hudry

(2012) Infant neural sensitivity to dynamic eye gaze is associated with later emerging autism. Current Biology 22: 338–342.

15.

Fein

Barton

Eigsti

(2013) Optimal outcome in individuals with a history of autism. Journal of Child Psychology and Psychiatry 54: 195–205.

16.

Gonzalez

Alpert

Shay

(1993) Autistic children on followup: change of diagnosis. Psychopharmacology Bulletin 29: 353–358.

17.

Gotham

Pickles

Lord

(2009) Standardizing ADOS scores for a measure of severity in autism spectrum disorders. Journal of Autism and Developmental Disorders 39: 693–705.

18.

Gotham

Risi

Dawson

(2008) A replication of the Autism Diagnostic Observation Schedule (ADOS) revised algorithms. Journal of the American Academy of Child and Adolescent Psychiatry 47: 642–651.

19.

Gotham

Risi

Pickles

(2007) The Autism Diagnostic Observation Schedule: revised algorithms for improved diagnostic validity. Journal of Autism and Developmental Disorders 37: 613–627.

20.

Gray

Tonge

Sweeney

(2008) Using the Autism Diagnostic Interview-Revised and the Autism Diagnostic Observation Schedule with young children with developmental delay: evaluating diagnostic validity. Journal of Autism and Developmental Disorders 38: 657–667.

21.

Hess

Landa

(2012) Predictive and concurrent validity of parent concern about young children at risk for autism. Journal of Autism and Developmental Disorders 42: 575–584.

22.

Kim

Lord

(2012a) Combining information from multiple sources for the diagnosis of autism spectrum disorders for toddlers and young preschoolers from 12 to 47 months of age. Journal of Child Psychology and Psychiatry 53: 143–151.

23.

Kim

Lord

(2012b) New Autism Diagnostic Interview-Revised algorithms for toddlers and young preschoolers from 12 to 47 months of age. Journal of Autism and Developmental Disorders 42: 82–93.

24.

Kim

Thurm

Shumway

(2013) Multisite study of new Autism Diagnostic Interview-Revised (ADI-R) algorithms for toddlers and young preschoolers. Journal of Autism and Developmental Disorders 43: 1527–1538.

25.

Le Couteur

Haden

Hammal

(2008) Diagnosing autism spectrum disorders in pre-school children using two standardised assessment instruments: the ADI-R and the ADOS. Journal of Autism and Developmental Disorders 38: 362–372.

26.

Lord

Risi

DiLavore

(2006) Autism from 2 to 9 years of age. Archives of General Psychiatry 63: 694–701.

27.

Lord

Rutter

DiLavore

(1999) Autism Diagnostic Observation Schedule (ADOS). Los Angeles, CA: Western Psychological Publishing.

28.

Lord

Rutter

DiLavore

(2012) Autism Diagnostic Observation Schedule (ADOS-2). 2nd ed. Manual (Part I). Torrance, CA: Western Psychological Services.

29.

Lord

Shulman

DiLavore

(2004) Regression and word loss in autistic spectrum disorders. Journal of Child Psychology and Psychiatry 45: 936–955.

30.

Lord

Storoschuk

Rutter

(1993) Using the ADI–R to diagnose autism in preschool children. Infant Mental Health Journal 14: 234–252.

31.

Luyster

Gotham

Guthrie

(2009) The Autism Diagnostic Observation Schedule-toddler module: a new module of a standardized diagnostic measure for autism spectrum disorders. Journal of Autism and Developmental Disorders 39: 1305–1320.

32.

Molloy

Murray

Akers

(2011) Use of the Autism Diagnostic Observation Schedule (ADOS) in a clinical setting. Autism 15: 143–162.

33.

Mullen

(1995) Mullen Scales of Early Learning. AGS ed. Circle Pines, MN: American Guidance Service.

34.

Newcombe

(1998) Interval estimation for the difference between independent proportions: comparison of eleven methods. Statistics in Medicine 17: 873–890.

35.

Oosterling

Roos

de Bildt

(2010) Improved diagnostic validity of the ADOS revised algorithms: a replication study in an independent sample. Journal of Autism and Developmental Disorders 40: 689–703.

36.

Piven

Harper

Palmer

(1996) Course of behavioral change in autism: a retrospective study of high-IQ adolescents and adults. Journal of the American Academy of Child and Adolescent Psychiatry 35: 523–529.

37.

Reichow

(2012) Overview of meta-analyses on early intensive behavioral intervention for young children with autism spectrum disorders. Journal of Autism and Developmental Disorders 42: 512–520.

38.

Richler

Bishop

Kleinke

(2007) Restricted and repetitive behaviors in young children with autism spectrum disorders. Journal of Autism and Developmental Disorders 37: 73–85.

39.

Risi

Lord

Gotham

(2006) Combining information from multiple sources in the diagnosis of autism spectrum disorders. Journal of the American Academy of Child and Adolescent Psychiatry 45: 1094–1103.

40.

Rogers

Estes

Lord

(2012) Effects of a brief Early Start Denver model (ESDM)-based parent intervention on toddlers at risk for autism spectrum disorders: a randomized controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry 51: 1052–1065.

41.

Roid

Sampers

(2005) Merrill-Palmer – Revised Scales of Development (Swedish version). Wood Dale, IL: Stoelting.

42.

Rutter

Le Couteur

Lord

(2003) Autism Diagnostic Interview Revised (ADI-R). Los Angeles, CA: Western Psychological Services.

43.

Sigman

Ruskin

Arbeile

(1999) Continuity and change in the social competence of children with autism, Down syndrome, and developmental delays. Monographs of the Society for Research in Child Development 64: 1–114.

44.

Sparrow

Cichetti

Balla

(2005) Vineland-II: Vineland Adaptive Behavior Scales. Circle Pines, MN: AGS Publishing.

45.

Turner

Stone

(2007) Variability in outcome for children with an ASD diagnosis at age 2. Journal of Child Psychology and Psychiatry 48: 793–802.

46.

Venter

Lord

Schopler

(1992) A follow-up study of high-functioning autistic children. Journal of Child Psychology and Psychiatry 33: 489–507.

47.

Ventola

Kleinman

Pandey

(2006) Agreement among four diagnostic instruments for autism spectrum disorders in toddlers. Journal of Autism and Developmental Disorders 36: 839–847.

48.

Wechsler

(2004) The Wechsler Intelligence Scale for Children (Swedish version 2005). 3rd ed. Stockholm: Pearson Assessment.

49.

Wiggins

Robins

(2008) Brief report: excluding the ADI-R behavioral domain improves diagnostic agreement in toddlers. Journal of Autism and Developmental Disorders 38: 972–976.

50.

Yirmiya

(2010) Editorial: early prevention and intervention—the five W (and one H) questions. Journal of Child Psychology and Psychiatry 51: 1297–1299.

51.

Zwaigenbaum

Bryson

Rogers

(2005) Behavioral manifestations of autism in the first year of life. International Journal of Developmental Neuroscience 23: 143–152.