The Search for an Early Intervention Outcome Measurement Tool in Autism

Abstract

Evidence is accumulating that early intervention can be effective in improving the skills of young children with autism spectrum disorder. However, the science is hampered by the lack of agreed “gold standard” tools for the measurement of progress and outcome. What is required is a reliable, valid, and sensitive measure of change in the core domains of autism, which can be undertaken blind to group and time. This article explores the use of a promising measure of change, for which reliability, validity, and sensitivity to change over a lengthy period have been previously demonstrated. Pilot data indicate that, despite some sensitivity to change over a short period of time, it does not capture treatment effects more effectively than an existing diagnostic tool. Future directions for the ongoing search are suggested, including consideration of how to achieve sensitivity to differential change as well as to change over time.

Keywords

intervention outcome measurement joint attention behavioral assessment autism spectrum disorder

Introduction

Autism spectrum disorders (ASD) are neurodevelopmental, lifelong conditions diagnosed using a set of behavioral criteria (American Psychiatric Association, 2013; World Health Organization, 1994), and characterized by persistent deficits in social communication and social interaction, along with restricted, repetitive patterns of behavior, interests, or activities. Children with ASD are typically diagnosed between 2 and 6 years of age (Boyd, Odom, Humphreys, & Sam, 2010; Dababnah, Parish, Tuner Brown, & Hooper, 2011). Around 1% of children are recognized at school age to have ASD (Baird et al., 2006), though recent estimates from the United States have risen to near 1.5% (Centre for Disease Control, 2014). The spectrum encompasses individuals with a wide range of intellectual and developmental abilities. In the past decade, there has been an increase in ASD intervention research, with concurrent improvement in the methodological quality of these studies (Charman, 2011; Magiati, Tay, & Howlin, 2012; National Research Council, 2001; Rogers & Vismara, 2008). For young children, the focus of specific intervention approaches is usually on the core social-communication deficits, often using strategies to enhance the quality of reciprocal interaction with parents and caregivers, enabling them to support children’s development of effective communication (Kasari et al., 2005; Oono, Honey, & McConachie, 2013).

It is well recognized that progress in research on effectiveness of early intervention is hampered by the use of highly variable outcome measures (Bolte & Diehl, 2013; Cunningham, 2012; Matson, 2007; Spence & Thurm, 2010; Wolery & Garfinkle, 2002). There are a number of reasons for this state of affairs. First, studies vary in the primary goal of intervention—overall function, reducing particular ASD difficulties, improving child functional skills, or enhancing quality of life for the child and/or family. Each of these implies different intervention strategies and outcome measures. Second, some studies opt to measure proximal outcomes (i.e., close to the target of a particular intervention approach), which generally show better intervention effects than distal (i.e., more functional and generalized) outcomes. However, an emphasis on proximal outcomes can be misleading in relation to the impact of interventions in day-to-day living. The third area for consideration for intervention researchers has to do with external validity. The dilemma here is that subjective (particularly family-reported) measures are those with the greatest external validity, as it is the experience of children and families that interventions particularly want to improve; however, such ratings are prone to expectation and placebo effects within interventions as parents cannot (or almost never) be blinded. A final challenge is to determine outcome measures that are responsive to change but also stable over the different settings that children experience. Sensitivity in measures is often limited in studies involving heterogeneous samples of children by floor and ceiling effects (floor effects when children are not ready to change, and ceiling effects when they have already mastered the skill). For all these reasons, the search for an appropriate early intervention outcome measure in autism is complex.

Even if the focus of an intervention is on early social communication, an outcome measure needs to combine a series of characteristics which are hard to reconcile. It needs to be directly observed or assessed, to minimize subjectivity, unlike the existing tools which measure progress using parent report (e.g., Cohen, Schmidt-Lackner, Romanczyk, & Sudhalter, 2003; Stone, Coonrod, Pozdol, & Turner, 2003). Given the limited length of some intervention trials (e.g., Kasari, Paparella, Freeman, & Jahromi, 2008; McConachie, Randle, Hammal, & Le Couteur, 2005; Rogers et al., 2012), it should be sensitive enough to pick up small changes in behavior but also robust enough to be readministered in a fairly short period of time.

There are some apparently suitable assessment measures for key skills such as language and adaptive behavior; however, difficulties in these areas are not specific to individuals with autism, and the tools’ validity in ASD may not have been formally explored (Cunningham, 2012; Hudry et al., 2010). To measure the core aspects of autism, one solution might be to use an autism-specific diagnostic tool such as the recently updated Autism Diagnostic Observation Schedule–2 (ADOS-2; Lord et al., 2012). However, expecting a change in the overall diagnostic status may not be reasonable, as has been shown in recent reports of randomized controlled trials (RCTs) of early intervention (Dawson et al., 2010; Green et al., 2010), and researchers have explored alteration of scoring rules in an attempt to increase sensitivity (e.g., Aldred, Green, & Adams, 2008; Green et al., 2010). Furthermore, diagnostic tools such as the ADOS are not designed to show a change over a few months, even with the newer calibrated scores of severity (Gotham, Pickles, & Lord, 2009). Therefore, to be useful, it is likely that the measured behaviors will target change in specific subcomponents of the social-communication and repetitive behaviors assessed in diagnosis.

There are available directly assessed specific measures of early communicative and social behaviors such as the Communication and Symbolic Behavior Scales–Developmental Profile (CSBS-DP; Wetherby & Prizant, 2002) and the Early Social Communication Scales (ESCS; Seibert, Hogan, & Mundy, 1982). However, they have the disadvantage of having been developed and standardized on typically developing (TD) young children (CSBS-DP to 24 months, ESCS to 30 months), and are thus likely to create a ceiling effect in scores for many preschool children with ASD.

The same objection of a narrow age-band of applicability applies to many coding schemes for child social-communication skills in the context of directly observed parent–child interaction (e.g., Adamson, Bakeman, Deckner, & Romski, 2009; Feldman, Greenbaum, Mayes, & Erlich, 1997; Kasari, Freeman, & Paparella, 2006; Wan et al., 2012). Other studies bypass this issue by creating novel, bespoke codes for evaluation of direct observation (McConachie et al., in press). However, there are a number of disadvantages. Bespoke coding schemes tend to measure skills very close to the intervention target, and thus fail to evaluate generalization of treatment effect. Without adequate investigation of the measurement properties of a novel coding scheme, a finding of lack of significant change could be attributable either to a lack of intervention effect or to insensitivity of the measure. Finally, the use of novel schemes prevents comparison of results across studies.

This article presents pilot data on the use of a directly observed measure of social interaction skills in young children with ASD, which attempts to address many of the limitations of other measures cited above. The tool used is the Social Orienting Continuum and Response Scale (SOC-RS; Mosconi, Reznick, Mesibov, & Piven, 2009). This tool is an objective, directly observed measure of behavioral change. It addresses the core features of autism in the social-communication domain and has been established for use with an ASD sample from the outset. Its sensitivity to change over a short period of time and to differential change between groups is as yet untested and will be the focus of this investigation.

SOC-RS

The SOC-RS focuses on social orienting behaviors, which are key early emerging impairments in autism. The SOC-RS was developed with a longitudinal sample of children with ASD, aged 2 years old at Time 1 and seen again 2 years later at Time 2 (Mosconi et al., 2009). These were matched with two cross-sectional cohorts of chronological-age and gender-matched TD children, one cohort for each time point. The measure uses the standard assessment procedure of the ADOS (Lord, Rutter, DiLavore, & Risi, 1999) as a source of behavioral data. Videos of a full Module 1 or Module 2 ADOS are observed and scored by a trained observer according to the SOC-RS handbook (Mosconi, Fletcher-Watson, McConachie, Reznick, & Piven, 2010). Five behaviors are scored for frequency: These are social referencing, response to joint attention, initiation of joint attention, orienting to name, and social smiling. The SOC-RS scoring system takes into account the number of opportunities to produce a behavior, not just the behavior frequency alone. For example, a child can only “orient to name” when someone calls his or her name.

The principal output of the SOC-RS is the Social Orienting Composite, calculated by combining z scores for each observed behavior. The SOC-RS can also produce a combined joint attention score, which incorporates both responses to and initiation of joint attention.

Good inter-rater reliability for each behavior (all intra-class correlations [ICCs] > .78) has been reported, as well as convergent validity between the Social Orienting Composite and the socialization subscale of the Vineland Adaptive Behavior Scale (VABS; Sparrow, Cicchetti, & Balla, 2005; evidence from Mosconi et al., 2009). Mosconi and colleagues also found that the measure discriminated between the ASD and TD samples, both at each time point and in terms of pattern of change over time. These promising data indicate that the SOC-RS may be a good candidate as a sensitive measure of change in key behaviors for preschool children with autism.

In this new study, we, therefore, aimed to determine whether the SOC-RS was sensitive to differential change under two specific conditions. First, does the SOC-RS pick up change over a shorter period of time (7 months rather than 2 years) than in the original SOC-RS development study (Mosconi et al., 2009)? Second, does the SOC-RS measure a substantial treatment effect when comparing groups who have and have not shown behavioral change in response to an intervention?

Method

Design

Our method in this study was to identify a known between-group treatment effect to ascertain whether a new tool, the SOC-RS, would be sensitive enough to detect that effect. We, therefore, compare children predefined as exhibiting progress, or not, in social and communicative development over the period of interest. The groups are small to maximize the difference between the two samples (“extreme groups”). This allows us to model whether the SOC-RS is capable of measuring a differential change between groups, which we can be confident already exists.

Participants

Participants were 20 preschool-aged children (M_age = 36.1 months) with a diagnosis of ASD, confirmed by the Autism Diagnostic Interview–Revised (ADI-R; Lord, Rutter, & Le Couteur, 1994) and the ADOS (Lord et al., 1999). The children were divided into two groups (n = 10 per group; see Table 1): labeled progress and no progress, based on whether they had exhibited improvement in social-communication behaviors over a period of 7 months. The selection process is detailed below. Groups were matched at Time 1 on VABS composite score, Macarthur Communicative Development Inventory (MCDI) score, and ADOS social and communication score. There were no group differences in the average time that the children were visible on camera during each of their ADOS assessments.

Table 1.

Characteristics of Selected Groups of Children.

Characteristic	No progress (n = 10)	Progress (n = 10)
Gender ratio (m:f)	9:1	8:2
Diagnosis at Time 1 (autism:asd)	7:3	7:3
ADOS module, Time 2^a (1:2)	9:1	9:1
	M (SD)	M (SD)	p value	95% CI
Age at Time 1 (months)	33.4 (7.28)	38.8 (6.34)	.09	[−11.1, 1.01]
VABC Time 1	62.3 (6.82)	64.9 (8.36)	.46	[−9.8, 4.6]
MCDI Time 1 (words produced)	53.0 (118.6)	89.5 (85.6)	.44	[−133.7, 60.7]
ADOS CSI Time 1	15.5 (4.67)	13.7 (4.72)	.40	[−2.61, 6.21]
	M (SD) [range]	M (SD) [range]
MCDI change (words)	+6.4 (10.2) [−5, +26]	+63.9 (79.2) [0, +265]
ADOS CSI change (range)	+1.6 (3.06) [+8, −1]	−7.8 (4.83) [−3, −15]

Note. Reduction = less impaired. ADOS = Autism Diagnostic Observation Schedule (Lord, Rutter, DiLavore, & Risi, 1999); CI = confidence interval; VABC = Vineland Adaptive Behavior Scales, Composite (Sparrow, Cicchetti, & Balla, 2005); MCDI = Macarthur Communicative Development Inventory (Fenson et al., 1993); CSI = Communication and Social Interaction algorithm score.

Ratio of children receiving a Module 1 versus a Module 2 ADOS at Time 2. All children received a Module 1 at Time 1.

Participant Selection

The children were selected from an original sample of 51 children involved in a controlled intervention study (see Note 1; McConachie et al., 2005). In that study, the parents of children with ASD, aged 24 to 48 months, had either immediate or delayed access to the “More Than Words” course (Sussman, 1999), group parent training, which aims to improve parents’ skills in interacting and communicating with their child. The intervention was found to have a positive effect on the immediate-access group relative to the (delayed-access) controls, increasing children’s vocabulary and parents’ use of facilitative interaction strategies. Following the 7-month follow-up assessment, control group parents attended the course, and assessment was conducted after a further 7 months.

We selected children according to whether they exhibited substantial progress or not after their parent(s) attended a “More Than Words” course (see Table 1). Progress was measured by reduction in the child’s score on the ADOS social and communication algorithm (no progress range = +8 to −1; progress range = −15 to −3) and secondarily using vocabulary increase as measured by the MCDI (Fenson et al., 1993; no progress range = −5 to +26 words; progress range = 0 to +265 words).

Procedure

Among other measures, at each assessment visit the child took part in an ADOS (Lord et al., 1999) administered by a trained researcher using standardized assessment procedures and toys. The films (about 30 min per participant, per time point) of these ADOS assessments were then used as the raw data for SOC-RS ratings collected in this study.

Scoring the SOC-RS

The SOC-RS codes behaviors in five categories. The rater codes the number of instances of the following behaviors for all times when the child’s head is visible on screen:

Referencing: episodes in which the child looks at another person’s face.

Social Smiling: events in which the child smiles at an adult with the clear intention of sharing emotion with them.

Orienting to Name: episodes in which the child’s name is called. The ratings distinguish between episodes in which the child responds or does not respond.

Joint Attention Initiation: episodes in which the child directs another person’s attention to an object, for the purposes of sharing attention in that object.

Joint Attention Responding: episodes in which the child is given a joint attention cue by another person. As above, ratings distinguish between episodes in which the child responds or does not respond.

Ratings

Two raters were trained on the SOC-RS using guidelines and advice provided by the measure’s originator, Dr. M. Mosconi. Both raters were blind to time and to group status. Based on a sample of 10 tapes (25% of the full sample), the average of inter-rater ICCs across all five measures of the SOC-RS was .855, with a range from .707 (Orienting to Name) to .966 (Joint Attention Initiation).

Analysis

The SOC-RS produces continuous variables, based on simple frequency counts, for each of the five items listed above. In addition, a Social Orienting Composite can be constructed by converting each continuous item score into a z score, and then summing these. Two items can also be used to provide categorical scores. For Orienting to Name, categorization is based on the press number at which the children responds, using only the specific ADOS presses for scoring (and ignoring other instances in which the child’s name is used). A dichotomous responder/non-responder code can be calculated for Joint Attention Responding based on whether or not the child responds to one of the early ADOS presses, using eye-gaze and head turn only. Children are scored as “responders” if they respond to one of the early presses, or “non-responders” if they do not respond until later, when pointing cues are added.

Each of these variables was calculated and explored in our analysis. We used ANOVAs on each item score and on the Social Orienting Composite to look for the main effects of time and interactions between group and time. The former result tests whether the SOC-RS is sensitive to change over a short (7 month) period, and the latter reveals whether the SOC-RS is sensitive to differential intervention effects. In addition, chi-square tests are employed to identify differential distributions of participants at each time point in Orienting to Name and Joint Attention Responding categories. Finally, we used correlations to check for relationships between the SOC-RS scores and other established measures, as a test of validity.

Results

Effects of Group and Time

ANOVAs on individual SOC-RS items revealed the main effects of Time in Referencing rate only (see Table 2) but not for other items. In addition, we found moderate effect sizes for the joint attention responding and joint attention total items, though the difference did not achieve significance. These effects of time compare favorably with the already-established change in MCDI scores, but have smaller effect sizes than for the ADOS social-communication algorithm score. There were no significant interactions between Time and Group for the individual items.

Table 2.

Effects of Time on SOC-RS Scores, With MCDI and ADOS for Comparison.

Item	Time 1	Time 2	p	Effect size (d)	95% CI
Item	M (SD)	M (SD)	p	Effect size (d)	95% CI
SOC	0.000 (0.525)	0.311 (0.552)	.022	0.577	[−0.587, −0.035]
Referencing rate	0.607 (0.300)	0.824 (0.342)	.010	0.676	[−0.376, −0.058]
Social smiling rate	0.065 (0.069)	0.065 (0.071)	1.00	0.000	[−0.042, 0.042]
Name trial^a	4.90 (3.13)	5.00 (3.32)	.898	0.031	[−1.71, 1.51]
JA responding	0.015 (0.022)	0.024 (0.021)	.095	0.419	[−0.021, 0.001]
JA total	0.000 (0.739)	0.488 (1.32)	.054	0.474	[−0.987, 0.012]
MCDI expressive score	71.3 (102.4)	106.5 (122.4)	.013	0.315	[−61.6, −5.44]
ADOS CSI^a	14.6 (4.66)	11.6 (7.14)	.003	0.830	[0.57, 5.66]

Note. SOC-RS = Social Orienting Continuum and Response Scale; MCDI = MacArthur Communicative Development Inventory; ADOS = Autism Diagnostic Observation Schedule; CI = confidence interval; JA = Joint Attention; CSI = Communication and Social Interaction.

Reduction = less impaired.

A main effect of time was seen for the Social Orienting Composite, with a moderate effect size (d = 0.577), though the interaction of Time and Group failed to reach significance (p = .077; see Table 3 and Figure 1). The mean change (between Time 1 and Time 2) in the Social Orienting Composite was 0.079 for the no progress group and 0.543 for the progress group. The difference in the degree of change between the two groups corresponds to an effect size of d = 0.79, a large effect, indicating that with a larger sample an interaction might be apparent. This is illustrated in Figure 1, showing the Time 1 and Time 2 Social Orienting Composite scores by group. Once again, effect sizes for the Social Orienting Composite are greater than for the MCDI but smaller than that captured by the ADOS social-communication algorithm score.

Table 3.

Effects of Group and Time on SOC-RS, MCDI, and ADOS Scores.

Item / Group	Time 1	Time 2	Effect size (d)
Item / Group	M (SD)	M (SD)	Effect size (d)
SOC
No progress	0.119 (0.537)	0.198 (0.540)	0.147
Progress	−0.119 (0.513)	0.424 (0.569)	1.004
Referencing rate
No progress	0.613 (0.329)	0.742 (0.261)	0.437
Progress	0.600 (0.286)	0.905 (0.405)	0.883
Social smiling rate
No progress	0.073 (0.056)	0.066 (0.080)	0.103
Progress	0.057 (0.082)	0.064 (0.065)	0.095
Name trial^a
No progress	6.50 (2.01)	5.80 (3.55)	0.610
Progress	3.30 (3.30)	4.20 (3.04)	0.284
JA responding
No progress	0.008 (0.017)	0.012 (0.020)	0.216
Progress	0.021 (0.025)	0.036 (0.014)	0.769
JA total
No progress	−0.173 (0.625)	0.038 (1.32)	0.217
Progress	0.173 (0.834)	0.938 (1.21)	0.749
MCDI expressive score
No progress	53.0 (118.6)	59.4 (120.4)	0.054
Progress	89.5 (85.6)	153.6 (110.5)	0.654
ADOS CSI^a
No progress	15.5 (4.67)	17.3 (3.71)	0.430
Progress	13.7 (4.72)	5.90 (4.65)	1.66

Note. SOC-RS = Social Orienting Continuum and Response Scale; MCDI = MacArthur Communicative Development Inventory; ADOS = Autism Diagnostic Observation Schedule; JA = Joint Attention; CSI = Communication and Social Interaction.

Lower score = less impaired.

Figure 1.

The interaction between Group and Time for Social Orienting Composite score.

Categorical Differences

Categorical scores were created for both the Orienting to Name and Joint Attention Responding items, in accordance with the SOC-RS manual, and as described above. Chi-square testing indicates significant between-group differences in Joint Attention Responding at Time 2 only, χ² = 7.5, p = .006, and in Orienting to Name at Time 2 only, χ² = 4.34, p = .037. This indicates that these categorical variables did detect differential change over time between groups.

Relationships Between the SOC-RS and Other Measures

To check the measure’s validity, a series of correlational analyses were performed to examine how SOC-RS scores related to the other measures collected: Vineland Adaptive Behavior Scales, Composite (VABC), MCDI, and ADOS social-communication algorithm (with a Bonferroni correction to p = .004). Only the Joint Attention Total produced correlations with other measures, as shown in Table 4.

Table 4.

Correlations Between SOC-RS Items (Combining Time Points and Groups) and Other Measures.

Other Measures	SOC-RS items
Other Measures	Referencing rate	Social smiling rate	Name trial	JA responding	JA total
VABC	r = −.031, p = .898	r = .025, p = .918	r = −.025, p = .916	r = .417, p = .067	r = .501, p = .024
MCDI	r = .306, p = .189	r = .336, p = .148	r = −.385, p = .093	r = .474, p = .035	r = .824, p < .000
ADOS	r = −.501, p = .024	r = −.292, p = .211	r = .455, p = .044	r = −.596, p = .006	r = −.657, p = .002

Note. SOC-RS = Social Orienting Continuum and Response Scale; JA = Joint Attention; VABC = Vineland Adaptive Behavior Scales, Composite; MCDI = MacArthur Communicative Development Inventory; ADOS = Autism Diagnostic Observation Schedule.

Individual Data

One strength of the small size of the groups under analysis is the possibility of exploring individual data. Figure 2 shows degree of change by individual participant in the Social Orienting Composite, ADOS social-communication algorithm, and MCDI expressive vocabulary score. The SOC-RS and MCDI scores have been transformed (multiplied by 10 and divided by 10, respectively) to facilitate their being represented on the same y axis. This illustrates the significant variability among children in the degree to which they exhibit change according to different measures. The SOC-RS scores do not always correspond to the pattern set by the MCDI and ADOS when initially identifying the two groups. At least three children in the no progress group appear to be exhibiting a large degree of improvement measured by the SOC-RS (Participants 4, 6, and 7), and others who were preidentified as having made progress do not have this reflected in their SOC-RS scores (Participants 17 and 20).

Figure 2.

Individual data showing degree of change between Time 1 and Time 2 in ADOS, MCDI, and SOC-RS scores.

Discussion

The SOC-RS development study (Mosconi et al., 2009) suggested that this new measure had potential to be used as a sensitive measure of treatment effect in intervention studies with young children with ASD, using blinded rating of behavior observed during assessment with the ADOS and focusing on key social targets of intervention. In this “proof-of-concept” study, the SOC-RS was used to rescore previously collected data from a sample of 20 children with ASD whose parents had taken part in an intervention. These children were identified a priori as having exhibited either a very small or very large change in their behavior between assessments at Time 1 and Time 2 approximately 7 months apart. This degree of change was identified using change in score on the ADOS social-communication algorithm and additionally by change in vocabulary using the MCDI. It was thought that comparison of these two extreme groups provided a good initial test of the suitability of the SOC-RS as an outcome measure. Our specific research questions were as follows:

Research Question 1: Is the SOC-RS sensitive to change over time?

Research Question 2: Can the SOC-RS detect a known, substantial treatment effect between groups?

In regard to the first question, the Social Orienting Composite score and one individual item (Referencing Rate) were sensitive to change over time—even though the period being studied was only 7 months, compared with 2 years in the original development study (Mosconi et al., 2009). When considering detection of treatment effect, again there were some significant results. Children in the progress group were more likely at Time 2 (and not at Time 1) to be categorized as “Responders” during both the joint attention and response to name presses of the ADOS. This effect indicates that the categorical scores which can be extracted from the SOC-RS may be more sensitive to differential change than the continuous variables which comprise the overall composite score.

Nevertheless, overall the SOC-RS was not sensitive to group by time interactions, and thus does not seem suitable as a measure of treatment effect. This lack of sensitivity is particularly striking, given the fact that our selection of groups means that we can be confident that a differential between-group effect over time does exist. Even when considering effect sizes for non-significant findings (given the small sample used in this study), we find that the SOC-RS scores are no more sensitive to change than the ADOS social-communication algorithm used in our original definition of the progress/no progress groups. Overall, evidence was not found that the SOC-RS could become a useful new measure of intervention efficacy.

One possible reason for this lack of sensitivity is that the SOC-RS draws on footage from a standardized assessment procedure (the ADOS), which may limit variability in the range of spontaneous social-communication behaviors observed. In addition, it measures only five dimensions of social and communication behavior. A more comprehensive measure assessing a broader range of spontaneous social and communication skills could have a greater chance of detecting subtle differences in change over time between groups.

Limitations of the Study

This proof-of-concept study analyzed data from a very small sample of children with ASD. As early intervention trials gain in methodological quality, it is becoming increasingly rare for studies to work with samples this small; therefore, it could be argued that this was an unfair test of the SOC-RS’s measurement sensitivity or suitability for use in intervention trials. However, our preselection of groups showing progress and no progress over the period being studied means that we can be confident that a treatment effect does exist, and that the SOC-RS failed to identify it sufficiently. Intervention and control groups in trials normally are found to have overlapping distributions of change scores, and some control group participants may make a substantial progress. Therefore, it is crucial that we can be confident that the chosen measurement tool is capable of detecting small but meaningful treatment effects, when studies are appropriately powered.

There is evidence of a difference in age between the two groups analyzed in this report, with a mean difference between groups of about 5 months. We considered whether the lack of SOC-RS effects could be partially attributable to this age difference. However, the original development of the SOC-RS and its use with a longitudinal sample studied at 2 and 4 years of age suggest that age effects should not have played a major part in our findings.

Strengths and Limitations of the SOC-RS

Are we closer to finding a reliable and valid early intervention outcome measurement tool in autism? The SOC-RS was selected for investigation because it meets some of the requirements. The behaviors targeted represent a concise summary of the key social and communicative skills, which are impaired in very young children with ASD, and there is no additional burden on the child where ADOS is already being used in assessment before and after intervention. The SOC-RS has been shown in its development study (Mosconi et al., 2009) to be a valid and reliable approach to directly measuring skill change over time in young children.

However, this current investigation using an extreme groups approach did not find that the SOC-RS showed greater sensitivity to change than the ADOS diagnostic social-communication algorithm in these key proximal intervention outcomes. Given that the SOC-RS uses the ADOS as a sample for coding, this finding has strong pragmatic implications. In clinical practice, where resources are limited, the time needed to derive SOC-RS codes from ADOS footage may not add value to evaluations of longitudinal individual change or intervention effect. The time-intensive nature of the process of training in the SOC-RS further limits its practicality as both a research and a clinical tool. Finally, the SOC-RS, by using the ADOS as a data sample, does not provide a completely naturalistic measure of social and communication behaviors.

Future Directions

So the search continues. One approach currently in development by Lord and colleagues is the Brief Observation of Social and Communication Change (BOSCC, formerly known as the Autism Diagnostic Observation Schedule–Change [ADOS-C]; Carr, Colombi, MacDonald, & Lord, 2011; Colombi, Carr, MacDonald, & Lord, 2011). This measure applies a subset of ADOS-style ratings, with an extended scale range, to play-based interaction between an adult and the young child with ASD. The use of this more naturalistic setting may provide the key to sensitive measurement of treatment effect, by permitting the child to engage in a wide range of spontaneous behaviors, rather than constraining their activities. The conclusion of the extensive work required to develop and test a new objective change-measurement tool is eagerly awaited by the autism research community.

A recent systematic review of the measurement properties of tools which monitor progress and outcomes of children below the age of 6 years with ASD has found significant limitations in the scope of available tools, while presenting evidence on those which appear the most robust (McConachie et al., in press). Significant development work on measurement of outcomes is still required to advance the field of early autism research and to allow meta-analysis across studies to support the strength of conclusions about which are the most effective interventions.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by Flexibility and Sustainability Funding awarded by Northumberland Tyne & Wear NHS Foundation Trust.

Notes

References

Adamson

Bakeman

Deckner

Romski

(2009). Joint engagement and the emergence of language in children with autism and down syndrome. Journal of Autism and Developmental Disorders, 39, 84–96.

Aldred

Green

Adams

(2008). A new social communication intervention for children with autism: Pilot randomised controlled treatment study suggesting effectiveness. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 45, 1420–1430.

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.

Baird

Simonoff

Pickles

Chandler

Loucas

Meldrum

Charman

(2006). Prevalence of disorders of the autism spectrum in a population cohort of children in South Thames: The Special Needs and Autism Project (SNAP). The Lancet, 368, 210–215.

Bolte

E. E.

Diehl

J. J.

(2013). Measurement tools and target symptoms/skills used to assess treatment response for individuals with autism spectrum disorder. Journal of Autism and Developmental Disorders, 43, 2491–2501.

Boyd

B. A.

Odom

S. L.

Humphreys

B. P.

Sam

A. M.

(2010). Infants and toddlers with autism spectrum disorder: Early identification and early intervention. Journal of Early Intervention, 32, 75–98.

Carr

Colombi

MacDonald

Lord

(2011, May). Measuring response to intervention with the Autism Diagnostic Observation Schedule–Change (ADOS-C). Poster presented at the International Meeting for Autism Research, Philadelphia, PA.

Centre for Disease Control. (2014). Prevalence of autism spectrum disorder among children aged 8 years-autism and developmental disabilities monitoring network, 11 sites, United States, 2010. Morbidity and Mortality Weekly Report. Surveillance Summaries, 63(2), 1–22.

Charman

(2011). Glass half full or half empty? Testing social communication interventions for young children with autism. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 52, 22–23.

10.

Cohen

I. L.

Schmidt-Lackner

Romanczyk

Sudhalter

(2003). The PDD Behavior Inventory: A rating scale for assessing response to intervention in children with pervasive developmental disorder. Journal of Autism and Developmental Disorders, 33, 31–45.

11.

Colombi

Carr

MacDonald

Lord

(2011, May). Developing a measure of treatment outcome: The Autism Diagnostic Observation Schedule–Change. Poster presented at the International Meeting for Autism Research, Philadelphia, PA.

12.

Cunningham

A. B.

(2012). Measuring change in social interaction skills of young children with autism. Journal of Autism and Developmental Disorders, 42, 593–605.

13.

Dababnah

Parish

S. L.

Turner Brown

Hooper

S. R.

(2011). Early screening for autism spectrum disorders: A primer for social work practice. Children and Youth Services Review, 33, 265–273.

14.

Dawson

Rogers

Munson

Smith

Winter

Greenson

. . . Varley

(2010). Randomized controlled trial of an intervention for toddlers with autism: The early start Denver model. Pediatrics, 125(1), e17–e23.

15.

Feldman

Greenbaum

C. W.

Mayes

L. C.

Erlich

S. H.

(1997). Change in mother-infant interactive behavior: Relations to change in the mother, the infant, and the social context. Infant Behavior & Development, 20, 151–163.

16.

Fenson

Dale

P. S.

Reznick

J. S.

Thal

Bates

Hartung

J. P.

. . . Reilly

J. S.

(1993). MacArthur Communicative Development Inventory: Users’ guide and technical manual. San Diego, CA: Singular Publishing.

17.

Gotham

Pickles

Lord

(2009). Standardizing ADOS scores for a measure of severity in autism spectrum disorders. Journal of Autism and Developmental Disorders, 39, 693–705.

18.

Green

Charman

McConachie

Aldred

Slonims

Howlin

, . . . The PACT Consortium. (2010). Parent-mediated communication-focused treatment in children with autism (PACT): A randomised controlled trial. The Lancet, 375, 2152–2160.

19.

Hudry

Leadbitter

Temple

Slonims

McConachie

Aldred

. . . Charman

(2010). Preschoolers with autism show greater impairment in receptive compared with expressive language abilities. International Journal of Language & Communication Disorders, 45, 681–690.

20.

Kasari

Freeman

Paparella

(2006). Joint attention and symbolic play in young children with autism: A randomized controlled intervention study. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 47, 611–620.

21.

Kasari

Freeman

Paparella

Wong

Kwon

Gulsrud

(2005). Early intervention on core deficits in autism. Clinical Neuropsychiatry, 2, 380–388.

22.

Kasari

Paparella

Freeman

Jahromi

(2008). Language outcome in autism: Randomized comparison of joint attention and play interventions. Journal of Consulting and Clinical Psychology, 76, 125–137.

23.

Lord

Rutter

DiLavore

Risi

(1999). Autism Diagnostic Observation Schedule: WPS edition. Los Angeles, CA: Western Psychological Services.

24.

Lord

Rutter

DiLavore

Risi

Gotham

Bishop

(2012). Autism Diagnostic Observation Schedule, 2nd edition (ADOS-2). Los Angeles, CA: Western Psychological Corporation.

25.

Lord

Rutter

Le Couteur

(1994). Autism diagnostic interview-revised: A revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. Journal of Autism and Developmental Disorders, 24, 659–685.

26.

Magiati

Tay

X. W.

Howlin

(2012). Early comprehensive interventions for children with autism spectrum disorders: A critical synthesis of recent review findings. Neuropsychiatry, 2, 543–570.

27.

Matson

J. L.

(2007). Determining treatment outcome in early intervention programs for autism spectrum disorders: A critical analysis of measurement issues in learning based interventions. Research in Developmental Disabilities, 28, 207–217.

28.

McConachie

Randle

Hammal

Le Couteur

(2005). A controlled trial of a training course for parents of children with suspected autism spectrum disorder. Journal of Pediatrics, 147, 335–340.

29.

McConachie

Parr

J. R.

Glod

Hanratty

Livingstone

Oono

I.P.

. . . Williams

(in press). Systematic review of tools to measure outcomes for young children with autism spectrum disorder. Health Technology Assessment.

30.

Mosconi

Fletcher-Watson

McConachie

Reznick

J. S.

Piven

(2010). Handbook for the Social Orienting Continuum and Response Scale (SOC-RS). Retrieved from www.ncl.ac.uk/cargo-ne/measures.html

31.

Mosconi

Reznick

J. S.

Mesibov

Piven

(2009). The Social Orienting Continuum and Response Scale (SOC-RS): A dimensional measure for preschool-aged children. Journal of Autism and Developmental Disorders, 39, 242–250.

32.

National Research Council. (2001). Educating children with autism. Washington DC: National Academy Press.

33.

Oono

I. P.

Honey

E. J.

McConachie

(2013). Parent-mediated early intervention for young children with autism spectrum disorders (ASD). Cochrane Database of Systematic Reviews, The Cochrane Library, 2013(4). doi:10.1002/14651858.CD009774.pub2

34.

Rogers

S. J.

Estes

Lord

Vismara

Winter

Fitzpatrick

. . . Dawson

(2012). Effects of a brief Early Start Denver Model (ESDM)-based parent intervention on toddlers at risk for Autism Spectrum Disorders: A randomized controlled trial. Journal of the American Academy of Child & Adolescent Psychiatry, 51, 1052–1065.

35.

Rogers

S. J.

Vismara

L. A.

(2008). Evidence-based comprehensive treatments for early autism. Journal of Clinical Child & Adolescent Psychology, 37, 8–38.

36.

Seibert

Hogan

Mundy

(1982). Assessing interactional competencies: The Early Social Communication Scales. Infant Mental Health Journal, 3, 244–245.

37.

Sparrow

S. S.

Cicchetti

D. V.

Balla

D. A.

(2005). Vineland Adaptive Behaviour Scales (2nd ed.). San Antonia, TX: Pearson Assessments.

38.

Spence

S. J.

Thurm

(2010). Testing autism interventions: Trials and tribulations. The Lancet, 375, 2124–2125.

39.

Stone

W. L.

Coonrod

E. E.

Pozdol

S. L.

Turner

L. M.

(2003). The Parent Interview for Autism-Clinical Version (PIA-CV): A measure of behavioral change for young children with autism. Autism, 7, 9–30.

40.

Sussman

(1999). More than words: Helping parents promote communication and social skills in children with autism spectrum disorder. Toronto, Ontario, Canada: Hanen Centre.

41.

Wan

M. W.

Green

Elsabbagh

Johnson

Charman

Plummer

(2012). Parent-infant interaction in infant siblings at risk of autism. Research in Developmental Disabilities, 33, 924–932.

42.

Wetherby

Prizant

(2002). Communication and Symbolic Behavior Scales–Developmental Profile, First Normed Edition. Baltimore, MD: Paul H. Brookes.

43.

Wolery

Garfinkle

A. N.

(2002). Measures in intervention research with young children who have autism. Journal of Autism and Developmental Disorders, 32, 463–478.

44.

World Health Organization. (1994). International Classification of Diseases (ICD10). New York, NY: Author.