The diagnosis of autism in community pediatric settings: Does advanced training facilitate practice change?

Introduction

With an estimated prevalence of 1 in 88 (Centers for Disease Control and Prevention (CDC), 2012), the accurate and effective identification of young children with autism spectrum disorder (ASD) represents a pressing public health and clinical care issue. An increasing body of literature supports substantial gains in cognitive and adaptive functioning for groups of young children with ASD who receive intensive specialized intervention services (Dawson et al., 2010; Warren et al., 2011). The American Academy of Pediatrics (AAP) guidelines have endorsed universal screening for ASD at 18 and 24 months of age and at any point when the caregiver expresses concerns (Johnson and Myers, 2007). Ideally, children screened at-risk would receive comprehensive, expert multidisciplinary evaluations in a time-sensitive manner, but significant limitations in community resources make it quite common for families to wait substantial intervals of time (6–12 months or more) for comprehensive evaluations (Nicholas et al., 2009; Zwaigenbaum and Stone, 2006). These delays contribute to significant familial distress and may limit access to ASD-specialized intensive intervention (Bailey, 2008; Nicholas et al., 2009; Warren and Stone, 2011). AAP practice guidelines do call for immediate referral to local Birth-to-Three/Part C systems subsequent to positive screens, but in many instances, a specific ASD diagnosis is required to access higher intensity and specialized intervention (Lord and Richler, 2006; Stahmer and Mandell, 2007). Children may be able to access some developmental intervention services through early intervention systems while they await diagnostic clarification, but the number of hours of service received without an ASD diagnosis is often drastically different from recommended levels (Stahmer and Mandell, 2007).

The situation is complicated further by the use of screening instruments that may often stress sensitivity and result in high rates of false-positive screens. This suggests that without significant additional resources in place, wide-scale screening for ASD at young ages may in fact increase wait times for appropriate diagnostic assessment (Nicholas et al., 2009). Given the limitations of available screening instruments and community resources, some researchers have gone as far as to suggest that there may be reason to question systematic population-based screening for ASD (Al-Qabandi et al., 2011). The growing focus on the value of early intervention, large-scale awareness initiatives, and parental demand for appropriate assessment of ASD concerns suggest that movement toward more limited screening is unrealistic. Given this context, pediatric providers need accurate and time-efficient approaches not only for screening but also for evidence-based methodologies for moving from screening to efficient diagnostic identification and subsequent treatment.

This study builds upon and extends a pilot training study designed to reduce delays in providing ASD-specialized early intervention services to young children (Warren et al., 2009). This initiative represents collaboration between the state chapter of the AAP, the Birth-to-Three/Part C system, and a university-based hospital and regional referral center. The pilot training was explicitly designed with the goal of providing community pediatricians training in a methodology for accurate diagnostic identification of ASD within their own practice setting. Specifically, the training was designed to teach enhanced diagnostic consultation and screening procedures to physicians and pediatric medical providers so that they may conduct time-sensitive (approximately 1 h) autism-focused assessments with children aged 18–36 months. The assessment framework involved the following components: (a) formal training on using the Modified Checklist for Autism in Toddlers (M-CHAT) and interview (Robins et al., 2001), (b) administration and scoring of the Screening Tool for Autism in Toddlers and Young Children (STAT), a level-2 screening instrument normed on children aged 24–35 months with extended scoring systems for 3-year-olds (Foss-Feig and Stone, 2007) and children as young as 14 months (Stone et al., 2008), that provides formal scores related to key social communication skills yielding a total ASD risk cutoff score subsequent to an interactive assessment of 15–20 min (Stone, et al., 2004), (c) use of brief, developmentally sensitive parent interview targeting core ASD symptoms (Warren et al., 2009), and (d) identification of appropriate billing and coding procedures. Following the training, participating pediatricians implemented the assessment framework (i.e. the 1-h consultation model) in their respective practices and subsequently referred a consecutive series of patients for independent evaluation at a university-based clinic. Results of forced-choice classification revealed 71% agreement but also a tendency toward overidentification of ASD in children with other neurodevelopmental complexities. Feedback from providers revealed that the forced-choice classification failed to provide a mechanism for capturing diagnostic uncertainty. Furthermore, the small sample of the pilot training, the intentional preselection of involved providers, and the lack of data concerning actual practice change limited our ability to comment on the potential viability of such a program as a methodology for enhanced identification of ASD statewide.

The current investigation expands this earlier pilot work by presenting a more comprehensive evaluation of this training model. A broader sample of pediatric providers received training, program evaluation data were collected over a longer time frame, and diagnostic accuracy was evaluated in an additional independent nonoverlapping sample of providers now allowing for diagnostic uncertainty.

Methods

Results

Practice change

The practice change questionnaire was returned by 26 of 27 (96%) participants. The one nonrespondent was a community pediatrician from the initial pilot sample who failed to complete training (Warren et al., 2009). Given the focus on training community pediatric providers, participating specialty providers (i.e. three developmental and behavioral pediatricians, one neurologist) were excluded from primary analyses. A majority of providers (91%) returning the survey indicated routine use of ASD-specific screening prior to training; in all cases, this involved use of the M-CHAT questionnaire, without the follow-up interview. Prior to training, the majority of providers (77%) indicated a routine practice behavior of referring children who screened positive for ASD on the M-CHAT questionnaire for outside evaluation without additional within-practice assessment (Table 1).

OPEN IN VIEWER

Table 1.

ASD practice characteristics before and after training.

	Pretraining	Posttraining
Utilize ≥1 formal ASD screening measure	91%	95%
Assess children with STAT	0%	64%
Conduct independent ASD assessment	23%	68%

ASD: autism spectrum disorder; STAT: Screening Tool for Autism in Toddlers and Young Children.

Paired-samples t-tests revealed pre- to posttraining differences indicating increased comfort level for discussing ASD diagnoses in young children. The mean level of comfort after training (M = 4.05, SD = 0.65) was significantly greater than the level of comfort prior to training (M = 2.50, SD = 1.06; t = 8.45; p < 0.001). There was also a substantial increase (t = 2.32; p < 0.05) in the number of young children who received a preliminary ASD diagnosis from involved providers during the most recent year of practice (M = 5.19, SD = 8.59) compared to the year prior to training (M = 2.81, SD = 6.84). On average, the number of children diagnosed within practice by providers involved in this training increased by 85%. Respondents also reported a significant change in their measured perception of the appropriateness of primary care providers issuing formal autism diagnoses pretraining (M = 2.27, SD = 1.03) compared to posttraining (M = 3.91, SD = 0.92; t = −7.66, p = < 0.001). At our follow-up measurement, an overwhelming majority of providers (91%) indicated that they felt it was appropriate (ratings ≥ 3) for “a child to receive a diagnosis from his primary care provider without or before a referral for more comprehensive evaluation.”

McNemar’s test revealed a large statistically significant shift (p < 0.01) in identified practice behavior, with 68% reporting that following the training they would be likely to conduct within-practice assessments for ASD (Table 1). Finally, the respondents indicated several barriers to using the methods from the training, endorsing a lack of appropriate referrals, time constraints, and a lack of resources and intervention services available to families postdiagnosis (Table 2).

OPEN IN VIEWER

Table 2.

Endorsed barriers to practice change.

	%
Time constraints	50
Financial constraints	36
Lack of appropriate referrals	32
Lack of resources postdiagnosis	27
Difficulties procuring early intervention	14

Clinical confirmation

Six of the eight local community pediatric providers attending the trainings within the catchment area of the lead training university referred families for independent evaluation within the time interval afforded (6 months posttraining). Of the two not completing such referrals, one pediatrician moved practice settings and the other cited few referrals. Of the 16 children referred for independent evaluation, 14 participated, with two families failing to keep scheduled appointments and respond to additional contact attempts. The ages of the 14 children (13 boys, 1 girl) who completed the confirmation assessment ranged from 24 to 42 months (Table 3). Eight children received a forced-choice ASD classification by the referring pediatric provider, while six received a forced-choice nonspectrum classification. The number of children referred by each provider ranged from one to four.

OPEN IN VIEWER

Table 3.

Child characteristics of confirmation sample.

Measure	ASD (n = 8)			Non-ASD (n = 6)			Total (N = 14)
	M	SD	Range	M	SD	Range	M	SD	Range
Age (months)	32.6	7.6	25–45	30.5	4.3	25–35	31.7	6.3	25–45
ADOS-CSS	7.0	2.3	4–10	2	0.9	1–3	4.9	3.1	1–10
MSEL-ELC	72.5	6.4	44–88	78.2	7.1	62–114	71.5	23.4	44–114
VABS-II-ABC	58.8	16.4	64–85	84.2	23.4	69–84	74.7	7.0	64–85

ADOS-CSS: Autism Diagnostic Observation Schedule–Calibrated Severity Score; MSEL-ELC: Mullen Scales of Early Learning–Early Learning Composite; VABS-II-ABC: Vineland Adaptive Behavior Scales, Second Edition–Adaptive Behavior Composite; SD: standard deviation; ASD: autism spectrum disorder.

In 12 cases, the pediatric providers indicated confidence in their ASD determination, with providers noting uncertainty of an ASD diagnosis based on their own evaluation in 2 cases. Based on forced-choice classifications, there was diagnostic agreement in 86% of cases (12/14; one false-positive, one false-negative) with comprehensive evaluation ratings. When providers were able to explicitly note uncertainty and when uncertain cases were counted as agreements, agreement rose to 93% across the sample (one false-negative).

Discussion

This study presents results from a training initiative developed to provide community pediatric providers from across the state with an evaluation structure for successful diagnostic classification of young children suspected of ASD within their own practices following screening. The most promising finding that emerged over the course of this training program was that many pediatric providers documented significant changes not only in ASD-related diagnostic attitudes but also in actual implementation of within-practice ASD assessment over a fairly substantial time interval.

This replication of the diagnostic confirmation procedures from the initial pilot study suggested that most young children assessed by community providers using this model were accurately classified (86%–93%). Taken together with results from the initial pilot confirmation procedure, wherein 71% of children were accurately identified within a pure forced-choice diagnostic framework (Warren et al., 2009), results suggest that advanced training in structured ASD interactive assessment and interviewing can facilitate accurate diagnostic classification for some children within community pediatric settings. If such documented diagnostic accuracy was held within a larger and more generalizable sample of both children and providers, this would represent a substantial improvement on the most widely used ASD screening instruments (Baird et al., 2000; Dietz et al., 2006; Pinto-Martin et al., 2008). Specifically, the positive predictive value of the M-CHAT between 16 and 30 months has reported to be as low as 11% when used in isolation (notably the only screener used by our sample prior to training) and 65% when used in combination with the accompanying interview (Kleinman et al., 2008). Such psychometric improvement could potentially help facilitate a host of individualized clinical decision-making opportunities for clinicians and families, in terms of pursuing additional diagnostic and treatment services.

The ultimate success of the current model and other related training programs requires not only demonstration of improved understanding of enhanced identification methods but also significant evidence of actual change in practice. Although significant barriers related to time and financial pressure were noted, a majority of those participating in training reported substantial changes related to primary diagnostic assessment practices. While some providers were conducting independent assessment for ASD for young children within their practice prior to training through the use of an ASD-specific screening questionnaire, no providers were using a standardized assessment instrument or ASD-specific interview. Following the training, there were substantial increases in the number of diagnoses made within respective provider practices (85%). Providers also documented significant shifts in their comfort discussing ASD diagnoses along with a corresponding dramatic shift in endorsed practice behaviors when emergent concerns about ASD were noted. Prior to the training, providers primarily referred cases outside their practice for independent evaluations. Following training, they often chose to conduct their own independent evaluation prior to, or in place of, such a referral. Unfortunately, this study methodology did not tease apart whether practitioners were in fact providing diagnoses in place of or prior to expert evaluation. This is certainly a very important differentiation that needs further study to determine how such models are best situated within diagnostic best practices. With this significant limitation noted, the reported practice changes of participating providers suggest that incorporating specific ASD interactive assessment practices is possible for certain providers and that this training may actually be a preferable practice model for these same providers. Such changes were seen as the result of a relatively brief (2-day) training with very low level of technical support and across a fairly large service geography. This suggests potential for broad-scale implementation, without substantial resource investment on the part of providers or existing diagnostic referral centers.

First, as a whole, the available data from this training program suggest that experienced pediatric providers receiving advanced training in ASD-specific interactive evaluation and interviewing can be quite accurate in their diagnostic decisions and successfully incorporate a time-efficient diagnostic consultation model into their practices. However, several significant methodological limitations of this study should be taken into account. Our cumulative diagnostic confirmation samples were nonrandom and included a relatively small number of children. This was due in part to geographical barriers for some state providers to refer children to the host institution. Geography, although a strength in demonstrating change across locations with varying issues related to access/practice, hampered our ability to provide diagnostic confirmation for all families. Second, this training represented an initial exposure and starting point for providers in terms of incorporating this tool into practice. Time, referrals, and support resources were initially seen as barriers toward implementation of the protocol and limited our ability to pull in large number of participants. Finally, we did not have substantial resources to provide pro bono diagnostic assessment/evaluation services for a large number of families over extended periods of time. As such, the characteristics of children and families successfully classified and those possibly misclassified are lacking.

In addition, most providers participating had substantial years of experience, were highly motivated to participate, and were already routinely conducting brief ASD screening. This may raise questions regarding the baseline level of skill, motivation, and experience necessary for achieving both accurate diagnostic classifications and practice change across provider groups. While it was not unexpected that providers responding to this training opportunity throughout the state represented a subgroup of specialized pediatric health-care providers, future replication of the training program predominately involving general pediatricians and nurse practitioners is needed to better evaluate the program’s broader potential impact and utility. Finally, a limitation of the present methodology was the reliance on retrospective self-report methodology to determine practice change. Despite intentions to limit need for detailed recall of practice behaviors and attitudes prior to training, rates at which providers may have overestimated training effect could not be measured, thus warranting future evaluation utilizing prospective reporting. More specific data are needed to better measure whether shifts in practice behavior translated into accelerated access to ASD intervention services.

Despite demonstrating potentially high levels of diagnostic agreement within community practice settings that rival that of experienced psychological providers (Turner and Stone, 2007), the question of whether such brief pediatric focused diagnostic models should represent ideal models of care must be considered. Utilization of briefer assessment models will undoubtedly contribute to more errors in definitive classification. Misclassification of ASD can hold powerful, long-lasting negative consequences for children and families (Johnson and Myers, 2007; Warren and Stone, 2011). As such, attention must be paid to both the potential risks associated with brief diagnostic models and the potential benefit of expedited, appropriate, and intensive early intervention services for a greater number of young children identified with ASD. The impact of briefer models of diagnostic identification on families, even when accurate, needs further clarification. Most of the literature examining family reaction to diagnosis suggests increased satisfaction in relation to shorter intervals of time from concern to diagnosis and fewer encounters with providers to receive a diagnosis (Brogan and Knussen, 2003; Goin-Kochel et al., 2006). However, there is considerable evidence from other areas of pediatric illness indicating that the way in which diagnostic information is conveyed has a significant impact on familial coping, stress, and behavior (Marvin and Pianta, 1996). This study provides strong data from the participating pediatric providers that brief, within-practice ASD assessment models appear appropriate; however, data were not obtained directly from families about such models. Future study is needed to determine agreement between primary provider diagnosis and subsequent diagnostic evaluation by an expert in ASD diagnosis. Finally, while it might be preferable to utilize the concept of “ASD risk” rather than definitive diagnosis to garner intervention services for very young children, many service systems currently require specific diagnosis in order to receive intensive ASD intervention, and a full revision of eligibility and service models may not be realistic. At the same time, there is great concern about potential harms that may come from overconfidence, misclassification, and overdiagnosis of ASD. This tension between ideal care and earlier effective diagnosis is certainly not easily resolved, and the current work represents only a preliminary investigation into a potential pathway for expediting accurate diagnostic consultation regarding ASD at early ages.

Conclusion

Given the deleterious consequences of lengthy waits for comprehensive diagnostic evaluations during a presumed critical window of neurobehavioral plasticity, the potential impact of training programs for advancing within-practice ASD identification and assessment may be quite powerful. The recent AAP parameters for ASD screening have constructed a practice recommendation based upon substantial, high-quality evidence. However, like many practice statements (Belamarich et al., 2006), these recommendations have been offered without providing specific models for addressing the service and care issues that are the direct result of enhanced screening. The question of how to address the results of enhanced ASD screening on a population level is a pressing one for both pediatric primary care and referral specialty providers. This article describes an initial attempt to improve the ability of one state system to respond to screening initiatives via a collaborative training pathway between regional practices and a university-based hospital and referral center. At rates of 1 in 88, collaborative methods for advancing effective within-practice identification of ASD are quite likely necessary for pragmatic functions of enhanced screening. It should be noted that this study represents only a preliminary investigation of such a collaborative methodology, and more research is necessary to understand how best to facilitate accurate identification methods as well as effective treatment of children with ASD across pediatric systems of care.