A Multi-Trait Multi-Method Examination of Psychometric Instrument Performance in Autism Spectrum Disorder

Methods

We report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study.

Participants

Data came from multiple case-control studies of children diagnosed with ASD (e.g., McAuliffe et al., 2017, 2020). Participants were recruited from local schools, from those who receive services at the Kennedy Krieger Institute and express interest in research participation, and from community clubs/organizations. The collection and analysis of data were approved by the Johns Hopkins Medicine Institutional Review Board. All data were collected by psychology associates supervised by doctoral-level psychologists. Inclusion for ASD criteria consisted of age 8.0 to 12.9 years, meeting criteria based on the ADOS-G or ADOS-2, module 3 (Lord et al., 2000, 2012) and final diagnosis by a single child neurologist specialized in the diagnosis of ASD. Some studies additionally required meeting criteria via the Social Communication Questionnaire (Rutter et al., 2003), Autism Spectrum Screening Questionnaire (Ehlers et al., 1999), and/or the Autism Diagnostic Interview-Revised (ADI-R) (Lord et al., 1994). In addition, participants were required to have had a Full-Scale IQ (FSIQ) > 80, unless there was a 12-point or greater index discrepancy in index scores, in which case either the VCI, VSI, or FRI must be > 80. Participants also had no co-occurring genetic or developmental disorder, or epilepsy. Further screening for psychiatric history was conducted using the Kiddie Scale for Affective Disorders and Schizophrenia (K-SADS) (Kaufman et al., 2013). Participants with Attention-Deficit/Hyperactivity Disorder (ADHD), Anxiety Disorders, and OCD were included, as there is a substantial amount of co-occurrence among these conditions and autism (Brookman-Frazee et al., 2018; Ewen & Shapiro, 2005; Rong et al., 2021; Simonoff et al., 2008). Eighty-three children met the inclusion criteria. See Table 1 for demographic information. 1.5% of eligible participants meeting ASD criteria for inclusion were excluded due to behavior (i.e., inability to complete all tasks of their respective study—not necessarily the measures reported here), 6% were excluded based on IQ criteria, and none was excluded based solely on an exclusionary psychiatric diagnosis.

OPEN IN VIEWER

Table 1.

Demographic Data of Study Participants.

Age	10.3 years (1.45, 8–12)
Sex	M = 70; F = 13
Race (% Caucasian)	81%
Co-occurring ADHD	47%
Co-occurring anxiety disorder	<0.1%
WISC-IV FSIQ	101.4 (16.8; 71–138)
WISC-V FSIQ	100.1 (15.3; 74–125)
ADOS total	14.23 (4.03; 7–27)
SRS total	75.0 (11.1; 50–90)
WISC-IV WMI Standard	97.92 (16.87, 59–135)
WISC-V WMI Standard	97.06 (18.52, 74–130)
BRIEF WM T	66.32 (10.00, 45–84)
BRIEF Inhibition T	64.02 (12.18, 35–88)
NEPSY Statue total	21.67 (7.52, 3–30)
DCDQ Grand DCD total	48.34 (11.73, 18–77)
PANESS total	41.98 (13.64, 13–73)
WISC-IV DSF Scaled	9.50 (3.27, 3–18)
WISC-V DSF Scaled	9.06 (2.71, 5–16)
Conners Inattention T	68.78 (12.44, 43–90)

Note. Values represent means and (SD, ranges), unless otherwise noted. ADHD = attention-deficit/hyperactivity disorder; WISC-IV = Wechsler Intelligence Scales-IV; FSIQ = Full-Scale IQ; ADOS = Autism Diagnostic Observation Schedule; SRS = Social Responsiveness Scale; WMI = Working Memory Index; BRIEF = Behavior Rating Inventory of Executive Function; DCDQ = Developmental Coordination Disorder Questionnaire; PANESS = physical and neurological examination of subtle signs; DSF = digit span forward.

Statistical Analysis

We had both performance-method and report-method data in support of a total of five psychological domains (traits) (Table 2). We modeled each measurement according to its trait and method; for participant i, trait j, and method k, we have the following statistical model:

y i j k = μ j k + α i j + β i k + ε i j k ,

where μ_jk was a fixed mean effect of the jth trait and kth method, α i j ~ N ( 0 , σ j 2 ) , β i k ~ N ( 0 , τ 2 ) , and ε i j k ~ N ( 0 , γ 2 ) . The correlation structures were set such that α i j , β i k , and ε i j k were independent of each other. There are several aspects of this model worth discussing. First, we chose linearity and separability with additive symmetric errors, as it is a parsimonious model and the general first starting point for most analyses. Setting the alpha and beta terms as random effects was done since having parameters grow as fast as or faster than the sample size creates statistical issues. The trait and method effects were separated for parsimony, including an interaction between these results in a large increase in the number of parameters and a more saturated model.

OPEN IN VIEWER

Table 2.

Seven Psychological Domains (Traits, Constructs) Examined, and the Report- and Performance-Method Instruments Used to Assess Each.

Trait	Parent rater	Performance
Inattention	Conners-3	Digit Span Forward
Working memory	Behavior Rating Inventory of Executive Functioning (BRIEF) Working Memory	Wechsler Intelligence Scale for Children Working Memory Index
Inhibition	BRIEF Inhibition	NEPSY Statue
Motor coordination	Developmental Coordination Questionnaire	Physical Assessment of Neurological Soft Signs
ASD core features	SRS total score	ADOS total score

Note. BRIEF = Behavior Rating Inventory of Executive Function; ASD = autism spectrum disorder; SRS = Social Responsiveness Scale; ADOS = Autism Diagnostic Observation Schedule.

Data were normalized to account for directionality of severity and the use of raw and scaled scores. To estimate the parameters in the model designed above, we estimated μ j k , σ j 2 , τ 2 , γ 2 using Maximum Likelihood Estimation from the structural equation modeling package in STATA version 17 (StataCorp, LLC, College Station, TX).

Results

Sixty-eight participants had the WISC-IV, and 15 had the WISC-V. Thirty-five participants had the SRS, and 48 had the SRS-2. Thirty-three participants had the ADOS-G, and 50 had the ADOS-2. A few participants had a computed FSIQ < 80, in line with the caveat relating to index-factor discrepancy, as described above.

Parent-rating instruments were susceptible to a greater amount of method-attributed variance in each psychological domain (trait) compared to that of the corresponding performance measures (Table 3). The aggregate estimate of shared variance attributable to method among parent-rating instruments was 57%, whereas the aggregate estimate of shared variance attributable to the performance method was 22%.

OPEN IN VIEWER

Table 3.

Percentage of Shared Factor Variance Explained by Method for Each Trait of Interest.

Trait	Rater	Performance
Attention	88.2	74.5
Working memory	91.2	71.8
Inhibition	91.7	37.2
Motor coordination	72.8	59.0
Autism core features	84.1	53.3

Table 3 reflects the percentage of the variability in each instrument score that is attributable solely to the method. Note that percentages do not add to 1 since they are percentages of factor variability. Usual care should be taken when comparing normalized percentages across traits. The key conclusion, however, is that the percentage of score variability attributable to method is greater for rater instruments than for performance instruments (rather than across-trait comparisons). This pattern is true not only across all domains (traits) but also within each and every domain measured.

Discussion

Our results indicate that, among autistic children with a restricted IQ range, a substantial percentage of between-individual variation seen in parent-report scores can be attributed to a bias that is inherent to rater methodology. That is to say that an individual’s score on a particular trait-specific rating scale is based to a large degree on how the individual is rated (by a parent, in this analysis) in a nonspecific fashion rather than the individual’s true level of expression of the targeted psychological trait. This large rater-method bias could explain, to a degree, the lack of correlation in ASD samples between ADOS and SRS (Leung et al., 2016), despite the fact that they were both designed to measure core ASD features, or between performance and rater measures of sensory reactivity (Wodka et al., 2016), as the rater bias effect may overwhelm true correlations between the performance and rater instruments targeting the same trait.

We know that this across-trait correlation of rater-instrument scores is not solely a function of true covariance of inter-individual abilities across traits (e.g., due to biological variation that affects performance in multiple psychological domains), because performance measures showed a substantially smaller degree of covariance across traits. (The mathematical alternative, that rater instruments truly capture real between-trait covariance and that the performance instruments show greater within-individual, between-trait differences due to enhanced susceptibility to random noise, seems less likely on its face.) Future work using samples from multiple raters per participant (e.g., parents and teachers) could better triangulate the specific role of the rater in the across-trait covariance; the convenience sample used did not include information from multiple raters. Moreover, future work may also evaluate the psychometric properties associated with different types of raters (self, parent, and teacher).

Although our results were primarily dependent on within-method covariation across traits, we are also conscious that our model also takes into account the within-trait correlation between instrument method. Here, there is a question about the degree of alignment of targeted traits within performance and rater pairs targeting the same trait. In choosing a performance instrument and a rater instrument for the same trait, our goal was to select instruments that would be viewed as representing the designated trait in a way that matches clinical practice. That said, we are also conscious that perfect alignment around the same trait is challenging. For one thing, the empirical delineation of the “primitive,”“most fundamental,” or “irreducible” traits/domains within Neuropsychology or Clinical Psychometrics is often limited (Poldrack & Yarkoni, 2016). Although the psychometric field has empirical approaches for defining the boundaries of a construct (Cronbach & Meehl, 1955; Nunnally & Bernstein, 2010; Strauss & Smith, 2009), and highly rigorous methods for the empirical validation of constructs have been proposed in research contexts (e.g., Shallice & Cooper, 2011, §3.3), actual clinical and research practice has often generated novel constructs and distinctions “ad lib and ad hoc” (Poldrack & Yarkoni, 2016, p. 588, citing W. R. Uttal). As one example, what level of granularity represents true and natural distinctions, for example, attention versus sustained attention versus auditory sustained attention?

The second consideration is that all instruments’ scores are sensitive not only to the traits they are purported to measure but are also sensitive to forms of artifact (such as bias within rater instruments, or the quality of the prior night’s sleep within performance instruments) as well as to other level of function of other psychological traits/domains (such as general intelligence tested via verbal IQ scales being sensitive to language ability/disability). We have generally spoken about the rater-instrument bias term observed within this analysis as being a product of rater bias, but it is equally plausible that rater instruments could be sensitive to a broader range of nontargeted psychological traits than performance instruments. Again, having rater data with ≥2 raters per participant would help disentangle these possibilities, as would multiple rater and performance instruments per trait (under the assumption that each instrument is sensitive to different confounds, which will average each other out when combining across instruments). Moreover, the use of instruments representing an even more diverse range of methods (e.g., interview and ethological observation) would help better characterize the testing artifacts and sensitivity to nontarget psychological domains than the current work. Doing so would involve a larger sample as well as other analysis methods, such as structural equation modeling. Such work may be particularly important within the study of core autism symptoms, where our conceptual models (compared with other psychological domains) are still under evolution. Optimally, the field would also subject each instrument used to be subject to a comprehensive task analysis (Friedrich & Rader, 1997) based on an empirically defined ontology of psychological traits (Poldrack & Yarkoni, 2016).

In our sample of autistic children, a condition that is especially heterogeneous in its presentation, the bias identified in rating forms further motivates the use of in-person testing for nuanced neuropsychological assessment in children with ASD, whether for research or clinical purposes. We could also envision future work to develop statistical adjustments against rater bias within a battery of rater measures, to render them more effective at making performance distinctions across traits.

The limited range of IQ, verbal skill, and core symptom presentation mean these results are not generalizable to the entirety of the autism spectrum. The visibility and impact of psychological variation in attention, working memory, inhibitory, motor coordination, and core autistic features may be different in autism groups with IQs different from the ones reported here. In fact, the evidence that exists suggests that the discriminatory ability of tests intended to measure autistic core features may perform less well in populations exhibiting IQs lower than in our current sample (Havdahl et al., 2016), whether or not the cause is method-related bias. The male-to-female ratio in this sample was higher than the typically reported 4:1 ratio for autistic individuals, and sex differences were not analyzed. However, more heavily male-dominated ratios in ASD samples have been reported among higher-IQ-range samples (de Giambattista et al., 2021; Werling & Geschwind, 2013), as was the case here.

While the applicability of the current project is toward understanding how best to characterize the psychological profile of individuals already diagnosed with ASD, future work may leverage transdiagnostic samples (see Tang et al., 2023) to understand how psychometric instruments function at the borderlands of differential diagnosis and with regard to co-occurring conditions (for the instruments used here, most notably Attention-Deficit Hyperactivity Disorder and Intellectual Disability).

In conclusion, the method by which psychometric data are gathered in ASD has an impact on interpretation. Data collected by surveying parent observations of their child’s behavior share much more variance (over 50%) than data collected by directly measuring the child’s performance on a task. As such, parent-rated scales may be no more specifically descriptive of inter-individual differences on the construct they are reportedly measuring than they are with the mélange of factors that goes into parent report across traits.