Evaluation of the Autism Spectrum Rating Scales in a Diverse,Nonclinical Sample

Abstract

Despite the changing racial/ethnic demographics in the United States, few studies exist that evaluate autism spectrum disorder (ASD) screening and diagnostic assessment measures for their cultural and linguistic responsiveness. The purpose of this study was to evaluate the structure of the Autism Spectrum Rating Scales (ASRS) in a diverse sample of parents with children (nonclinical sample) between the ages of 6 and 18 years (N = 405). Confirmatory factor analyses, factor correlations, and the evaluation of item loadings were used to examine the structure of the ASRS across cultural groups. Results yielded cross-cultural differences. Implications and directions for future research are discussed.

Keywords

cultural/cross-cultural validity factor analysis autism spectrum disorder parents

Autism spectrum disorder (ASD) is characterized by deficits in social communication and interaction skills and the presence of restricted and repetitive patterns of behavior, interests, or activities (American Psychiatric Association, 2013). Children with ASD may receive a medical diagnosis of ASD and/or special education services under the eligibility category of autism. Identifying children with ASD in both medical/clinical and educational settings is imperative as it leads to service provision. In particular, early identification of ASD is ideal as the outcomes associated with treatment at earlier ages are greater (e.g., Zwaigenbaum et al., 2015).

Appropriately identifying ASD can be challenging for practitioners. Given the complexity of ASD and the frequency at which mental health and medical problems co-occur with the disorder (e.g., Cubala-Kucharska, 2010; Simonoff et al., 2008), an interdisciplinary evaluation team with members who have expertise in the disorder is recommended (e.g., Johnson & Myers, 2007; Ozonoff, Goodlin-Jones, & Solomon, 2005). Moreover, a comprehensive evaluation that includes measuring several domains of functioning is also strongly encouraged (e.g., Campbell, Ruble, & Hammond, 2014). Although there are various instruments across domains of functioning (e.g., cognitive, language, adaptive) that can be used to provide a holistic picture of strengths and weaknesses that should be included in an ASD evaluation, there are several ASD-specific tools available to help clinicians determine whether ASD is present and, if so, to gauge symptom severity and inform intervention planning.

There are several available instruments to evaluate ASD-specific symptoms. The Autism Diagnostic Observation Schedule–Second Edition (ADOS-2; Lord et al., 2012) and the Autism Diagnostic Interview–Revised (ADI-R; Le Couteur, Lord, & Rutter, 2003) are viewed as the “gold standard”; however, many clinicians across various settings may not use either of these measures due to the time-intensive administration and training requirements. Even for clinicians who do utilize these tools, other instruments are likely to be included as ASD evaluations should utilize multiple measures. Other commonly used instruments include rating forms, which provide perspectives from parents/caregivers and/or teachers. For example, although the majority of school psychologists report using rating forms, only some use the ADOS during school-based ASD evaluations (Aiello, Ruble, & Esler, 2017). School psychologists who do not use the ADOS are most likely to utilize rating forms to inform their evaluation and identification practices (Akshoomoff, Corsello, & Schmidt, 2006). There are several ASD-specific rating forms available to clinicians. The selection of appropriate, valid, and reliable measures is crucial. This is particularly true when conducting ASD evaluations for culturally and linguistically diverse (CLD) populations. The definition of culture is broad and includes a range of defining characteristics of a group of people, such as socioeconomic status, race and ethnicity, or religion. We focus on racial and ethnic diversity in this article, with a specific emphasis on Black, Latinx, and White populations.

Disparities in ASD Identification

Although there has been substantial research focused on disparities of multiple health conditions (e.g., Institute of Medicine, 2002), limited research has been conducted pertaining to disparities in ASD identification. Early research in this area has found disparities in ASD identification particularly related to race/ethnicity and socioeconomic status. Mandell, Listerud, Levy, and Pinto-Martin (2002) found that Black children were more likely to be identified with ASD at a later age if they were receiving Medicaid, an indicator of socioeconomic status, than White children. Furthermore, in another study by Mandell, Ittenbach, Levy, and Pinto-Martin (2007), Black children with ASD were more likely to be first diagnosed with conduct disorder or adjustment disorder than White children. In a study of a community sample, researchers found that Hispanic children were less likely than White children to be diagnosed with ASD at all, and that Black children with ASD were diagnosed at older ages than White children (Centers for Disease Control and Prevention, 2006). ASD disparities have also been documented in the educational identification of ASD. Sullivan (2013) found that in comparison with White students, Asian students were more likely to receive a special education identification of ASD. Furthermore, in comparison with White students with ASD, Hispanic students were less likely to receive this identification. This research also demonstrated an extreme variability in these prevalence rates across U.S. states (Sullivan, 2013).

The most current data regarding disparities in ASD diagnosis and identification are available through the Centers for Disease Control Autism and Developmental Disabilities Monitoring (ADDM) Network. ADDM prevalence rates are obtained from reviewing diagnostic and educational identification records of 8-year-old children from 11 states. In 2018, the combined estimated prevalence of ASD among White children was 7% greater than among Black children and 22% greater than that among Hispanic children. Notably, individual states vary extensively regarding prevalence among racially diverse children. For example, in three states, the White to Black ASD prevalence ratios were statistically significant, and in nine states, the prevalence of ASD was higher among Black children than among Hispanic children. Only in one state were there no statistically significant differences in the ASD prevalence ratio among White, Black, and Hispanic children. Variability among certain racial groups was also large, as Asian/Pacific Islander children ranged from 7.9 cases per 100 in one state to 19.2 cases per 1,000 in another state (Baio et al., 2018).

Disparities in ASD identification can significantly affect the educational, social, and vocational outcomes for children with ASD. If a child receives a delayed diagnosis of ASD or a misdiagnosis, they are missing critical early intervention services. Cultural expectations pertaining to social communication or behaviors may contribute to disparities in identification, as identification measures may not represent culturally responsive behavioral expectations or concerns of family members. For example, Zhang, Wheeler, and Richey (2006) found that eye contact with adults and pointing with the index finger is considered inappropriate within the Chinese culture. Norbury and Sparks (2013) reported cultural differences in pretend play, public displays of emotion, and engagement with toy dolls. Issarraras, Matson, Matheis, and Burns (2018) found differences in concerns specific to social development across parents from different racial/ethnic groups with children in an early intervention program. Social communication behaviors are typically part of an ASD assessment and will likely be influenced by culture. Limited research exists pertaining to ASD identification tools and potential differences in performance based on racially/ethnically diverse groups.

Cultural Responsiveness of ASD Measures

Despite the changing racial/ethnic demographics in the United States and the increased numbers of ASD evaluations being conducted, there is limited information about how well these evaluations accurately identify children with cultural or linguistic diversity. Due to the recognized disparities in ASD identification within racially/ethnically diverse populations, it is possible that the assessment methodologies practitioners are currently using are not appropriate for all racial/ethnic populations due to cultural differences in perceived concerns of parents pertaining to ASD. Few studies exist that evaluate ASD screening and diagnostic assessment measures for their cultural and linguistic responsiveness. Furthermore, few publishers include sufficient data pertaining to racial/ethnic populations when validating and standardizing ASD assessments.

In one study, Harris, Barton, and Albert (2014) evaluated four ASD diagnostic tools and six ASD screeners for their cultural and linguistic responsiveness. After reviewing each tool using a research-based checklist of quality indicators, the authors found extreme variability in culturally and linguistically responsive quality indicators. In particular, the ADOS fared the worst of all tools. Furthermore, screening tools for ASD had higher quality indicators than ASD diagnostic tools. Thus, this study identified the need for improved ASD assessments for CLD populations, particularly diagnostic tools (Harris et al., 2014). It is possible that racially/ethnically diverse populations are misidentified and underidentified with ASD due to the assessment practices that professionals utilize.

The Current Study

One of the more recently developed and frequently utilized rating forms across various settings is the Autism Spectrum Rating Scales (ASRS; Goldstein & Naglieri, 2010). The ASRS measures ASD-related symptoms, behaviors, and features in children aged 2 to 18 years using parent and teacher reports. There are separate forms for children aged 2 to 5 years and 6 to 18 years. Short forms are also available for screening, treatment, and progress monitoring purposes. Raters are asked to read statements asking about their child’s/student’s behavior and determine the frequency on a scale ranging from 0 (never) to 4 (very frequently). Results yield a total score; three subdomain scores, which include Social/Communication (SC; all ages), Unusual Behaviors (UB; all ages), Self-Regulation (SR; ages 6-18 only); and nine treatment scales (Goldstein & Naglieri, 2010).

The current study focuses on the ASRS Parent Form (6-18) and will only report its associated psychometric information. We suggest readers interested in the psychometrics of other ASRS forms consult additional resources (e.g., Goldstein, Naglieri, & Ozonoff, 2009; Goldstein, Naglieri, Rzepa, & Williams, 2012). We chose to focus on parents (vs. teachers) because they frequently serve as reporters in ASD evaluations in both medical/clinical and educational settings. Furthermore, parent reporters are able to provide unique cultural perspectives, which might affect disparities in ASD identification. Finally, parental concern and understanding of ASD may influence the referral, evaluation, and intervention processes. The ASRS Parent Form evidences strong internal consistency (α = .97). Internal consistency for the individual subscales also yield strong reliability with the following coefficients: SC α = .95, UB α = .95, and SR α = .92. The ASRS Parent Form demonstrates strong test–retest reliability (r = .92). Test–retest reliability over a 2- to 4-week interval ranged from r = .90 to .91 for the SR and SC/UB subscales, respectively. The ASRS Parent Form also has strong interrater reliability (r = .89). Subscale interrater reliability coefficients range from .84 to .92 (Goldstein & Naglieri, 2010).

Content validity for the ASRS Parent Form was demonstrated by mapping items onto essential ASD diagnostic criteria according to the Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; DSM-IV-TR; American Psychiatric Association, 2000). Criterion validity was assessed through correlation with three other validated ASD rating forms: the Gilliam Autism Rating Scale–Second Edition (GARS-2; Gilliam, 2006), the Gilliam Asperger’s Disorder Scale (GADS; Gilliam, 2001), and the Childhood Autism Rating Scale (CARS; Schloper, Reichler, Rochen-Renner, 1988). The ASRS Parent Form is strongly correlated with the GARS-2 (r = .63) and moderately correlated with the GADS (r = .54) and CARS (r = .40). No correlations with regard to the ASRS Parent Form and the ADOS-2 or ADI-R were conducted; however, all measures were developed using the core ASD symptomology from the DSM-IV-TR making them conceptually consistent (Goldstein & Naglieri, 2010; Simek & Wahlberg, 2011).

An exploratory factor analysis (EFA) was conducted to determine construct validity. Results from the EFA suggests the ASRS Parent Form is a three-factor structure that measures social communication, unusual behaviors, and self-regulation as seen on the subscales (Goldstein & Naglieri, 2010). Notably, no confirmatory factor analysis (CFA) was conducted to corroborate this result. EFAs provide preliminary information regarding factor structure; however, the process is exploratory in nature, less conservative, and occurs in the earlier stages of measure development. CFAs are generally conducted following EFAs to confirm the EFA results and occur in later stages of measure development (Cabrera-Nguyen, 2010). The authors also conducted several EFAs across race/ethnicity to demonstrate applicability to diverse groups. Although positive that potential differences in measure structure across racial/ethnic groups were discussed, the use of replicating EFAs does not fully address the issue. More in-depth analyses including CFAs and measurement invariance are needed. The purpose of the current project is to conduct the first step in evaluating the structure of the ASRS in a diverse sample of parents of children between the ages of 6 and 18 years in a nonclinical sample.

Method

Participants

Parents (N = 405) with children between the ages of 6 and 18 years without disabilities participated in the current study. Participants self-identified as Black (n = 118), Latinx (n = 106), or White (n = 181). Mean participant age and number of children significantly differed across racial/ethnic groups. White parents (M = 36.9, SD = 7.5) were significantly older than Black (M = 33.6, SD = 4.7) and Latinx (M = 33.2, SD = 6.0) parents. Although the majority of parents within each racial/ethnic group had one child, significant differences still emerged. Similar significant differences were seen in parent income, employment, and child gender. The research regarding the impact of parent demographics on rating form responses is limited and, to the best of our knowledge, has yet to indicate parent age, number of children, or employment impact responses on behavior rating scales. Some studies have found that wealthier mothers are more likely to report fewer symptoms of ASD in their children than mothers with lower incomes (Piper, Gray, Raber, & Birkett, 2014). There is also evidence that parents of females have more concerns about ASD symptoms, particularly related to social communication, than do parents of males (Kirkovski, Enticott, & Fitzgerald, 2013). These differences may affect parent ratings. As such, this sample may be subject to response biases. See Table 1 for detailed participant descriptive information.

Table 1.

Participant Demographic Information Across Independent Racial/Ethnic Subgroups.

	White	Black	Hispanic
	n = 181	n = 118	n = 106	p
Parent’s age				<.001
	36.9 (7.5)	33.6 (4.7)	33.2 (6.0)
Native English speaker				.086
Yes	178 (98.3%)	114 (96.6%)	99 (93.4%)
No	3 (1.7%)	4 (3.4%)	7 (6.6%)
Parent’s gender				.281
Female	92 (50.8%)	53 (44.9%)	44 (41.5%)
Male	89 (49.2%)	65 (55.1%)	62 (58.5%)
Family income				.002
Less than US$20,000	13 (7.2%)	22 (18.6%)	16 (15.1%)
US$20,000-US$34.999	29 (16%)	17 (14.4%)	17 (16%)
US$35,000-US$49,999	27 (14.9%)	23 (19.5%)	23 (21.7%)
US$50,000-US$74,999	46 (25.4%)	21 (17.8%)	26 (24.5%)
US$75,000-US$99,999	26 (14.4%)	25 (21.2%)	16 (15.1%)
More than US$100,000	40 (22.1%)	10 (8.5%)	8 (7.5%)
Parent employment				.007
Employed full time(40 or more hours per week)	141 (77.9%)	98 (83.1%)	89 (84%)
Employed part time(less than 35 hr per week)	25 (13.8%)	18 (15.3%)	17 (16%)
Other	15 (8.3%)	2 (1.7%)	0 (0%)
Parent education				.217
Less than high school	4 (2.2%)	2 (1.7%)	0 (0%)
High school or GED	11 (6.1%)	4 (3.4%)	6 (5.7%)
Some college	32 (17.7%)	14 (11.9%)	7 (6.6%)
Associate’s degree	19 (10.5%)	10 (8.5%)	12 (11.3%)
Bachelor’s degree	84 (46.4%)	69 (58.5%)	62 (58.5%)
Professional degree	31 (17.1%)	19 (16.1%)	19 (17.9%)
Number of children				<.001
1	76 (42%)	80 (67.8%)	59 (55.7%)
2	67 (37%)	31 (26.3%)	44 (41.5%)
3	23 (12.7%)	3 (2.5%)	3 (2.8%)
4 or more	15 (8.3%)	4 (3.4%)	0 (0%)
Child’s age				.06
	9.5 (3.5)	8.6 (2.6)	9.0 (2.9)
Child’s gender				.015
Male	110 (60.8%)	55 (46.6%)	68 (64.2%)
Female	71 (39.2%)	63 (53.4%)	38 (35.8%)

Note. Child age and gender reported for the youngest child with no disability. GED = general educational development.

A comparison of our sample with the ASRS sample used to conduct the EFA is also important. Goldstein and Naglieri (2010) combined the clinical (N = 596) and normative (N = 960) samples for the ASRS 6 to 18 parent form to perform the EFA on which this study was based. The ASRS clinical sample included more males (n = 444, 75%) than females (n = 149, 25%), as was the case in our sample. Our sample included a greater proportion of Black and Latinx individuals, as compared with 85 (14.2%) and 79 (13.3%), respectively, in the ASRS clinical sample. Finally, our sample was slightly more educated than the clinical validation sample. Most parents in the ASRS clinical sample had high school diplomas or completed some college. The majority of our respondents had at least bachelor’s degrees. The additional 960 participants from the ASRS normative sample included in the EFA exhibited similar distributions of race, White (n = 597, 62.2%), Black (n = 137, 14.3%), and Latinx (n = 142, 14.8%). Gender was equally distributed in the normative sample with 480 male (50%) and 480 female respondents (50%). In sum, our sample is more educated than the samples used in the measure’s original development and deliberately includes a greater proportion of Black and Latinx parents.

Procedures

The following procedures were completed after approval from the university institutional review board. Participants were recruited through Amazon’s Mechanical Turk (MTurk). All data were collected electronically on MTurk via Qualtrics. MTurk is a virtual crowdsourcing marketplace where requesters (e.g., researchers, businesses) recruit “workers” (individual people) to complete tasks, such as surveys, online. Requesters create a human intelligence task (HIT) that briefly describes the task. Workers use the MTurk website to find HITs in which they would like to participate. Workers on MTurk are unable to have multiple accounts and are given unique IDs to reduce the likelihood of the same participant completing an HIT multiple times (Chan & Holosko, 2016). Individuals using MTurk can be anyone above the age of 18 years with Internet access.

Overall, MTurk holds promise as a method for researchers to collect high-quality data in an efficient manner (Buhrmester, Kwang, & Gosling, 2011). The low cost, ease of participant recruitment, and accessibility to diverse participants are benefits to using MTurk (Buhrmester et al., 2011; Chan & Holosko, 2016). Research also suggests that MTurk participants attend to tasks comparably to convenience samples and complete interactive tasks similarly to those in traditional research laboratory settings (Hauser & Schwartz, 2016; Thomas & Clifford, 2017). For additional information regarding MTurk, readers are encouraged to visit their website (https://www.mturk.com) or review relevant publications on the use of MTurk in research (e.g., Chan & Holosko, 2016).

Workers interested in our study selected our HIT, consented to participation, and responded to screening questions to ensure they met eligibility criteria (i.e., were at least 18 years old; self-identified as Black, Latinx or Hispanic, or White; and had at least one child between the ages of 6 and 18 without a disability). If eligibility criteria were met, participants completed demographic information about themselves and their youngest child in that age range followed by the ASRS 6 to 18 parent version. To increase the validity of participant responses, attention check (e.g., Berinsky, Margolis, & Sances, 2013) items were randomly inserted in the Qualtrics survey and only MTurk workers with a master’s qualification were able to complete our study. The master’s qualification is awarded to workers who have continual accurate and high-quality work across a variety of tasks and over time. Participants who incorrectly answered these questions were omitted from the sample. Participants were paid US$1 upon completion of the survey.

Data Analysis

Descriptive analyses of participant demographics were first computed. Internal consistency, as captured by Cronbach’s alpha, was analyzed for the three subscales as outlined by the ASRS technical manual (Goldstein & Naglieri, 2010). A CFA was performed using the three-factor subscales for the ASRS. Several model-fitting indices were employed as structural parameters: chi-square test of model fit, comparative fit index (CFI), and the root mean square error of approximation (RMSEA). The CFA was first fit to the combined sample and then to each independent subgroup. Fit indices to determine model fit and significance came from recommendations presented by Pendergast, von der Embse, Kilgus, and Eklund (2017) and Kline (2015). Factor correlations and item analyses were also conducted.

Results

CFA Model Fit

All three subscales exhibited a high degree of internal consistency on the combined sample (α = .968 UB, .934 SR, .947 SC). Results for CFA fit to the combined sample and the three subgroups are presented in Table 2. The combined sample yielded unsatisfactory model fit indices, χ²(1,707) = 7,969.99, p < .001; CFI = .861, RMSEA = .095, p < .001. The White subsample fit slightly better, χ²(1,707) = 3,298.78, p < .001; CFI = .891, RMSEA = .072, p < .001. In contrast, the Black subsample yielded much worse fit, χ²(1,707) = 3,380.04, p < .001; CFI = .887, RMSEA = .091, p < .001. The Latinx subsample also fit the CFA poorly, χ²(1,707) = 3,061.098, p < .001; CFI = .873, RMSEA = .087, p < .001.

Table 2.

Confirmatory Factor Analysis on the Combined Sample and Independent Subsamples by Race.

Model fit information	Independent subsamples
	Combined	White	Black	Latino
	N = 405	n = 181	n = 118	n = 106
Chi-square test of model fit
Chi-square estimate (df = 1,707)	7,969.994	3,298.783	3,380.037	3,061.098
p	<.0001	<.0001	<.0001	<.0001
RMSEA
RMSE estimate	0.095	0.072	0.091	0.087
90% CI	0.093, 0.097	0.068, 0.075	0.087, 0.096	0.082, 0.091
Probability RMSEA ≤ .05	<.0001	<.0001	<.0001	<.0001
CFI	0.861	0.891	0.887	0.873
TLI	0.856	0.886	0.883	0.868
Chi-square test of model fit for the baseline model
Chi-squared estimate (df = 1,770)	46,805.610	16,309.863	16,572.123	12,434.514
p	<.0001	<.0001	<.0001	<.0001
WRMR
WRMR value	2.859	1.729	1.823	1.678
Partial model results	Estimated (SE)	Estimated (SE)	Estimated (SE)	Estimated (SE)
Reverse-worded item: loadings
Item 4 (SC)	−.730 (0.048)***	−.902 (0.084)***	−.153 (0.099)	.858 (0.079)***
Item 5 (SR)	−.329 (0.057)***	−.917 (0.130)***	−.068 (0.095)	.332 (0.067)***
Pairs of latent variables: correlations
SR and UB	.883 (0.012)***	.785 (0.027)***	.946 (0.013)***	.926 (0.014)***
SC and UB	−.258 (0.032)***	−.459 (0.043)***	−.031 (0.063)	.324 (0.058)***
SC and SR	−.161 (0.035)***	−.342 (0.046)***	−.069 (0.071)	.346 (0.061)***

Note: Three-factor CFA fit with 60 ordinal items for a total of 303 free parameters using the robust weighted least squares estimator (WLSMV) estimator utilizing a delta parameterization with probit link. RMSEA = root mean square error of approximation; RMSE = root mean square error; CI = confidence interval; CFI = comparative fit index; TLI = Tucker–Lewis index; WRMR = weighted root mean square residual; SC = Social/Communication; UB = Unusual Behaviors; SR = Self-Regulation; CFA = confirmatory factor analysis.

***

p < .001.

Factor Correlations

When evaluating estimated correlations between latent factors, the combined sample indicated the SR and UB subscales to be significantly and strongly correlated with each other (r = .88), but significantly and slightly negatively correlated with the SC subscale (r = −.26 and −.16, respectively). Again, the White subgroup exhibited similar significant correlations (SR and UB r = .79, SC and UB r = −.46, SC and SR r = −.34). In both the Black and Latinx subgroups, the degree of correlation between the SR and UB subscales was also significant but much stronger (r = .95 and .93). Conversely, the UB and SR subscales were significantly correlated with the SC subscale in the Latinx subsample to a lower degree (r = .32 and .35). Although not significant, these correlations were negative among Black parents.

Item Loadings

The subgroup CFAs also revealed disparities in the loading of two items. Specifically, Items 4 (SC subscale; question inquiries about showing emotion) and 5 (SR subscale; question focuses on following directions), the only reverse-scored items, correlated differently in the various racial subgroups. In the combined sample, both items loaded negatively (−.73 and –.33, respectively) as expected, though Item 5 evidenced a weaker loading on its factor. In the White subsample, both items displayed negative loadings on the respective factors, but to a higher degree (−.90 and −.92). For the Black subsample, however, neither items loaded significantly, although the estimates for the loadings were negative (−.15 and −.07). Both items loaded significantly in the opposite direction within the Latinx subsample (.86 and .33).

Discussion

Combined sample CFA results did not support the three-factor model that was found in the EFA from the initial validation of the ASRS (Goldstein & Naglieri, 2010). These results do not confirm the three-factor model (UB, SR, SC) specific to ASD for a diverse sample. More notably, subsample CFA results differed by race/ethnicity, which also fails to corroborate EFA findings from the initial validation of the ASRS. For White participants, the results suggested limited support for the three-factor model. Conversely, the three-factor model was not supported for Black and Latinx participants. These results indicate that the factor structure of the ASRS may not be comparable across White, Black, and Latinx parents.

A closer examination of factor relations indicated that, for the combined sample, the SR and UB subscales were strongly correlated with one another and negatively correlated with the SC subscale. These results are conceptually logical as it can be expected that the presence of more self-regulation difficulties and unusual behaviors results in poorer social communication skills. When examining subscale relations across subsamples, the same patterns among the SR and UB were evident for White participants. Although the SR and UB subscales were still correlated with one another for Black and Latinx participants, the correlation was notably higher. Correlations to this degree may indicate convergence or two strongly related constructs. Specifically, for Black and Latinx parents, the latent constructs of self-regulation and unusual behaviors may not be distinct or more highly related in comparison with White parents.

When evaluating the relation between the SR and SC subscales and UB and SC subscales across subsamples, the White subsample again performed similarly to the combined sample. Interestingly, the SR and UB subscales were not significantly negatively correlated with the SC subscale for Black participants and were inversely correlated among Latinx participants. In contrast to White parents, it is possible that the construct of unusual behaviors or self-regulation issues may not relate to social communication skills for Black parents. Behaviors that indicate strong social communication skills may have little to do with how Black parents perceive unusual behaviors and self-regulation problems, whereas White parents may view behaviors associated with poor social communication as unusual and indicative of poor self-regulation skills. For Latinx parents, these constructs are positively correlated. Superficially, this relationship appears unusual as it implies that stronger social communication skills are related to more self-regulation problems and unusual behaviors. These results may be better explained by how different cultures define and understand constructs themselves rather than the cultural differences in relationships between constructs. For example, the items used to describe behaviors associated with these three ASRS subscales likely align with the values, behaviors, and beliefs associated with the mainstream culture in the United States and likely best reflect those who identify similarly. The relationship between these subscales is best understood when the items are appropriately measuring the related constructs. Although the relationships between the UB and SR subscales are consistent across groups, the discrepancy in relationships appears to arise from the SC subscale. This may indicate that the items measuring social communication on the ASRS may not accurately reflect how individuals outside of White mainstream culture define and understand the construct. Although presently a hypothesis, the results from analyzing subscale relationships in conjunction with previous research highlighting the cross-cultural differences in social communication and behaviors (Norbury & Sparks, 2013; Zhang et al., 2006) support the idea of that of the three ASRS subdomains, SC may be most unstable across cultures. Further research investigating how cultures define social communication and interaction is needed to best inform the development of culturally responsive ASD rating scales.

Results also yielded differences in specific item loadings, namely, Items 4 (SC, showing emotion) and 5 (SR, following directions). According to the EFA, in the validation of the ASRS (Goldstein & Naglieri, 2010), these items negatively loaded on the SC and SR subscales, respectively. CFA results indicated similar loadings for the combined sample, thus aligning with the EFA. Subsample analyses indicated that White parents showed similar results; however, these items were more negatively loaded. For Black parents, these items did not significantly load on any factor. For Latinx parents, both Items 4 and 5 loaded on to the SC and SR subscales, but positively. It is important to mention that Item 5 showed a positive, but weak loading. These results may suggest that, for White parents, showing less emotion and following instructions more frequently are related to poorer social communication skills and fewer self-regulation problems. For Black families, showing little emotion and not following directions may not be related to social communication skills and self-regulation problems. In contrast, Items 4 and 5 may not contribute to determining social communication and self-regulation for this population. Most interestingly, showing less emotion may be indicative of higher social communication skills for Latinx parents. These results align with previous research indicating cultural differences in the expression and value of emotion (e.g., Tsai, Knutson, & Fung, 2006). Given that these items appear differentially important across racial/ethnic groups, including them in the ASRS for Black and Latinx parent populations may not be appropriate and contribute to incorrect interpretations.

Although more research on the ASRS within racially and ethnically diverse populations is needed, the findings of this study have several implications for professionals. First, it is recommended that professionals consider whether giving the ASRS to certain populations, particularly Black and Latinx parents, will yield beneficial and interpretable results for ASD identification purposes. The results of this study indicate that there are differences by race and ethnicity in parent responses, particularly in regard to social communication, self-regulation, and unusual behavior. Although we need to further investigate these differences in future research, it is likely that cultural beliefs and experiences affect responses on the ASRS. Thus, differential scores on the ASRS by race/ethnicity may affect ASD identification, particularly in Black and Latinx populations. Professionals must use caution when interpreting the ASRS with Black and Latinx populations until more research is conducted with this measure. Given the disparities in ASD identification, particularly among Black and Latinx populations, it is imperative that professionals utilize culturally responsive ASD identification practices. It is recommended that if professionals utilize the ASRS with Black or Latinx populations, that the family is extensively involved and their cultural expectations and beliefs are integral pieces of data, the identification process is multidisciplinary in nature, the assessment process includes multiple data sources, and more information in the areas of social communication, self-regulation, and unusual behavior is obtained from sources other than the ASRS.

Limitations

One limitation of the current study is the inherent restrictions associated with survey methodology. First, participants may not be truly representative of the population due to various issues, including sampling bias. For example, participants self-selected to participate in the study. In addition, the current survey was conducted electronically on MTurk, which limits participants to those having access to the Internet as well as MTurk accounts, thus potentially affecting generalizability of results. Although sample limitations are always a potential in survey research, the use of MTurk may have benefits over other survey data collection strategies (e.g., students on college campuses) as it may result in more diverse samples representative of the broader population. Another limitation is participant response quality. It is a possibility that some participants may have answered items dishonestly or rushed through the survey, thus not answering questions accurately due to inattention. We attempted to control for attention and accuracy concerns by not including participants in the analysis who completed the survey in an unreasonably short amount of time or answered attention questions incorrectly. We also recruited participants who were identified as “Masters” (respondents who are high performing) by MTurk. An additional limitation is that participants were unable to ask clarifying questions about items as the survey was administered online. Another limitation is sample size. Although there is no sample size requirement for CFA, a larger sample would be beneficial during subsample analyses specifically given the number of items (i.e., 60) included in the model. Finally, the current study focused on only three cultural groups—Black, Latinx, and White. Subsequently, results from this study are only applicable to these specific groups.

Future Research

More research is needed in the area of culturally responsive ASD assessment, especially related to measure development and validation. Future research specific to the applicability of the ASRS across diverse populations is also needed. First, replication of this study with more samples collected via other recruitment efforts to further elaborate on our findings is recommended. Second, conducting analyses such as measurement invariance and differential item functioning is recommended to better understand how the ASRS functions across diverse populations. Evaluating the structure of the ASRS in other racial/ethnic groups as well as other cultural groups (e.g., socioeconomic) should also be conducted. Moreover, future research should evaluate how other various cultural factors other than race/ethnicity may account for structural differences in the ASRS. Also, the current study only focused on the ASRS Parent Form (6-18); additional studies examining the structure of the ASRS Parent Form (2-5), Teacher Forms for both age groups, and Short Forms for both parents and teacher respondents across both age in diverse samples is warranted.

It is also important for future research to evaluate how other commonly used ASD rating forms function within and across diverse populations. In addition to specifically evaluating the psychometrics of measures, more research on understanding the diagnostic symptoms of associated behaviors and how they present cross-culturally is also recommended. For example, how do different cultures define symptoms and related behaviors of ASD (e.g., social communication) and how can we best measure them? Knowing conceptual differences may influence the questions included in updated and newly created measures to ensure they are more culturally responsive.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Maryellen Brunson McClain

References

Aiello

Ruble

Esler

(2017). National study of school psychologists’ use of evidence-based assessment in autism spectrum disorder. Journal of Applied School Psychology, 33, 67-88. doi:10.1080/15377903.2016.1236307

Akshoomoff

Corsello

Schmidt

(2006). The role of the Autism Diagnostic Observation Schedule in the assessment of autism spectrum disorders in school and community settings. The California School Psychologist, 11, 7-19.

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text rev.). Washington, DC: Author

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: Author.

Baio

Wiggins

Christensen

D. L.

Maenner

M. J.

Daniels

Warren

. . . Dowling

N. F.

(2018). Prevalence of autism spectrum disorder among children aged 8 years: Autism and developmental disabilities monitoring network, 11 sites, United States, 2014. Mmwrmorbidity and Mortality Weekly Report, 67(6), 2-23.

Berinsky

A. J.

Margolis

M. F.

Sances

M. W.

(2013). Separating the shirkers from the workers? Making sure respondents pay attention on self-administered surveys. American Journal of Political Science, 58, 739-753.

Buhrmester

Kwang

Gosling

S. D.

(2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5.

Cabrera-Nguyen

(2010). Author guidelines for reporting scale development and validation results in the journal of the society for social work and research. Journal of the Society for Social Work and Research, 1, 99-103.

Campbell

J. M.

Ruble

L. A.

Hammond

R. K.

(2014). Comprehensive developmental approach assessment model. In Wilkinson

L.A.

(Ed.), Autism spectrum disorder in children and adolescents evidence-based assessment and intervention in schools (pp. 51-73). Washington, DC: American Psychological Association.

10.

Centers for Disease Control and Prevention. (2006). Mental health in the United States: Parental report of diagnosed autism in children aged 4–17 years—United States, 2003–2004. Morbidity and Mortality Weekly Report, 55, 481-486.

11.

Chan

Holosko

M. J.

(2016). An overview of the use of Mechanical Turk in behavioral sciences; Implications for social work. Research on Social Work Practice, 26, 441-448.

12.

Cubala-Kucharska

(2010). The review of most frequently occurring medical disorders related to etiology of autism and the methods of treatment. Acta Neurobiologiae Experimentalis, 70, 141-146.

13.

Gilliam

J. E.

(2001). Gilliam Asperger’s Diagnostic Scale. Austin, TX: PRO-ED.

14.

Gilliam

J. E.

(2006). Gilliam Autism Rating Scale (GARS-2) (2nd ed.). Austin, TX: PRO-ED.

15.

Goldstein

Naglieri

J. A.

(2010). Autism Spectrum Rating Scales (ASRS): Technical manual. Toronto, Ontario, Canada: Multi-Health Systems Inc.

16.

Goldstein

Naglieri

J. A

Ozonoff

(2009). Assessment of autism spectrum disorders. New York, NY: Guilford Press.

17.

Goldstein

Naglieri

J. A.

Rzepa

Williams

K. M.

(2012). A national study of autistic symptoms in the general population of school-age children and those diagnosed with autism spectrum disorders. Psychology in the Schools, 49, 1001-1016.

18.

Harris

Barton

E. E.

Albert

(2014). Evaluating autism diagnostic and screening tools for cultural and linguistic responsiveness. Journal of Autism and Developmental Disorders, 44, 1275-1287.

19.

Hauser

D. J.

Schwartz

. (2016). Attentive turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48, 400-407.

20.

Institute of Medicine. (2002). Unequal treatment. Washington, DC: National Academies Press.

21.

Issarraras

Matson

J. L.

Matheis

Burns

C. O.

(2018). Differences in developmental concerns of young children with autism spectrum disorder across racial/ethnic groups. Developmental Neurorehabilitation, 22, 174-179.

22.

Johnson

C. P.

Myers

S. M.

(2007). American academy of pediatrics council on children with disabilities: Identification and evaluation of children with autism spectrum disorders. Pediatrics, 120, 1183-1215.

23.

Kirkovski

Enticott

P. G.

Fitzgerald

P. B.

(2013). A review of the role of female gender in autism spectrum disorders. Journal of Autism and Developmental Disorders, 43, 2584-2603.

24.

Kline

R. B.

(2015) Principles and practice of structural equation modeling (4th ed., ProQuest Ebook Central). Guilford Publications. Retrieved from https://ebookcentral-proquest-com.dist.lib.usu.edu/lib/usu/detail.action?docID=4000663

25.

Le Couteur

Lord

Rutter

. (2003). Autism diagnostic interview-revised (ADI-R). Los Angeles, CA: Western Psychological Services.

26.

Lord

Rutter

DiLavore

Risi

Gotham

Boshop

S. L.

(2012). Autism diagnostic observation schedule: (ADOS-2) manual (Part 1): Modules 1-4 (2nd ed.). Torrance, CA: Western Psychological Services.

27.

Mandell

D. S.

Ittenbach

R. F.

Levy

S. E.

Pinto-Martin

J. A.

(2007). Disparities in diagnoses received prior to a diagnosis of autism spectrum disorder. Journal of Autism and Developmental Disorders, 37, 1795-1802.

28.

Mandell

D. S.

Listerud

Levy

S. E.

Pinto-Martin

J. A.

(2002). Race differences in the age at diagnosis among Medicaid-eligible children with autism. Journal of the American Academy of Child & Adolescent Psychiatry, 41, 1447-1453.

29.

Norbury

C. F.

Sparks

(2013). Difference or disorder? Cultural issues in understanding neurodevelopmental disorders. Developmental Psychology, 49, 45-58.

30.

Ozonoff

Goodlin-Jones

B. L.

Solomon

(2005). Evidence-based assessment of autism spectrum disorders in children and adolescents. Journal of Clinical Child and Adolescent Psychology, 34, 523-540.

31.

Pendergast

L. L.

von der Embse

Kilgus

S. P.

Eklund

K. R.

(2017). Measurement equivalence: A non-technical primer on categorical multi-group confirmatory factor analysis in school psychology. Journal of School Psychology, 60, 65-82.

32.

Piper

B. J.

Gray

H. M.

Raber

Birkett

M. A.

(2014). Reliability and validity of Brief Problem Monitor, an abbreviated form of the Child Behavior Checklist. Psychiatry and Clinical Neurosciences, 68, 759-767.

33.

Schloper

Reichler

Rochen-Renner

(1988). The Childhood Autism Rating Scale. Torrance, CA: Western Psychological Services.

34.

Simek

A. N.

Wahlberg

A. C

. (2011). Test review: Autism Spectrum Rating Scales. Journal of Psychoeducational Assessment, 29, 191-195.

35.

Simonoff

Pickles

Charman

Chandler

Loucas

Baird

(2008). Psychiatric disorders in children with autism spectrum disorders: Prevalence, comorbidity, and associated factors in a population-derived sample. Journal of the American Academy of Child & Adolescent Psychiatry, 47, 921-929. doi:10.1097/CHI.0b013e318179964f

36.

Sullivan

A. L.

(2013). School-based autism identification: Prevalence, racial disparities, and systemic correlates. School Psychology Review, 42, 298-316.

37.

Thomas

K. A.

Clifford

(2017). Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments. Computers in Human Behavior, 77, 184-197.

38.

Tsai

J. L.

Knutson

Fung

H. H.

(2006). Cultural variation in affect valuation. Journal of Personality and Social Psychology, 90, 288-307.

39.

Zhang

Wheeler

J. J.

Richey

(2006). Cultural validity is assessment instruments for children with autism from a Chinese cultural perspective. International Journal of Special Education, 21, 109-113.

40.

Zwaigenbaum

Bauman

M. L.

Choueiri

Kasari

Carter

Granpeesheh

. . . Natowicz

M. R.

(2015). Early intervention for children with autism spectrum disorder under 3 years of age: Recommendations for practice and research. Pediatrics, 136, 60-81.