Psychometric properties of questionnaires and diagnostic measures for autism spectrum disorders in adults: A systematic review

Autism-Spectrum Quotient (AQ)

The evidence for the psychometric properties of the AQ published since the NICE (2014) guidelines and covered in the current review is not consistent, varying according to study design and recruitment source (Baron-Cohen et al., 2001). The sensitivity and specificity of the AQ-50 and the AQ-10 were good (⩾80%) when comparing archival clinical data from adults with ASD, against a general population group (Booth et al., 2013). However, this is not equivalent to the population of subjects presenting to ASD diagnostic assessment settings. In such settings, case recognition and the need to consider both differential diagnosis and the identification of additional mental health problems can be challenging.

In a Japanese study (Nishiyama et al., 2014) recruiting individuals with a known diagnosis of ASD, and a comparison general population sample, the internal consistency and construct validity of short versions of the AQ were inconsistent. A strength of the study was that a battery of screening measures were administered to all participants, including the general population comparator group, to identify any co-occurring mental health conditions.

A further study recruiting individuals with schizophrenia (Lugnegard et al., 2015) from a psychiatric clinic, adults with ASD and individuals from the general population, found that ASD and schizophrenia were both significantly associated with a higher AQ-50 score. In addition, only the AQ version with a binary response option discriminated reliably between ASD and schizophrenia and only on one subscale (attention switching; Lugnegard et al., 2015). There was no significant difference in AQ-50 scores between individuals with ASD and those with schizophrenia when using the 4-point response option. Participants were diagnosed prior to the study, though diagnoses of ASD and schizophrenia were confirmed in this study using the DISCO-11 (Wing et al., 2002) and the SCID-1 (Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders, 4th ed.; DSM-IV; First and Gibbon, 2004), respectively. However, such purposive recruitment of individuals with a defined specific diagnosis is not comparable to the use of the AQ in ASD diagnostic assessment clinical settings.

In the fourth article (Sizoo et al., 2015), individuals attending an ASD diagnostic clinic completed abbreviated versions of the AQ prior to a clinical assessment. The comparison group again consisted of individuals from the general population. Reported specificity (more than 66%) and sensitivity (57%–62%) were lower than in the Booth et al. (2013) study. A strength of the study was the recruitment of participants from ASD clinics and administration of the AQ prior to their full assessment, which is similar to procedures in clinical settings.

Finally, Ashwood et al. (2016) reported findings using the subject and informant versions of the AQ-50 and the AQ-10. Participants were consecutively referred to an ASD assessment clinic and had high rates of comorbid mental health conditions. Across both AQ versions, sensitivity was above 71% but specificity was less than 38%, and the relationship between the AQ scores and the Autism Diagnostic Interview–Revised (ADI-R) and the Autism Diagnostic Observation Schedule–Generic (ADOS-G) was weak. Thus, the AQ did not reliably identify those individuals who did not have ASD. The presence of mental health conditions (such as a generalised anxiety disorder) also increased the risk of false-positive ASD diagnoses using the AQ-10 (Ashwood et al., 2016). This study highlights the limitations of the AQ in a more ecologically valid setting compared to studies using non-clinical general population comparison groups.

Thus, overall, the findings from these studies suggest that due to low levels of specificity, the AQ is not a reliable indicator of which people should progress to a full ASD assessment.

Ritvo Autism Asperger Diagnostic Scale–Revised

When the Ritvo Autism Asperger Diagnostic Scale–Revised (RAADS-R) was used in an ASD assessment service (Sizoo et al., 2015) and compared to a general population group, sensitivity was good (73%) but specificity was low (58%) (Ritvo et al., 2011). This indicated some problems accurately identifying those without ASD (Sizoo et al., 2015). Limitations in this study are similar to those reporting on the AQ, including the use of a case–control design and purposive recruitment of a general population comparison group that lacked mental health or developmental assessment. A study strength was that participants were recruited from ASD assessment services and administered the RAADS-R prior to full ASD assessment. The low levels of specificity reported suggest that the RAADS-R was not a reliable indicator of those who should progress to a full ASD assessment.

The psychometric properties of a 14-item version (the RAADS-14) were tested by Eriksson et al. (2013) in an ASD group, a clinical control group (individuals with ADHD, mood, psychotic and borderline personality disorders) and a general population group. The ASD group had significantly higher scores compared with the clinical control group. However, although both sensitivity (97%) and specificity (95%) were good with general population controls, the specificity for comparison with the clinical control group was reduced (to 46% with ADHD; Eriksson et al., 2013). Comparison to a psychiatric control group was clinically relevant, but study limitations again included case–control design and the lack of a consistent use of ASD and/or other mental health assessments in the control group; in addition, some participants were recruited online and for these individuals diagnoses were self-reported and not confirmed.

Social Responsiveness Scale 2

Full and shortened versions of the adult self-report Social Responsiveness Scale 2 (SRS-2) were translated into Japanese (Nishiyama et al., 2014) and completed by individuals recruited from the general population and from clinical services (Constantino and Gruber, 2005). These versions had good internal consistency but were correlated with symptoms of psychoticism and distress as measured by the Brief Symptom Inventory (BSI; Derogatis and Melisaratos, 1983) and the K10 (Kessler et al., 2002). A strength of the study was that the general population group was administered a battery of mental health screening measures; a limitation was diagnosis of participants prior to the study.

A Japanese translation of the SRS informant version (Takei et al., 2014) demonstrated good sensitivity in both genders, and good specificity in men with ASD, compared to individuals in a control group with mental health difficulties. However, specificity in women was reduced (61%) indicating limitations in identifying women who did not have ASD (Takei et al., 2014). A strength of this study was confirmation of diagnosis by the research team; a limitation was that only individuals in the ASD group completed a reference standard (the ADOS), and the control group had no clinical assessment to exclude the presence of ASD.

Social Communication Questionnaire

The Social Communication Questionnaire (SCQ; German version) was completed by parents or professional carers (in the majority of cases), of adults with intellectual disability and suspected ASD as part of routine clinical assessments of consecutive admissions to an intellectual disability mental health service (Berument et al., 1999; Sappok et al., 2015a). Interestingly, the SCQ current version (assesses the last 3 months) was found to be more effective than the lifetime version; the best combination of sensitivity (89%) and specificity (66%) was achieved by raising the SCQ cut-score for ASD from 15 to 18. When the focus was on lifetime development, specificity decreased to 48%. This finding may have been because the SCQ was completed by a professional carer who may not have known about a subject’s early development, rather than a family member. About 87% of included participants had a moderate/severe intellectual disability, so the findings may not generalise to adults with milder intellectual disability (Sappok et al., 2015a). A strength of the study was that no exclusions were made on grounds of neurodevelopmental or mental health comorbidities, increasing the clinical validity of the sample.

Cross-cultural validity of the SCQ current version was examined in a study recruiting adults with intellectual disability from mental health services in Germany, the United Kingdom and the United States (Sappok et al., 2017). In contrast to the previous study (Sappok et al., 2015a), a lower cut-score of 13 was found optimal and made for sensitivity of 87% and specificity of 58%. More severe levels of intellectual disability were associated with higher scores. In some cases, administration of the SCQ was prior to an ASD assessment so informants completing the measure were blind to diagnosis; however, variations in administration of the SCQ in different countries may have introduced bias. SCQ scores were influenced by gender and country with males and those recruited in Germany having higher scores.

Social Communication Questionnaire for Adults with Intellectual Disabilities

International data (from Germany, the United States and United Kingdom) were collected from individuals recruited via intellectual disability mental health settings with the SCQ for Adults with Intellectual Disabilities (SCQ-AID) completed by carers or researchers (Derks et al., 2017). The new algorithm with a cut-point of nine demonstrated sensitivity of 81% suggesting the shortened version has clinical utility though as found by Sappok et al. (2015a, who used the original version) specificity was low (62%). A strength of the study was the international recruitment of participants; however, the authors noted some differences in diagnostic process across sites. Further independent evaluation of this new measure is needed.

The Diagnostic Behavioural Assessment for ASD–Revised

When the Diagnostic Behavioural Assessment for ASD–Revised (DiBAS-R) was completed by a relative or staff member supporting an adult accessing an intellectual disability service, rates of 81% sensitivity and specificity were found when comparing those with and without ASD (Sappok et al., 2014, 2015b). These findings using this new measure are encouraging, and a strength of the study is that diagnoses were made contemporaneously therefore replicating circumstances from clinical settings. Further evaluation in other populations of adults with intellectual disability is required.

In a study of consecutive referrals to a specialist intellectual disability psychiatric clinic in Germany, DiBAS-R sensitivity was 82% and specificity 67%. Diagnostic accuracy was better in adults with mild/moderate intellectual disability than those with severe intellectual disability (for whom specificity decreased to 34%; Heinrich et al., 2017). A strength of the study was that the DiBAS-R was completed by carers, prior to a full diagnostic assessment, and was therefore representative of the clinical process.

One study used a combination of the DiBAS-R and the Autism Checklist (ACL) and is described below (Mutsaerts et al., 2016).

Autism Checklist (ACL)

In a psychiatric clinic for adults with intellectual disability, Mutsaerts et al. (2016) reported that combined use of the ACL and the DiBAS-R led to improved sensitivity of 95% (when there was a positive screen on at least one measure) compared to when each measure was used alone (91%/75%, respectively; Sipes and Matson, 2014). Combined use also led to an increase in specificity to 88% (when there were positive screens on both measures) compared to using each measure separately (75%; Mutsaerts et al., 2016). The inclusion of data collected from people referred to clinic was a study strength.

Autism Diagnostic Interview-Revised (ADI-R)

The ADI-R was evaluated with adults without intellectual disability who were consecutively referred (mostly by self or a relative) to an ASD diagnosis clinic in Italy (Fusar-Poli et al., 2017; Lord et al., 1994). Overall, the ADI-R had low sensitivity (43%) but good specificity (95%). In contrast, across the subscale domains sensitivity was acceptable (more than 60%) and specificity was good (more than 79%), with the exception of the restricted, repetitive and stereotyped behaviours domain (just 37%).

When the ADI-R and ADOS-G were combined, sensitivity was still low (42%). The authors noted that the ADI-R may work less well with higher-functioning adults because a significant number of items rely on developmental history, and so more subtle presentations later in life may be missed (Fusar-Poli et al., 2017). It was a strength of the study that those administering the ADI-R were blind to DSM-5-based clinical consensus and that a third of participants already had a psychiatric diagnosis.

Autism Diagnostic Observation Schedule (second edition) Module 4 revised algorithm

Hus and Lord (2014) reported sensitivity and specificity of the new revised Module 4 algorithm to be above 80% in a study including previous research participants and individuals presenting to a developmental disabilities clinic (the control group comprised individuals with a variety of Diagnostic and Statistical Manual of Mental Disorders (4th ed.; text rev.; DSM-IV-TR disorders (Lord et al., 1999, 2012). The severity scores were mostly independent of verbal IQ and race. A limitation of the study was the retrospective design, and exclusion of some individuals with a relative with ASD, so these new algorithms do require replication in other groups.

In a Dutch study of individuals with ASD and three different control groups, De Bildt et al. (2016) reported a sensitivity of 61% for the ADOS Module 4. However, specificity varied according to the comparator group: 95%–100% for individuals recruited from non-clinical or forensic settings, compared to 22%–50% for individuals with schizophrenia (De Bildt et al., 2016). Items that were endorsed by the ASD group but not by individuals with schizophrenia included ‘quality of rapport’, ‘conversation’, ‘quality of social responses’ and ‘highly specific topics’. Limitations in the study design included the small sample size and the use of archival video data with individuals being diagnosed prior to the study.

In the third study, Pugliese et al. (2015) investigated individuals referred for an ASD assessment and previous research participants (many of whom met criteria for a range of diagnoses) using the Autism Diagnostic Observation Schedule (second edition) (ADOS-2) Module 4 and a cut-point of 8. The study found that while sensitivity was 85% (80%–89% across research sites), specificity was somewhat lower at 72% (62%–91%). The results were best for women and those with above or below average verbal IQ. Limitations in the study design included variability in the reference standards used to confirm a diagnosis of ASD and omission of any description of blinding during administration of the tests.

Langmann et al. (2017) investigated the performance of the original and revised ADOS-2 Module 4 algorithms using retrospective data. Participants were adults and adolescents with suspected ASD who had been referred to specialist ASD diagnostic clinics in psychiatry services. The revised algorithm demonstrated slightly better performance overall (sensitivity 86% and specificity 80% at cut-point 8) compared to the original algorithm. However, specificity for adults with a diagnosis of personality disorder compared to ASD was poor (individuals with personality disorder comprised 58% of false-positive cases). In addition, diagnostic accuracy was less for women and older individuals. However, a possible limitation to the generalisability of these findings was that less than a quarter of those recruited were female and only 25% were over age 24 years.

Fusar-Poli et al. (2017) evaluated the ADOS-2 Module 4 with adults whose IQ was more than 70 and who were consecutively referred to an ASD diagnosis clinic in Italy. The majority of participants were self or relative referrals. The original ADOS-2 Module 4 algorithm (Lord et al., 2012) had sensitivity of more than 86% across all domains. Sensitivity of the revised algorithm (Hus and Lord, 2014) was 87%. Specificity was 83% for the original algorithm but reduced on individual domains (63%) and was 74% on the revised algorithm. Strengths of the study included the avoidance of a case–control design, and that those administering the ADOS-2 were blind to DSM-5-based clinical consensus. In addition, and as would be expected in a clinical setting, a proportion of those recruited had existing psychiatric diagnoses (37%).

Maddox et al. (2017) found the original and revised algorithms for the ADOS-2 Module 4 (Hus and Lord, 2014; Lord et al., 2012) had a sensitivity of 100%, while specificity was 74% and 70%, respectively. A strength of the study was recruitment of participants from psychiatric settings; however, study findings indicated a high number of participants with psychosis falsely met criteria for ASD and that the social-communication domain was particularly limited in discriminating between the two.

Autism Mental State Examination

The Autism Mental State Examination (AMSE) was reported to have good sensitivity and specificity for discriminating ASD from other disorders using the ADOS-G or the ADI-R (Grodberg et al., 2012, 2014). Strengths of the study design included blinded administration of the measure and the consecutive recruitment of individuals who self-referred for assessment of ASD.

Structured Questionnaires (SQs)

The new information reported compliments the overall NICE (2014) recommendations regarding SQs. Most SQs show limited utility in identifying those who did not have ASD (Ashwood et al., 2016). This finding was evident in studies recruiting participants in clinically realistic settings (e.g. Ashwood et al., 2016) compared to studies using a case–control design with a general population comparator (e.g. Booth et al., 2013). There were particular problems differentiating ASD from schizophrenia, which may be in part a consequence of the potential overlap in symptoms (e.g. flattened affect and aspects of social communication) and/or underlying genetic phenotype (Cross-Disorder Group of the Psychiatric Genomics, 2013). Given the low specificity, the AQ and the RAADS-R should not in our opinion be used on their own for screening and case recognition in clinical settings. Based on published data, this would apply to all versions of the AQ including the abbreviated 10-item version recommended by NICE (2014). Regarding the SRS, at the time of the NICE (2014) recommendations there were no diagnostic accuracy studies that met guideline inclusion criteria. New evidence suggests that the accuracy of the SRS (Japanese translation) for adults may be limited in women, and in the presence of symptoms of psychoticism or distress as measured using screening tools (Nishiyama et al., 2014; Takei et al., 2014). Evidence for the diagnostic accuracy of the SCQ with adults was not available previously. In this review, the SCQ was found to be more effective when using a higher cut-off score in a German cohort (Sappok et al., 2015a). However, when an international cohort was recruited (Germany, United States and the United Kingdom), a lower cut-score improved diagnostic accuracy (Sappok et al., 2017). Where professional carers completed the current SCQ (without reference to developmental history) for adults with intellectual disability in a clinical setting, this showed better sensitivity and specificity than the lifetime version (Sappok et al., 2015a). This highlights the potential difficulty of gathering a developmental history to support the diagnosis of ASD in adults, which has implications for the choice of diagnostic tools used. A new and shortened version (the SCQ-AID) used in an international study had good sensitivity but low specificity; however, the authors note some variation in diagnostic rates and processes across countries (Derks et al., 2017). This review found no new evidence regarding the Asperger Syndrome Diagnostic Interview (ASDI) or the Adult Asperger Assessment (AAA).

Diagnostic Measures (DMs)

Studies assessing the psychometric properties of DMs (such as the ADOS-2 Module 4) reported some success identifying ASD among clinic referrals though limitations remained. The ADOS-G and ADI were recommended by NICE for use with adults with and without intellectual disability. New evidence shows that the new ADOS-G algorithms had good sensitivity and specificity in a study with a case–control design (Hus and Lord, 2014). This was to a certain extent replicated (specificity was acceptable) in individuals consecutively referred to a psychiatric clinic with a variety of diagnoses (Fusar-Poli et al., 2017). However, new studies also indicated that in clinical groups, specificity was reduced for schizophrenia, personality disorder and psychosis (De Bildt et al., 2016; Langmann et al., 2017; Maddox et al., 2017). Scores were additionally found to interact with gender and IQ although further studies are required to replicate these findings before definite conclusions can be made (Langmann et al., 2017; Pugliese et al., 2015). The ADI-R demonstrated mixed results in adult ASD clinic referrals, having good specificity but low sensitivity (Fusar-Poli et al., 2017). The AMSE showed promising evidence as a new measure; further studies in other settings and by teams independent of the authors are required (Grodberg et al., 2014).

Combinations of tools

NICE (2014) recommended using a battery of tools to support the diagnostic process. This was based on evidence from studies using SQs and DMs in isolation, and at the time the guidelines were published there were no diagnostic accuracy studies regarding use of tools in combination. In this review, only two studies evaluated the use of a combination of tools. In adults with ASD and intellectual disability, using the ACL and the DiBAS-R improved the sensitivity and specificity of these tools (Mutsaerts et al., 2016). For adults without an intellectual disability, combining the ADI-R and the ADOS-G improved specificity but reduced sensitivity (Fusar-Poli et al., 2017). Sensitivity of the combined measures was lower than to using the ADOS-G alone; the authors suggest that the ADI-R may be less sensitive to the more subtle presentation of ASDs diagnosed in adulthood (Fusar-Poli et al., 2017). This new evidence whereby combined use of these measures improved diagnostic effectiveness is in accordance with NICE (2014) recommendations regarding a battery of measures (Mutsaerts et al., 2016).