Abstract
We aimed to adapt the Modified Checklist for Autism in Toddlers to Turkish culture. The Modified Checklist for Autism in Toddlers was filled out independently by 191 parents while they were waiting for the well-child examination of their child. A high screen-positive rate was found. Because of this high false-positive rate, a second study was done in which the Modified Checklist for Autism in Toddlers was administered by health-care staff in a short interview with two groups of parents. The first group (the high-risk group) comprised 80 children aged 18–36 months, who were initially diagnosed with pervasive developmental disorders. The second group (the low-risk group) comprised 538 children of the same age, who were followed regularly by the well-child clinic. Two screen positives were found in the low-risk group. These two children, a random sample of 120 children from the low-risk group, and all the high-risk group were invited to a clinical evaluation. The diagnostic power of the Modified Checklist for Autism in Toddlers was assessed against clinical diagnosis and the Childhood Autism Rating Scale. The positive predictive value of the Modified Checklist for Autism in Toddlers was found to be 75%. Our findings led us to conclude that the Modified Checklist for Autism in Toddlers is a useful tool in Turkey for screening of pervasive developmental disorders in primary care, but in our culture, it is completed more accurately when health-care personnel ask the parents the questions. This study shows that Modified Checklist for Autism in Toddlers screening should be adapted based on culture and setting.
Introduction
Autistic Disorder is a lifelong neurodevelopmental disorder classified under the umbrella of pervasive developmental disorders that can be diagnosed during early childhood. It is characterized by impairments in communication and socialization skills, restricted interests, and repetitive behaviors (APA, 2000). The prevalence of autistic disorder reported in many studies in the last decade has been estimated to be about 20 in 10,000 (Gillberg et al., 2006; Nassar et al., 2009), and the estimated rate for all the autism spectrum disorders (ASD) is about 1 in 88 (Centers for Disease Control and Prevention (CDC), 2012). A recent study conducted in a South Korean community (Kim et al., 2011), which included 55,266 children aged 7–12 years, reported a prevalence rate of 2.64%, which is higher than previous studies. Long-term follow-up studies on ASD show an unfavorable outcome. Besides carrying the core symptoms of autism into adulthood, most of these individuals have additional mental health problems (Cederlund et al., 2008; Hofvander et al., 2009).
Available data suggest that with earlier detection, the outcome of autism is gradually improving; for example, more individuals are able to live independently. Research focused on early diagnosis and early manifestations of autism symptomatology has increased in recent years (Coonrod and Stone, 2005). Some screening measures have been developed to identify children at risk and to refer them for diagnostic evaluations. These screening measures involve two approaches to identify children at risk of autism within the general population. The first approach is to use nonspecific screening measures to identify children at risk of a broad range of developmental problems. The second approach is to use a screening test that specifically targets the symptoms of autism (Hofvander et al., 2009). However, in Turkey, none of these screening methods are being used and the prevalence of Autistic Disorder has not yet been studied.
The Modified Checklist for Autism in Toddlers (M-CHAT) is one of the specific measures designed for use in pediatric settings to identify toddlers at risk of autism (Robins et al., 2001). The psychometric properties of this instrument have been examined in several languages and cultures; however, there is no study investigating the use of M-CHAT in Turkish. Therefore, the aim of this study was to adapt the M-CHAT to Turkish culture using a sample aged 18–36 months from Istanbul.
Materials and methods
Description of the M-CHAT
The M-CHAT was designed as a self-administered, parent questionnaire for regular pediatric visits to screen for autism in toddlers (Robins et al., 2001). The M-CHAT includes 23 “yes” or “no” items. The “yes” response is normal for 19 items, but is abnormal for item numbers 11, 18, 20, and 22. A child is considered as screen positive at the initial screening if he or she has abnormal answers for 2 of the 6 critical items or 3 of any 23 items. The critical items on the M-CHAT are as follows: item 2 (interest in other children), item 7 (proto-declarative pointing), item 9 (bringing objects to show the parent), item 13 (imitating), item 14 (responding to name), and item 15 (following a point). If the child screens positive, it does not constitute a diagnosis but indicates significant risk of autism, suggesting the need for evaluation with a gold standard test to diagnose autism. The M-CHAT was originally validated for children between 16 and 30 months of age, but many studies have used an upper age limit of 36 months or more (Canal-Bedia et al., 2011; Snow and Lecavalier, 2008).
Setting
This study was carried out at the Department of Pediatrics and Department of Child and Adolescent Psychiatry. The Well-Child and the Child Neurology Divisions of the Department of Pediatrics and the Autism Clinic of the Department of Child and Adolescent Psychiatry conducted this study. The well-child clinic follows up children between the ages of 0 and 6 years in Istanbul. Although it is part of a university clinic, it serves as a community-based clinic and accepts all referrals. It is not a clinic where only children who are at risk are being followed. Therefore, the population of this clinic represents the general population. Follow-up of physical development, nutrition (breastfeeding consulting, vitamin and mineral supplementation), vaccination and screening (phenylketonuria, hypothyroidism, biotinidase deficiency, eye problems, hearing, etc.) is the main focus of this clinic. This study protocol was approved by the Ethics Committee of our university. The M-CHAT was translated and back translated to Turkish.
Study 1
The M-CHAT questionnaire was filled out by the mother and/or father while they were waiting for the well-child consultation for child aged 18–36 months (n = 191), as in the original study (Robins et al., 2001). The completed forms were scored by the first author (B.K.). Children who screened positive on the M-CHAT (n = 94; 49.2%) were contacted by telephone by health-care staff; 48 could be reached and received a follow-up telephone call to review their answers. Of these 48 children, 42 passed the M-CHAT during the telephone interview. Since the rate of false positives (42 out of 48) was much higher than previous studies with the M-CHAT, we decided to change our method.
Study 2
The previous forms were invalidated and the health-care staff (a nurse and a psychologist) were shown by the study group how to complete the questionnaire while interviewing the parents. The interviews were standardized. All the interviews were performed by the same health-care staff and at the same place. The questions in the M-CHAT were directed to the parents by health-care staff in a one-to-one interview, with explanations about the questions when necessary. This interview took about 5–7 min, about the same time as the self-administered form. There was no telephone follow-up.
Participants
The questionnaire was administered to two groups. The first group (the high-risk group) comprised 80 children aged 18–36 months, who were initially referred to the child neurology department due to social–communicative impairment and diagnosed by a pediatric neurologist with a probable diagnosis of pervasive developmental disorders. This group was waiting for a final diagnostic interview with an autism expert. The second group (the low-risk group) comprised 538 children in the same age range without any known developmental and physical health problems attending the well-child clinic for regular visits. This group did not include the previous 191 individuals. The M-CHAT was administered to all children in both groups aged between 18 and 36 months. Children who had a physical disability or clear mental retardation and children who attended without parents or primary caregivers were excluded from the study to decrease the high false positivity of the M-CHAT.
The health-care staff received the answers from the parents for all 23 test items. Parents were told that they had to answer all the questions as “yes” or “no,” and if they were unsure about the answer, it was explained to them that they should choose the most appropriate answer for their child. The health-care staff first read the items, and if the parents could not adequately understand the questions, the staff member explained further.
The M-CHAT questionnaires were scored by the first author (B.K.). All available subjects in the high-risk group (n = 60) and a random sample of the low-risk group (120 subjects) were invited to a clinical evaluation. All families who were invited to the clinical evaluation accepted the offer of evaluation. In addition to applying Diagnostic and Statistical Manual of Mental Disorders (4th ed., text revision; DSM-IV-TR) criteria, the Childhood Autism Rating Scale (CARS) was applied to measure the severity of autistic symptoms. CARS is the sole instrument for measuring the severity of autistic symptoms that has been validated in Turkish (İncekas et al., 2009). The clinical evaluation was done at least 6 months after the M-CHAT was completed, when the children were at least 2 years of age. Although the clinical evaluation was done 6 months after the M-CHAT, these children were in routine follow-up in the child neurology clinic and were referred to educational programs by their own physicians.
The clinical evaluation and completion of CARS were done by three clinicians: one child neurologist and two child psychiatrists under the supervision of the second author (N.M.M), who has 20 years of experience in child psychiatry and is a founder of the autism clinic in our university. The results of the M-CHAT were compared with the results of the CARS and the diagnostic evaluation and the positive predictive value (PPV) of the M-CHAT our study group was calculated.
Statistics
Data were analyzed using SPSS (10.0) for Windows, and statistical significance was assessed using the chi-squared test and analysis of variance (ANOVA). Screen-positive response rates in different groups were analyzed by comparing two independent proportions (Saunders and Trapp, 1990).
Results
Study 1
Ninety-four of 191 (49.2%) children screened positive on the M-CHAT, and only 48 of 94 (44.7%) children could be reached by telephone follow-up to confirm the test results. Of these 48 children, 42 passed the M-CHAT during the telephone interview. Six children (12.5%) screened positive after the telephone follow-up. We referred these six children to a child psychiatrist, but could not follow them up further.
Study 2
The algorithm of the study is presented in Figure 1. There were 292 boys (54.3%) and 246 girls (45.7%) in the low-risk group and 60 boys (75%) and 20 girls (25%) in the high-risk group. The high percentage of boys in the high-risk group was statistically significant (p < 0.001). The mean age of the children was 27.15 ± 5.05 months (mean ± SD) in the low-risk group and 23.94 ± 5.14 months in the high-risk group (p < 0.001) (Table 1).
Algorithm of the study (clinical evaluation was done using CARS and DSM-IV-TR).
Demographic characteristics of the participants in study 2.
SD: standard deviation.
All children (80/80) in the high-risk group screened positive on the M-CHAT. Two of 538 children in the low-risk group, one 19-month-old girl and one 18-month-old boy, screened positive on the M-CHAT. The girl had 10 abnormal responses (items 1, 5, 6, 7, 8, 9, 15, 19, 20, 21; abnormal response to three critical items), and she was diagnosed with Rett syndrome; the boy had six abnormal responses (items 5, 6, 7, 14, 19, 22; abnormal response to two critical items), and he had the diagnosis of Autistic Disorder at his follow-up evaluation.
Clinical evaluation based on CARS and DSM-IV-TR diagnostic criteria was done with 60 children from the high-risk group (20 caregivers from the high-risk group could not be reached) and 120 children randomly selected from the low-risk group (n = 536). Two subjects with a diagnosis of Autistic Disorder and Rett syndrome, who were detected during the M-CHAT screening, were excluded from the random selection (Figure 1). The mean CARS score for all children from the high-risk group (n = 60) was significantly higher than that of the children from the low-risk group (n = 120) (p < 0.001). Of the 60 children from the high-risk group, 45 met the diagnosis of Autistic Disorder according to the DSM-IV-TR criteria with a total CARS score ≥ 30; and 15 children did not have the diagnosis of ASD according to DSM-IV-TR criteria with a total CARS score < 25. These 15 children had some clinical concerns, such as language delay, but all were out of the autism spectrum. The improvement in this group could be related to either early intervention or diagnostic overlap/misdiagnosis at the first visit. From our low-risk group (n = 120), only two children showed developmental problems; one girl and one boy with CARS scores of 29.5 and 29, respectively. When the M-CHAT questions were administered to them at 18 months of age, they each had two abnormal responses (none of the six critical items). They had a history of developmental regression after the age of 18 months; the girl was diagnosed with pervasive developmental disorder–not otherwise specified (PDD-NOS) at 2 years of age, and the boy was diagnosed with PDD-NOS at the age of 30 months. The results of the M-CHAT and CARS tests were compared, and the diagnostic power of the M-CHAT score was evaluated against a CARS cutoff of 25, to distinguish ASD from the non-ASD population (Chlebowski et al., 2010). The PPV of the M-CHAT was 75% for our study group (Table 2).
The diagnostic power of the M-CHAT test in Turkish compared with CARS test and DSM-IV-TR criteria for autism.
M-CHAT: Modified Checklist for Autism in Toddlers; CARS: Childhood Autism Rating Scale; DSM-IV-TR: Diagnostic and Statistical Manual of Mental Disorders (4th ed., text revision).
The distribution of abnormal responses to the M-CHAT items is shown in Table 3. The abnormal response rate was above 50% for questions 2, 5–9, 11, 14, 15, and 17–23 and below 20% for questions 1, 3, 12, and 16 in the high-risk group. The results for the critical items in the M-CHAT for the high-risk group in relation to the results of the CARS test are presented in Table 4.
The distribution of abnormal responses to the M-CHAT in low- and high-risk groups when the questionnaire was completed by the health-care staff interviewing the parent.
A: abnormal response rate to the test item in the high-risk group with CARS score > 30; B: abnormal response rate to the test item in the low-risk group; C: abnormal response rate to the test item in the high-risk group with CARS score < 30. NS: nonsignificant; M-CHAT: Modified Checklist for Autism in Toddlers; CARS: Childhood Autism Rating Scale.
Comparing two independent proportions.
Distribution of the abnormal responses for the six critical questions of the M-CHAT in the high-risk group who failed or passed the CARS.
A: abnormal response rate to test item in the high-risk group with CARS score > 30; B: abnormal response rate to test item in the high-risk group with CARS score < 30. NS: nonsignificant; M-CHAT: Modified Checklist for Autism in Toddlers; CARS: Childhood Autism Rating Scale.
Comparing two independent proportions.
Discussion
Studies show that early developmental and behavioral interventions, especially before 3 years of age, improve the prognosis of children on the autistic spectrum; delay in diagnosis and appropriate intervention, especially after 5 years of age, decreases the effectiveness of the intervention (Johnson and Myers, 2007). Therefore, early detection and diagnosis of ASD would seem to be crucial for the outcome of this group. Screening tools are an important part of the diagnostic procedure. This study is the first to evaluate the psychometric utility of a well-known autism-specific screening instrument, the M-CHAT, in Turkey.
The first important point is how the M-CHAT was applied. It was originally developed to be filled in by the children’s caregivers. In our study, formerly, 191 of the M-CHATs were filled in independently by the parents, and the parents of children who failed received a confirmatory telephone call, at which time the completed M-CHAT is reviewed to clarify the selected answers and ascertain whether the child has in fact screened positive on the checklist. Only 12.5% of the children who screened positive remained positive after the phone call, pointing to a very high false-positive rate for parent-filled checklists. The change of the administration method and filling in the forms by the health-care staff according to parental information gave rise to more valid assessments. Low predictive value of the M-CHAT was reported to be the biggest shortcoming of the screener in unselected populations before the telephone interview (Kleinman et al., 2008). Kleinman et al. reported that 185 of the 385 children who failed the initial parental screening were confirmed to be failing in the telephone follow-up (48%). This fourfold difference in concordance rates can be related to socioeconomic, educational, and cultural factors. First, the setting of our study is a university well-child clinic following children mostly from low and middle socioeconomic classes; therefore, their parents may have had some difficulty understanding the written items of the M-CHAT. The average years of schooling of the adult population (6.1 years) is less than the developed Western countries (e.g. 12 years in the United States), and the rate of women having high school or above education is lower (18.2%). Second, people in our culture are rarely familiar with checklists, and in daily life, many people avoid following written instructions and usually prefer verbal ones. From our previous studies using self-rated questionnaires, we noticed that people respond more readily and accurately when a professional asks the question in a face-to-face interview. Therefore, completing the M-CHAT verbally is a more familiar procedure for most families.
It took approximately 5–7 min for health-care staff to administer the M-CHAT, which is not significantly longer than self-administration by English readers. Health-care staff involvement may be considered a drawback for this method, but this approach decreased the false positivity of the test significantly without additional modifications. So, it can be concluded that in our society, the M-CHAT is a useful instrument when it is administered by health-care staff interviewing parents.
Another point that should be clarified is our rationale for including the high-risk group in this study. We included the high-risk group with a probable diagnosis of pervasive developmental disorder to increase the probability of including children with autism without having to screen a very large number of children. This group included significantly more boys than the low-risk group, which included children with no known health problems. Since most of the children in the high-risk group for autism had a diagnosis of autism, this difference in the sex ratio was unavoidable due to the high percentage of boys among children with autism. The boy/girl ratio is reported to be 2/1 to 4/1 in the literature (Swinkels et al., 2006). In our study, this ratio was 3/1 in the high-risk group. The mean age of the children was also significantly different between the high- and low-risk groups, but all children in the study groups were aged between 18 and 36 months.
Many studies have evaluated the M-CHAT as a autism-specific screening tool in different countries (Canal-Bedia et al., 2011; Eaves et al., 2006; Eldin et al., 2008; Inada et al., 2011; Perera et al., 2009; Pinto-Martin et al., 2008; Robins et al., 2001; Wong et al., 2004). The methodology in our study had similarities with the studies of Robins and Dumont-Mathieu (2006). These studies have sufficient number of children from both low- and high-risk groups. The PPV in our study (0.75) was similar to the PPVs obtained in the studies by Robins et al. (0.80 and 0.74). Our PPV was for our study group not of the M-CHAT in Turkish.
In our study, the M-CHAT was administered at the age of 18–36 months. Clinical diagnosis and the CARS test were used to confirm the diagnosis of autistic spectrum disorder after 24 months of age. The time period between these tests was at least 6 months, and early educational intervention programs were initiated for children who screened positive with the M-CHAT before confirmation of the diagnosis by the study group. We observed that some children who had autistic characteristics at the time of the M-CHAT had no autistic characteristics when the clinical evaluation and CARS test were administered. Similar observations have recently been reported in the literature. This change may be due to false positives on the M-CHAT or amelioration of the autism symptoms due to the early educational interventions, but it was not possible to determine which of these applied. Because of the possibility of undocumented improvement in some children, we hypothesize that the true PPV of the M-CHAT might be higher than that found in our study.
Another important issue is the critical items of the M-CHAT. Robins et al. (2001) defined items 2, 7, 9, 13, 14, and 15 as critical in the M-CHAT because of the high diagnostic power of these items. In our study, the ratio of abnormal responses in the high-risk group to the low-risk group was high for questions 2, 5, 7, 9, 10, 14, 15, 17, 19, and 21 and low for questions 1, 3, 11, 12, 13, and 22. The difference between our results and those for the original M-CHAT is that there is less diagnostic power for item 13, but higher diagnostic power for test items 5, 10, 17, 19, and 21. The abnormal response rate for item 11 was 12.5% in the low-risk group, so the prognostic value of this item was very low.
This study suffers from some limitations. First, due to the unavailability of Autism Diagnostic Interview–Revised (ADI-R) and Autism Diagnostic Observation Schedule (ADOS) in Turkish, we were not able to make the diagnosis of autism based on internationally accepted interview form. Second, our study included a relatively small community sample. Third, the follow-up period (6 months) is relatively short, so to have more confidence in the psychometric properties of the M-CHAT in Turkey, we need a longer follow-up period.
However, this study has some strengths. First, this is one of the few studies on screening for autism in a developing country. Second, this is the first study in our country to adapt an autism-specific screening measure to our culture; therefore, this study describes a method for using the M-CHAT in a specific setting and culture. Third, all diagnostic interviews and clinical evaluations were done by child psychiatrists and child neurologists who had at least 4 years of experience working with autistic children, and all interviews were done under the supervision of the second author (N.M.M) who has 20 years of research experience and a clinical background in this field. This study is the first step to assess the psychometric utility of the M-CHAT in Turkish. We plan to conduct a new study with a larger sample and longer follow-up to assess the validity and reliability of the M-CHAT.
To sum up, in our study, self-administered completion of the questionnaire by parents was revealed by follow-up telephone interview to show many false positives. There were also significant problems with reaching the parents by telephone. The first important finding of this study is that the PPV of the M-CHAT increases when the questionnaire is filled out by the health-care staff in conjunction with the parents. The time needed for the health-care staff to fill out the questionnaire is negligible, and the requirement for additional parental clarification by interview disappears. The second important finding of this study is that the M-CHAT is a useful tool in Turkey for screening children for PDD in primary care after the age of 18 months. This study shows that the M-CHAT is a useful screening tool when it is adapted based on culture and setting.
Footnotes
Acknowledgements
We would like to thank Professor Deborah Fein for sharing her invaluable ideas with us during the course of this study and also for her and Diana L. Robins’ gracious permission to us for using M-CHAT in Turkish. In addition, we would also like to thank foundation of TODEV for the logistic support, Nilüfer Kayacık Doğan, Nevin Adalı, Ceren Can and the students of Istanbul Medical Faculty for their assistance in data gathering.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.