Examining Clinics for Children with Autism: The Autism Translating To Treatment Study

Abstract

Objectives:

Certain clinical providers specialize in providing complementary and integrative medicine (CIM) therapies for children with autism spectrum disorder (ASD). Because many of these providers and their patients/families have reported substantial improvement, the authors developed an online platform to carefully examine these clinical practices. The initial goal was to examine the feasibility of prospective data collection in this setting. The larger goals were to characterize the tests and treatments used in these clinics; examine associations between specific treatments, biomarkers, and improved outcomes; and identify promising treatments for future study.

Design:

Prospective cohort study.

Setting:

Four CIM clinics specializing in treating children with ASD.

Patients:

Children with ASD age 2–8 years.

Interventions:

The study protocol provided no interventions, but all interventions provided by the CIM clinical providers were recorded.

Outcome measures:

Aberrant Behavior Checklist (ABC); Social Responsiveness Scale (SRS); and instruments that assessed sensory sensitivity, language, gastrointestinal (GI) symptoms, pediatric quality of life, and caregiver strain.

Results:

Fourteen children were enrolled (mean age, 4.4 years). Over 3 months, the total behavior score (ABC) decreased (improved) from 110.8 to 103.8 (change, −7.0; 95% confidence interval [CI], −27.9 to 13.9), and the total social responsiveness score (SRS) decreased (improved) from 133.8 to 127.2 (change, −6.6; 95% CI, −30.5 to 17.3), but these changes were not statistically significant. Similarly, caregiver strain and pediatric quality of life decreased (improved) but by a nonsignificant amount. More severe GI symptoms and more severe ASD symptoms were associated with lower quality of life (p < 0.001).

Conclusions:

Barriers to successful data collection were identified. Despite these challenges, this study could confirm interesting associations between data elements, highlighting the future value of similar systems for improving evidence-based care in this population.

Introduction

Complementary and integrative medicines (CIMs) are commonly used by families in an effort to improve outcomes for their children with autism spectrum disorder (ASD). CIM has been defined by the National Center for Complementary and Integrative Health (NCCIH) as “health care approaches developed outside of mainstream Western, or conventional, medicine.” CIM was previously referred to as complementary and alternative medicine, or CAM, but the term “integrative” has replaced “alternative” in some descriptions of these practices, and CIM is used here. Previous surveys have found that 32%–95% of families report ever using CIM treatments for their child with autism.^1

–6 Variations in the estimates of the prevalence of CIM use are likely related to differences in survey methods (different populations, definitions of CIM, and survey distribution), but clearly CIM use is very common. In the most recent and largest population-based survey of CIM use among children with autism to date (1084 respondents), 88% of families reported ever using CIM while 69% reported using CIM in the past 3 months.⁵ Among families reporting CIM use, the average number of therapies used recently was two.⁵

One study found that the most common reasons for deciding to use CIM therapy were parental concerns regarding medication safety and side effects.² CIM therapy for ASD is more common among families with greater parental education, younger child age, and child prescription drug use,⁵ as well as among children with more severe autism symptoms.²

Several prior studies have assessed parental perceptions of efficacy of CIM for ASD, although methods varied widely. One study reported that parents felt that 75% of CIM therapies were beneficial.⁶ Another assessed both CIM and traditional therapies and found that 51% of parents reported a gluten-free or casein-free diet was beneficial.⁷ In the most recent study, parents were asked about efficacy of individual CIM therapies, and ratings of “effective” ranged from 20% to 84%; the most highly rated CIM products were melatonin (78.8%) and probiotics (65.1%), and the most effective CIM practices were at-home massage (84.2%), yoga/t'ai chi/qi gong (80.1%), and special diets (71.6%).⁵

The scientific literature has limited information from randomized controlled trials and meta-analyses regarding the efficacy of CIM for ASD. To date, eight systematic reviews of CIM therapies for ASD have been published by the Cochrane Collaboration, and only one therapy (music therapy) was found to have evidence supporting efficacy for ASD.⁸ There was insufficient evidence to support the efficacy of vitamin B6 plus magnesium,⁹ auditory integration training,¹⁰ gluten or casein-free diets,¹¹ chelation,¹² secretin,¹³ acupuncture,¹⁴ and omega-3 fatty acids¹⁵ in these systematic reviews.

Recently, the NCCIH published a featured health topic on autism, highlighting that there is “very little high quality research on complementary health approaches for ASD.”¹⁶ This publication noted that music therapy may improve social and behavioral skills and that melatonin may help with sleep problems, based on a systematic review published in 2011.¹⁷ The NCCIH also noted that there is insufficient evidence to determine efficacy for omega-3 fatty acids, acupuncture, massage, mindfulness, and oxytocin, among others. However, lack of scientific evidence does not prove lack of efficacy. It is possible that effective CIM therapies have not been adequately studied, and parental reports of efficacy along with review articles^18,19 suggest that some CIM therapies have promising evidence and should be further examined.

Certain clinical providers specialize in providing CIM therapies for children with ASD. Because many of these providers and the patients/families that they manage have reported substantial improvement, the authors sought to collaborate with and closely examine these practices with the following goals: (1) characterize the tests and treatments used in these clinics, (2) determine whether there are associations between specific treatments and improved outcomes, (3) determine whether specific biomarkers (laboratory tests) are associated with symptom severity and/or predictive of response to treatment, and (4) identify promising biomarkers and treatments that could be examined in future studies.

Materials and Methods

Planning conference

On January 27, 2012, a planning conference was held in San Francisco, California, and included key stakeholders in the area of CIM treatment for ASD. Attendees included clinical practitioners with experience providing CIM treatments to children with ASD, clinical investigators who conduct studies of CIM, funders, providers of CIM services, representatives of advocacy groups, and parents of affected children. The goal of the conference was to share ideas regarding the development of a standard set of laboratory tests that might be used to evaluate a child with ASD and a list of CIM treatments that might be effective and worthy of further evaluation. Although an extensive list of both tests and treatments was generated, there was not a clear consensus on which tests or treatments should be used in a child with ASD. Therefore, it was determined that the prospective study (described here) should not require clinicians to use specific tests or treatments but rather should examine different practices to determine which tests and treatments were most commonly used. Clinicians were asked about their interest in participating in the subsequent study.

Clinical sites

Four clinical sites in four different states agreed to participate in this study, which we named the Autism Translating To Treatment (AT3) study. All sites engaged in further collaborative planning for the collection of data. Sites included Portland, Oregon; Denver, Colorado; Little Rock, Arkansas; and Wilton, Connecticut. Each site has expertise in evaluating children with ASD and providing CIM treatments.

Selection of outcome measures

An extensive literature review was conducted to identify outcomes reported in prior studies examining interventions for children with autism with the goal of selecting a standard set of outcomes to include in AT3 and future studies. This process attempted to balance the need for a complete assessment while limiting the response burden for busy parents and clinicians. A previous systematic review identified 195 prospective clinical trials assessing treatment response in ASD published between 2001 and 2010.²⁰ In those studies, 289 unique measurement tools were used, but there was unfortunately very limited consistency in the selection of specific instruments. Only 3 outcome tools were used in more than 3% of the studies: the Aberrant Behavior Checklist (ABC, 5.0%), Clinical Global Impression (CGI, 4.6%), and Vineland Adaptive Behavior Scale (VABS, 3.9%). For this study the ABC was chosen; the VABS was not included because this instrument takes longer to complete (20–60 minutes), and the CGI was not included because it requires evaluation by a clinician experienced with this instrument (and clinicians experienced with this measure were not available at each site). The Social Responsiveness Scale (SRS) was chosen; this instrument has been used in recently published randomized controlled trials of interventions for ASD.^21,22 Also selected were instruments that assessed symptoms or other key variables in the following areas: sensory sensitivity, language, gastrointestinal (GI) symptoms, pediatric quality of life, and caregiver strain. In addition, a detailed form was developed to obtain the medical history for each child (including birth and family history, medical problems, prior medications, CIM, and other treatments) (Fig. 1).

FIG. 1.

Study outcomes.

Development of online platform for data entry

One of the goals of the study was to develop a user-friendly, online platform where users could easily log on from home or work, securely respond to surveys, and enter other relevant clinical information. After an extensive evaluation of multiple web-based services, a designer for the AT3 study was selected (Ilumivu, Cambridge, MA) and development began on August 23, 2013. The initial platform was developed during the 5-month period September 2013 through January 2014. The platform allowed parents to read about the study, go through an online informed consent process, and complete surveys that were provided on their own unique study page. Each study site was trained to use the online platform through conference calls conducted in February 2014.

Compliance with ethical standards, approval, and enrollment

The Committee on Human Research at the University of California San Francisco approved the study. Clinicians at each study site posted flyers that described the study, and interested families contacted a research assistant and received a detailed explanation of the study procedures. Informed consent was obtained from all parents/caregivers. Enrollment opened in May 2014 and continued through April 2015 (1 year).

Eligibility criteria

Children age 2–8 years with a diagnosis of autism (confirmed by a clinician at each site using Diagnostic and Statistical Manual of Mental Disorders, 4th edition, criteria) were eligible for study participation. The only exclusion criterion was being unable or unwilling to complete the surveys.

Sample size and statistical analyses

For this initial pilot study, the goal was to enroll and follow 5 participants per site (total of 20 participants) for a total period of 6 months. Experience with this pilot study would then inform future decisions about protocol changes, staffing, platform modifications, respondent burden, and feasibility to conduct a larger study. The analyses planned a priori were to examine (1) demographic characteristics of enrolled participants, (2) changes in each of the study outcome measures over time, (3) correlations between baseline variables, and (4) correlations between laboratory values or treatments and changes in outcome measures.

All data analyses were performed by using Stata 14 Statistical Software Package (Stata Corp, College Station, TX). The changes in outcome measures from baseline to follow-up (3 months) shown in Table 1 were compared by using t-tests. The exploratory relationships (correlations) between selected continuous measures were assessed via a linear mixed model adjusting for time.²³ All reported p-values are two sided and were not adjusted for multiple testing, as recommended by Rothman, to reduce the likelihood of missing important associations and to allow for the evaluation of each association in the context of other factors.²⁴

Table 1.

Changes in Outcome Measures from Baseline to Follow-Up (n = 6 Participants)

Outcomes	Baseline ^a	Follow-up (3 mo) ^a	Δ baseline to follow-up (95% CI)	p-Value
Aberrant Behavior Checklist	110.8 (28.8)	103.8 (32.9)	−7.0 (−27.9 to 13.9)	0.43
Hyperactivity	36.3 (10.3)	34.3 (12.3)	−2.0 (−14.4 to 10.4)	0.70
Irritability	25.3 (6.8)	24.7 (8.2)	−0.7 (−3.7 to 2.4)	0.59
Lethargy	28.5 (7.9)	25.7 (8.7)	−2.8 (−7.9 to 2.2)	0.21
Inappropriateness	8.3 (4.4)	8.3 (3.8)	0.0 (−2.4 to 2.4)	1.00
Stereotypic behavior	12.3 (4.8)	10.8 (4.3)	−1.5 (−5.6 to 2.6)	0.39
Social Responsiveness Scale	133.8 (34.6)	127.2 (25.9)	−6.6 (−30.5 to 17.3)	0.49
Social awareness	17.2 (3.0)	15.6 (1.7)	−1.6 (−4.8 to 1.6)	0.24
Social cognition	23.8 (4.9)	22.4 (4.0)	−1.4 (−6.9 to 4.1)	0.52
Social communication	43.4 (12.8)	45 (10.6)	1.6 (−6.1 to 9.3)	0.60
Social motivation	24.0 (5.3)	23.4 (5.1)	−0.6 (−1.7 to 0.5)	0.21
Autistic mannerisms	25.4 (11.3)	22.6 (9.8)	−2.8 (−11.8 to 6.2)	0.44
Caregiver Strain	33.5 (11.7)	32.2 (11.3)	−1.3 (−7.7 to 5.0)	0.61
Pediatric Quality of Life	33.8 (9.2)	33 (12.0)	−0.8 (−11.2 to 9.6)	0.84
Abbreviated Sensory Profile	30.4 (14.9)	29.4 (17.0)	−1.0 (−6.5 to 4.5)	0.67
GI Severity Index	9.3 (4.8)	10.5 (3.3)	1.2 (−2.0 to 4.3)	0.38

Values are expressed as mean (standard deviation).

Results

Patient characteristics

During the 1-year enrollment period, 24 parents completed “consent to be contacted” forms and were contacted by a research assistant. Of these, 20 completed the online informed consent process and enrolled in the study, and 14 completed the baseline surveys. Only 6 of 14 (43%) participants completed any follow-up survey data at the first time point (3 months), and only 2 of 14 (14%) participants completed 6-month follow-up information (Fig. 2). All study participants remained in their original CIM clinical practice, and the lack of completion of follow-up survey data was reported by parents and clinicians to be due to competing demands on time.

FIG. 2.

Study timeline: planning, approval, database, enrollment. IRB, Institutional Review Board.

The characteristics of the 14 enrolled participants are shown in Table 2. Thirteen of 14 children were male (93%), and the overall mean age was 4.4 years; 79% of patients were receiving one or more therapies, 79% were taking at least one supplement, 21% were using CIM, and 57% were on a special diet. Table 2 illustrates a sample of the wide range of information that can be collected in a web-based study.

Table 2.

Baseline Characteristics of Enrolled Participants

Characteristic	Value
Sex, n (%)
Male	13 (93)
Female	1 (7)
Race/ethnicity, n (%)
White	7 (50)
Hispanic	1 (7)
Mixed (white and other)	3 (21)
Undisclosed	3 (21)
Average age (yr)	4.4
Average parental age at delivery (yr)
Mother	30.4
Father	31.3
Child using therapies, n (%)
Total	11 (79)
Speech therapy	9 (64)
Occupational therapy	7 (50)
Applied behavior analysis	5 (36)
Physical therapy	5 (36)
Picture exchange communication system	4 (29)
Early Start Denver Model	1 (7)
Social skills group	1 (7)
Child using supplements, n (%)
Total	11 (79)
Injectable B12	6 (43)
Fish oil	4 (29)
Melatonin	5 (36)
Probiotics	6 (43)
Vitamin D	3 (21)
Child using CIM, n (%)
Total	3 (21)
Chiropractic	3 (21)
Chelation	1 (7)
Homeopathic	1 (7)
Naturopathic	1 (7)
IVIG	1 (7)
Child using special diet
Total	8 (57%)
Gluten free	8 (57%)
Casein free	7 (50%)
Other	4 (29%)

CIM, complementary and integrative medicine.

Change in outcome measures over time

The average scores at each time period as well as the change in scores are shown in Table 1 for the 6 patients who completed the 3-month follow-up surveys. The total ABC decreased (improved) from 110.8 to 103.8 (change, −7.0; 95% confidence interval [CI], −27.9 to 13.9), and the total SRS score decreased (improved) from 133.8 to 127.2 (change, −6.6; 95% CI, −30.5 to 17.3); however, these changes were not statistically significant. Similarly, caregiver strain and pediatric quality of life decreased (improved) but by a nonsignificant amount. Only 2 of the 16 scales and subscales increased (worsened) over time (SRS social communication subscale and GI severity index), but these changes were also not statistically significant.

Relationship between outcome measures

The relationship between autism symptom severity (as measured by the ABC and SRS) and pediatric quality of life versus worsening GI symptoms in shown in Figure 3. As demonstrated in the figure, increasing (more severe) scores on the GI severity index were positively correlated with worsening behavior (ABC), worsening social responsiveness (SRS), and more impaired pediatric quality of life. Although the correlations all visually show this relationship, only the correlation between GI symptoms and pediatric quality of life was statistically significant (p < 0.001). Similarly, Figure 4 demonstrates that as ASD symptom severity increased, both caregiver strain and pediatric quality of life also worsened (increased). These correlations were significant only for the changes in pediatric quality of life (p < 0.001).

FIG. 3.

Scatterplot showing the relationship between Gastrointestinal (GI) Severity Index and the outcome measures: Aberrant Behavior Checklist (ABC), Social Responsiveness Scale (SRS) and Pediatrics Quality of Life. Higher scores on the ABC, SRS, and GI severity index indicate worsening symptoms.

FIG. 4.

Scatterplot showing the relationship between changes in the ABC and SRS and changes in Caregiver Strain and Pediatrics Quality of Life. Higher scores on the ABC, SRS, Caregiver Strain, and Pediatrics Quality of Life indicate more severe symptoms.

Laboratory testing

The laboratory testing function of the online platform was used for only 3 study participants and did not capture enough data to provide meaningful summaries and associations. Figure 1 lists the many laboratory tests by laboratory and by test type.

Effect of CIM treatments on outcome measures

Data collected on the various CIM treatments for each participant varied widely, and it was not possible to examine the relationship between treatments and outcomes given the small sample size and limited follow-up.

Discussion

The overarching goal of this investigation was to assess the effects of CIM treatment on the behavior, sociability, and quality of life of children with autism by closely examining the practices of expert clinicians. To accomplish this, a comprehensive online platform was developed for systematic implementation, data collection, and analysis. The study had several important limitations, which we have grouped into three main areas: nonstandardized laboratory methods, high respondent burden, and poor usability of the online platform. While attrition or loss to follow-up is an inevitable event that is prominent in many longitudinal studies, the low completion rate in this study did not allow for an examination of the efficacy of CIM treatments or identification of potential biomarkers. However, this pilot effort demonstrated the potential value of online data collection and identifies methodologic issues that will aid in the planning of future studies.

Potential value of systematic, online data collection in autism

Although the study did not have an adequate sample size or follow-up to identify associations between specific treatments/biomarkers and participant outcomes (which could have provided a “signal” to suggest a potentially effective treatment or useful biomarker), the study did demonstrate the potential value of examining relationships between the collected data elements. For example, in this small set of 14 enrolled participants, there was a clear and visually evident relationship between worsening GI symptoms and both symptoms of autism and quality of life of affected children (as reflected by worsening scores on the ABC and SRS and pediatric quality of life; Fig. 3). This finding in a small set of patients is consistent with prior research, which shows a strong correlation between autism severity and GI symptoms.²⁵ Even in this very limited dataset, it was possible to perform a relatively simple analysis that confirmed findings of prior research, suggesting that a larger and more complete longitudinal data set has enormous potential to investigate many key issues in autism, from natural history to prognosis to identification of potentially useful biomarkers and treatments. As noted below, addressing the barriers to effective data collection will allow for more thorough data and greater potential for discovery.

Nonstandardized laboratory methods

The high variation in the number and types of laboratories and tests engendered the development of a lengthy and complex data entry process, which led to very limited entry of laboratory data. With eight different clinical laboratories, it was not possible to develop a system in which participant laboratory data were automatically downloaded into the secure data storage. Furthermore, study clinicians did not have a standardized or consistent method of laboratory assessment, so enrolled children had different sets of laboratory tests, limiting the ability to examine the trend or association with any given test. The proposed solution to this problem is to identify a set of tests that are considered to be standard and useful for the evaluation and management of all children with ASD that could be performed on all participants in future studies. The initial experience with the planning conference was not able to accomplish this task because of variations in provider practice. However, recent developments in the area of metabolomics may offer a potential solution. The field of metabolomics analysis has improved markedly in the past decade, and it is now possible to conduct one metabolomics test on a sample of serum (or urine or saliva) and detect as many as 1300 different unique chemicals that make up an individual's “metabolome.” A recent analysis found that children with autism have as many as 80 different metabolites with statistically significant differences from typically developing children,²⁶ and these metabolites might serve as biomarkers to diagnose autism, guide selection of treatments, or monitor treatment efficacy. Further studies to examine the utility of metabolomics in autism are ongoing.

High respondent burden

As mentioned earlier, respondent burden is a critical factor affecting response rate in survey studies. In this pilot study, caregivers reported that the total time it took to complete the questionnaires at each time period ranged from 130 to 160 minutes. Studies have shown that the length of web-based questionnaires negatively affects response rate and compliance.^27
–29 In one study, participation rates dropped from 75% to 63% when it was preannounced that the survey would last 10 or 30 minutes, respectively.²⁹ In the same study, those who took the 10-minute survey had a higher completion rate than those who took the 30-minute survey. In addition, studies have reported that follow-up contacts or reminders after the initial response significantly increased response rate for both online and mail-in surveys, but the timing and frequency of reminders must be carefully considered so as not to annoy participants.²⁸ Furthermore, studies have also shown that monetary incentives can act as extrinsic motivators that increase response rate.^28,30 Other factors that may negatively affect the respondent's perception of burden to web-based surveys include complex password access, lengthy download times, and technical issues with the website. The issue of respondent burden requires a careful balance of the desire to be thorough with assessments while maximizing completion rates.

In the current study, the respondent burden was too large, and a 2- to 3-hour initial assessment likely reduced enthusiasm for continued study participation. Both clinicians and parents who participated in this study provided feedback that there were extremely busy caring for children with ASD, which makes respondent burden an even more critical factor to evaluate when planning future studies in this population. On the basis of feedback from study participants, baseline assessments should be limited to 30 minutes or less and follow-up assessments to 15 minutes or less.

Utility and usability of online platform

The complex design of the AT3 online data entry system contributed to low adherence and high dropout. On the basis of experience with this study and participant feedback, both usability and utility are essential for improving study adherence. Usability refers to the ease of use of the system. Caregivers of children with autism and other neurodevelopmental disorders are already overburdened. Therefore, it is imperative that any online data collection platform minimizes interactions that demand high cognitive load or lengthy tasks. In this context, important usability features are simplicity of design and low maintenance. Utility refers to the usefulness of the system: in other words, can the system user identify a benefit from use? For higher utility, a system should support real-time data inference and on-demand data access. For example, a system with high utility might allow the data to be autoanalyzed and presented to users in the form of simple graphs or charts that summarize important findings.

On the basis of participant feedback in the current study, the authors initiated the development of a new system using participatory design, a common method in the technology design community whereby the designer works closely with the target user to collaboratively iterate on the design of a technology.³¹ The goal of the revised system is to support (a) evidence-based care by enabling caregivers to quantify the effects of interventions and (b) integrated care by allowing caregivers to share progress within their network. The new system will have custom dashboards depending on the user's role. For data analysis and inference, various standard and custom graphical data visualizations will be incorporated, and this will allow caregivers to track key life events and assess a child's progress within this context. Furthermore, this platform will have a built-in scheduling component to monitor data entry and remind caregivers about pending and current data entry tasks. The revised system should have immediate and obvious benefits and will therefore be more compelling and lead to markedly improve adherence, data collection and quality, and scientific discovery.

Conclusion

This pilot study provides insight into the potential benefits of online longitudinal data collection in children with autism, in both CIM and traditional medical settings. Despite many limitations, including a large and nonstandard set of laboratory assessments, a large respondent burden, and poor completion rates, this study was able to identify interesting associations between data elements. For example, the results have confirmed prior research that found that an increase in the severity of GI problems is associated with more severe symptoms of ASD. Similarly, in this small group of patients, more severe symptoms of autism were associated with lower quality of life. The study experience and participant feedback also identified several areas that may improve adherence and data quality in future studies, and these include using a standard laboratory assessment, limiting respondent burden, and designing systems with high usability and utility so that the platform user can quickly recognize tangible benefits to study participation. Ultimately, this study advances the technique of using online data collection to improve clinical research, personalized treatment plans, and collaborative care based on each child's unique profile.

Footnotes

Acknowledgments

This work was supported by the Robert's Shoes Fund and the Jane Botsford Johnson Foundation.

Author Disclosure Statement

Dr. Hendren reports research grants from Curemark, BioMarin, Roche, Shire, Sunovion, Autism Speaks, Vitamin D Council, Advisory Board for Curemark, BioMarin, Neuren, and Janssen. All other authors declare that no competing financial interests exist.

References

Christon

, Mackintosh

, Myers

. Use of complementary and alternative medicine (CAM) treatments by parents of children with autism spectrum disorders. Res Autism Spectr Disorde, 2010; 4:249–259.

Hanson

, Kalish

, Bunce

, et al. Use of complementary and alternative medicine among children diagnosed with autism spectrum disorder. J Autism Dev Disord. 2007; 37:628–636.

Harrington

, Rosen

, Garnecho

, Patrick

. Parental perceptions and use of complementary and alternative medicine practices for children with autistic spectrum disorders in private practice. J Dev Behav Pediatr, 2006; 27(2 Suppl):S156–161.

Levy

, Hyman

. Use of complementary and alternative treatments for children with autistic spectrum disorders is increasing. Pediatr Ann, 2003; 32:685–691.

Owen-Smith

, Bent

, Lynch

, et al. Prevalence and predictors of complementary and alternative medicine use in a large insured sample of children with autism spectrum disorders. Res Autism Spectr Disord, 2015; 17:40–51.

Wong

, Smith

. Patterns of complementary and alternative medical therapy use in children diagnosed with autism spectrum disorders. J Autism Dev Disord, 2006; 36:901–909.

Goin-Kochel

, Mackintosh

, Myers

. Parental reports on the efficacy of treatments and therapies for their children with autism spectrum disorders. Res Autism Spectrum Disord, 2009; 3:528–537.

Geretsegger

, Elefant

, Mossler

, Gold

. Music therapy for people with autism spectrum disorder. Cochrane Database Syst Rev, 2014(6):CD004381.

Nye

, Brice

. Combined vitamin B6-magnesium treatment in autism spectrum disorder. Cochrane Database Syst Rev, 2005(4):CD003497.

10.

Sinha

, Silove

, Wheeler

, Williams

. Auditory integration training and other sound therapies for autism spectrum disorders. Cochrane Database Syst Rev, 2004(1):CD003681.

11.

Millward

, Ferriter

, Calver

, Connell-Jones

. Gluten- and casein-free diets for autistic spectrum disorder. Cochrane Database Syst Rev, 2004(2):CD003498.

12.

James

, Stevenson

, Silove

, Williams

. Chelation for autism spectrum disorder (ASD). Cochrane Database Syst Rev, 2015(5):CD010766.

13.

Williams

, Wray

, Wheeler

. Intravenous secretin for autism spectrum disorder. Cochrane Database Syst Rev, 2005(3):CD003495.

14.

Cheuk

, Wong

, Chen

. Acupuncture for autism spectrum disorders (ASD). Cochrane Database Syst Rev, 2011(9):CD007849.

15.

James

, Montgomery

, Williams

. Omega-3 fatty acids supplementation for autism spectrum disorders (ASD). Cochrane Database Syst Rev, 2011(11):CD007992.

16.

National Center for Complementary and Integrative Health. Featured health topic: autism spectrum disorder. Online document at: https://nccih.nih.gov/health/autism. Accessed May 12, 2016 .

17.

Rossignol

, Frye

. Melatonin in autism spectrum disorders: a systematic review and meta-analysis. Develop Med Child Neurol, 2011; 53:783–792.

18.

Bent

, Hendren

. Complementary and alternative treatments for autism part 1: evidence-supported treatments. AMA J Ethics, 2015; 17:369–374.

19.

Lofthouse

, Hendren

, Hurt

, Arnold

, Butter

. A review of complementary and alternative treatments for autism spectrum disorders. Autism Res Treat, 2012:870391.

20.

Bolte

, Diehl

. Measurement tools and target symptoms/skills used to assess treatment response for individuals with autism spectrum disorder. J Autism Dev Disord, 2013; 43:2491–2501.

21.

Bent

, Hendren

, Zandi

, et al. Internet-based, randomized, controlled trial of omega-3 fatty acids for hyperactivity in autism. J Am Acad Child Adolesc Psychiatry, 2014; 53:658–666.

22.

Hardan

, Fung

, Libove

, et al. A randomized controlled pilot trial of oral N-acetylcysteine in children with autism. Biol Psychiatry, 2012; 71:956–961.

23.

Vittinghoff

, Glidden

, Shiboski

, McCulloch

. Regression Methods in Biostatistics: Linear, Logistic, Survival, and Repeated Measures Models. New York: Springer, 2012.

24.

Rothman

. No adjustments are needed for multiple comparisons. Epidemiology, 1990; 1:43–46.

25.

Adams

, Johansen

, Powell

, Quig

, Rubin

. Gastrointestinal flora and gastrointestinal status in children with autism—comparisons to typical children and correlation with autism severity. BMC Gastroenterol, 2011; 11:22.

26.

West

, Amaral

, Bais

, et al. Metabolomics as a tool for discovery of biomarkers of autism spectrum disorder in the blood plasma of children. PloS One, 2014; 9:e112445.

27.

Cook

, Heath

, Thompson

. A meta-analysis of response rates in web or Internet-based surveys. Educat Psychol Measure, 2000; 60:821–836.

28.

Deutskens

, De Ruyter

, Wetzels

, Oosterveld

. Response rate and response quality of internet-based surveys: an experimental study. Marketing Lett, 2004; 12:21–36.

29.

Galesic

, Bosnjak

. Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opin Q, 2009; 73:349–360.

30.

Goritz

. The impact of material incentives on response quantity, response quality, sample composition, survey outcome, and cost in online access panels. Int J Market Res, 2004; 46:327–345.

31.

Jacko

. The Human-Compter Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications. 3rd ed. New York: CRC Press, 2002.