Social Validity of Evidence-Based Practices and Emerging Interventions in Autism

Abstract

Although social validation of the goals, methods, and outcomes of evidence-based practices (EBPs) in autism treatment is a significant factor in their selection and effective use, EBPs are typically identified on the basis of the technical soundness of research without consideration of social validity. The authors investigated EBPs and emerging treatments identified by the National Autism Center (NAC) and National Professional Development Center on Autism Spectrum Disorders (NPDC) to determine which interventions have evidence of social validity, and the types of social validation addressed. A review of 828 articles cited by the NAC and NPDC determined that only 221 articles (26.7%) demonstrated direct evidence of the measurement of social validation. Of seven social validity categories analyzed, only consumer satisfaction, clinically significant behavioral change, and socially important dependent variables were consistently reported. A list of EBPs with varying levels of social and empirical validation is presented, and implications for future research are discussed.

Keywords

autism autism spectrum disorder evidence-based practices social validation applied behavior analysis special education

Although the field of autism intervention and treatment is still emerging, a lack of knowledge about what to do should no longer be considered a significant challenge facing service providers. Recent large-scale, systematic, and robust research reviews have identified interventions with strong empirical evidence of effectiveness in some outcome areas. The results of these reviews provide support for the use of specific treatments with individuals who have autism spectrum disorder (ASD), especially within certain subpopulations. Two initiatives, in particular, the National Autism Center’s (NAC; 2009) National Standards Project (NSP) and similar efforts by the National Professional Development Center on Autism Spectrum Disorders (NPDC; Odom, Collet-Klingenberg, Rogers, & Hatton, 2010; Wong et al., 2014), have resulted in the dissemination of lists of empirically validated treatments that can be implemented by teachers, parents, and therapists. Although differing somewhat in methodology, scope, and definitions (NPDC, 2013), the NSP and NPDC studies independently and reliably identified autism interventions and treatments considered to be “established” and “confirmed,” respectively, as evidence-based practices (EBPs) in autism. The NPDC list originally specified 24 confirmed EBPs, 18 of which were also identified as established treatments by the NSP. Additional EBPs have recently been added (and one was removed) by the NPDC (Wong et al., 2015).

Both the NSP and NPDC reviews developed criteria to systematically analyze the content (i.e., the “quality, quantity, and consistency,” NAC, 2009, p. 31) of research findings and their corresponding strength of evidence. Both research teams sought to ensure adequate empirical evidence was available to reliably determine the efficacy of treatments for individuals with ASD before labeling any intervention as an established, confirmed, or emerging EBP.

Despite this encouraging development, research in special education, applied behavior analysis (ABA), and other therapeutic fields suggests empirical evidence of effectiveness is often not sufficient to ensure the widespread use of potentially effective interventions, treatments, and programs (e.g., Bodfish, 2004; Carter, 2010). Although using EBPs in the public education of children with autism and other disabilities is mandated by federal regulations (e.g., Individuals With Disabilities Education Improvement Act [IDEIA], 2004; No Child Left Behind [NCLB], 2002) and state and local education policies, many service providers resist their use (Bodfish, 2004) or continue to implement practices demonstrated by research to be ineffective (Cook & Odom, 2013; Lilienfeld, Marshall, Todd, & Shane, 2015). And, even though the delivery of effective programming based on scientific evidence is emphasized by professional standards and codes of ethics in special education (Council for Exceptional Children, 2013), ABA (Behavior Analyst Certification Board, 2010), and most other therapeutic professions providing autism services (e.g., American Occupational Therapy Association, 2010), significant challenges remain in the widespread effective implementation of research-demonstrated EBPs in public schools and other settings, including homes, clinics, and centers (Cook & Cook, 2011; Mesibov & Shea, 2011). A variety of factors related to the users of EBPs, the practices themselves, and their institutional context (Cook & Odom, 2013; Rapp et al., 2010) have been reported to negatively affect the effective use of EBPs. These factors include a lack of knowledge of EBPs, limited availability of training, low competence of service providers, high costs, lack of time for planning and preparation, and difficulties using some interventions, among many others. Limited social validation is another possible factor affecting the routine application of EBPs.

Social validity can generally be defined as consumer satisfaction with the goals, procedures, and outcomes of programs and interventions (Alberto & Troutman, 2008; Wolf, 1978). The extent to which a particular intervention or treatment receives social validation plays a significant role in whether the treatment is adopted and implemented (Kazdin, 1977; Kern & Manz, 2004). According to Carter (2010), “. . . just because a program is considered effective does not mean that it will be considered appropriate by those closely involved in implementation of the program, or that it should even be implemented in the first place” (p. 7). Thus, without social validation, there is perhaps less chance autism EBPs will be used effectively in homes, schools, and clinics (Bodfish, 2004; Carter, 2010; Kazdin, 1977; Schwartz & Baer, 1991; Wolf, 1978). It can be argued that social validity is a necessary additional consideration in the selection and effective use of EBPs in autism.

Extending previous seminal work investigating the quality of intervention research for individuals with disabilities (Horner et al., 2005; Gersten et al., 2005), Reichow and colleagues (Reichow, Doehring, Cicchetti, & Volkmar, 2011; Reichow, Volkmar, & Cicchetti, 2008) identified quality indicators of robust autism research, including seven specific components of social validation. These social validity indicators include (a) socially important dependent variables (i.e., society would value the changes in outcome of the study), (b) time- and cost-effective intervention (i.e., the ends justify the means), (c) comparisons between individuals with and without disabilities, (d) a behavioral change that is large enough for practical value (i.e., it is clinically significant), (e) consumers who are satisfied with the results, (f) independent variable manipulation by people who typically come into contact with the participant, and (g) a natural context. Further research investigating key aspects of social validation can provide important additional evidence to support the identification and use of specific interventions for individuals with ASD across the autism spectrum and life span.

Fortunately, increased attention has been given to the social validation of autism interventions and treatments. Similar to the NSP and NPDC results, researchers have identified autism interventions and treatment components, as well as specific target skills, which have been socially validated by parents, special educators, and public school administrators (Callahan, Henson, & Cowan, 2008) and by behavior analysts and other service providers (Dickson et al., 2014). Callahan et al. (2008) asked parents, special education teachers, and administrators to socially validate essential components of autism programs based on their perceived importance as part of comprehensive school-based programming. These authors reported a rank-ordered list of intervention components with broad-based support for their use organized into a comprehensive treatment model referred to as IDEAL, an acronym of the model’s five functional areas, including (a) individualized programming, (b) data-based decision making, (c) use of empirically demonstrated interventions, (d) active collaboration, and (e) a focus on long-term outcomes. The IDEAL Model components (see Table 1) were further reliably analyzed to determine whether they aligned with practices typically associated with ABA, the Treatment and Education of Autistic and Related Communication-Handicapped Children Model (TEACCH), both ABA and TEACCH, or neither approach (Callahan, Shukla-Mehta, Magee, & Wie, 2010). A total of 24 subject matter experts with certification and/or extensive training and experience (eight board certified behavior analysts, eight TEACCH-trained practitioners, eight experts with training in both models) rated which IDEAL Model components represented a “defining feature” of ABA or TEACCH, or both models, as they are commonly recognized and practiced with students with ASD. Overall agreement with the authors’ assigned categories was 70.2% (Callahan et al., 2010, p. 76).

Table 1.

Summary of Evidence of SV.

IDEAL Model intervention components/SV rating (SD)/model affiliation: ABA, TEACCH, or BOTH	EBP/emerging practice/articles cited (n)		Evidence of SV within NSP and NPDC articles		Articles reporting social validity across Reichow, Doehring, Cicchetti, and Volkmar (2011) SV categories (n)
	NSP	NPDC	Articles reviewed^a (n)	Articles reporting social validity (n/%)	Socially important dependent variables	Time- and cost-effective	Individual with and without disabilities	Clinically significant behavioral change	Consumer satisfied with results	Independent variable × People in typical context	Natural context
Social skills strategies/6.69 (0.56)/BOTH	Social skills package^b/16	Social skills training/15	28	13/46.4%	3	4	5	5	11	1	4
Structure and routines/6.62 (0.73)/TEACCH	Schedules/12	NA/Structured work systems^c	12	6/50.0%	3	3	1	5	5	1	1
Communication skills strategies/6.57 (0.70)/BOTH	Behavioral package/228	Functional communication training/12	234	35/15.0%	14	7	1	23	26	10	10
FBA/functional analysis/6.54 (0.69)/ABA	Behavioral package/228	FBA/10	236	35/14.8%	14	9	1	23	26	8	8
Motivational procedures^d/6.53 (0.64)/BOTH	Antecedent package/99	Antecedent-based intervention/32	121	22/18.2%	10	4	3	5	16	3	3
Effective reinf./6.42 (0.80)/BOTH	Behavioral package/	Reinf./43	253	35/13.8%	15	8	1	24	26	9	9
Effective reinf./6.42 (0.80)/BOTH	228	Reinf./43	253	35/13.8%	15	8	1	24	26	9	9
Task analysis/6.40 (0.79)/BOTH	Behavioral package/	Task analysis/7	235	33/14.0%	15	6	2	23	24	8	9
Task analysis/6.40 (0.79)/BOTH	228	Task analysis/7	235	33/14.0%	15	6	2	23	24	8	9
Prompting, shaping, chaining, and fading/6.39 (0.79)/ABA	Antecedent package/99	Prompting/33	128	23/18.0%	16	3	4	7	19	2	5
Incidental teaching/natural. teach./6.32 (0.78)/BOTH	Natural. teaching strategies/32	Natural. intervention/10	37	17/45.9%	3	6	4	14	13	3	3
Peer-mediated intervention/6.32 (0.82)/BOTH	Peer training package/33	Peer-mediated instruction and intervention/15	42	15/35.7%	8	5	6	13	3	5	4
Incentive programs based on positive reinf./6.18 (0.94)/ABA	Behavioral package/228	Reinf./43	253	35/13.8%	15	8	1	24	26	9	9
Intensive 1:1 instruction trials/6.17 (1.00)/ABA	Behavioral package/228	Discrete trial training/13	240	32/13.3%	14	6	1	22	23	8	8
Preferred reinforcers/6.16 (1.06)/ABA	Antecedent package/99	Antecedent-based intervention/32	121	22/18.2%	10	4	3	5	16	3	3
Visual activity schedules/visual supports/6.16 (1.09)/TEACCH	Schedules/12	Visual supports/18	26	17/65.4%	13	7	4	7	16	1	2
PRT/6.15 (0.90)/BOTH	PRT/14	PRT/7	17	6/35.3%	0	1	0	4	5	2	0
Individual. desks and furniture, other classroom materials/6.10 (1.03)/BOTH	Antecedent package/99	Antecedent-based intervention/32	121	22/18.2%	10	4	3	5	16	3	3
Formal reinf. assessments/6.08 (0.95)/ABA	Behavioral package/228	Reinf./43	253	35/13.8%	15	8	1	24	26	9	9
Differential reinf. strategies/6.06 (1.00)/ABA	Behavioral package/228	Differential reinf. of alternative incompat. or other behavioral/26	241	32/13.3%	14	6	1	22	23	8	8
Stimulus control/6.06 (1.10)/ABA	Antecedent package/99	Antecedent-based intervention/32	121	22/18.2%	10	4	3	5	16	3	3
Progress. and constant time delay/6.05 (0.99)/ABA	Antecedent package/99	Time delay/12	107	18/16.8%	11	2	3	3	14	2	4
Adult-directed strategies/6.02 (0.97)/ABA	Modeling/50	Prompting/33	83	17/20.5%	5	2	2	14	8	1	3
Student preferences and/or interests/6.02 (1.03)/BOTH	Antecedent package/99	Antecedent-based intervention/32	121	22/18.2%	10	4	3	5	16	3	3
Systematic prompts and prompt hierarchies/5.99 (1.10)/ABA	Antecedent package/99	Prompting/33	128	23/18.0%	16	3	4	7	19	2	5
Modifying physical environment/5.95 (1.06)/TEACCH	Antecedent package/99	Antecedent-based intervention/32	121	22/18.2%	10	4	3	5	16	3	3
Modeling/video modeling/5.93 (1.09)/ABA	Modeling/50	Video modeling/32	66	26/39.4%	10	5	2	17	11	2	1
Extinction/5.91 (1.24)/ABA	Reductive package^b/33	Extinction/11	44	1/2.3%	1	1	0	1	1	0	0
Physical classroom organizers/5.75 (1.28)/TEACCH	Schedules/12	NA/Structured work systems^c	12	6/50.0%	3	3	1	5	5	1	1
Technology and other augmentative communication devices/5.74 (1.30)/BOTH	Technology-based treatment^b/19	Technology-aided instruction and intervention/20	37	7/18.9%	6	2	0	3	6	2	1
Pictorial storyboard simulations and/or social stories/5.72 (1.15)/TEACCH	Story-based intervention package/21	Social narratives/17	31	23/74.2%	13	5	2	14	13	3	7
PECS/5.71 (1.31)/TEACCH	PECS^b/13	PECS/6	17	6/35.3%	3	3	0	3	5	1	2
Self-management strategies/5.70 (1.10)/BOTH	Self-management/21	Self-management/10	27	9/33.3%	5	4	1	6	6	2	4

Note. ABA = applied behavior analysis; TEACCH = Treatment and Education of Autistic and related Communication-handicapped Children; EBP = evidence-based practices; SV = social validation; NSP = National Standards Project (National Autism Center); NPDC = National Professional Development Center on Autism Spectrum Disorders; FBA = functional behavior assessment; PRT = pivotal response training; PECS = Picture Exchange Communication System; NAC = National Autism Center.

The number of articles reviewed equals the total number of articles cited by the NSP and NPDC minus the number of articles appearing on both reviews. ^bConsidered an “emerging practice” by the NAC. ^cNot applicable (NA) because “more stringent evidence reduced previous evidence” and the intervention is no longer considered an EBP by the NPDC. ^dPreferred items and natural reinforcers.

More recently, Dickson et al. (2014) investigated core skills important to individuals with autism across the life span using an assessment instrument consisting of direct and indirect measures of skills. Two hundred forty-five parents, teachers, and therapeutic staff responsible for implementing effective treatment programs were asked to indicate which items they considered to be foundational. The results provided a measure of social validation for specific treatment targets in the domains of discrimination, communication, social skills, self-help, health and safety, and recreation/physical education. The authors concluded these socially validated skills, such as making requests for assistance or desired items, responding to one’s name, and following one-step directions, can be used as a starting point for instructional programming for individuals with ASD (Dickson et al., 2014).

Now that researchers have reliably identified potentially effective interventions from which parents, teachers/therapists, and administrators can choose, research attention can begin to address specific factors affecting how these consumers select and implement EBPs (Cook & Odom, 2013). A critical first step in this process is to help service providers identify from among the relatively large list of available interventions and treatments those that, ostensibly, have the greatest likelihood of making a positive impact on specific target deficits of their children, students, and clients (Odom, 2009; Odom, Cox, Brock, & the NPDC, 2013). The results of additional social validity research can help inform this process. For example, identifying which specific social validation components are most critical for the successful implementation of EBPs, and which aspects of social validity are explored most and least frequently in the literature, can provide new insights for practitioners and researchers.

The purpose of this study is to identify the extent to which autism interventions and treatment components are reported to have both social and empirical validation and to investigate the types of social validity commonly addressed by autism researchers and practitioners. The resulting list of EBPs could serve as a starting point for service providers designing and implementing autism programming. Identifying socially validated EBPs could help address (and possibly eliminate) some of the barriers to effective implementation of EBPs, and address the pervasive research-to-practice gaps in autism programming (Dingfelder & Mandell, 2011). Most importantly, a prioritized list of validated interventions and training resources could ultimately result in improved outcomes for individuals with ASD and their families.

Method

Participants and Procedures

Preliminary analysis

Two analyses were conducted to identify the extent to which empirically demonstrated EBPs also have evidence of social validation. For the first analysis, a team of three researchers representing doctoral-level expertise in special education, ABA, autism assessment, and early childhood education compared the correspondence of empirically validated treatments and interventions using the NSP and NPDC lists of established, confirmed, and emerging EBPs with a list of 60 evidence-based intervention components that were socially validated using a mail survey by 187 parents, special education teachers, and public school administrators (Callahan et al., 2008). The authors included treatments categorized as “emerging” by the NSP. These treatments are supported by a minimum of one group design or two single-subject design studies with a minimum of six participants, with adequate scientific merit, and no conflicting results reported. In addition, beneficial treatment effects have been reported for these interventions for at least one dependent variable/specific target behavior (NAC, 2009). Previously, survey respondents rated each of the 60 intervention components on a 7-point Likert-type scale based on their perceived importance of the intervention as part of a high-quality autism program, where a rating of 7 indicated the component was “Absolutely Important (the component is an indispensable part of a high-quality autism program)” and a rating of 1 meant the component was “not at all important (the component is totally irrelevant and/or unrelated to a high-quality autism program).”

Inter-rater agreement

For this preliminary study, the research team independently analyzed each of the socially validated intervention components of the IDEAL Model to identify corresponding treatments empirically validated by the NSP and/or NPDC reports. Each member of the research team was instructed to thoroughly read designated NSP and NPDC materials to become knowledgeable about the research and analysis procedures and the treatment definitions used. The researchers were directed to independently review each of the 60 IDEAL Model components and determine whether the components directly aligned with any of the 24 treatments identified as “established” or “emerging” by the NSP and/or those “confirmed” as an EBP by the NPDC. Descriptions of the identified EBPs and IDEAL Model components were used by the study authors during the review and alignment phase of this analysis. The NSP information included the name of the intervention package, if applicable, as well as descriptive information about the various interventions making up the package. For example, the NSP’s “Antecedent Package” was defined as including behavior chain interruption, behavioral momentum, choice, and cueing/prompting, among other components. The results of the independent ratings were analyzed for agreement. Twenty-three of the 24 NSP/NPDC EBPs aligned directly with one or more of the socially validated IDEAL Model intervention components. Inter-rater agreement across all intervention components was 89.4%. The research team met to address all items for which there was less than 100% agreement among the three raters. This process clarified definitions and resulted in 100% agreement among all raters for all items.

Outcomes

A total of 31 IDEAL Model components were determined to align directly with an NSP/NPDC EBP. Although typically a single IDEAL Model component aligned with just one EBP, in some cases, more than one IDEAL Model component corresponded with a particular EBP. For example, two IDEAL Model components, “The use of visual activity schedules . . .,” and “The use of physical classroom organizers . . .,” were determined to align with the EBP “Visual Supports” (NPDC)/”Schedules” (NSP).

Expert validation process

The corresponding IDEAL Model components and NSP/NPDC EBPs were systematically validated by experts in the field of autism. Using Qualtrics software, Version 13, of the Qualtrics Research Suite (Qualtrics, 2013), a link to an online survey was sent to seven persons with extensive experience and training in autism intervention, including university faculty members teaching autism graduate courses, and public school and center-based practitioners with board certification in ABA and/or formal training in the TEACCH Model. The experts were asked to validate that the socially validated IDEAL Model components represented the same interventions and treatments within the NSP/NPDC studies. Respondents were instructed to indicate agreement (“Yes”) or disagreement (“No”) that the NSP/NPDC EBP referred to the same treatment or intervention as conceptualized within the IDEAL Model. For example, the experts were asked to determine whether the IDEAL Model intervention component, “The use of incidental teaching or naturalistic teaching . . .,” represented the same intervention as the NSP’s “Naturalistic Teaching Strategies” and the NPDC’s “Naturalistic interventions.” All but one expert validated 100% of the items as representing the same interventions. The overall mean percentage agreement of the expert validators was 99.6% (range = 96.7%–100%), indicating a very high level of agreement that the socially validated IDEAL Model intervention components represent the same treatments empirically validated by the NSP and NPDC.

Secondary analysis

Because the procedures implemented to determine the social validity of the IDEAL Model components addressed only a limited aspect of social validation—that is, the perceived value of specific intervention components by consumers within public school autism programs—a second analysis was conducted to determine whether broader evidence of social validation exists for EBPs and emerging practices in autism. For this analysis, an expanded research team consisting of the original members and four additional PhD-level researchers reviewed all articles cited within the NSP and NPDC reviews for the corresponding socially validated IDEAL Model components. The purpose was to determine whether the author(s) directly addressed social validation, and to identify the types of social validity reported. A total of 828 unique articles within the reference lists for the relevant NSP and NPDC EBPs were reviewed.

Articles were searched using an automated search function and at least the following search terms: valid, social, and satis. In situations where the automated search was inoperable or failed to locate social validity information, the researchers reviewed the full article to identify possible references to social validity and social validation. Researchers were directed to search every article for language and terminology not captured in the automated searches.

The research team used an intentionally conservative approach to reliably identify direct evidence of measurement of social validation. That is, the researchers coded articles as demonstrating direct evidence of social validation only if the author(s) specifically collected social validity data. The research team only considered the explicit language of the author(s) using the words, headings, phrases, and/or concepts appearing in the articles which clearly addressed or referred to social validity or social validation. Most often, this information was located within sections of the article labeled “Social Validation,” or “Parent Satisfaction,” or similar headings. In rare cases, the reviewers located relevant information embedded within other sections of the articles. In all instances in which the article authors examined social validity, at least one Reichow et al. (2011) category was identified and coded.

Finally, to determine the types of social validation most often addressed by autism researchers, article reviewers assessed the author(s)’ stated evidence of social validation and assigned the information to one or more of the Reichow et al. (2011) categories of social validity. For example, the statement “Social validation measures provided by 30 raters indicated that perceived positive changes in performance had occurred in child behaviors, and training procedures were acceptable” could be assigned to Reichow et al.’s categories of “Clinically significant behavioral change” and “Consumers satisfied with the results,” respectively.

Inter-rater agreement

Of the 828 articles reviewed within the reference lists of the NSP and NPDC EBP summary reports, a total of 201 articles (24.3%) were reviewed and coded by at least two different researchers to determine inter-rater reliability in assessing evidence of social validity and agreement among the reviewers for assignments of social validity evidence to the Reichow et al. (2011) categories. A reliability coefficient was calculated for each article by dividing the number of rater agreements by the number of agreements plus disagreements and multiplying the results by 100 to obtain a percentage of agreement. Agreements and disagreements were compared for each of the seven Reichow et al. (2011) categories, as well as for the basic question, “Did the article contain evidence of social validity?” An agreement was defined as both reviewers scoring the Reichow et al. category and answering the evidence of social validity question the same. There was 100% agreement regarding whether articles contained evidence of social validity. Across all articles in which social validation was identified and those articles in which social validity was not identified, overall inter-rater agreement on the assignment of social validity evidence to the Reichow et al. categories was 97.4%. For those articles only in which social validity was identified, overall inter-rater agreement was 88.4% (range = 62.5%–100%). Within these articles, inter-rater agreement for each Reichow et al. social validity category was as follows: (a) socially relevant Dependent Variables = 84.4%, (b) time- and cost-effective = 88.8%, (c) persons with and without disabilities = 93.3%, (d) clinically significant behavioral change = 82.2%, (e) consumers satisfied with the results = 88.8%, (f) Independent Variable by people in typical context = 86.6%, and (g) a natural context = 84.4%. When inter-rater agreement was not reached, the raters discussed how they coded the specific information in the article and whether it fit the established coding criteria. After independently re-rating these articles, consensus was reached in all cases.

Results

IDEAL Model Alignment With NSP and NPDC

Table 1 contains a list of the 31 socially validated IDEAL Model autism interventions that correspond directly with EBPs and emerging treatments empirically demonstrated to be effective for individuals with ASD. As noted above, each IDEAL Model intervention component was systematically determined to represent the same specific EBP or treatment/package validated by the NPDC and NAC, respectively. Complete verbatim descriptions of the IDEAL Model Intervention Components in Table 1 may be found in Callahan et al. (2008). The social validation ratings on the 31 interventions range from 5.70 to 6.69 (M = 6.17) on a 7-point scale, indicating a high to very high level of support by key consumers for the use of these treatments in autism programming within schools, clinics, and homes.

Table 1 includes a designated model affiliation associated with each intervention, based on earlier research (Callahan et al., 2010). Callahan and colleagues previously analyzed whether socially validated autism interventions were associated with ABA, TEACCH, both models, or neither model. Prior results suggested the primary consumers of autism programming value intervention components inherent within both ABA and TEACCH above treatments associated with either model alone.

Several of the socially validated EBPs were identified by the NSP as emerging, versus established, practices. These EBPs were “Social Skills Training Groups/Social Skill Package,” “Computer Aided Instruction/Technology-Aided Instruction and Intervention,” “Picture Exchange Communication,” and “Extinction.”

Four established EBPs had more than one corresponding IDEAL Model intervention component. “Antecedent-Based Intervention/Antecedent Package” was represented by six practices socially validated within the IDEAL Model, including the use of specific motivational procedures, student preferences, stimulus control, physical modifications to the instructional environment, and others. The effective use of “Reinforcement/Behavioral Package” was supported by three socially validated interventions.

Social Validity Within NSP and NPDC EBPs

As can be seen in Table 1, the secondary analysis identified additional evidence of social validity for all of the NSP and NPDC EBPs aligning with the IDEAL Model. Across all EBPs, the mean percentage of articles in which the authors directly reported evidence of social validity was 26.7%. However, the percentage of articles in which the authors reported social validity was variable, ranging from a low of 2.3% for the use of extinction to 74.2% for the use of Pictorial Story Board Simulations/Social Stories (see Table 1).

Of the Reichow et al. (2011) categories, researchers most often addressed social validity by reporting evidence within the categories of “Consumers Satisfied with the Results,” “Clinically Significant Behavioral Change,” and “Socially Important Dependent Variables.” Across all EBPs, more than 73% of the research articles in which social validation evidence was present addressed consumer satisfaction with the specific EBP, while approximately half of the articles directly addressed whether the intervention targeted socially important variables, and whether the treatment had clinically significant benefits. Relatively few researchers addressed the time and cost-effectiveness of interventions, and where or with whom the interventions were implemented. Table 2 shows the overall mean percentage of articles containing social validation evidence for each of the Reichow et al. (2011) categories.

Table 2.

Mean Percentage of Articles Reporting Social Validity Within Reichow, Doehring, Cicchetti, and Volkmar (2011) Categories Across All EBPs.

	Socially important dependent variables	Time- and cost-effective	Individuals with/without disabilities	Clinically significant behavioral change	Consumers satisfied with the results	IV by people in typical contact	A natural context
Mean % of articles	48.4	27.4	11.5	55.1	73.3	17.4	19.7

Note. EBPs = evidence-based practices.

Analyses of social validity of the EBPs and emerging treatments by their affiliated model (i.e., ABA, TEACCH, or BOTH ABA and TEACCH) were conducted. EBPs associated with the TEACCH Model (N = 6; M = 48.9%) had a much higher overall percentage of articles addressing social validity than treatments associated with the ABA approach alone (N = 13; M = 17.0%) or those representing BOTH ABA and TEACCH (N = 12; M = 26.1%). The results of a one-way, between-subjects analysis of variance indicated a statistically significant difference between TEACCH and ABA, and between TEACCH and the BOTH group of interventions, F(2, 28) = 12.949, p < .000, MS_error = 161.522, α = .05. The analysis of variance did not identify a statistically significant difference between the ABA treatments and those in the BOTH category.

Discussion

This research provides new information for consumers of autism interventions. For the first time, the literature addressing EBPs and social validity has been systematically integrated to help parents, educators, therapists, and others access research findings that include both evidence of effectiveness and validation of support for the goals, procedures, and outcomes of autism treatments. As a result, service providers and administrators now have a menu of research-validated interventions that can serve as a blueprint for treatment implementation, strategic planning, and program evaluation to improve outcomes, establish training priorities (Hughes, Combes, & Mehta, 2012), and build effective communities of practice (Cook & Odom, 2013; Hall, 2015).

Adding a social validation component to efforts by the NAC, NPDC, and others to identify and define EBPs supports long-standing and renewed calls for making social validity a routine part of program application and evaluation (Carter, 2010; Foster & Mash, 1999; Kazdin, 1977; Reichow, 2011; Schwartz & Baer, 1991; Wolf, 1978). Recommendations to refine and expand social validation in behavior change research have been made and, some might argue, ignored, for decades beginning with the seminal work of Kazdin (1977) and Wolf (1978). Fifteen years later, Kennedy (1992) assessed that only 20% of articles published in the journals Behavior Modification and the Journal of Applied Behavior Analysis addressed measures of social validity, leading to additional questions about how best to define, conceptualize, and measure the construct of social validation. The results of our research suggest the leading behavioral journals continue to report a low rate of social validity measurement in their autism treatment studies. For example, of 182 articles published in the Journal of Applied Behavior Analysis, only 19 studies (10.4%) directly reported the measurement of social validation. If social validation research is to have a maximum impact on improving the effectiveness of applied interventions, researchers must make renewed efforts to address fundamental issues about “when to use social validity measures, and what measures are most appropriate” (Kennedy, 1992, p. 154). In this study, the low rates of social validity observed in several of the Reichow categories underscores the continuing need for exploring potentially important differences in the way researchers value, select, implement, and report social validation measures.

It is important to note that establishing the social validity of any particular treatment is not a substitute for fully examining the intervention’s efficacy in other relevant ways (Schwartz & Baer, 1991). Further evaluation is necessary to help specify the salient steps or components of EBPs which result in maximum impact and widespread adoption by practitioners—important elements of implementation science (Odom, 2009). In addition, research to investigate interactions between key factors, such as the age and severity of students and clients, and the types of treatment settings, is warranted.

Limitations and Future Directions

Given the processes implemented in previous studies leading to this research, wherein researchers solicited social validation ratings of autism intervention components with widely disseminated positive outcomes, it is not surprising key stakeholders would broadly support their use. Thus, additional research is needed to extend these preliminary findings to assess consumer satisfaction with the goals, procedures, and outcomes of a larger array of practices in homes, schools, and clinics. In addition, the conservative approach used by this research team to identify evidence of social validity undoubtedly resulted in under-reporting the percentage of NSP and NPDC articles containing social validity evidence. For example, most of the articles reviewed detailed the results of interventions with outcomes that could be considered to be clinically significant. However, because no effort was made by the authors to directly address the concept of social validity, these articles were not coded as providing evidence of social validation. If future research was conducted with a broader definition of what constitutes social validity, the results would likely indicate higher levels of social validation than reported here. Furthermore, these results are limited to only those studies included in the NSP and NPDC reviews. It is possible different levels of social validation could be reported in autism research not cited by the NSP and NPDC. In addition, our analyses of the combined articles including both the NAC packages (e.g., the 228 articles within the Behavioral Package) with the more discrete NPDC treatments (e.g., the 10 articles for Functional Behavior Assessment [FBA]) are problematic. In this example, with only two articles in common, the majority of the Behavioral Package studies clearly targeted interventions other than FBA. Future research should attempt to identify specific articles within the larger NAC packages, which more closely match the corresponding treatments in the NPDC and IDEAL research. Finally, an important future direction for autism treatment research is to maximize the potential value of lists of effective interventions such as those presented here and disseminated by the NAC and NPDC. It is potentially problematic for service providers that the listed interventions have been classified using a variety of conceptualizations and methods. Autism researchers should identify inherent problems with such classification schemes and move toward investigating common principles and practices that underlie the effective application of evidence-based treatments.

Conclusion

These results affirm that empirically demonstrated autism treatments believed to be important by parents, teachers, therapists, and administrators have evidence of social validity. It is encouraging that all of the EBPs identified as established, confirmed, or emerging by the NAC and NPDC have at least a minimal level of social validation to complement the empirical evidence of effectiveness. At the same time, with less than 27% of the reviewed research articles directly addressing social validity, and in light of the wide range of reported social validity among EBPs, it is clear that the overall evidence for social validation remains limited. In addition, the relatively low rates of social validity for some treatments considered to be standard parts of comprehensive autism programming (e.g., the use of positive and differential reinforcement, Discrete Trial Training, and extinction) suggests much more work needs to be accomplished in this important area.

Although it is beyond the scope of this article, we support the future development of improved processes, including standardized frameworks for socially validating EBPs in autism intervention (Schlosser, 1999). For now, however, immediate actions can include consideration of policy changes by the editors of journals in applied research, which would require authors to directly and clearly report how social validity was measured. Researchers themselves should also begin to include descriptive social validity information within all intervention research manuscripts submitted for publication.

As part of the evolution of autism research and intervention from a focus on what works to why and how specific treatments can be implemented with fidelity to bridge the research-to-practice gap, it is important that researchers and practitioners consider social validity when making instructional and therapeutic decisions.

Footnotes

Acknowledgements

The authors wish to thank the expert validators for their valuable assistance with this research.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Alberto

P. A.

Troutman

A. C.

(2008). Applied behavior analysis for teachers (8th ed.). Columbus, OH: Pearson.

American Occupational Therapy Association. (2010). Occupational therapy code of ethics and ethics standards (2010). Retrieved from http://www.aota.org/Practice/Ethics.aspx

Behavior Analyst Certification Board. (2010). BACB guidelines for responsible conduct for behavior analysts. Retrieved from http://www.bacd.com/index.php?page=57

Bodfish

J. W.

(2004). Treating the core features of autism: Are we there yet? Mental Retardation and Developmental Disabilities Research Reviews, 10, 318–326.

Callahan

Henson

Cowan

(2008). Social validation of evidence-based practices in autism by parents, teachers, and administrators. Journal of Autism and Developmental Disorders, 38, 678–692.

Callahan

Shukla-Mehta

Magee

Wie

(2010). ABA versus TEACCH: The case for defining and validating comprehensive treatment models in autism. Journal of Autism and Developmental Disorders, 40, 74–88.

Carter

S. L.

(2010). The social validity manual: A guide to subjective evaluation of behavioral interventions. Boston, MA: Academic Press.

Cook

B. G.

Cook

C. C.

(2011). Unraveling evidence-based practices in special education. The Journal of Special Education, 47, 71–82.

Cook

B. G.

Odom

S. L.

(2013). Evidence-based practices and implementation science in special education. Exceptional Children, 79, 135–144.

10.

Council for Exceptional Children. (2013). Special education professional ethical principles and practice standards for special education professionals. Retrieved from http://www.cec.sped.org/Standards/Ethical-Principles-and-Practice-Standards?sc_lang=en

11.

Dickson

C. A.

MacDonald

R. P. F.

Mansfield

Guilhardi

Johnson

Ahearn

W. H.

(2014). Social validation of the New England Center for Children-Core Skills Assessment. Journal of Autism and Developmental Disorders, 44, 65-74. doi:10.1007/s10803-013-1852-5

12.

Dingfelder

H. E.

Mandell

D. S.

(2011). Bridging the research-to-practice gap in autism intervention: An application of diffusion of innovation theory. Journal of Autism and Developmental Disorder, 41, 597–609.

13.

Foster

S. L.

Mash

E. J.

(1999). Assessing social validity in clinical treatment research: Issues and procedures. Journal of Consulting and Clinical Psychology, 67, 308–319.

14.

Gersten

Fuchs

L. S.

Compton

Coyne

Greenwood

Innocenti

M. S.

(2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71, 149–164.

15.

Hall

L. J.

(2015). Sustaining evidence-based practices by graduated special educators of students with ASD: Creating a community of practice. Teacher Education and Special Education, 38, 28–43. doi:10.1177/0888406414558883

16.

Horner

R. H.

Carr

E. G.

Halle

McGee

Odom

Wolery

(2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165–179.

17.

Hughes

H. L.

Combes

B. H.

Mehta

S. S.

(2012). Managing autism: Knowledge and training in autism spectrum disorders among special education administrators in Texas. Journal of Special Education Leadership, 25(2), 90–98.

18.

Individuals With Disabilities Education Improvement Act (2004). 20 U.S.C., § 1415 (reauthorization of the Individuals With Disabilities Education Act of 1990).

19.

Kazdin

A. E.

(1977). Assessing the clinical or applied significance of behavior change through social validation. Behavior Modification, 1, 427–452.

20.

Kennedy

C. H.

(1992). Trends in the measurement of social validity. The Behavior Analyst, 15, 147–156.

21.

Kern

Manz

(2004). A look at current validity issues of school-wide behavior support. Behavioral Disorders, 30, 47–59.

22.

Lilienfeld

S. O.

Marshall

Todd

J. T.

Shane

H. C.

(2015). The persistence of fad interventions in the face of negative scientific evidence: Facilitated communication for autism as a case example. Evidence-Based Communication Assessment and Intervention, 8, 62–101. doi:10.1080/17489539.2014.976332

23.

Mesibov

G. B.

Shea

(2011). Evidence-based practices and autism. Autism, 15, 114–133.

24.

National Autism Center. (2009). Addressing the need for evidence-based practice guidelines for autism spectrum disorders (National Standards Report: The National Standards Project). Randolph, MA: Author.

25.

National Professional Development Center. (2013). The NPDC on ASD and the National Standards Project. Retrieved from http://autismpdc.fpg.unc.edu/evidence-based-practices

26.

No Child Left Behind (NCLB) Act of 2001, Publ. L. No. 107–110, § 115, Stat.1425 (2002).

27.

Odom

S. L.

(2009). The tie that binds: Evidence-based practice, implementation science, and outcomes for children. Topics in Early Childhood Special Education, 29, 53–61. doi:10.1177/0271121408329171

28.

Odom

S. L.

Collet-Klingenberg

Rogers

Hatton

(2010). Evidence-based practices in interventions for children and youth with autism spectrum disorders. Preventing School Failure, 54, 275–282.

29.

Odom

S. L.

Cox

A. W.

Brock

M. E.

, the National Professional Development Center. (2013). Implementation science, professional development, and autism spectrum disorders. Exceptional Children, 79, 233–251.

30.

Qualtrics. (2013). Qualtrics (Version 13). Provo, UT: Author.

31.

Rapp

C. A.

Etzel-Wise

Marty

Coffman

Carlson

Asher

. . . Holter

(2010). Barriers to evidence-based practice implementation: Results of a qualitative study. Community Mental Health Journal, 46, 112–118.

32.

Reichow

(2011). Developments, procedures, and application of the evaluative method for determining evidence-based practices in autism. In Reichow

Doehring

Cicchetti

D. V.

Volkmar

F. R.

(Eds.), Evidence-based practices and treatments for children with autism (pp. 25–39). New York, NY: Springer.

33.

Reichow

Doehring

Cicchetti

D. V.

Volkmar

F. R.

(2011). Evidence-based practices and treatments for children with autism. New York, NY: Springer.

34.

Reichow

Volkmar

F. R.

Cicchetti

D. V.

(2008). Development of the evaluative method for evaluating and determining evidence-based practices in autism. Journal of Autism and Developmental Disorders, 38, 1311–1319. doi:10.1007/s10803-007-0517-7

35.

Schlosser

R. W.

(1999). Social validation of interventions in augmentative and alternative communication. Alternative and Augmentative Communication, 15, 234–247.

36.

Schwartz

I. S.

Baer

D. M.

(1991). Social validity assessments: Is current practice the state of the art? Journal of Applied Behavior Analysis, 24, 189–204.

37.

Wolf

M. M.

(1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11, 203–214.

38.

Wong

Odom

S. L.

Hume

K. A.

Cox

A. W.

Fettig

Kucharczyk

. . . Schultz

T. R.

(2014). Evidence-based practices for children, youth, and young adults with autism spectrum disorder. Chapel Hill: Frank Porter Graham Child Development Institute, Autism Evidence-Based Practice Review Group, The University of North Carolina.

39.

Wong

Odom

S. L.

Hume

K. A.

Cox

A. W.

Fettig

Kucharczyk

. . . Schultz

T. R.

(2015). Evidence-based practices for children, youth, and young adults with autism spectrum disorder: A comprehensive review. Journal of Autism and Developmental Disorders, 45, 1951–1966. doi:10.1007/s10803-014-2351-z