Abstract
Criteria for evaluating behavior support programs are changing. Consumer-based educational and behavioral programs, such as School-Wide Positive Behavior Support (SWPBS), are particularly influenced by consumer opinion. Unfortunately, the need for and use of social validity measures have not received adequate attention in the empirical literature related to these increasingly utilized behavioral programs. The purpose of this article is to illustrate the importance of social validity and to present an in-depth discussion of its history, merits, and practical applications relevant to the expanded scope of Positive Behavior Intervention and Support, specifically, SWPBS. In addition, the importance of social validity in bridging the gap between research and practice is discussed; current limitations are noted; and suggestions are made for future direction regarding this critical area of assessment.
Keywords
Criteria for evaluating behavior support programs are changing. Simply creating theoretically and technically sound intervention strategies is no longer sufficient: Strategic plans must align with the environment in which interventions are implemented (Albin, Lucyshyn, Horner, & Flannery, 1996). Researchers, implementers, and evaluators are increasingly urged to consider stakeholder opinion (Francisco & Butterfoss, 2007). Strategic plans must consider the needs and perceptions of those who are responsible for change (i.e., principals and administrators), for those who implement interventions (i.e., teachers, paraprofessionals, and parents), and for those who are impacted by interventions (i.e., students, teachers, and parents).
Faced with the current demands of implementing research-based practice to address behavioral and academic needs, school administrators and teachers are overwhelmed with a multiplicity of plans and strategies promising results, though not necessarily proven effective in their specific school and with their specific population of students (Miramontes, Marchant, Heath, & Fischer, 2011; Reimers, Wacker, & Koeppl, 1987).
In deciding which programs to implement, educators and researchers must carefully evaluate specific programs based on applicability to their specific population. To ensure successful program implementation, those in charge must consider more than statistical descriptions of the proposed program’s reliability and validity. Of paramount importance, those in charge must determine the particular program’s value from the perspective of specific stakeholders it proposes to serve.
Social Validity
In the current climate of evidenced-based intervention, we often lose sight that it is not solely the proposed intervention that leads to desired change, it is the buy-in of stakeholders and the fidelity of implementing evidence-based interventions that ensures success. The power of teacher buy-in increases the likelihood of successful implementation of interventions, particularly at the school-wide level (Biggs, Vernberg, Twemlow, Fonagy, & Dill, 2008; Stauffer, Heath, Coyne, & Ferrin, 2012).
Through evaluations of social validity (how well an implemented program is accepted and embraced by stakeholders), today’s researchers have at their disposal a broader range of data to better inform change, making improvements in previously unsuccessfully implemented social programs. Although polling stakeholder opinions is not a new concept, its methodology, particularly in educational settings, is still evolving. Thus, researchers must define who potential stakeholders are; what these stakeholders consider to be socially relevant goals, procedures, and outcomes; and how these stakeholders’ responses affect the integrity of program implementation.
In regard to defining who, according to E. G. Carr et al. (2002), stakeholders may include “teachers, employers, parents, friends, and the target person” (p. 5). Carr et al. also emphasized that stakeholders have evolved from “passive participants” to the role of “active participants” who “participate as partners to build the vision, methods, and success criteria” for interventions (p. 8). Similar to Carr et al.’s terminology, throughout this article, the terms stakeholder and consumer are used interchangeably.
Social Validity of School-Based Behavioral Interventions
Consumer-based educational and behavioral programs, such as School-Wide Positive Behavior Support (SWPBS), are particularly influenced by consumer opinion. Specifically, Positive Behavior Intervention and Support (PBIS) emphasizes data collection and analysis to inform decision making (Sugai et al., 2000). To ensure implementation fidelity, some SWPBS programs are currently collecting treatment fidelity data through school-wide evaluation assessments, such as the School-Wide Evaluation Tool (SET; Horner et al., 2004). Unfortunately, fidelity data are often collected and reviewed without considering social validity data (Lane, Kalberg, Bruhn, et al., 2009; Miramontes et al., 2011).
Subsequently, unheeded directives to collect social validity data in tandem with program implementation and evaluation data have resulted in a poorly defined stakeholder voice, with subsequent data failing to reflect stakeholders’ input regarding socially relevant goals, procedures, and outcomes. For example, J. E. L. Carr, Austin, Britton, Kellum, and Bailey (1999) reviewed all full-length articles published in the Journal of Applied Behavior Analysis (JABA) between 1968 and 1998, specifically searching for studies that included measures of both treatment outcome and treatment acceptability. Considering publications during this 31-year period, the purpose of Carr et al.’s review was to determine the trend of researchers collecting social validity data within applied behavior analysis research, specifically in analog and naturalistic settings. Their findings indicated that only 12% of JABA studies between 1968 and 1998 reported treatment outcome data related to social validity, with a stronger trend beginning in the 1970s.
In a more recent article, Clarke and Dunlap (2008) specifically tracked research involving data-based interventions for students with disabilities, determining which articles included measures of social validity. They reviewed articles published between 1999 and 2005 in three journals: Journal of Positive Behavior Interventions (JPBI), JABA, and Education and Training on Mental Retardation and Developmental Disabilities (ETMRDD). Their findings indicated that of the total number of published research articles meeting their criteria for inclusion, 31% of JPBI’s intervention articles included some measure of social validation, as compared with 3% of JABA’s and 20% of ETMRDD’s intervention articles.
Based on the findings of J. E. L. Carr et al. (1999) and Clarke and Dunlap (2008), the majority of researchers publishing in the area of behavioral interventions do not typically report social validity data. Given the importance of stakeholder buy-in, initial and ongoing consideration of social validity is of paramount importance. These literature reviews should send a strong message to researchers and practitioners who are involved in SWPBS: We must do a better job in collecting, analyzing, and reporting social validity data.
Barriers to Collecting Social Validity Data
Moving beyond the mere tally of articles including social validity, Kern and Manz (2004) discussed limitations of current social validity data and collection procedures. Concurring with J. E. L. Carr et al. (1999), these researchers also noted a limited use of social validation measures. In addition, they identified shortcomings in the few social validation procedures that were utilized. Similar across all levels of the SWPBS model (primary, secondary, and tertiary levels), Kern and Manz noted inconsistent and inadequate definitions of the term stakeholders, inadequate representation of all stakeholders, and inadequate assessments investigating SWPBS goals, procedures, and outcomes. In particular, Kern and Manz identified significant limitations in the range of stakeholders included in social validation procedures and minimal social validity assessments conducted with respect to secondary- and tertiary-level models within SWPBS models. However, from Lane, Kalberg, Bruhn, et al.’s (2009) perspective, less attention has been directed at assessing primary prevention efforts within SWPBS models. Nonetheless, Kern and Manz align with Lane, Kalberg, Bruhn, et al. in supporting Carr et al.’s concern that social validity continues to be overlooked within behavioral intervention research.
This limited assessment of social validity, particularly with school-based efforts, is problematic for two reasons. First, without ongoing methods for soliciting stakeholder feedback on procedures and outcomes, researchers lack information for predicting the acceptability of implemented programs. Second, if social validity measures are not routinely utilized, the development and quality of these measures are jeopardized. Subsequently, with limited market demand, social validity measures will not be readily available for use (J. E. L. Carr et al., 1999). Unfortunately, the need for and use of social validity measures have not received adequate attention.
Increasingly promoted as an effective model for school-based behavioral intervention, the drive to implement PBIS is gaining momentum across the educational landscape (E. G. Carr, 2007). As such, PBIS is primed for stakeholder evaluation. In preliminary stages of implementing SWPBS, polling stakeholder opinion offers particular benefits. Stakeholder opinion and feedback are critical in establishing an information baseline for future analytical comparisons, as well as informing current program implementation. In addition, social validity data offer feedback on which to base suggestions for future improvements.
Purpose
The purpose of this article is to illustrate the importance of social validity assessment and to present an in-depth discussion of its history, merits, and practical applications in respect to SWPBS. In addition, this article discusses the importance of social validity in bridging the gap between research and practice, identifies current limitations, and offers suggestions for future direction regarding this critical area of assessment.
Definition of Social Validity
In its simplest form, social validity is a measure of how well a social program is embraced by those who are targeted to benefit from it. In 1978, Wolf first introduced the concept of social validity as a measure of three distinct areas: (a) the social significance of identified treatment goals, (b) the social appropriateness of procedures utilized in achieving treatment goals, and (c) the social importance of research effects and outcomes.
The first and primary intention for establishing social significance of treatment goals is to evaluate whether identified goals serve clients. Goals that do not serve clients are considered socially invalid (Kern & Manz, 2004). Similarly, treatment acceptability is concerned with social significance of treatment procedures, eliciting information regarding the community’s judgment that treatment methodology is appropriate, fair, and reasonable (Kazdin, 1977). To be perceived as socially valid, treatment steps must be feasible, simple, and cost-effective (Kern & Manz, 2004). Finally, measuring the social importance of treatment outcomes considers whether the results (outcomes) are meaningful to key stakeholders.
Based on public opinion, social validity assessments are designed to evaluate the potential success and viability of a social program (Kazdin, 1977; Wolf, 1978). In the business world, consumer satisfaction has long been assessed to predict the likelihood of a product’s success (Elliott, 1988; Elliott, Witt, Galvin, & Peterson, 1984; Kazdin, 1977; McMahon & Forehand, 1983). In reviewing the literature, however, Schwartz and Baer (1991) noted that the term social validation had strayed from its original meaning. Across time, social validation began to describe a variety of assessments that focused on evaluating the effectiveness of interventions. As originally prescribed, a measure of social validity assessed a program’s viability, determining whether a program had a reasonable chance of succeeding based on participants’ perceptions (Schwartz & Baer, 1991). As such, social validity data have been described and measured from two different perspectives, (a) participants’ impressions of proposed interventions and (b) the success of interventions, with measured progress serving as an indicator of social validity.
Schwartz and Baer (1991) further noted that original social validity assessments involved a two-step process. First, an accurate and representative sample of stakeholders was assembled and their opinions solicited and collected. Second, to ensure successful implementation and ultimately stakeholder satisfaction, the stakeholder opinions were used to inform both the content of the intervention and the manner in which it was implemented. This process guaranteed that the intervention was valued by the affected community of stakeholders.
Schwartz and Baer (1991) specifically identified four categories of stakeholders: (a) direct stakeholders, (b) indirect stakeholders, (c) members of the immediate community, and (d) members of the extended community. Direct stakeholders are those individuals directly affected by the product or treatment: the primary stakeholders. For school-based interventions, the direct stakeholders are the students (Gresham & Lopez, 1996; Schwartz & Baer, 1991). Indirect stakeholders are those individuals who purchased the intervention and/or imposed the intervention onto direct stakeholders. Examples of indirect stakeholders include parents, teachers, and school administrators working within the school system (Gresham & Lopez, 1996; Schwartz & Baer, 1991). Members of the immediate community may or may not be directly involved in the process of program implementation, but they regularly interact with both the direct and indirect stakeholders (Schwartz & Baer, 1991). These individuals include bus drivers, nonteaching school personnel, and volunteers. Finally, members of the extended community are stakeholders who live within the school district boundaries but do not regularly interact with the direct or indirect stakeholders (e.g., tax payers, school boards, district employees working in other schools, and law enforcement personnel; Gresham & Lopez, 1996).
Once an accurate and representative sample of stakeholders is assembled and polled, the second step is to use the resulting information to determine the program’s value to the community. The main goal of this process is not to gather false praise for a proposed program, but to gather useful information about potential program weaknesses, implementation problems, and stakeholder perceptions regarding the program’s future success and impact (Schwartz & Baer, 1991). Behavioral programs that tap into what matters most to key stakeholders have the potential to be as successful in practice as they are in theory (Kazdin, 1977).
Ingredients of Social Validity
Researchers should continuously strive to evaluate effectiveness of proposed and applied interventions. Professionals are responsible for the quality of research and for its impact on those directly affected by it. Carnine (1997) noted that to effectively judge research data, three areas should be evaluated: trustworthiness, usability, and accessibility. Trustworthiness is defined as the confidence practitioners have in research findings that support potential programs considered for implementation. Usability refers to the practicality of implementing research-based practice in real-world settings. In addition, accessibility designates the extent to which research findings are available to interested stakeholders.
Educators have legitimate concerns about the quality of educational research findings in terms of applicability within real classrooms, considering the complexity of school dynamics (Carnine, 1997). In addition to technical considerations that build confidence in these programs (e.g., methods are research-based and outcome assessment data support proposed interventions), educators must consider the likelihood of a given intervention or social program being accepted and used by practitioners, administrators, and other knowledgeable stakeholders (Kauffman, 1996). Furthermore, educators want and need basic information about proposed program goals and practical guidelines to achieve those goals.
Research findings that support specific programs and intervention strategies must focus on topics that are important to educators. In addition, information must be clearly communicated. If goals and research support for these programs do not align with educators’ top priorities, then programs will not be successfully implemented and supported over time by educators, regardless of affirming data. In respect to introducing SWPBS, the necessity of social validity cannot be underestimated (Carnine, 1997).
Furthermore, Carnine (1997) specified how measures of social validity assess the alignment of programs and research with factors important to educators—trustworthiness, usability, and accessibility. The mere act of valuing and seeking stakeholder opinion sets the foundation for a program’s trustworthiness. When researchers specifically evaluate a program’s alignment with key stakeholders’ priorities, the resultant findings can indicate the likelihood of that program being successfully utilized by educators. In other words, social validity carries the power to indicate the likelihood of a program’s potential utility. Finally, if social validity assessments indicate a program’s alignment with educators’ priorities, then these programs have a greater potential to reach across school communities, increasing a program’s accessibility to all stakeholders.
As previously outlined, social validity carries considerable potential for bridging the gap between research and practice. However, as research-based interventions and programs are typically established in a top-down pattern (Child Trends, 2008), this research-to-practice gap is steadily increasing (Kern & Manz, 2004), plaguing and stifling potential progress for researchers and educators. These conditions emphasize the critical importance of social validity assessments in educational research, including SWPBS efforts.
Tailoring social validity assessments to target key stakeholders’ areas of greatest concern empowers these individuals as they become part of the process and make informed choices regarding interventions. Prior to implementing a program, social validity measures gauge key stakeholders’ level of support for a particular program and identify underlying resistance to implementing programs and interventions. Schwartz (1991) noted that stakeholders who make informed choices are often the most satisfied, and satisfied stakeholders strengthen program viability. Ultimately, research-based interventions that are both teacher/school friendly and aligned with student needs are successful because these interventions target the needs of key stakeholders.
SWPBS
As researchers’ knowledge of behavioral and academic interventions expands, social validity assessments also contribute important information about the makeup of successful school environments. Prominent among recent research-based practices is the multitiered model of SWPBS. In contrast to the traditional punitive and reactionary methods commonly associated with controlling students’ behavioral challenges, PBIS offers constructive ways to preemptively address the function of behaviors in both environmental and individual contexts.
Sugai et al. (2000) reviewed and defined PBIS as “a general term that refers to the application of positive behavioral interventions and systems to achieve socially important behavior change” (p. 133). Of particular note is their mention of the model’s central goal of enacting behavior changes that are socially significant. By seeking to deliver socially significant results, this particular model is desirable for a variety of stakeholders within schools (e.g., teachers, students, parents, and administrators), considering the needs of each participant when determining intervention goals.
Rather than reactively addressing complications subsequent to problematic behavior, SWPBS interventions are designed to proactively address potential behavioral challenges. In addition, SWPBS is based on data-driven measures and relies heavily on data to guide the decision-making process. Sustained use of research-validated practices focuses on maximizing student behavioral change (Gresham, 2004; Sugai & Horner, 2002). With a foundational model of team problem solving, SWPBS functions on a three-tiered conceptual approach in identifying and addressing problems. The overall model utilizes primary, secondary, and tertiary levels of intervention.
The primary level is also labeled the universal level due to its applicability across all students, settings, and staff (Kern & Manz, 2004). Interventions on this level are designed to target students who are equipped with general education skills (George, White, & Schlaffer, 2007). The secondary level is designated as the targeted interventions level because it provides services for those students who have been classified as at risk of problem behaviors. At this level, services are more specialized and tailored to go beyond the services provided at the primary level. The tertiary level in this three-tier model addresses complex ongoing behavior problems that affect approximately 1% to 5% of the school population (George et al., 2007). Supports at this level are individualized to provide students with intensive intervention.
SWPBS and Social Validity
One of the fundamental philosophies of PBIS is the idea that although humanistic values should not replace empiricism, these values should certainly inform empiricism (E. G. Carr et al., 2002). Stakeholder participation is fundamental to the success of SWPBS. Moving from the traditional expert-driven methodology to a stakeholder-driven methodology, SWPBS encourages a collaborative system (Sugai et al., 2000) which functions as a support network, undoubtedly contributing to its success with systems-level change (E. G. Carr et al., 2002; Sugai et al., 2000). In SWPBS, for example, decisions are developed, implemented, and evaluated by the school system as a whole, fostering ownership and social validity among its key stakeholders (Scott, 2007). This direct involvement encourages stakeholders to make informed choices which, as mentioned earlier, contribute to the program’s viability.
The manner in which key stakeholders are integrated into planning, implementation, and evaluation processes involved in SWPBS makes this a unique system (E. G. Carr et al., 2002; Scott, 2007; Sugai et al., 2000). It is therefore no surprise that key stakeholders’ opinions should be evaluated when considering future successful implementation and development of SWPBS programs. Social validity assessments are vital components in overall evaluations of SWPBS programs because this assessment piece informs researchers on a fundamental attribute of SWPBS implementation and development—stakeholder participation.
Assessment of social validity in SWPBS
The assessment process for all of the important areas of social validity mentioned by Wolf (1978), Carnine (1997), and Kauffman (1996) can be carried out by using rating scales or surveys that collect self-reported information (Finn & Sladeczek, 2001). Some of the research in the area of positive behavior support has attempted to incorporate social validity assessments (Bohanon et al., 2006; Houchins, Jolivette, Wessendorf, McGlynn, & Nelson, 2005; Lyst-Miltich, Gabriel, O’Shaughnessy, Meyers, & Meyers, 2005; McCurdy, Mannella, & Eldridge, 2003). Because significant information is gathered using self-reported data, it is important to carefully consider the fidelity of these measures and their corresponding data and to take appropriate steps toward successfully conducting such assessments.
For example, McCurdy et al. (2003) conducted teacher perception surveys regarding a SWPBS model implemented in an ethnically and racially diverse inner-city elementary school. The survey was designed to gather data on teachers’ self-reported perceptions regarding the SWPBS model, including the acceptability of this new model, overall satisfaction with the model, generalizability of the model, and positive outcomes. The staff satisfaction questionnaire was administered to school staff during each of the first 2 years of implementation. Results showed high levels of teacher satisfaction and support for the program’s continuation. Although researchers gathered favorable data regarding teachers’ acceptance of this new model, feedback was incomplete. Several key stakeholders and individuals with indirect involvement were not surveyed. Expanding social validity assessment to include more than one group of stakeholders would have provided additional information to further bridge the gap between research and practice.
Further efforts to bridge the research-to-practice gap have included collecting social validity data from stakeholders at the universal level of SWPBS implementation. For example, Bohanon et al. (2006) measured staff perceptions associated with the process of implementing a new SWPBS program in one of Chicago’s urban high schools. More specifically, this study was conducted to measure the new program’s social validity. Each year during the first 2 years of implementation, researchers administered the Effective Behavior Support (EBS) Survey, designed to measure educators’ current perceptions of SWPBS.
The survey was administered in small groups composed of key personnel—defined by the researchers as those “who came into direct contact with the students during the school day” (Bohanon et al., 2006, p. 134). Year 1 to Year 2 results demonstrated a strong increase in support for the priority of the program. This information was later used in selecting SWPBS priorities for action planning during Years 3 and 4.
In discussing limitations and future directions, Bohanon et al. (2006) commented on the scarcity of research data concerning planning and implementation at more individualized levels of SWPBS. This observation emphasizes the need for future research to examine prevention and intervention efforts associated with the second and third tiers of SWPBS. Furthermore, future endeavors should be guided by feedback gathered from stakeholders involved in second- and third-tier support, specifically social validity assessments targeting these levels of intervention. This is critical because, depending on the level of intervention, teachers’ interests and priorities may vary. Understanding these differences will affect the effectiveness of interventions implemented on each particular level of intervention.
In an interesting effort to gather stakeholder input, Kincaid, Childs, Blasé, and Wallace (2007) used a systematic process to understand the barriers and facilitators of implementing a SWPBS initiative in Florida. By 2005, Florida’s behavioral support project had implemented SWPBS in more than 100 schools across the state. From these efforts, several questions emerged. In an effort to provide answers, researchers invited eight “high implementing” SWPBS schools to participate in a group process. Each group was initially instructed not to interact with other members while answering two open-ended questions. Group members were to individually write responses to the following questions: (a) “What have been the barriers to implementing SWPBS in your school or district?” and (b) “What has facilitated the implementation of SWPBS at your school or in your district?” (Kincaid et al., 2007, p. 176). After writing their responses, group members took turns reading their answers aloud in a round-robin format.
Summarized responses describing perceived barriers included a total of 21 themes, more than half of which reflected issues related to staff buy-in. More specifically, lack of staff buy-in was characterized by poor communication, resulting in miscommunications and confusion surrounding simple procedures and desired goals. By identifying specific factors impacting implementation of SWPBS, the research team was able to successfully outline strategies to improve the program’s success. This valuable information gathered by Kincaid et al. (2007) demonstrates the power of social validity assessments in identifying and addressing problems and shaping more effective programs.
Recent examples of successful implementation of SWPBS
In recent years, researchers have increased efforts to consider and assess the social validity of PBIS by investigating a broader array of contexts, expanding measurement techniques, and increasing the scope of PBIS. Intending to glean reliability and validity data for the Primary Rating Intervention Scale, a Likert-type teacher survey included 17 questions regarding social validity. This survey was adapted from an earlier version, the Intervention Rating Profile-15 (Witt & Elliott, 1985). Lane, Kalberg, Bruhn, et al. (2009) pre-assessed the social validity of SWPBS slated for implementation in a cluster of schools. Prior to implementing the program, Lane, Kalberg, Bruhn, et al. surveyed 617 teachers in 11 elementary schools, 3 middle schools, and 5 high schools. This research endeavor was unique in that it provided a standardized approach for collecting social validity data at the universal level; social validity data were gathered prior to implementing SWPBS; and data were gathered across multiple settings serving K-12 students. Results indicated strong social validity across educators serving all three age groups of students (elementary, middle school, and high school).
Another group of researchers conducted a descriptive evaluation with an early childhood, program-wide SWPBS model using a mixed triangulation design (Frey, Park, Browne-Ferrigno, & Korfhage, 2010). Minimal early childhood initiatives have been conducted or empirically investigated where the initiative’s key elements mirror SWPBS; therefore, the findings from this study are considered unique. Focus groups were conducted to assess social validity of the proposed program. Discussion leaders followed a 9-question interview protocol (Frey et al., 2010). Interviews were taped, transcribed, and coded by theme. In addition to focus groups, teacher surveys (Inventory of Universal Practices for Promoting Children’s Social Competence) were administered to investigate teachers’ perceptions of the degree to which practices were being implemented. Classroom observations also provided another piece of evidence verifying treatment integrity. This study’s findings indicated that although stakeholders typically conveyed support of programs goals, they struggled to implement interventions and follow procedures as prescribed. They also acknowledged the need to modify procedures and assessment efforts to better align with teachers’ capacity to implement interventions and to be adequately supported by school resources.
On the other end of the spectrum, Farkas and colleagues (2011) evaluated the impact of SWPBS elements within an alternative setting for students in Grades 5 to 12 identified with emotional behavior disorders or identified as other health impaired. Both student and teacher measures indicated that SWPBS Tier 1 strategies produced meaningful outcomes.
Miramontes et al. (2011) added to this body of research with their recent efforts in measuring perceptions of school personnel regarding the goals, procedures, and outcomes of a statewide SWPBS model. Results from the surveyed stakeholders were positive, demonstrating high levels of overall satisfaction. However, based on participants’ feedback, several areas were identified for improvement, including the amount of paperwork required for successful implementation and the practicality of implementing and adhering to program procedures.
The recent works of Lane, Kalberg, Bruhn, et al. (2009); Frey et al. (2010); Farkas et al. (2011); and Miramontes et al. (2011) demonstrate growing efforts in the area of assessing social validity when implementing SWPBS. Another commonality across these four research studies was that each research group attempted to concurrently collect treatment fidelity data alongside social validity data. These efforts are unique in that treatment fidelity data are infrequently reported within behavioral intervention research, particularly in combination with social validity data (Miramontes et al., 2011). In future SWPBS research endeavors, Miramontes et al. recommended an increased emphasis on collecting both treatment fidelity data and social validity data.
Social Validity Survey Design
Based on recent literature, successful implementation of SWPBS relies heavily on a number of individual system dynamics that combine to create the appropriate environment conducive to change. This literature demonstrates that although some studies have utilized social validity assessments in gauging stakeholders’ perceptions and buy-in regarding SWPBS (Lane, Kalberg, Bruhn, et al., 2009; Upreti, Liaupsin, & Koonce, 2010), more research is needed to increase use and strengthen utility of assessing social validity. Clearly, several untapped resources to measure social validity of SWPBS are available (Carter, 2010; Lane, Kalberg, Bruhn, et al., 2009; Lane, Kalberg, & Menzies, 2009). In addition, models of successful implementation of SWPBS are provided in publications by Farkas et al. (2011) and Lane, Kalberg, Bruhn, et al.(2009) for improving the implementation of SWPBS.
Considering that the inherent purpose of collecting social validity data is to solicit information directly from stakeholders, surveys are one of the most efficient ways to gather this type of data. The literature contains various recommendations for designing survey questions for social validity questionnaires. To examine how these questionnaires can remain true to the intentions of social validity assessments, the recommendations of Wolf (1978), Fawcett (1991), Reimers et al. (1987), and other noted social validity experts are to be discussed in the following paragraphs. In addition, specific types of questions and recommendations for measuring social validity and stakeholder satisfaction are reviewed.
Wolf (1978) proposed three questions that should guide the framework for designing social validity assessments:
Are the specific behavioral goals important and relevant to what the targeted stakeholders want?
Are treatment procedures acceptable to stakeholders?
Are stakeholders satisfied with both the predicted and unpredicted outcomes?
Wolf referred to these critical questions as the basis for assessing social validity.
In a seminal article, Fawcett (1991) extended Wolf’s framework and outlined considerations to increase the likelihood of attaining meaningful results from social validity assessments. First, Fawcett suggested that to assess the social significance of treatment goals and treatment outcomes, researchers must use descriptive labels such as the following: “important,” “complete,” or “representative.” Second, in assessing the appropriateness of procedures, Fawcett recommended the use of more global language such as “overall satisfaction” or “willingness to recommend to others.” Third, to minimize confusion with terminology, Fawcett advised assessors to prepare specific instructions for participating stakeholder judges (pp. 236–237).
Measuring judgments of social validity put forth by Wolf (1978) should include language outlined in the literature as conducive to targeting measures of acceptability. However, a good social validity questionnaire should begin with an item written to measure participants’ perceived understanding of the model’s goals and procedures. This is consistent with a suggestion made by Reimers et al. (1987) that assessing understanding should precede attempts to measure stakeholders’ perceptions of acceptability.
Once understanding is evaluated, questions intended to measure the first judgment of social validity (i.e., program goal acceptability) may be written using the terms important and acceptable, as outlined by Fawcett (1991). To measure the second judgment of social validity (acceptability of program procedures), questions should use the language “willingness to use, given time constraints” and “willingness to recommend to others” (Kern & Manz, 2004, p. 54). To measure the third judgment of social validity (program outcomes), questions should use terms such as “improved outcomes” and “made a positive impact” (Lane & Beebe-Frankenberger, 2004, p. 101).
In addition, Lane and Beebe-Frankenberger (2004) suggested writing items to specifically address secondary and tertiary levels of SWPBS. This is an important consideration because success may vary depending on the level of intervention and the unique needs of students, teachers, and parents associated with that level of intervention.
In addition to attending to the language used in surveys, Tourangeau, Couper, and Conrad (2004) suggested researchers consider a number of visual heuristics that may impact responses to survey questions. For example, spacing between descriptor options, order of the descriptors, and groupings of related questions were found to impact participants’ responses. To prevent visual heuristics from interfering with meaningful responses, social validity questionnaires should be designed with equal spacing between descriptors and with descriptors arranged with the least positive responses on the left, moving to more positive responses on the right. Furthermore, paper/pencil stakeholder surveys have demonstrated lower response rates than web-based surveys (Kamps et al., 1998; McCarthy & Shrum, 2000; Ransdell, 1996). Therefore, when researchers opt to conduct paper/pencil surveys, they should consider increasing response rates by offering rewards and other incentives (Kamps et al., 1998).
Current Limitations and Future Directions
Despite repeated recommendations to evaluate social validity, researchers continue to overlook this important source of data and fail to report information regarding social validity (Carnine, 1997; E. G. Carr, 2007; Gresham & Lopez, 1996; Kauffman, 1996; Kern & Manz, 2004; Lyst-Miltich et al., 2005; Schwartz & Baer, 1991; Scott, 2007). Furthermore, assessment and reporting of social validity often fail to reflect the critical purpose as originally outlined by Wolf (1978) and Kazdin (1977). To strengthen future research, researchers propose attending to the following details related to social validity: (a) clarify definitions of the term stakeholders; (b) ensure adequate representation of stakeholders; (c) conduct adequate assessments that investigate the program’s goals, procedures, and outcomes, including all levels of the SWPBS model (primary, secondary, and tertiary levels); and (d) track social validity across time, assessing social validity prior to program implementation (consumer buy-in), during implementation, and following an extended period of implementation (Kern & Manz, 2004; Lane Kalberg, Bruhn, et al., 2009). By attending to these recommended improvements, researchers will redirect the purpose of social validity assessments back to its original purpose—evaluating whether the goals of implemented programs best serve the client and impact positive and desired change (Kern & Manz, 2004).
To thoughtfully evaluate the goals of treatment programs and models such as SWPBS, researchers would be wise to evaluate stakeholders’ perceptions of treatment goals on an inclusive scale and ensure adequate representation of indirect stakeholders (teachers, administrators) responsible for primary stakeholders (students). To date, few publications demonstrate that these evaluations are being conducted (Lyst-Miltich et al., 2005; McCurdy et al., 2003).
Categorically speaking, key stakeholders should include parents, paraeducators, teachers, administrators, and related service providers (i.e., school psychologists, social workers, and school counselors). Results should be compared across these stakeholder groups, including those who are and are not actively involved with program development and implementation. Determining differences of social validity between these groups would contribute to the breadth and depth of information gathered from these reports.
Conclusion
Evaluating stakeholder opinions and perceptions is a critical piece of implementing and conducting SWPBS. By engaging in efforts to assess social validity, such as those described in this publication, researchers and practitioners collaborate to bridge the gap between research and practice. One potential outcome is that implementation of SWPBS strategies is sustainable because data (including perception data) document strategies as both effective and acceptable. A second outcome of engaging in these efforts is the increased potential for creating workable guidelines to inform meaningful change.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was partially funded from the Utah State PBIS Project—ABC-UBI.