Abstract
Social competence intervention studies published from 1970 to 2008 for preschoolers were reviewed for reports of social validity assessment. Analysis of 90 studies indicated that nearly 27% (n = 24) of studies reported at least one measure of social validity assessment for the goals (n = 7), procedures (n = 8), or effects (n = 19). The methods used for these assessments were described, including information about the participants, time of assessment, and results. Trends in the rate of social validity assessment, methodological concerns, and recommendations for further social validity assessments are discussed.
When applied behavior analysis was in its infancy, Wolf (1978) advised researchers that it was not enough to provide society with effective interventions to impact behavior but that researchers needed to determine whether society valued the interventions. Wolf argued that if consumers did not “like” or value an intervention, they would not use it, no matter how effective. Kazdin (1977) and Wolf first introduced the term social validity to mean the degree to which an intervention has social importance or is valued by consumers. Wolf recommended that consumers participate in social validation assessment measures for the following three dimensions of social validly: (a) goals—degree to which target behaviors are valued by consumers, (b) procedures—degree to which target behaviors are acceptable to consumers, and (c) effects—degree to which the outcomes of a study are satisfying for consumers. Stolz (1981) suggested that despite purposeful dissemination of interventions from the field of applied behavior analysis, interventions with low social validity have a lower probability of being adopted.
The development of social competence is a critical component of early development, but for some children with developmental delays, the acquisition of these skills does not happen naturally. A large body of empirical research, much of it from researchers using applied behavior analysis, exists on effective social competence interventions for young children (Brown & Conroy, 2002; Elliott & Gresham, 1993; Mastropieri & Scruggs, 1985–1986; McEvoy, Odom, & McConnell, 1992; Odom et al., 1999; Sainato & Carta, 1992). Social competence interventions tend to use environmental arrangement; child-specific, peer-mediated strategies; or a combination of these approaches to facilitate the development of social skills in young children (Odom et al., 1999). Despite the abundance of effective procedures, early childhood special education classroom teachers implement social competence interventions procedures at moderate to extremely low levels (McConnell, McEvoy, & Odom, 1992). The rate of implementation of social competence interventions is influenced by a variety of factors. Elliott, Witt, Galvin, and Peterson (1984) found a relationship between the acceptability of applied behavior analysis interventions and amount of time and skills required for implementation. Elliott and colleagues also found that teachers were more accepting of interventions if the presenting behavior problem was severe. Odom, McConnell, and Chandler (1993) reported that teachers identify limited time, resources, and access to peers without disabilities as barriers to implementation of social competence interventions.
Given the variety of reasons a teacher may choose not to adopt or implement an effective intervention, applied behavior analysis researchers have been assessing the social validity, or acceptability, of interventions at an increasing rate (Carr, Austin, Britton, Kellum, & Bailey, 1999). Although reviews have been conducted addressing the use of social validity assessment in applied behavior analysis research (Carr et al., 1999; Fawcett, 1991; Kennedy, 1992; Schwartz & Baer, 1991; Storey, 1996), or more specifically in early intervention and early childhood special education (Odom & Strain, 2002), none have focused on the social competence intervention research for preschool children. This review fills a small but important gap in the existing literature. The purpose of this review is to examine the social validity assessment methodologies used in the preschool social competence intervention research from 1970 through 2008 to determine (a) the rate and trends of social validity assessment use during this period, (b) what dimensions (e.g., goals, procedures, effects) are being assessed, and (c) what methods are being used.
Method
Study Selection
Studies were included in this review if they (a) had at least 50% or more of the study participants between the ages of 3 and 5 years, (b) addressed issues of social skills and had an intervention specifically designed to effect the social competence of the participants, (c) were classroom based (i.e., intervention procedures conducted in the child’s classroom or other school space typically used by children such as a playground), (d) used single-subject research methodology, and (e) were published in a peer-refereed journal.
The author located studies through (a) an electronic search using Psych INFO and ERIC, (b) a list of articles provided by the first author of the Chandler and Lubeck (1992) review of generalization and maintenance in preschool children’s social skills intervention, (c) a manual search of nine peer-reviewed journals from 1990 to 2008 (see Table 1 for a list of journals), and (d) ancestral searches using the reference sections of found articles and reviews of the intervention literature (Elliott & Gresham, 1993; Mastropieri & Scruggs, 1985–1986; Odom et al., 1999; Sainato & Carta, 1992). Ninety social competence intervention articles were located that met all of the selection criteria. The number of articles found by journal is shown in Table 2.
List of Journals Hand Searched From 1990 to 2008
Sources for Articles on Social Interventions for Young Children
Analysis of the Studies
First, articles were coded using code sheets to indicate (a) whether the article used a social validation assessment, (b) the type of social validation measurement (e.g., goals, procedures, effects), (c) what method of assessment was used (e.g., rating scale, survey, interview), and (d) who participated in the social validation measurement (e.g., family, peers, target child).
Determining whether social validity occurred in an article was complicated by the variety of opinions expressed—often conflicting—about the definition of social validity and the appropriate measures for determining social validity. For example, Hawkins (1991) recommended assessing the habilitative validity of an intervention by comparing behavior rates of average or exemplary persons with the behavior rates of participants for an objective measure of a program’s habilitative validity. Kennedy (2002) recommended assessing the maintenance of behavior change as an objective indicator of social validity.
Schwartz and Baer (1991) reminded us that social validity was never intended as a measure of a program’s effectiveness and is intended to be a secondary measure to the more objective measures used in applied behavior analysis research (Wolf, 1978). Despite the author’s agreement with Schwartz and Baer, for the purpose of coding, the original description of social validity provided by Wolf and the more recent objective measures recommended by Hawkins (1991) were used. Reports of anecdotal conversations were not included as a method of social validity assessment unless they were a component of another more structured assessment method. For example, conversations were reported if they were conducted while viewing videos from different phases of the intervention.
During the coding of articles, an additional code sheet was used for each additional type of social validation assessment. For example, if a study used a questionnaire to determine how acceptable teachers thought the procedures were, as well as their satisfaction with the effects, two code sheets were completed. Separate code sheets were used for each social validation assessment methodology.
Interrater agreement
Reliability of the article coding was measured using a method developed by Symons, Koppekin, and Wehby (1999). After the first author coded all of the articles using the code sheets, a graduate student coded 20% of the articles. Purposeful sampling was used to ensure that 50% of the articles used in the reliability check reported social validity assessment procedures to provide a more accurate assessment of coder agreement. Training was provided, including a codebook with definitions, and the two reviewers practiced coding articles that were not included in the reliability check.
An agreement was scored every time the two coders entered the same code for a specific item on the coding protocol; a disagreement was coded when the two coders entered different codes on a specific item on the coding protocol. If both reviewers agreed that a social validation measure was not used, then scoring stopped. If reviewers agreed that a social validation measure was used or if only one of the reviewers thought that a social validation measure was used, then scoring of interrater agreement continued. Another agreement or disagreement was scored each time the reviewers agreed or disagreed on one of the following: (a) what type of social validation measurement occurred (e.g., goals, procedures, or effects), (b) who participated in the social validation measurement (e.g., family or peers), and (c) what method of assessment was used (e.g., rating scale or survey). Reliability was determined by dividing agreements by agreements plus disagreements and multiplying the quotient by 100.
Results
Interrater Agreement
The overall agreement between the two independent raters for all the coded items combined was 83.3%. The percentage agreement for the coding of yes or no signifying whether social validity assessment occurred was 93.8%. Agreement was 90.9% for type of assessment, 81.8% for who participated in the assessment, and 88.9% for the method used for the assessment. Overall agreement may appear low given that the variables are static; however, reporting of social validity in some articles was ambiguous.
Rates and Trends
Of the 90 articles reviewed, 26.67% (n = 24) of them conducted and reported one or more social validation assessment. Specifically, 7.78% (n = 7) assessed the validity of goals, 8.89% (n = 8) assessed the validity of procedures, and 21.11% (n = 19) assessed the validity of effects. Some studies conducted assessments for more than one type of social validity. Table 3 shows the specific type(s) of social validity assessed in each of the 24 articles. The rate of social validity assessment in the preschool social competence intervention literature has been steadily increasing since the 1970s with a slight plateau starting in 2000 (see Figure 1).
Levels of Social Validity Assessment
Rate of social validity assessment in preschool social competence intervention.
Social Validity Assessment of Goals
Seven studies (7.78%) conducted a social validation measurement to determine which target behaviors were the most important to consumers (see Table 3). Wolf (1978) explained that researchers can rely on the consumers of the interventions to make judgments about the goals of an intervention through measures such as interviews or rating scales. Social validity assessment methods used were reviewing the Individual Educational Plan (IEP), consultation, and questionnaires.
Reviewing the IEP
Researchers from four studies reviewed target children’s IEPs when selecting specific goals for individual children. Although there is demonstrated variability in the level of contribution family and other team members have in the generation of the IEP document (Martin et al., 2006), reviewing the IEP has been included as a method for social validity assessment because it is one strategy researchers have to access information about participant educational goals. Two of these studies increased the rate of low-probability requests that were social in nature and unique for each target child based on their IEP (Davis, Brady, Hamilton, McEvoy, & Williams, 1994; Davis & Reichle, 1996). Strain, Hoyson, and Jamieson (1985) consulted IEPs of target children to help “fine tune” the already selected goals. Laushey and Heflin (2000) reviewed IEPs of the target children to confirm that enhanced socialization was a goal for participants. The target behaviors from most of these studies were already determined by researchers before consulting the IEPs (Davis et al., 1994; Davis & Reichle, 1996; Strain et al., 1985). However, researchers aimed to make intervention goals more socially valid for individual children by reviewing the goals already selected by the IEP teams.
Consultation
Four of the studies consulted with consumers to help in the selection of goals. In the two studies described previously (e.g., Davis et al., 1994; Davis & Reichle, 1996), the target behaviors were customized based on the combination of students’ IEPs and consultation with families, teachers, and educational assistants. Laushey and Heflin (2000) combined the review of IEPs and conducted a focus group during which four teachers, one speech-language pathologist, and a parent selected the specific target behaviors to be targeted for the children (e.g., turn taking). Robertson, Green, Alper, Schloss, and Kohler (2003) reported assessing the validity of target behaviors through “subjective evaluation.” Kazdin (1982) described subjective evaluation as the soliciting of preferences by others who are in a position to judge about the behaviors that are the target for an intervention. Robertson and colleagues, however, do not report any methods or participants involved in the subjective evaluation process. Most of the studies provided a copy of all the target behaviors generated (Davis et al., 1994; Laushey & Heflin, 2000; Robertson et al., 2003).
Questionnaire
Ducharme and Holborn (1997) gave questionnaires to 10 teachers and 25 parents. The questionnaire listed 10 possible target behaviors and asked participants to rank them in order of importance. In addition, room was provided for other target behaviors to be added. A unique feature of Ducharme and Holborn’s study was the additional validity assessment of their target behaviors at the end of the study to see whether they were still valued by consumers.
Sunhwa, Sainato, and Davis (2008) gave a questionnaire to three early childhood special educators and three parents of children participating in the study, asking whether they considered the research questions to be useful and valid. Sunhwa and colleagues explained that the purpose of the study was to increase the social interactions of children with autism, and data were collected on the target behaviors of prompted and unprompted social behavior.
Social Validity Assessment of Procedures
Eight studies (8.89%) conducted and reported some sort of social validation assessment of procedures (see Table 3). For a social competence intervention to be implemented in classrooms, the intervention procedures need to be acceptable to classroom teachers (Hall & Didier, 1987; Reimers, Wacker, & Koeppl, 1987) and other consumers of the intervention. The studies reviewed used a variety of methods to assess the acceptability of intervention procedures such as conversation, questionnaires, surveys, focus groups, interviews, videos, and rating scales.
Conversations
Sometimes researchers report anecdotal conversations with consumers (e.g., teachers, families) during which the consumer affirms the acceptability of an intervention procedure. An example of conversation reporting is provided by Morrison, Sainato, Benchaaban, & Endo (2002) who described a conversation with a teacher who indicated that the intervention would continue to be used and that the children enjoyed it. Zanolli and Daggett (1998) reported a conversation with the family of a target child who had reservations regarding the use of food to reinforce behavior. The researchers altered their intervention procedures for the particular child by reinforcing desired behavior with affection instead of food. Reporting anecdotal conversations with consumers about the acceptability of an intervention is certainly related to social validity. These conversations were not included in the rate and trend totals for this review.
Questionnaires and surveys
Questionnaires and surveys were the most common method used to assess the social validity of procedures. Spohn, Timko, and Sainato (1999) gave questionnaires to early childhood teachers and speech-language pathologists. Storey, Danko, Ashworth, and Strain (1994) gave classroom assistants a questionnaire at the end of each phase of their study. Reagon, Higbee, and Endicott (2006) gave surveys and questionnaires to the mother of a target child and to a sibling who participated in the intervention.
Three of the studies using questionnaires and surveys reported the questions used. Storey and colleagues shared the content of a questionnaire used, and Hundert and Hopkins (1992) provided a copy of the five-item survey given to six teachers. Garfinkle and Schwartz (2002) gave classroom and support staff a five-question, 5-point, Likert-type scale survey (e.g., The intervention was something I could do in my classroom.) and four open-ended questions with room provided to make suggestions for changes to the intervention. All three studies provided clear results describing the positive responses of participants. A unique feature of the Garfinkle and Schwartz survey is that it was given anonymously, presumably allowing participants to feel less inhibited about giving negative feedback about the validity of procedures.
Focus groups and interviews
Two (2.33%) studies reported focus groups or interviews to assess the validity of procedures. Laushey and Heflin (2000) conducted a focus group with teachers, a speech-language pathologist, and a family member. Spohn et al. (1999) conducted interviews with children regarding what they liked about the intervention procedure and what parts worked best. Spohn et al. was the only study to report measures assessing the acceptability of an intervention from the perspective of the adults and children involved in the procedures.
Video and rating scales
Two of the studies assessed the social validity of procedures by asking consumers to evaluate videos of children from different phases of the study. Oke and Schreibman (1990) showed video clips to 22 undergraduate students who completed a rating scale concerning the children’s affect (Dunlap, 1984; Dunlap & Koegel, 1980). Sunhwa et al. (2008) had 20 early childhood special education professionals read a description of an intervention procedure and then watch 2-min segments collected during various phases of the study. Professionals were asked whether they thought the intervention would be easy to use within the context of regular classroom routine and whether they would use the intervention considering the time and effort required. Results from both studies were positive but assessed different aspects of the procedures. Oke and Schreibman assessed the social validity of procedures by asking adults to rate the mood and behavior of children participating in the procedure, whereas Sunwa and colleagues assessed the validity of procedures from the perspective of professionals regarding the feasibility of the intervention.
Social Validity Assessment of Effects
Of the 90 studies found, 19 (21.11%) assessed the social validity of the effects of intervention. In an effort to assess consumer satisfaction with the outcomes of interventions, researchers used a variety of methods, including sociometric measures, peer-comparison data, questionnaires, surveys, rating scales, checklists, videos, and interviews.
Sociometric measures
Sociometric measures are used to obtain ratings from a peer group about how much children like to play with each member of their class. Sociometrics can be a valuable tool for providing an evaluation of a child’s social competence and level of acceptance or rejection from the perspective of the children themselves (Asher, Singleton, Tinsley, & Hymel, 1979). Five studies used sociometric measures for assessing the social validity of a study’s effects.
The most commonly used sociometric measure was a rating scale developed by Asher et al. (1979) or an adapted version of this measure involving children sorting pictures of classmates into three boxes—one box with a smiling face for “likes a lot,” a straight line mouth for “liked a little,” and a sad face for “doesn’t like at all.” Researchers have used the Asher et al. sociometric measure at different times during their studies. Hundert and Houghton (1992) gave the measure during each phase of their study and then during a follow-up phase. McGee, Almeida, Sulzer-Azaroff, and Feldman (1992) used sociometric measures before and 5 months after the intervention. Odom and Watts (1991) used sociometric measures after the intervention was over.
Sainato, Maheady, and Shook (1986) assessed the social validity of the effects of an intervention with two different types of sociometric measures: (a) a variation of the How I Feel Toward Others assessment (Agard, Veldman, Kaufman, & Semmel, 1978) in which children marked one of four faces, ranging from a frown to a smile and a question mark for “I don’t know,” describing how they feel about a given peer, and (b) a peer nomination technique in which children identified their “best friend.” Sainato and colleagues administered these measures to children 5 times, once during each phase of their study. English, Goldstein, Shafer, and Kaczmarek (1997) used another type of sociometric measure before and after intervention called the Friendship Train Instrument (McConnell & Odom, 1986) in which children chose preferred passengers for a train ride from a collection of peer photographs.
The majority of the studies using sociometric measures for the dimension of effect found no change (English et al., 1997), ambiguous results that were inconsistent across target children (Odom & Watts, 1991), or ambiguous results that decreased during the intervention phase (Hundert & Houghton, 1992). Only two of the studies reported a positive increase in target children’s sociometric ratings (McGee et al., 1992; Sainato et al., 1986).
Peer-comparison data
Van Houten (1979) recommended using an objective measure to assess the social validity of an intervention’s effect by establishing a typical level of performance criteria through the collection of data on the rates of behaviors in comparison children. Some would argue that peer-comparison data are not a measure of social validity given that it is not a subjective value judgment from consumers on the value of an intervention (Schwartz & Baer, 1991). Peer-comparison measures, however, have been included in this review because researchers have reported using them and have labeled peer-comparison measures as social validity assessment.
Peer-comparison data were used in six of the social competence intervention studies (Anita & Kreimeyer, 1987; Ballard & Crooks, 1984; Hundert & Hopkins, 1992; Hundert & Houghton, 1992; Osnes, Guevremont, & Stokes, 1986; Robertson et al., 2003). These researchers collected social interaction data on comparison children who were enrolled in the same classroom as the target children. Most of the studies reported the rates of target behaviors for comparison children (Anita & Kreimeyer, 1987; Hundert & Hopkins, 1992; Hundert & Houghton, 1992; Robertson et al., 2003). Anita and Kreimeyer (1987) and Hundert and Houghton (1992) also reported ranges for the rates of behaviors in comparison children. The results of the peer-comparison data were ambiguous for the majority of the studies (Anita & Kreimeyer, 1987; Ballard & Crooks, 1984; Hundert & Houghton, 1992).
Questionnaires and surveys
To assess the social validity of study effects, five of the studies used questionnaires and surveys. Participants were family members (Reagon et al., 2006), teachers and parents (Ducharme & Holborn, 1997), only teachers (Hundert & Hopkins, 1992; Storey et al., 1994), or teachers and support staff (Garfinkle & Schwartz, 2002; Spohn et al., 1999). Most of the studies included a copy of the survey or questionnaire in their report (Ducharme & Holborn, 1997; Garfinkle & Schwartz, 2002; Hundert & Hopkins, 1992; Storey et al., 1994). All of the studies reported positive results.
Rating scales and checklists
Five studies used rating scales or checklists for assessing the effects of social competence interventions. McGee et al. (1992) gave the Pictorial Scale of Perceived Competence and Social Acceptance for Young Children (Harter & Pike, 1980, 1984) to the participants’ teachers at the beginning and end of the study. The Pictorial Scale of Perceived Competence and Social Acceptance for Young Children rates children on cognitive competence, physical competence, and peer acceptance.
Most researchers using rating scales or checklists combined them with video. Storey and colleagues (1994) had 12 parents and 18 early childhood experts rate video clips of target children taken from each phase of the study. Shafer, Egel, and Neff (1984) had three teachers and a paraprofessional rate video clips from the baseline and posttraining phases regarding the quantity of target child and peer play. English et al. (1997) showed video-clip segments to 11 early interventionists who then completed a rating scale for the two dimensions of quantity and quality of play for target children. Morrison et al. (2002) had 12 early childhood education teachers evaluate video from pre- and postintervention of the target children and complete an adapted version of the Hawaii Preschool/Kindergarten Survival Skills Checklist (McCormick & Kawate, 1982) including 12 items about specific skills (e.g., spontaneously engage in play activities). Finally, Sunhwa and colleagues (2008) had 20 early childhood special educators watch video clips of target children taken from each of the experimental conditions and then complete a checklist regarding the target child’s ability to make requests, play, share, and respond to peers. Sunhwa and colleagues put the video clips into random order so the teachers would not know what condition the clip was from.
Most of the studies clearly reported what was on the rating scale (English et al., 1997; McGee et al., 1992; Shafer et al., 1984). Morrison and colleagues (2002) indicated generally what was in the 12 items in the skills checklist. Most of the studies reported positive results (English et al., 1997; McGee et al., 1992; Shafer et al., 1984; Sunhwa et al., 2008).
Interviews
Three studies used interviews to evaluate the social validity of the effects of interventions. Bay-Hinitz, Peterson, & Quilitch (1994) interviewed teachers at the end of their study and provided a clear description of all the questions asked during the interview. Spohn and colleagues (1999) interviewed children who participated in the intervention but did not provide a list of these questions. Bay-Hinitz and colleagues reported ambiguous or mixed results from their interviews, whereas Spohn and colleagues reported positive results. Morrison and colleagues (2002) reported on conversations that were interview like in that they were planned and conducted in conjunction with viewing pre- and post video clips of participants with parents and teachers. Sunhwa and colleagues (2008) asked teachers follow-up questions about how effective an intervention was for increasing the social interactions of children with autism after viewing video clips of target children from various phases of the study.
Discussion
The assessment of social validity in the social competence intervention literature for preschool children has increased since the 1970s, and about half of the articles published over the past decade have included reports of social validity assessments. This increasing trend suggests that many researchers agree on the value of social validity assessment. Still, there is great diversity in the methods used, aspects of the study selected for assessment (e.g., goals, procedures, or effects), and the types of consumers selected to participate in assessments. The results of this review allow the reader to survey the broad and diverse terrain of social validity assessment in the social competence intervention literature. Given this vantage point, a few issues surface that deserve discussion: (a) optimal timing of social validity assessments, (b) types of consumers selected to participate, (c) subjective versus objective measures, (d) standards for reporting, and (e) the role of anonymity.
Timing of Assessments
The assessment of an intervention’s effect can only be measured at the end of a study. The social validity assessment of goals and procedures, however, can be assessed at the beginning, during, and after an intervention. This allows consumers to help shape procedures and target behaviors and provide feedback throughout and after an intervention. Most social validity assessment of goals occurred before the intervention (Davis et al., 1994; Davis & Reichle, 1996; Laushey & Heflin, 2000; Robertson et al., 2003; Strain et al., 1985; Sunhwa et al., 2008) with the exception of Ducharme and Holborn (1997) who collected data before and after the intervention. Ducharme and Holborn provide an example of how to incorporate the feedback of direct consumers regarding the goals of an intervention over the entire course of the study.
The social validity assessment of procedures was always conducted after the intervention was over. This means that researchers missed the opportunity to determine how consumers perceived an intervention procedure before they actually implemented it. The perceptions of consumers about a behavioral intervention they have not experienced are valuable as this may reflect the opinions of teachers and families who could potentially “shop” through the vast social competence intervention. When teachers are asked to implement an intervention that they find undesirable or unacceptable, their level of implementation and quality of implementation are going to be effected. Assessing the social validity of procedures with consumers before they are implemented would allow researchers the ability to increase the compatibility of behavioral interventions with early childhood classroom culture and consumer values as well as make interventions more socially valid for individual children.
Types of Consumers
The identification of consumers for social validity assessments can be a complex challenge. Schwartz and Baer (1991) recommended that consumers include people with varying degrees of participation with an intervention: (a) direct consumers who are the primary recipients of the intervention or people who are directly involved in the intervention (e.g., the target children) and all others involved in implementation such as teachers or peers, (b) indirect consumers who are people strongly affected by the effects of the intervention but who are not direct consumers (e.g., family members who do not participate directly in the intervention), (c) members of the immediate community who are people who interact with the direct and indirect consumers (e.g., children on the playground), and (d) members of the extended community who are people who may never have direct contact with the consumers of the intervention. Schwartz and Baer recommended that social validation assessments be extended to include previously underrepresented consumers.
Although the majority of consumers participating in the social validity assessments in this review were from the first two levels described by Schwartz and Baer (1991), direct or indirect consumers (e.g., teachers, professionals, families, and peers), there are studies that conducted social validity assessments with indirect consumers. Two of the studies assessed the social validity of procedures by asking consumers who were members of the extended community to evaluate videos of participants from various phases in the study. Oke and Schreibman (1990) showed video clips from phases of the study to 22 undergraduate students who completed a rating scale concerning the children’s affect. Sunhwa et al. (2008) asked 20 professionals in early childhood special education to read a description of an intervention procedure and report whether it would be easy to implement within a student’s routine. One of the benefits of using indirect consumers is that a larger number of participants can be involved in the assessment.
Subjective Versus Objective Measures
Van Houten (1979) and Hawkins (1991) have advocated for the use of objective measures when assessing social validity as opposed to the subjective assessment of consumer satisfaction or opinion. Hawkins explained that the term social validity is misleading because social validity assessments are evaluating consumer satisfaction, and not all behavior interventions are targeting social behaviors. Although objective measures of social validity exist in the social competence intervention literature, Schwartz and Baer (1991) explained that social validity assessment was never intended to be objective. Regardless of this difference in opinion, the only objective measure found in this review was the use of peer-comparison data to evaluate the social validity of an intervention’s effect, and no studies was found that used an objective measure to assess the social validity of procedures or goals (Anita & Kreimeyer, 1987; Ballard & Crooks, 1984; Hundert & Hopkins, 1992; Hundert & Houghton, 1992; Osnes et al., 1986; Robertson et al., 2003).
Standards for Reporting
Schwartz and Baer (1991) explained that social validity assessment is intended to be a secondary measure in addition to more objective measures used in applied behavioral research. Given the amount of data collected and reported in single-case research designs (Kazdin, 1982) and the page limits in peer-reviewed journals, it is understandable that the reporting of social validity assessment is often brief and incomplete. Rather than provide a list of studies that did not provide clear description of social validity measures and results, the following discussion will focus on some of the studies that did report the necessary information with an understanding that this is not always the case.
Most, but not all of the studies using questionnaires, surveys and interviews for social validity assessment provided a list of questions used (Bay-Hinitz et al, 1994; Ducharme & Holborn, 1997; Garfinkle & Schwartz, 2002; Hundert & Hopkins, 1992; Storey et al., 1994). One study by Garfinkle and Schwartz (2002) included means and ranges when reporting the results of Likert-type questions and provided quotes from open-ended questions asked during interviews. Only two articles that used peer-comparison data on rates of social interaction included ranges for the means (Anita & Kreimeyer, 1987; Hundert & Hopkins, 1992). The standards and scientific rigor of social validity assessments should be held to a higher standard. Although social validity assessment will continue to be secondary to the primary purpose of a given behavioral intervention study, the studies discussed above can be used as guides for researchers seeking to increase the quality of reporting.
Role of Anonymity
Garfinkle and Schwartz (2002) expressed concern that direct consumers of their intervention may have been hesitant to give negative feedback on social validity assessments due to relationships developed with researchers. Even though questionnaires were anonymous, Garfinkle and Schwartz expressed their hesitation regarding the positive feedback from consumers. This raises an important issue for social validity assessment with direct consumers. Are direct consumers reluctant to give negative feedback on social validity assessments due to the relationships that are developed during the study? Garfinkle and Schwartz’ provision of anonymity is worth considering in future social validity assessments that lend themselves to anonymous response (e.g., questionnaires or surveys). Social validity assessments with large numbers of indirect consumers may also be a strategy to decrease the social pressure to give positive responses.
Limitations
The first author conducted electronic, manual, and ancestral searches from 1970 through 2008, and it is certain that some articles were missed. In addition, the inclusion criteria that the study must address issues of social skills and have an intervention designed to affect the social competence were sometimes difficult to judge. Preschool social competence intervention studies target a wide range of behaviors that can sometimes make it difficult to determine whether the study is a social competence intervention. Hurley, Wheby, and Feurer (2010) identified 80 different target behaviors used in the social competence intervention literature for preschool children illustrating the complex nature of preschoolers’ social interactions. Inclusion criteria for this study were further complicated by the overlap between the goals of the communication intervention and social competence intervention literature. Given the intertwined relationship between the domains of social and communication competence, it is understandable that the goals for interventions from both bodies of literature are often similar. Finally, there are studies that target behaviors related to social competence, such as increasing functional toy play that do not address social interactions with peers (Paterson & Arco, 2007). It would have been beneficial to conduct an interrater reliability measure for the selection and rejection of articles included in the review.
Conclusions
Wolf (1978) charged researchers with developing and using better methods and systems for asking consumers about the value of their work. Behavior interventionists can use the examples of social validity assessment and issues raised in this review as a resource for developing and planning future social validity assessments. In addition, researchers can refer to studies with the purpose of assessing the social validity of the social competence intervention preschool literature. Odom et al. (1993) assessed the acceptability of procedures with 131 teachers, and Hurley et al. (2010) assessed the value of behaviors goals with 36 early childhood educators. Given the increasing rate of social validity assessment, it is clear that more researchers see these measures as important. Horner and colleagues (2005) included the assessment of social validity as a critical defining features and indicators of quality in single-subject research methodology. Although consensus is growing about the importance of social validity assessment, we must take a critical look at the methods used, timing of assessments, thoroughness of reporting, number of consumers, and the role of anonymity in the assessment of social validity. Before an intervention can have a positive effect on the behavior of a child, it must be selected and then implemented by consumers. Consumer feedback provided through social validity assessment is a valuable resource for researchers to facilitate the transition of work beyond publication to implementation in classrooms.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.