Abstract
In the past two decades, evidence-based status has been a coveted credential for many nonprofit organizations hoping to legitimize their programs or interventions. Several formal registries exist to provide a collection of health and prevention programs evaluated by experts and deemed “evidence-based.” While registries offer positive benefits like allowing for a centralized listing of approved programs, there have been concerns about issues pertaining to the process of obtaining the evidence-based credential. Namely, some of the criticisms include the use of inappropriate study designs, the lack of consistent evaluation of evidence provided in support of programs, as well as program creators being involved in the evaluation that ultimately shows positive program effects. Using focus groups of prevention specialists, this study explores the quest for evidence-based status. The results show themes of vindication, acting as a resource, and perceptions of relevant others informing the deeper meaning of motivation for pursuit of evidence-based status. Additionally, emergent themes of program iteration and evolution inform program preparation. The article shows that while placement on an evidence-based registry is a highly sought-after achievement, many program creators fail to understand the evaluation process for admittance as well as the potential criticisms of the lists.
Introduction
In the past two decades, the concept of “evidence-based program” (EBP) status has been the object of desire for many nonprofit and human service agencies. EBPs are “collections of practices that are done within known parameters (philosophy, values, service delivery structure, and treatment components) and with accountability to the consumers and funders of those practices” (Fixsen, Blase, Metz, & Van Dyke, 2013, p. 213). Stated in another way, EBPs undergo critical assessment and intensive evaluation and have demonstrated positive intervention outcomes (Rychetnik & Wise, 2004). In the health promotion, prevention, and intervention fields, several registries exist and offer credentialing of “evidence-based” status to programs that meet specific requirements. Some of these include the Substance Abuse and Mental Health Services Administration’s (SAMHSA) National Registry of Evidence-Based Programs and Practices (NREPP), the U.S. Department of Justice’s (USDOJ) “Blueprints” program, and the Office of Juvenile Justice and Delinquency Prevention (OJJDP) Model Programs Guide (for others, see Escoffery, Carvalho, & Kegler, 2012; Hill, Alpi, & Auerbach, 2010).
The concept of a registry is positive in that it means to “reduce the time lag between the creation of scientific knowledge and its practical application in the field” (Wright, Zhang, & Farabee, 2012, p. 959). Registries provide an outlet for sharing (Hill et al., 2010) in a centralized location for organizations to select from programs that showed clear evidence of positive benefits to the service population. Gaining placement on the EBPs list suggests that the program meets a set of rigorous standards. Specifically, applicants are required to show through properly designed studies, study replications, subsequent publications, and presentations that the program is ready to be disseminated for public use. In this way, being on the EBP list was like a badge of approval “insinuating proof through official recognition of their evidence-based status” (Wright et al., 2012, p. 960). Additionally, many public policies have evolved to favor the use of these EBPs (Armstrong, Waters, Crockett, & Keleher, 2007; Tibbits, Bumbarger, Kyler, & Perkins, 2010).
While there are positive implications of obtaining the evidence-based credential, there are several critical issues about the process of obtaining the credential worth noting. For example, registries boast a set of rigorous admission standards required to be met by applicants. Not only does the study design have to be a true randomized or quasi-experimental design, it also has to be conducted properly and the data analyzed appropriately (Steinberg & Luce, 2005). However, several scholars have found that research quality was inconsistently assessed on the registry lists; programs submitted evidence that did not, in fact, meet the outlined minimal criteria for admission; and many organizations exhibited a conflict of interest by having the creators of the program also act as the evaluator (Axford & Morpeth, 2013; Wright et al., 2012). As such, the very evidence that achieved the program’s evidence-based status is, therefore, suspect. If there are critical issues with the evidence-based credentialing system, why do nonprofit organizations and government human service agencies put such a high priority on attaining evidence-based status?
This research explores the perception of program developers who are trying to attain evidence-based status through lists like those offered by SAMHSA, USDOJ, or OJJDP. Using qualitative research, the researcher conducted focus groups of program developers from the nonprofit and government sectors. Many previous studies evaluate EBPs or refer to evidence-based practice in medicine. A plentiful literature exists exploring the implementation (Fixsen et al., 2013; Self-Brown, Whitaker, Berliner, & Kolko, 2012), sustainability (Tibbits et al., 2010), or community perception (Ramanadhan et al., 2012) of EBPs. Far fewer researchers have investigated the journey toward achieving admittance on the EBP lists. The present study aims to expand on knowledge surrounding the quest for evidence-based status and to fill the gap in the literature. This study challenges existing practices and advocates for the development of new policies pertaining to the evaluation of EBPs. As such, the results of this research are pertinent to health promotion specialists, human services organizations, evaluation professionals, and researchers alike.
Literature Review
Perhaps one of the most critical appraisals of EBPs registry lists came from Wright et al. (2012). Wright et al. analyzed the vetting process for programs that applied and gained admittance to the NREPP list. The authors, though acknowledging the good intentions of the list, observed several significant flaws with the programs that had successfully made it through the rigorous application process. They found issues with the quality of research as rated by SAMHSA professionals, the sample sizes being inappropriately small, and evidence that program creators were also acting as program evaluators. Specifically, of the 31 programs reviewed, many were rated as having questionable quality or lacked dissemination readiness, 17 programs achieved a rating that highlighted methodological flaws or lack of empirical evidence while 15 of the programs were rated poorly on readiness for dissemination (Wright et al., 2012). In addition to the rankings being inconsistent and low, the authors also noted that many of the interventions submitted support studies that had an inappropriately low sample size. In fact, most studies that were reviewed contained less than 100 participants. The last finding revealed that many of the program creators also participated in the evaluation of the program. This creates clear conflict of interest as those parties interested in obtaining evidence-based status for their program are also responsible for furnishing evidence of the program’s effectiveness.
Providing a different perspective, Axford and Morpeth (2013) critically assessed why EBPs failed to be fully accepted in children’s services agencies. Axford and Morpeth (2013) identified scientific, ideological, cultural, organizational, and professional critiques of EBPs. The authors argued that the philosophical basis for determining whether a program is evidence-based is “fundamentally flawed” (p. 269). The authors cautioned the reliance entirely on empiricism underestimates other possible ways of assessment. Additionally, randomized control trials that are typically seen as a gold standard (Monette, Sullivan, & DeJong, 2010) are inadequate and potentially unethical. Meanwhile, quasi-experimental designs do provide ways to control for spurious factors but can require complexity of design and large sample sizes (Axford & Morpeth, 2013). In either case, results from even rigorous designs can potentially mislead causing programs to be deemed erroneously as “evidence-based.” And, Axford and Morpeth, like Wright et al. (2012), also noted the problematic tendency of program developers to be involved in the evaluation.
In a different approach, Escoffery et al. (2012) evaluated a training program that coached health educators to define evidence, assess constituent needs, locate an EBP, choose a program that fits the community, and adapt, implement, and evaluate EBPs. Between the pretest and posttest, the results showed significant increases across all metrics. The largest increases in knowledge occurred to defining steps to adapt EBPs, the ability to discuss factors in each phase, and the ability to locate EBPs on registries. The findings suggested that training programs can increase the competencies of health educators to find, assess, and use EBPs or strategies.
Two consistent themes emerged from the literature each having implications for policy development. First, scholars agree that a clear demarcation of the meaning of evidence-based is currently lacking (Mihalic & Elliott, 2015; Steinberg & Luce, 2005; Wright et al., 2012). Wright et al. (2012) noted “the lack of a clear definition of what ‘evidence based’ implies” as a reason for variation across types of submitted evidence (p. 958). Similarly, Rychetnik and Wise (2004) asserted that “concepts of evidence vary among professional, disciplinary and social groups” (p. 248). Additionally, Mihalic and Elliott (2015) explained that the registries each use different terminologies to classify the programs in a way that can be confusing to the reader. Second, several authors asserted that the quality of evidence submitted to these registries should be further scrutinized (Mihalic & Elliott, 2015; Steinberg & Luce, 2005; Wright et al., 2012). Mihalic and Elliott (2015) asserted that the registry lists show variation in the focus and criteria for assessing the evidence of the programs submitted for acceptance. This variation can be misleading to those individuals seeking programs to select and implement. Wright et al. (2012) found many concerning inconsistencies about the evidence submitted in support of the programs. Steinberg and Luce (2005) noted that critical evaluation of the quality of a study should assess methodologically sound research design as well as the execution of said design. As such, if inconsistencies exist about the definition of what it means to be “evidence-based,” and if the supportive studies submitted to gain admittance to the different registries are not scrutinized appropriately for methodological soundness, then what truly is the value and appeal of being on the registries? Development and implementation of policy to address these two critical challenges would be conducive to the further promotion and proper assessment of health programs.
Method
The study was conducted in central Pennsylvania. First, approval was obtained from the institutional review board of a large public research university in Pennsylvania. Next, the researcher conducted focus groups to obtain perceptions on the topic of interest. Focus groups embrace the idea that many decisions are made in a social context (Patton, 2002). The focus group design was, therefore, selected to allow participants to hear each other’s responses and to foster discussion about the evidence-based status acquisition process.
A call for participation was sent through an outreach e-mail listserv of approximately 100 constituents and was announced at a local community event in which 34 nonprofit leaders attended. Criterion sampling was used as all invited parties were program creators or managers in the process of attaining evidence-based status. The researcher cultivated sensitizing concepts prior to executing the focus groups. Sensitizing concepts are defined as “loosely operationalized notions . . . that can provide some initial direction to a study as a fieldworker inquiries into how the concept is given meaning in a particular place or set of circumstances being studied” (Schwandt, 2001, as cited in Patton, 2002, p. 278). To create focus for the research moving forward, four different sensitizing concepts were shaped: reason for pursuing EBS, program planning, awareness of evaluation process, and blind interest. The researcher then conducted two focus groups in the summer of 2014. A semistructured protocol was used as a reference point to encourage both groups to answer the same questions (see the appendix).
The focus groups consisted of four and three participants, respectively. Seasoned qualitative researchers suggest that groups should have 6 to 10 participants (Patton, 2002) or 6 to 12 participants (Brown, 1999) but can allow for less depending on the level of homogeneity of the group. For example, Green and Hart (1999) allowed for groups as small as three as it was supported by the context of the research. In the case of this study, each group was highly homogenous and consisted of all females who worked in the health promotion field. Each group consisted of one African American female, with the remainder identifying as Caucasian. Each group contained a director-level participant with over 25 years of experience in her respective prevention areas, and the other participants were prevention specialists at the program level each with at least 10 years in the field. The public and nonprofit sectors were both represented. On average, the focus groups lasted approximately 35 minutes.
The researcher transcribed the focus group data immediately after the conclusion of each event in an effort to gather the most data. With the two focus groups, responses were consistent and saturation of themes occurred. Patton’s (2002) process was employed for data analysis. Specifically, the researcher reviewed the transcript data and made notes about important responses. Next, the researcher organized the data into categories through the convergence process (Patton, 2002). Then, patterns in the responses were used to generate emergent themes. The researcher executed this process over multiple iterations to ensure that the data were internally homogenous and externally heterogenous. As a method of ensuring quality in the data, the researcher used member checks in the focus groups by listening to the participant commentary, summarizing what was heard, and repeating it back to them (Patton, 2002). Additionally, the researcher kept an audit trail during the analysis. The audit trail is a chronological list of decisions made during the course of data analysis further allowing for a record of process.
Results
This section provides a summary of the deeper meanings and subsequent emergent themes that surfaced from the focus group data (see Figure 1). The intended audience for this section includes nonprofit entities who may be interested in obtaining access to an EBP registry, community members, academics, and program evaluators.
Deeper Meanings and Emergent Themes
Motivation for Pursuit
Three emergent themes that related to the deeper meaning of motivation for pursuit included vindication, becoming a resource for others, and perception of relevant others.
Vindication
It was clearly stated in both focus groups the desire to achieve evidence-based status as vindication for the hard work involved in the creation of the program. It was suggested that admittance to the list would validate the efforts made to create and refine the curriculum. Specifically, one individual contributed it “feels good to be able to say that we took something that did not exist, wrote this, and people really like it.” Another participant stated, “The ultimate goal was to create a curriculum program that worked and that we knew worked.” For the participants in the focus groups, achievement of evidence-based status offered justification and reassurance about their commitment to the development of their programs.
Becoming a Resource
In addition to feelings of justification with the achievement of evidence-based status, both focus groups mentioned the importance of being a resource for other agencies. For example, one participant stated, “It [admission to the list] enables us to become a resource regarding these kinds of programs.” Another participant added a story about previous research conducted on a different program. In her case, the research was publicized on a website, which then resulted in her being contacted by people asking for data and information about the program because it was promising as pertaining to outcomes.
While there remained a general willingness and enthusiasm about becoming a resource for other prevention professionals, apprehension was also expressed by the respondents. For example, when asked what the perceived drawbacks are for gaining admission to the list, respondents expressed trepidation. One participant suggested that becoming a resource for others can create the environment for being inundated with requests. Another expressed concern around the use and replication of the program. She stated what if “someone takes your model and replicates it and something bad happens?” In the cases where being a resource was perceived as potentially negative, the contributors focused on the feeling of exposure and loss of control once the program is listed.
Perception of Relevant Others
Complementary to being a resource for others, the focus groups asserted that gaining admission to the registries would change the way their organization was perceived by relevant others. This theme emerged from two dominant perspectives: participants referenced their peer groups and similar organizations, and participants referenced their funders or parent agencies. One participant shared that gaining admission to the lists would “bolster our professional credibility in the field.” The shared perspective among groups was that being on the lists would make other entities aware of their legitimacy. Another individual indicated that the biggest value in being on the list would be the ability of her organization to count the program in its required reporting totals. In this case, the participant was coming from a county-level government agency that was required to report data to the parent agency. Her sentiment “I think the biggest value is being able to put it in the system as an e-b count” shows that the parent agency has placed a premium on the use of programs that have achieved evidence-based status.
Program Preparation
In addition to motivation for pursuing evidence-based status, both focus groups highlighted the trials and tribulations as pertaining to the preparation process. Three themes emerged that related to the deeper meaning of program preparation including registry expectations, number of edit iterations, and evolution of the program.
Registry Expectations
Participants in the focus groups were asked about their familiarity with the registries. Each group expressed familiarity, though knowledge of the registries was clearly uneven between the groups. For example, when asked if familiar with the registry lists, one participant shared, “Not a lot. I know that there are lists.” Meanwhile, the other group had shared the specific list to which they were attempting to gain admission stating, “SAMHSA NREPP is most attractive because that is the field we work in. It is the list people go to first when they are looking for evidence-based programs.” When probing further about the application process, one group had minimal understanding of study design such that they requested an explanation of the concept of a comparison group. In that discussion, one participant wrestled with the concept asking methodological questions like “When is it too early to do that [have a comparison group]?” and “So, the comparison group is the group that would not participate [in the intervention]?” On the other hand, the focus group having fuller knowledge of the lists expressed awareness of the required quasi-experimental design. For example, one participant shared,
The comparison group is a thorn in our side due to ethical reasons. Trying to pick who is going to get the service and who is not . . . both groups could use it! In our hearts we’re counselors and preventionists. We don’t want to see any kid left behind.
However, in their expression of the concerns pertaining to the specific design, they shared no indication of the capacity required to execute such prerequisites, and they expressed no doubt or cynicism that the program applicant evaluation process could potentially be flawed.
Iterations
The sheer number of iterations that the participants experienced as they prepared their programs for submission was noted in both focus groups. In both focus groups, the participants admitted to cultivating their programs from the ground up and continuing to rework them over a period of years. One participant even shared that her program was reworked over a period of 10 years. The participants discussed their programs with pride and exhaustion in their voices. One stated, “On a personal level, this is one of the most exciting, time consuming, wear-us-down projects. An adventure!” The same participant told her story about how they had to “tear up the program” and rebuild it or finesse it over seven or eight sizeable iterations.
Evolution
The last theme to emerge was the need to evolve. Here the focus was on the acceptance of change as modifications were integrated into their program. One participant stated the following:
I think that you have to be prepared to make changes to it [program]. I mean you can’t get locked into it. . . . It certainly evolved and just hearing this makes me think we have to make changes. You can’t be so locked into the design. You have to be willing to make changes.
Additionally, a participant while giving insight on the best learned lessons of the many iterations her program experienced expressed the importance of “being precise. To have a good editor. Be patient. Listen. And, try not to be defensive.” In both focus groups, the participants expressed humility in coming to a place of peace with the evolutions that their respective programs underwent.
Discussion
The results contribute to the literature by showing why evidence-based status is such a coveted accomplishment. However, and perhaps more interesting, there are perceptions that illustrate the paradox of attempting to achieve evidence-based status though the process is quite flawed. For example, one of the most surprising findings was the general lack of awareness of the evaluation process for admission to these lists. As suggested, the literature notes a lack of clarity surrounding the definition of evidence-based (Mihalic & Elliott, 2015; Steinberg & Luce, 2005; Wright et al., 2012). This could explain the unfamiliarity pertaining to how evidence is evaluated and the ultimate implications on underlying assessment of program impact. One focus group was less informed about the registry evaluation process than the other. The less informed group had awareness of the registries and wanted to be on the list. The more informed focus group, on the other hand, knew of the lists and explained that they were displeased with the quasi-experimental design component for ethical reasons. Still, neither focus group expressed an understanding of the potential problems associated with the review process. The results suggest that neither have these participants questioned the ultimate merit of programs on the registries nor have they considered that the badge of evidence-based status may be nothing more than a misunderstood, inaccurate, yet highly sought-after, designation.
Though counterintuitive, the act of working toward the evidence-based registry goal without full comprehension of process can be understood through the lens of social cognitive theory (Bandura, 1986). Social cognitive theory suggests that human actions are influenced by three interacting mechanisms: behavior, personal or cognitive factors, and the external environment. Over time, these three factors compel an individual to act in a certain way and subsequently to further influence his or her environment; a phenomenon known as reciprocal determinism (Bandura, 1986). Stated differently, individuals learn through the observation of others and will aim to replicate their behaviors. Efforts are then continued, and behaviors adopted, based on positive motivational incentives. As applied to the findings of this study, individuals or groups working in the health promotion space observe peers cultivating programs to improve the health of a community and with the intent of being deemed an EBP. Individuals, perceiving value in the observed behavior of others, then model their efforts and activities to keep with normative industry practice while striving to maximize benefits to the target population. Ultimately, individuals become particularly driven by the positive motivational incentive of potentially achieving evidence-based status within one of the regarded registries.
In the same light, a curious phenomenon occurred during the call for participation for the study. When the announcement was made of the focus groups to discuss the journey that program creators have experienced while working toward achieving evidence-based status on a registry list, the researcher received many inquiries of a similar nature. The individuals, when informed of the opportunity to participate in a focus group about the EBP registries, mistook the invitation as a way to actually obtain “evidence-based status.” The individuals contacted the researcher and stated that they wanted to become evidence-based. When the researcher asked follow-up questions, some of the callers did not even have programs created to evaluate. This experience shows that the term evidence-based is not only irresistible and pervasive but also highly misunderstood.
Last, being deemed an EBP is a key determinant for securing intervention funding. As such, it was not surprising to hear both focus groups express eager interest in obtaining evidence-based status. What was unexpected was the motivation for becoming evidence-based as a mechanism of self-confirmation. There was a clear and expressed desire to feel vindication for the amount of effort and time put into growing these programs from inception. These entities feel that becoming credentialed as evidence-based would validate their experiences and show that their hard work has paid off. As Wright et al. (2012) suggested, evidence-based status was perceived as a badge of honor. This was additionally supported by the expressed fatigue as pertaining to the amount of edits and iterations that their programs endured. And while the participants submitted to the need to evolve, the program creators expressed perseverance with the excessive duration of their journeys.
Limitations and Conclusion
As with any research, this study was not devoid of limitations. For example, while focus groups as a qualitative method of inquiry provide a deep and rich understanding of the phenomenon, they cannot be generalized. However, the researcher provided a detailed description of the endeavor as a way to increase transferability for other scholars interested in exploring similar occurrences. Next, the results showed a need for enhancing the interview protocol pertaining to one of the questions. Specifically, the question surrounding the registries should be expanded in future research to include awareness and more directed questions assessing the registry evaluation procedures. Finally, as previously stated, recruitment of participants who fit the criteria presented an obstacle to the study. Many more program creators came forward with an overall misunderstanding of the invitation to participate. As such, future research could employ techniques to increasing participation such as clarity of research concept in the initial invitations or by using current registry applicant lists as a potential sampling frame.
The participants of the study showed an obvious desire to gain admittance to the registries though the lists are not fully understood by the program creators. Participants shared that they would feel a sense of justification by becoming evidence-based and by being able to act as a resource for peers. In this sense, it was clear that participants cared about what others thought of their program. Also in support of this notion, participants expressed the level of professional credibility that they would enjoy with evidence-based status. Complimentary to these motivations, the program creators explained the number of iterations that their programs experienced to continually improve the product. They conceded to the notion of evolution in a way that showed humility in their process. In conclusion, the results of these two focus groups provided a fuller understanding of the experiences of those program creators seeking admittance to the lists and laid the foundation for future research exploring the process of pursuing evidence-based status.