State Department and Provider Agency Utilization of Evidence-Based Program Registries in Behavioral Healthcare and Child Welfare

Abstract

Evidence-based program registries (EBPRs) are web-based compilations of behavioral healthcare programs/interventions that rely on research-based criteria to rate program efficacy or effectiveness for support of programmatic decision-making. The objective was to determine the extent to which behavioral health decision-makers access EBPRs and to understand whether and exactly how they use the information obtained from EPBRs. Single State Authorities (SSAs) and service provider agencies in the areas of behavioral health and child welfare were recruited nationally. Senior staff (n = 375) responsible for the selection and implementation of programs and/or policies were interviewed by telephone concerning their visits (if any) to 28 relevant EBPRs, the types of information they were seeking, whether they found it, and how they may have used that information to effect changes in their organizations. At least one EBPR was visited by 80% of the respondents, with a median of three different registers being visited. Most visitors (55%) found all the information they were seeking; those who did not desired more guidance or tools for individual program implementation or were unable to locate the program or practice that they were seeking. Most visitors (65%) related using the information obtained to make changes in their organizations, in particular to select, start or change a program, or to support the adoption or improvement of evidence-based clinical practices. EBPRs were shown to be important resources for dissemination of research-based program effectiveness data, leading to increased use of evidence-based practices in the field, but the study also identified needs for greater awareness of EBPRs generally and for more attention to implementation of specific recommended programs and practices.

Keywords

evidence-based programs evidence-based program registries program effectiveness best practices program evaluation behavioral health

Introduction

Behavioral health providers and statutory agencies at all levels of government and across such diverse disciplines as addiction, mental health, nursing, education, and criminal justice are increasingly tasked with implementing effective treatment and prevention policies and practices that are supported by evidence of effectiveness using high quality evaluation methods. (D’Angelo et al., 2017; Data Coalition, 2017; Eval4Action, 2021; Lundgren & Rieckmann, 2011; Pew Results First Clearinghouse Database, 2021; White House, 2021; Zimerman, 2013). With respect to behavioral healthcare, the current Biden administration’s plan to address the drug addiction challenge includes supporting and expanding access to evidence-based treatment, prevention, and harm reduction (Psychiatric Times, 2021). There is a massive amount of research and evaluation data on treatment programs and policy initiatives available to these decision makers. Reviewing and evaluating this research information often requires staffing, financial, and informational resources that are beyond the resources of even large state departments or provider agencies. Evidence-based program registries have emerged as an important tool for turning this research data into actionable information, so that decision-makers and service providers can identify and implement effective behavioral health programs and policies (Burkhardt et al., 2015; Horne, 2017; Westbrook et al., 2017).

Evidence-based program registries (EBPRs) are web-based compilations of programs/models, or systematic reviews of such programs/models, that apply research-based criteria to rate their efficacy or effectiveness for support of programmatic decision-making. Such EBPRs are typically operated by government agencies or non-profit organizations (Burkhardt et al., 2015; Gillen et al., 2013; Government Accountability Office 10-30, 2009; Kingston et al., 2016; Means et al., 2015; Westbrook et al., 2017). Most EBPRs assess the effectiveness of individal programs, while others focus on evidence reviews, meta-analyses, and assessment of different program models and clinical approaches. Websites that include EBPRs may also provide a variety of other resources to support evidence-based programming (Burkhardt et al., 2015).

EBPRs relevant to behavioral healthcare review and evaluate evidence-based programs (EBPs) related to substance misuse, mental health, child welfare, criminal justice, and some facets of the education system (i.e., dropout prevention) (Gillen et al., 2013; Goesling et al., 2017; Hennessy et al., 2006; Horne, 2017). These EBPRs carefully screen, select, and evaluate the supporting research studies of program effectiveness using explicit sets of criteria and standards of evidence (Gillen et al., 2013; Hennessy et al., 2006; Paulsell et al., 2014; Westbrook et al., 2017), and usually provide summaries of these evaluations in the form of an “evidence rating” meant to inform and facilitate decision-making (Burkhardt et al., 2015; Kingston et al., 2016; Means et al., 2015). More details describing EBPRs may be found in Burkhardt et al. (2015) and Means et al. (2015).

In each state in the U.S., the delivery of behavioral health services is regulated by a Single State Authority (SSA), that has the authority to oversee and regulate the provision of such services within the state, such as the Michigan Department of Health and Human Services (Rieckmann et al., 2015). Although a wide variety of audiences use EBPRs, given the existing statutory and funding pressures to use evidence-based programs (EBPs), the staff of these SSAs constitute an especially important potential audience for EBPRs (Burkhardt et al., 2015). Prior studies have examined specific EBPRs or special topics regarding EBPRs (Buckley et al., 2020; Green-Hennessy, 2013; Green-Hennessy, 2018; Horne, 2017; Mihalic & Elliott, 2015; Maranda et al., 2021; Paulsell et al., 2014, 2017; Stephenson, 2017; Walker et al., 2017; Zack et al., 2019). However, there is no systematic research on the extent to which behavioral health decision-makers access EBPRs generally and whether and how they actually use the information obtained from EPBRs. This study addresses this gap in the existing research by addressing the following primary research questions: (1) To what extent do SSA and service provider agency staff access EBPRs related to behavioral health & child welfare? (2) What types of information did the staff obtain and did it meet their needs? (3) Did the respondents use the information obtained and in what ways? (4) What did respondents like or find useful about the EBPR websites and what are their recommendations for improving them?

The scope of the present study includes 28 EBPRs relevant to behavioral healthcare. This is the universe of EBPRs in the English language as identified from the list published by Burkhardt et al. (2015) and from an updated search of the internet conducted by the authors. These EBPRs have generally been developed and sponsored independently of each other. Some encompass the entire field of behavioral healthcare while others are specialized—limited to specific behavioral health topics or clinical populations. There has been no discernable coordination in the conceptualization, development or funding of these EBPRS. Consequently, this creates possibilities both for duplication of effort and for gaps in the coverage of behavioral healthcare interventions.

Method

Overview

The present study relies on telephone interviews of staff in the SSAs for behavioral health, the SSAs for child welfare, service provider agencies for behavioral health, and service provider agencies for child welfare. Child welfare SSAs and service providers were explicitly included in the study because many of the services provided to children and their families fall within the scope of behavioral healthcare, for example, mental health services for children or Functional Family Therapy, (National Council for Behavioral Health, 2021). Additionally, many EBPRs include such child welfare-related interventions and services.

SSA and provider agency staff (n = 375) responsible for the selection and implementation of programs and/or policies were interviewed concerning their visits (if any) to any of 28 relevant EBPRs, the types of information they were seeking, whether they found it, and the actual or potential utility of that information for their organizations. No personal identifying information of respondents was recorded on the interview.

Recruitment of Behavioral Health and Child Welfare SSAs

The present study interviewed representatives from two types of agencies—SSAs responsible for oversight of behavioral health services and provider agencies responsible for the delivery of services to clients. For each group, both behavioral health and child welfare agencies were included. The respondents for each type of agency were recruited separately through the mechanisms detailed in the following paragraphs.

Nationally the SSAs for mental health and addiction services are consolidated in state departments of health, behavioral health, or human services, with few exceptions. The commissioners of the relevant departments are all members of the National Association of State Mental Health Program Directors (NASMHPD). In the few states where addiction treatment was separate from mental health (e.g., New York State), or where the mental health commissioner was not also the commissioner for addiction services, the study’s principal investigator contacted the addiction services department directly regarding participation. The District of Columbia (D.C.), while not a state, was included in the recruitment of state departments. A senior administrator of the NASMHPD Research Institute provided an introductory e-mail for the study to the department commissioners or directors. The study then followed-up with a request to interview one or more senior staff who were responsible for the selection and implementation of behavioral health interventions.

Child welfare SSAs were recruited with the assistance of the National Association of Public Child Welfare Administrators (NAPCWA), an affiliate of the American Public Human Service Association (APHSA). The leadership of all child welfare SSAs belong to this organization. A senior administrator of the APHSA provided an introductory e-mail for the study to the department commissioners or directors. The study then followed-up with a request to interview one or more senior staff who were responsible for the selection and implementation of behavioral health interventions.

Recruitment of Behavioral Healthcare and Child Welfare Provider Agencies

The recruitment of behavioral health and child welfare agencies involved the utilization of two major resources. The first resource was a contact list from the National Council for Behavioral Healthcare, which is the largest membership organization for such service provider agencies and which also includes agencies providing child welfare services. Member agencies were recruited through introductory e-mails to the agency directors by a senior administrator of the Council, several announcements in the Council’s newsletter, and by direct e-mails from the study citing the Council’s endorsement of the study. The research team worked from a membership list of 2565 agencies current as of October 2017. The request was to interview a senior staff member who was responsible for the selection and implementation of behavioral health interventions. Provider agencies were offered a $25 gift certificate for the participating staff member or for the agency. Two reminders were sent to initial non-responding agencies.

Because the foregoing method did not yield sufficient interviews from provider agencies, we also recruited service provider agencies through a second resource—a tailored list generated from Guidestar (2021), which is a comprehensive database of non-profit organizations in the U.S. Potential behavioral healthcare and child welfare provider agencies were selected according to the tax filing status of the organization (IRS Category), the major activity category of the organization (National Taxonomy of Exempt Entities [NTEE] Code), and the specific focus of the organization (NTEE Subcode) (National Center for Charitable Statistics, n.d.). A total of 2523 provider agencies met the inclusion criteria nationally. The research team sent e-mails to the directors of these agencies requesting to interview a senior staff member who was responsible for the selection and implementation of behavioral health interventions. Provider agencies were offered a $25 gift certificate for the participating staff member or for the agencies. Two reminders were sent to initial non-responding agencies.

Sampling Results

A flowchart of the sampling results appears in Figure 1. The results of contacting the SSAs responsible for behavioral healthcare and/or child welfare for administrator interviews were as follows. Forty-eight out of the 51 states (including D.C.), or 94%, were represented by departments with behavioral health responsibilities and 37 out of the 51 states (including D.C.), or 73%, were represented by departments with child welfare responsibilities. (Two states had a department that combined behavioral health and child welfare responsibilities.)

Figure 1.

Study Recruitment/Sampling Flowchart. 1. Nine provider agency staff are omitted from final sample due to having unclassifiable service areas.

Of 2565 invitations for interviews sent to the National Council member agencies, interviews were completed for 131 staff (5.1% of the agencies). Of 2825 invitations sent to provider agencies from the Guidestar-generated list, interviews were completed for 86 staff (3.0% of the agencies). (Response rates are considered in the Discussion.)

In total, 375 interviews were completed. Of these 375 completed interviews, 167 (44.5%) were from SSA staff, and 208 (55.5%) were from provider agency staff. Most respondents who completed an interview were from behavioral health agencies (n = 259, 69.1%), followed by child welfare agencies (n = 64, 17.1%) and then by agencies that dealt with both behavioral health and child welfare (n = 52, 13.9%). Of 375 respondents, 301 (80.3%) indicated that they had visited and were able to name an EBPR, forming the final analytic sample of 301 for the study.

The number of respondents exceeds the numbers of SSAs because, at our request, often the departments referred us to more than one administrator who met our inclusion criterion. All of these respondents were interviewed, so that multiple staff members from some SSAs were interviewed. This was considered important because the SSAs are invariably large and set policy and practice for a whole state across a range of relevant services. The average number of attempted interviews per state was 3.2, with a range of 1–9 respondents interviewed. The average number of respondents interviewed per agency was 1.65, with a range of 1 to 5 respondents being interviewed. The study only interviewed one staff member per provider agency. Interviewing one staff member per provider agency was considered reasonable, as provider agencies tend to be smaller with fewer staff than are found in the SSAs. Staff interviews were conducted between July 1, 2018 and October 9, 2020.

Measures

Contacts to schedule interviews were divided among three organizations which collected the data according to region of the U.S: Western Michigan University for central states; Rady Children’s Hospital—San Diego for western states; and Development Services Group Inc. for eastern states). Data were collected with a semi-structured telephone interview comprised of both fixed response and open-ended questions. We allowed for narrative qualitative responses important for capturing types of information that could not be anticipated or predicted. Specifically, the interview consisted of two kinds of questions—single choice questions (e.g., yes/no/maybe) and open-ended questions related to study variables (e.g., “what information are you usually looking for when you visited the website”). For the open-ended questions, there were no structured responses—respondents were allowed to freely respond to the question prompt. The questions that were asked are given in the text below and in the Appendix A.

Staff scheduled to be interviewed were e-mailed a list of 28 EBPRs prior to the interview. During the interview they were asked, “Have you visited any of the EBPRs [on the list]?” Individuals who responded that they had visited at least one of the EBPRs, or who said they visited and named a registry not on the list, were asked about their experiences with the registry site(s). Staff members who indicated that they had not visited any EBPRs were asked an alternate set of questions relating to where they accessed information on EBPs, whether such information was satisfactory, what further information they needed, and whether they would be likely to visit an EBPR website in the future.

Coding Procedures

The project managers developed a codebook for coding the narrative responses to interview questions by reviewing the responses provided and inductively developing the coding schemes based on those responses, according to standard procedures for thematic coding of qualitative data (Guest et al., 2012). The code development process was initiated by two senior researchers, who reviewed a sample of responses to the open-ended questions and inductively coded these responses. This initial set of codes was then revised by the principal investigator. Subsequently, coders from each of the partner organizations coded the same set of responses and were asked to provide feedback on ambiguous or confusing codes. The coding from each of the coders was then compared to identify those codes that provided the lowest coding agreements. These codes were further revised until any ambiguities were resolved. This final set of codes formed the codebook for coding all open-ended responses included in the present study.

Once the codebook was developed, interviews were randomly assigned to one of two collaborating organizations for coding. After training on the codebook, pairs of coders within each organization reviewed each response and reached a consensus about how to code that response. These consensus codes were then applied to the responses. Our main concern was that no response would be coded by a single coder. Up to three codes from the codebook could be applied to each response, and there was no minimum number of codes applied. Because the responses were coded by consensus by two researchers, no inter-rater reliability statistics were calculated. Tabular analysis is used to present the data.

Institutional Review Board (IRB) Review

The study was reviewed by the IRB of Western Michigan University which determined that the scope of work did not meet the federal definition of involving human subjects. The reviewers of the application at the National Institute on Drug Abuse also determined it was exempt from human subjects research regulations.

Results

Research Question #1

The respondents were provided with a list of 28 EBPRs prior to the interview; they were asked whether they ever visited any of those websites and if so, which ones they had visited at least once. Of the 375 respondents interviewed, 301 (80.3%) indicated that they had visited at least one EBPR website and were able to name one, forming the final analytic sample (Figure 1).

Table 1 shows the percentage of respondents who visited each of the 28 listed EBPRs; registries not on the list that were named are in the table footnotes. The most frequently visited registries (more than 30% of the respondents) were the National Registry of Evidence-Based Programs and Practices (NREPP), the Substance Abuse and Mental Health Services Administration (SAMHSA) Evidence-based Practices (EBP) Resource Center, the Office of Juvenile Justice & Delinquency Prevention (OJJDP) Model Programs Guide, the Suicide Prevention Resource Center (SPRC) and the California Evidence-Based Clearinghouse for Child Welfare (CEBC). Of respondents visiting at least one registry, 80.7% visited more than one. The mean number of registries visited was four and the median number was three. 12 respondents visited more than 10 registries, slightly positively skewing the distribution of registries visited. The results of chi-square analyses indicate statistically significant differences between agency types on which of the top 5 registries were visited. For example, 76.9% of SSA respondents reported visiting NREPP, while only 54.3% of provider agency respondents visited NREPP (Table 2, first question).

Table 1.

Frequency of Registry Visits by Those Who Visited at Least One Registry (N = 301 respondents).

Registry^a,b	%
National Registry of Evidence-based Programs and Practices (NREPP) (F)^c,d	63.1
Substance Abuse and Mental Health Services Administration (SAMHSA) Evidence-based Practices (EBP) Resource Center (F)	47.5
Office of Juvenile Justice & Delinquency Prevention (OJJDP) Model Programs Guide (F)	34.2
Suicide Prevention Resource Center (SPRC)	33.2
California Evidence-based Clearinghouse for Child Welfare (CEBC)	31.6
Evidence-based Practices for Substance Use Disorders (Univ. Of Washington)	17.3
Blueprints for Healthy Youth Development (Blueprints)	16.9
Child Trends/What Works^e	16.6
Youth.gov (F)	12.3
What Works Clearinghouse (WWC) (F)	12.0
Effective Child Therapy: Evidence-based Mental Health Treatment for Children & Adolescents	12.0
PracticeWise	11.3
Home Visiting Evidence of Effectiveness (HomVEE) (F)	10.0
CrimeSolutions.gov (F)	8.0
Cochrane Database of Systematic Reviews	8.0
Pew Results First Clearinghouse Database^f	7.6
Collaborative for Academic, Social, and Emotional Learning (CASEL)	7.0
Compendium of Evidence-based Interventions and Best Practices for HIV Prevention (CDC) (F)	7.0
Teen Pregnancy Prevention Evidence Review (F)	6.3
Connecticut Clearinghouse	5.6
Society of Clinical Psychology	5.0
What Works in Reentry Clearinghouse	5.0
Effective Interventions: HIV Prevention that Works (CDC) (F)	4.3
Clearinghouse for Military Family Readiness	4.0
Resource Center for Adolescent Pregnancy Prevention (ReCAPP)	2.7
Social Programs that Work	2.7
The Campbell Collaboration	1.3
Centre for Reviews and Dissemination (York University)	0.3
Other/Unknown	14.6

^aIncludes visiting several not on the initial list.

^bThe following registries were not on the registry list initially presented to respondents, but were named as registries they visited. Washington State Institute for Public Policy (3 respondents), Promising Practices Network (2 respondents), National Child Traumatic Stress Network (2 respondents), FriendsNRC.org (1 respondent), Boston University Mental Health Clearinghouse (1 respondent), The Community Guide (1 respondent).

^c(F) denotes federally administered registry. However, there are others that are federally funded but administered by private organizations.

^dNREPP was suspended in Jan. 2018 and replaced by the SAMHSA EBP Resource Center. Although the latter does provide “evidence reviews” of programs and practices, some embedded in more general publications, most of the new site is devoted to other kinds of resources to support EBPs. Some NREPP data are still accessible at the Pew Results First Clearinghouse Database (2021).

^eThe What Works component of the site has been dropped. Research syntheses based on the What Works database are available at https://www.childtrends.org/links-syntheses and the main Child Trends site contains additional evaluation studies of programs, models, and clinical approaches.

^fThe previous NREPP ratings for many programs are included on this site.

Table 2.

Inferential Statistical Analysis for Selected Questions.

Which of these registries have you visited? (Top 5 responses, N = 301)
	SSA (N = 117)	Provider (N = 184)	Chi-square
National Registry of Evidence-based Programs and Practices	76.9	54.3	15.66**
SAMHSA Evidence-based Practice Resource Center	18.8	65.8	63.24**
OJJDP Model Programs Guide	41.0	29.9	3.94*
Suicide Prevention Resource Center	33.3	33.2	.00
California Evidence-based Clearinghouse for Child Welfare	42.7	24.5	11.06**
What information are you usually looking for when you visit these websites? (N = 292)
	SSA % (N = 110)	Provider (N = 182)	Chi-square
Implementation information for a program (costs, training, implementation guides, fidelity scales, how to engage clients)	43.6	43.4	.00
Identify programs/models suitable for implementation	47.3	37.9	2.48
Information about effectiveness/strength of the evidence base or long-term outcomes of a program/model	50.9	29.1	13.91**
Have you used the information from any of these websites to make changes in your agency? (N = 298)
	SSA % (N = 116)	Provider % (N = 182	Chi-square
No	30.2	28.6	.391
Not sure	6.9	5.5
Yes	62.9	65.9
What changes have you made in your agency/organization as a result of visiting these websites? (N = 183)
	SSA (N = 67)	Provider % (N = 116)	Chi-square
Selected, started, changed, or implemented a program/model/practice	40.3	41.4	.05
Supported the adoption of evidence-based practices or improved current clinical practices	14.9	43.1	15.73**
Obtained/used resources to help with changing policy	26.9	28.4	.08

Note: Percentages are percent of respondents from each agency type that were coded with each code.

*Difference significant at p ≤ .05.

**Difference significant at p ≤ .01.

It is important to note that although each site includes, at least as one component, a registry or list of individual evidence-based programs or systematic reviews of programs (e.g., Cochrane), many have evolved to also offer a variety of training, technical assistance and other resources to support the dissemination of evidence-based programs and practices (e.g., the SPRC; the SAMHSA EBP Resource Center; the CEBC).

Research Question #2

When asked about what kinds of information they were seeking, the most frequent responses for 292 respondents involved implementation information for a program (costs, training, implementation guides, fidelity scales, how to engage clients) (43.5%); identifying programs/models suitable for implementation (41.4%); and Information about the effectiveness, strength of the evidence base or long-term outcomes of a program/model (37.3%).

The “implementation information” code includes information such as program costs, availability of manuals or other resources, staffing requirements, and training requirements, whereas the ”identifying programs/models” code means that the individual was searching for a suitable EBP to adopt. Chi-square analyses revealed no statistically significant differences between SSA and provider agency respondents in terms of seeking implementation information about programs and identifying programs to implement. There was a statistically significant difference between SSA and provider agencies on whether a respondent looked for information about the effectiveness of a program, with 50.9% of SSA and 29.1% of provider agency respondents endorsing this item (Table 2, second question).

When asked whether they found what they were looking for, 55.3% responded that they found what they were looking for “all of the time,” 20.6% indicated “most of the time,” 21.6% responded “sometimes or half the time,” and 2.4% responded that they had not found what they were looking for (n = 291). Regarding what kinds of desired information they could not find on the registries, the most frequent deficits noted by 161 respondents were: Detailed descriptions of programs/models (implementation or contact information, trainings, costs, fidelity scales) (32.4%); certain program or practices that respondent was searching for (27.6%); and Information about effectiveness with specific populations or treatment issues, including the full range of available programs (17.2%).

Those who visited more than one registry (n = 243) were asked whether they encountered any contradictory information in different registries—62.6% replied “no,” 19.3% replied “yes,” 16.5% replied “not sure,” and 1.6% did not respond. For those who perceived contradictory information, the interviewer asked about the nature of the contradictory information and how they may have resolved the contradiction(s). The most frequent problems noted by 41 respondents were that different registries gave different ratings to the same programs (53.7%) or used different standards or criteria to rate programs (34.1%). To resolve such contradictions, respondents most frequently used their own best judgment, including personal experience with a program (36.4%) or went to the primary evaluation studies to help make a decision (29.5%). Overall, the number of respondents who found contradictory information constituted a small portion of the overall sample.

Additional, less frequent responses to the interview questions pertinent to research question #2 are shown in the Appendix A.

Research Question #3

In response to the question “have you used any of the information from these websites to make changes in your agency,” 64.8% replied “yes,” 29.2% replied “no,” and 6.0% replied “not sure” (n = 298). A chi-square analysis revealed no significant differences between SSAs and provider agencies in whether the respondents made any changes in their organization or agency based on the information they obtained (Table 2, third question).

Respondents (n = 183) used the information most frequently at their agencies to select, start, change, or implement a program (41.0%); to support the adoption of evidence-based or improved clinical practices (32.8%); and to help with changing an agency policy (27.9%). The only significant difference was that provider respondents were more likely than SSA respondents to have supported the adoption of evidence-based practices or improvement of clinical practices (Table 2, fourth question).

Some verbatim responses help illustrate the sort of changes referred to by the staff more concretely: “We’ve chosen models using these registries - they have helped lead us to some practices that we had heard about, but didn't know enough, or that we found (in) this way”; “A nurturing parenting program has been implemented taken from the California Clearinghouse”; “We identified a program (for elderly drug users) on the SAMHSA website that met our needs, pitched it to our local funding sources, and we’ve been implementing it for five years”; “In one of the regulations we just developed for agencies to provide intensive behavioral health services - which referred to three sites, including NREPP - that they could use to justify their use of a specific program”; “We made a change about our tobacco policy, moving to a tobacco-free environment - we found information about that in SAMHSA best practice data.”

The most frequent reasons why some staff (n = 86) did not use the information from EBPRs to make any changes are that: Current programming and services are considered satisfactory (41.9%); lack of information from the registry that was relevant to their agencies’ needs (24.4%); and insufficient authority to make changes at their agency (9.3%).

Additional, less frequent responses to the interview questions pertinent to research question #3 are shown in the Appendix A.

Research Question #4

When asked what they liked most about the EBPR sites they visited, the most prevalent responses relate to what is termed “user friendliness,” a common desire, if not demand, of users of any website. Following were the accessibility of the research summaries, the consolidation of relevant programmatic information on one site and supportive resources for implementation of specific recommended programs (Table 3, first question).

Table 3.

Most Frequent Responses to Interview Questions for Research Question #4.

What do you like most about the websites (you visited)? (N = 263 respondents)	%
“User-friendly:” website is simple, consistent, visually appealing, good search engine, well organized, easy to locate the information you need.	49.8
Clear research descriptions or summaries with details and evidence about program effectiveness	24.7
All evidence-based program information consolidated in one place; “one stop shopping”	15.6
Website provides trainings, toolkits, manuals, webinars, etc.	11.8
What do you find is useful about the websites (you visited)? (N = 285 respondents)	%
Detailed descriptions of programs/models (implementation and contact information, trainings, costs, fidelity scales, etc.)	37.5
Website properties (easy to navigate, user friendly, good starting point, accessible, good research tool, etc.)	31.2
Evaluation data, effectiveness data, outcome data for programs/models	20.0
Able to find programs relevant to our clinical context, or able to identify if a current program is appropriate for our context	18.9
Good overview or synopsis of models or clinical issues	18.9
For the website that you selected as most useful: What makes it the most useful website? (N = 230 respondents)	%
Website properties (easy to navigate, user friendly, good starting point, accessible, etc.)	28.7
The depth or breadth of the information sources	27.8
Able to determine relevance and appropriateness of programs for own clinical or operational contexts, or find relevant programs	22.2
Site is familiar or trusted	21.7
Detailed descriptions of programs/models (implementation and contact information, trainings, costs, fidelity scales, etc.)	13.9
How could these websites be improved in your opinion? (N = 252)	%
Make them more user-friendly, accessible, easier to navigate, improve search tools, improve device-responsiveness	34.5
Not much could change, no changes needed	25.0
Provide more information to support implementation (costs, relevant populations, limitations, required trainings, implementation guides, etc.,)	16.3

Note: Percentage is percentage of respondents who gave a response classified in each category. Percentages add to more than 100% because respondents could be classified in more than one category.

As a follow-up question, interviewers asked about what was useful about the EBPRs they visited (Table 3, second question). The most frequent responses related to detailed description of programs/model and program implementation resources. Next were user friendliness, reiterating the previous finding; evaluation and effectiveness data; inclusion of programs relevant to the respondent’s clinical context or client population; and a good overview or synopsis of models or clinical issues.

If respondents visited more than one site, they were asked to identify the most useful site and what makes it the most useful (Table 3, third question). Their responses were generally consistent with previous responses in this sequence of questions: user friendliness, the extent of informational resources, and a recurring theme, the appropriateness or relevance of the programs reviewed for the respondent’s clinical context. Sites that are familiar and trusted and provide detailed information on program implementation, are also preferred.

Finally, interviewers inquired about respondents’ opinions about how the EBPR sites they visited could be improved (Table 3, fourth question). Enhanced user-friendliness was the most frequent recommendation, consistent with what respondents liked most about EBPRs which did have that characteristic. The second most common response was that nothing needed to be changed. The desire for more guidance or tools for program implementation was mentioned third most frequently, reiterating previous concern about this.

Additional, less frequent responses to the interview questions pertinent to research question #4 are shown in the Appendix A.

Discussion

Response Rates

The interview response rates for service provider agencies were lower than expected. However, this may be less potentially biasing than in other survey research because the present study can be interpreted as an examination of the utility (including perceived limitations) of EBPRs for staff who are motivated to access and use EBPRs.

Research Question #1

A large majority of respondents had visited at least one EBPR with a median of three EBPRs being visited among those who visited any. The two most frequently visited registries were NREPP and the EBP Resource Center, both previously or currently administered by the Substance Abuse and Mental Health Services Administration (SAMHSA). Note that SAMHSA stopped updating NREPP in January 2018 and made it inaccessible as of August 2018, as reported by the Pew MacArthur Results First Clearinghouse Database (2021). Although the latter still contains many NREPP program reviews, the link to its website directs visitors to a third-party archived version of the NREPP website, where available. This means that NREPP was effectively inaccessible for one half of the study data collection period. This potentially may have depressed its rate of use as shown in Table 1. However, the question about which registries were visited by the respondent did not have a time frame condition associated with it, which means that respondents could have reported ever visiting NREPP, regardless of whether it happened during the data collection period.

The fourth most frequently visited EBPR, the SPRC, is administered by a private entity but funded by SAMHSA. Because SAMHSA is by far the largest U.S. funder of addiction prevention and treatment and mental health services, as well as specifically suicide prevention services, it is understandable that state SSAs and service providers would be seeking programmatic guidance from these sites.

The third most frequently visited EBPR, the OJJDP Model Programs Guide, is operated by a federal agency, the Office of Juvenile Justice and Delinquency Prevention, which also has a large constituency dependent on this office for programmatic funding. The fifth most frequently visited registry, the CEBC, is funded by the state of California as part of its efforts to increase the use of EBPs in the child welfare and family service systems; while use of EBPs is encouraged, funding by the state is not tied directly to program ratings.

Following the top five registries, visitation frequency drops to more modest levels, whether or not the registries are federally administered. Similarly, Maranda et al. (2021) found state department website references to EBPRs were skewed to a small number of registries. This may be due to the specialized nature of many of the registries and the smaller constituencies for their specialized topics. But it may also be due to a lack of awareness of the full scope of the available registries, although we did not confirm this directly and have no evidence that checking more registries would result in improved programmatic decision-making. Anecdotally, when presented with our list of 28 registries, many of the respondents offered that they had never heard of most of them. Although the respondents who visited any registries were generally satisfied with the information they obtained (see below), registries they did not visit may also contain additional relevant and non-duplicative EBP information. Overall, It seems that it would be important to make the behavioral health and child welfare fields aware of the full breadth and depth of the EBPRs currently publicly available.

Research Question #2

The major types of information that respondents were seeking is generally consistent with typical stated purposes of the EBPRs, that is, “to support practitioners in the delivery of effective services” and “to inform the implementation of effective programs at the program design level” (Burkhardt et al., 2015). A majority (55%) of the respondents found what they were seeking from the EBPRs all the time, but the remainder indicated some degree of difficulty. When asked what desired information they could not find, almost one-third (32%) cited detailed program descriptions including implementation information for specific programs/models. Although many EBPRs offer general guidance and resources for EBP implementation, fewer include detailed instruction and guidance for the implementation of the individual programs or clinical practices that are recommended. For example, a recent examination of 11 prevention-related EBPRs found that few had standards about dissemination readiness of the recommended individual interventions or offered implementation information or individualized support to users (Buckley et al., 2020). NREPP appears to have been the only EBPR that included a separate rating for the adequacy of implementation information, which was termed “Readiness for Dissemination,” although that rating had not been applied to programs rated by NREPP after 2015. Increased attention to providing program implementation information on EBPRs would be beneficial to meeting the needs of users.

More than a quarter (28%) of the respondents could not locate the program or practice they were seeking on the registry sites they visited, a problem that might be solved by accessing a larger number of registry sites; but this is not guaranteed. Since there is no coordination among the population of registries, some relevant programs or practices may well not be reviewed by any of them.

There was also a concern regarding how effective a program might be with a specific client population or clinical issue that was not included in the existing program evaluations. This suggests that, after the effectiveness of a program/intervention with a primary population has been determined, there should be greater attention to conducting supplementary evaluations with special populations or focusing on less common but nonetheless significant clinical issues. Additionally, this may be a limitation of some EBPRs that have a narrower focus—for example, a register focused on juvenile justice issues may not list any programs that are also effective for adults.

About a fifth of staff visiting more than one registry stated they encountered contradictory information. Most of those stated that the conclusions or recommendations regarding specific programs, practices or models were not consistent across two or more registries. The relatively low proportion of staff who reported finding inconsistencies between registries seems to contradict documented inconsistencies in program ratings across registries reported by other studies (Zack et al., 2019; Walker et al., 2017; Means et al., 2015). However, lack of awareness of conflicts between registries in recommendations may simply mean that respondents were not attending closely to the existence of conflicts or not seeking guidance about a specific program or practice in more than one register at a time.

Research Question #3

The changes made at respondents’ agencies resulting from information obtained from EBPRs certainly are in accord with the intent of the registries. These findings are particularly important because they document that the respondents are not merely browsing these registry sites, but are seeking, often finding, and often utilizing, the information found to enhance evidence-based programming, clinical practice or related policies in their organizations. This use may not be directly instrumental in the sense that many respondents reported utilizing the registries in ways other than selecting new programs. Thus, the utility of EBPRs for staff solely seeking new programs may be different than for those looking for program implementation information.

Prior research has noted a declining rate of adoption of evidence-based treatments (Bruns et al., 2016). Since our sample consists predominantly of motivated users of EBPRs, as indicated above, our finding may not be inconsistent with prior research. Those who access EBPRs find them of value, but there are arguably many other staff and organizations who are not utilizing EBPRs or implementing programs recommended by them. This question cannot be answered until a study with a more representative, probabilistic sampling strategy is conducted.

There was a significant difference between the SSAs and provider agencies in whether they used the information to support adoption of new clinical practices or improvement of existing practices, with the provider agencies noting this more often than the SSAs (4th question, Table 2). This is likely because SSAs are not directly responsible for adopting new practices.

Research Question #4

The generally positive comments about the characteristics of the registry websites indicates that their user friendliness and the perceived value of their content will continue to facilitate their utilization by the behavioral healthcare field. Nevertheless, respondents also recommended several thoughtful improvements in format and content that deserve consideration wherever applicable. Many EBPRs need to continue to work to improve user friendliness still further. Additionally, respondents recommended that many EBPRs needed to provide more support for program implementation. Some respondents also suggested that the content from multiple EBPRs could be consolidated into fewer registries. This process was already begun by the Pew Results First Clearinghouse Database, which consolidates the reviews from nine other major EBPRs (including NREPP, the CEBC and the What Works Clearinghouse). However, more work could be done in this area.

Limitations

The staff sample is certainly a convenience sample. We cannot say how representative it is of behavioral health agency staff nationally. Additionally, there may also be a difference in the rate of participation of staff due to differences in recruitment procedures, in particular because provider staff were incentivized with a gift certificate to participate, whereas SSA staff were not.

Given that the present study uses a convenience sample, one could assume that the results obtained may vary depending on who is included. For example, if more state agency representatives were interviewed, then the item “information about what is working in other states” might have received a higher rate of endorsement in response to the question “what kinds of information are you usually looking for when you visit these websites?” Alternately, it may be that if fewer provider agencies were sampled, the item “implementation information” would have received less endorsement in response to the same question. Finally, it might be assumed that individuals who agreed to the interview may already be more interested in the use of EBPs and EBPRs, that is, are more motivated users of EBPRs.

The coding process used two coders per response, rather than using two coders rating responses independently. As a result, inter-coder reliability statistics were not obtained. Thus, it is unclear whether other codes would have produced greater or lesser agreement between raters. Initial resolution of coding disagreements during the development of the coding scheme may offset some of this uncertainty; however, the degree to which this is true cannot be assessed.

The study was not designed to evaluate the utility of specific EBPRs. Also, not all types of agencies that might use these EBPRs could be included in the current study, for example, state departments and provider agencies related to criminal justice services. This is probably why the criminal justice-related EBPR sites had lower utilization in this sample, although such sites do include extensive behavioral health interventions that would be relevant to the study respondents; there is overlap between behavioral healthcare and criminal justice populations.

Finally, it is unknown whether the results would differ if respondents had not been given a list of extant EBPRs, but instead had been asked to recall visits to websites that listed evidence-based programs.

Conclusions

Web-based evidence-based program registries (EBPRs) are an important potential resource for disseminating evidence of effectiveness for behavioral health programs, practices, and policies. The convenience sampling yielded respondents who usually accessed multiple EBPRs, although visits were skewed toward a small number of federally sponsored registries; SSAs and provider agencies should be made more aware of full range of EBPRs available to them. Respondents generally were seeking information consistent with stated purposes of the registries, except that there was also a desire for more instruction on implementing specific recommended programs, which many registries do not emphasize. Respondents also preferred sites with programs that could be identified as relevant or tailored to the respondent’s specific clinical context or client population. Respondents did not merely browse the registry sites; rather a majority were also able to relate changes they made at their agencies attributable to the information they obtained. Respondents considered user-friendliness of the sites as vital. The need for more and better program implementation guidance was mentioned often. EBPRs were shown to be important resources for dissemination of research-based program effectiveness data, potentially leading to increased use of EBPs, but needs for greater awareness of EBPRs generally and more attention to tools for implementing recommended programs and practices were also identified.

One caveat to the conclusions reached in the present study is that different users may have different experiences concerning the relative utility of the sites. It is reasonable to think that no one register can meet all the needs of all potential users. These EBPRs may not identify or rate a specific program or type of program that meets the needs of a special population, or they may not provide adequate implementation guidance to users who have already selected a program. Thus, one should not view EBPRs as the sole source of authoritative information—users need to seek additional sources of information about EBPs to support evidence-based decision-making for behavioral health.

Footnotes

Appendix

Appendix A

Research Question #2: Additional Responses
What kinds of information are you usually looking for when you visit these websites? (n = 292 respondents)	%
Information about a program/model that does not include strength of evidence or implementation resources (e.g., relevance to population, contact info, historical information)	17.5
Survey the existing EBPs in my field or get the latest knowledge about EBPs	17.5
Information that will help our agency respond to an RFP	10.6
General statistics and data on a behavioral health problem or services not specific to a particular program	6.2
Information about what is working in other states or localities.	5.5
Other resources, information, or tools not included in other categories	4.5
What desired information could you not find? (N = 161 respondents)	%
Detailed information about research or evaluations conducted	14.5
Most up to date information, or clarifications on what is new or outdated	7.6
Information about other implementations/lessons learned	7.6
Given a certain level of evidence, how close is the program to the next level? (What would it take to get to the next level?)	1.4
How to obtain funding	1.4
Links to local service providers	1.4
Information about a specific population/problem/topic	0.7
Other	14.5
In what way is the information contradictory? (N = 41 respondents)	%
Ratings can differ based on populations, subgroups, or other study-level details	14.6
Ratings may change over time	4.9
Programs may not appear in registry even though they are peer-reviewed	2.4
Other	2.4
How do you handle the contradictory information? (N = 44 respondents)	%
Review more websites or other sources of information to verify the information	18.2
Contact the program developer, other users, experts, or authorities to get more information	18.2
Refer to a statutory agency, funding source, or authority within my own organization	13.6
Review and assess program context, populations, cultural relevance, funds availability, agency values, etc.	13.6
Go with my most trusted registry	9.1
Other	2.0
Research question #3: Additional responses
Describe how you used the information from these websites to make changes in your agency? (N = 183 respondents)	%
Enabled response to a statutory or funding requirement, facilitated proposal development, or complied with a grant or contract	11.5
Expanded list of approved programs, developed new programs or expanded client access to programs	7.1
Facilitate proposal development or helped respond to an RFP	6.0
Selected service providers or hired new staff based on information from the registry	2.2
Learned how to assess and evaluate programs	1.6
Other	9.8
Describe why you did not use the information from these websites to make changes in your agency? (N = 86 respondents)	%
Changes are in progress	7.0
Does not have resources to make changes	3.5
Did not have state mandate for evidence-based programs/practices	2.3
Other	14.0
Research question #4: Additional responses
What do you like most about the websites (you visited)? (N = 263 respondents)	%
Provides a comprehensive list of programs that are connected to multiple clinical areas	10.6
Provide population-specific information and statistics on prospective program population for respondent’s agency.	8.7
Website provides a rating system where ratings are described enabling comparisons	5.3
Ease of linking to the program developer’s page	1.9
Help when applying for a grant; up to date in the field.	1.9
Other	7.3
What do you find is useful about the websites (You visited)? (N = 285 respondents)	%
They are a one-stop source of comprehensive information on programs/models	10.9
Increased confidence in programs already being used or considered for use; confirm that certain programs are evidence-based	7.4
Source of programs that are likely to be eligible for funding and reimbursement	2.1
Information about grant requirements or available funding opportunities	1.4
Language and definitions consistent with own state language and definitions	0.4
Other	13.7
For the website that you selected as most useful: What makes it the most useful website? (N = 230 respondents)	%
Source of programs that are likely to be eligible for funding and reimbursement; useful for grant applications needing evidence-based programs	10.4
Increased confidence in programs already being used or considered for use, confirm that certain programs are evidence-based	7.4
Good links to other sources; more information	6.1
Information is up to date	2.6
Other	10.9
How could these websites be improved in your opinion? (N = 252 respondents)	%
Integration or consolidation of information from all registries into fewer registries or streamline the information	11.1
Increase public awareness by improved advertising, or by sending out email alerts for information changes, updates, or major new findings	7.5
Update information more frequently; show frequency of updates and the way its updated	6.7
Provide information about how effectiveness ratings are determined or how programs are vetted	6.0
Show the evidence behind a practice and provide examples of effective programs	5.6
More consistent formatting and information presentation, customized download, information put in layman’s terms and less technical	5.2
Provide information on how widely the program is used by other local agencies, states, and jurisdictions/regions.	2.0
More interaction from website’s customer/technical support	1.6
Have a single basic review method that is informed by evolving understanding of methods in systematic review	1.2
Provide a quick way to verify outcome results for funding	0.4
Improve the accuracy and currency of contact information on the website	0.4
Other	13.1

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Grant # R01 DA042036 from the National Institute on Drug Abuse.

Acknowledgments

Robert Boruch and Paul Roman reviewed a prior draft of this manuscript. Mary Terzian was a collaborator at an early stage of the study. Mary Ramlow provided valuable administrative support.

Author Contributions

Magura: Funding acquisition; conceptualization; methodology; project administration; supervision; formal analysis; writing-original draft. Lee: Methodology; data curation; formal analysis; writing-review and editing. Abu-Obaid: Investigation; project administration; writing-review and editing. Landsverk: Funding acquisition; Investigation; writing-review and editing. DeCamp: Investigation; project administration; writing-review and editing. Rolls-Reutz: Funding acquisition; Investigation; project administration; writing-review and editing. Green: Investigation; project administration; writing-review and editing. Ingoglia: Investigation; project administration; writing-review and editing. Hollen: Investigation; project administration; writing-review and editing. Flagg: Investigation; project administration; writing-review and editing.

ORCID iDs

Stephen Magura

Whitney DeCamp

Jennifer Rolls-Reutz

References

Bruns

E. J.

Kerns

S. E.

Pullman

M. D.

Hensley

S. W.

Lutterman

Hoagwood

K. E.

(2016). Research, data, and evidence-based treatment use in state behavioral health systems, 2001–2012. Psychiatric Services, 67(5), 496–503. https://doi.org/10.1176/appi.ps.201500014

Buckley

P. R.

Fagan

A. A.

Pampel

F. C.

Hill

K. G.

(2020). Making evidence-based interventions relevant for users: A comparison of requirements for dissemination readiness across program registries. Evaluation Review, 44(1), 51–83. https://doi.org/10.1177/0193841X20933776

Burkhardt

J. T.

Schröter

D. C.

Magura

Means

S. N.

Coryn

C. L. S.

(2015). An overview of evidence-based program registries (EBPRs) for behavioral health. Evaluation and Program Planning, 48, 92–99. https://doi.org/10.1016/j.evalprogplan.2014.09.006.

Data Coalition (2017). Foundations for evidence-based policymaking act of 2018. P.L. 115-435. https://www.datacoalition.org/

D’Angelo

Pullmann

M. D.

Lyon

A. R.

(2017). Community engagement strategies for implementation of a policy supporting evidence-based practices: A case study of Washington state. Administration and Policy in Mental Health and Mental Health Services Research, 44(1), 6–15. https://doi.org/10.1007/s10488-015-0664-7

Eval4Action (2021). Available at: https://www.eval4action.org/

Gillen

A. C.

Elefantis

A. B.

Hodgson

A. B.

Hennessy

K. D.

(2013). The international reach of SAMHSA’s national registry of evidence-based programs and practices. International Journal of Mental Health, 42(4), 78–94. https://doi.org/10.2753/imh0020-7411420404

Goesling

Oberlander

Trivits

(2017). High-stakes systematic reviews: A case study from the field of teen pregnancy prevention. Evaluation Review, 41(1), 27–49. https://doi.org/10.1177/0193841X16664658

Government Accountability Office 10-30 (2009). Program evaluation: A variety of rigorous methods can help identify effective interventions. https://www.gao.gov/products/gao-10-30.

10.

Green-Hennessy

(2013). Coverage of mental health and substance misuse topics in the cochrane review system. Epidemiology and Psychiatric Sciences, 22(2), 155–162. https://doi.org/10.1017/S2045796012000455

11.

Green-Hennessy

(2018). Suspension of the national registry of evidence-based programs and practices: The importance of adhering to the evidence. Substance Abuse Treatment, Prevention, and Policy, 13(1), 26. https://doi.org/10.1186/s13011-018-0162-5

12.

Guest

MacQueen

K. M.

Namey

E. E.

(2012). Applied thematic analysis. : Sage.

13.

Guidestar (2021). https://www.guidestar.org/

14.

Hennessy

K. D.

Finkbiner

Hill

(2006). The national registry of evidence-based programs and practices: A decision-support tool to advance the use of evidence-based services. International Journal of Mental Health, 35(2), 21–34. https://doi.org/10.2753/imh0020-7411350202

15.

Horne

C. S.

(2017). Assessing and strengthening evidence-based program registries’ usefulness for social service program replication and adaptation. Evaluation Review, 41(5), 407–435. https://doi.org/10.1177/0193841X15625014

16.

Kingston

B. E.

Mihalic

S. F.

Sigel

E. J.

(2016). Building an evidence-based multitiered system of supports for high-risk youth and communities. American Journal of Orthopsychiatry, 86(2), 132–143. https://doi.org/10.1037/ort0000110

17.

Lundgren

L.M.

Rieckmann

(2011). Research on implementing evidence-based practices in community-based addiction treatment programs: Policy and program implications. Evaluation and Program Planning, 34(4), 353–355. https://doi.org/10.1016/j.evalprogplan.2011.02.001

18.

Maranda

M. J.

Magura

Gugerty

Lee

M. J.

Landsverk

J. A.

Rolls-Reutz

Green

(2021). State behavioral health agency website references to evidence-based program registries. Evaluation and Program Planning, 85, 101906. https://doi.org/10.1016/j.evalprogplan.2021.101906.

19.

Means

S. N.

Magura

Burkhardt

J. T.

Schröter

D. C.

Coryn

C. L. S.

(2015). Comparing rating paradigms for evidence-based program registries in behavioral health: Evidentiary criteria and implications for assessing programs. Evaluation and Program Planning, 48, 100–116. https://doi.org/10.1016/j.evalprogplan.2014.09.007.

20.

Mihalic

S. F.

Elliott

D. S.

(2015). Evidence-based programs registry: Blueprints for healthy youth development. Evaluation and Program Planning, 48, 124–131. https://doi.org/10.1016/j.evalprogplan.2014.08.004.

21.

National Center for Charitable Statistics . (2000). IRS activity codes. https://nccs.urban.org/publication/irs-activity-codes.

22.

National Council for Behavioral Health (2021). https://www.thenationalcouncil.org/.

23.

Paulsell

Del Grosso

Supplee

(2014). Supporting replication and scale-up of evidence-based home visiting programs: Assessing the implementation knowledge base. American Journal of Public Health, 104(9), 1624–1632. https://doi.org/10.2105/AJPH.2014.301962

24.

Paulsell

Thomas

Monahan

Seftor

N. S.

(2017). A trusted source of information: How systematic reviews can support user decisions about adopting evidence-based programs. Evaluation Review, 41(1), 50–77. https://doi.org/10.1177/0193841X16665963

25.

Pew Results First Clearing house Database (2021). Evidence-based policymaking: A guide for effective government. https://www.pewtrusts.org/en/research-and-analysis/data-visualizations/2015/results-first-clearinghouse-database.

26.

Psychiatric Times . (2021). https://www.psychiatrictimes.com/view/biden-administration-plan-narrows-in-on-drug-addiction-crisis.

27.

Rieckmann

Abraham

Zwick

Rasplica

McCarty

(2015). A longitudinal study of state strategies and policies to accelerate evidence-based practices in the context of systems transformation. Health Services Research, 50(4), 1125–1145. https://doi.org/10.1111/1475-6773.12273

28.

Stephenson

A. L.

(2017). Journey toward evidence-based status: Seeking admission to formal program registries. Health Promotion Practice, 18(5), 681–687. https://doi.org/10.1177/1524839916670575

29.

Walker

S.C.

Lyon

A.R.

Aos

Trupin

E.W.

(2017). The consistencies and vagaries of the Washington state inventory of evidence-based practice: The definition of “evidence-based” in a policy context. Administration and Policy in Mental Health and Mental Health Services Research, 44(1), 42–54. https://doi.org/10.1007/s10488-015-0652-y

30.

Westbrook

T. R.

Avellar

S. A.

Seftor

(2017). Reviewing the reviews: Examining similarities and differences between federally funded evidence reviews. Evaluation Review, 41(3), 183–211. https://doi.org/10.1177/0193841X16666463

31.

White House . (2021). Memorandum on restoring trust in government through scientific integrity and evidence-based policymaking. https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/27/memorandum-on-restoring-trust-in-government-through-scientific-integrity-and-evidence-based-policymaking/

32.

Zack

M.K.

Karre

J.K.

Olson

Perkins

D.F.

(2019). Similarities and differences in program registries: A case study. Evaluation and Program Planning, 76, 101676. https://doi.org/10.1016/j.evalprogplan.2019.101676.

33.

Zimerman

A. L.

(2013). Evidence-based medicine: A short history of a modern medical movement. Virtual mentor. American Medical Association. https://doi.org/10.1001/virtualmentor.2013.15.1.mhst1-1301