Use of Research Evidence by Juvenile Justice and Youth Service Professionals

Literature Review

Some reasons offered to explain research underutilization may be invalid. Popular explanations may be based primarily on anecdotal evidence and speculation, therefore comprising an unsatisfactory set of reasons for why social research is not applied enough. These reasons may center on negative stereotypes (Bogenschneider & Corbett, 2010). For example, academicians may believe that policymakers and practitioners refuse to incorporate their research results and ideas because they interfere with their personal, cultural, political, or economic goals. In turn, policymakers and practitioners may believe that academicians mostly conduct research that is irrelevant to practice or promotes special interest agendas. Anecdotal evidence may at times support such claims, but much more research is needed to validate them. Nonetheless, as Weiss et al. (2008) point out, the gap between research and policy and practice is likely caused by characteristics of the research realm, characteristics of the practical realm, and the relationship between the two realms.

Some of the underutilization of research evidence can indeed be attributed to the context, beliefs, and actions of policymakers. Policymakers have been accused of making rushed judgments about research evidence, demanding premature and oversimplified presentations of findings, changing the purpose of projects after they have already started, ignoring some results or dismissing them on false grounds, spending little time reading and analyzing reports (failing to give attention to details), losing interest in projects, and, generally, basing decisions about the role of research evidence in political agendas and ideology rather than quality of the research itself (Weiss et al., 2008). Jennings and Hall (2011) found in a national survey sample that to varying degrees, state agencies do consult and consider scientific evidence but that they also base policy and program decisions on political and constituent sources of information (e.g., governors, legislators, other agencies and officials, and interest groups).

In their review of dissemination and utilization of research evidence in health and social services, Osterling and Austin (2008) identified lack of awareness of research, isolation from knowledgeable colleagues, and inability to evaluate quality of research as individual-level barriers and lack of time on the job to review research, staff and management non-supportive of implementing research, and lack of authority to make changes as organizational-level barriers. Similarly, Jack et al. (2010) found in a qualitative study that senior child welfare agency administrators valued research evidence but identified several obstacles to its use: environmental-level problems include non-evidence-based political interference, biased use of evidence at the governmental-level, inadequate financial resources for implementation, and a focus on reacting to crises; organizational-level problems include lack of information resources and time to review studies; and individual-level problems include lack of research knowledge and skills by supervisors and frontline staff.

Underutilization is also attributable to researchers. Problems with the body of research evidence itself may stand in the way of its application to policy and practice. Its dissemination and use is inhibited by confusion over conflicting results in the literature and what constitutes evidence, the time lag between the time when research is produced and when it is put into practice, and lack of generalizability to a local context (Osterling & Austin, 2008). Wright, Zhang, and Farabee (2010, p. 6) ask an important question: “What qualifies as ‘evidence’ and who gets to produce it?” The authors point out that some evidence-based program (EBP) research has serious flaws in its methodology. In some cases, evaluators examine interventions that they carried out, which presents a conflict of interest. Thus, some programs have not been properly evaluated, leading to the possibility that ineffective programs were supported by the research or that effective programs were not. Another problem exists when research evaluates improperly implemented programs (Wright et al., 2010). Good research will likely show such a program to be ineffective, which will be deceiving because the true strategy was not tested. Thus, it is possible that some strategies are good ideas that were never given a proper chance.

Largely, the problems with research evidence are the problems with social research in general (Weiss et al., 2008). Few studies are carried out with ideal methodological rigor. Data are usually quite flawed, and researcher/evaluator bias (e.g., toward funding agencies) interferes with methodological integrity and full reporting. Problems also exist in the dissemination of research, as publication bias conceals potentially important research (e.g., journals are reluctant to publish null findings suggesting that a program is no better than another). Findings across studies are inconsistent or contradictory, which is reflected in registries and clearing houses that often have little overlap in their lists of programs. Finally, even good evidence may not be broadly applicable. Studies often focus on individuals or isolated environments and fail to account for larger contexts, findings are historically contingent and thus may become outdated as society and individuals change, and findings may not address immediate practical needs (Weiss et al., 2008).

Evidence-based policy and practice must be carried out within supportive organizational cultures and by individuals who champion its cause (Jack et al., 2010). However, policymakers and practitioners must be able to have faith in research evidence. They must be able to believe that: (a) policies, programs, and practices have been properly implemented; and (b) the research examining their effectiveness is sound. Understandably, having this kind of faith will be difficult for many decision makers. Thus, it is important to understand what policymakers and practitioners consider to be “evidence,” which may be quite different from the positions of academics. Some critics conceptualize evidence not simply as the results of evidence research but the results of research that meets rigorous criteria (Wright et al., 2010).

However, a gap between even good research evidence and policy/practice appears to exist (Wright et al., 2010). Several barriers to communication between researchers and practitioners exist. These include their differing priorities and backgrounds, power dynamics between the two, practitioners’ lack of access to research reports and articles, unclear practical implications of studies, scattered relevant literature, and incomprehensible statistical analyses and presentation of studies (Osterling & Austin, 2008).

Why does this gap exist? The “two communities” theory, according to Caplan (1979), maintains that “social scientists and policymakers live in separate worlds with different and often conflicting value systems, different reward systems, and different languages” (p. 459). Researchers often work primarily in academic settings such as universities and research centers, whereas practitioners and policymakers often work strictly in the field. This separation of physical environments seems to be matched by a separation of research and service delivery (Bogenschneider & Corbett, 2010; Caplan, 1979). Researchers and field professionals also have different constituents, preset agendas, time frames for results, emphases on written versus spoken communication, concerns regarding validity of evidence, knowledge priorities, and several other daily concerns that facilitate misunderstandings about the realities and meaning of each other’s work (Weiss et al., 2008). Scholars and field professionals tend to discuss their work using professional jargon and dissemination outlets (e.g., journals vs. internal reports) to which each are unfamiliar, which makes establishing the “real world” relevance of research difficult. Because of these distant, potentially conflicting, work environments, researchers and field professionals find it difficult to understand each other’s needs and motives, collaborate, and bring research and practice together (Bogenschneider & Corbett, 2010; Caplan, 1979).

Efforts have been made to bridge this gap but many appear to be ineffective, despite that they are theoretically plausible and are based on good intentions. In an effort to develop strategies to influence social workers’ use of research, Gira, Kessler, and Poetner (2004) reviewed studies on use of research evidence in the health professions and identified strategies that received either weak or quite limited support. These were the following: (a) distributing or making available printed information such as guidelines, articles, journals, newsletters, and notebook compilations; (b) using local “opinion leaders,” persons with credibility and status regarding research evidence expertise, to influence colleagues; (c) continuous quality improvement reviews of practice at the organizational (not individual) level; (d) educational outreach visits by trained professionals who meet with providers to influence their performance; and (e) performance audit and feedback by experts, peers, and/or clients.

Ensuring that good research has a place in policymaking therefore requires more than just good intentions on the part of researchers, policymakers, and practitioners; bridges have to be formed between the two communities, which is no easy task given the nature of the obstacles just mentioned. Learning more about each other’s work environments would aid in forming better partnerships between researchers and policymakers and practitioners. For the academic, it is important to specifically understand how research is currently being acquired, viewed, and used (or not used) by field professionals. In their study on measuring the policy impact of health research, Lavis, Ross, McLeod, and Gildiner (2003) distinguished push, pull, and joint activities. Improving user-pull qualities of research and producer–user exchanges (joint action) will likely be of more immediate concern to field professionals than producer-push activities. Although it is important for researchers to promote their work, there is much to be gained by exploring the perspectives of field professionals. Perhaps policymakers and practitioners desire and attempt to apply research evidence but are unable or hesitant to do so for a variety of reasons.

The Current Study

In criminal justice, research evidence–based policy and programs appear to be emphasized more in corrections (Myers & Spraitz, 2011), directing more attention toward outcomes of decisions made by lawmakers and administrators in governmental agencies. Past research has paid little attention to the experiences of professionals who work more directly with delinquent and at-risk youth. While agency policy and practice is heavily guided by mandates from governmental and other authorities, local administrators and practitioners have much discretion in determining the policy and practices of their agencies. Ideally, these professionals seek out information on research evidence, which includes more than just evaluation studies, in making decisions. Thus, they represent an important population to study with respect to how research evidence is used in determining responses to crime and other problem behaviors.

The current study explores research use in a user community defined as professionals at agencies providing services to at-risk and offending youth. Research on user communities aim to understand the extent to which polices and practices are affected by research (Davies & Nutley, 2008). This study begins to extend the type of work done primarily in health and education into juvenile justice and child welfare. It seeks to add new information to prior findings on field professionals’ perceptions of barriers to using research evidence (e.g., Bogenschneider & Corbett, 2010; Nelson, Leffler, & Hansen, 2009). A select group of stakeholders was examined to assess their knowledge of research evidence, whether or not they use it to guide practice, and why or why not they use it.

Sample

The sample consists of members of a county-level disproportionate minority contact (DMC) steering committee in a Midwestern state. As a local site for a statewide initiative, the committee was formed to develop a multi-pronged implementation plan to reduce the disproportionate representation of children of color in the county’s child welfare and juvenile justice systems, and assess the results of this implementation over time. The committee consists of local leaders—managers, supervisors, and middle-level practitioners—representing a variety of areas including juvenile and criminal justice, social services, mental health, education, and faith-based organizations. Charged to develop and evaluate evidence-based and community-based strategies, the committee provides a valuable sample for exploring the use of research evidence in policy and practice.

A total of 35 professionals participated in 4 focus groups. Each group consisted of 8 to 10 participants. Ten participants identified themselves as male, 21 identified as female, and 4 did not indicate sex. Eleven identified as African American or Black, 19 identified as White, 2 identified as Hispanic or Latino, and 3 did not indicate race. Degrees held were as follows: 2 associates, 18 bachelor’s, 13 master’s, 1 Jurist Doctorate, and 1 non-indicated. Participants worked for child welfare service (7), law enforcement (6), mental health (5), court (5), juvenile justice or delinquency service (4), advocacy (4), and educational (4) organizations. Nine participants did not indicate the positions they held but included were 13 program supervisors, 6 executives, 5 department heads, 1 direct service provider, and 1 caseworker.

Data Collection Procedures

Procedures were developed in accordance with established focus group standards and principles (Berg & Lune, 2012; Scott, Garner, & Lewis-Elligan, 2013). The interview protocol was adapted from Nelson et al.’s (2009) focus group study of use of research evidence in education policy and practice to fit the particular focus of this study. The informed consent statement was derived from a model provided by the American Association for Public Opinion Research (2013).

Because of the nature of the participants’ work, assignment to groups was based on their time availability (random assignment was not possible). Participants received no direct payment but were provided with food and beverages during each session. Because participants already had a vested interest in the research topic, the researchers determined that the project is intrinsically rewarding enough to generate ample participation. Participation was strictly voluntary and confidential. Each participant was given a survey information sheet, and verbal informed consent was obtained at the outset of each focus group session.

To limit the burden placed on participants, the interviews were limited to 1½ hr. To make participants feel comfortable enough to share honest and candid views, and to further protect confidentiality, the discussions were not recorded. For each group, one researcher served as the sole moderator whereas another only took typed notes. This allowed each researcher to carefully concentrate on these tasks. Both researchers are skilled and experienced at performing their relative tasks. The qualitative data recorded therefore consist primarily of paraphrased comments accompanied by some direct quotes.

Participants were asked to candidly share their thoughts, perspectives, and concerns regarding two subjects: evidence-based recommendations to reduce DMC in the county and use of research evidence in general. This analysis is restricted to the latter. Prior to conducting groups the researchers developed themes, based on past studies, to be used to guide discussions. The moderator then posed these themes to all groups and used follow-up and probing questions as needed. The themes are listed below along with sample probing questions and statements.

Analytic Strategy

The strategy for data analysis was to first organize groups’ answers to the preceding theme questions, which involved a pre-established coding scheme. Simply put, the researchers wanted to clearly identify, understand, and present the information asked of the participants. However, the researchers also searched for themes that emerged in group discussions that were not easily categorized under the pre-established themes. This involved some amount of “open-coding.” Overall, the mission was to cover the data on use of research evidence comprehensively. In the next section, participants’ comments are presented “in their own words” as much as possible and key quotes are used.

Theme I: Awareness and Knowledge

Research evidence was generally seen as “things that work the best,” but some more specific references were made to program effectiveness: “a program has been studied and there is proof that it works,” “a proven program that produces certain outcomes with fidelity to the model,” and “a proven program and research has been done.” Discussions did not get much past these simple notions however, and groups did not offer specific types or examples of research evidence nor did they express a strong familiarity with the body of research evidence.

Some critical perceptions were offered, including that the term research evidence or evidence-based is or was used “loosely.” Some participants commented, “I would be concerned that people use the term” and “it’s just numbers—what we’ve been seeing for years is numbers” but acknowledged that an evidence base is needed to get funding for programs. Two participants went into depth, represented by the following excerpts from the note-taker’s notes. Indicating their lack of faith in research evidence, they identified two major problems—resistance to evidence-based change and generalizing results—and their solutions—create an effective strategy for “buy-in” and account for local needs:

Like the others I think there needs to be more education and support. What tends to happen is that we have so much ingrained tradition and the way things used to be and how things have been implemented. When people come with a new agenda into the community there is a lot of pushback. Evidence-based practice is a positive and useful approach but for me, like others, what is missing is something that helps promote the change to make sure that enough people understand why change is being made and why it is important. My position is implementing a process of change . . . Even though there is a lot of training and research, when you come against tradition and familiar ways of looking at things, there is a lot of tendency to go around and feel that you may be attacking the way someone has worked and thought for 20 years and that is very threatening to people. It is difficult for people to hear change without being threatened. It is important to include education in providing opportunity to show why what we are doing doesn’t always work; it is too easy to revert back if they don’t understand why the change is happening.

Evidence-base depends on what the evidence is. I see a lot of stuff when it comes to school safety and law enforcement. They say it is evidence-based, but when I look at it, it isn’t evidence-based for my community. Some try to take advantage of that. The evidence is what you pull from this community; it isn’t what they found in Philly or Chicago. It has to be unique to these families, when the community understands where decisions are coming from and that community is part of the process—talking to parents and finding out what they need.

Theme II: Extent and Nature

Participants tended to be familiar with policy and programs labeled “evidence-based” or “best practices” but rather unfamiliar with, or skeptical of, the research evidence behind them. For example, one stated “I don’t think DHS uses any directly, but we contract with agencies who do—evidence-based parenting models for example” and another stated “Allegedly the risk assessment is supposed to be evidence-based, but is skewed.” These concerns are also expressed in the following excerpts from notes:

We may start out with a model, but veer away from it because of lack of funding. [Program] got to the point to where we were just working with kids, teaching them what their parents should do, but didn’t work with the parents so we ended up doing more harm than good . . . We are required to find an evidence-based model, and if there isn’t one, then do something else. I can come up with a model, but if we don’t keep fidelity then it is wrong and doesn’t work. We need to look with more scrutiny and not put dollars in it.

We do best practice in policy, but don’t use evidence-based practice at [government agency]. That is [government agency] culture. I haven’t met a policy maker yet. Have they done what we do?

A good evidence-based model will tell you that it has worked in an urban setting, or rural setting. A rural model might not work in an urban setting. EB models tell you what type of setting/population it works with.

. . . When EBP was becoming an issue, we had someone write an EBP guide on what works with kids. IDDT was used first, probably wouldn’t do it that way again. This occurs at every level of organization.

However, as shown in the following excerpts, two participants did distinguish between “best” and “evidence-based” practices. Interestingly, one is more skeptical of “best practices” and the other is more skeptical of “evidence-based practices.”

In the last 4-5 years we have been moving away from “best practice” where each teacher did what they wanted. We now require EBP and want to know if it is replicated, if it is urban, if it is a “gold standard” study. We actually surpassed scores in other communities. Teachers don’t always like it and are uncomfortable when told they can’t do it the way they have always done it. We try to include all stakeholders.

A best practice example is life skills classes to kids. We did a pre- and post-test. Everyone said they got something out of the program. This is offered to kids in diversion. They say it is helping but it is not an “evidence-based practice” because it is not supported by a specific study or research. Lots of people are out there helping but because they aren’t strictly EBP, they don’t fall under that.

Also, when the facilitator asked the group to talk about how they ensure fidelity in program implementation, two participants responded to the issue. The following excerpts from notes exhibit an understanding of the meaning and importance of “evidence-based” and “best practice” as they pertain to programming.

We do a lot to insure fidelity—data tracking, showing fidelity at each step. To call it evidence-based you have to stay within those lines. This takes a lot of time and effort at all levels. There is accountability so that we don’t throw around the word if we aren’t really doing it. Time has to be put in to provide the fidelity and the reports. We do training and there is a funding issue to be able to keep the evidence-based program—cost and accountability. If we aren’t on the same page, it can affect the evidence base.

Programs implemented with fidelity to a model had an outside evaluator who holds implementation accountable. The independent evaluator makes sure that fidelity is followed. It is too easy to slip. We have phenomenal evidence-based practices, but they had money to run the randomized trials. There are lots of best practice programs but no money to prove it and get to that level: parents as teachers program, nurturing parent program, and DBP for kids in residential shelter. We do use those programs. The one that is truly evidence-based is the one with an independent evaluator.

Theme III: Improving Quality, Accessibility, and Practical Usefulness

Participants appeared to be unable to speak about the quality, accessibility, and utility of research evidence behind policies and practices. They even frequently went into discussion about general social work issues, making comments such as “I can’t fix the kids without educating the parents,” “parent attitudes need to change,” “community police officers would be so good,” “you are a product of your surroundings—they see their buddy acting a certain way and they follow,” “no value in working anymore—no one works,” and “teachers can’t teach anymore—have to worry about bureaucratic stuff.” However, as described in the next section, a distinct separate theme emerged at this point in the group interviews.

Emergent Theme: Improving EBP Implementation

Although they may not have discussed research evidence itself, participants had much to say about improving the implementation of EBPs. The following quotes and excerpts from notes reveal that the groups were perhaps more familiar and concerned with carrying out programs and developing policy and practices.

Involve more field supervisors in decisions regarding research and what gets implemented. I don’t even see that research unless I’m looking it up on my own.

and frontline staff . . . Trainings need to be appropriate—need to be restructured to be more beneficial.

I think the policy makers need to come and work in the field for a year before they are writing policy.

Instead of cultural competence training, we need to learn how to partner with families—find a way to partner with families better.

Stick with one plan and try to work it out. Ideas get implemented or tried, then after a year it fades away somewhere—consistency in following through with programs.

Off to good start—folks coming in and saying that the training sucks is a good problem to have . . . We’ll try some evidence-based models and some will fit and some won’t.

We need to bring everyone to the table. If we are busting our butts but someone else isn’t, then we will see the results of that.

Community buy-in—you see something new start up and you don’t have everyone there. Then it comes and it goes and that is worse than doing nothing at all. The community needs to know and there needs to be buy in.

From a system of care perspective: I work in a lot of collaboratives but we struggle to work it down. We get some at the table, but it is really hard to take it down—getting the staff that work for us to have the same collaborative spirit and belief.

Ongoing assistance with conflict management—there is no common lexicon or language. Collaboration and commitment: I don’t see peers at the meeting—need to stratify and include people at all levels, right down to the ground. We’ll learn to relate to one another in a positive way

Open connections: if it is a police officer, he shares with the schools, schools share with CMH, and so on when dealing with the same client. Otherwise it’s piecemeal—each agency doing its own EBP.

need to have the commitment and cultural competency and understanding where they are coming from, and what their needs are and somehow incorporating that into meeting their individual needs.

Besides pray, collaboration—not chasing after the dime, but make it more user-friendly so agencies are working together and not competing.

Education/training: training not available to everyone. Through education we learn about all the interventions.

We’re hiring the wrong folks probably and there’s not enough education going on.

Theme IV: Factors Inhibiting Use

As stated earlier, two of the groups were able to discuss and additional theme: factors that interfere with their use of research evidence in policy and practice. Again, groups discussed program implementation instead of research evidence itself. Most comments addressed problems with running EBPs, especially lack of funding, training, and education, and other resources. Participants commented,

Something I have dealt with—they could be really good for white middle class kids, but not for urban African American youth. There is money needed but also takes teachers and staffing.

Long-term commitment needed . . . minimum of 5-7 years. We sometimes lose focus due to turnover—a person running it leaves. The latest one is asking for a 3 year commitment from the person running the program . . . We work within our own circles for so long.

Expectations are higher but funding is less.

Sustainability is important. We can do it all but if people don’t believe in it and don’t think it will work, then it won’t. We need champions of the process at the times when people start throwing up their hands. So difficult—how do we impact community mindsets?

Data: none of this is anything if we can’t show the outcomes that show kids are doing better.

I have a list of people waiting to be in classes. We’re doing the best we can. We take to the limit of the classes . . . We are looking for something. There is a lack in there and when we look for a program, the first thing we look for is EB.

Manpower is huge—someone on the list and someone waiting to be trained. What am I going to do?

We have 20+ years working with the CAN Council. In our community there is fear that if I share with you, you might take my client list and get my grant. We’re still very fragmented—lots of EBP are pushed on them by the state. You might be using one program but it only has two checks. Someone else finds a program that has five checks and that is what a funding source wants you to use. What we end up doing is chasing the dime and that’s where you go, and that prohibits doing really great long term studies because we may have a really great program for a number of years but then have to change, to get services for families; that is what we do.

Summary

Directly, focus group participants exhibited limited awareness and knowledge of research evidence itself, though a few mentioned evaluation issues. This is an obvious problem in that for professionals to use research evidence, they must have awareness and knowledge of it. However, groups were very familiar with programming based on research evidence. The groups’ awareness and knowledge focused on the basic notion that some proof exists that an intervention works, though a few participants went beyond this to point out that research evidence has its flaws and limitations. Professionals closely involved with implementation (daily practice) may be concerned with whether or not an intervention is evidence-based, and works for them, more than issues regarding the production (e.g., methodological and theoretical) and dissemination (e.g., quality of publication outlet) of research evidence.

Similarly, when asked to address the extent and nature of their use of research evidence, groups tended to show that they are familiar with policy and programs labeled “evidence-based” or “best practices” but rather unfamiliar with, and sometimes skeptical of, the research evidence behind them. Discussions did not reveal that participants typically seek out research evidence on their own to make decisions. Perhaps field professionals’ use of research evidence is limited to what is externally required of them. If they do not perceive EBPs as legitimate or applicable, imposed use could foster unfavorable views of research evidence that inhibit their use of it.

Groups seemed to be unable to address how to improve the quality, accessibility, and utility of research evidence. Of course, they had limited awareness and knowledge of research evidence itself which would make it difficult to offer solutions to problems with its use. The groups did however introduce the emergent theme of improving EBP implementation. Here, participants identified major problems in policy and practice and offered solutions such as involving supervisors and frontline staff more in decision-making, using community-coordinated approaches, fostering community buy-in, and more accessible education and training. The two additional discussions of factors inhibiting use of research evidence also focused on implementation, addressing obstacles such as lack of funding, training and education, and other resources. Again, field professional may spend more time scrutinizing implementation of research-based interventions rather than the production of the research.

Implications

The participants’ focus on program implementation is consistent with the “two communities” argument that field professionals and academics live in two different “worlds” (Caplan, 1979). The world of the field professional revolves around implementation and daily practice. Working in the “trenches,” they may have little time to become familiarized with the body of research evidence itself, and be more concerned with the fact that a program or practice is known to be evidence-based and that they have to give all of their attention to carrying out their programs or giving direct services. More research is needed to discover whether lack of time and limited role duties are primary factors preventing field professionals from becoming directly involved with research activities and research evidence literature, and whether lack of faith in research evidence is also a major factor. Although some participants were critical, they rarely appeared to be hostile toward research evidence because it conflicted with their cultural values or self-interests.

The participants’ concerns with implementation also reflect that applying research evidence in everyday practice is no easy task. Research evidence does not automatically prescribe what to do under special circumstances requiring practitioner discretion. As other research shows, the implementation of evidence-based practices in service delivery is a complicated process in which a variety of expected and unexpected problems may arise, including those having to do with program fit to the needs of clients, staff and practitioner training, and organizational and external support (Aarons & Palinkas, 2007; Durlak & DuPre, 2008; Maher et al., 2009). The possibility that problems with implementing evidence-based practices have a feedback effect that illegitimates research evidence in the minds of practitioners should also be explored.

Although more research in this area is needed, it is clear that solutions, ways to decrease the gap between research and policy/practice, are needed. One strategy for increasing use of research evidence is through “imposed use.” In the United States, federal funding agencies have made grants contingent on demonstrating that programs are based on scientific evidence of success. In the words of Weiss and colleagues (2008), governmental funding agencies thus act as a “Fairy Godmother” to researchers who wish for their work to have an impact on policy. In essence, policymakers dependent on funding have to use research evidence, whether they want to or not. The problem with imposed use, however, is that it assumes that the evidence basis is strong. As mentioned earlier, this is often not the case as studies may be biased, methodologically unsound, and non-generalizable to other settings. Also, authorized “experts” (e.g., members of panels) who decide which programs are best supported by research evidence may have biases such as self-interest in certain programs. These issues make it difficult for policymakers and practitioners to genuinely believe in and use evaluation evidence; they may instead approach adherence to effectiveness principles as simply a bureaucratic task (Weiss et al., 2008). This concern was expressed by some participants in the current study, who complained that evidence-based models that did not apply to their particular needs or circumstances (e.g., type of community or population) were imposed on them, and that funding criteria used to decide the level of evidence basis of a program (scoring systems) are arbitrary or unfair.

Perhaps problems with imposed use can be fixed, making it a better strategy for ensuring the use of evidence in policymaking and programming. For example, funding agencies could do more to promote innovation, instead of “forcing” programmers to imitate existing models (which may not work under some circumstances). Even then, imposed use will not be a panacea for bridging the gap between research and policy/practice. Other approaches are needed.

One approach is to directly involve “third party” public policy and evaluation firms. These firms often play the liaison role by bringing academics and policymakers and practitioners “to the table,” sharing the realities of the field with universities and research centers and sharing new knowledge generated by researchers with agencies. One of the keys for increasing the likelihood of implementing evidence-based practice is the development of strong public/private partnerships that include all three partners (academics, policy and evaluation firms, and policymakers and practitioners). Colleges and universities are not necessarily charged with implementing knowledge, but they do focus on generating knowledge. Practitioners are not always in a position to access and utilize this knowledge. The sole purpose of public policy and research/evaluation firms is to produce information that helps decision makers make better decisions. Partnerships among these three types of organizations are seldom established and sustained, which contributes to the minimal implementation of evidence in the field.

Engaging in system reform efforts, which is what the implementation of evidence-based practice is mostly about, is very time-consuming and will not occur overnight. Practitioners who work in human services fields typically do not have the time, expertise, or resources to sustain the effort over time and academics often do not have federal grants and resources to stay engaged in long-term system reforms that involve training, program expansion, and the modifications of policy and procedure. Public policy and research/evaluation firms can be considered “catalytic organizations” that take the lead on strategic thinking, coaching, and implementing what is often discussed in conceptual conversations among practitioners and empirical researchers. This is not to say, however, that third parties which offer research and evaluation services do not have their own agendas. Many are advocacy groups with special interests. Thus, care must be taken when selecting a third party with which to collaborate. The potential for bias in other research contexts (e.g., academic and self-study) is not eliminated in public policy and research firms, which just underscores the importance of transparency and scrutiny of research and decision-making processes. Stakeholders concerned about these issues may want to involve “neutral” third-party firms, though advocacy groups that utilize sound methodology are also valuable partners.

Of course, engaging policy research firms, particularly over the long term, requires additional and substantial resources to do it right. With current limited funding that must be directed toward implementing programs, there may be little or no funds available to spend on third-party groups. In fact, some firms would not exist if community leaders and stakeholder groups did not collaborate to secure new grant funds, reallocate existing agency funds, and attend to other matters needed to stay involved. Policy research firm interventions work best when they are community-based collaborative efforts, as opposed to public or private agencies working within their own silos. Hence, the ability of policy research firms to bridge research and policy/practice is contingent on communities learning how to work together so that they can make decisions together, and generate and allocate resources together.

Conclusion

Results of the current study must be viewed within limitations stemming from the nature of the sample as well as the dual topic of discussion. In addition to use of research evidence, DMC was also discussed (though with an emphasis on evidence-based strategies to reduce DMC). Obviously, use of research evidence in other contexts must be studied to identify common problems faced by professionals. Also, the sample included a variety of types of professionals. On one hand, this gives the study a valuable element of diversity. On the other hand, it prevented insights into how specific professional communities perceive and use research evidence. Thus, this study provides but a beginning to understanding field professionals’ experiences in using research evidence, and more research is needed.

Like many qualitative studies, this one used a non-representative sample and is thus very limited in its generalizability. Needed are more focus group studies as well as larger surveys. More focus groups would provide a better basis for understanding common problems in using research evidence, not only by increasing the number of studies but by varying the types of samples and research evidence themes. For example, studies of local youth service agency administrators’ use of research evidence in providing services to juvenile offenders would be valuable. Larger theory-based surveys producing quantitative data are also needed to provide more generalizable results. In fact results of focus groups studies could contribute to the development of survey instruments.

Despite its limitations, the current study suggests a possible reason for the underutilization of research evidence in juvenile justice and youth services. Rather than having strong methodological, theoretical, or ideological objections, the nature of research evidence itself did not appear to be a prominent issue for group participants. An important question is, then, why is it not a prominent issue for them? The sample was rather unique in that it consisted of managers, supervisors, and practitioners with local authority, in contrast to lawmakers and public policymakers who are often examined in other studies. The focus groups’ emphasis on implementation suggests that practical limitations (performing daily job duties) prevent professionals from becoming more aware and knowledgeable of research evidence and its utility, though this possibility must be explored with future research.