Abstract
Objectives:
Explore how judges and attorneys define, acquire, interpret (i.e., determine the accuracy and relevancy), and use research in their decision-making in delinquency cases.
Methods:
We conducted semi-structured interviews with 30 judges, 15 prosecutors, and 13 defense attorneys. We used stratified purposeful sampling, stratifying participants by region of the U.S. and urbanicity.
Results:
Judges and attorneys have a sound understanding of how research can enhance their work. Typically, judges and attorneys acquire research from intermediaries. Beyond being a conduit for research, intermediaries play an important role in vetting the quality of research and identifying viable recommendations for practice. While practitioners are willing to use research, they feel that their ability to do so is limited by factors such as state policy, funding, and inaccessibility of research.
Conclusions:
While we caution generalization of the findings, this study contributes to the evidence-base on the use of research by documenting that judges and attorneys most often use research conceptually (i.e., research changes their perspective which then changes their behavior). Although respondents also reported using research-based tools to make specific decisions (instrumental use), many reported overriding research when they felt it conflicted with their judgment, suggesting that political use of research may be prevalent.
The juvenile justice system has undoubtedly been shaped by research. For example, research is cited in the U.S. Supreme Court decisions that abolished the juvenile death penalty and banned the use of life without the possibility of parole for youth (Anon 2005). Moreover, increasingly, juvenile justice practitioners are engaging in data-driven decision-making and using research-based tools, such as risk and needs assessments, to develop youths’ case plans. In fact, as of 2017, 42 states have either state statutes or probation agency policies that support the use of risk assessments (“Juvenile Justice Geography Policy and Practice (JJGPS)” 2017). Additionally, state policymakers have shown an increasing commitment to evidence-based practices, policies, and programs. For instance, as of 2014, 28 states have agencies with administrative regulations that require evidence-based practices and another 18 states have statutes that support a commitment to evidence-based programs and practices (JJGPS 2017). Beyond this, juvenile incarceration rates have declined by more than 50 percent in the last decade (Sickmund et al. 2017)—a phenomenon that experts have attributed, at least in part, to research demonstrating the ineffectiveness of incarceration (Love et al. 2016). This trend of research-based reforms is likely to continue, as the 2018 reauthorization of the Juvenile Justice and Delinquency Prevention Act (JJDPA) emphasizes the use of evidence-based programs and practices and requires states to develop state plans that are “supported by or take account of scientific knowledge regarding adolescent development and behavior” (Lewis 2018:9).
This picture of research-informed justice systems stands in stark contrast to that painted by the reporter Sarah Koenig, which follows a year in a Cuyahoga County courthouse: This is possibly the most profound and least examined question in the building. What works? The court doesn’t gather statistics on sentencing, and that’s true for most of the country, by the way. No data that says defendants in Cuyahoga County do better after six months of probation than after three years of probation, or, in terms of re-offending, four years in prison yields better results than seven years in prison. We just don’t know. (Koenig, 2018)
Research lends support to Ms. Koenig’s insights; for example, a 2014 study by the Council of State Governments Justice Center found that as of 2013, only about a quarter of states examined youth recidivism by lengths of stay in facilities (13 states) or by participation in different services (11 states). In fact, there were 11 states that did not track youth recidivism at all (Council of State Governments Justice Center 2014). Gaps between research and practice are further evidenced by the dispositions youth commonly receive, as research indicates that just five percent of eligible youth receive evidence-based services (Greenwood 2008).
Conditions that Shape the Use of Research in Practice and Policy
Scholars studying the use of research evidence have identified several typologies for how research is used by practitioners and policymakers (Nutley, Walter, and Davies 2007; Pelz 1978; Tseng 2013; Weiss 1977, 1979). These typologies, which are not mutually exclusive, include instrumental use, which refers to situations where research is sought out and is then directly applied to answering a question; conceptual use, which refers to situations where research influences or enlightens how individuals think about issues, problems, and solutions; tactical use, sometimes referred to as political use, which occurs when individuals use research to justify an existing stance or point of view; imposed use, which occurs when practitioners are forced to use research, such as when research use is tied to funding; and finally, process use which refers to knowledge that is generated from the process of collaborating with researchers. Although these typologies of research use have been integrated into studies focused on youth-serving systems, such as education, health, and child welfare (e.g., DuMont 2015; Tseng 2012, 2013), to date, we are unaware of any studies applying this framework to juvenile justice settings.
To date, there have only been a few studies that have examined the conditions that promote the use of research in the juvenile justice system (Johnson, Lebold, and Elam 2016; Love et al. 2016; Pesta et al. 2017). Across these studies, data have been collected from an array of juvenile justice stakeholders, including direct service providers, correctional administrators, state policymakers, technical assistance specialists, law enforcement, researchers, and experts. Common barriers to the use of research identified across these studies include: 1) difficulties practitioners and policymakers face in accessing and interpreting research; 2) unsupportive leadership; 3) budgetary restrictions; and 4) a mistrust of research. A common facilitator of research use is positive, sustained interactions between the research, policy, and practice communities, including both direct interactions (e.g., research practice partnerships) and indirect interactions through intermediaries, such as professional development organizations. While these studies represent a step forward in advancing our understanding of how practitioners and policymakers use research in their decision-making, they are limited by the fact that they do not share a conceptual framework for defining how research is used. The use of a unifying framework can advance our understanding of the use of research in the juvenile justice system because it facilitates our ability to compare findings across different studies within the juvenile justice system (e.g., predictors of conceptual research use may be different than the predictors of political use) as well as to assess the generalizability of findings to other youth-serving systems.
Further highlighting the importance of having common definitions of research use, emerging evidence suggests that the gap between research and practice varies depending on how the research is being used (Love et al. 2016). For example, a recent qualitative study, which included roundtable discussions and interviews held with juvenile justice practitioners, technical assistance providers, and national experts, found that research is much more likely to be integrated into state and agency policies and regulations than it is to be integrated into practitioners’ day-to-day interactions with youth (Love et al. 2016). Notably, Love and colleagues found that Many interviewees also noted that although research supports keeping youth in the community and matching them with appropriate services, there are no clear guidelines on how to do so…All interviewees were familiar with evidence-based practices, but several explained that although evidence-based practices are often promoted or required to access funding, practitioners do not always understand how to effectively operationalize them. Many stakeholders do not know what practices or programs to select, when to adopt them, or how to properly implement them with the youth they serve. (P. 4)
Similarly, participants in this study noted that probation officers often set goals for youth based on court-imposed mandates rather than what the research identifies as best practices. Further, this study found that although research-based tools, such as risk assessments, are commonly adopted across jurisdictions, they are often not implemented as intended due to common implementation challenges that result in frequent overrides (e.g., lack of buy-in, insufficient knowledge of the research behind the tool).
Love and colleagues’ findings are aligned with those from other studies’ examining systems’ implementation of evidence-based practices and programs. For example, results from a study examining the implementation of a risk and needs assessment found that over a third of respondents were either unsure or disagreed that the assessment results should be used in case planning (as cited in Sullivan 2019:198). In addition, a study on the implementation of 46 evidence-based programs in juvenile justice and child welfare systems found that many evidence-based programs remained “stuck on the shelf” because implementors of these programs lacked the resources, expertise, or incentive to implement the programs at scale (Neuhoff, Loomis, and Ahmed 2017). Taken together, the findings from these studies suggest that beyond promoting the overall use of research, more work is needed to ensure that data and research are being used accurately, appropriately, and fairly.
Implementation Science
Although significant findings from random assignment studies have fueled substantial optimism; too often, these programs fail to yield similar positive findings when replicated or “scaled up” (Supplee and Metz 2015). For example, of the 369 programs reviewed in the Office of Juvenile Justice and Delinquency Prevention’s (OJJDP) Model Programs Guide, a compendium of evidence-based programs, just 68 programs were rated as effective (OJJDP 2020). Moreover, finding few unqualified successes, Berman and Fox (2016) likened the rate of failure of research-based reform efforts in the justice system to that of new start-up businesses.
Research finds that programs that fail to implement an intervention as intended are less likely to be successful (Carroll et al. 2007). Indeed, one meta-analysis of over 500 evaluations of programs for children and adolescents found that interventions that were correctly implemented had two to three times greater impacts than programs that were not (Durlak and DuPre 2008).
Researchers in the field of implementation science—a field focused on identifying factors that promote the successful installation and maintenance of a new evidence-based program, policy, or practice—have identified multiple “implementation drivers” that increase the likelihood of successful adoption of a new policy, program, or practice. These implementation drivers are often grouped into three overarching categories: competency drivers, organizational drivers, and leadership drivers (Fixsen et al. 2013). Competency drivers are focused on the stakeholders who directly implement the new practice or policy and include factors such as hiring qualified frontline staff; providing frontline staff with ongoing training, support, supervision, and coaching; and ensuring frontline staff have sufficient buy-in. Organizational drivers are focused on the infrastructure, systems, procedures, and climate within organizations needed to successfully implement a new practice or policy and include factors such as user-friendly data systems and administrative and managerial processes. Finally, leadership drivers are related to the strategies and skills that organizational leadership need to successfully install and maintain the new practice or policy (Derrick-Mills and Winkler 2016; Fixsen et al. 2013; Vincent et al. 2018). These implementation drivers are multi-layered in the sense that they attend to both factors within an organization (the “inner context;” e.g. leadership, organizational climate, and frontline staff competencies and buy-in) and external to an organization (the “outer context;” e.g. state policy and funding, formal collaborative agreements across agencies; (Aarons, Hurlburt, and Horwitz 2011; Sullivan 2019)). In addition to being multi-layered, implementation drivers are also transactional and interdependent. For example, an agency’s climate, an organizational driver, can influence frontline staff’s readiness to learn new skills and buy-in for changing the status quo (competency drivers).
Recently, researchers have begun to use implementation science as a framework for studying the adoption of research-based reforms in juvenile justice systems (Derrick-Mills et al. 2019; Derrick-Mills and Winkler 2016; Sullivan 2019). For example, OJJDP released a brief based on two evaluations, that used an implementation science lens to identify drivers of successful implementation of risk assessments (Vincent et al. 2018). The competency drivers identified include timely staff training and staff perceptions of the validity and the usefulness of the instruments. Key organizational drivers were ongoing support, supervision, and training of frontline staff; standardizing procedures across all stakeholder groups involved; and having a strong, user-friendly data system in place. Leadership drivers that were important include leaders’ engagement of both frontline staff and key stakeholders outside of probation agencies, such as juvenile court judges, in strategic planning efforts.
Barriers to the successful implementation of evidence-based programs, practices, and policies can be exacerbated in the context of juvenile justice systems. One reason for this is because the successful implementation of research-based reforms in the justice system often requires collaboration across multiple autonomous agencies, such as juvenile court judges, prosecutors; defense attorneys; probation offices; police departments; and direct service providers—each of which may have distinct organizational cultures, attitudes toward existing processes, constituencies they report to, and mission focus (Sullivan 2019). For example, a state legislature may pass a law requiring automatic diversion for certain types of offenses, but the prosecutor’s office could have discretion in how to apply that law or in what charges they file. Similarly, a jurisdiction may have a policy to not place youth who have low risk scores in pre-trial detention but may be left with no choice if it is unsafe for a youth to go home and there are no alternative out-of-home placements available.
A second challenge to the implementation of research-based reforms in justice systems lies with the wide variability across jurisdictions. For example, juvenile justice systems often depend upon networked partners and treatment providers to deliver services (Howell et al. 2013; Sullivan 2019), each of which may vary in their fidelity of implementation of an evidence-based program. Similarly, a juvenile court in an urban area may have the resources to have a single judge oversee all juvenile delinquency cases while a more rural juvenile court may have a single judge who oversees multiple types of cases (e.g., delinquency, dependency, domestic relations, civil matters, and adult misdemeanors).
A third challenge concerns program fidelity more specifically, as evidence-based programs are not often validated across diverse populations of youth, and practitioners are not always equipped to determine which adaptations are appropriate. Relatedly, program staffing and resources in juvenile justice settings rarely align with evidence-based programs’ implementation guidelines, making it difficult to implement them with fidelity. Finally, many programs lack funding for the ongoing training of staff and the development and maintenance of data systems that support implementation (Butts and Roman 2011; Lipsey et al. 2010).
Current Study
Across the past decade numerous research-based reforms have been implemented in juvenile justice systems throughout the country. However, research has shown that 1) states’ progress in implementing evidence-based reforms has been uneven (National Academies of Science, Engineering, and Medicine 2019); and 2) among jurisdictions that have implemented reforms, quality of implementation varies—an issue that may stem from the difficulties practitioners face in integrating research into their day-to-day practice (Hussemann and Liberman 2017; Love et al. 2016; Vincent et al. 2016). Accordingly, more information is needed on how to effectively install and maintain evidence-based practices in juvenile justice settings.
Implementation science offers a useful framework for identifying factors—both within and external to an organization—that increase the likelihood an evidence-based reform will be implemented successfully. However, to successfully identify the necessary implementation drivers (i.e., competency drivers, organizational drivers, and leadership drivers), researchers must have a nuanced understanding of the day-to-day operations of the juvenile justice system. Relatedly, findings from studies examining the conditions that promote research use in the justice system have emphasized the importance of relationships between the research community and the practice and policy communities (Johnson, Lebold, and Elam 2016; Love et al. 2016; Pesta et al. 2017)—a finding that aligns with those from studies on the use of research evidence from other fields (Chen 2010; DuMont 2015; Jensen, Hoagwood, and Trickett 1999).
Unfortunately, to date, researchers have made only limited efforts to engage key stakeholders in juvenile justice systems (Laub 2016). In particular, few, if any, studies have focused on research use in the juvenile court system. The neglect to do so is problematic for several reasons. First, juvenile court judges, prosecutors, and defense attorneys make several key decisions throughout juvenile justice proceedings (Sarri et al. 2001). For example, prosecutors are often cited as the practitioners with the most discretion in their decision-making, and have the authority to affect a variety of case processing decisions, including the decision to formally charge a juvenile, sentencing options, and, in many jurisdictions, the decision to transfer or waive a youth to the adult criminal justice system (Sarri et al. 2001). Judges also have considerable discretion. For instance, although a court intake officer can make recommendations for pre-trail detention, a judicial officer can override that decision. Similarly, while probation officers are often charged with case management, after adjudication, the judge has sole decision-making authority regarding disposition, though, notably, probation officers, defense attorneys and prosecutors may submit disposition recommendations for the judge’s review (Leiber and Peck 2015; Sanborn 1996; Sarri et al. 2001). Given the critical role juvenile court judges and attorneys play, it is unsurprising that research has shown that attaining the buy-in of juvenile court stakeholders is key for successfully implementing evidence-based reforms (Love et al. 2016; Vincent et al. 2018).
In addition to their powerful role in decision-making during juvenile justice proceedings, given that most states’ juvenile probation departments are either overseen by the judicial branch or are a shared function of both the judicial and executive branches (National Center for State Courts 2012), judicial leadership can also influence the practices of juvenile probation agencies. Finally, understanding the perspectives of juvenile court judges and attorneys is important because these individuals must necessarily consider factors other than social science research (e.g., legal precedent and evidence) when making decisions—a point recently illustrated in an opinion editorial written by a juvenile court judge: It is often an impossible calculation, equally rooted in science, facts, law and prediction. Judges cannot always get it right. Judges consistently feel the tension between the comfort of stronger sentences and the optimism of alternatives to longer sentences. Neither is always the right answer (Rogoff 2019).
Thus, the purpose of this study was to better understand how juvenile court judges and attorneys use research in their decision-making. There were four key research questions driving this study: How do juvenile court judges and attorneys define, acquire, interpret, and use research? What are the conditions that facilitate or inhibit the use of research? Are there critical junctures in juvenile justice proceedings where research is most likely to be used? What are best practices in conducting, translating, and disseminating research?
To answer these research questions, we conducted semi-structured interviews with judges, prosecutors, and defense attorneys working in the juvenile justice system.
The current study makes three contributions to the field. First, the study provides initial insight into how attorneys and judges—key stakeholders who have yet to be included in studies on research use in juvenile justice systems—are using research in their decision-making. Given the consistent findings that failure to obtain buy-in from juvenile courts can serve as a roadblock to implementing research-based reforms (e.g., Love et al. 2016; Vincent et al. 2018), learning from this stakeholder group may be particularly critical to the success of any research-based reform effort. Second, this study is based upon a framework of research use that has been used to examine the use of research in other fields, such as education and health (DuMont 2015; Tseng 2012, 2013), which facilitates our ability to examine how well our findings align with those of similar studies conducted in other youth-serving systems. Finally, this study synthesizes judges’ and attorneys’ suggestions for how researchers can produce more meaningful and useful information for this critical stakeholder group, which can aid future efforts to bridge research and practice in the juvenile justice system.
Methods
Sample
We conducted interviews with 30 judges and 28 attorneys (15 juvenile prosecutors and 13 juvenile defense attorneys). Given that juvenile justice systems vary between (JJGPS 2017; National Research Council 2013) and within states—particularly across rural and urban communities (Blackmon, Cain, and Livermore 2015; Feld 1991; Hawkins and Kempf-Leonard 2010)—we used stratified purposeful sampling (Ritchie et al. 2014), stratifying participants by region of the U.S. (Northeast, Southwest, Midwest, West, and Southeast) and urbanicity (urban, suburban, and rural). With the use of stratified purposeful sampling, our sampling process was designed to result in a sample of practitioners whose experiences and jurisdictions reflected the broader experiences of practitioners nationwide. We stopped recruitment, however, after reaching thematic saturation, the point in which additional interviews yield redundant data, and, thus, no new thematic information was gained (Charmaz 2006). Our sample size is consistent with existing research that finds that saturation, is often reached after five to nine interviews with members of the same group (Patton 2002).
As can be seen in Table 1, respondents were approximately evenly split by geographic region. Nearly half of participants reported that their jurisdictions predominately served youth from urban communities, just under a third were from jurisdictions that served youth from rural communities; and approximately one-fourth were from jurisdictions that mostly served suburban youth. Most judges reported that they oversaw both delinquency cases and dependency (child welfare) cases (43 percent, n = 13), followed by judges who oversaw all family court cases (e.g., dependency, delinquency, and divorce; 27 percent, n = 8), and judges who had a more general jurisdiction, overseeing all family, civil, and adult misdemeanor cases (27 percent, n = 8). Only one judge exclusively oversaw delinquency cases. In contrast, nearly half of attorneys (46 percent, n = 12) handled juvenile delinquency cases exclusively, with the remaining handling some mix of juvenile delinquency and other types of cases. Finally, about half of participants were female and most were seasoned professionals, with half having a tenure of 11 or more years.
Summary of Respondent Characteristics.
Notes. Urbanicity is based off of respondents’ perception of the population of youth seen in their court; Region is defined as follows, Midwest: Illinois, Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, North Dakota, Ohio, South Dakota, Wisconsin; Northeast: Connecticut, Delaware, District of Columbia Maine, Maryland, Massachusetts, New Hampshire, New Jersey, New York, Pennsylvania, Rhode Island, Vermont; Southeast: Alabama, Arkansas, Florida, Georgia, Kentucky, Louisiana, Mississippi, North Carolina, South Carolina, Tennessee, Virginia, West Virginia; Southwest: Arizona, New Mexico Oklahoma, Texas; West: Alaska, California, Colorado, Hawaii, Idaho, Montana, Oregon, Utah, Washington, Wyoming.
We recruited participants using referrals as well as through identification on state or county websites. Of the 30 judges who participated in the study, 18 were identified through state or county websites, eight were referred by judges who were ineligible to participate in the study because they were not currently overseeing delinquency cases, and four were referred through our study’s expert advisory panel. To identify judges through websites, we selected counties listed in the National Center for Health Statistics’ 2013 NCHS Urban-Rural Classification Scheme for Counties (Ingram and Franco 2014). After we selected a county, trained research assistants identified the juvenile court in that jurisdiction and then selected a juvenile court judge to recruit. We reached out to a total of 265 judges. A total of 94 (35.5 percent) judges did not respond to our requests after three follow-up requests and 76 (28.7 percent) were dropped from follow-up because we filled our respondent quota for that region and urbanicity. Of the 95 judges who responded, 48 (50.5 percent) declined to participate, 30 (31.6 percent) were interviewed, and 17 (17.9 percent) were not eligible to participate because they were not currently overseeing delinquency cases.
We used a similar process to recruit attorneys. Twelve attorneys were referred by judges who participated in the study and 11 attorneys were referred by other attorneys. We identified five attorneys through state or county websites. We reached out to a total of 208 attorneys, attorneys’ offices, or contacts at professional organizations (such as the Center for Children and Youth Justice and states’ defense and prosecuting attorney associations) for referrals. A total of 113 (54.3 percent) did not respond to our requests after three follow-up requests and 30 (14.4 percent) were dropped from follow-up because we filled our respondent quota for that region and urbanicity. Of the 65 attorneys who did respond, 11 (16.9 percent) declined to participate, 28 (43.1 percent) were interviewed, and 26 (40.0 percent) were not eligible to participate because they were not currently working on delinquency cases.
In total, we contacted approximately 370 juvenile justice practitioners, of whom, 44 percent responded. Of those who responded, 37 percent declined to participate, and 27 percent were not eligible. Collectively, our study response rate was approximately 18 percent. Given our low response rate, the generalizability of our findings is limited. Nevertheless, we hope that the recommendations and insights from this study will provide a useful contribution to efforts to better understand how research is used by judges and attorneys in the juvenile justice system.
Procedures
Prior to data collection, an Institutional Review Board approved all study protocols and procedures. Four trained interviewers conducted semi-structured interviews by phone. Interviews with judges were conducted between November 2016 and June 2017. Interviews with attorneys were conducted between May 2017 and February 2018. Interviewers asked participants to verbally consent to participate and to be recorded. A trained research assistant took notes during interviews that were unable to be recorded (n = 3). The median interview length was 55 minutes.
Instruments
We developed semi-structured interview protocols based on a framework for understanding the use of research evidence outlined in a Social Policy Report, “The Uses of Research in Policy and Practice” (Tseng 2012). We followed this framework, in an effort to examine the extent to which our findings generalized across other disciplines (e.g., DuMont 2015). We first developed the judicial interview protocol and then adapted it to create the attorney protocol. Two expert consultants, a researcher and a juvenile court judge, and the study’s expert advisory panel—which was comprised of attorneys, researchers, technical assistance specialists, and advocates working in the juvenile justice system—reviewed each protocol. In addition, the judge and attorney protocols were piloted with a juvenile court judge and a juvenile prosecutor, respectively.
In the interviews, participants were asked how they: 1) define or distinguish research from other forms of information; 2) acquire research; 3) interpret research (i.e., determine the research is of high quality and relevant to their work); and 4) use research in their decision-making—both generally and at each juncture of a delinquency case (referral, intake, waiver/transfer to the adult system, detention, adjudication, and disposition). The protocol also included questions about conditions that impede or facilitate research use (e.g., organizational climate, funding, leadership, and the severity of the offense the youth committed). Key constructs assessed by the protocols are summarized in the Analysis section, below. A secondary data analysis guide (which details constructs assessed by the protocols), copies of the protocols, and de-identified interview transcripts are archived at the National Archive of Criminal Justice Data.
Analysis
The study team uploaded transcriptions (and notes from the three interviews we were unable to record) into Nvivo 10 (SR International Pty Ltd 2014), a software package designed to facilitate the analysis of qualitative data. Our team used a combination of codes, including both codes established a priori and emergent codes that arose from analyzing the data. From a deductive standpoint, we developed a priori codes aligned with our research questions based on the typologies of research use (i.e., instrumental, conceptual, tactical or political, imposed, and process use) established by Carol Weiss (Weiss 1979), which have been used in studies on research use across multiple disciplines (e.g., Tseng 2012, 2013).
We also used an inductive approach to gain information about key concepts of interest such as defining, interpreting, and acquiring research as well as the different types of research use. The study team held debriefing sessions concurrent to data collection to identify preliminary themes and develop the initial emergent coding scheme. We then updated and refined this coding scheme using an iterative approach of open, axial, and selective coding (Krueger 2014). This inductive coding technique, drawing from grounded theory, allows codes or themes that are unanticipated—but critical to our analysis—to emerge. For example, we learned in the first several interviews that in-person trainings and continuing education were a primary way that practitioners acquired research. In subsequent interviews, we began asking participants what kinds of research-based information were included in these trainings, and whether their confidence in this information varied depending on who provided the training. Coders identified areas of the interview that related to certain themes; then coded relationships between different themes; and finally, coded pieces of the interview that helped to connect all of the themes into a single narrative or story (Creswell 2012; Trochim 2006).
Coders included entire sentences and/or paragraphs as appropriate to provide context to any one code and met regularly to review the interrater reliability. The coding team met regularly to update these definitions based on themes that were identified in the interviews and to compare classifications of typologies of research use. For a priori and inductive codes, the study team met to discuss inconsistencies and clarify code and node definitions for any codes with a Cohen’s kappa coefficient below 0.61 and nodes with coding density that measured below 0.70, as this indicates problems with coder agreement (Krippendorff 2018; Landis and Koch 1977). Coders recoded interview transcripts after addressing these inconsistencies. Entire transcripts were coded for respondent characteristics (displayed in Table 1). This allowed us to examine differences by respondent characteristics.
The combination of deductive and inductive coding techniques allowed the study team to holistically explore how practitioners navigate and use social science research, and to learn how the contexts in which practitioners operate influence which research use typology or combination of typologies they use. For example, from an inductive standpoint, we learned that practitioners’ use of research changes depending on a variety of factors such as the stakes of the decision and how reputable practitioners deem the research information to be. This gave us more nuanced information about practitioners’ research use in each of the a priori typologies we explored.
Findings
In the following section, we describe the themes that were identified through our analysis across our four research questions. We end with synthesizing respondents’ recommendations for researchers aiming to produce information that is useful in applied settings. Given that common themes were identified across the three groups of respondents (i.e., judges, prosecutors, and juvenile defenders), results are presented for all respondent groups in aggregate, with differences between groups highlighted when applicable. Our team coded all incidences when respondents discussed core components of each research question, which were defined a priori (i.e. defining, acquiring, assessing, and using research, as well as typologies of research use). The team regularly met to identify and define themes within each component.
How Do Juvenile Justice Judges and Attorneys Define, Acquire, Interpret, and Use Research?
Defining research
At the outset of the interview, we asked participants to define social science research in their own words. Overall, respondents provided thorough definitions that aligned well with the study team’s own definition. Specifically, most judges and attorneys (n = 45) defined research as a systematic, investigative process that tests hypotheses and produces information from which conclusions can be drawn. For example, one attorney defined research as the: …scientific method. It is either studying previous research or previous experiments to try to come to a new understanding of material, or it could also be [the] conducting of the experiment itself, and then figuring out if the conclusions are consistent with what maybe the person has hypothesized the research would say…
A handful of respondents provided less thorough definitions of research but explained that they were familiar with what constituted social science research more broadly. For example, one judge noted lightheartedly, “It’s like pornography, I know it when I see it.” A few attorneys (n = 4) initially provided definitions of legal research rather than social science research. In these cases, the interviewers read respondents a scripted definition of social science research to ensure that the participants were thinking about social science research for the duration of the interview.
Acquiring research
We also asked respondents where they obtained social science research and whether they felt it was easy or difficult to obtain. Generally, respondents reported that it was easy to access research-based information and that they frequently received new research-based information. Although respondents reported that acquiring research information was easy, they found it much more difficult to go back and find a piece of research when they needed to recall details from the source later. Thus, acquiring research was a two-part process for practitioners: first, they physically got access or exposure, and second, they knew how to re-access that information when they were ready to consume it.
One of the primary ways that respondents (judges: n = 24, attorneys: n = 13) received research was through the trainings and conferences they attended to maintain their continuing legal education credits. For example, 25 respondents (judges: n = 16; attorneys n = 9) reported attending trainings that included presentations of research on at least an annual basis. Although many respondents (judges: n = 11; attorneys: n = 5) attended national trainings and conferences, such as those put on by the National Council of Juvenile and Family Court Judges (NCJFCJ), state and local trainings and conferences were even more popular, with 18 participants reporting they attend these events on a regular basis. These local events were preferred over national conferences due to time and resource constraints. For example, one judge stated: On one hand you want to get the training; on the other hand, you have to sit in court to do the job…I do some voluntary stuff here and there, but if you’re gonna get the job done, you have to sit in court. So probably what [my jurisdiction is] doing is about right. You know, two multi-day conferences a year, and then maybe [I] try to take advantage of an opportunity or two a year to do additional [training].
Respondents reported that these annual trainings and conferences incorporated research on a variety of topics, including adolescent brain development (n = 45), evidence-based practices or programs (n = 40), and trauma or trauma-informed care (n = 30). Notably, only a few respondents (judges: n = 5, attorneys: n = 1) reported attending an orientation training that included social science research when they began their job.
Most respondents (n = 48) noted that they preferred to receive new research during in-person trainings so that they can ask questions and focus on the information being presented to them during a set-aside time; still, some participants (n = 11) mentioned that written materials can also be useful. Respondents stated that they most often acquire written materials through email from colleagues or listservs. These email listservs include both listservs internal to their place of work as well as listservs of external organizations—most typically intermediary organizations (e.g., professional development organizations, government agencies, and applied research centers or think tanks). Although respondents reported that they frequently received research via email, due to the volume of email they receive, they rarely had time to filter through and digest it. Moreover, participants explained that the volume of email they receive can also make it difficult to return to the information once time had passed. When respondents had time to browse through research in their email, most respondents noted they would only click the link to a research article when the subject line or first few sentences of the email indicated there was something directly relevant to the population they work with or to a problem they had been struggling with. Respondents noted a variety of other factors that can influence whether they opened and read an article, including whether the email came from a source they trusted. An active search for relevant research could be triggered on occasions when judges and attorneys needed information about a specific topic (n = 35). For example, one prosecutor noted: I have looked up a few articles about the harm of cyber porn or revenge porn. I’ve tried to include that in my sentencing recommendations, because I do think that’s something that judges don’t quite understand or appreciate the harm of. That’s the one area where I have actually sought out some information, and some of our detectives actually do training for that type of crime as well.
Although respondents generally reported that research was widely available, they noted obstacles to finding research information when they needed it. For example, one defense attorney stated: I think it’s difficult [to find research] and much of it is just because of the time factors of not having that much time to look…I guess, I, probably historically, have a mixed rate of success in being able to find things that are really helpful and possibly giving up on other matters.
In line with this sentiment, a few participants (n = 6) recommended that researchers develop an online clearinghouse that summarizes juvenile justice related research.
Assessing the relevance and quality of research
We also asked participants to describe how they determined whether research was valid and/or applicable to their own work. We identified two processes that practitioners used when making this assessment: determining individual-level relevance and assessing the rigor and trustworthiness of the information.
As mentioned above, often, participants struggled to filter through the plethora of research available (online or through email) to find information relevant to their work. To determine the relevancy of research, respondents noted that they considered the topic of the article, whether the study sample was aligned with the population they serve, and whether the study findings have practical application to their work (n = 16). One defense attorney said: If I have a 12-year-old client and the research was completed on 16 to 17-year-olds, I don’t think that’s gonna be high quality for this particular case. Maybe if my client was 16 or 17…Just factors, as part of the research or part of the study, if that makes sense. If it’s a spectrum of ages, then that could be helpful, but if the research was done on someone that’s three years older than my juvenile, then I don’t think that’s gonna be helpful for my juvenile because a 13-year-old definitely has a different mindset than a 16-year-old.
After identifying a relevant research article or summary, respondents reported that they had to evaluate whether the information was trustworthy. A commonly reported problem among respondents was that it is hard to trust research findings because often, results from different studies can conflict with one another. For example, one prosecutor said, “You can find research on anything you want to find research on. You can find statistics that support or go against what you say all the time.” Judges and attorneys reported that this was particularly true for research findings summarized on the internet, and because of this, it was particularly meaningful when all studies came to the same conclusion (e.g., as was the case with the plethora of articles around adolescent brain development).
Participants reported using a variety of metrics to determine the quality of the research they come across. Most commonly, respondents reported assessing quality by the source that shared the research and/or the organizations that produced or funded the research (n = 15). Many respondents stated that they were more likely to trust information that came from sources they perceived to be impartial, specifically noting that they would not trust research from anyone with an “agenda.” For example, one defense attorney shared: Well, if I hear it on NPR, I’m pretty much gonna accept it as being true. If I read a journal, which I don’t do, [but] if I read an established journal, especially if it comes from an academic area, then I’d probably accept that as legitimate. I wouldn’t just accept anything on the internet as legitimate. Just because it’s on the internet doesn’t mean it’s true.
Sources that practitioners trusted as “impartial” varied. For example, one prosecutor noted “[I]f research came from the Innocence Project…Or came from a group that has an agenda, then I don’t think it carries as much weight as somebody that’s non-biased or…Isn’t taking a position on a particular issue.” Respondents also admitted that it was not always clear what the source behind research was and whether it was biased. As one defense attorney noted “Are there studies funded by those that privately contract with the state for beds? I don’t know. That wouldn’t surprise me, but I just don’t know if we get to that level of ever knowing.”
A small subset of respondents reported examining characteristics of the study (e.g., whether the sample population was similar to the population the respondent worked with; the study design; n = 13), whether the research was peer-reviewed (n = 4), and whether the research was accepted by their colleagues (n = 4). For example, one prosecutor stated: Given that I’m a skeptic on pretty much any research, not just this topic, the first thing I want to know is how was your research designed? What’s your sample size, what [does] your control group look like? Is your control group really an adequate control group for what you’re trying to measure?
Many respondents reported, however, that it was hard to determine whether research was high quality or rigorous. As one defense attorney noted “In terms of whether the methodology behind it is something that could be criticized, or whether there’s flaws. I don’t think many of us would…Really have the knowledge base to be able to identify that.”
Finally, respondents noted that they needed to feel as though the researchers understood the nuance inherent in delinquency cases. As one prosecutor noted: For example, if I’m reading research and it’s saying that no child should be detained ever, for example, I’m not going to receive that because I know that there are kids on my dockets that I’m terrified for them to be let off detention. Because I don’t think they’re going to survive longer than a couple of weeks, because they’ve placed themselves at such great risk…I need to see that they fully understand how serious many of these issues are, how terrifying they are.
Along these lines, many respondents noted that researchers would benefit from spending time in juvenile courts or shadowing judges and attorneys.
Using research
The team met to look for patterns of research use and then compared these patterns to the typologies of research use that were defined ahead of data collection. We identified evidence of conceptual, instrumental, and political research use.
Judges and attorneys described several ways in which research aided in their decision-making and could improve practice in the juvenile justice system. For example, one judge said: We are here to try to come up with decisions to do the best by that person and to me the research is really the illuminating piece of that decision. It is the backdrop of the whole decision-making process for me.
Others, however, emphasized that research is only one piece of the puzzle in their decision-making. In these cases, respondents tended to take in research findings and have them in the back of their mind as they navigated their work rather than directly applying a research finding to a specific case. This type of research use, which we categorized as conceptual use, was reported by 44 respondents and was more common among judges than attorneys (n = 28 judges; n = 8 prosecutors; n = 8 defense attorneys). For example, in response to a question about how he uses research, one judge explained “to use computer words, it gets assimilated into my database.” Echoing this sentiment, a defense attorney explained, “typically the research is used in a more general sense than an actual study being incorporated into a motion.” Many respondents shared examples of how research changed their thinking on a given issue over time. For example, one judge said that as a result of attending a seminar put on through the Annie E. Casey Foundation’s Juvenile Detention Alternatives Initiative, he relied less on detention: “I went from routinely having somewhere between eight and 20 kids in jail, to—I think this morning I had two.”
Respondents also discussed using research-based information to make specific decisions about a case (n = 45), which we categorized as instrumental use. This appeared in the use of expert testimony, evidence-based assessments, and information about evidence-based programs in their jurisdictions. Although the use of evidence-based tools, such as risk assessments was commonly reported, respondents questioned the extent to which these tools were validated, based on quality data, and consistently applied. Respondents also varied in their perceptions of how much weight should be given to those assessments. For example, one judge said: I think rather than me issue something on my arbitrary gut sense…It ought to all be about rehabilitation and what will result in the best outcome for this juvenile. To me the research is the fundamental in making this decision.
In contrast, another judge stated, “I use [risk assessments] as a tool to enhance my judgment, but in the end, I think a judge has to go with their experience…I can’t let someone else’s research overrule my judgment.”
About half of respondents mentioned that research is, albeit infrequently, brought up during cases—most commonly through expert testimony (n = 31). For example, one defense attorney noted: It’s not super common…But I just got off the phone earlier today, talking to a therapist who’s gonna testify in support of some kids being able to be returned home at our next hearing. And he’ll, I’m sure, cite research about best practices in returning kids after they’ve been removed due to substance abuse.
Use of research during a case varied based on respondent type: over half of judges (n = 19) and defense attorneys (n = 7) reported that research was cited or brought up during a case, compared to about a third of prosecutors (n = 5). Further, although nearly all judges reported using research, attorneys reported that it was rare for judges to bring up research during a case. Similarly, although most attorneys reported using research themselves, they reported that it was rare for their opposing counsel to use research.
Respondents provided varying explanations for the disparate use of research among different practitioner types. One defense attorney noted that social science research tends to support the defense’s position more often than it supports the prosecution’s: We [defense attorneys] can get big advantages out of using [research] because the state doesn’t use it very well and doesn’t think about it as much. And a lot of the research supports our kids’ positions better than it supports the state’s…A lot of things that the state operates off of, like conventional wisdom about things…“Well, you know, this kid’s really messing up in placement. And we shouldn’t reward him by giving him a less restrictive placement when he’s messing up in the more restricted placement.” That’s conventional wisdom, right? And then we operate from a research base. We say, “You know, if a kid’s been in placement for more than six months, the number of gains they’re gonna make probably are pretty limited if we keep them there longer.” That’s what the research tells us. So the research really supports our position and undermines theirs. And I feel like that’s often the case…
Contradicting the notion that only defense attorneys incorporate social science research, several prosecutors reported that they incorporate research into their decision-making to make the decision that is best for both the youth and the community. As one prosecutor noted: Maybe I might make a decision based on that research and the facts of this youth who’s in front of me to maybe think about a diversion program. Even before I file a case against this young person, I might be thinking about what program is available to me based on some of the research that has informed me, and the facts that this youth is presenting that I might be able to safely divert this young person from court involvement. It’s hard for me to even tell you, but it informs every part of our work because it does explain some of the behavior. If you have the behavior explained, then maybe you could think about: What is it that I can put into place, or ask the court to put into place to alleviate the risk that this youth is presenting? It does inform almost every decision point.
There were a few instances where respondents felt that research was used politically; accusations of political use were brought up by both defense attorneys and prosecutors. For example, in response to a question regarding how research is typically used in juvenile justice cases, one prosecutor noted “[I]t’s usually used by the defense lawyer. It’s not usually used by me. They use it as a reason to justify the accused’s behavior.” On the other hand, one defense attorney noted “[District attorneys] have got the law and the goose-stepping police state on their side. So, they don’t need empirical research and stuff to show that all this has had an awful effect ‘cause they don’t care about the effect. They want their statistic of an arrest.”
What are the Conditions that Facilitate or Inhibit the Use of Research?
Many respondents spoke about barriers and facilitators to their use of research. Most commonly, respondents referenced time and resource constraints (n = 24). They noted that, heavy caseloads often prevented them from keeping up to date on the latest research and explained that in some jurisdictions—often in rural jurisdictions—lack of available services or other resources can limit the extent to which decisions can be made that are most aligned with research findings. For example, one judge said, “[I]f there isn’t an open bed, I can have all the research in the world but not be able to use it.” Similarly, judges and attorneys reported simply not having access to research as a barrier (n = 15). Issues with access included both journal paywalls as well as the availability of studies on what interventions work best, for whom, and when. For example, one prosecutor stated: Using research, in general, can be frustrating when you don’t have a subscription to the journal articles where it’s written up. You’re always depending on that secondhand take on it and that can be a little frustrating. But at the same time, I don’t have enough time to read all of the articles out there and do the rest of my job, so I need to have research that is broken down into easy to use, bite-sized bits.
Respondents also noted that state policies and court regulations influenced their ability to use research (n = 15). For example, some prosecutors reported that they were precluded from incorporating non-legal factors into their decision-making: The way I do my job is largely dictated by the legislature of the state. All the research surrounding adolescent brain development is interesting and I think it can and should guide lawmakers but until they change the statutes, I am bound by those statutes in doing my job. Information related to research, recidivism, everything else, while it’s interesting, it has very little impact on the way I do my job, because I don’t get to change the rules based on my opinion. Because if I do that, then I am no longer doing my job the way I’m supposed to and there are also constitutional issues that come into play. As I have said in court more than once, this is a court of law not a court of research. That’s the end.
In addition to these more formal policies and procedures, a few practitioners mentioned the perceptions of others’ as barriers. For example, one judge noted, “[M]y greatest frustration is when people don’t know the research and assume you are being ‘soft’, when you are just doing something that is developmentally appropriate.”
Finally, participants reported that the nature of a given case (n = 19) influenced their ability to use research. Generally, judges and attorneys expressed a sense that in the presence of contention or uncertainty, there is often a greater demand for research to inform the decision-making process. Still, others noted that when cases were especially contentious or had a lot of media attention, they were less able to use research to inform their decision-making.
Are There Critical Junctures in Juvenile Justice Proceedings Where Research Is Most Likely to be Used?
We also asked respondents about how research was used during different junctures of a case, to identify whether typologies of research use depended on the stage of case processing. Overall, we found that typologies of research use were not contingent on case processing stage. Instead, respondents told us that there were some stages which allowed for inclusion of research information in decision-making and others that did not.
Respondents felt that research was most useful during decisions related to diversion, detention, and disposition. Specifically, 15 judges and eight attorneys said that research on the effectiveness of incarceration and/or research-based tools such as risk assessments were commonly used in decisions about detention. Additionally, 21 judges and six attorneys indicated that research is used during disposition. Practitioners felt that evidence-based tools, such as dispositional matrices and risk and needs assessments, the availability of evidence-based programs, or their general understanding of current research literature can serve as a backdrop, or determining factor, when deciding what disposition a youth should receive.
Practitioners had mixed perceptions on the use of research during adjudication. Many (n = 15) felt that research should never be used in this juncture. For example, one judge explained, “When you adjudicate, that decision has to be based on the [legal] evidence presented in court, so I don’t know how anybody could use research for that.” In contrast, there were a few respondents who noted that research was critical in determining criminal intent and culpability, and research on the context of confessions and eyewitness statements, was critical for informing adjudication decisions. Although respondents noted that research was relevant to decisions regarding transferring or waiving youth to the adult criminal justice system, they were often not able to use research due to state statutes, or because other decision-makers were involved in decisions regarding transfers/waivers to adult court. Finally, respondents noted that they had little involvement in the referral and release junctures and, as such, could not describe how research is used in these junctures. Similarly, respondents who were judges and defense attorneys noted that they are typically not involved in intake decisions.
What Are Best Practices in Conducting, Translating, and Disseminating Research?
We asked participants how researchers could make their research more useful to practitioners. Differences did not vary by practitioner type, or by the type(s) of research use that the practitioner engaged in. Respondents reported that researchers should ensure that they are writing in a clear, succinct, jargon-free manner (n = 10). Beyond clear presentation of findings, respondents emphasized that researchers need to provide clear, actionable implications for practice and policy. For example, one judge noted: I’ve been to conferences before where people have left the room saying, “Wow, that’s really interesting. That’s great information. Oh my gosh, could you believe that? But why did I sit there for 90 minutes and listen to that, ‘cause it’s not gonna be helpful to me, because there’s no way to practically apply it or utilize it.” So, I think you would always want to have a portion of the presentation at the end, “So, what does this mean to me?” or, “How can I use this to make the lives of my kids better?”
Some participants suggested that researchers include juvenile justice practitioners in the research process (n = 6). Similarly, a few respondents recommended that researchers spend time observing courts and learning about the context of juvenile justice proceedings (n = 3). For example, one prosecutor explained: I think that if researchers had a firm sort of good understanding of what we do and what sort of day-to-day versus the ivory tower type thing, I think that would be helpful because sometimes researchers don’t [know]. They read about how things are supposed to work, but don’t necessarily have an idea of how things actually work. So, I think if they had a practical knowledge, then that would be beneficial in terms of making their research better for our use.
Respondents noted that inclusion in the research process should consist of practitioners working with researchers and funders to develop research questions as well as partnering to interpret the findings and translate the findings into actionable practices and policies to implement.
Discussion
Our study explores how juvenile justice judges and attorneys define, acquire, interpret, and use social science research as well as the conditions that shape that use. Results from this study suggest that overall, judges and attorneys have a sound understanding of what research is and how it can be used enhance their work. Additionally, the results of this study emphasize the value of researchers partnering with practitioners to more fully understand the individuals and agencies involved in the day-to-day operations of the juvenile justice system. Moreover, these results highlight the value of using implementation science as a lens to guide the implementation of research-based reforms as we found evidence of multiple implementation drivers, including competency drivers, such as the need for staff training and buy-in; organizational drivers, such as the need to have quality data systems and supervision structures in place to monitor performance; and leadership drivers, such as the need to authentically engage partner agencies in strategic planning efforts. In addition to implementation drivers, the study findings lend support to the notion that implementation drivers are multi-layered and interdependent. For example, many respondents indicated that although their inner organizational context (e.g., leadership, organizational climate, and staff competencies) supported the use of research, factors external to their organization, such as state policy and availability of evidence-based programs, precluded them from being able to use research.
A particularly interesting finding is that conceptual research use was most prevalent among our respondents, particularly judges. Many respondents described “ah-ha moments” where exposure to research findings changed their thinking about a topic which led to a subsequent change in their behavior. For example, several judges noted that research on adolescent brain development, the effects of trauma, and the effectiveness of evidence-based alternatives to incarceration shifted their views on detention and incarceration which in turn made them less likely to detain or commit youth.
Our finding that practitioners are most likely to use research conceptually is aligned with research from other fields which finds that policymakers commonly engage in conceptual research use (Bogenschneider, Corbett, and Parrott 2019; Farrell and Coburn 2016; Yanovitzky and Weber 2020). Yanovitzky and Weber (2020) suggest that conceptual research use is likely prevalent among policymakers because, much like juvenile court practitioners, policymakers must consider the needs, interests, and inputs of different stakeholders to make decisions about complex problems and therefore cannot rely on a single piece of information to make a decision.
Drawing parallels between conceptual research use and schema theory, an information processing model of perception and cognition (Axelrod 1973), Yanovitzky and Weber argue that research-based information is its most consequential when it has the ability to change a decision-maker’s schema or “filter” for how they process, organize, and interpret information.
The finding that judges tended to use research more conceptually may also explain our finding that attorneys underestimate the extent to which judges use research. Additionally, this finding suggests that it may be difficult to measure judges’ research use through more objective measures, such as coding references in case files, transcriptions, or court minutes.
Our finding that risk and needs assessments are often used inconsistently is aligned with findings from other studies (e.g., Love et al. 2016; Sullivan 2019; Vincent et al. 2016), indicating that continued training, coaching, and supervision around the use of risk and needs assessments is likely needed to support any research-based reform that incorporates the use of risk assessments. It is important to note, however, that when respondents reported overriding the assessment, it was because they questioned the accuracy of the assessment rather than just being opposed to using the tool. Accordingly, continued assessment and validation of risk and needs instruments is needed—both to ensure that the tool is working as intended as well as to demonstrate to stakeholders how the tool can be used to help them achieve their mission more effectively.
Related to skepticism around risk and needs assessments, although judges and attorneys were willing and sometimes eager to use research, many respondents expressed concerns about the credibility of research, often questioning whether the producers or funders of a particular piece of research had a hidden agenda. While, many respondents noted that it was important to look at the study methodology—mostly the composition of the study sample—numerous participants either struggled to articulate how to determine whether the methods and statistical approaches used were appropriate or expressed that they just didn’t have sufficient training to do so. Accordingly, many respondents noted that they assessed the credibility of a piece of research by whether they felt that it came from a trusted source—whether it be a colleague, academic institution, or intermediary organization. In fact, many respondents noted that their state agencies and professional development organizations served as filters for research, vetting out research that was unreliable or biased. This finding underscores the critical role of the academic community in helping practitioners and intermediary organizations identify low quality research and pseudo-science, as the use of such research can lead to great harm, such as that which was used to develop the “super predator” myth (Larson and Carvente 2017).
Recommendations for Research, Practice, and Policy
Although participants said that research can enhance their decision-making, they noted that too often research was not useful to them or it was inaccessible. Many respondents lamented about journal paywalls, jargon, the length of articles, and researchers failing to make concrete or realistic recommendations. Participants often paired their critiques of research with recommendations for making research more accessible and useful to judges and attorneys. These critiques and recommendations for researchers are summarized below.
First, participants noted that they rarely have time to read lengthy articles. Instead, they preferred receiving research through in-person presentations, brief jargon-free summaries of research findings, and data visualizations. Interestingly, although respondents emphasized the need for these short, user-friendly summaries, they felt that it was important for them to have the ability to dive deeper into the methods and suggested that all summaries include links to more detailed information. Similarly, participants noted that they preferred in-person trainings because they 1) occur during time set aside for professional development; and 2) provide an opportunity for them to ask the researcher questions. Participants reported that presentations were further strengthened when they included a practitioner who could share how they used the research to inform their day-to-day practice.
Second, practitioners reported feeling like much of the research they came across wasn’t fully relevant to their work. Most often, this was because researchers either failed to include recommendations for practice or they included recommendations that were either too vague to be implemented or were not feasible. Participants felt that this issue stemmed from researchers having a limited understanding of the day-to-day operations of the juvenile court system.
As such, many respondents noted that researchers would benefit from spending time interacting with juvenile court systems to better understand the legal framework within which they operate and the day-to-day functioning of the juvenile justice system. Respondents felt that the engagement of researchers in the daily operations of juvenile court systems will help to ensure that research is designed and reported in a way that will ultimately be useful for decision-makers.
Beyond wanting researchers to spend more time immersed in the daily operations of juvenile court systems, judges and attorneys indicated that they desired increased engagement from the research community in the implementation of studies. By and large, respondents spoke positively of their involvement in research-based reform efforts and desired to collaborate with researchers so that researchers could produce more relevant research. Judges also reported being “hungry for data” and wishing that they had access to their local data to better understand the youth in their jurisdictions and the effectiveness of different dispositions. Researchers who produce high quality, policy relevant information can leverage this openness to engagement by including practitioners in the formative stages of research design, on advisory panels, and engage them when interpreting results and making recommendations based on findings. Beyond designing and conducting studies that meet the needs of practitioners, by partnering with practitioners, researchers can better identify implications and recommendations for practice and policy based on their study findings.
Participants reported they also felt research wasn’t relevant to their work when it wasn’t based on the population of youth they serve. Participants expressed frustration trying to identify which populations of youth the research was based on. To address this issue researchers should clearly define the population their results are generalizable to and tag their research with relevant key terms so that their information is discoverable through internet searches. Additionally, researchers should create descriptive titles that include information on the characteristics of the samples the study is based on. Finally, researchers should ensure they are creating evidence-bases built on diverse populations of youth, in particular, respondents noted a need for more information regarding youth living in rural communities, females, and American Indian/Alaska Native youth.
Third, respondents repeatedly noted that they had limited access to findings published behind paywalls. Accordingly, they recommended that researchers take every step possible to make their findings publicly available. This could be achieved through paying open access fees or publishing in open access journals, and/or getting approval from a journal to share a practitioner-friendly summary that links to the full journal article. Moreover, given that judges and attorneys often have requirements for continuing education, researchers should consider partnering with professional development organizations or state agencies to leverage ongoing, in-person, or web-based trainings as a venue for presenting relevant research findings. Sharing research in this manner not only leads to an efficient use of the limited time judges and attorneys have for professional development, it provides an opportunity for practitioners to engage in a dialogue with researchers about how the research can inform their work.
Beyond disseminating research, participants felt that state agencies and their professional organizations were critical for vetting the quality and relevance of research, synthesizing research, identifying practical implications of research findings, and getting access to the research. These professional organizations include larger national organizations such as the American Bar Association and the National Council of Juvenile and Family Court Judges, as well as local and state agencies. Judges and attorneys from rural jurisdictions felt that their local and state agencies were a critical conduit for obtaining research, as they often did not have the time or funding to attend national conferences. As such, researchers should consider intermediary organizations as key stakeholders when developing dissemination and outreach plans, particularly if they lack the time, resources, and expertise to effectively translate and disseminate research to practitioner audiences.
In addition to the recommendations for researchers outlined above, our study’s findings also have several implications for practice and policy. First, given that practitioners report receiving the majority of their research through trainings, juvenile justice systems may want to ensure that research is incorporated into all relevant professional development and orientation trainings. Additionally, jurisdictions may want to consider requiring or incentivizing practitioners’ attendance of these trainings, particularly trainings for probation officers, who often are charged with implementing evidence-based tools, such as risk assessments yet have limited access to the research in which these tools were based upon. Drawing on lessons learned from implementation science, these trainings should introduce the background and philosophy underlying the new approach as well as be interactive and focus on building both staffs’ skills and buy-in (Sullivan 2019). In addition, booster trainings, technical assistance, and coaching should be made available to assist with translating the new skills or knowledge into practitioners’ day-to-day interactions with youth.
Although practitioners report receiving a great deal of research-based information, they report a strong need for local data and evaluations of the effectiveness of research-based tools, various dispositions, and other reform efforts. As such, juvenile justice systems should consider investing in research practice partnerships with researchers, encouraging the use of local data to inform reform efforts, and requiring implementation and outcome evaluations of reform efforts and research-based tools. Research practice partnerships may be particularly beneficial at the outset of a new reform initiative, as research has shown that implementation proceeds more smoothly when staff have an opportunity to engage in the planning process, express their concerns, learn about the rationale for the change, and understand how the change can benefit their work (Bryson 2007; Meyers, Durlak, and Wandersman 2012; Sullivan 2019).
The importance of developing these multi-agency partnerships, which also aligns with the National Research Council’s (2013) recommendation to establish state-level stakeholder groups to guide reform efforts, was particularly emphasized by judges, who felt that they were uniquely qualified to convene and lead these groups. Judges felt that these partnerships should include all parties involved in implementing the new approach, including those from different agencies. Including members from different agencies can assist in both garnering the support of the different agencies, which is particularly important when the agencies have different orientations or constituents (Sullivan, 2019). For example, respondents who had more positive perspectives toward research use tended to participate in multi-agency groups to review research and data, whether it be through formal participation in a research-based initiative such as the Juvenile Detention Alternatives Initiative or more informal get togethers, like a monthly brownbag series hosted in a judge’s courtroom.
Finally, given the stark contrast between rural, urban, and suburban jurisdictions, state policymakers and state agencies should consider providing more resources to rural jurisdictions as well as consider peer trainings or peer learning collaboratives where practitioners in rural jurisdictions can be trained in research by colleagues who have greater access to research.
Study Limitations
Although our study offers insight into the use of research evidence in the juvenile court system, there are several limitations that warrant consideration. First, because the findings from this study are based on a small sample, they may not generalize to the larger population of judges and attorneys overseeing delinquency cases. Our findings are strengthened, however, through our use of stratified purposeful sampling, as we interviewed respondents from a total of 30 states, representing a cross-section of juvenile justice practitioners across urban, suburban, and rural jurisdictions throughout each region of the U.S. Nevertheless, we acknowledge that the respondents who agreed to participate in our study may not be representative of all judges and attorneys in their jurisdiction. Second, although our response rate is similar to other studies with juvenile court practitioners (e.g., Grindall 2003; Mears et al. 2010), we acknowledge that the low response rate may further limit the generalizability of these findings. The alignment of our findings with those from other fields examining the use of research evidence (e.g., DuMont 2015; Tseng 2012) as well as studies conducted on research use in the juvenile justice system (e.g., Pesta et al. 2017), however, bolsters our findings. Third, generalizability of this study may be further limited by the assumption that those who volunteer to participate in a research study may have more positive perceptions of research than those who decline to participate. We were, however, able to include three respondents who were very wary of the use of research in juvenile justice proceedings so the perspectives of those skeptical of research are also included in these findings. Finally, the perspectives included in this study are limited to those of juvenile court practitioners and thus may not be representative of other stakeholders in the juvenile justice system, such as probation officers, who may use research in their day-to-day work with youth and families. Our decision to focus on juvenile court stakeholders was intentional, however, as prior studies examining the research-to-practice gap have focused on other professionals working in the juvenile justice system, such as senior level practitioners in state agencies, state-level policymakers, mid-level practitioners, technical assistance specialists, researchers, and experts (Johnson, Lebold, and Elam 2016; Love et al. 2016; Pesta et al. 2017). Nevertheless, more research is needed, particularly that which is grounded in a shared conceptual framework, to better understand how various juvenile justice practitioners engage social science research in their decision-making.
Conclusion
This study represents a unique effort to illuminate the ways in which juvenile court practitioners engage with, evaluate, and apply social science research to their work. Although researchers have made substantial contributions to the juvenile justice evidence base in recent years, more can be done to ensure that high quality, policy relevant research reaches judges, attorneys, and other actors in juvenile justice systems. In particular, the results of this study highlight the value of researchers partnering with practitioners to fully understand the characteristics and operations of the different agencies and individuals involved in the juvenile justice system. Practitioners in this study expressed that they are eager and willing to incorporate research findings into their work so the onus is on researchers to ensure that their work is translated for the right audiences and incorporates tangible action steps that can be applied in real world settings.
Footnotes
Authors’ Note
The opinions, findings, and conclusions or recommendations expressed in this article are those of the authors and do not necessarily reflect those of Child Trends.
Acknowledgments
We would like to thank members of our project team (Elizabeth Jordan, Kristin Anderson Moore, Miranda Carver Martin, Ann Schindler, Garet Fryer, and Michael Bradley), our expert consultants (Timothy Ross and Judge William Thorne), members of our expert advisory panel (Kent Berkley, Ana M. Bermúdez, Mark Greenwald, Naomi Goldstein, Jessica Feierman, David Meyers, and Jennifer White), and National Institute of Justice staff (Eric D. Martin and Natasha Parrish) for their contributions to the project.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
This project was supported by Award No. #2015-R2-CX-0014, awarded by the National Institute of Justice, Office of Justice Programs, U.S. Department of Justice. The opinions, findings, and conclusions or recommendations expressed in this article are those of the authors and do not necessarily reflect those of the Department of Justice.
