Assessing Youth Level of Service/Case Management Inventory Implementation Outcomes: Lessons from Five Diverse Pennsylvania Counties

Abstract

We examined implementation outcomes several years after rollout of the Youth Level of Service/Case Management Inventory (YLS/CMI) risk/need assessment (RNA) tool in five diverse Pennsylvania county juvenile probation offices. Offices had policies to direct the use of the YLS/CMI, and officers tended to view the tool favorably, complete it, and apply it in their work. However, there were also variations in the extent of implementation. These seemed related to differences in office leadership and climate, implementation and quality assurance strategies, probation officers’ support for reforms, and the broader stakeholder environment. Results are largely consistent with implementation science principles.

Keywords

risk assessment risk-need-responsivity juvenile justice probation implementation

Introduction

Structured risk/needs assessment tools (RNAs) are the cornerstone of decision-making within the “Risk-Needs-Responsivity” (RNR) model (Andrews et al., 2011; Bonta & Andrews, 2010). However, while a substantial body of literature speaks to the predictive validity of RNAs (Gendreau et al., 1996; Latessa et al., 2009; Vose et al., 2008), and research shows they contribute to offender success when properly used (Luong & Wormith, 2011; Vieira et al., 2009), the extent to which practitioners actually use RNAs in line with RNR principles is variable in practice. While some short-term follow-up evaluations of well-resourced pilot RNA implementations show improvements in decision-making (Vincent, Paiva-Salisbury, et al., 2012; Vincent et al., 2016; Young et al., 2006), research in more routine community corrections contexts tends to indicate only partial adherence to RNA policies (Miller & Maloney, 2013; Viglione et al., 2015). It is, therefore, important to understand how these tools can be effectively implemented and sustained in local settings. This article helps address this question by documenting the patterns of use of the Youth Level of Service/Case Management Inventory (YLS/CMI) RNA in local Pennsylvania counties several years after its initial state-wide introduction, and examining the processes that have shaped outcomes.

Background Literature

Risk/Need Assessment (RNA) Tools

Contemporary RNAs score individual characteristics to produce empirically validated “risk” scores (Gottfredson & Moriarty, 2006), allowing clients to be classified into groups reflecting their likelihood of recidivism. These tools also score “needs”—namely dynamic risk factors that are susceptible to change through intervention. “Fourth generation” RNAs support ongoing assessment and case planning, and often measure “responsivity” factors that affect client responses to treatment (Bonta, 1996).

Particularly in their later generations, RNAs support key principles central to the RNR model (Andrews et al., 2011; Lowenkamp et al., 2006). According to the risk principle, the intensity of service should be matched to the offender’s risk. The need principle indicates that interventions should target the specific needs of an offender that lead to criminal behavior. And the responsivity principle affirms the superiority of cognitive and social learning interventions (“general responsivity”) and asserts that interventions should be tailored to the characteristics of offenders (“specific responsivity”) including, for example, their strengths, motivation, learning ability, and demographic characteristics (Andrews et al., 2011; Dowden & Andrews, 2004).

RNA Use in Practice

Studies of pilot RNA implementations (including the YLS/CMI in some Pennsylvania counties) suggest they produce positive changes in practitioner decision-making in the first year or so (Vincent et al., 2016; Vincent, Paiva-Salisbury, et al., 2012; Young, et al. 2006). However, studies in more routine contexts—removed in space and time from pilot implementation—offer a more cautious assessment. While practitioners do use RNAs, they also show patterns of significant underutilization (Haas & DeTardo-Bora, 2009; Hannah-Moffat et al., 2009; Whiteacre, 2004). One national survey of community corrections practitioners found that, while they tended to fill out RNAs when required, practitioners often made decisions that did not correspond with their results (Miller & Maloney, 2013). Similarly, an ethnographic study showed that adult probation officers filled out RNA tools but made little reference to RNAs, case plans, or criminogenic needs in client meetings (Viglione et al., 2015).

Gaps between policy and street-level practitioner practice are well documented (Brodkin, 2011; Viglione, 2017) and are epitomized by Lipsky’s (1980) “street level bureaucrats,” who use their discretion to adapt formal policy mandates to the rigors of frontline practice. Correctional environments may be a particular challenge for reform implementation, given a culture that may prioritize traditional risk or liability concerns and bureaucratic structures that may resist change (Farrell et al., 2011; Lynch, 1998; Viglione, 2017).

Implementation Science

Scholars have paid increasing attention to policy implementation strategies within organizations, contributing to the development of a body of “implementation science” literature. For example, Damschroder et al.’s (2009) Consolidated Framework for Implementation Research (CFIR), depicts implementation as contingent on five major domains: the nature of the intervention itself, including its suitability for practitioners; the character of the outer setting, including peer institutions and external mandates; the inner setting, including the organizational structure and “implementation climate” (such as the organizational fit of the intervention, systems of rewards and incentives, readiness for change, and leadership); the nature of individuals within the organization, including their knowledge and attitudes; and the implementation process itself, including planning, engagement, evaluation and reflection, and the cultivation of buy-in from stakeholders.

The National Implementation Research Network (NIRN) “implementation frameworks” (Bertram et al., 2015; Fixsen et al., 2005) offers an action-oriented implementation model that emphasizes the “drivers” of implementation. These include: competency drivers that develop practitioner skills and confidence through selection, training and coaching; organization drivers such as performance tracking, problem solving, data systems, and efforts to maintain support; and leadership drivers concerned with management in moments of stability and uncertainty. Authors also emphasize the unfolding multi-year process of implementation, and highlight the importance of continued attention to program fidelity after initial implementation has passed. Case study evidence on RNA implementation echoes these insights, emphasizing leadership, training, monitoring, and the challenges of staff resistance (Ferguson, 2002; Vincent, Guy, et al., 2012; Young et al., 2006).

The Current Study

Informed by the literature on RNA use and implementation science, the current study examines the use of the YLS/CMI juvenile risk assessment, a fourth generation RNA (Hoge & Andrews, 2002). It focuses on five Pennsylvania case study county juvenile probation departments, several years after the YLS/CMI’s state-wide phased introduction.

Context

In Pennsylvania, a state of 12.8 million people, juvenile probation is a patchwork of autonomous county-based agencies that, nonetheless, have strong policy coordination through state leadership organizations. The state-led effort to implement the YLS/CMI proceeded in four staggered phases across counties (2009, 2010, 2011 and 2012, respectively), with 66 out of 67 ultimately implementing the tool.

The implementation effort appeared well planned and energetic, and gave attention to ongoing problem-solving, adaptation, and quality assurance. Planning and training on the YLS/CMI for early phases relied on significant input from external consultants. In later phases, the state adopted a “train-the-trainer” model, whereby probation officers trained during earlier phases underwent certification to become “master trainers,” and helped train and coach staff in counties newly implementing the tool. The rollout was also supported by training on principles of evidence-based practices more generally, and the use of tools for counties to assess organizational readiness. A further supporting strategy was the convening of regular conference calls (grouped by implementation phase) allowing staff from more experienced YLS/CMI counties to help problem-solve implementation challenges being experienced by new or less experienced sites. YLS/CMI implementation ultimately became part of a broader evidence-based juvenile justice reform strategy in Pennsylvania from 2010.

Importantly, state reform leaders’ attention to YLS/CMI implementation had continued until the time of this research. Leaders continued to focus on frontline staff’s fidelity to the YLS/CMI, for example through ongoing measurement of YLS/CMI completion, the continuous training and certification of local master trainers, and the regular convening of quarterly conference calls across counties. They also responded to implementation problems through specific interventions. This included, for example, a recent redesign of a case plan template to address operational challenges in the field.

Research Questions

While state organizations expected county probation staff to complete the YLS/CMI and use it to inform dispositional recommendations and case planning, county implementation teams were ultimately left to adapt local YLS/CMI policies to fit local contexts and sensibilities. The research asks two primary questions about these efforts, several years after introduction of the YLS/CMI: (1) What implementation patterns are observed in the five counties? (2) What factors account for variations in implementation patterns? Results are reviewed in the broader context of the implementation science literature.

Data and Methods

The research first adopted a mixed methods approach, relying on both quantitative and qualitative data, to describe the extent and patterns of implementation—in relation to local policies, officer completion of the YLS/CMI, and officer application of assessment tool results. It then used qualitative data to identify plausible explanations for variations in implementation patterns, informed by existing literature. We purposively selected case study counties to ensure variation in size, location, demographics (including level of urbanization), phase of YLS/CMI implementation, and leadership involvement in state reform efforts (Table 1 provides details) to ensure variations in implementation experiences.

Table 1.

Demographic and Juvenile Justice Characteristics of Case Study Counties.

County	YLS/CMI rollout phase	No. JPOs (2016/17; range reported only)	All juvenile dispositions per officer (JCJC 2015)	Probation, consent decree, and placement dispositions per officer (JCJC 2015)	County population size (ACS 2015)	Urban/rural predominant by population (2010 census)
A	1 (2009)	40–49	27.7	13.4	Medium (100–500 k)	Urban
B	2 (2010)	40–49	21.9	9.2	Large (>500 k)	Urban
C	3 (2011)	>100	19.2	5.4	Large (>500 k)	Urban
D	2 (2010)	10–19	10.6	6.9	Small (<100 k)	Rural
E	4 (2012)	<10	21.2	7.4	Small (<100 k)	Rural

Notes: Probation, consent decree, and placement dispositions all typically require active JPO supervision. The number of JPOs is reported as a range, to limit the possibility of deductive disclosure of county identities. JPOs in County E were embedded within a larger juvenile and adult probation agency (some JPOs listed here also have adult probation responsibilities). JPO = juvenile probation officers (of all ranks); JCJC = Juvenile Court Judges Commission; ACS = American Community Survey.

Qualitative Data: Interviews, Observations, and Policies

In addition to collecting and reviewing local policies (e.g., organizational charts, official YLS/CMI and case plan policies, and court report templates), we collected and analyzed a range of interview and observational data (fieldwork was conducted between 2016 and 2018), as described below (and summarized in Table 2).

Table 2.

Qualitative Research Subjects from Case Study Counties (from Interviews and Observations).

	Core interviews	YLS/CMI case study interviews	Observed subjects	All subjects
Line probation officers	56	38	72	91
Community-based	22	19	40	44
School-based	6	5	7	9
Specialized	16	5	11	19
Intake/assessment	12	9	14	19
Supervisors	21	0	2	22
Chief/Deputy Chief/Assistant Chief	9	0	0	9
Total	86	38	74	122

Interviews with probation officers

We conducted 86 open-ended mostly in-person “core” interviews with officers from a range of roles and ranks. For smaller counties (D and E) we interviewed all or most officers, while in larger counties (A, B, and C), we interviewed a sub-sample of officers, ensuring coverage of a range of ranks and roles. Interviews used a topic guide focused on the officer’s role and background, their experience with the YLS/CMI and other reform efforts, and their perspective on the broader organization. We generated notes from these interviews, mostly informed by digital recordings. Additional follow-up interviews were conducted with 38 probation officers (mostly by telephone) who were actively using the YLS/CMI during the course of their day-to-day duties. These interviews focused on specific examples (usually two per officer) of how they completed and made decisions using the YLS/CMI, resulting in 81 separate cases in total. The interviews used a topic list designed to shed light on officers’ reasoning when scoring the YLS/CMI and making decisions informed (or not) by YLS/CMI scores.

Observations of probation officers

We undertook a total of 198 hours of direct observations, evenly distributed across counties, involving over 70 frontline probation officers. The scheduling of observations was shaped by convenience and officer availability, and was in part contingent on the decisions of office managers and administrators. Observations were recorded as brief hand-written notes, followed by the elaboration of detailed fieldnotes within the following day or two (Emerson et al., 2011).

Interviews with state reform leaders

We also conducted a mix of in-person and phone interviews with 12 state reform leaders during the second half of 2017. These included administrators, probation officers of varied ranks, and consultants who had played key roles in the design or implementation of reforms, with a topic guide that centered on these experiences. These were all recorded and transcribed.

Analysis

We analyzed core case study interview transcripts and observational fieldnotes by importing them into NVIVO qualitative data analysis software. We subjected core interviews to thematic coding, tapping into prior areas of interest, while also allowing codes to evolve inductively. We also produced some analytic matrices/spreadsheets to summarize views, practices and policies revealed by interviews, using codes shaped by prior research and inductive themes. We reviewed reform leader interviews, without conducting detailed coding, to build a broad story of the YLS/CMI implementation process. We coded observational data to identify discrete “interactional episodes,” typically between officers and clients (Viglione et al., 2015). These mostly included assessment meetings, supervision contacts, and court hearings. We recorded key features of these episodes in fieldnotes, and we created spreadsheets to distinguish episodes according to their characteristics. We also reviewed YLS/CMI examples and coded them into a spreadsheet structure; this supported descriptive analysis of the strategies used to complete them and the decisions taken or contemplated about the youth. Finally, local county policies were summarized, according to key policy components, in spreadsheet format. This facilitated comparison and description.

Officer Survey

Data collection

We conducted a state-wide electronic web-based survey in the summer of 2018. The survey instrument was informed by the qualitative research, and also included some measures based on earlier, related surveys (Miller & Maloney, 2013, 2015). Following preliminary pilot-testing, the deputy director of the Juvenile Court Judges Commission (JCJC) emailed the survey to county probation chiefs, with a request to forward to probation staff. Outreach included a pre-survey email, and an initial and three reminder survey emails at roughly 1, 3, 5, and 7-week intervals (with an opportunity for participants to enter a drawing for one of five $20 store vouchers). Researchers also made some follow-up phone calls and e-mails to counties with limited responses. Here, we analyze 155 responses from Counties A through D, with county response rates ranging from 55% to 88% (County E produced too few responses to analyze separately).

Analysis

Analysis focused primarily on descriptive summaries of single survey items, focused on how often officers performed a range of YLS/CMI-related behaviors (all mapping to a 0-10 scale). Additionally, we used two attitude scales, based on earlier measures developed in a prior survey of PA juvenile probation officers (Miller & Maloney, 2015). A support for the YLS/CMI scale (range 0–4, alpha = 0.95) was based on eight 5-point agree/disagree items based on statements such as, it “provides helpful guidance to probation officers they wouldn’t otherwise have,” and “involves the appropriate measures of risk characteristics”. We also used a broader support for risk/need assessment (in place of practitioner judgment) scale (range 0–4, alpha = 0.90). This used five 5-point agree/disagree items based on statements that included: “Real people are naturally better than assessment tools at assessing needs,” and “Assessment tools are better than subjective individual judgments at assessing risks.”

Results

We first present findings concerning the local YLS/CMI policies adopted, officer’s attitudes towards the tool, and officers’ use of it in their routine work. We then offer hypotheses to explain variations in YLS/CMI implementation patterns within and between counties.

County Policies

Local YLS/CMI policies were, to some extent, dynamic. Even during our research, there were modifications made to policy documents, and new policies introduced. There had also been recent innovations in local policy—including the introduction in most settings of a revised “field case plan” to replace an older, unpopular, case plan template.

Notwithstanding, at the time of the research, each county policy required officers to complete an initial electronic YLS/CMI at or soon after intake (usually by specialized assessment personnel) and then follow-ups (by supervising officers), minimally at 6-month intervals, and at case closing. Decisions and recommendations (e.g., dispositions, case planning, supervision, and services) were to be shaped by the YLS/CMI risk and need information and, less prescriptively, by responsivity factors. Decisions relied variously on service matrices (Counties A, C and D) linking risk/need profiles to interventions, a case plan document to help action case management goals (Counties B, C, D, and E, with A’s under revision), and written protocols linking risk to supervision frequency (active in Counties C, D, and E, with A and B’s under development). Officers were also expected to share some YLS/CMI information with service providers and court stakeholders. All counties used some proprietary evidence-based exercises with clients during contacts, presumably informed by the YLS/CMI (e.g., “Carey Guides” or “Brief Intervention Tools”). YLS/CMI use was further supported by “booster trainings,” approval/sign off by supervisors, supervisory case reviews, and reviews of statistical reports.

There were, however, some notable county policy differences. Policies in Counties B and E were the least developed, lacking some core policy elements and having no provision for routine case reviews. County A’s policies were apparently the most developed, embracing many of the core and supporting elements found across other counties, while also deploying some novel innovations. The latter included the direct provision of extensive in-house group programs, quality assurance mechanisms ensuring alignment of YLS/CMI with practice (including a recently introduced observational quality assurance protocol to be conducted by probation supervisors), and a strong emphasis on the verbal presentation of the YLS/CMI in court. More details of policy variations in the five counties can be found in Miller and Maloney (2020).

Staff Attitudes Toward Risk/Need Assessment

Probation officers appeared more positive than negative towards the YLS/CMI. This was evident from generally positive county mean scores for the support for the YLS/CMI scale and the broader support for risk/need assessment (in place of practitioner judgment) scale (Figure 1). Interview accounts echoed these findings, with views emphasizing the YLS/CMI’s validity and utility for decision-making. For example, one officer explained “I think it’s the best resource we have to get information and identify risk and to steer us toward what programs to use and the amount of contacts that are necessary.” However, some more skeptical views were at times also voiced. These emphasized the inferiority of the YLS/CMI to the insights and experience of probation officers, and concerns about the extra workload generated.

Figure 1.

Mean attitudes to the YLS/CMI and risk/need assessment principles among county juvenile probation officers (survey, scores range from 0 to 4; n = 154). Both comparisons show statistically significant differences across the four counties (based on Kruskal-Wallis tests; p < .01).

County differences were, again, evident. They were seen in (statistically significant) differences by county in survey attitude scales (Figure 1), with County A the most positive and County B the least. Similarly, officers from County B tended to express negative views in interviews more strongly and frequently than officers in other counties.

YLS/CMI Completion Practices

Officers tasked with completing the YLS/CMI indicated they tended to complete the YLS/CMI when required, albeit not always. This is illustrated by survey results (see Figure 2), and is supported by interviews with officers (and with state leaders involved in monitoring YLS/CMI use). As one officer said: “it’s just one of those things that you know you have to do. And it’s done and you do it.” However, interview evidence also suggests that officers sometimes did not get their follow-up assessments done, at least on time.

Figure 2.

Means of officer YLS/CMI adherence measures (survey; scores range from 0 to 10; sample n = 86–117). Items described as “(neg.)” have been reverse scored so that positive numbers indicate adherence. Items with asterisks (*) show statistically significant differences between the four counties (based on Kruskal-Wallis tests; minimum p < .05).

Officers also mostly seemed committed to completing the YLS/CMI carefully and honestly. Survey measures (see Figure 2) suggest they reviewed relevant data sources and did not rush or manipulate scores, though they were sometimes inclined to use upward overrides. In reviews of completed YLS/CMIs, frontline officers could mostly provide justifications for the criminogenic needs factors they checked when asked, suggesting a good faith effort at completion.

However, we saw some diversity in the character of YLS/CMI completion. Perhaps as many as half of the 81 YLS/CMI examples we reviewed showed potential discrepancies with state scoring guidelines, apparently because of officers tending to adapt (rather than ignore) the guidelines. Officer interviews, including discussions of YLS/CMI examples, also suggest that upward overrides were sometimes used to upgrade the overall risk category for more serious crimes (most notably sex offending).

There was also some variation in the process followed to complete YLS/CMI. Both interviews and observations suggest efforts were greater for the initial assessment compared to follow-ups. The former seemed to consistently involve sit-down meetings with youth and a parent (sometimes together, sometimes separately; depending in part on office norms and policy), while follow-up assessments often relied on officers’ working knowledge of the client. Among the 21 sit-down assessments to inform the YLS/CMI that we observed (17 initial assessments and four re-assessments), officers also varied as to whether they followed a written question script, referred directly to the YLS/CMI (in a couple of cases they even shared the YLS/CMI instrument directly with the youth for their review), or conducted a looser conversation that elicited YLS/CMI information indirectly. Most still seemed to touch on the key YLS/CMI domains, however.

Finally, the survey showed some variation across counties in YLS/CMI completion practices (Figure 2). Notably, there was statistically significant variation for completing the YLS/CMI when required, with Counties A and D scoring higher than Counties B and C.

Applying the YLS/CMI in decision-making and case-planning

Consistent with local policy, and broader risk, need and responsivity principles, officers to varying degrees seemed to use the YLS/CMI to inform decisions. Decision-making survey measures (Figure 2) were generally positive (though sometimes a little lower than completion scores). In interviews, need and risk categories (though not so much responsivity) were among the factors interviewees highlighted when asked how the YLS/CMI affected decision-making. For example: “the need on the YLS is going to be in the case plan,” “I try to touch the top two [criminogenic needs] during supervision,” and “low [risk] kids you obviously see less.” And, among example YLS/CMIs reviewed with officers, institutional placement was more common, and the range of services greater, for higher risk clients. Service decisions were also typically relevant to at least some YLS/CMI needs, with services targeting low scoring YLS/CMI domains in only a minority of cases.

In line with the survey (Figure 2), however, interviews highlighted other decision-making criteria that competed with YLS/CMI results in decision-making. For example, these could involve a concern over the nature of the client’s offense. One interviewee noted: “there are other things that I might include that . . . are not necessarily related to criminogenic needs . . .” subsequently clarifying “we take into consideration consequences, because they aren’t necessarily here because they did something great.” Some emphasized their own judgments alongside raw tool results, for example: “If I know he is a heroin user, I’m going to up my supervision of him and put another program to help him kick the habit – based more on what the kids are doing rather than sole focus on the risk score.” Some highlighted relevant non-criminogenic needs such as mental health, with one officer noting: “when [we have] a juvenile with severe mental health issues . . . it’s not going to address that.” And officers often raised questions about the accuracy of the YLS/CMI, for example: “When you get the YLS from intake, you have to take that with a grain of salt, because they’ve only had the case for a short amount of time . . . [and it’s] mostly self-reported information.” Significantly, a number of officers also seemed to place little store at all in YLS/CMI assessment results, as one noted: “Typically, when I receive a case, the YLS isn’t something that I would typically turn to for help.”

Finally, both survey scores (Figure 2) and interviews suggest County A had probably the strongest overall commitment to using and applying YLS/CMI assessments, and County B probably the weakest. Thus, there are some statistically significant differences in relevant measures which showed some sharp contrasts between these two counties (Figure 2). We also heard probation officers in County B most often play down the YLS/CMI as a basis for decision-making in their work. Even then, however, it is notable that some officers in County B highlighted a recent shift towards greater use of the YLS/CMI in decision-making, following renewed management emphasis on applying the instrument in case planning.

Applying the YLS/CMI in routine supervision

Local policies often encouraged officers to address YLS/CMI-defined criminogenic needs in their regular interactions with clients—something we were able to check through our observations. Evidence, in practice, was mixed. Across 112 observed routine supervision contacts (excluding assessment events), officers mentioned the YLS/CMI with their clients in just one in 10 occasions. However, in two in five encounters, they made some reference either to the YLS/CMI, a case plan document, or client goals—probably a more meaningful measure of the YLS/CMI’s influence (these frequencies seem lower than survey measures in Figure 2, however). Officers also discussed topics aligned with YLS/CMI needs in almost all observed encounters, outweighing non-criminogenic needs areas (such as mental health, physical health, or housing). This discussion was, however, uneven. Family and education were routinely emphasized (discussed about half and three quarters of the time, respectively), while leisure/recreation, personality/behavior, and attitude/orientation were discussed less frequently (respectively about one time in five), suggesting officers were inconsistent in their attention to different criminogenic domains. More generally, just over half of routine officer-client supervision encounters had some kind of rehabilitative orientation (about six in every 10), often seeming to help address criminogenic needs, through problem-solving, client reflection, or work on client strategies or skills.

Local policies also variously required the sharing of YLS/CMI assessment results with system stakeholders. In the survey (Figure 2) officers indicated routinely discussing the YLS/CMI in court review and disposition hearings, while in the 69 observations of such hearings (that were not short adjournments), officers referenced the YLS/CMI about a third of the time. However, across 21 observed meetings between officers and clients in placement facilities, we never saw officers discuss the YLS/CMI with service staff (though only five occasions included any extended conversations with these staff).

While the small numbers of observations in individual counties can only be suggestive, they show some variations, with Counties A and B again seeming to present stronger and weaker examples. In observed routine officer-client supervision contacts, County A saw frequent references to the YLS/CMI, case plan or goals (9 out of 20 contacts—a similar prevalence to Counties C and D), most often saw supervision contacts take a rehabilitative orientation (16 out of 20 encounters), and most often saw officers reference the YLS/CMI in review or dispositional court hearings (on 14 out of 18 occasions). In County B, officers were least likely to make references to the YLS/CMI, case plan, or goals in observed supervision contacts (in eight out of 24 contacts), or to take a rehabilitative orientation in their client contacts (in nine out of 24 contacts). Moreover, in observed dispositional and review hearings in County B, the YLS/CMI was rarely referenced. Survey results (Figure 2) also show a statistically significant difference in verbally referencing the YLS/CMI in court, again with officers in Counties A and B the most and least likely to report this; however, there were no statistically significant variations in officers’ self-reported references to the YLS/CMI during client-officer meetings.

Explaining Divergent Implementation Outcomes

Our analysis suggests that counties had developed concrete policies to guide the use of the YLS/CMI, and that staff were broadly supportive of the tool, regularly filled it out when required, and tended to apply it in decision-making and other activities. However, we also saw important variations. Counties A and B, in particular, probably most clearly exemplified stronger and weaker overall patterns of implementation, with counties C, D, and E seeming to fall somewhere in-between. Moreover, even within counties, patterns were not consistent, given variations across individual officers. Drawing on the accounts given by interviewees, and comparisons across counties and subjects, we offer some hypotheses about processes that may help account for these variations.

Office leadership and climate

Counties’ embrace of the YLS/CMI and other juvenile justice reforms seemed profoundly related to county leadership, which in turn related to broader aspects of office climate. According to one state reform leader, the counties “out in front [on reform]…my perception is that they were doing pretty good work anyway. So they probably recruited the right type of people, had the right type of culture…[and] they were ready for this and primed for these types of changes.”

Variations across case study counties were consistent with this idea. Thus, County A had had a stable chief who had been centrally involved in the state leadership, and whom colleagues recognized as an energetic champion of reform. County B, meanwhile, had for several years experienced leadership providing only lukewarm support for the YLS/CMI and related juvenile justice reforms. However, reform efforts in this county had since been reenergized under a new chief seeking to reboot this agenda.

There was also evidence that counties had begun implementation from different states of readiness. In County D, for example, where the YLS/CMI replaced an existing RNA, officers who discussed the earlier assessment tool saw the YLS/CMI as a welcome improvement. This stood in contrast to other counties, where the initial rollout of the YLS/CMI was often met with more resistance.

Structural characteristics of offices may also have been relevant. The largest office (County C) seemed to face some distinct challenges winning support and buy-in for reform, apparently owing to the greater physical (and social) distance between frontline staff and office leaders. Working arrangements in County B, where officers lacked a permanent desk and worked mostly from the field, may have impeded reform efforts because of reduced staff cohesion here. Interestingly, officer workload did not clearly map onto variations in implementation successes, with average dispositions per officer higher in County A than B, despite the former’s stronger patterns of implementation (Table 1).

Implementation and quality assurance strategies

Implementation and quality assurance strategies seemed to have shaped implementation outcomes, in part linked to formal policies in local counties. For example, in County A, where we saw most frequent reference to the YLS/CMI in observed court hearings, there was an explicit and unique policy requiring the YLS/CMI assessment results always to be discussed in these hearings. Or in County D, which had a written policy requiring a “placement meeting. . .to determine which placement facility best meets the identified needs of the clients as well as the YLS/CMI areas to target. . .,” we observed an example of such a meeting, in which multiple probation officers were gathered in a conference room to discuss a youth’s placement. Fieldnotes recorded this conversation, which highlighted how “the juvenile has been doing well at home and for the most part was doing well in school….[and] had far less issues with his mother.” While the YLS/CMI was not explicitly invoked, criminogenic needs were part of this conversation.

The strategies county leaders had used to message reforms and engage staff in reform activities may also have mattered. There was some evidence that resistance to reforms was less when they were meaningfully explained and made relevant to officers’ professional goals. One county leader noted:

The big feedback I got a lot a lot of times was. . . . “it was never explained. . .nobody ever said it to me. . .where it’s about the kid”. It was kinda like brought to them [by prior administrators]. . . . “I don’t believe in this stuff either, but this is what we’re doing”. . .We are. . .repairing some of that now. . .it’s a unified message coming out. . . . “Are you invested in changing kids and making the community safer? . . .You are? Good. Let me talk about the ways you can enhance your skill sets. . .”. . .The YLS is a big piece of that.

It is also plausible that the direct involvement of officers in the development and delivery of new policies and programs was associated with greater buy-in. This dynamic seemed relevant in County A, where officers were often involved directly in the delivery of client group services, and shaped ongoing reforms through committee involvement and other kinds of input. It was even used intentionally to try to bring along more resistant officers, as one subject described: “When our department has had those resistant PO’s…[we] put them on a committee then to have an opportunity to make a change.”

Specific quality assurance strategies also appeared significant. All counties required some form of supervisory involvement and sign-off of key YLS/CMI-related tasks and documents, and all participated in booster trainings in which officers’ scoring of the YLS/CMI was assessed. Additionally, two counties (A and D) had written policies requiring regular supervisory reviews focused on ongoing YLS/CMI alignment with case management (and County C had a more weakly stated version of this). Differences in policies, but also differences in supervisors’ style within counties, seemed to relate to variations in how the YLS/CMI was used and applied in supervision and case management. Thus, some interviewees described examples of a more bureaucratic approach to supervisory monitoring, which prioritized the timely completion of paperwork over officer practices and decision-making. As indicated by one supervisor: “They have given me the case plan, because we need a supervisor signature. I then…check it off that they did it.” The supervisor subsequently observed: “I don’t know if they are keeping track of it or if they even come back to it. . .” By contrast, other interviewees described a more substantive approach, in which supervision and case management practices were also assessed. The situation in County A stood out as a particularly strong example, as one officer described: “Supervisors are constantly reviewing caseloads and. . . want to know what’s the top criminogenic need . . .and what you’re doing to address it.”

Officer orientation

A number of dynamics had apparently shaped patterns of staff support for the YLS/CMI and other reforms. One was the recruitment of younger staff who were often very open to the new ways of working, as one office leader said: “The new staff coming in, they gobble that stuff. They love it.” More complicated was the situation of longer-serving veteran staff who had begun their careers significantly prior to reforms. While a few had apparently chosen to retire or move jobs when confronted with reforms, others had stuck around. Some of these, it seemed, remained resistant even after several years, as one officer leader noted: “there are certain staff here that just do things a certain way for years and they are just going to do things the same way, no matter. . .” There were, however, a substantial number of veteran staff who, while in some cases initially resistant, had adapted and come to see the value of reforms. As one officer described: “15 years of doing almost the same thing. . . .I was seething, when all this stuff was happening. . . .It all changed. . . .Now I’m [helping lead a piece of the county’s work to implement reforms]. . . .I’m sure the research will show me that arrests rates. . . .and placement rates are down. . . .and those are all good things.”

Stakeholder context

Successful implementation outcomes seemed to rely, to some extent, on support from juvenile court stakeholders. As one state reform leader remarked: “the places where we ran into problems. . .it was always some key person was left out and wasn’t buying in and they were digging their heels in. . .it could have been DA’s, public defenders. . .or a judge didn’t buy it. . .” Meanwhile, more supportive relationships could provide implementation leverage. One probation leader described how he had used the judge to try to influence his own county probation officer colleagues, at a time when the chief was not actively supporting the YLS/CMI reforms: “we eventually had to go to the judge to get her to say [to our staff] ‘yes, you’re going to do this and yes I support this.’”

Another issue concerns the range and quality of services available to youth, which granted officers flexibility in responding to YLS/CMI assessments. This could be shaped by probation leaders’ efforts to cultivate and develop services in their communities, for example through quality assurance, outreach to new providers, or developing in-house programs. The diversity of programs was particularly well developed in County A, as one officer observed: “We have so much variety. . .anything identified in the YLS can be addressed either through one of our [internal] programs, or looking for a service provider to address it.”

Discussion

Despite positive examples of RNA implementation in short-term pilot follow-up studies (Vincent et al., 2016; Vincent, Paiva-Salisbury, et al., 2012; Young et al., 2006), research in routine settings indicates RNAs are regularly underutilized among frontline practitioners (Miller & Maloney, 2013; Viglione et al., 2015). This study sought to examine implementation patterns several years after a well-planned, energetic, and collaborative statewide effort to implement the YLS/CMI, and to explore reasons for variations.

The statewide effort had led to concrete impacts on local county operations. All had explicit policies to complete and apply the YLS/CMI, thus supporting risk, need, and general responsivity principles (though with apparently less attention to specific responsivity principles). Findings moreover suggest a substantial—although not always consistent—level of practitioner adherence to these broad policies and principles. However—and in line with literature on “street-level-bureaucrats” (Lipsky, 1980)—they did not rigidly prescribe practitioners’ behavior, and we saw officer discretion applied in deciding how YLS/CMI assessments would be conducted, scored, and incorporated into decision-making and practice. This discretion was shaped by broader concerns, such as the character of clients’ offenses, client accountability, individualized professional judgments, working knowledge of clients, and non-criminogenic client needs.

We also observed substantial variability in levels of implementation, both between and within counties. Evidence suggests that strong local office leadership and climate, strategies to support implementation and quality assurance (in part underscored by the formulation of relevant policies), probation officers’ support for reforms (shaped by recruitment, staff turnover, and effective staff engagement), and a strong stakeholder environment (including diverse youth services), all plausibly contributed to implementation outcomes. Based as they are on a case study approach, however, these represent hypotheses that need further testing.

We should also note some limitations to our methods. We were particularly reliant on self-report information, provided through interviews and surveys, which may have been subject to biases. Additionally, there may have been some selection bias in the officers we were able to observe, some behavioral adjustments made by officers being observed, and some recall biases in the observational fieldnotes we wrote. Notwithstanding, our different data sources did tend to reinforce one another, giving us greater confidence in the conclusions we have drawn. Perhaps a more profound limitation arose from the timing of our data collection. Historical processes shaping early stages of implementation may have been less apparent several years after the introduction of the YLS/CMI, when our research was conducted. This may have led to an incomplete picture of the implementation process.

Conclusions

Overall, our findings are largely consistent with implementation science literature. They highlight factors facilitating implementation that are similar to the “drivers” described by Fixsen and colleagues’ (Bertram et al., 2015; Fixsen et al., 2005), including those related to leadership and organizational performance monitoring. They show the relevance of external and internal contexts, and the importance of “implementation climate” (including organizational readiness and leadership) highlighted by Damschroder et al. (2009). The gradual unfolding process of YLS/CMI implementation over time also echoes Fixsen et al.’ (2005; Bertram et al., 2015) description of a sequential, staged model of implementation. They also have synergy with prior RNA implementation case studies, which emphasize how leadership, staff engagement, policy development, and partnership with juvenile justice stakeholders promote implementation success (Ferguson, 2002; Vincent, Guy, et al., 2012; Young et al., 2006).

There were also challenges, recognized in the broader implementation literature, that seemed particularly relevant to the YLS/CMI and likely other RNAs. Notable is their susceptibility to “paper implementation” (Fixsen et al., 2005) which involves the bureaucratic demonstration of compliance rather than substantive changes in behaviors. Some officers completed or received the YLS/CMI as required, but did not always apply it in their decision-making and supervision. Similarly, quality assurance efforts often focused on the “paper trail” of RNA compliance rather than the application of results in substantive officer practice. Given that RNAs are, in essence, documents, it is not surprising that they are susceptible to this problem.

The research also suggests some tensions with established implementation science. One concerns the distinction between “inner” and “outer” settings often highlighted (e.g. Damschroder et al., 2009; Greenhalgh et al., 2004). While county probation offices can be seen as inner contexts for reform, and state organizations, peer county probation offices, and local court and service provider stakeholders part of an outer setting, in practice these lines were blurred. In some cases, internal county staff and participants were simultaneously state reform advocates and leaders participating in state level organizations and committees. Furthermore, juvenile justice stakeholders (such as judges, attorneys and service providers) were important local reform partners but also had direct responsibility for the youth that probation officers supervised. These characteristics undermined a clear boundary between inner and outer reform contexts.

Additionally, we did not see a tidy distinction between initial implementation and a “full implementation” (Bertram et al., 2015) or “sustainment” phase (e.g. Aarons et al., 2011), as highlighted in the literature. After several years using the YLS/CMI in Pennsylvania, state organizations and local county probation departments continued to innovate and adapt YLS/CMI policies and practices. Perhaps the complexity of adapting varied practices to adhere to risk, need and responsivity principles means that RNA implementation is always, to some extent, a work in progress, requiring ongoing adaptation, trial and error.

The findings we present highlight some important successes and challenges in the long-term progress and sustainability of YLS/CMI implementation in Pennsylvania, and detail the processes that seem to shape these outcomes. We hope our insights contribute to an implementation science tailored to the application of RNAs. We also hope they provide a useful reference point for those introducing RNAs into correctional settings and promoting adherence to RNAs in routine practice.

Footnotes

Acknowledgements

The authors would like to thank all the partners and participants in this project, including all the Pennsylvania juvenile probation officers and supporting staff who gave their time to assist with this research, the Pennsylvania Juvenile Court Judges’ Commission, and the Pennsylvania Council of Chief Juvenile Probation Officers.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The second author of this study, Dr. Carrie Maloney, has been paid as a consultant by the PA Association of Chief Justice Probation Officers and the PA Juvenile Court Judges Commission for work focused on the development of a juvenile detention screening instrument. This has not involved work directly related to the YLS/CMI, the key focus of this article.

Funding

This project was supported by Award No. 2015-R2-CX-0015, awarded by the National Institute of Justice, Office of Justice Programs, US. Department of Justice. The opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect those of the Department of Justice.

Author Biographies

Joel Miller, an associate professor at Rutgers’ School of Criminal Justice, has seen employment in the UK Home Office, the University of Malaga in Spain, and the Vera Institute of Justice, New York. His research focuses on community corrections, police accountability, and environmental criminology.

Carrie Maloney is an associate professor at East Stroudsburg University of PA. Previously employed by the New Jersey Juvenile Justice Commission, Dr. Maloney’s research focuses on risk-assessment and classification in juvenile justice, accountability of justice system practitioners, and the implementation of evidence-based practices.

Courtney S. Harding works for the Pennsylvania Office of the Attorney General and has a PhD from Temple University. She is particularly interested in the relationship between homelessness, poor health, substance abuse, and justice system involvement, and the policy and service implications for “frequent utilizer” groups at this intersection.

Krissinda Palmer has worked in several juvenile justice settings, including youth detention centers and residential treatment facilities for juvenile offenders. Her current research focuses on violence against women, the sociology of law, and the treatment of juvenile offenders with intellectual disabilities.

Jesse Brey is pursuing a doctoral degree from Temple University in criminal justice. His research focuses on criminal and juvenile justice policies, particularly in the area of corrections, and the ways these impact inequalities in urban communities.

Juan R. Sandoval is a PhD student in the Department of Criminology, Law & Society at the University of California, Irvine. He does research on community supervision, criminal record data accuracy, and rural communities. He also has experience serving as an adult probation/parole officer in the state of New Mexico.

References

Aarons

G. A.

Hurlburt

Horwitz

S. M.

(2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23.

Andrews

D. A.

Bonta

Wormith

J. S.

(2011). The risk-need-responsivity (RNR) model: Does adding the good lives model contribute to effective crime prevention? Criminal Justice and Behavior, 38(7), 735–755.

Bertram

R. M.

Blase

K. A.

Fixsen

D. L.

(2015). Improving programs and outcomes: Implementation frameworks and organization change. Research on Social Work Practice, 25(4), 477–487.

Bonta

(1996). Risk-needs: Assessment and treatment. In Harland

A. T.

(Ed.), Choosing correctional options that work: Defining the demand and evaluating the supply (pp. 18–32). Sage.

Bonta

Andrews

(2010). Viewing offender assessment and rehabilitation through the lens of the risk-needs-responsivity model. In McNeill

Raynor

Trotter

(Eds.), Offender supervision: New directions in theory, research and practice (pp. 19–40). Routledge.

Brodkin

E. Z.

(2011). Putting street-level organizations first: New directions for social policy and management research. Journal of Public Administration Research and Theory, 21(Suppl. 2), 199–201.

Damschroder

L. J.

Aron

D. C.

Keith

R. E.

Kirsh

S. R.

Alexander

J. A.

Lowery

J. C.

(2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(1), 50.

Dowden

Andrews

D. A.

(2004). The importance of staff practices in delivering effective correctional treatment: A meta-analysis of core correctional practices. International Journal of Offender Therapy and Comparative Criminology, 48, 203–214.

Emerson

R. M.

Fretz

R. I.

Shaw

L. L.

(2011). Writing ethnographic fieldnotes. University of Chicago Press.

10.

Farrell

J. L.

Young

D. W.

Taxman

F. S.

(2011). Effects of organizational factors on use of juvenile supervision practices. Criminal Justice and Behavior, 38(6), 565–583.

11.

Ferguson

J. L.

(2002). Putting the “what works” research into practice an organizational perspective. Criminal Justice and Behavior, 29(4), 472–492.

12.

Fixsen

Naoom

Blase

Friedman

Wallace

(2005). Implementation research: A synthesis of the research. University of South Florida.

13.

Gendreau

Little

Goggin

(1996). A meta-analysis of the predictors of adult offender recidivism: What works!. Criminology, 34(4), 575–608.

14.

Gottfredson

S. D.

Moriarty

L. J.

(2006). Statistical risk assessment: Old problems and new applications. Crime & Delinquency, 52, 178–200.

15.

Greenhalgh

Robert

Macfarlane

Bate

Kyriakidou

(2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4), 581–629.

16.

Haas

S. M.

DeTardo-Bora

K. A.

(2009). Inmate reentry and the utility of the LSI-R in case planning. Corrections Compendium, 34(1), 11–16.

17.

Hannah-Moffat

Maurutto

Turnbull

(2009). Negotiated risk: Actuarial illusions and discretion in probation. Canadian Journal of Law and Society, 24(3), 391–409.

18.

Hoge

R. D.

Andrews

D. A.

(2002). Youth level of service/case management inventory: User’s manual. Multi-Health Systems.

19.

Latessa

Smith

Lemke

Makarios

Lowenkamp

(2009). Creation and validation of the Ohio risk assessment system: Final report. University of Cincinnati.

20.

Lipsky

(1980). Street-level bureaucracy: Dilemmas of the individual in public services. Russell Sage Foundation.

21.

Lowenkamp

C. T.

Latessa

E. J.

Holsinger

A. M.

(2006). The risk principle in action: What have we learned from 13,676 offenders and 97 correctional programs? Crime & Delinquency, 52(1), 77–93.

22.

Luong

Wormith

J. S.

(2011). Applying risk/need assessment to probation practice and its impact on the recidivism of young offenders. Criminal Justice and Behavior, 38(12), 1177–1199.

23.

Lynch

(1998). Waste managers? The new penology, crime fighting, and parole agent identity. Law and Society Review, 32(4), 839–870.

24.

Miller

Maloney

(2013). Practitioner compliance with risk/needs assessment tools: A theoretical and empirical assessment. Criminal Justice & Behavior, 40, 716–736.

25.

Miller

Maloney

(2015). Organizational determinants of practitioner compliance with a juvenile risk-needs assessment tool. Paper presented at the American Society of Criminology annual meeting, Washington, DC.

26.

Miller

Maloney

(2020). Operationalizing risk, need, and responsivity principles in local policy: Lessons from five county juvenile probation departments. The Prison Journal, 100(1), 49–73.

27.

Vieira

Skilling

Peterson-Badali

(2009). Matching court-ordered services with youths’ clinically identified treatment needs: Predicting treatment success for young offenders. Criminal Justice and Behavior, 36, 385–401.

28.

Viglione

(2017). Street-level decision making acceptability, feasibility, and use of evidence-based practices in adult probation. Criminal Justice and Behavior, 44, 1356–1381.

29.

Viglione

Rudes

D. S.

Taxman

F. S.

(2015). Misalignment in supervision: Implementing risk/needs assessment instruments in probation. Criminal Justice and Behavior, 42(3), 263–285.

30.

Vincent

G. M.

Guy

L. S.

Grisso

(2012). Risk assessment in juvenile justice: A guidebook for implementation. MacArthur Foundation.

31.

Vincent

G. M.

Guy

L. S.

Perrault

R. T.

Gershenson

(2016). Risk assessment matters, but only when implemented well: A multisite study in juvenile probation. Law and Human Behavior, 40(6), 683.

32.

Vincent

G. M.

Paiva-Salisbury

M. L.

Cook

N. E.

Guy

L. S.

Perrault

R. T.

(2012). Impact of risk/needs assessment on juvenile probation officers’ decision making: Importance of implementation. Psychology, Public Policy, and Law, 18(4), 549.

33.

Vose

Cullen

F. T.

Smith

(2008). The empirical status of the Level of Service Inventory. Federal Probation, 72, 22–29.

34.

Whiteacre

(2004). Case manager experiences with the LSI-R at a federal community corrections center. Corrections Compendium, 29(6), 1–5.

35.

Young

Moline

Farrell

Bierie

(2006). Best implementation practices: Disseminating new assessment technologies in a juvenile justice agency. Crime & Delinquency, 52(1), 135–158.