Sustainability of Evidence-Based Practices: Risk-Need-Responsivity in Probation 7 Years Later

Abstract

This quasi-experimental, pre-post study investigated the sustainability of risk-needs assessment (RNA) and the risk-need-responsivity (RNR) practices in five juvenile probation offices across two states 7 years after rigorous implementation. The study evaluated adherence to practices and facilitators of sustainability via interviews with leadership (N = 10) and probation officers (N = 84) and a seventh-year cohort of youth propensity score matched to a cohort from the first year of implementation (n = 730 per state cohort). Probation officers in both states reported relatively high adherence to RNR-based practices, and one state significantly improved adherence to its RNA administration policies and the risk principle. Adherence to the risk principle in placement and service referral decisions regressed in the other state. Essential ingredients for sustainment included maintaining staff training and monitoring, evolving RNR practices through the adoption of complementary evidence-based practices, and having court personnel who enable best practices.

Keywords

juvenile justice risk-needs assessment risk-need-responsivity sustainability implementation

State and local juvenile justice systems have experienced substantial reform over the last 15 years resulting in significant reductions in juvenile arrest, adjudication, and incarceration (Hockenberry & Puzzanchera, 2019) and increased adoption of various evidence-based practices (Juvenile Justice Geography, Policy, Practice & Statistics, 2021) to reduce recidivism. One of the most well-studied and widely recommended evidence-based practices in justice settings (National Research Council, 2013) is the risk-need-responsivity (RNR) model (Andrews & Bonta, 2010). The RNR framework has three primary principles. The risk principle suggests the system should reserve intensive interventions (e.g., confinement, more services) for the highest risk cases. The need principle stipulates case management practices should provide programming that targets individuals’ dynamic risk factors that influence their offending (criminogenic needs). The responsivity principle indicates programming should consider how well the styles and modes of programming are matched to individuals’ attributes that may affect treatment response. For the RNR approach to be fully realized, it requires agencies to adopt a valid risk-needs assessment (RNA) to accurately identify individuals’ risk for reoffending and their key dynamic risk factors (e.g., impulsivity and substance use) to target with services.

Despite the widespread adoption of RNAs and RNR, there are many challenges faced by systems when implementing these practices, including how to translate information from RNAs into case management decisions. For example, probation studies have shown that case plans are often not linked to the results of RNAs as the RNR model prescribes (e.g., Miller & Maloney, 2013; Viglione et al., 2015). Moreover, probation officers may misunderstand how to use the practices or simply do not buy-in to their value (Ingel et al., 2022). However, translating RNAs and the RNR approach into probation practice requires buy-in from not only probation officers (POs) but also other key stakeholders throughout a jurisdiction (e.g., judges, attorneys, and probation leadership). The representatives of all groups must be engaged in the development of strong policies and implementation procedures (Vincent, Guy, & Grisso, 2012) to cultivate stakeholder buy-in. Studies of RNAs or RNR use in criminal-legal settings that incorporated strong stakeholder-engagement and guidance from researchers report more positive findings (e.g., Viljoen et al., 2019), than those without this engagement.

While strong implementation-related studies of RNA and RNR are slowly increasing, a critical next step is to investigate whether the implementation of these practices is sustained over time. To date, there are very few studies on the sustainability of evidence-based practices or reform efforts of any sort in legal settings (e.g., Miller & Palmer, 2020). Moreover, most studies reporting limited adherence to RNR practices do not disentangle whether this outcome was due to an initial implementation problem versus a sustainability problem. To address this gap, the current study gathered an additional wave of data from juvenile probation offices in two states 7 years after the comprehensive implementation of RNA and RNR-based practices.

Implementation of Fidelity and Sustainability

With the substantial uptake of evidence-based practice reforms in justice settings, it is imperative for criminal-legal research to not measure impacts (such as recidivism) without also measuring fidelity. Implementation strategies are the methods used to enhance the adoption, implementation, and sustainability of innovations (e.g., RNR, assessment strategies, and new interventions) in practice settings (Proctor et al., 2013). To achieve improved outcomes, new practices need to be paired with effective implementation strategies (e.g., revising written procedures, obtaining stakeholder buy-in, and designing decision-support tools) and delivered in a system that supports the new practice (Fixsen et al., 2019). Implementation fidelity is “the degree to which an intervention was implemented as prescribed in the original protocol or as it was intended by the developers” (Proctor et al., 2011, p. 69). A core component of implementation fidelity is practitioners’ (in this case, judges, POs, probation leadership) adherence to new practices (Carroll et al., 2007), which relates to both their behaviors and their attitudes.

Sustainability refers to whether an evidence-based practice has been incorporated into routine practice over time (Moullin et al., 2020). Trajectories of reform efforts can take many forms, with a common form being to simply adopt the reform in name but never actually relate it functionally to practice (Taylor, 2005). The adoption of RNAs in legal settings is no exception. Several studies of POs indicate they may complete their office’s RNA routinely but then report little to no use of the tool in decision-making (Haas & DeTardo-Bora, 2009; Miller & Maloney, 2013). The handful of observational and cross-sectional studies have shown there is little evidence of POs following RNR principles, such that low-risk youth may still get multiple services (Fabelo et al., 2015) or POs spend more supervision time on surveillance-related issues than on addressing needs (Viglione, 2019). When fidelity is lacking, agencies are prone to stop using new practices within 2 years (Loman et al., 2010). Sustained trajectories of reform, on the other hand, can take different forms. One involves the new practice overtaking whatever preceded it so completely that it is institutionalized as the status quo (sustained implementation). Another trajectory is to achieve a dynamic equilibrium by adjusting and adapting evidence-based practice to a continually changing environment (innovative; Taylor, 2005).

Studies of sustainability should examine multiple constructs. In their synthesis of the literature, Moore et al. (2017) defined sustainability as encompassing five constructs: (1) for a specific period of time, (2) the program or strategy continues to be delivered, and/or (3) the individual (e.g., practitioner) change in behavior necessary for the new strategy has maintained; (4) the strategy and practitioners’ behavior change may evolve or adapt while (5) continuing to produce benefits for individuals/systems. In addition, studies of sustainability must consider contextual factors (Moullin et al., 2020; Stirman et al., 2012) that can act as facilitators or barriers to new practices (e.g., sociopolitical climate, practices of agency partners, leadership; Rocque et al., 2014). In the case of RNR, one must remember that POs do not operate in a vacuum. They operate within a context that either enables their ability to follow RNR (e.g., courts permit time for RNAs to be completed pre-disposition to inform diversion or service decisions) or hinders it (e.g., judges set all expectations in court orders before an RNA).

The Current Study

Implementation science underscores the importance of long-term studies that follow the implementation of RNAs and RNR to assess whether and how practices are sustained.

The current study is from the Long-Term RNR Sustainability Study, a quasi-experimental study of the sustainability of RNA and RNR practices in five county-run probation offices in two states 7 years after their RNA and RNR implementation. Two probation offices were in a Southern state, which implemented the Structured Assessment of Violence Risk in Youth (SAVRY; Borum et al., 2006), and three were in a Northeastern state, which implemented the Youth Level of Service/Case Management Inventory (YLS/CMI; Hoge & Andrews, 2011). All offices originally followed a researcher-guided, stakeholder-engaged, implementation strategy (Vincent, Guy, & Grisso, 2012) for their RNA and RNR. The current study compares their seventh-year outcomes (seventh-year cohort) to outcomes after their first year of implementation in 2009 and 2010 (first-year cohort) as reported in the Risk-Needs Assessment in Juvenile Probation Implementation Study [RNAJP] (Vincent et al., 2016; Vincent, Paiva-Salisbury, et al., 2012).

In 2009 and 2010, the five juvenile probation offices implemented policies based on RNR procedures. The policies of interest to the current study include (a) administration of a valid RNA prior to disposition for every youth referred or adjudicated, with a few exceptions, and (b) alignment of multiple decisions with the risk principle (e.g., disposition and placement recommendations, number of service referrals) such that low-risk youth received little intervention and high-risk youth received intensive intervention. In a quasi-experimental pre-post design measuring outcomes up to 18 months after implementation (first-year cohort), the RNAJP study reported strong adherence to RNA administration policies in three of five probation offices (Vincent et al., 2016). As a result of high adherence, these effective implementers had significant improvement in at least three of the five outcomes studied (reduced placement rates, less severe dispositions, supervision levels and number of services aligned with risk levels, and recidivism), while the remaining two (one per state) with poor adherence had little change in case processing outcomes. However, the initially poor implementers may have achieved better outcomes over time. It can take 2 to 4 years for agencies’ implementation efforts to have an impact on the people served (Fixsen et al., 2019).

The current study used data from the five probation offices to investigate the sustainability of RNR practices. The researchers defined sustainability using the first four constructs identified by Moore et al. (2017), which pertain to implementation outcomes as opposed to impacts (fifth construct). For the first construct (period of time), we selected a duration of 7 years based on findings from other human service agencies indicating at least 7 years was the average duration for sustained implementers (Loman et al., 2010). For the second construct (strategies continue to be delivered), we conducted interviews with probation office leadership to examine whether the RNA and RNR-practices continued to be delivered over time. For the third construct, we examined sustained practitioner behavior change using three methods: (1) seventh-year POs’ self-reported adherence to RNR in their decisions, (2) first- vs. seventh-year cohorts’ adherence to RNA administration policies (i.e., did every youth receive an RNA at the prescribed time?), and (3) first- vs. seventh-year cohorts’ adherence to the risk principle in placements and service referrals. Comparisons between the first- and seventh-year cohorts quantified whether practices regressed (worse than the first year), sustained (same high adherence as the first year), or improved (better than the first year) over time. Information from leadership interviews documented whether RNR practices evolved or adapted (fourth construct) and identified facilitators and barriers.

The primary research questions were: (1) whether both states continued to deliver their RNA and RNR procedures seven years after implementation despite both having had one initially poor implementer site and (2) whether both states sustained or improved practitioner behavior change as indicated by the three adherence measures. We hypothesized both states would continue to deliver their RNA and RNR-based procedures and would not differ significantly in measures of practitioner behavior change.

Method

Sites (Probation Offices)

The researchers conducted this study in the five RNAJP study county-run probation offices; three in a Northeastern (NE) state and two in a Southern (SO) state. The sites originally were selected because they matched between states in terms of their rural versus urban location, volume of youth, and base rates of any out-of-home placements. All sites implemented the RNA and RNR practices office-wide in 2009 except Site 2 in the Northeastern state (NE Site 2) initially piloted the YLS/CMI with only select POs, and NE Site 3 piloted the YLS/CMI in just one probation unit to start. The current study procedures accommodated for these exceptions by (1) not conducting a cohort comparison of adherence to the RNA administration policy in NE Site 2 (only youth cases receiving a YLS/CMI were included in the first-year cohort, giving the impression of 100% adherence) and (2) the seventh-year cohort in NE Site 3 comprised only youth cases seen in the same probation unit as the first year.

Probation Leadership and Officer Samples

The first author conducted one in-person interview per site with the chief probation officer (or equivalent) and manager (N = 10) to document system or programmatic changes occurring over the 7-year period. All but one of these leaders had been working in the office when the RNA was first implemented. Research staff conducted phone interviews with 84 (Southern = 38, Northeastern = 46) of the 87 probation officers and supervisors from the five sites who were responsible for conducting, using, or supervising the use of RNAs. The missing three POs were on leave. Most POs interviewed (91.67%) were responsible for conducting initial RNAs with youth at the time of the referral or adjudication, while the others were responsible for only reassessments or were supervisors who reviewed but did not conduct RNAs. Collectively, the PO sample had an average of 7.15 years of experience in their current position, 38 had been working in the office since the first-year cohort, and 10 POs had only been hired in the past year.

Table 1 reports the characteristics of the full sample of POs and by state. POs in the Northeastern state reported having had significantly more training (includes booster training) on the YLS/CMI (M = 10.83, SD = 7.04) than POs in the Southern state did on the SAVRY (M = 2.84, SD = 2.33), had been working in their current position for significantly longer, and were older, on average. There was not a significant state difference in the number of POs who had been hired after the first-year cohort was gathered.

Table 1:

Probation Officer Characteristics by State (N =84)

	Overall(N = 84)		Northeastern(n = 46)		Southern(n = 38)		Statistic (df)	p	d/OR	95% CI
	n	%	n	%	n	%	Statistic (df)	p	d/OR	95% CI
Sample
# POs interviewed (% all POs)	84	96.55	46	95.83	38	97.44
# POs interviewed for cohort #1	38	45.24	23	50.00	15	39.50	χ² (1, 84) = .93	.34
Current position
Supervisor	13	15.47	7	15.22	6	15.79
Field or intake PO	61	72.62	34	73.91	27	71.05
Other	10	11.90	5	10.87	5	13.16
# POs who conduct initial RNAs	77	91.67	45	97.83	32	84.21
# New POs (< 12 months)	9	10.70	5	10.90	4	10.50	χ² (1, 84) = 0.03	.96
Experience
Months in current position (M, SD)		85.84 (87.71)		103.82 (95.93)		63.97 (71.88)	t(70) = 2.15	.03	0.41
Months working in JJ (M, SD)		180.34 (108.39)		204.22 (105.30)		151.45 (106.31)	t(82) = 2.28	.02	0.50
RNA trainings received (M, SD)		7.21(6.73)		10.83(7.04)		2.84(2.33)	t(56) = 7.22	<.001	1.52
Characteristics
Gender: Female	33	39.29	15	32.61	18	47.37	χ² (1, 84) = 1.90	.17	1.86	[0.77, 4.51]
Race/Ethnicity							χ² (1, 84) = 17.07	<.001	0.45^a	[0.27, 0.63]
Non-Latinx White	57	67.86	39	84.78	18	47.37
Non-Latinx Black	20	23.81	3	6.52	17	44.74
Other (e.g., Latinx, Asian)	7	8.33	4	8.69	3	7.89
Education							χ² (1, 84) = 0.06	.81	1.11	[0.46, 2.69]
Some college or bachelor’s degree	52	61.90	29	63.04	23	60.52
Master’s degree	32	38.09	17	36.96	15	39.47
Age (M, SD)		40.65 (9.56)		41.59(9.39)		39.53(9.77)	t(82) = 0.98	.33	0.21

Note. Significant differences are in bold. The reference group was Northeastern. CI = confidence interval; JJ = Juvenile Justice; N = Sample size; OR = Odds Ratio; % = Percentage of state sample; POs = probation officers; RNA = risk-needs assessment.

V reported for effect size.

Youth Sample

Researchers received administrative data from the 7^th-year cohort of youth using all continuous cases that were expected to receive an RNA according to each office’s policy (Northeastern state = youth referred to the court, Southern state = youth adjudicated, with a few office-specific policy exceptions) starting January 1^st, 2017, until a sufficient number of cases was obtained in each site (Southern n = 337; Northeastern n = 552). Researchers initially selected youth for inclusion in the 7^th-year cohort who matched youth in the pre-implementation cohort from the original RNAJP study using 1:1 nearest neighbor matching (without replacements) procedures to equate the cohorts. Propensity-score matching (PSM) is a technique commonly used in observational studies to reduce potential bias resulting from differences in relevant characteristics between control and treated groups (Rosenbaum & Rubin, 1983). Because the current study compares only the first-year and seventh-year cohorts, researchers evaluated the match between these two cohorts, within probation offices, using a full matching procedure (Stuart et al., 2011) to ensure these cohorts were balanced within sites. By making use of all individuals present in the dataset, a full matching method entails categorizing individuals into a series of matched sets or subclasses. Each set includes both treated and untreated individuals, ensuring the inclusion of at least one individual from each group. Variables were included in the matching procedure if they were demographic characteristics that were not matched naturally (e.g., gender was constant across years so did not require balancing) or were associated with out-of-home placements (e.g., living situation, substance use, and mental health history; Brookhart et al., 2006). Researchers conducted a series of bivariable comparisons to examine associations between each covariate and outcome, applying a p < .05 criterion for covariate inclusion in each site’s propensity score analysis (Schafer & Kang, 2008). To evaluate improvement in covariate balance, absolute standardized mean difference (ASMD) was assessed both before and after matching (Rubin, 2001) by examining the percentage balance improvement (i.e., reduction in imbalance) in the standardized mean difference. PSM results are included in the Supplemental Material Table SA available in the online version of this article.

The Southern sites showed similar covariate distributions between the treated and control groups (i.e., only 1 of the 20 total covariates across both sites had ASMD > 0.20), suggesting an improvement in fit following matching. However, the Northeastern sites still showed poor distributional balance with certain covariates (i.e., 8 of the 28 total covariates across all three sites had ASMD > 0.20). Eleven covariates showed a reduction in balance after matching, indicated by negative percent balance improvement estimates; however, most covariates still had an acceptable ASMD of ≤ 0.20. In part, the observed imbalance with certain covariates for the Northeastern sites was due to (a) the change in sampling in NE Site 3, leading to youth having less serious offense histories in the seventh-year cohort because they were assessed earlier in the process (intake) than in the first-year cohort (post-disposition), and( b) possible overall shifts in the characteristics of youth entering the system in the seventh-year (e.g., they had higher rates of behavioral health treatment than the first year). Consistent with acceptable threshold values, covariates that showed an ASMD above 0.20 following matching and were significantly related to a respective outcome were included in regression analyses to create double robustness and decrease residual bias (Schafer & Kang, 2008). Weights generated from the PSM procedure also were included in subsequent regression models.

The final matched samples included 297 youth per cohort in the Southern state and 433 per cohort in the Northeastern state. The matched cohorts were roughly 24% female, 50% Black, and 15.50 years of age, on average. Only 28% had a violent index offense and the average number of prior offenses was under one (see Supplemental Material Table SB available in the online version of this article).

Measures and Procedures

Risk-Needs Assessment Instruments

The Southern state implemented the SAVRY (Borum et al., 2006), a 24 risk-item factors and six protective factor-item instrument that follows the structured professional judgment (SPJ) approach. The SPJ approach involves raters (e.g., POs) providing a categorical risk rating of a youth’s overall risk for violence (i.e., low, moderate, high) based on their professional judgment following appraisal of the relevance of all item ratings. The Northeastern state implemented the YLS/CMI (Hoge & Andrews, 2006), a static/dynamic actuarial instrument. The YLS/CMI differs from the SAVRY in that risk level is determined based on total scores (a sum of 42 risk items) used to assign youths to an actuarial-based categorical risk level. Both instruments have comparable predictive validity for reoffending, and both significantly predict both violent and non-violent reoffending (Olver et al., 2009). Both instruments require the same information gathering and time to conduct, both had excellent inter-rater reliability among POs in the first-year cohort (Vincent, Guy, Fusco, & Gershenson, 2012), and there were no differences in their implementation fidelity or impact in the first-year cohort (Vincent et al., 2016).

Office Leadership Interview: Continued Delivery of Risk-Needs Assessment and RNR

In-person interviews with leadership at each probation office documented whether the RNAs and RNR continued to be delivered (Question 1) and system or programmatic changes that could have facilitated or prevented the sustainability of RNR-based practices. The audio-recorded interviews queried significant changes or events over the past 7 years in the office and in the state system in the following areas: RNA policies, probation department structure, staff resources, office or stakeholder agency leadership changes (e.g., prosecutor, defense attorney, court), staff training, adoption of new reform practices, and availability of services.

Probation Officer Interview: Adherence to RNA and RNR Principles

Consenting POs completed an interview with one of five masters-level research assistants. Interviewers asked POs questions to gauge adherence to RNR-based practices within three case management areas: Disposition recommendations, service referrals, and recommendations following probation violations. Each section started with an open-ended question to understand the factors involved in POs’ decision-making without prompting socially desirable responding (e.g., “What information do you consider in your disposition recommendations? What factors are most important?”). Interviewers audio-recorded, transcribed, and entered the responses into a database. The first author and one interviewer independently coded each of the responses to open-ended questions as “no mention of the RNA,” ‘mentioned the RNA directly’ (includes reference to the RNA’s semi-structured interview), or “mentioned the RNA indirectly” (mentioned specific dynamic risk factors but did not the RNA or its need areas by name). Final ratings were based on the consensus between coders.

Open-ended questions were followed by four direct questions within each case management area asking POs to rate their use of the SAVRY or the YLS/CMI on an 8-point scale (0 = never to 7 = always). These questions and rating scale were adapted from a national survey designed by Miller and Maloney (2013) and asked how often do they (1) make a recommendation that corresponds with the RNA risk level (or that targets the need areas identified by the RNA for services), (2) make a more restrictive recommendation than the RNA risk level indicates (or disregard need areas identified on the RNA), (3) make a less restrictive recommendation than the RNA risk level indicates (or target needs not identified on the RNA), and (4) make a recommendation without consulting the RNA. Interviewers asked for an example when POs gave responses of two or higher to questions 1 thru 4. Finally, POs were asked to rate how useful they found the RNA to be for the respective decision on an eight-point scale (0 = not at all useful to 7 = extremely useful).

Administrative Data: Adherence to RNA Administration Policies

The researchers used administrative data to record whether and when an RNA had been completed for each youth in both cohorts and whether the timing matched the policy. The NE site’s policies were to complete the YLS/CMI with all court-referred youth pre-adjudication (with a few site-specific exceptions, such as excluding diverted youth) unless the youth or family refused, in which case the YLS/CMI was to be completed within 10 days after an adjudication hearing. The SO site’s policies generally stipulated POs were to administer the SAVRY post-adjudication and pre-disposition. SO Site 1 did not require the SAVRY for deferred or unsupervised cases and permitted POs to administer the SAVRY within 10 days of disposition if the pre-disposition assessment was not possible. RNA policy adherence rates were calculated as the percentage of youth in each cohort who received an RNA at the time stipulated, divided by the number of youth cases for whom the RNA should have been completed as per the respective sites’ policy. The calculations allotted a 10-day window to account for date errors.

Administrative Data: Adherence to the Risk Principle

We evaluated adherence to the risk principle using administrative data from youth in the two cohorts who received the YLS/CMI or SAVRY to examine whether risk level was positively correlated with two case management outcomes: Restrictive placements and number of service referrals. Restrictive placements included secure (detention, correctional institutions) and non-secure residential facilities, excluding psychiatric or child welfare placements. Services were defined as any community-based or placement-related services aimed at treatment or rehabilitation (e.g., mentoring programs, Functional Family Therapy, counseling), and did not count accountability-related interventions such as community service and electronic monitoring. The study tracked service referrals rather than whether services were received because the referrals were less likely to confound PO and court decisions without service availability or refusals. Placements and service referrals were recorded until each youths’ case was closed, or to the end of the study follow-up period, whichever came first. The minimum follow-up periods were held constant for both the first- and seventh-year cohorts within each site, ranging from seven (SO Site 1) to 11 months (NE Site 1). However, average lengths of follow-up still varied by cohort, making it essential to control for follow-up days when comparing cohorts.

Data Analyses

To examine whether both states had sustained or improved practitioner behavior change (Question 2) along the three measures of adherence researchers conducted all analyses at the state-level, except adherence to RNA administration policies due to site-specific policy differences. To examine PO self-reported adherence in the seventh year, we calculated mean PO responses on the 8-point scale questions and the proportion that mentioned using their RNA for each area of decision-making according to researcher consensus ratings. To test the hypothesis of the absence of state-level differences in PO adherence to RNR, analyses first identified covariates to include in state comparisons by examining significant state differences in PO characteristics (using chi-squares and t-tests) and testing these in regression models to identify variables significantly associated with the self-report measures of RNR adherence. Next, researchers tested state differences using chi-squares for dichotomous consensus ratings (logistic regressions if covariates were present) and t-tests for POs’ eight-point scale ratings (Analyses of Covariance if covariates were present).

For the second measure of practitioner behavior change, chi-squares compared RNA policy adherence rates (number of youth receiving the RNA at the correct time/total number of new youth eligible to receive the RNA) between matched cohorts within sites. Rates were quantified as significantly increasing (improved), staying constant (sustained), or significantly decreasing (regressed). For explanatory purposes, we also calculated overall rates of completion of the RNA (number of cases that received an RNA/total number of youth eligible to receive an RNA), regardless of whether the RNA was completed at the time dictated by policy.

For the last measure of practitioner behavior change, researchers examined adherence to the risk principle within each cohort by determining whether risk level was positively correlated with youth ever receiving a restrictive placement (chi-squares) and their number of service referrals (analysis of variance; ANOVA). Placement and service data analyses excluded youth who were lost at follow-up due to transfer to adult court or moving out of state (n = 38) or who never received an RNA, and therefore, did not have a risk level (n = 198). Moderated hierarchical regressions (logistic for placement data and linear regression for service data) tested whether cohort membership (first year or seventh year) moderated the associations between risk level and the respective outcome. The first blocks included the PSM weights, days of follow-up (to account for variable follow-up lengths), and any unmatched variables from the PSM identified as covariates. The second block included risk level (0 = low, 1 = moderate, and 2 = high/very high) and cohort (0 = first year or 1 = seventh year) and the third block added an interaction term for risk level by cohort. Significant interaction terms indicated adherence either improved or regressed in the seventh year, depending on the direction of the term. Non-significant interactions indicated sustained adherence. These patterns were used to gauge state-level differences in the last two measures of behavior change.

Results

Continuing to Deliver the RNA and RNR Practices (Question 1)

According to the leadership interviews, consistent with the hypothesis, both states were using their RNA across all counties seven years after their initial implementation. All five study sites were using the RNAs office-wide, including the two sites that started by piloting the YLS/CMI in only part of their office seven years prior. No probation office amended its policies regarding how RNAs were to be considered in decisions (e.g., case plans should include youths’ top need areas identified by the RNA). Only one office significantly changed its RNA administration policy. Four years after initial implementation, NE Site 3 upgraded its policy to complete the YLS/CMI pre-adjudication rather than post-disposition. This was in alignment with the other state offices and enabled risk and the needs to influence more case processing decisions.

Programmatic and Contextual Changes Influencing Sustainability (Question 1)

According to statements from Northeastern site leadership, many significant state-level initiatives occurred to promote juvenile justice system reform following the adoption of the YLS/CMI. The juvenile act was modified to require juvenile justice agencies to use evidence-based practices and indicated probation offices would not receive grants unless they adopted the YLS/CMI. A state entity designed the Juvenile Justice System Enhancement Strategy, which outlined a process of system reform that included the YLS/CMI, RNR, and motivational interviewing, followed by changing youth behavior through quality case management, evidence-based practices, graduated response, and skill-building. This entity provided financial resources for probation offices to get advanced training on case management and skill-building, including statewide adoption of the Carey Guides Brief Intervention Tool. The state entity and probation chief’s committee instituted a process for biannual recertification of state YLS/CMI master trainers and PO booster training twice a year on the YLS/CMI and case plan training.

At the site-level, each probation office used state resources to maintain internal master trainers who in turn provided YLS/CMI booster training annually. Each probation office instituted routine supervisory reviews of the quality of YLS/CMIs. NE Sites 1 and 2 also implemented supervisory reviews to ensure the services recommended by POs were addressing needs identified on the YLS/CMI. In addition, NE Site 2 supervisors obtained rigorous supervisor training and added the quality of POs’ YLS/CMIs and need-to-service matching in case plans as considerations in job performance evaluations. Since implementing the YLS/CMI in 2009, every site had changes in its probation leadership, eliminated their intensive probation units, adopted graduated response protocols, received motivational interview training, and had changes in their judiciary and assistant district attorneys, some of whom were “tougher on crime.” Law enforcement in NE Site 3’s district implemented a school diversion program, resulting in substantial reductions in delinquency referrals. This reduction made it feasible for the office to conduct the YLS/CMI with all youth pre- instead of post-adjudication. NE Sites 1 and 2 had officers trained in the Carey Guides to promote needs-based supervision contacts. The chief in NE Site 1 stated that the “job of probation looks different, with the YLS/CMI and motivational interviewing being fully integrated into their work now..”

In the Southern state, the two study sites were among a handful of locally operated probation offices but were still affected by some state-level changes. There were two relevant state-level initiatives since their first year of implementation. First, the payer of community services shifted from the state juvenile justice agency to multiple managed care organizations. These organizations covered evidence-based programs, such as Multi-Systemic Therapy, which led to an increase in their availability and use. Second, in 2013 the state adopted detention standards that reduced the number of youths who could be sent to these facilities. Both sites had changes in their judiciary, probation leadership, and assistant district attorneys, one of which increased the use of post-petition diversion (SO Site 1). Both sites adopted motivational interviewing and graduated response approaches. Unlike the Northeastern state, there were no statewide initiatives to maintain master trainers or promote case planning skills. SO Site 1 gave POs booster training on the SAVRY annually, included SAVRY and case plan training as part of the orientation for new POs, and maintained quality assurance by supervisors routinely checking SAVRYs and case plans. Conversely, SO Site 2 only had one SAVRY booster since 2010, and supervisors routinely checked the quality of case plans but not SAVRYs. The Southern state’s POs had completed significantly less RNA and RNR training than their Northeastern counterparts (see Table 1).

Probation Officer Behavior Change by State: Self-Reported Adherence (Question 2)

As indicated in Table 1, POs in the Northeastern state had significantly more experience working in juvenile justice and in their current position than POs in the Southern state. Regression models indicated months of experience was significantly associated with POs’ use of their RNA in two areas: consensus ratings for POs’ use of their RNA in service recommendations; B = .01(.01); Exp[B] = 1.01; CI .95 = [1.00, 1.01], p = .05; and POs’ ratings for making service recommendations without consulting their RNA; B = .27(.01); CI = [.001, .01], p = .02. Thus, we included months of experience as a covariate for adherence to RNR in service recommendations to remove the effects of state differences in PO characteristics before testing our hypothesis that states would not differ in POs’ self-reported adherence.

Table 2 provides the results of PO interviews overall and by state. The number of PO respondents varied by case management area because not every PO had a role in each area of decision-making. With respect to open-ended questions (first row within each case management area in Table 2), over half of POs (63.77%) referenced the RNA for disposition recommendations, approximately 55% for service recommendations, and 26% when recommending responses to a violation. Overall, POs’ mean responses on rating scales indicated the RNA affected their decisions most of the time (M = 5.14 for probation violations to M = 6.19 for services; 7 = Always). Moreover, ratings indicated their recommendations infrequently departed from the RNA risk level or need areas (M = 1.12 for disregarding needs to M = 2.10 for being more restrictive with probation violations). Examples of circumstances in which POs stated they might be more restrictive than the RNAs’ risk level included sex offender cases, cases where they felt the family needed intensive services, or when the youth acquired new charges. A few POs provided an example indicating a general orientation toward punishment; “Sometimes you have to do that to make a point to the juvenile or sometimes they just don’t get it.” The most frequent departure from the RNA was to target needs not identified on the RNA with services (M = 2.91, SD = 1.90). Examples of such instances according to these POs often involved responsivity factors, such as mental health or learning disabilities, or needs coming to light after the initial assessment; “There may be things that they [youth] don’t answer truthfully on the SAVRY, or that their parents don’t know about . . . Or things that we find out about. . .” With respect to usefulness ratings, the lowest mean rating was for violations (M = 4.72, SD = 2.06).

Table 2:

Probation Officer Interview Ratings for Their Use of Risk-Needs Assessments in Decisions by State

	Overall(N = 84)	NE state(n = 46)	SO state(n = 38)	Statistic(df)	p	d/η_p²	OR [95% CI]
	M (SD)	M (SD)	M (SD)	Statistic(df)	p	d/η_p²	OR [95% CI]
Disposition recommendations	n = 69	n = 40	n = 29
Mentioned using RNA^a	63.77%	65.00%	62.07%	χ²(1) = 0.06	.80		.88 [0.33, 2.38]
Rec corresponds with risk	5.52 (1.46)	5.15 (1.66)	6.03 (0.94)	t(67) = −2.58	.01	.65
More restrictive rec than the risk	1.72 (1.35)	1.88 (1.40)	1.52 (1.27)	t(67) = 1.09	.28	.27
Less restrictive rec than the risk	1.88 (1.53)	2.08 (1.48)	1.62 (1.59)	t(66) = 1.22	.23	.29
Rec without consulting the RNA	1.00 (1.68)	1.13 (1.86)	.83 (1.42)	t(67) = 0.72	.47	.18
Useful for disposition recs	5.03 (1.71)	4.65 (1.85)	5.55 (1.35)	t(67) = −2.23	.03	.56
Service recommendations	n = 76	n = 43	n = 33
Mentioned using RNA	55.27%	69.76%	36.36%	B = −1.22	.02		.29 [0.11, 0.80]^b
Target needs from RNA	6.19 (1.26)	6.24 (1.05)	6.12 (1.49)	F(2,70) = 0.89	.42	.03^c
Disregard some needs from RNA	1.12 (1.57)	1.40 (1.58)	.76 (1.50)	F(2,70)=1.70	.19	.05^c
Target needs not identified in RNA	2.91 (1.90)	3.00 (2.08)	2.79 (1.67)	F(2,70) = 0.70	.50	02^c
Rec without consulting RNA	1.64 (1.20)	2.10 (2.16)	1.06 (1.62)	F(2,70)=4.30	.02	11^c
Useful for service recs	5.63 (1.56)	5.45 (1.48)	5.85 (1.64)	t(73) = −1.10	.28	.25
Recommending responses to probation violations	n = 69	n = 39	n = 30
Mentioned using RNA	26.09%	30.77%	20.00%	χ² (1)= 1.02	.31	.56	.56 [0.18, 1.73]
Rec corresponds with risk	5.14 (1.82)	4.67 (2.02)	5.77 (1.33)	t(67) = −2.58	.01	.64
More restrictive rec than the risk	2.10 (1.71)	2.46 (1.68)	1.63 (1.65)	t(67) = 2.04	.04	.50
Less restrictive rec than the risk	1.78 (1.69)	2.03 (1.68)	1.47 (1.68)	t(67) =1.37	.17	.33
Rec without consulting the RNA	1.67 (2.25)	2.05 (2.50)	1.17 (1.78)	t(67) = 1.64	.10	.40
Useful for probation violations	4.72 (2.06)	4.49 (2.23)	5.03 (1.79)	t(67) = −1.10	.28	.27

Note. Ratings ranged from 0 (Never) to 7 (Always). Significant findings are in bold. NE = Northeastern; SO = Southern; CI = confidence interval; OR = Odds Ratio; RNA = risk-needs assessment; Rec = recommendation; ANCOVA = analysis of covariance.

“Mentioned using RNA” refers to the open-ended questions and consensus ratings. ^b Results are Exp[B] and its 95% CI from logistic regression conducted to control for months of experience working in juvenile justice. Reference group was Northeastern. ^c ANCOVAs conducted to include months in current position as a covariate. Effect size reported is η_p^2.

Contrary to the hypothesis that states would not differ, t-test or analysis of covariance results identified significant state-level differences in POs’ self-reported adherence to RNR in every case management area with medium effect sizes. Southern POs gave significantly higher ratings than Northeastern POs when asked if disposition and violation recommendations corresponded with risk level (see Table 2). Northeastern POs were most apt to make more restrictive recommendations following violations than the risk level would indicate. With respect to service recommendations, Northeastern POs mentioned using their RNA almost three times as often as Southern POs upon open-ended questioning, B = −1.222(.512); Exp[B] = 0.295; p = .017; CI = [.108, .804], but also reported making recommendations without consulting their RNA more often, on average, F (2,70) = 4.30, p = .017, η_p² = .11. With respect to usefulness ratings, the only state-level difference was that Southern POs gave significantly higher ratings for the usefulness of the RNA in disposition recommendations than Northeastern POs.

Practitioner Behavior Change: Adherence to RNA Policies (Question 2)

Table 3 provides the YLS/CMI or SAVRY completion and policy adherence rates for each cohort by site with chi-square results comparing policy adherence rates. Contrary to the hypothesis, states significantly differed such that Northeastern sites significantly improved and Southern sites regressed. NE Site 3 had marked improvement with youth in the seventh year cohort being over four times as likely as those in the first year cohort to receive the YLS/CMI pre-disposition. NE Site 1 significantly improved by about 10%, which was a smaller effect (odds ratio [OR] = 1.64), but this site had less improvement to make than NE Site 3. Change in adherence could not be evaluated for NE Site 2 due to its partial implementation procedure in the 1^st year of implementation; however, it had excellent adherence in the 7^th year (95.19%). In the Southern state, SO Site 1 significantly improved its rates of policy adherence (OR = 3.01), however, SO Site 2 regressed from approximately 57% of its youth having a completed SAVRY pre-disposition in the 1^st year to only 25% in the 7^th year (OR = .25).

Table 3:

Change in Adherence to Risk/Needs Assessment Administration Policies by State and Site

	1st Year cohort		7th Year cohort		Comparing rates of adherence				Sustainability
	RNA completion rate, n (%)	Policy adherence rate, n (%)	RNA completion rate, n (%)	Policy adherence rate, n (%)	χ² (df)	p	OR	95% CI^a	Sustainability
Northeast State(N = 433 per cohort)	334(77.10)	285 (66.90)	412(95.20)	339(78.30)
Site 1 (n = 221)	190(85.97)	152 (68.78)	209(94.57)	173(78.28)	5.13(1,442)	.02	1.64	[1.07, 2.51]	Improved
Site 2 (n = 104)	104(NA)	104(NA)	99(95.19)	99(95.19)					NA
Site 3 (n = 108)	40(37.04)	29(28.71)	104(96.30)	67(62.04)	23.34(1,209)	< .001	4.06	[2.27, 7.25]	Improved
Southern State(N = 297 per cohort)	261(88.47)	212 (75.18)	248(83.8)	164(68.62)
Site 1 (n = 205)	185(90.24)	165 (82.50)	180(87.80)	142(93.42)	9.24(1,352)	< .01	3.01	[1.44, 6.30]	Improved
Site 2 (n = 92)	76(82.61)	47(57.32)	69(75.00)	22(25.29)	17.93(1,169)	< .001	.25	[0.13, 0.48]	Regressed

Note. Significant findings are in bold. RNA = risk-needs assessment; Completion Rates = number of youth with RNAs completed at any point during supervision/the total number of youth court referrals (in the Northeast) or adjudications (in the South) for which a SAVRY or YLS/CMI was expected as per the sites’ policy; Policy adherence Rate = number of youth with RNAs completed at the time dictated by policy (e.g., pre-adjudication, pre-disposition)/the total number of youth court referrals (in the Northeast) or adjudications (in the South) for which it was expected as per the sites’ policy; CI = confidence interval; NA = not applicable.

Chi-squares compared rates between the adherence to policy rate cells.

Practitioner Behavior Change: Adherence to the Risk Principle (Question 2)

Among youth with a completed YLS/CMI in the Northeastern state’s first-year cohort (n = 334), the risk levels were 48.50% low, 45.21% moderate, 6.29% high. The seventh-year cohort (n = 412) had a significantly higher percentage of low-risk (57.52%), and lower percentage of moderate-risk (36.17%) youth than the first-year cohort, χ²(2) = 6.60, V = .04, CI [.03, .17], p = .04. Among youth with a completed SAVRY in the Southern state’s 1^st-year cohort (n = 260), the risk levels were 40.38% low, 43.46% moderate, 16.15% high. The percentage of youth in the seventh-year cohort with a SAVRY (n = 248) rated as low risk (30.24%) was significantly lower than the first-year cohort, χ²(2) = 5.94, φ = .11, CI [.04, .19], p = .05. SO Site 2 significantly decreased from 13 (1^st-year) to only two high-risk youth (7^th-year) while the portion of high-risk youth in SO Site 1 increased (14.15% to 27.22%).

Table 4 reports the results of adherence to the risk principle for placement and service decisions by cohort and state. Overall rates of youth going to restrictive placements stayed relatively constant in both states between the first and seventh years of RNA implementation (SO = 31.90% versus 32.69%; NE = 18.70% versus 21.09%, respectively). In the Northeastern state, chi-square analyses indicated risk was significantly related to whether youth received a placement in both cohorts (see Table 4). A moderated logistic regression analysis including the PSM weights, days of follow-up, and unbalanced covariates (any violent priors, number of prior offenses), demonstrated risk level was significantly associated with placements at the second block (Exp[B] = 2.41, CI [1.42, 4.07], p = .001), and the interaction term between cohort and risk level was not (Exp[B] = 1.14, CI [0.57, 2.29], p = .71), indicating sustained adherence over time. In the Southern state, placements were significantly related to risk level in the first year but not in the seventh year (see Table 4) where low-risk youth were as likely to be placed (32.00%) as high-risk youth (34.00%). A moderated hierarchical logistic regression including the PSM weights and days of follow-up (there were no covariates) demonstrated risk level was significantly associated with placements at the second block (Exp[B] = 1.93, CI [1.28, 2.91], p = .002), but the interaction term between cohort and risk level also was significant (Exp[B] = 0.51, CI [.29, .91], p = .02), indicating a significant regression in adherence.

Table 4:

Change in Adherence to the Risk Principle in Case Management Areas by State

% Placed^a	1st Year cohort(Southern n = 257; Northeastern n = 332)						7th Year cohort(Southern n = 245; Northeastern n = 403)						Sustained
% Placed^a	Lown (%)	Modn (%)	Highn (%)	χ²(df, N)	P	V[95% CI]	Lown (%)	Modn (%)	Highn (%)	χ²(df, N)	p	V[95% CI]	Sustained
NE State	15(9.35)	34(22.67)	13(61.90)	36.70(2,332)	<.001	.33[.22, .46]	21(9.01)	49(34.03)	15(57.69)	55.83(2,403)	<.001	.37[.28, .47]	Sustained
SO State	21(20.00)	40(35.40)	21(53.85)	16.13(2,257)	<.001	.25[.14, .37]	24(32.00)	39(32.50)	17(34.00)	.06(2,245)	.97	.02[.02, .18]	Regressed
Service referrals^b	M(SD)	M(SD)	M(SD)	F(df, N)	P	η²	M(SD)	M(SD)	M(SD)	F(df, N)	P	η²
NE State	0.89(1.64)	1.62(2.20)	2.57(2.87)	9.71(2,331)	<.001	.06	1.19(1.51)	2.03(1.99)	2.85(2.17)	17.44(2,402)	<.001	.08	Sustained
SO State	1.75(1.74)	2.04(1.39)	2.49(1.25)	3.43(2,256)	.03	.03	2.23(2.29)	2.37(2.03)	3.02(2.01)	2.34(2,244)	.10	.02	Regressed

Note. Significant findings are in bold. CI = confidence interval; NE = Northeastern; SO = Southern.

Cells represent the percentage of youth within each risk level who received a restrictive placement at any point during supervision. ^b Means represent the average number of service referrals received by youth within each risk level.

With respect to service referrals, ANOVA results indicated the number of referrals significantly increased across youths’ risk levels in both states in the first-year cohorts, but with only a small effect in the Southern state (η² = .03; see Table 4). In the Northeastern state, the association remained significant in the seventh-year cohort with a medium effect (η² = .08). A moderated hierarchical linear regression including PSM weights, days of follow-up, and unbalanced covariates (any violent priors, number of prior offenses, any prior mental health treatment, and any prior substance use treatment) demonstrated risk level was significantly associated with service referrals in the Northeastern state at the second block (β = .09, CI [.03, .54], t = 2.21, p = .03), and the interaction term between cohort and risk level was not significant (β = −.18, CI [−.62, 33], t = −.60, p = .55), indicating the association between risk and services sustained over time. In the Southern state’s seventh-year cohort, although the average number of referrals remained in the expected direction with high-risk youth receiving an average of 3.02 (SD = 2.01) referrals and low-risk youth receiving 2.23 (SD = 2.29), the association was not significant. Consequently, the moderated regression including days of follow-up and PSM weights (there were no other covariates) demonstrated risk level was not significantly associated with services at the second block (β = −.70, CI [−3.78, 0.41], t = −1.58, p = .11), but the interaction term between cohort and risk level was significant (β = −.48, CI [−8.48, −.18], t = -2.05, p = .04) indicating adherence regressed. Results of covariate analyses and regressions are included in Supplemental Material Tables SC and SD available in the online version of this article.

Discussion

This was the first multi-site study to examine the sustainability of RNR and RNA in probation offices, using four sustainability constructs (Moore et al., 2017). Prior studies reported POs do not use their RNAs in decisions regularly and/or do not follow the RNR model (e.g., Dyck et al., 2018; Viglione et al., 2015), but whether these findings are a result of initial poor implementation or lack of sustainability is unknown. The current study addressed this knowledge gap by revisiting five probation offices seven years after a rigorous, stakeholder-engaged implementation of RNA and RNR to examine whether states sustained, improved, or regressed over time, as well as facilitators and barriers to sustainability.

Continued Delivery of Risk-Needs Assessment and RNR Principles

Consistent with the hypothesis that both states would have sustained their delivery of RNA and RNR, all five probation offices were still delivering their RNA and some RNR-based practices. In fact, the two offices that originally started with only partial office pilots moved to full implementation of their RNA by the seventh year. Moreover, both states had rolled out the implementation of their RNAs in juvenile probation statewide. One initially ineffective probation office (NE Site 3) with an original YLS/CMI administration policy that was not conducive to RNR-based practices vastly improved its policy and adjusted its probation structure to conduct assessments earlier in the process. This agency had considerably poor adherence to practices in its first year (less than 40% of youth had received a YLS/CMI) yet achieved striking gains in implementation fidelity, a 95% RNA administration adherence rate by Year 7, due to policy changes from new leadership.

Evolving and Adapting the Evidence-Based Practices

Since initial implementation, programmatic changes in RNA administration and RNR-based practices occurred indicating agencies adapted practices over time. Leadership interviews revealed clear state differences—the Northeastern sites benefited from statewide initiatives that promoted evidence-based practices. The sociopolitical climate in the state enabled strong implementation of RNR practices that evolved through the adoption of complementary innovations (e.g., motivational interviewing, training POs in skill-building) and statewide support to maintain fidelity to the YLS/CMI, and case plaining through booster trainings, internal coaching, and quality assurance.

Conversely, in the Southern state, state-level initiatives were fewer but increased accessibility to evidence-based community services and limited use of detention. Data indicated these changes were not followed by reductions in rates of restrictive placements and the number of services provided to youth increased regardless of risk level. There was no state-level support for quality assurance strategies or staff training. The office that did not sustain internal training or strong quality assurance procedures seemingly experienced a degradation in the validity of SAVRYs given the substantial decline in youth rated as high risk (13 in the first year, 2 in the seventh year) and increase in youth rated low-risk (56.52%). This risk composition is inconsistent with national reductions in arrest (presumably for lower risk youth) and was inconsistent with the other Southern office.

Probation Officer Behavior Change: Self-Reported Adherence to RNR

The hypothesis that states would not differ in PO-reported adherence was partially supported. Overall, POs reported relatively high adherence to the risk principle for disposition recommendations and to the need principle (> average 5 on an 8-point scale, with higher numbers indicating stronger adherence). However, the Southern state was significantly less likely than the Northeastern state to report the use of their RNA in service decisions in their open-ended responses. The limited use of the RNA in case planning may partially explain the Southern states’ regressed adherence to the risk principle in the numbers of services referred; by the seventh year, low-risk youth were receiving about as many services as high-risk youth in the South. Open-ended questions across POs indicated departures from criminogenic needs in service decisions were generally to address responsivity factors (e.g., mental health needs), which is consistent with RNR and may represent an evolution in practice as both states adopted mental health screening. Adherence to the risk principle when recommending responses to probation violations was low across the board. POs reported lower perceptions of the usefulness of RNAs in probation violation recommendations, possibly explaining the lack of change in restrictive placement rates in both states (most occur following a violation of probation rather than at disposition). The favorable perceptions of the RNAs usefulness in other decisions may be a sign that RNR practices had been institutionalized to some degree in both states, and POs were not merely complying as has been found nationally (Miller & Maloney, 2013) but were in fact bought in. An experimental study of POs across this Northeastern state provides further evidence of strong PO adherence to risks and needs, with a few departures (Miller & Palmer, 2020).

Broad Practitioner Behavior Change: Adherence to RNA Policies and Risk Principle

We examined broader practitioner behavior change via adherence to the administration of the RNA and adherence to the risk principle, which is dependent on both PO behavior and an enabling court environment. For example, in states like the Southern state where probation attempts to complete an RNA with youth’s post-adjudication and pre-disposition, whether the RNA is completed within this timeframe is dependent on the court’s willingness to allow the time in-between hearings. Only judges have the authority to impose restrictive placement decisions, which may or may not follow PO recommendations. Likewise, the number of services to which youth are referred is somewhat controlled by POs but most juvenile courts include the services youth are to attend in the court orders, meaning it is partly or wholly determined by the court.

Inconsistent with the hypothesis that states would not differ, there were meaningful state differences in the maintenance of practitioner behavior change. Despite the apparent institutionalization of the RNA and RNR among POs in both states, administrative data painted a different picture of the sustained fidelity of practices. The Northeastern state significantly improved its adherence to administering the RNA to every youth on time (pre-adjudication) and sustained its strong adherence to the risk principle in decisions. In part, this was due to a change in probation leadership in NE Site 3, which led to a significant change in administration policy (from post to pre-disposition). Incidentally, this new chief was the only leader who was not employed in the office for the first-year cohort.

Results were the opposite in the Southern state. Leadership interviews indicated there was less buy-in to the RNR approach among their new judges and district attorneys than in the previous court administration, which was reflected in their administrative data. The Southern office (SO Site 2) with the strictest policy, which was reliant on judges bifurcating adjudication and disposition hearings, had a significant regression in policy adherence (57–25%) and consequently, the lowest RNA completion rate of any site (75% of eligible youth). SO Site 1 had improved administration adherence, but this was mainly the result of an influx in youth handled informally, in which case the SAVRY was not required. SO Site 1’s administration policy was more feasible than SO Site 2 in that POs could complete the RNA within 10 days post-disposition if necessary.

The approach of conducting RNAs post-disposition may have been more feasible than pre-disposition RNAs but it was clearly not more effective. Collectively across both sites in the Southern state, adherence to the risk principle in case management significantly regressed over time. In the seventh-year cohort, low-risk youth received too many services and were just as likely to be placed as high-risk youth. In part, this may have been a result of the seemingly low validity of PO SAVRY ratings in SO Site 2 (only two youth in the seventh-year cohort were rated as high risk), indicating youth may have been falsely designated as low risk. However, the pattern was similar in SO Site 1 where there was no evidence of low-validity SAVRY ratings. It seems courts, and possibly POs, were frequently making decisions without the benefit of a valid RNA in both jurisdictions, either because they did not have it or because they did not see a reason to use it. In sum, the Southern state’s sustainability was hindered by system changes and turnover in court personnel resulting in an absence of court buy-in, as well as insufficient quality assurance and staff monitoring.

Study Limitations

This study has several limitations. First, this study speaks more to the sustainability of the risk principle than the need or responsivity principles, which are considerably more complex (e.g., Drawbridge et al., 2019; Vieira et al., 2009), involve examining additional stakeholders and barriers due to service availability (Haqanee et al., 2014), and were beyond the scope of this paper. Second, we did not control for site-level differences in the analyses of PO data due to small numbers in some sites (n ranged from 8 to 22). Third, we were unable to achieve an acceptable balance in the PSMs between cohorts in two sites but accounted for this by including covariates in the adherence to risk principle cohort-comparisons. Fourth, data from only two time-points is a limitation that inhibits the measurement of fluctuations over time, particularly with respect to the use of the RNA in placement and service-related decisions. However, it was clear the delivery of RNAs and RNR continued for the entire seven-year period and that buy-in by POs was maintained despite any potential fluctuations. Nonetheless, the lack of multiple time-points prevents pinpointing which systemic or practice changes had the largest influence on sustainability.

Adherence rates to the administration of the RNA for Northeastern offices, where youth receive the YLS/CMI prior to adjudication, were not able to account for youth or parent refusals to be interviewed prior to an adjudication hearing or for youth not showing up to appointments. Situations like these and occasional unconventional court practices (e.g., youth may be automatically disposed to placement or sent to adult court) lead to the strong argument that offices should not be expected to have a 100% adherence to RNA administration policies. Eighty-five percent is an effective and feasible benchmark that still results in a positive impact on case processing (Drawbridge et al., 2019). Finally, on the surface, it may appear state differences in sustainability were due to the use of different instruments, particularly because the SAVRY involves some judgment. This judgment may have led to inaccurate risk levels or less use of need areas (which are less clear than in the YLS/CMI) leading to greater reliance on judgment by POs in decisions. It is difficult to disentangle the effects of instrumentation differences from the effects due to dramatic differences in (a) ongoing and training and quality assurance procedures and (b) court and prosecutor buy-in, both of which were significantly poor in one SO site yet strong in all NE sites. The difference in instrumentation did not result in state-level differences in fidelity to the RNA or RNR procedures in the first-year cohorts. This points to the more plausible explanation of sustainability differences being due to dramatic state-level differences in ongoing training and quality assurance practices. For example, the validity of SAVRY risk levels appeared to have degraded in SO Site 2 (only two high-risk youth in the seventh year) but not in SO Site 1 where more training was provided.

Conclusions and Recommendations

Overall, this study of the sustainability of RNA and RNR principles adds further support to single-state findings regarding successful juvenile justice reforms when strong implementation procedures are followed (Rocque et al., 2014). Research consistently demonstrates that effective implementation strategies for evidence-based practices are critical for achieving positive outcomes in real-world settings (Fixsen et al., 2009). The initial strategies followed by probation offices in this study included orientations and involvement with multiple stakeholder groups to promote buy-in, development of RNR-based policies and decision-support tools (e.g., case plans aligned with need and responsivity areas and service matrix), and facilitation of a master trainer model and supervisory reviews. The initial implementation and intervention-level outcomes (Vincent et al., 2016) far surpassed that of post hoc observational studies in sites with initially strong adherence to their RNA administration procedures (e.g., Dyck et al., 2018; Viglione, 2019). Also, this study demonstrated agencies that struggle with implementation initially can evolve and achieve implementation fidelity and strong outcomes (e.g., NE Site 3).

The state differences in sustainability uncovered by the current study point to facilitators and barriers to sustaining RNR-based practices. These fell into the three categories of drivers identified by implementation science (Fixsen et al., 2019). Drivers included the benefits of supportive leadership at the state and local levels innovating practices through the adoption of complementary initiatives (leadership drivers), regular local training and supervisory monitoring (competency drivers), and continued buy-in from system partners, namely court officials (organizational drivers). Conversely, the Southern state demonstrated some barriers to sustainability, including turnover in judges and district attorneys in the absence of education from probation leadership about the importance of the RNR approach, and lack of supervisory quality assurance measures and ongoing training. Key implications of the findings include:

Poor implementation outcomes in the first year or two of implementing RNAs and RNR do not preclude later implementation success if the sociopolitical climate and leadership enable and promote evidence-based practices.

Sustaining several aspects of RNR requires behavior change from not only probation officers and strong leadership, but also from court stakeholders (Harvell et al., 2018). Restrictive placements are dependent on the courts, and in large part, so are service referrals due to the inclusion in court orders. These aspects of case management are unlikely to adhere to the risk principle if the court is not bought into and/or is not receiving RNA information pre-disposition unless the courts permit POs to have control over service planning. Studies should attend to whether courts are enabling or disabling.

Probation leaders should adopt an ongoing strategy for educating court personnel on their RNA and RNR to manage turnover. Promoting buy-in for evidence-based practices must be ongoing, it is not a “one and done.” State-level support for evidence-based practices can have a strong influence in this regard.

Policies to administer the RNA prior to adjudication are more effective and sustainable than post-adjudication/pre-disposition policies because they are not dependent on the courts delaying the disposition hearing and ensure courts have access to risk and need information prior to disposition decisions. Any pre-adjudicatory procedures must be paired with protections against self-incrimination (see Vincent, Guy, & Grisso, 2012).

Coaching, booster training, and supervisory coaching or quality assurance within probation offices are critical; state-supported training will promote sustainability.

Evolution and adaptation are important parts of sustainability (Moore et al., 2017; Taylor, 2005) and are essential as the real-world multi-level, context in which these practices are delivered changes over time. Likewise, our benchmarks for successful implementation and sustainment should account for these changes. We should not expect 100% adherence to RNA administration policies or to the risk and need principle, given the importance of integrating the responsivity principle.

Future research should examine whether agencies maintaining RNR practices sustain the benefits for justice-involved youth (Moore et al., 2017). These benefits may be conceptualized via intervention-level outcomes such as reductions in supervision levels and rates of formal processing, reductions in the use of restrictive placements, improved need to service matching, and reductions in recidivism. Studies of the sustainability of justice reform efforts are scarce. Future sustainability studies should focus attention toward developing sophisticated measures of practice adaptations, the impact of changing contextual factors, and which implementation supports are most effective for sustaining practice (e.g., written polices, coaching, stakeholder buy-in, data-tracking and continuous quality improvement methods).

Supplemental Material

sj-docx-1-cjb-10.1177_00938548241229193 – Supplemental material for Sustainability of Evidence-Based Practices: Risk-Need-Responsivity in Probation 7 Years Later

Supplemental material, sj-docx-1-cjb-10.1177_00938548241229193 for Sustainability of Evidence-Based Practices: Risk-Need-Responsivity in Probation 7 Years Later by Gina M. Vincent, Dara C. Drawbridge, Spencer G. Lawson, Kristina Todorovic and Rachael T. Perrault in Criminal Justice and Behavior

Supplemental Material

sj-docx-2-cjb-10.1177_00938548241229193 – Supplemental material for Sustainability of Evidence-Based Practices: Risk-Need-Responsivity in Probation 7 Years Later

Supplemental material, sj-docx-2-cjb-10.1177_00938548241229193 for Sustainability of Evidence-Based Practices: Risk-Need-Responsivity in Probation 7 Years Later by Gina M. Vincent, Dara C. Drawbridge, Spencer G. Lawson, Kristina Todorovic and Rachael T. Perrault in Criminal Justice and Behavior

Supplemental Material

sj-docx-3-cjb-10.1177_00938548241229193 – Supplemental material for Sustainability of Evidence-Based Practices: Risk-Need-Responsivity in Probation 7 Years Later

Supplemental material, sj-docx-3-cjb-10.1177_00938548241229193 for Sustainability of Evidence-Based Practices: Risk-Need-Responsivity in Probation 7 Years Later by Gina M. Vincent, Dara C. Drawbridge, Spencer G. Lawson, Kristina Todorovic and Rachael T. Perrault in Criminal Justice and Behavior

Supplemental Material

sj-docx-4-cjb-10.1177_00938548241229193 – Supplemental material for Sustainability of Evidence-Based Practices: Risk-Need-Responsivity in Probation 7 Years Later

Supplemental material, sj-docx-4-cjb-10.1177_00938548241229193 for Sustainability of Evidence-Based Practices: Risk-Need-Responsivity in Probation 7 Years Later by Gina M. Vincent, Dara C. Drawbridge, Spencer G. Lawson, Kristina Todorovic and Rachael T. Perrault in Criminal Justice and Behavior

Footnotes

Authors’ Note:

The authors thank the probation leadership and officers at the study sites. We also thank our statisticians and research associates, Lisa Lundquist, Cassandra Beinemann, Maxwell Christensen, Danielle Rieger, and Lauren McDowell. This research was funded by the Office of Juvenile Justice and Delinquency Prevention (PI: Vincent, 2016-JF-FX-0057) and the John D. and Catherine T. MacArthur Foundation (07-88401-000-HCD) Models for Change Initiative Research Network.

ORCID iDs

Gina M. Vincent

Spencer G. Lawson

Supplemental Material

Supplemental Material is available in the online version of this article at .

Gina M. Vincent, PhD, is Co-Director of the Law and Psychiatry Program and a professor in the Implementation Science & Practice Advances Research Center (iSPARC) at the UMass Chan Medical School. She studies implementation of risk assessment and risk-need-responsivity, as well as mental health and substance use in justice settings.

Dara C. Drawbridge, PhD, is an assistant professor in the Law and Psychiatry Program and Implementation Science & Practice Advances Research Center (iSPARC) at the UMass Chan Medical School. She provides implementation support for, and studies, the implementation of risk-need-responsivity, risk-needs assessment, and case management in justice settings.

Spencer G. Lawson, PhD, is a senior research scientist in the Law and Psychiatry Program and Implementation Science & Practice Advances Research Center (iSPARC) at the UMass Chan Medical School. His research considers the practical implications for managing and preventing behavioral health conditions among people in criminal-legal settings, with attention also placed on evaluating the implementation and validity of risk instruments.

Kristina Todorovic, PhD, is an assistant professor at the University of Southern Indiana. She studies extralegal factors that influence jurors’ decisions and perceptions of child witnesses’ credibility. Her research contributes to the development of evidence-based investigative interview techniques with children in a forensic context.

Rachael T. Perrault, MA, is a project director in the Law and Psychiatry Program and the Implementation Science & Practice Advances Research Center (iSPARC) at UMass Chan Medical School. She studies the implementation of risk assessment and risk-need-responsivity, as well as decision-making in juvenile justice settings.

References

Andrews

D. A.

Bonta

(2010). Rehabilitating criminal justice policy and practice. Psychology, Public Policy, and Law, 16(1), 39–55. https://doi.org/10.1037/a0018362

Borum

Bartel

Forth

(2006). Manual for the structured assessment for violence risk in youth (SAVRY). Psychological Assessment Resources.

Brookhart

M. A.

Schneeweiss

Rothman

K. J.

Glynn

R. J.

Avorn

Stürmer

(2006). Variable selection for propensity score models. American Journal of Epidemiology, 163(12), 1149–1156. https://doi.org/10.1093/aje/kwj149

Carroll

Patterson

Wood

Booth

Rick

Balain

(2007). A conceptual framework for implementation fidelity. Implementation Science, 2(1), 40. https://doi.org/10.1186/1748-5908-2-40

Drawbridge

Todorovic

Winters

Vincent

G. M.

(2019). Implementation of risk-need-responsivity principles into probation case planning. Law and Human Behavior, 43, 455–467. https://doi:10.1037/lhb0000351

Dyck

H. L.

Campbell

M. A.

Wershler

J. L.

(2018). Real-world use of the risk-need-responsivity model and the level of service/case management inventory with community-supervised offenders. Law and Human Behavior, 42, 258–268.

Fabelo

Arrigona

Thompson

M. D.

Clemens

Marchbanks

M. P.

(2015). Closer to home: An analysis of the state and local impact of the Texas juvenile justice reforms. Council of State Governments Justice Center.

Fixsen

D. L.

Blase

K. A.

Naoom

S. F.

Wallace

(2009). Core implementation components. Research on Social Work Practice, 19(5), 531–540. https://doi.org/10.1177/1049731509335549

Fixsen

D. L.

Blase

K. A.

Van Dyke

M. K.

(2019). Implementation practice & science. Active Implementation Research Network.

10.

Haas

S. M.

DeTardo-Bora

K. A.

(2009). Inmate reentry and the utility of the LSI-R in case planning. Corrections Compendium, 34(1), 11–16.

11.

Haqanee

Peterson-Badali

Skilling

(2014). Making “what works” work: Examining POs’ experiences addressing the criminogenic needs of juvenile offenders. Journal of Offender Rehabilitation, 54(1), 37–59. https://doi.org/10.1080/10509674.2014.980485

12.

Harvell

Love

Pelletier

Warnberg

Willison

J. B.

Winkler

M. K.

(2018). Bridging research and practice in juvenile probation. https://ojjdp.ojp.gov/sites/g/files/xyckuh176/files/pubs/252234.pdf

13.

Hockenberry

Puzzanchera

(2019). Juvenile court statistics 2017. National Center for Juvenile Justice.

14.

Hoge

R. D.

Andrews

D. A.

(2006). Youth Level of Service/Case Management Inventory: User’s manual. North Tonawanda, NY: Multi-Health Systems.

15.

Hoge

R. D.

Andrews

D. A.

(2011). Youth level of service/case management inventory 2.0. Multi-Health Systems.

16.

Ingel

S. N.

Davis

L. R.

Rudes

D. S.

Hartwell

T. N.

Drazdowski

T. K.

McCart

M. R.

Chapman

J. E.

Taxman

F. S.

Sheidow

A. J.

(2022). Misunderstanding and sensemaking among juvenile probation officers working with evidence-based practices. Victims & Offenders, 17(7), 975–993. https://doi.org/10.1080/15564886.2021.1985670

17.

Juvenile Justice Geography, Policy, Practice & Statistics. (2021). Evidence-based practices. http://www.jjgps.org/juvenile-justice-services#risk-assessment?year=2020

18.

Loman

S. L.

Rodriguez

B. J.

Horner

R. H.

(2010). Sustainability of a targeted intervention package: First step to success in Oregon. Journal of Emotional and Behavioral Disorders, 18(3), 178–191.

19.

Miller

Maloney

(2013). Practitioner compliance with risk/needs assessment tools: A theoretical and empirical assessment. Criminal Justice and Behavior, 40(7), 716–736. https://doi.org/10.1177/0093854812468883

20.

Miller

Palmer

(2020). Juvenile probation officer decision-making in a reforming state: Assessing the application of evidence-based principles. Criminal Justice and Behavior, 47(9), 1136–1155. https://doi.org/10.1177/0093854820925112

21.

Moore

J. E.

Mascarenhas

Bain

Straus

S. E.

(2017). Developing a comprehensive definition of sustainability. Implementation Science, 12(1), 1–8. https://doi.org/10.1186/s13012-017-0637-1

22.

Moullin

J. C.

Sklar

Green

Dickson

K. S.

Stadnick

N. A.

Reeder

Aarons

G. A.

(2020). Advancing the pragmatic measurement of sustainment: A narrative review of measures. Implementation Science Communications, 1(1), 1–18. https://doi.org/10.1186/s43058-020-00068-8

23.

National Research Council. (2013). Reforming juvenile justice: A developmental approach. The National Academics Press. https://doi.org/10.17226/14685

24.

Olver

M. E.

Stockdale

K. C.

Wormith

J. S.

(2009). Risk assessment with young offenders: A meta-analysis of three assessment measures. Criminal Justice and Behavior, 36(4), 329–353. https://doi.org/10.1177/0093854809331457

25.

Proctor

Silmere

Raghavan

Hovmand

Aarons

Bunger

Griffey

Hensley

(2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7

26.

Proctor

E. K.

Powell

B. J.

McMillen

J. C.

(2013). Implementation strategies: Recommendations for specifying and reporting. Implementation Science, 8(1), 1–11. https://doi.org/10.1186/1748-5908-8-139

27.

Rocque

Welsh

B. C.

Greenwood

P. W.

King

(2014). Implementing and sustaining evidence-based practice in juvenile justice: A case study of a rural state. International Journal of Offender Therapy and Comparative Criminology, 58(9), 1033–1057. https://doi.org/10.1177/0306624X13490661

28.

Rosenbaum

P. R.

Rubin

D. B.

(1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41

29.

Rubin

D. B.

(2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services and Outcomes Research Methodology, 2(3–4), 169–188. https://doi.org/10.1023/A:1020363010465

30.

Schafer

J. L.

Kang

(2008). Average causal effects from nonrandomized studies: A practical guide and simulated example. Psychological Methods, 13(4), 279–313. https://doi.org/10.1037/a0014268

31.

Stirman

S. W.

Kimberly

Cook

Calloway

Castro

Charns

(2012). The sustainability of new programs and innovations: A review of the empirical literature and recommendations for future research. Implementation Science, 7(1), 1–19. https://doi.org/10.1186/1748-5908-7-17

32.

Stuart

E. A.

King

Imai

(2011). MatchIt: Nonparametric preprocessing for parametric causal inference. Journal of Statistical Software, 42, 1–28. https://doi.org/10.18637/jss.v042.i08

33.

Taylor

J. E.

(2005). Sustainability: Examining the survival of schools’ comprehensive school reform efforts. American Institutes for Research.

34.

Vieira

T. A.

Skilling

T. A.

Peterson-Badali

(2009). Matching court-ordered services with treatment needs: Predicting treatment success with young offenders. Criminal Justice and Behavior, 36(4), 385–401. https://doi.org/10.1177/0093854808331249

35.

Viglione

(2019). The risk-need-responsivity model: How do probation officers implement the principles of effective intervention? Criminal Justice and Behavior, 46(5), 655–673. https://doi.org/10.1177/0093854818807505

36.

Viglione

Rudes

D. S.

Taxman

F. S.

(2015). Misalignment in supervision: Implementing risk/needs assessment instruments in probation. Criminal Justice and Behavior, 42(3), 263–285. https://doi.org/10.1177/0093854814548447

37.

Viljoen

J. L.

Shaffer

C. S.

Muir

N. M.

Cochrane

D. M.

Brodersen

E. M.

(2019). Improving case plans and interventions for adolescents on probation: The implementation of the SAVRY and a structured case planning form. Criminal Justice and Behavior, 46(1), 42–62. https://doi.org/10.1177/0093854818799379

38.

Vincent

G. M.

Guy

L. S.

Fusco

S. L.

Gershenson

B. G.

(2012). Field reliability of the SAVRY with probation officers: Implications for training. Law and Human Behavior, 36, 225–236. https://doi.org/10.1007/s10979-011-9284-2

39.

Vincent

G. M.

Guy

L. S.

Grisso

(2012). Risk assessment in juvenile justice: A guidebook for implementation. John D. and Catherine T. MacArthur Foundation. www.NYSAP.us

40.

Vincent

G. M.

Guy

L. S.

Perrault

R. T.

Gershenson

B. G.

(2016). Risk assessment matters, but only when implemented well: A multi-site study in juvenile probation. Law and Human Behavior, 40, 683–696.

41.

Vincent

G. M.

Paiva-Salisbury

Cook

N. E.

Guy

L. S.

Perrault

(2012). Impact of risk/needs assessment on juvenile probation officers’ decision-making: Importance of implementation. Psychology, Public Policy, & the Law, 18, 546–576.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB

0.04 MB

0.02 MB

0.03 MB