Abstract
This study examined the feasibility of and fidelity to risk/needs assessment, mental health screening, and risk-need-responsivity (RNR)-based case planning within juvenile probation in two states. The researcher-guided implementation effort included the Massachusetts Youth Screening Instrument-2 (MAYSI-2), Structured Assessment of Violence Risk in Youth (SAVRY), and policies to prioritize criminogenic needs while using mental health services only when warranted. Data from 53 probation officers (POs) and 553 youths indicated three of five offices had high fidelity to administration and case planning policies. The interrater reliability (n = 85; intraclass correlation coefficient [ICC][A, 1] = .92 [Northern state] and .80 [Southern state]) and predictive validity (n = 455; Exp[B] = 1.83) of SAVRY risk ratings were significant. There was an overreliance on mental health services; 48% of youth received these referrals when only 20% screened as having mental health needs. Barriers to fidelity to RNR practices in some offices included assessments not being conducted before disposition, lack of service availability, and limited buy-in from a few stakeholders.
Considerable attention has been directed toward advancing juvenile justice reform by translating the evidence for reducing delinquent offending into practice (National Research Council, 2013). Case planning is a particularly important area for translating research evidence into practice. Quality case planning can improve the chances of justice-involved individuals’ success, whereas poor quality planning can set them up for failure. Case planning involves a series of interrelated functions to provide coordination, including assessment, planning, linking, monitoring, and advocacy (Monchick et al., 2006). In justice settings, the most widely tested model for individualizing case planning in a manner that will reduce recidivism has been risk-need-responsivity (RNR; Andrews & Bonta, 2010; Gendreau et al., 2006). The RNR model emphasizes use of services that target individuals’ malleable risk factors.
In juvenile justice settings, mental health needs are another critical area for consideration in case planning due to the high prevalence rates of these needs (e.g., Drerup et al., 2008; Teplin et al., 2002). A recent study of juvenile probation case planning found mental health services frequently took precedence over other risk-related services (Haqanee et al., 2014), which is contrary to the existing evidence about how to reduce recidivism (e.g., McCormick et al., 2017) and highlights challenges with implementing best practice. To address these challenges and better integrate risk and mental health needs into case plans, researchers assisted juvenile probation agencies with implementing risk/needs assessment and mental health screening procedures, along with RNR-based case planning practices. The purpose of this study was to examine the implementation outcomes (feasibility and fidelity) of this RNR-driven, researcher-guided, case planning process in multiple juvenile probation offices.
RNR and Mental Health
The RNR framework suggests intensive programming should be reserved for higher risk cases because lower risk cases do as well or better with minimal intervention (risk principle; Hoge & Andrews, 2011). Programming should target an individual’s dynamic risk factors influencing their offending (need principle) to achieve the largest reduction in recidivism. Programming also should consider how well the styles and modes of service are matched to an individual’s attributes that may affect treatment response (responsivity principle). To accomplish this, agencies must first conduct a valid risk/needs assessment that accurately identifies each individual’s risk level and dynamic risk factors to which the case plan will be tailored. Dynamic risk factors, also referred to as criminogenic needs, are malleable characteristics that strongly influence youth offending behavior. Examples of youth criminogenic needs include negative peer associations, personality traits (e.g., impulsivity, poor anger control, lack of empathy), and procriminal thinking. Decades of longitudinal and epidemiological research (e.g., Farrington, 1989; Lipsey & Derzon, 1998) has documented significant associations between dynamic risk factors and reoffending.
Mental health needs and diagnoses generally are viewed within the RNR model as noncriminogenic needs (and not relevant to case planning unless symptoms are severe) or as responsivity factors. Mental health needs can be an important factor in case planning because symptoms moderate the benefits a youth would otherwise receive from services that target criminogenic needs (responsivity factor). However, with the exception of some specific circumstances (e.g., symptoms of attention-deficit hyperactivity disorder), mental health needs are not considered risk factors for recidivism. Indeed, Andrews and Bonta (2010) maintained that criminogenic needs were sufficient for prediction of recidivism without considering formal diagnoses. Several studies of youth have supported this sentiment by demonstrating that neither youths’ mental health diagnoses nor their clinical assessment scores have been associated with, or predictive of, recidivism after accounting for criminogenic needs (Guebert & Olver, 2014; McCormick et al., 2017; Schubert et al., 2011). Moreover, receipt of mental health treatment alone, even evidence-based treatment, does not appear to affect justice-related outcomes (McCormick et al., 2017; Skeem et al., 2011), whereas services targeting criminogenic needs do (McCormick et al., 2017; Vieira et al., 2009). Nonetheless, Haqanee and colleagues (2014) found mental health needs often took precedence over criminogenic needs in juvenile probation officers’ (POs’) case plans. Because the prevalence of mental health symptoms (e.g., Vincent et al., 2008) and diagnoses (e.g., Wasserman et al., 2010) in juvenile justice settings is undeniably high, inclusion of mental health services in case plans often may be warranted. However, prioritization of mental health services over services targeting criminogenic needs is not in alignment with effective approaches for recidivism reduction.
Importance of Implementation and the Implementation Process
Shifting probation case planning practice to be in alignment with the evidence requires strong implementation procedures that involve key stakeholders within the jurisdiction (Vincent, Guy, & Grisso, 2012). Multiple studies indicate POs generally do not link case plans to risk/needs assessments as the RNR model prescribes (Haas & DeTardo-Bora, 2009; Miller & Maloney, 2013; Peterson-Badali et al., 2015; Viglione et al., 2015). These findings are likely an artifact of poor implementation practices. There is evidence that the use of decision support tools and appropriately structured case plan documents improve POs’ use of risk/needs assessments (Bosker & Witteman, 2016; Viljoen et al., 2019). Moreover, probation offices with strong implementation procedures and diligent supervisors have significantly better implementation of RNR than offices with less oversight (e.g., Nelson & Vincent, 2018).
Moving from the “let it happen” to a “making it happen” approach (Greenhalgh et al., 2004), the Risk Assessment and Behavioral health Study (RABS) involved researchers assisting juvenile probation offices in two states with implementation of a predispositional risk/needs assessment and mental health screening system designed to improve case planning. The RABS project used components of the Active Implementation Frameworks model (Fixsen et al., 2019) and the risk assessment implementation guidebook (Vincent, Guy, & Grisso, 2012) in this effort. The system included implementation of a valid risk/needs assessment for dispositional planning (the Structured Assessment of Violence Risk in Youth [SAVRY], Borum et al., 2006), a valid mental health screening tool (the Massachusetts Youth Screening Instrument-2 [MAYSI-2] Grisso & Barnum, 2006), and an RNR-based case planning process that integrated mental health needs. We briefly describe the implementation steps here. More detail can be found elsewhere (Vincent, Paiva-Salisbury, et al., 2012).
The RABS project researchers selected juvenile justice agencies with buy-in from a Northern and a Southern state. In each state, the implementation effort was driven by a lead agency and high-level decision-makers (juvenile justice stakeholders: POs, judges, attorneys, service providers, and representatives from the Department of Mental Health). In the Northern state, the state juvenile justice agency responsible for probation and corrections was the lead and the Administrative Office of the Courts was the lead in the Southern state, which had a county-run system. The researchers facilitated each state’s selection of their validated risk/needs assessment by presenting the research evidence (including differential predictive validity by race) for different instruments, ultimately resulting in both states selecting the SAVRY.
Researchers worked through an enhanced, standardized implementation process with an implementation committee in each state for 1 year (Vincent, Paiva-Salisbury, et al., 2012). The enhancements pertained to adding methods for integrating mental health needs into case planning. The process addressed the three main implementation drivers (Fixsen et al., 2019) by working with stakeholders to promote buy-in for RNR-based decision-making (leadership drivers), developing written policies and decision-support tools (organizational drivers), and providing training to judges, POs, and service providers (competency drivers). Activities included (a) conducting judge and PO orientation training on risk assessment, RNR, and the contributions of mental health to elicit buy-in for use of the SAVRY in disposition decisions (the services in case plans were dictated by court order for most jurisdictions); (b) stakeholder-engaged development of policies for administration and use of the instruments in disposition recommendations and case plans; (c) development of decision-support tools (e.g., standardized case plan, customized service matrices guiding selection of mental health, and risk reduction services); (d) integration of instruments and service data into case management systems; and (e) supervisor quality assurance checklists.
Following the five implementation activities, POs underwent 2.5 days of training on the SAVRY and MAYSI-2, and both POs and service providers underwent a day of training on case planning. The case plan training emphasized how to match needs to services and that mental health services alone were insufficient for reducing recidivism. The training and policies directed POs and service providers to refer youth with only criminogenic needs to only risk reduction services, and to refer youth with both criminogenic needs and significant mental health needs to both risk reduction and mental health services. All were instructed that youth should not be referred to mental health services in lieu of risk reduction services except in the case of low-risk youth who present with only mental health needs, a group that should be diverted. The training also included decision-support tools for integrating mental health needs into disposition recommendations (a recommendations template) and case planning (a structured case plan template and matrix listing each agency’s services by criminogenic need area).
Current Study
This multisite, implementation outcomes study used mixed qualitative and quantitative data from the RABS project to examine the feasibility of and the fidelity to the predispositional risk/needs assessment and mental health screening system. Proctor et al. (2011) defined implementation outcomes as the “effects of deliberate and purposive actions to implement new treatments, practices, and services” (p. 65). Feasibility is the extent to which a new innovation can be successfully used within an agency or setting (Karsh, 2004; Proctor et al., 2011). We examined the feasibility of the instruments and procedures using qualitative data from POs to gauge whether jurisdictions could successfully use the instruments to in case planning.
Fidelity is “the degree to which an intervention was implemented as it was prescribed in the original protocol or as it was intended by the developers” (Proctor et al., 2011, p. 69). Fidelity is a complex construct but a core component is adherence (Carroll et al., 2007), which we examined with respect to three policies. First, we evaluated POs’ adherence to instrument administration policies. Based on effectiveness findings from other probation offices (Vincent et al., 2016), we defined high fidelity as 90% or more of eligible youth cases receiving the SAVRY and MAYSI-2. Second, we examined adherence to policies pertaining to the use of instruments in making referrals, mainly whether POs used the MAYSI-2 to make evaluation referrals, and the SAVRY to make service referrals. Fidelity would be high if POs reported adhering to these procedures most of the time, and if the percentage of youths referred for an evaluation was comparable to the national percentage of probation intake youth cases “screening in” on the MAYSI-2 (roughly 26%, Vincent et al., 2017). Third, we examined adherence to RNR-based case planning policies regarding integration of criminogenic and mental health needs. A reasonable benchmark for high fidelity to this policy would be for the proportion of youth cases referred to (a) a mental health service to be lower than the proportion screened in on the MAYSI-2 because screening tools are expected to have high false-positive rates (Grisso, 2005), and (b) risk reduction services to be comparable to the proportion rated as moderate to high risk on the SAVRY. In addition, we examined whether there was fidelity to conducting the SAVRY as intended by the developers by assessing POs’ interrater reliability (IRR) and predictive validity.
Method
Sample and Participants
This study included four county-run probation offices from a Southern state and all nine state-run probation offices within a Northern state. Researchers gathered deidentified administrative and file data (e.g., youth demographic and court referral information, SAVRY and MAYSI-2 results, service referrals from case plans) on all youth cases and interview data from consenting POs. The institutional review board at the University of Massachusetts Medical School approved this study.
Youth Cases
Researchers identified youth cases for inclusion based on criteria set by the probation offices as to the point in court processing (preadjudication or predisposition) where youth would receive the MAYSI-2 and SAVRY. The initial sample comprised 856 youths charged with delinquency offenses, which constituted all court-referred or adjudicated youth at the relevant court processing point in the sites over a 9-month period (starting in July or August 2015 and ending in March or April 2016, depending on the state). The Northern state sample was 150 consecutive sentenced youths because this state was unable to implement the SAVRY predisposition due to multiple system issues. The sample from the Southern state comprised the 706 youths seen at the points where each office conducted the instruments. In one office, this was all youth cases referred to the court (n = 135); for two offices, it was all youth cases petitioned before they were adjudicated, which excluded youth diverted or receiving nolle prosequi (n = 153); in the largest office, the MAYSI-2 was administered at intake and the SAVRY was conducted postadjudication/predisposition. To keep the full study sample proportionate across offices in the Southern state, the researchers drew a random sample of half of the adjudicated youths from this large office (n = 418). The sample of 856 youths was 21.60% female, averaged 15.71 years of age (SD = 1.61), and was 50.00% Black, 32.20% non-Hispanic White, and 17.80% Other race (often biracial). Their ethnicities were 7% Hispanic.
Probation Officers (POs)
The PO sample included the 53 POs, supervisors, and intake officers (Northern state n = 20, Southern state n = 33) who reported they had conducted the SAVRY and/or MAYSI-2 at least once by the time of their research interview, which was conducted 4 to 5 months into implementation. The PO sample was 68.52% female, 75.93% White, and 20.37% Black. They had 18.10 median years (SD = 9.17 years) experience working with juvenile justice-involved youth. POs in the Northern state were significantly more likely to be White (90.00%) than those in the Southern state (67.65%; χ2 (2) = 10.70, p = .005, Φ = .45) and had more years of experience (M = 1.79, SD = 0.52 vs. M = 1.16, SD = 0.80, respectively); t(51.39) = 3.54, p = .001, d = .93, 95% confidence interval (CI) = [39.62, 143.39].
Measures and Procedures
The procedures involved multiple methods of data gathering. First, three trained master’s-level research assistants conducted phone interviews with the eligible POs. Interviewers gathered both quantitative and qualitative information to gauge POs’ perceptions of feasibility and their self-reported adherence to (a) the MAYSI-2 and SAVRY administration policies and (b) the policies regarding use of the instruments in referrals. Second, researchers obtained youth-level data from administrative data systems (e.g., prior offenses, petition and disposition dates, demographic information, SAVRY and MAYSI-2 results, admissions and release dates from placements), POs’ written case notes, hard copy of social history reports, and disposition recommendation forms. Research staff contacted POs directly to obtain missing data if at all possible. Youth-level data collection spanned a 14-month period from July or August 2015 (implementation date) to September or October 2016 resulting in a minimum 6-month to a maximum 14-month follow-up period.
To examine POs’ IRR on the SAVRY, each probation office randomly selected youth cases to be rated by two independent parties. Every PO had at least one of their cases double rated. The primary rater was always the assigned PO who interviewed the youth and a parent, and the second rater was usually a PO or supervisor who observed the interview, reviewed collateral information, and independently rated the SAVRY. For feasibility purposes, in 36.47% of cases, the second rater was a master’s-level intern. The master’s-level interns completed the same SAVRY training as the POs.
POs’ Adherence to Policies
Audio-recorded PO interviews included multiple questions to measure PO adherence to policies. First, to measure adherence to instrument administration procedures, interviewers asked POs at what point in the process they conducted the instruments, whether any were missed, and why. To measure adherence to use of the MAYSI-2, interviewers asked POs about the percentage of their cases referred for a more comprehensive mental health evaluation based on the MAYSI-2 results. To measure adherence to use of the SAVRY in service recommendations to the court and/or case plan referrals, interviews included four questions adapted from a national survey of risk assessment use in probation settings designed by Miller and Maloney (2013). The questions asked POs to rate their level of use of the SAVRY on a scale from 0 (never) to 7 (always) regarding how often they (a) “. . . target the criminogenic needs identified by the SAVRY?,” (b) “. . . disregard some criminogenic needs that are identified by the SAVRY?,” (c) “. . . target needs that are not identified by the SAVRY?,” and (d) “. . . make a decision or recommendation about services without consulting the SAVRY?”. Whenever a PO responded to any questions in the negative, interviewers asked what were the “biggest barriers to completing/using the instrument?”.
SAVRY
The SAVRY (Borum et al., 2006) was designed to assess violence risk in adolescents aged 12 to 18 years; however, it is also a valid assessment of nonviolent reoffending (Olver et al., 2009). It comprises six protective factor items and 24 risk factor items, 10 of which are static and 14 are dynamic. Risk factors are rated as low, moderate, or high. The SAVRY uses a structured professional judgment approach as opposed to a formulaic actuarial approach. This means the final determination of a youth’s overall level of risk is the PO’s summary risk rating (low, moderate, high risk) based on the PO’s professional judgment after appraising the most relevant risk factors for the case. Meta-analyses indicate the SAVRY has good predictive validity for reoffending in a variety of youth populations (average areas under the curve [AUCs] of 0.71; Olver et al., 2009; Singh et al., 2011) from different racial/ethnic backgrounds (e.g., Perrault et al., 2017).
The POs underwent extensive training on the SAVRY over a 3-month period and prior to any data collection. They attended a 2-day workshop on the SAVRY and policies for its administration, followed by adequate completion of three post-training, practice case vignettes, receiving feedback after each case. POs were taught to rate the SAVRY based on all available collateral information (e.g., school records, psychological reports) and semi-structured interviews with the youth and with a caregiver. Training included how to use the SAVRY to identify criminogenic need areas based on item combinations identified by Nelson and Vincent (2018). Agency data systems were programmed to be consistent with this need-identification approach to ensure POs were considering the proper dynamic risk factors.
MAYSI-2
POs received a half-day training on completion of the MAYSI-2 and policies for its administration and use. The MAYSI-2 is a valid brief, self-report screening tool for use in the juvenile justice system to identify youth who might have mental health needs (Grisso & Barnum, 2006). The MAYSI-2 was nationally normed for juvenile justice youth aged 12 through 17 years and contains 52 yes/no questions scored into six clinical scales (alcohol/drug use, angry-irritable, etc.) and one nonclinical scale (traumatic experiences scale). Youth completed the MAYSI-2 on a computer that read the questions aloud. Probation office policies identified youth with elevated self-reported mental health symptoms (critical mental health cases) if they scored above the caution cutoff on the Suicide Ideation Scale and/or above the warning cutoff on any other two clinical scales. These youth “screened in” and would need some form of response.
Service Referrals
For each youth case, researchers obtained administrative data on the services referred and the service agency to which youth were referred over a 6-month minimum to a 14-month maximum period from each youth’s disposition dates (average 10-month follow-up). The researchers defined service referrals as rehabilitation-oriented services, including referrals to residential treatment and inpatient facilities and excluding services the youth initiated prior to their court involvement or that were referred by the parents. Referrals did not count court conditions that would be considered sanction-oriented (e.g., electronic monitoring, community service). We counted service referrals made for all youth cases, regardless of whether they received dispositions that would not necessarily include a mandate for services (e.g., diverted, minor sanction). Youth who did not receive any service referrals were coded as “none” if they received dispositions that would be expected to involve services (e.g., probation, unsupervised probation), and were coded as “not applicable” if they received dispositions that would normally not involve services. Services received while in correctional facilities could not be obtained.
The researchers categorized services into two types: mental health or risk reduction. Mental health services included counseling, psychiatric inpatient stays, residential mental health settings, and evidence-based programs that would be expected to affect mental health (e.g., cognitive-behavioral therapy, dialectical behavior therapy). Evaluations were not counted as services because an evaluation alone would not be considered a treatment. Services addressing only substance use were coded as risk reduction because substance use is a risk factor. Risk reduction services were those that addressed at least one criminogenic need area. For each service, researchers coded all the need areas the service would target based on each probation office’s service matrix, information on provider websites, and direct contacts with providers when necessary. For example, Parenting with Love and Limits was considered a risk reduction service for the family need area, and cognitive-behavioral therapy was considered a mental health service as well as a risk reduction service for both disruptive behavior and procriminal attitudes. Every probation office had access to multiple service options (minimum of four providers) within each criminogenic need area with the exception of substance use treatment in the Northern state (only one provider), and disruptive behavior and attitude-related services in one Southern office (only two provider options).
Recidivism
The definition of recidivism was any new petition (formal filing of charges) following the index offense and was categorized as violent (involved potential or actual harm to another individual) or any recidivism (included nonviolent and violent petitions and excluded violations and status offenses). Researchers gathered recidivism data from both juvenile and adult records. In the Southern state, data came from county-level administrative databases, and in the Northern state, data came from the family court’s statewide criminal record database. Time at-risk was calculated separately for each offense category after accounting for time spent in secure facilities, using the study’s follow-up date as the end point for youth who did not reoffend. The median recidivism follow-up period was 602.50 days (ranging 447–723 days).
Data Analyses
The majority of analyses testing adherence to policies involved chi-squares or mean comparisons (t tests) of interview responses and types of service referrals. Analysis of qualitative data involved transcribing POs’ responses to open-ended questions and categorizing these into themes based on consensus between two researchers. The SAVRY IRR analyses used two-way random, single measurement, absolute agreement-type intraclass correlation coefficients (ICC(A, 1), McGraw & Wong, 1996). Fleiss (1986) suggested the following classifications for ICC1s. ICC1 values of ≥.75 indicate excellent agreement, .60 to .74 indicate good agreement, and .40 to .59 indicate fair agreement. Cox proportional hazards regression models evaluated the predictive validity of the SAVRY for recidivism. Cox regression is a semi-parametric survival analysis that accounts for variable times at-risk by examining the proportion of cases that are surviving the time to a specific event (recidivism). The hazard ratio (Exp[B]) is the preferred index for interpretation. It is the ratio of hazards between two individuals whose values on the variable of interest differ by one unit when predictor variables are held constant, if applicable (Hosmer et al., 2008). For example, an Exp[B] equal to two for the SAVRY risk level would indicate that, at any day during the study period, the likelihood of recidivism for a youth rated as high risk would be twice that of a youth rated as moderate risk.
Results
Did POs Find the New System Feasible?
The researchers examined the feasibility (the extent to which a new innovation can be used successfully within an agency, Karsh, 2004; Proctor et al., 2011) of the predispositional risk assessment and mental health screening system via research interviews with POs 4 to 5 months after implementation. The majority of POs (83%) reported that they used the SAVRY in their case management decisions and all POs (100%) reported they used the MAYSI-2 for referrals for evaluations. However, many POs reported barriers to completion of the instruments and to the use of the SAVRY in their case planning. The qualitative data indicated barriers fell into four major themes: system-related obstacles, stakeholder buy-in, policy and training challenges, and data management systems.
System-Related Obstacles
Most system-related obstacles were present only in the Northern state with the only obstacle identified by both states being lack of service availability. In the Northern state, despite buy-in from many stakeholders, the state agency was never able to get the SAVRY and MAYSI-2 completed predisposition as intended. The end result was that the court specified the services to be completed by youth in the court orders before POs completed the SAVRY. POs reported that when the court-ordered services were not in alignment with SAVRY results, their only options were to recommend additional services to the youth and family to do voluntarily or to file a probation violation to change the court orders. There were three primary obstacles preventing assessments from being conducted predisposition in the North.
The first obstacle was the agency structure. The same overarching agency had oversight of both juvenile justice and child welfare. POs were held to the child welfare regulations so, in addition to the SAVRY, they still had to complete a “needs” assessment and accompanying case plan designed for child welfare settings. POs were trained how to use the child welfare case plan to capture and address SAVRY need areas, but they reported that the process was not user-friendly. This structural obstacle was exacerbated by turnover in leadership at the child welfare agency that prevented the juvenile justice department from being permitted to (a) stop using the child welfare assessment and (b) use the RNR-based case plans designed for this study. In addition, turnover in the chief justice position led to a 10-month delay in obtaining the essential court approval for POs to complete the SAVRY predisposition. This delay was not due to lack of buy-in but rather the time involved in replacing the chief justice. Finally, concerns from the PO’s union over the heavier workload involved in administering the SAVRY to the pool of youth preadjudication, as opposed to the smaller pool of youth receiving a disposition, presented another obstacle that was not rectified by the end of this study.
Stakeholder Buy-in
Although buy-in was present from the majority of key stakeholders in both states, lack of buy-in from some key leadership and some relatively seasoned POs presented barriers. As noted, the Northern state lacked buy-in from the new leader of the overarching child agency. In the Southern state, one judge who had initially expressed buy-in for the SAVRY and was part of the implementation committee changed her mind shortly after implementation for reasons unknown. As such, many POs reported they did not complete the SAVRY for youth in this judge’s court. Many Southern POs reported they were not making SAVRY and RNR-guided disposition and service recommendations to the court unless the presiding judge expressly asked for them, which was partially due to limitations in the buy-in of some POs.
Some POs made statements indicative of resistance to change, particularly in the Southern state where POs were accustomed to merely monitoring compliance to the court orders rather than to creating case plans. Several seasoned POs in both states (five or fewer per state) provided responses indicating that they did not individualize case management based on the SAVRY because they either had “standard methods they had been using for years” or because their “experience was more informative.” Several of the seasoned POs also provided responses indicating they were focusing more on mental health issues than on risk factors. With respect to the MAYSI-2, a few POs in the Northern state reported they were uncomfortable making “clinical judgments” about the need for mental health consultations and felt mental health screening was “too clinical” for their role.
Policy and Training Challenges
A common barrier to the feasibility of the administration policies in the three offices completing the instruments preadjudication was scheduling difficulties with the youth and family and occasional (but rare) refusals to be interviewed prior to the adjudication hearing. Moreover, in one of the offices that intended to complete the instruments only for youth who would be petitioned to the court (meaning the youth would receive a formal filing to have an adjudication hearing), some assessments were missed because POs inaccurately predicted at the time of a youth’s intake whether the youth would be petitioned. A few POs in the offices administering the SAVRY preadjudication also expressed concerns with the accuracy of assessments because youth and families were often untruthful.
A common procedural barrier to completing the SAVRY that was expressed in both states was difficulty completing the youth and parent semi-structured interviews that researchers provided to POs. POs had difficulty understanding that “semi-structured” meant to use it simply as a guide rather than going through each question. A related limitation in the Northern state was the administrator’s policy that POs had to complete their SAVRY interviews over three visits with the youth and family, one of which needed to be in the home. POs described this approach as “cumbersome.” Finally, the supervising PO in one site misinterpreted the SAVRY administration policy and their training and did not permit its administration to youth younger than 12 years, older than 18 years, or those charged with a sex offense.
Data Management System Issues
A primary obstacle in the Southern state pertained to their case management data system. Most of the counties had not been using this data system prior to the RABS project. Thus, POs were trained on this data system, at the same time they were learning the SAVRY, MAYSI-2, and case planning. Moreover, problems with the data system (e.g., inaccurate youth case numbers) resulted in difficulty locating some SAVRYs and MAYSI-2s, which POs had indicated they completed. This barrier was noted most frequently in the largest Southern office.
Did POs Adhere to the SAVRY and MAYSI-2 Administration Policies?
Of the total 856 eligible cases, MAYSI-2 results were only available for 461 (53.9%) youths and the SAVRY was available for 477 (55.6%) youths. Only 384 of the 856 eligible youths had both the MAYSI-2 and the SAVRY, resulting in an overall poor adherence rate of 45.00%. However, this rate is deceiving due to variability across sites and particularly poor administration in one site. Adherence to the SAVRY administration policy was high in the Northern state (100%) and two Southern sites (90.69% and 90.37%), moderate in one Southern site (67.27%), and exceptionally poor in one Southern site (21.77%). Adherence to the MAYSI-2 administration policy was fairly high in the Northern state (89.33%) and one Southern site (89.63%), and moderate (70.90% and 72.09%) to poor (23.21%) in the other sites. Southern POs’ explanations for the missing SAVRYs were the feasibility issues noted previously, including problems with locating data in their system. POs stated they did not administer the MAYSI-2 to youth who were noticeably unstable and were hospitalized without being screened.
In light of the poor adherence in some sites, the final sample of youths for analyses was less than the original 856. We defined the final sample as cases with at least one instrument (SAVRY or MAYSI-2) completed, resulting in a sample of 553: 150 postdisposition youths from the Northern state and 403 predisposition youths from the Southern state (247 evaluated preadjudication and 156 from the random sample of postadjudication/predisposition youths). The sample of 553 youths was 24.05% female, averaged 15.68 years of age (SD = 1.55), and was 49.01% Black, 37.40% non-Hispanic White, and 13.59% Other race. Almost 11% were of Hispanic ethnicity. The majority committed a nonviolent offense (66.90%) with the most common being theft or burglary (25.73%). This final sample differed significantly from the 303 youths that did not receive either instrument by (a) race, such that the final sample had a lower proportion of Black youths (53.3% vs. 73%, respectively) and a higher proportion of non-Hispanic White youths (43.7% vs. 25.1%, respectively); χ2 (4) = 25.67, p < .001, Φ = .19, and (b) ethnicity, such that there was a significantly higher proportion of Hispanic youths (16.4% vs. 2.4%); χ2 (1) = 5.84, p = .016, Φ = .12. Youth in the final sample also were more likely to have committed a violent index offense (33.1%) than those excluded (24.1%); χ2 (1) = 7.56, p = .006, Φ = .09. These differences were due to the demographics of youth (a) at the sites with the strongest adherence and (b) who were most likely to be formally processed (e.g., violent offense).
Of the 461 youths with a MAYSI-2, 20.40% (n = 94) scored as a critical case and this rate did not differ significantly between states (Northern = 19.40%, Southern = 20.80%; χ2 (1) = 0.11, p = .74, Φ = .02). Of the 476 youths with a SAVRY, only 17.90% (n = 85) were rated as high risk, 41.20% were moderate risk (n = 196), and 41.00% were low risk (n = 195). The Northern state had significantly fewer low-risk youth (28.00%) than the Southern state (46.91%) due to the postdisposition nature of the Northern sample; χ2 (2) = 15.24, p < .001, Φ = .18.
Did POs Adhere to Policies Pertaining to Use of the MAYSI-2 and SAVRY?
With respect to the MAYSI-2, POs in the Northern state reported that they only made referrals for mental health evaluations for 12% of youths they screened, on average, and POs in the Southern state said they referred 28%. Regarding adherence to use of the SAVRY, POs averaged 5.54 (SD = 1.99) in their responses to whether they targeted criminogenic needs identified by the SAVRY in their service referrals, meaning they used the SAVRY for referrals most of the time. The Southern state POs’ responses indicated they targeted criminogenic needs significantly more often (M = 5.94, SD = 1.77) than the Northern state (M = 4.85, SD = 2.11); t (52) = 2.03, p = .05, d = .56, 95% CI = [0.02, 2.17]. On average, POs in both states reported rarely disregarding needs that were identified by the SAVRY (M = 1.11, SD = 1.63), but occasionally targeting needs not identified in the SAVRY (M = 3.15, SD = 2.18). In addition, POs reported rarely making service referral recommendations without consulting the SAVRY (M = 1.72, SD = 2.50). Responses to these last three questions did not significantly differ between states.
Did POs Adhere to RNR-Based Case Planning Policies?
The following analyses included the sample of 437 youths from the total 553 youths who spent some time on community supervision (excludes those dismissed or diverted [n = 94, 16.99%] or in a secure facility the whole study period [n = 22, 4.84%]) and received a disposition that would be expected to involve mandated rehabilitation-oriented services. Only 295 of the 437 youths had any service referrals documented. The most common type of single service referral was a mental health-related service (47.8% of the 437 youths expected to receive any service). More than half of the cases (52.6%; n = 229) received a referral to some form of risk reduction service, which may appear higher than mental health services; however, these referrals were dispersed across different criminogenic needs. The most common type of risk reduction service was in the family need area, such as family counseling or parenting classes (24.5%). This was followed by peer-related (23.4%), procriminal attitude (22.9%), disruptive behavioral (22.0%), and substance use–related services (18.8%). The least common form of service referral was education-related (14.7%). POs referred about one third of youths (34.10%; n = 149) to both a mental health and at least one risk reduction service and 13.50% (n = 59) to only mental health services, 51.00% of whom were moderate to high risk.
Although it is not a perfect measure of adherence to the case planning policy, we also examined whether types of service referrals significantly differed based on MAYSI-2 critical case status. Youth who screened in on the MAYSI-2 should have been most likely to receive a mental health evaluation, which in turn could have resulted in recommendations for mental health services. Among youths who received a MAYSI-2 and some form of community supervision (n = 365), MAYSI-2 critical case status was not associated with referrals to mental health services; χ2 (1) = 0.18, p = .68, Φ = .02. Specifically, 52.1% (n = 38) of youths who scored as a critical case versus 49.3% (n = 144) of youths who did not score as a critical case were referred to a mental health service and this did not differ significantly by state. The odds of a critical case receiving a mental health service referral, relative to a noncritical case, was 1.14 (95% CI = [0.60, 2.16], p = .70) in the South, and 1.08 (95% CI = [0.46, 2.54], p = .87) in the North. Figure 1 in the Supplemental Materials displays the percent of youths by MAYSI-2 critical case status who were referred to (a) a mental health–related service, (b) a risk reduction service by the need area addressed, and (c) both a mental health and at least one risk reduction service (this included services that addressed both mental health and risk, such as cognitive-behavioral therapy. There were no significant differences in any types of referrals made for youth based on their critical case status, and this relationship did not differ significantly by state.
As a whole, the findings suggest sites did not implement the need and responsivity principles into their case planning with fidelity. However, there were site-level differences with one probation office in the Southern state evidencing particularly high fidelity. Adherence to the administration policy in this site was 90.69% (121 of 135 court-referred youths). Twenty-two percent of these scored as a critical case on the MAYSI-2 and 17% were high risk on the SAVRY. Among the 91 youths receiving dispositions that would warrant receipt of services, the most common type of referral was still to mental health services (62.6%); however, 68.1% received a referral to some type of risk reduction service. Almost half received a referral for a procriminal attitude (57.1%) and/or a disruptive behavioral (48.4%) service. More than half of the youths (52.7%) received both a mental health and risk reduction service. Roughly 10% of youths received only a mental health service, all but one of whom was low risk.
Did POs Complete the SAVRY With Fidelity?
With respect to IRR, in the Northern state, 18 cases rated by two independent raters (i.e., two trained POs or one PO and a trained research assistant) demonstrated excellent agreement on the SAVRY risk level (ICC(A, 1), = .92, p < .001) and on all but two need areas. In the Southern state, a total of 67 cases from the four offices also had excellent rater agreement on the SAVRY risk level (ICC(A, 1), = .80, p < .001) and on all but one need area (see Table 1). To examine predictive validity, we conducted recidivism analyses on the sample of 455 youths who had a SAVRY completed before they reoffended (excluded five cases that reoffended earlier) and who spent some time in the community over the median 602.50-day follow-up period (excluded 22 youths [4.84%] who were in a residential or secure facility the entire study period). Of these 455 youths, 92 (20.02%) received a new petition for any offense and only 38 (8.40%) received a petition for a violent offense. Cox regression analyses indicated the SAVRY risk level and most criminogenic need areas significantly predicted any and violent recidivism (see Table A: Supplemental Materials).
Probation Officer Interrater Reliability on SAVRY Overall Risk Level and Need Areas
Note. SAVRY = Structured Assessment of Violence Risk in Youth; ICC = intraclass correlation coefficient; CI = confidence interval.
p < .05. **p < .01. ***p < .001.
Discussion
The feasibility of and fidelity to the predispositional risk/needs assessment and mental health screening system was mixed and varied by jurisdiction. Lack of feasibility was tied largely to the extent of system-related obstacles, the point in the juvenile justice process at which the sites set their policies for administering the SAVRY and MAYSI-2, and data management issues. Not surprisingly, the sites with the greatest threats to feasibility (e.g., the system-related obstacles in the Northern offices, and data system issues in the largest Southern office) had the lowest fidelity to the new practices. This study lends further evidence to the notion that whether case planning procedures follow RNR principles with fidelity is not merely a function of PO competencies and strong written policies. Similar to other studies, the quality of case planning was largely affected by other factors in the jurisdictions, including judge buy-in (Shook & Saari, 2007), whether the assessment was conducted prior to disposition (Vincent et al., 2016), and data system issues (Vincent et al., 2018). The take-home message is that POs, and probation offices for that matter, do not operate in a vacuum. Instead, they operate within a system that imposes processes that can act as barriers or as facilitators to implementing evidence-based practices with fidelity (Harvell et al., 2018).
Did POs Adhere to Policies for Administration and Use of the Instruments?
Using the benchmark of a 90% adherence rate, three of five sites had high fidelity to their SAVRY administration policies and two had relatively high fidelity to their MAYSI-2 administration policy. Sites with a policy that involved every youth receiving the instruments early in the process had better adherence than those administering instruments only for youth expected to be petitioned, or for youth between the adjudication and disposition hearings. In the case of the latter, courts often do not bifurcate the adjudication and disposition hearings to allow for completion of a postadjudication assessment, even when judge buy-in is present; thus, assessments can be missed. Sites completing the instruments preadjudication ran into some scheduling challenges; however, these challenges did not result in adherence rates as low as sites that opted to administer the SAVRY to only youth expected to be petitioned or to youth postadjudication. Although policies for completion of assessments for all youths preadjudication resulted in stronger adherence, it is critical to note that this process required strong legal protections against self-incrimination, which requires collaboration with judges and attorneys.
With respect to using the instruments as intended, POs reported high fidelity to the MAYSI-2 use policy. POs in the Southern state reported referring roughly 28% for an evaluation, on average, and POs in the Northern state reported referring only 12% of their youth. Because states did not significantly differ in their proportion of youth scoring as critical cases on the MAYSI-2 (20.40% overall), the disparity in referrals between states was likely an artifact of differences in the point at which the MAYSI-2 was implemented. Youth receiving the MAYSI-2 postdisposition (Northern) would have been more likely to have already received a psychological evaluation than youth receiving the MAYSI-2 predisposition (Southern). The 28% of youths referred for an evaluation in the Southern offices was higher than their proportion of critical cases but was comparable to MAYSI-2 national norm data indicating 26% of youths at probation intake typically screen in (Vincent et al., 2017).
According to POs’ self-reports, fidelity to their use of the SAVRY in case planning was high in the Southern offices where they completed assessments predisposition, but significantly lower in the Northern offices, which was an artifact of most services being determined by the court prior to the SAVRY assessment. Northern POs reported frustration with not being able to conduct the SAVRY predisposition because their options for making service referrals that would address criminogenic need areas were limited by court orders. In sum, most POs understood how the instruments were to be used in case planning and reported high fidelity when the system permitted it. Some of the more seasoned POs were an exception to this finding.
Did POs Adhere to Case Planning Policies Regarding Mental Health and Risk Reduction Services?
Results from administrative data indicated that the need and responsivity principles were not implemented with fidelity in the majority of offices over the full 14-month study period. The likelihood of youth being referred to a mental health service while on community supervision was about 50-50 and this was unrelated to screening and assessment results. Given the rate of critical cases on the MAYSI-2, we would have expected less than 20% of cases to receive mental health services. Instead, use of mental health services was more than twice that (roughly 48%). Fidelity to the policies regarding risk reduction services was better. Approximately 60% of cases were moderate to high risk on the SAVRY indicating some form of risk reduction service(s) was needed and 52.6% were referred to risk reduction services. However, half of the 59 youths who received only mental health services were moderate to high risk. Also, the rates of referrals to risk reduction services addressing need areas with the strongest associations with recidivism among youth (e.g., peers, personality/behavior problems, see Perrault et al., 2017; Peterson-Badali et al., 2015) were relatively low.
These results indicate that there tended to be an overreliance on mental health services in case plans. The low fidelity to this aspect of the case planning procedures was due to a variety of factors. Probably the biggest factor was the inability of the Northern state to administer the instruments predisposition, meaning the courts were deciding which services would be in a case plan without the benefit of a risk/needs assessment. Instead, Northern courts relied on predispositional mental health evaluations, which were ordered regularly. The problem is that many juvenile justice jurisdictions consider the services youth are expected to attend to be part of the disposition and court-ordered conditions, which are determined by the court. Thus, when the disposition hearing occurs before a risk/needs assessment is completed, it can greatly limit fidelity to RNR principles. This problem led to the National Council of Juvenile and Family Court Judges (2017) proposition that judges “cease imposing ‘conditions of probation’ and instead support probation departments’ developing, with families and youth, individualized case plans . . .” (p. 2). Aside from convincing courts to change their court order practices in this way, the other solution is for courts to be presented with assessment results prior to disposition hearings to inform their decisions about the services stipulated. The conclusion is that unless one is working in a jurisdiction where the courts already defer to probation to create individualized case plans, fidelity to RNR principles requires both courts and POs to change their practices. RNR must be implemented in “jurisdictions” rather than in “probation offices.”
Two other barriers prevented high fidelity to the RNR-based case planning process. First, similar to qualitative findings from other PO studies (Vincent, Paiva-Salisbury, et al., 2012), the seasoned POs tended to be most resistant to RNR concepts and use of the SAVRY in guiding service decisions and instead reported relying heavily on mental health services. An additional barrier to targeting risk reduction services over mental health services was lack of availability of risk reduction services, a common issue expressed by POs that promulgated their overreliance on mental health services. Similar to other studies (Haqanee et al., 2014), many POs stated that mental health services were their “go-to” referrals due to limited options. Examination of each office’s service matrix and providers utilized in their service data indicated disruptive behavior and attitude/orientation services were limited relative to mental health services in at least two offices. This is problematic in that a high rate of mental health referrals for justice-involved youth who may be better served by risk reduction services can put undue strain on the mental health system while youth in greater need may be falling through the cracks. Another barrier was POs’ impressions that clinicians were addressing criminogenic needs in counseling. There is little reason to believe this is true, given counseling services appear to be ineffective for reducing recidivism (McCormick et al., 2017; Skeem et al., 2011). One study determined mental health treatment was essential for reducing recidivism; however, it was substance use disorder treatment that led to the reported reductions (Hoeve et al., 2014).
Other lessons learned came from observations of the probation office that had exceptionally high fidelity. This county-run office had four critical benefits. First, their policy was to assess and screen all youth immediately following their referral to the court. Second, they had strong probation supervisors who provided coaching to their staff and reviewed or collaborated on every SAVRY and case plan. Third, this agency had strong judge buy-in, such that the judge requested the POs’ service recommendations for all cases prior to making his disposition decisions. Finally, this office was the most well resourced in terms of providers who could offer evidence-based risk reduction services in most every need domain.
Did POs Complete the SAVRY With Fidelity?
On a positive note, the evidence suggested POs were completing the SAVRY with fidelity. The IRR of their overall risk ratings and most need areas was good to excellent. Moreover, the SAVRY overall risk rating and most of the need areas significantly predicted new petitions over an average 10-month period. The hazard ratios indicated an increase in the risk level on the SAVRY overall (and within the disruptive behavioral problems need area) was associated with twice the likelihood of reoffending violently at any point over the study period (see Table A in Supplemental Materials). The significant prediction of the SAVRY risk level for recidivism when completed by POs in the field is consistent with other studies (Viljoen et al., 2019; Vincent et al., 2016) and lends more credibility to the use of structured professional judgment instruments by POs. One limitation with the SAVRY expressed by some POs was that parents and youth were not always truthful, which indicated some did not remember the SAVRY should not be rated based on self-reported interview information alone. This point must be reinforced by supervisors on an on-going basis.
Limitations
There were a few notable limitations of this study. The most significant limitation was that the only measure of mental health needs was the MAYSI-2, which is a screening tool and not a diagnostic instrument. Although our interest was in acute mental health symptoms, the challenges posed by use of a blunt screening measure is that it is unknown how many youths who were “screened-in” actually needed mental health services; a full mental health assessment is needed to determine that. Nonetheless, there should have been a significant, positive association between screening results and referrals to mental health services.
A limitation to assessing PO adherence to the case planning policies using administrative data was that there was no failsafe way to distinguish service referrals that stemmed from a PO recommendation versus an independent decision by the court. Many of the mental health service referrals may have been due to court decisions. Another issue was the data system in the Southern state where the limited accuracy of youth court identifiers and disposition data entry was a significant hindrance. The researchers rectified data issues as much as possible by working closely with the state’s data department; however, the POs in the largest Southern site thought their adherence to the administration policies was significantly better than the data suggested.
Finally, there were some limitations with the SAVRY fidelity testing. First, the test of IRR would have been more rigorous if the second raters had conducted independent interviews of youth and parents rather than observing the same interview, which could inflate agreement. Our design of double-rating cases based on the same information was similar to other PO field reliability studies (Kennealy et al., 2017; Vincent, Guy, Fusco, et al., 2012), and use of a more optimal design was not feasible for youth or staff.
Recommendations and Future Directions
This implementation outcomes study should be viewed as a fidelity assessment of an initial implementation phase (Fixsen et al., 2019). The timeframe for this study spanned a little over 1 year after implementation; thus, it points to barriers to overcome within the first year. Implementation science suggests it requires about 2 to 4 years for adoption of a new practice to become fully operational to the point where there is an impact on the consumer (Barrett et al., 2008; Fixsen et al., 2005). Data like those reported here should be gathered in the first year of implementation and used to revise policies and provide booster training to improve fidelity. This study also points to six barriers and recommendations for justice agencies that are implementing RNR-based concepts into case planning.
First, changes in leadership and unsupportive opinions of external partners who are key to the process, mainly judges (Vincent et al., 2016), are significant barriers to implementation and can delay a new process from achieving its full potential. Juvenile justice agencies should start negotiations with probation unions as early in the process as possible. Second, juvenile justice systems that fall within the same umbrella agency as child welfare will be unable to implement risk assessment and RNR feasibly and with fidelity unless they are able to follow separate, juvenile justice-specific policies. Needs assessments and case plans conducted for child welfare cases examine different concerns (e.g., child safety). Third, postdisposition risk/needs assessments are entirely ineffective for implementation of RNR-based case planning unless courts are willing to allow POs leeway in their later case planning rather than including it all in the court orders (also see Harvell et al., 2018).
Fourth, cumbersome data systems that are viewed by staff as infeasible and do not track the essential data are significant barriers to implementation and continuous quality improvement (Vincent et al., 2018). Agencies with comprehensive and user-friendly data systems already in place will do better than agencies requiring this level of data entry from POs for the first time. Fifth, as was evident in the probation office with strong adherence, diligent supervisors that coach staff are a key driver of good fidelity (see Fixsen et al., 2019). Implementation efforts should create quality assurance procedures and provide separate supervisor training on these procedures and coaching upfront to change the behavior of seasoned staff. Another strategy that can work with staff who lack buy-in is to add case planning and risk/needs assessments to their job performance evaluations. Finally, strategies for educating juvenile justice agencies about how to identify mental health–related services that are most likely to be effective in reducing recidivism should be considered. For example, clinicians who follow a manualized cognitive-behavioral therapy program would be preferred over those providing unstructured counseling. There also may be value in training mental health providers how to address criminogenic needs.
This study lends important insights for research on probation practices. Researchers should routinely report when and how a new practice was actually implemented before examining whether it “works” or whether POs are using their risk/needs assessments properly. This includes reporting practices across the whole jurisdiction, which affects whether POs can do their job. Using this study as an example, we would have simply concluded POs were not following RNR principles if we did not know their SAVRYs were conducted too late in the process. Key areas to document include when the risk/needs assessment was conducted, the court practices for writing conditions, judge buy-in, and service availability. We also recommend researchers conduct more feasibility assessments and examine whether incorporating the lessons learned from these studies into refining policies and procedures leads to higher fidelity.
Supplemental Material
sj-docx-1-cjb-10.1177_00938548211008491 – Supplemental material for Risk-Need-Responsivity Meets Mental Health: Implementation Challenges in Probation Case Planning
Supplemental material, sj-docx-1-cjb-10.1177_00938548211008491 for Risk-Need-Responsivity Meets Mental Health: Implementation Challenges in Probation Case Planning by Gina M. Vincent, Rachael T. Perrault, Dara C. Drawbridge, Gretchen O. Landry and Thomas Grisso in Criminal Justice and Behavior
Footnotes
Authors’ Note:
The authors thank the administrators, probation supervisors, and officers at the study sites, who shall remain unnamed to preserve confidentiality. We also thank our research associates, Cassandra Beinemann, Christina DeSimone, Max Christensen, Andrew Nickerson, and Kathryn Riddinger. This research was funded by the Office of Juvenile Justice and Delinquency Prevention (PI: Vincent, 2014-JF-FX-0001) and the John D. and Catherine T. MacArthur Foundation as an extension of the Models for Change Initiative Research Network (105740-0).
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
