Abstract
The Positive Achievement Change Tool (PACT) is a popular risk-assessment tool used in juvenile justice systems across the country. This study is an effort at revalidating the PACT in the context of a large Southern city by utilizing days to recidivism in the context of survival analysis. The current research is concerned with whether the PACT’s risk classifications are effective in predicting recidivism. Findings indicate that the PACT effectively predicts recidivism better than chance and that recidivism is especially common early on in probation, with most incidents taking place within the first 200 days. However, the predictive validity of the PACT is not strong for the study population, and its implementation may benefit from context-specific reforms.
Introduction
The purpose of risk assessments in juvenile justice is to predict recidivism, and a valid tool will be successful in doing so (Baglivio & Wolff, 2019). The predictions made by risk assessments designate subjects as belonging to risk categories such that some are classified as being more at-risk for recidivism than others, and they make their forecasts based on things like social, family, and criminal history (Phillips et al., 2005). Informed by the results of risk assessments, juvenile justice officials make decisions on how best to treat the clients who come under their supervision. One common application of risk assessment data is to facilitate the least at-risk justice system clients being subject to the least intrusive sanctions and for more intensive sanctions to be applied to those who are most likely to reoffend (Skeem et al., 2017; Vose, 2016). To be clear, the data collected by risk assessments are commonly used to inform legal decisions made with respect to justice-involved youth (Kleeven et al., 2022), though this application is not necessarily intrinsic to the overall objective of predicting the severity of recidivism risk.
Modern risk assessments are based upon the Risk, Needs, and Responsivity (RNR) model, which helps them to account for several important concepts that are considered relevant to recidivism and client welfare (Campbell et al., 2018; Ho & Rocheleau, 2021). The Risk Principle refers to proportionality, such that interventions are matched to clients’ expected risks to recidivate (McCafferty, 2017), based on static and dynamic risk factors. Static risk factors include things that are stable such as demographic characteristics and behavioral history, while dynamic risk factors refer to changeable features like social attitudes, substance abuse, and other patterns of behavior (Shapiro et al., 2018). These dynamic factors are what the Needs Principle addresses, since problems here identify what treatment needs a client has, which in turn inform what interventions should be implemented to meet them. The Responsivity Principle addresses how interventions can be tailored to clients and includes consideration of things like learning styles, personal motivations, and cognitive abilities (Baglivio & Wolff, 2019; McKenzie, 2018).
Risk assessment has become ingrained within modern juvenile justice practice but is at risk of being perceived as a panacea to the myriad challenges within juvenile justice systems (Gottfredson & Moriarty, 2006). It is meant to meet systemic needs, but its failure can invite devastating consequences, including wasted money and misdirected punishment. Therefore, it is necessary to constantly interrogate whether risk assessment instruments are effective by empirically validating them (Arnold et al., 2018; Mulvey et al., 2016). The factors that influence risk in one population may not apply as well elsewhere, and the usefulness of individual items in an assessment may also be variable across populations, geography, and time (McCafferty, 2016; Mulvey et al., 2016; Skeem et al., 2017).
Moreover, demographic characteristics may interact with risk assessments in unforeseen ways. For example, instruments may fail to account for cross-cultural exposure to risk factors and thus attribute a higher likelihood of recidivism to subjects from minority backgrounds (Schmidt et al., 2020). Similarly, gender is known to predict delinquency, with male youth being more likely to recidivate regardless of their age (Farrington, 1986; Gottfredson & Hirschi, 1990; Moffitt, 1993), which can complicate risk assessment by assigning higher levels of risk to female subjects in an attempt at gender neutrality (Hamilton et al., 2019). Most assessments are thought to be white- and male-referenced, and tools may be built with those kinds of justice system clients in mind (Hilterman et al., 2016; Moore & Padavic, 2011). To the extent that this is true, it might facilitate the well documented racial disparities in outcomes that exist within juvenile justice (Leiber & Fix, 2019), as well as slighting female justice system clients who may have different needs than their male peers (Ho & Rocheleau, 2021).
Time is also a critical aspect to consider when investigating recidivism risk and its assessment. Research on adults has shown that the months immediately following release from prison are especially unstable and that the risk of recidivism tends to stabilize and diminish after that (Huebner & Berg, 2011). Further, probation clients’ perceptions of the legitimacy of sanctions can be variable with time, which can have implications for their propensity for compliance with various supervision expectations (McNeill & Robinson, 2013). Time clearly matters in the context of adult criminal justice, and periods of heightened recidivism risk are expected to be found here for juveniles.
In short, there never comes a point beyond which validation and/or revalidation of risk assessment tools becomes unnecessary, particularly when such tools are used on large populations for which they were not originally designed. The purpose of this research is to help to meet these needs, which will be done by examining the Positive Achievement Change Tool and how it has functioned within the juvenile justice system in a major Southern city in the US.
The PACT
The PACT focusses on the so-called “Big Eight” recidivism indicators, which are education/employment history, family relationships, emotional and mental health, leisure/recreation, peer relationships, substance abuse, criminal orientation and thinking, and residential stability (Andrews & Bonta, 2010). Proceeding from these, the PACT results in three scores for each juvenile it assesses, which measure Social History Risk, Criminal History Risk, and Overall Risk to Re-offend.
A juvenile’s Social History Risk score reflects criminogenic factors over 21 different indicators. While younger children necessarily have less social history, the field of risk assessment has long considered the social circumstances within the home as being meaningful for predicting aggression in children as young as 6, and thus its use here is consistent with general practices within risk assessment research (Enebrink et al., 2006). Social History Risk scores range from 8 to 18, with higher scores indicating that a juvenile’s social environment has more criminogenic factors. All items in the Social History section are weighted equally regardless of a youth’s age, and the behavioral questions consider behaviors occurring within the last 6 months. The total Criminal History Risk scores range from 0 to 31, with higher scores indicating more recidivism risk severity.
The PACT has two iterations, which are the PACT pre-screen and the PACT full assessment. The pre-screen employs 46 items, while the full assessment uses 126. The pre-screen is conducted by the probation department’s court intake officers, who utilize a semi-structured motivational interviewing method to gain information from both the subject and their parents (when possible). Motivational interviewing involves a counselor and a client (Rollnick & Miller, 1995), in this case an intake officer and a juvenile, and its goal is to avoid conflict while learning of a client’s static and dynamic risk factors.
Motivational interviewing also aims to provide a constructive discussion about how the subject can change their attitudes and behaviors to be more pro-social (Rollnick & Allison, 2004). This process involves the intake officer asking open-ended questions, showing affirmation, and promoting reflective listening via expressing empathy (Miller & Rollnick, 2002). All probation department court staff and juvenile probation officers in the study sample were trained in these methods and are required to take a 16-hour PACT/Case Plan Training before employment (Miller & Rollnick, 2002). The training familiarizes personnel with the techniques and purposes of motivational interviewing so they can obtain information from youth for use in the PACT pre-screen and amended PACT pre-screen risk assessment (Miller & Rollnick, 2002). The intake officer or juvenile probation officer must complete a pre-screen during the intake process, and an amended pre-screen 10 days before the youth appears in court for the alleged criminal offense or offenses which brought them into contact with the juvenile justice system. It is following these interviews that the PACT pre-screen combines domain scores to generate the Social History Risk and Criminal History Risk scores used to create the final classification of risk.
PACT Validity and Research Questions
Many studies have attempted to validate risk assessment instruments, and findings vary by instrument and population (Schwalbe et al., 2006). Assessments have often been shown to be effective to some degree better than chance (Vose, 2016), but there is variation in how well a given tool performs, and with respect to which population (Holtfreter & Cupp, 2007). For example, studies have revealed substantial variation in tool effectiveness between counties in the same state (McCafferty, 2016), across racial lines (Andretta et al., 2019), and by gender (Pusch & Holtfreter, 2018). Risk assessments seem typically to have higher predictive validity for white, male offenders and often perform less well for other populations (Campbell et al., 2018). Metrics like education, mental health status, living arrangements, and employment history seem to be predictively robust across jurisdictions, but other indicators like peer influence and criminal history have demonstrated more sporadic utility depending on jurisdiction and population (McCafferty, 2016).
Though the PACT was developed in Washington State by Barnoski (2004), it has been adopted in some form (e.g., the YASI, Back on Track) in over 20 states (Andrews & Bonta 2010; Hamilton et al., 2019), including the one in which the city studied here is located. Given the proliferation of the tool’s use, it is necessary to investigate whether the PACT is effective in a densely populated urban environment for which it was not initially developed. Accordingly, the first research question of this paper is whether the PACT is predictively valid in the city studied here. This question will be answered concerning Overall Risk to Reoffend and whether the risk designations are valid such that juveniles who are assigned high-, moderate-, and low-risk recidivate at the predicted frequencies relative to one another. The paper’s second research question is whether there are any identifiable time points that seem especially relevant to recidivism risk across the PACT’s risk designation levels.
This research includes two corresponding hypotheses. Because contemporary risk assessment tools prioritize protection of the public from recidivism and emphasize effective treatments for clients (Andrews et al., 2006), the first hypothesis is that the PACT will be predictively valid to some degree and that the risk designations will prove valid relative to one another. Also, since time-varying factors such as living and employment conditions have been suggested to be important to recidivism (Scott et al., 2014), the second hypothesis is that there will be approximate periods during which recidivism risk is elevated.
Current Study
Earlier research has demonstrated that the PACT has moderate predictive utility across racial groups, gender, type of referral event (e.g., misdemeanor, felony), and jurisdictions (Baglivio, 2009, 2015; Baglivio & Jackowski, 2013; Hamilton et al., 2015; Martin, 2012; Winokur-Early et al., 2012). The current study is novel due to the fact it is the second study besides Hutchins’s (2019) research to explore the PACT pre-screen assessment’s ability to predict time till recidivism among a sample of Texas juvenile justice-involved youth, and the first do so by tracking subjects for 36 months rather than 12. Further, the study population examined here is very large relative to some other studies (Hempel et al., 2013; Schwalbe, 2007).
Method
Sample
A PACT pre-screen was completed on every youth who was referred to juvenile probation within the city, except when the juvenile probation department closed the cases at the intake point and there would be no court action. This study utilizes a retrospective convenience sample that included subjects who were administered the PACT pre-screen as part of community supervision between March 7, 2017 and March 7, 2019. This allowed for those youth who had the last PACT pre-screen or amended PACT pre-screen on March 7th, 2019, to have an ample amount of time to recidivate, since the end of the observation period was March 7th, 2020.
The rationale for using the PACT pre-screen or amended PACT pre-screen was to examine predictive utility for youth who would not have received post-court interventions due to the pre-screen assessment being conducted during the intake procedure. Also, only juveniles who resided within the county at the time of their risk assessment were included in the analyses because the county-specific recidivism data used here did not allow for anything else. Further, in Texas, youth can be sentenced to a determinate or an indeterminate sentence, meaning the youth may start on juvenile probation, or be sentenced within the Texas Juvenile Justice Department (TJJD) for a determinate petition. Before a juvenile reaches the age of 19, they can be transferred to the adult system, including prison, parole, and adult probation to complete the remainder of their sentences. The movement of juveniles through both the adult and juvenile justice systems would make it prohibitively difficult to track youth who had been released as an adult outside of the juvenile justice system, and so such cases were not included here. The final sample consists of 2,758 subjects.
Measures
Dependent variable
Time until violations of probation stemming from a new delinquency charge is the dependent variable. The recidivism information used here was obtained from the county-wide database, and it is important to again note that information about adult recidivism and recidivism occurring outside the county was not tracked through this system and is therefore not included here. For this study, recidivism of juveniles aged 10 to 17 years and time to recidivism are tracked. Only violations of probation (VOP) for a new delinquency charge and misdemeanor B or more severe charges were included as recidivating events (e.g., a new criminal offense). The rationale for not including status offenses such as class C misdemeanors or CINS (conduct in need of supervision) offenses is because the PACT was created to predict new criminal charges rather than status offenses. This approach is consistent with previous research conducted by Martin (2012), who utilized misdemeanor offenses and probation violations for a new criminal charge as the dependent variable to assess the PACT’s ability to predict time to recidivism. The survival time had a mean of 416.00 days (SD = 239.00) and a range between 2 and 1,046 days, as indicated below in Table 1.
Descriptive Statistics for All Variables.
Note. N = 2,758. M = Mean; SD = Standard Deviation.
Yes.
Independent variables
The independent predictor variables included in this study were race/ethnicity, gender, PACT Social History Risk to reoffend, PACT Criminal History Risk to reoffend and PACT Overall Risk to reoffend. In this study, the last PACT pre-screen before the recidivating event was used and included in the analysis as the independent predictor variable. The sample included one assessment for each juvenile. Therefore, some juveniles in the dataset had the first PACT pre-screen as their primary indicator variable, and other juveniles had the re-assessment as their primary indicator used for analysis.
Analytic Strategy
The Receiver Operator Characteristic (ROC) post-estimation technique was used here to quantify the probability that a randomly selected recidivist will have a higher score on the PACT pre-screen instrument than a randomly selected non-recidivist (Rice & Harris, 1995). This probability is constructed by measuring the area between the “true-positive rate” (the rate at which the PACT pre-screen accurately predicted recidivism) and that of the “false-positive rate” (the rate at which the PACT pre-screen predicted recidivism which did not occur), and this is what is known as the Area Under the Curve (AUC; Barnes et al., 2016; Mossman, 1994; Rice & Harris, 1995).
AUC scores could theoretically cover a range from 0.0 (no predictive validity, the model is always wrong) to 1.00 (perfect predictive validity, the model is always correct), and are interpreted as an effect size (Baglivio, 2009, 2015; Baglivio & Jackowski, 2013; Winokur-Early et al., 2012). This means that the larger the AUC value, the more accurate the tool is in its predictions (Mossman, 1994; Rice & Harris, 1995, 2005). Guidelines for establishing AUC cut points and interpreting effect sizes suggest that AUC values of 0.556 to 0.638 are considered small/weak, values of 0.639 to 0.713 are considered medium/moderate, and values of 0.714 and above are considered large/strong effects (Rice & Harris, 2005).
The predictive validity of the PACT is measured by Harrell’s C values, which are equivalent in survival analysis to the AUC/ROC measure used elsewhere and entail equivalent cut-points (Glen, 2020). Harrell’s C-index (also known as the concordance index) is a measure of goodness of fit for models that produce risk scores (Harrell et al., 1982). It is often used to evaluate risk models in survival analyses in which censored data is common. If risk models for the PACT are predictively valid, juveniles who had shorter survival times should have higher risk scores, and those who had longer survival times should have lower risk scores. This analysis will reveal the predictive validity of the PACT and notable timeframes for recidivism risk within the study population.
Though survival analysis is commonly used in medical research (Fawcett, 2006; Obuchowski, 2005), it is applicable to other settings. Here, it is used to analyze juveniles’ recidivism. Cox regression was also employed to examine the predictive validity and relative contribution of the PACT pre-screen Overall Risk to Reoffend, Criminal History Risk, and Social History Risk in predicting time until recidivism during the 36-month follow-up period. The dependent variable is time at risk until recidivation/end of the study period, and is based on the cumulative survival function, meaning the proportion of subjects on probation who “survive” the study period by not recidivating. The risk period began when the juveniles received their initial sentence to probation and continued for 36 months. Time to recidivism was measured in days from the initial sentence to probation, ranging from 0 to 1,095. For non-recidivists, time at risk was calculated according to the juvenile’s court date and the end date of the study (March 7, 2020), or if the juvenile’s probation sentence ended before then.
The hazard ratio reported here indicates how changes in the dependent variable either increased or decreased by one unit of the predictor variable or if the predictor variable is nominal compared to the reference category. The hazard ratio size of effect indicates the contribution that each covariate provides, meaning that larger hazard ratios indicate a greater influence of specific covariates. Because Cox regression is a non-parametric analysis, it does not make any specific assumptions about the distribution of the independent variables, although the independent variables should be correlated with the dependent variable. Youth who recidivate after the end of the follow-up period in the sample are assumed to have not failed because of the censoring characteristic. Censoring can induce Type I errors, because of the potential to misidentify those who fail after the end of the study timeframe. Randomly censored youth have specific end dates within the dataset depending on when the youth was arrested, and the probation sentence termination fell within the total observation study period. The time at the final date of the study is categorized as random and right censored. According to Allison (1984), right censoring is seen as non-informative, and does not influence time until the observed event within the data and the presumed hazard ratio. The survival analysis regression allows for a time of entry into the study, which is included to account for any biases induced by censored data (Allison, 1984).
Results
Descriptive Statistics
During the study period, 2,758 juveniles had an initial PACT pre-screen and were supervised on probation. About 34.88% of the sample, or 962 juveniles, recidivated during the study period. 7.61% (n = 210) of the sample were white, 45.83% (n = 1,264) were Hispanic, and 45.94% (n = 1,267) were African American. The youth in the sample were categorized as one race/ethnic group because the county data examined here did not allow for blended racial categorizations. The racial groups in this dataset are: Asian, African American, Hispanic, American Indian, Other, Unknown, and White. None of the youth in the current dataset were reported as unknown. Further, the dataset categorizes Hispanic as a race rather than an ethnicity.
Males made up 79.66% (n = 2,197) of the sample. The juveniles in the sample had an average age of 14.82 (SD = 1.20). Further, there was a roughly even categorical split regarding risk level, with 33.72% (n = 930) being low-risk, 33.14% (n = 914) being moderate-risk, and another 33.14% (n = 914) in the high-risk level. Criminal History Risk was not as evenly split, with only 8.67% (n = 239) scoring in the low-risk level. A majority of 60.55% (n = 1,670) scored in the moderate-risk level, and another 30.78% (n = 849) scored in the high-risk level. For Social History Risk, 39.49% (n = 1,089) scored as low-risk, another 39.20% (n = 1,081) scored in the moderate-risk level, and the remainder of 21.32% (n = 588) scored in the high-risk level. The number of days that passed from the time a youth had been assessed via the PACT pre-screen or had the amended PACT pre-screen assessment conducted until a youth was sentenced to probation was calculated (M = 45.73 days, SD = 62.11). Descriptive statistics are displayed in Table 1.
Harrell’s C
Table 2 displays the Harrell’s C between the Overall Risk to Reoffend and its ability to predict time until recidivism, the end of the study period, or the subject’s completion of their probationary term. The results indicated that the PACT risk score predictions were weak (Overall Risk: 0.60, p < .001; Criminal History Risk: 0.52, p < .001; and Social History Risk: 0.52, p < .001) (Newson, 2010).
Results of Harrell’s C analyses for PACT Risk Levels.
Note. N = 2,758. 95% confidence intervals(CIs) shown in brackets.
p < .001.
Area Under the Curve (AUC)
Displayed in Table 3, the effect sizes across the three domains of the PACT scores were also weak (Overall Risk: AUC = 0.58, p < .01; Criminal History Risk: AUC = 0.55, p < .01; and Social History Risk: AUC = 0.57; p < .01), which yielded substantively similar results in the Harrell’s C statistics.
Results of AUC Analyses for PACT Risk Levels (N = 2,758).
Note. N = 2,758. AUC = Area Under The Curve; SE = Standard Error; 95% confidence intervals (CIs) shown in brackets.
**p ≤ .01.
Multicollinearity was also addressed within the multivariate model. The variance inflation factor (VIF) was utilized to assess multicollinearity within the regression model. If a VIF is higher than 5 for any of the independent variables in the model, there is an issue with multicollinearity (Mehmetoglu & Jakobsen, 2016). The highest VIF value in Model 1 was 3.81 (which was the race variable), and the other variables had VIF values ranging from 1.00 to 3.81. Covariate-specific and global tests for independent variables were included in the Cox regression analysis. Based on the individual chi-square analyses and the global chi-square analysis, there was no evidence that the proportional-hazard assumption was violated.
Survival analysis
The results of the Cox regression Model 1, shown in Table 4 below, indicate that Hispanics were more likely to reoffend within the follow-up period. Hispanic subjects had a risk of recidivating 1.61 times greater than their white peers. Similarly, African Americans were 1.98 times as likely to recidivate as whites. Males were 1.89 times more likely than females to recidivate. Juveniles at moderate-risk to reoffend were 1.38 times as likely to recidivate as those where were low-risk on the PACT pre-screen. When the subjects in the sample scored high on the PACT pre-screen, their risk of recidivating increased by 1.70 times those at low risk.
Cox Regression Analysis Utilizing PACT Overall Risk to ReOffend, Gender, Race/Ethnicity as Main Predictor Variables.
Note. SE = Standard Error; 95% confidence intervals (CIs) shown in brackets.
Yes.
p ≤ .05. ***p ≤ .001.
Figure 1 below illustrates the success of the PACT’s risk classifications. All three show the expected results that low-risk juveniles survived the study period most successfully, followed by those considered to be at moderate-risk, with high-risk subjects recidivating most quickly and most frequently. These results indicate some validation for the effectiveness of the PACT risk assessment tool.
Kaplan-Meier survival analysis for overall risk to reoffend.
The Kaplan-Meier Curves shows the survival of subjects on probation over time, including when subjects dropped out of the study or were studied for different lengths of time due to different probation sentences. The interval measured is days survived until a qualifying recidivating event, and the survival probability is defined as the number of subjects surviving divided by the number at risk. Juveniles who recidivated, dropped out, or who were otherwise removed are not counted in the at-risk category. Censored cases are those that are lost before the conclusion of the study period and are not counted in the denominator when calculating the survival probability (Rich et al., 2010).
The vertical axis represents the estimated probability of survival for the hypothetical cohort study period, but not the true percentage of juveniles who survived. The horizontal axis represents the study period time, in this case 1,095 days (36-months). All cases start at the top of the vertical axis (e.g., y-axis; 1.0% or 100%), which establishes the proportion of the sample who has not experienced the “death” event (i.e., recidivism). The length of each of the three horizontal lines represents the survival duration for that interval (e.g., known survival time interval), and the survival estimates of a given duration are representative of the cumulative probability for the study period. The censored cases are indicated in the Kaplan-Meier curve as tick marks which do not cause the termination of the survival time for other juveniles in the sample (Rich et al., 2010).
Figure 1 illustrates the Kaplan-Meier Curves and risk table, and displays youth who recidivated during the study period. The numbers in the parentheses below the survival curve signify youth who recidivated, whereas the numeric values without parentheses are youth who did not. About 28% of the total number of juveniles in the sample (511 out of 2,758) recidivated within 200 days of starting probation. This is, in turn, 82% of the total number of juveniles who recidivated (511 out of 962).
Discussion
The first hypothesis predicted that the PACT would be predictively valid to some degree and that the risk designations would prove valid relative to one another. This hypothesis was supported by the results, but not strongly. To reiterate, a Harrell’s C value of greater than 0.5 is better than chance (Harrell et al., 1982), which means that the PACT pre-screen does work to a degree for all three of its scores. However, for predictive validity to be considered “good,” the Harrell’s C should be above 0.7 (Glen, 2020). These results fall well short of that score across all three dimensions, though the Overall Risk to Reoffend comes closest at 0.60. The AUC statistics for the PACT pre-screen Criminal History Risk level, PACT pre-screen Social History Risk level, and PACT pre-screen overall risk level are all weak/small effects, according to Rice and Harris (2005).
Previous studies that have validated the PACT in Florida have reported AUC values of 0.59 (Baglivio, 2009), 0.59 (Baglivio & Jackowski, 2013), and 0.63 (Winokur-Early et al., 2012). There are three published studies to date that have explored the predictive utility of the PACT within a sample of Texas youth (Hutchins, 2019; Martin, 2012; McKenzie, 2018), though only Hutchins and Martin employed survival analysis. Martin (2012) utilized a 12-month follow-up period and combined 912 full PACT assessments with 2,205 pre-screen assessments for a total sample size of 3,117 youth. Martin (2012) also found an AUC value of 0.60 for the PACT’s Overall Risk to Reoffend. Consistent with these findings, the Overall Risk to Reoffend was a significant predictor of future recidivism (Martin, 2012). More recently, Hutchins (2019) utilized only PACT pre-screen assessments to predict recidivism and reported an AUC value of 0.561, whereas the current study had an overall risk AUC of 0.58 using PACT pre-screen assessments.
These AUC values are lower than most previous studies (see Baglivio, 2009, 2015; Martin, 2012; McKenzie, 2018; Schwalbe, 2008), which could be because the PACT pre-screen had not previously been validated within the probation department before implementation. It could also be due to methodological differences, since previous studies (Baglivio, 2009; Martin, 2012) used a combination of PACT pre-screen assessments and full PACT assessments, which could increase the predictive utility of the PACT. Further, as already noted, prior studies used a shorter follow-up period (Andrews et al., 2012; Baglivio & Jackowski, 2013).
However, the risk categorizations do work well relative to one another, as shown by most of the hazard ratios. The PACT Overall Risk to Reoffend hazard ratios show that high-risk juveniles are more likely to reoffend than moderate-risk subjects, and low-risk juveniles recidivate least. This finding was consistent with work conducted by Baglivio (2009), who found similar results. Low-risk youth were less likely to reoffend compared to moderate-risk youth, and moderate-high risk youth were less likely to reoffend compared to youth who were at high risk to recidivate, which is a modest but fundamental validation of the tool’s use here.
The second hypothesis predicted that there would be approximate periods during which recidivism risk was elevated. Since most recidivism occurred prior to the 200-day mark, this hypothesis was supported, and the Cox regression analysis showed that each increase in risk classification (low-, moderate-, and high-risk to reoffend) was associated with an increase in observed recidivism. This is displayed in the Kaplan Meier estimates in Figure 1. The 200-day mark is also the approximate point at which the recidivation levels for moderate- and high-risk categories diverge. Several factors could explain this, including a differential decay in service effectiveness that begins to manifest at that time for the risk categories, or it could be that circumstances within the lives of subjects on probation start to assert greater influence then. Having over 80% of recidivism occurring within 200 days regardless of risk level establishes the necessity for juvenile justice practitioners within this juvenile justice department to be vigilant about matching the criminogenic needs of youth to effective treatment services early on, and if these needs are not met then rapid recidivism will remain likely (Luong & Wormith, 2011).
Limitations
It is important to consider the limitations of the current study and to comment on future research about the PACT pre-screen risk assessment. One such limitation is that there was a restricted range of recidivism information available due to geographical constraints, such that recidivism taking place outside the study jurisdiction was not captured. Also, information on a subject who committed a new offense but was processed through the criminal justice system as an adult was not available, and recidivism that occurred outside the county analyzed here was also not captured. Further, within this jurisdiction, juveniles can be processed and charged as an adult for certain offenses starting at age 17. Given that subjects aged 16 and above represented about 35% of the current sample, it is likely that recidivism was not fully captured for some of the sample.
Convenience sampling also limits the generalizability of the findings outside of the study population. This limitation is minimized somewhat by the need for location-specific validation, but it is worth mentioning. Also, there was no information collected on the level of training that individual intake officers had before implementing the instrument, raising potential questions of data reliability. The social history domain notably requires the intake officer to make subjective assessments based on their interactions and knowledge of the subject, so the PACT pre-screen requires a significant amount of professional judgment which is not accounted for in this study and may influence intake officers' overall risk ratings.
Further, there is a need to explore the quality of services provided to juveniles along with their levels of engagement with those services through the study period (Ashford & Gallagher, 2019). Future work should incorporate these dimensions as control variables to see if they improve the effectiveness of PACT scores over time and to see if participants had a lower risk of recidivism than non-participants. Finally, the first 200 days after the start of probation appears to be an especially fraught period, and juvenile justice agencies would be well-advised to pay close attention to their charges during this important timeframe.
Conclusion
In the city studied here, the PACT was effective in predicting recidivism, but not nearly to the degree that policymakers would hope for. This means that improvement in its deployment in the study jurisdiction is crucial, which could require more/better training for juvenile justice officers or better empirical assessment of which indicators are most useful in predicting recidivism within the county. It is not consistent with best practices to simply import a tool developed elsewhere without diligently and continually evaluating its effectiveness in a new environment and as applied to a novel population (Vincent et al., 2016). Properly implemented risk assessment instruments can help juvenile justice officials preserve taxpayer dollars by limiting residential placement to youth that require serious attention and pose the most risk to community safety (Andrews et al., 2006), as well as protecting that safety and the well-being of juveniles. The benefits of quality implementation of risk assessment are too substantial to overlook, and the evidence here suggests that there is meaningful room for improvement in that regard for this jurisdiction.
Further, attention should be paid to periods during which risks for recidivism are elevated, which these results indicate happens early on, within 200 days or approximately 6 to 7 months after the start of probation. These findings are consistent with research conducted by Baglivio (2014), who found substantial recidivism within 180 days. These finding could be explained by the tendency of youth to have shorter probation sentences (such as 6 months) or because of changes in their circumstances that aggravate their proclivity to offend during that timeframe. Examples of such changes that have been identified in the literature include reintroduction to antisocial peers who contribute to criminal thinking, relapses regarding substance use, and economic stress related to employment (Cohen & Vanbenschoten, 2014). Another possibly is that a youth’s criminogenic needs are not being matched well with treatments, since in such cases recidivism happens more quickly (Vieira et al., 2009).
The mission of juvenile justice can be made more successful with the proper implementation of well-designed risk assessment tools, but to do so requires ongoing validation efforts and an awareness that what works in one place and time may not be as effective in other contexts. This work is a contribution to those efforts, and it is hoped that with enough such scholarship the enterprise of risk assessment can be improved along with the lives of the vulnerable populations who are its subjects.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
