The Predictive Validity of the Post-Conviction Risk Assessment Among Federal Offenders

Abstract

In the U.S. federal court system, the Probation and Pretrial Services Office (PPSO) uses a tool known as the Post-Conviction Risk Assessment (PCRA) to assess offender risk and identify challenges offenders face while under supervision. This article evaluates the PCRA and its components to determine its usefulness as a predictive tool for evaluating risk. Overall, we find that the PCRA is an effective tool for classifying offenders as it is currently designed, achieving a level of predictive validity comparable with its competitors. Notably, we also find that the PCRA effectively differentiates offenders early in supervision terms, and that its predictive power diminishes as time under supervision lengthens. Finally, the strength of PCRA classification appears to vary with offense type. The PCRA performs well for some offenses including drug, violent, and property offenses, but provides less utility in reliably predicting less common offenses.

Keywords

risk assessment federal probation validation PCRA supervision

Introduction

For decades, supervision agencies have used risk assessment tools to gauge the risks and needs of offenders as they transition out of prison and return to society. These tools are an important part of the supervision model because they help supervision officers to identify which offenders are at the greatest risk of failure and provide effective mediums for officers to identify specific challenges that offenders face. Moreover, this strength of application drives researchers and practitioners to search for ways in which the accuracy and effectiveness of risk assessment instruments can be improved.

The federal courts are no exception. Although the U.S. federal judicial system has made use of various schemes for risk categorization since the 1970s, the recent growth of popularity in dynamic risk assessments has prompted the Probation and Pretrial Services Office (PPSO) within the Administrative Office of the U.S. Courts to develop a new, more expansive tool known as the Post-Conviction Risk Assessment (PCRA). Like its predecessors (i.e., the Risk Prediction Scale 80 [RPS-80] and Risk Prediction Instrument [RPI] described on page 8), the PCRA is meant to gauge offenders’ risks during the post-conviction supervision period. Unlike its predecessors, the PCRA is meant to improve effectiveness of supervision by identifying a greater number of risk, need, and responsivity factors (Andrews, Bonta, & Hoge, 1990), and by reporting observed changes to these factors over time.

After its development in 2009 to 2010, agency-wide implementation of the PCRA began in 2011. Since then, only a small handful of studies have assessed the effectiveness of the PCRA as it has been applied to cohorts of offenders coming into federal supervision (Cohen, Lowenkamp, & VanBenschoten, 2015; Lowenkamp, Holsinger, & Cohen, 2015). Nevertheless, establishing a body of literature as to whether and to what extent the PCRA is an effective tool for classifying offenders and offender risk is an important exercise; established tools, such as the PCRA, are often candidates for potential adoption by agencies looking to introduce, expand, or reform their use of risk assessments. To that end, the goal of this article is to assess the predictive validity of the PCRA as it has been realized in the field and to explore how various elements of the PCRA do or do not contribute to its utility in application. Our intent is to answer the following central research question:

Research Question 1: How well does the PCRA predict failure in the post-conviction period?

Our article also contributes to the existing literature by describing the utility of the PCRA in predicting rearrests throughout the length of a supervision term and for specific offenses.

Risk Assessment in Criminal Justice

The formal use of risk assessments as a means of identifying individuals at greater or lesser risk of committing crimes after release from jail or prison has been a part of criminal justice thinking since the early 1900s (Burgess, 1928). Since then, assessments have become a staple feature in the design and delivery of post-confinement supervision. From a public safety perspective, they provide a means of identifying offenders who require additional supervision, those who may benefit from additional treatment, and/or those who may benefit from other interventions designed to facilitate successful reintroduction into society. From a fiscal perspective, they allow agencies to target their resources toward the most effective uses and reduce ineffective spending. For these reasons, researchers are continually searching for better ways to identify and characterize offender risk and looking for ways to improve the predictive accuracy of risk assessment instruments used in that pursuit (Gendreau, Little, & Goggin, 1996).

At their core, most risk assessments are models that weigh factors associated with risk to stratify offenders into ordinal “risk categories” roughly corresponding to the probability of reoffending. Most models use some (or all) of the following factors as predictors: offending history, substance abuse, family relations, and peer delinquency (Schwalbe, 2008). Early assessments (also called first generation risk assessments) placed a heavy emphasis on intuitive judgments from law enforcement officers and correctional experts of the time (Bonta & Andrews, 2007). Assessments rooted in clinical judgment making had some appeal in that they are driven by the opinions of experienced professionals whose subjective judgment is relatively informed. In that sense, assessments had a sort of informal credibility and flexibility in their design and application. However, a significant drawback to this approach more generally is that it often lacked accuracy and reproducibility (Dawes, Faust, & Meehl, 1989). Conversely, researchers have found that actuarial models based on quantitative and statistical principles consistently outperform models based on clinical judgment (Gottfredson & Moriarty, 2006). Over time, the use of first generation tools has given way to a second generation of actuarial instruments, emphasizing evidence over intuition.

Actuarial models of risk assessments have proven to have several advantages to practitioners. First, they can be consistently applied over time and over offenders by a large set of individuals administering an assessment. They also tend to be significantly less resource intensive. Although clinical tools require highly trained individuals to administer and evaluate the results of an assessment, actuarial tools typically have a shallower learning curve and require less time to administer and score (Dawes et al., 1989). In terms of implementation cost and predictive utility, actuarial instruments are functionally superior and more economically viable, though implementation itself may be also vulnerable to sources of error in certain settings and circumstances (Lowenkamp, Holsinger, Brusman-Lovins, & Latessa, 2004; Lowenkamp, Latessa, & Holsinger, 2004). At the same time, second generation instruments rely exclusively on risk factors that are static in nature—that is, factors that are immutable such as criminal history and substance abuse history. And although static factors of risk are extremely useful in the overall prediction of risk, they do not take advantage of information as it is changing over time, where those changes are also associated with the risk of reoffending (Bonta & Andrews, 2007).

The introduction of dynamic factors (i.e., factors changing over time—employment status, treatment status, etc.) into actuarial models of risk assessment has led to the development of a third generation of instruments. These instruments not only emphasize the changing needs and circumstances of offenders in predicting risk, they are intended to act as a vehicle for correctional staff to actively reduce risk by tailoring supervision efforts to meet the needs of offenders. In that way, the application of these third generation instruments reflects a larger, more general movement toward the risk-need-responsivity (RNR) model as an approach to supervision (Andrews, Bonta, & Hoge, 1990).

Finally, and most recently, a fourth generation of instruments has begun to emerge, the aim of which has been to improve upon third generation instruments by expanding the breadth of factors used to measure the risks, needs, and responsivity of offenders and allowing officers to tailor interventions based on an offender’s learning styles and abilities (Bonta & Andrews, 2007; Johnson, Lowenkamp, VanBenschoten, & Robinson, 2011; Lowenkamp, Johnson, Holsinger, VanBenschoten, & Robinson, 2013). In addition, these instruments integrate an offender’s criminogenic needs and responsivity factors into a probation officer’s case management system, allowing for the more efficient implementation of supervision or treatment (Andrews, Zinger, et al., 1990). Although this expansion is rightly motivated by purposes other than pure risk prediction (e.g., in the interests of effective treatment allocation), current research is also considering what statistical trade-offs (if any) are made in risk prediction between instruments that emphasize static vs. dynamic factors (Yang, Wong, & Coid, 2010). Moreover, supervision objectives driving these expansions suggest a rationale for periodic reassessments (also growing in popularity) that provides independently useful simple prediction (i.e., reassessments are a way to monitor treatment needs and offender status).

Risk Assessment in the U.S. Federal system

Past Instruments

The U.S. federal judicial system has made use of risk categorization since the 1970s, and has adopted official evidence-based models of risk assessment since the 1980s. In 1982, the Federal Judicial Center identified and tested four existing models of risk assessment to serve as a base for what would be the first instrument implemented on a national level, the RPS-80. A decade later, the instrument was redesigned to improve predictive accuracy. This new tool, known as the RPI, was created using a multivariate analysis of supervision cases and found to be a more accurate model than its predecessor and competitors. This new model scored offenders on a scale of 0 to 9 based on age at the start of supervision, number of prior arrests, use of a weapon in the instant (i.e., conviction) offense, employment status, history of drug and alcohol abuse, prior absconding, education, and living situation (i.e., with a spouse and/or children) at the start of supervision.

Until 2011, the RPI remained the primary risk assessment tool for post-conviction supervision used by PPSO. However, the rising popularity of dynamic risk assessments prompted U.S. federal courts to develop a new tool that expanded assessments to reflect changes in offender circumstances and offer more targeted recommendations for the case supervision plan. This latest tool called the PCRA is the subject of this article.

PCRA Construction and Validation

The process of constructing and validating the PCRA began in 2009, with the results of this exercise published in Johnson et al. (2011). For this exercise, the authors conducted multivariate analyses using data from federal presentence reports, existing risk assessments, criminal history record checks, and other data contained in the agency’s internal database to identify predictive variables. Overall, their analyses examined three sample groups—one group for construction and two groups for validation. Their samples were derived from offenders released to federal supervision between October 2005 and August 2009, consisting of roughly 185,000 offenders in total, of which just more than 100,000 were used for analysis.

The selection of initial testing items came from a variety of sources, including existing research, input from officers, existing risk assessment instruments, and recommendations of a small set of districts directly involved in tool development. Items were tested to see how well they predicted rearrests after the start of supervision using logistic regression models. The time frame for rearrests covered in their analyses extended as long as 60 months after the start of supervision. Significant predictors were identified from the model and assigned 1 to 3 points based on their correlation with rearrests. The authors summarize the PCRA’s predictive ability using the area under the curve–receiver operating characteristics (AUC-ROC). They found that the AUC exceeded 0.7 in all validation samples, suggesting that the instrument had good predictive validity in both the short-term and longer term follow-up periods. In a subsequent paper, Lowenkamp et al. (2013) extended their analysis to assess the interrater agreement of the instrument and validate predictions using a sample of assessments completed by probation officers. They found that the instrument predicted arrests reliably from assessment results based on administrative data or officer-completed assessments. They also found high rates of interrater agreement, ranging from 87% to 100%.

Method

PCRA Design

Overall, the PCRA is divided into two main components: (a) an officer assessment and (b) an offender assessment. The officer assessment is a 30-item assessment that is administered, as the name suggests, by the individual probation officer supervising the offender. Assessment items capture a mixture of static and dynamic risk factors and are collected at regular intervals. For the officer assessment, officers are asked to conduct an initial assessment at the outset of supervision, with no restriction placed on the timing of reassessments. Understandably, this creates considerable variation in the actual timing of PCRA administration, though, in practice, officer reassessments typically coincide with the timing of case plan reviews, occurring at 6-month and 12-month intervals. For this assessment, officers collect data across six structured domains of risk: criminal history, education and employment, drug and alcohol issues, social networks, cognitions, and other factors. In addition, officers collect data about responsivity factors that are not related to risk, but instead predict an offender’s receptiveness to treatment.¹

Within each domain, specific items are recorded by officers. Some items are used to compute the final risk assessment score (i.e., PCRA score). We refer to these as “scored items”; there are 15 such items. There are also some items that are collected by officers but which are not used for computing PCRA scores. We refer to these as “unscored items,”² Most scored items are assigned a value of either 1 or 0; however, two items, “age at supervision start” and “number of prior arrests,” can score values up to 2 and 3, respectively. The appendix describes each of the PCRA items appearing in the officer assessment along with the scoring value for the final PCRA score. Overall, the highest possible PCRA score from the officer assessment is 18. Because the officer assessment is conducted multiple times over a supervision period, the PCRA score has the flexibility to change over time. Finally, offenders are placed into risk classifications depending on the overall PCRA: “low risk” for scores from 0 to 5; “low/moderate risk” for scores from 6 to 9; “moderate risk” for scores from 10 to 12; and “high risk” for scores from 13 to 18.

The goal for this study is to test whether and to what extent the constructs and classifications set forth by the PCRA are an effective tool for predicting offender failure under supervision. To that end, this analysis focuses solely on assessing the predictive value of the scored items from the officer assessment. In addition to scored and unscored officer items, offenders also complete self-assessments at the outset of supervision, followed by additional reassessments conducted on an annual basis. These offender assessments are not utilized in determining PCRA classifications and we do not discuss them further.

Data Sources

Data for this analysis come primarily from data housed in the internal case management database for the PPSO, known internally as Probation and Pretrial Services Automated Case Management System (PACTS). PACTS data comprise 521 tables covering post-conviction and pretrial supervision; they are complex and wide-ranging. Some tables describe characteristics of supervision terms such as start and end dates. Others provide definitions, for example, codes representing race/ethnicity categories. Most importantly, PACTS records a host of information that is relevant to the analysis including data on offender assessments/characteristics, sentencing outcomes, conditions of supervision, programming and treatment and intermediate outcomes such as offender employment, payment of fines and changes in substance use. PPSO also augments these internal data with data from external sources including data from the Federal Bureau of Investigation (FBI). We discuss the use of these data in the following sections.

Sample of Offenders

Our analytic sample is derived from admission cohorts of offenders received into federal community supervision from the start of the fiscal year (FY) in 2005 (October 1, 2004) to the end of the FY in 2013 (up to September 30, 2013).³ Because our interest is limited to evaluating the PCRA assessment, we exclude all offenders who never receive a PCRA while under supervision. Thus, the bulk of our sample (n = 171,131 or 79%) is made up of offenders received into supervision after implementation of the PCRA began in FY2011; the remainder (n = 44,905 or 21%) are offenders who were admitted before FY2011 and received a PCRA sometime after supervision start. We exclude offenders when records of their initial arrest (i.e., the arrest that led to their initial incarceration in federal prison) are missing (n = 18,220).⁴ In addition, we exclude offenders with other charges pending (n = 22,279)⁵ and offenders serving terms of supervision not described as “probation” or “terms of supervised release” (n = 10,905).

Overall our sample is comprised of 216,036 total PCRAs administered to 139,239 unique offenders across 141,446 individual terms of supervision.⁶ Twenty-one percent of these terms began prior to FY2011, 26% began in FY2011, 33% began in FY2012, and the remaining 20% began in FY2013. Approximately 58% of the terms in our sample were ongoing as of the end of our observation window (March 15, 2014), 29% had ended with a successful completion and the remaining 13% ended either by revocation, death or transfer. For ongoing terms, the median time observed was 3 years (SD = 1.2); for successfully completed terms, median time observed was 1.8 years (SD = 1.1). Among those rearrested, the median time observed until rearrest was slightly more than 1 year.

Arrest Data

Arrest data for this analysis come from data sources outside of PACTS. Specifically, arrest data are assembled by PPSO from arrest records extracted via a web-based application used by probation officers to access criminal history information for supervised offenders, known as Access to Law Enforcement Systems (ATLAS) and from the FBI’s Computerized Criminal History (CCH) system. Arrest strings contained in the records themselves are parsed and converted into 10 broad offense categories used by the Administrative Office of the U.S. Courts: Violent, Property, Drug, Sex Offense, Firearms (e.g., possession of weapon), Escape/Obstruction (e.g., perjury), Public Order (e.g., drunk and disorderly), Technical (e.g., failing to appear in court), Immigration (e.g., facilitating an illegal entry), and Other (e.g., refusal to pay court fines).⁷ The length of observable criminal history available from these arrest data extends to March 15, 2014.

There is no single way for defining recidivism; however, PPSO defines recidivism as rearrest for new criminal activity.⁸ Thus, for our analysis, we define the recidivism as a rearrest for a serious offense occurring within 6 months of the date the PCRA was administered.⁹ Only those PCRAs administered to an offender under active supervision are considered, so that every offender is at liberty to be rearrested starting on the day they received the PCRA. Offenders who have no interruptions to supervision over the 6 months following a PCRA are considered a success. Offenders who are rearrested for a serious offense within 6 months after a PCRA but who would have otherwise been observed for the entire window are considered a failure. Offenders who are not rearrested for a serious offense within 6 months, but who end supervision before 6 months have elapsed (either because they revoke [n = 2,588] or their term expires [n = 761]), do not enter the computation.

The 6-month window is chosen for two reasons. First, because the PCRA is a relatively new assessment, recidivism measures defined over a follow-up that is too lengthy significantly limits the size of our available sample and the resulting power of our analysis. Second, the PCRA itself appears to be administered frequently, on 6-month and 12-month windows, where reassessments provide the most up-to-date available information. Thus, it makes sense to focus on how the PCRA improves near-term predictions, rather than predictions over longer time horizons where information is subject to change.¹⁰ It is also important to note that models using only officer items define the start of the follow-up window using the assessment date from the officer assessment. Conversely, models using both officer and offender items define the start date as the latest assessment date between the officer–offender assessment pair.¹¹

PCRA Assessment Data

Data for the officer and offender assessments are taken directly from PPSO’s internal PACTS database. Several features of the PCRA make it a challenging measure to apply in analysis. First, the use of the PCRA does not have a long history. Initial implementation of the PCRA as an instrument only began as long ago as October 2010, and universal application by districts was arguably not achieved until the following year. This makes sample size a challenge in some instances of our analysis. Taken together with the fact that outcome data (i.e., arrest data, discussed above) only extend as far as March 2014, the PCRA as a predictor cannot easily be evaluated for long supervision terms or long follow-up (i.e., rearrest) windows.

Second, most PCRAs are not completed until shortly after supervision has begun, several weeks in many cases. In theory, recording lags mean that offenders who reoffend very quickly are not represented in our analysis, whereas such a group could be captured if PCRAs were completed for them before supervision started. For this analysis, 20,244 offenders were dropped because they reoffended before a PCRA was completed for them.¹² This type of selection implies that (a) our analysis understates the true rates of failure for the larger population of federal offenders overall, and (b) our conclusions may or may not generalize to the unmeasured group. To the extent that this is a group we want to represent, this is a limitation of our analysis.

A third challenge for analysis is that the PCRA is administered multiple times over the course of supervision, with data elements that are updated to reflect changing information. This differential timing means that we must consider (a) how to incorporate changing information during a supervision term, and (b) how the predictive power of PCRA assessments varies according to when it is administered during a supervision term.¹³ Moreover, new evidence suggests that the number of offenders with changes in scores over time is not trivial and that such changes positively predict subsequent success or failure in the supervision period (Cohen et al., 2015). Our solution is to stratify our analysis, grouping PCRAs together according to when they were administered within a supervision term. We create a total of six groupings: PCRAs administered (a) within the first 3 months, (b) between 3 and 6 months, (c) between 6 and 12 months, (d) between 12 and 24 months, (e) between 24 and 36 months, and (f) after 36 months. In all cases, the upper bound is inclusive to the group. In cases where more than one PCRA was administered within a window (n = 11,555, or ≈5%), we take only the first PCRA. Most of these instances were cases for the unbounded group (i.e., PCRAs administered after 36 months).

Data Analysis

To investigate how well the PCRA predicts failure, we compute the average rate of rearrest by groupings based on risk assignments. This summary metric is both straightforward and informative and is computed over the entire sample of offenders given PCRA assessments. If the PCRA instrument is effective, then greater associated risk should be significantly correlated with higher observed rates of failure. In addition, we summarize the accuracy of PCRA scores using a commonly reported metric based on ROC analysis. ROC analysis is used in the assessment of classification models and works by comparing rates of true positive classification with rates of false positive classification at classification cutoff values. The area under the ROC curve (also called the AUC) summarizes how accurately the classifier (i.e., the PCRA score) separates observed failures from observed successes and provides a useful tool for assessing and selecting optimal classification designs (Mossman, 2013).¹⁴

To assess the contributions of individual PCRA items, we estimate a nonlinear logistic regression with scored PCRA items as covariates. Specifically, we estimate:

logit (\Pr (Y_{j} = 1)) = \sum_{i = 1}^{I} β_{i} (P C R A_{i j}) + α

where Y_j is the dependent measure (= 1) if offender j was rearrested while under supervision and within 6 months of the start of supervision, PCRA_ij is the recorded score for the ith PCRA item for offender j, β_i is the estimated coefficient for the ith PCRA item, and α is a constant term.

Results

As described earlier, we stratify our results into groupings separated by when each PCRA was administered (i.e., when during supervision). We do this for two reasons. First, the PCRA itself is a dynamic assessment and we want to allow for this flexibility in our evaluation. Second, because the latent characteristics of offenders under successful supervision for long periods are not likely to be the same as those continuing only a short time, it seems unlikely that the PCRA has the same overall predictive power throughout the entire course of supervision. Table 1 summarizes the number of offenders with complete PCRAs, stratified by risk level (in the rows) and the window of time since supervision in which a PCRA was administered (in the columns). It is important to note that because multiple PCRAs can be administered to the same offender over time and because we stratify our analysis by the window of time in which a PCRA was given, the same offender can be represented across multiple columns.¹⁵

Table 1:

Number of Offenders, by Risk Level and Time PCRA Was Administered During Supervision (N = 216,036)

Risk Level	Months Into Supervision When PCRA Is Administered
Risk Level	Within 3 Months	>3 to 6 Months	>6 to 12 Months	>12 to 18 Months	>18 to 24 Months	>24 to 36 Months	>36 Months
High	6,002	1,294	1,443	624	560	396	151
Moderate	17,216	4,594	5,506	2,743	3,029	2,273	1,031
Low/moderate	31,814	10,587	13,254	9,373	10,283	9,228	5,245
Low	29,547	9,204	11,197	8,461	8,423	8,071	4,487
Total	84,579	25,679	31,400	21,201	22,295	19,968	10,914

Note. PCRA = Post-Conviction Risk Assessment.

According to this table, our data contain 84,579 PCRAs that were both (a) administered within the first 3 months of supervision and (b) administered at least 6 months before the end of the observation window (March 15, 2014). Of these roughly 84,000 PCRAs, around 7% result in a high risk designation and almost 73% result in a low or low/moderate risk grouping. By contrast, PCRAs administered more than 24 months after the start of supervision show these proportions change dramatically, to roughly 2% and 88%, respectively. A significant factor driving this shift is no doubt the selection of high risk offenders out of the sample as time under supervision progresses, although other factors obviously contribute as well. Nevertheless, this shift confirms the point that the predictive value of the PCRA is unlikely to be uniform over time.

As noted earlier, we measure a rearrest event as any rearrest for a serious offense occurring within 6 months of the date the PCRA was administered. Table 2 summarizes the proportion of offenders rearrested while under supervision within 6 months of the PCRA. Consistent with Table 1, the columns denote the time when a PCRA was administered, and the rows denote risk level. Overall, this table shows two things. The first is that the PCRA appears to do a good job at differentiating offender risk overall. For example, offenders labeled as high risk based on a PCRA in the first 12 months are rearrested 65% to 75% more often than moderate offenders of the same window (i.e., 15% vs. 9%). Likewise, those same moderate offenders are rearrested at twice the rate (i.e., 100% more often) of low/moderate offenders of the same window (9% vs. 4%). Calculations of the area under the ROC (i.e., the AUC) for this 12-month window lie between 0.732 and 0.740. Taken together, these findings confirm that the PCRA is a powerful, informative tool for assessing offender risk.

Table 2:

Rate of Rearrest Over a 6-Month Period Following a PCRA, by Risk Level and Time PCRA Was Administered During Supervision (N = 216,036)

Risk Level	Months Into Supervision When PCRA Is Administered
Risk Level	Within 3 Months	>3 to 6 Months	>6 to 12 Months	>12 to 18 Months	>18 to 24 Months	>24 to 36 Months	>36 Months
High	14%	16%	15%	10%	12%	11%	7%
Moderate	8%	9%	9%	7%	7%	6%	6%
Low/moderate	4%	4%	4%	4%	4%	3%	3%
Low	1%	1%	1%	1%	1%	1%	2%
AUC (PCRA total score)	0.732	0.740	0.740	0.690	0.712	0.682	0.650
95% CI	[0.724, 0.739]	[0.727, 0.753]	[0.727, 0.752]	[0.671, 0.708]	[0.694, 0.730)	[0.662, 0.703]	[0.619, 0.681]

Note. PCRA = Post-Conviction Risk Assessment; AUC = area under the curve; CI = confidence interval.

The second feature of the PCRA demonstrated in the table is its predictive power appears to diminish as the time spent under supervision continues. Using the AUC as a barometer for predictive power, this table shows that PCRAs administered in the first 12 months of supervision have the strongest power (AUC around 0.74), those administered in Years 2 and 3 have less power (around 0.70), and those administered after 3 years have the least power (0.65). Although determining what level of the AUC is considered “good” depends upon the context in which the value is being judged (Mossman, 2013), many researchers agree that values of 0.74 are considered good and that a difference of magnitude 0.1 (i.e., roughly 0.74 vs. 0.65) reflects a material difference in the strength of prediction (Drew, Wiersma, & Huettmann, 2011; Hanley & McNeil, 1982). We note that rates of reoffending are generally lower in later time windows than in earlier windows, which may increase the difficulty in discriminating between risk groups and contribute to the decrease in predictive power as time since supervision increases.

A likely explanation for why the PCRA appears less predictive at later points in supervision is that the risk pool of offenders is changing over the length of supervision. Specifically, the stock of offenders measured at the start of supervision includes a higher concentration of high risk offenders, relative to stock populations measured after some time (e.g., after 1 year of supervision), which will include fewer high risk offenders as they fail over time at a higher rate. Importantly, the PCRA has not been separately calibrated to compute predictions for various arrangements of offenders. Therefore, it must be true that predictive accuracy is better for some arrangements and worse for others. That said, the use of dynamic risk factors in the PCRA partly offsets this difficulty (Cohen et al., 2015).¹⁶

The PCRA score does a better job at differentiating offender risk for the most common offense types than for less common offense types. Table 3 provides an illustration. This table reports 6-month rearrest rates by risk level (in the rows) and by the type of rearrest offense (in the columns). Unlike the previous table, Table 3 limits the results to the sample offenders with PCRAs administered during the 3-month window from the start of supervision (n = 84,579). This table shows that PCRA classifications seem to provide the best information for four types of rearrest offenses: (a) drug offenses, (b) violent offenses, (c) property offenses, and (d) the unknown offense group. This result seems sensible given that these offense types make up the majority of the rearrest events that the PCRA is calibrated to predict. Also, given that these offense types are among the most serious within this group, greater strength of prediction for these specific offenses is likely to be (more) optimal from a policy perspective. Still good information is provided for offenses related to sex offenses. Finally, this table shows that the PCRA provides less utility in reliably predicting uncommon offenses such as public order, escape/obstruction, and immigration offenses. Reduced predictive accuracy among these offense types is not surprising as these crimes were given less weight to the calibration of the PCRA.

Table 3:

Proportion of Observed Rearrests, by Risk Level and Offense Type, for PCRAs Administered Within the First 3 Months of Supervision Start (n = 84,579)

Risk Level	Drug	Viol.	Prop.	Unk.	Immi.	Weap.	Oth.	Esc./Obstr.	Sex Offense	Public Order
High	4.1%	4.0%	3.3%	0.8%	0.3%	0.4%	0.4%	0.5%	0.2%	0.0%
Moderate	2.6%	2.2%	2.0%	0.5%	0.2%	0.3%	0.2%	0.2%	0.1%	<0.0%
Low/moderate	1.1%	0.9%	1.0%	0.3%	0.1%	0.1%	0.1%	<0.0%	0.1%	<0.0%
Low	0.2%	0.2%	0.3%	0.1%	0.3%	<0.0%	<0.0%	<0.0%	<0.0%	0.0%
AUC (PCRA total score)	0.752	0.759	0.727	0.795	0.619	0.704	0.709	0.685	0.749	0.439
95% CI	[0.738, 0.765]	[0.747, 0.771]	[0.687, 0.768]	[0.749, 0.841]	[0.177, 1.000]	[0.641, 0.767]	[0.694, 0.725]	[0.651, 0.718]	[0.710, 0.789]	[0.393, 0.485]
Total rearrested across all risk groups	1,084	985	953	235	179	113	109	83	58	2

Note. PCRA = Post-Conviction Risk Assessment; Viol. = violent offenses; Prop. = property offenses; Unk. = offenses of unknown type; Immi. = immigration offenses; Weap. = weapons offenses; Oth. = other offenses; Esc./Obstr. = escape/obstruction offenses; AUC = area under the curve; CI = confidence interval.

Although the preceding two tables describe the predictive relationship for overall PCRA scores to rearrests, the last two tables describe the association between specific scored items in the PCRA and 6-month rearrests. Table 4 reports the proportion of rearrests for offenders grouped by item and items scores using all available PCRAs (N = 216,036) and ignoring PCRA administration date. Table 5 reports the relationships between items and rearrests as coefficients estimated by the model described in Equation 1.

Table 4:

Proportion of Observed Rearrests, by PCRA Item Score, Across All PCRAs (N = 216,036)

Description	Response	Recorded Score
Description	Response	0	1	2	3
Number of prior arrests	0 = none; 1 = one or two; 2 = three through seven; 3 = eight or more	1.2%	2.4%	4.1%	6.9%
Violent offense	0 = no; 1 = yes	2.6%	5.9%	—	—
Varied offending pattern	0 = 1 offense type; 1 = 2 or more	1.7%	5.2%	—	—
History of revocation/arrest while on supervision	0 = no; 1 = yes	2.3%	6.4%	—	—
Institutional adjustment status	0 = no or NA; 1 = yes	3.2%	6.9%	—	—
Age at supervision intake	0 = 41+; 1 = 26 to 40; 2 = 25 or less	2.4%	4.9%	6.6%	—
Highest education at supervision start	0 = HS or higher; 1 = less than HS/GED	3.6%	7.7%	—	—
Unemployment status	0 = employed; 1 = unemployed	3.5%	7.4%	—	—
Good work assessment	0 = yes; 1 = no	3.6%	6.1%	—	—
Current alcohol problems	0 = no; 1 = yes	2.5%	4.6%	—	—
Current drug problems	0 = no; 1 = yes	3.3%	7.5%	—	—
Marital status	0 = married; 1 = not married	3.8%	6.7%	—	—
Unstable family situation	0 = no; 1 = yes	3.1%	6.2%	—	—
Lacks positive pro-social support	0 = no; 1 = yes	3.3%	5.8%	—	—
Attitude toward supervision and change	0 = motivated; 1 = not motivated	2.9%	5.5%	—	—

Note. PCRA = Post-Conviction Risk Assessment; HS = high school; GED = general education development.

Table 5:

Estimated Coefficients From the Logistic Regression Model of Rearrest Within 6 Months, for PCRAs Administered Within the First 3 Months of Supervision Start (n = 84,579)

Variable	Coefficient	OR	Wald Statistic	95% CI of Coefficient
Number of prior arrests	0.366***	1.441***	13.48	[0.313, 0.419]
Violent Offense	0.193***	1.214***	4.76	[0.114, 0.273]
Varied offending pattern	0.045	1.049	0.69	[−0.083, 0.174]
History of revocation/arrest while on supervision	0.373***	1.451***	8.69	[0.289, 0.458]
Institutional adjustment status	0.243***	1.275***	6.58	[0.169, 0.314]
Age at supervision intake	0.644***	1.904***	23.26	[0.589, 0.698]
Highest education at supervision start	0.183***	1.202***	5.00	[0.111, 0.255]
Unemployment Status	0.237***	1.267***	6.30	[0.163, 0.310]
Good work assessment	0.209***	1.232***	5.41	[0.133, 0.284]
Current alcohol problems	−0.015	0.984	−0.32	[−0.109, 0.077]
Current drug problems	0.160***	1.175***	4.09	[0.084, 0.238]
Marital status	0.182***	1.200***	3.55	[0.081, 0.283]
Unstable family situation	0.035	1.036	0.87	[−0.044, 0.114]
Lacks positive pro-social support	0.238***	1.268***	5.69	[0.155, 0.329]
Attitude toward supervision and change	0.240***	1.271***	5.29	[0.151, 0.329]
Constant	−5.389***	0.005***	−75.73	[−5.528, −5.259]
Pseudo R²	.0911

Note. The pseudo R² reported here is McFadden’s pseudo R², which represents the proportional reduction in the log likelihood such that higher values indicate a better model fit. There is no good intuition for the interpretation for this pseudo R², because unlike the R² from an OLS model, McFadden’s pseudo R² does not have an exact interpretation as the proportion of variability in the outcome accounted for by the model. As a consequence, pseudo R² is better applied in a relative setting, comparing different models of the same type, on the same data and predicting the same outcomes. PCRA = Post-Conviction Risk Assessment; OR = odds ratio; CI = confidence interval; OLS = ordinary least squares.

p < .05. **p < .01. ***p < .001.

Table 4 shows that across all PCRA items, higher item scores are correlated with a greater proportion of rearrests among offenders with those scores. For example, this table shows that 7.4% of unemployed offenders are rearrested while under supervision, as compared with 3.5% of employed offenders. Table 5 further confirms the importance of most PCRA items in predicting rearrests using a multivariate model. It shows that there is a significant predictive relationship among 12 of the 15 scored items, with all items having the relationship to the outcome that theory suggests. Of those 12, the number of prior arrests, history of revocation/arrest while on supervision, and offender age at intake are among the most informative. Of the three remaining items (varied offending pattern, current alcohol problems and unstable family situation), univariate results from Table 4 suggest a relationship should exist. The absence of this expected relationship in the Table 5 could be the result of collinearity in the model or because the marginal contributions of these variables to predictive power are small. Taken together, these tables confirm that the PCRA is a powerful assessment tool.

Discussion

Our analysis shows that the PCRA is an informative assessment tool as it is currently designed, achieving a level of predictive validity comparable with its competitors. Among all PCRAs administered during the first 12 months of supervision, high risk offenders are rearrested (within 6 months of the assessment) 65% to 75% more often than moderate offenders. Likewise, moderate offenders are rearrested at twice the rate of low/moderate offenders. We also find strong evidence that the predictive power of the PCRA diminishes over the life of the supervision term. Simply put, PCRAs administered toward the beginning of a supervision term have better predictive accuracy than those administered later in the term. The result is no doubt a reflection of the changing risk pool of offenders and/or changes in individual offender risk over time, but suggests that a recalibration of the PCRA may be a practical enhancement.

Using only the scored items, the PCRA effectively identifies those offenders at the greatest overall risk of reoffending while under supervision and effectively differentiates offender risk on the basis of certain types of offenses, namely, those of greatest practical importance. Our analysis of the PCRA’s predictive power, as measured by AUC, is in line with the results reported in Lowenkamp et al. (2013). Our analysis also indicates that each of the items in the PCRA identifies important risk factors practitioners should consider in evaluating offender risk of short-term reoffending, and that when these factors are used in combination, nearly all provide independently useful additional information. Secondary data analysis also shows the importance of the dynamic factors for the PCRA’s predictive ability.¹⁷ Moreover, our analysis does not control for possible supervision effects caused by the application of effective supervision strategies/conditions among higher risk offender. Thus, our estimates of PCRA classification effectiveness are inherently biased. That is also to say that, in the absence of supervision, the PCRA is likely to be a better predictor of risk than expressed in here.

Despite being an effective classification tool, it may be possible to improve PCRA classifications through some small and relatively simple adjustments to the PCRA design. For example, results from Tables 4 and 5 show that individual items predict risk differently, suggesting that reweighting scored items may produce improvements to prediction.¹⁸ Moreover, allowing the PCRA to be calibrated differently over the course of a supervision term, or for specific offenses, may increase its utility as a classifier of risk. In addition, this analysis does not consider how the integration of unscored items can positively affect the PCRA’s usefulness. Future analysis should explore this possibility.

Beyond these simple adjustments, more complex models have the potential to generate material improvements in prediction. State-of-the-art statistical learning approaches (such as regression trees and boosting regressions) are slowly making their way into the application of sophisticated risk classifiers (Berk & Bleich, 2013).¹⁹ The elements and construction of the PCRA may be able to exploit the advantages of these data-hungry approaches to classification. At the same time, such approaches often do nothing to inform officers about which risk factors matter most. Given that supervision officers are meant to actively manage offenders to mitigate risks when possible, classifications based on statistical learning approaches may be impractical.

There are several other important limitations to our analysis that have implications for the interpretation of these results. First, small samples appear to hinder analysis of PCRAs administered late in supervision terms. The data clearly indicate that PCRAs administered late in supervision terms have significantly less predictive accuracy than PCRAs administered earlier. However, given the small number of high risk offenders in the latest periods, it is hard to know whether and how these results might change with larger samples.

Another limitation to our analysis is that groups stratified by the date of PCRA administration do not equally represent admission cohorts. The reason for this is that the PCRA has a short history of implementation; thus, offenders with PCRAs administered well into their supervision term are necessarily from earlier cohorts (e.g., FY2009, FY2010), while offenders with “earlier” PCRAs are necessarily from later cohorts (e.g., FY2012, FY2013). To the extent that (a) there are systematic differences in offender cohorts, or (b) changes in the application of the PCRA over time (i.e., as officers have become more familiar with the instrument), comparisons of PCRA efficiency at various stages of supervision are potentially misleading, as they understate current PCRA effectiveness and overstate it prior effectiveness. There is currently no adequate, formal means of testing either of these two propositions using the current data; however, future analyses using more data would effectively solve this problem.

One additional limitation of the stratified groups we use for analysis is that they do not describe the heterogeneity of PCRA effectiveness within groups, though that heterogeneity is likely to exist. Although we select groups that roughly coincide with timelines related to PCRA use, groups that are defined more narrowly (e.g., PCRA administered in Month 1, Month 2, etc.) would provide a greater level of detail as to how PCRA timing and predictive accuracy are related. We cannot achieve this with our current data as narrower grouping require much larger samples (i.e., more years of PCRA data). We leave this as an investigation for future analysis.

Overall, we conclude that future applications of the PCRA can be improved, though it is unclear whether the potential benefits warrant the associated costs. Reweighting current PCRA items appears to produce marginal gains in predictive accuracy, but would incur costs in terms of updating IT systems, training, and documentation. One avenue for reweighting not explored in this article might be to normalize PCRA items for specific subgroups of interest. Consider, for example, that the PCRA might be calibrated differently on the basis of gender. Although secondary data analysis of our sample (81% male vs. 19% female) shows that the PCRA as it is designed predicts equally well for men and for women (results not shown), gender-norming risk instruments may yet provide additional improvements to accuracy and precision in risk prediction between groups and may provide more effective tools for managing offenders in general.²⁰ Again, however, the additional burden created by introducing greater flexibility in the design and application of the PCRA may be significant, and the value of this trade-off is unclear. At most, PPSO should explore the utility of expanding its PCRA design to incorporate the features demonstrated by sophisticated approaches to extend PCRA value.

Finally, PPSO should also explore the utility of expanding its definition of risk categorization. Currently, the PCRA uses only four differentiating categories: high risk, moderate risk, and so on. Predictive models need not be bound by the lumpiness imposed in these current categorizations (i.e., the uneven distribution of offenders across the four PCRA risk classifications, with most offenders being classified as “low” or “low/moderate” risk). PPSO may wish to explore whether expanding the class/number of categories of classification can provide a finer level of detail that is of practical importance to the effective delivery of supervision.

Footnotes

Appendix

Table A1:

Descriptions of PCRA Items From the Officer Assessment

Item	Item Description	Valid Answers	Scored
1.1	Arrested under age 18	A = no; B = yes	N
1.2	Number of prior misdemeanor and felony arrests	0 = none; 1 = one or two; 2 = three through seven; 3 = eight or more	Y
1.3	Violent offense	0 = no; 1 = yes	Y
1.4	Varied offending pattern	0 = 1 offense type; 1 = 2 or more	Y
1.5	Revocation/arrest while on supervision	0 = no; 1 = yes	Y
1.6	Institutional adjustment status	0 = no or NA; 1 = yes	Y
1.7	Age at intake to supervision	0 = 41+; 1 = 26 to 40; 2 = 25 or less	Y
2.1	Highest education at supervision start	0 = HS or higher; 1 = less than HS/GED	Y
2.2	Unemployment status	0 = employed PT/FT, disabled and receiving benefits; 1 = student, homemaker, unemployed or retired but able to work	Y
2.3	Number of jobs in past 12 months	A = 1; B = none or more than 1	N
2.4	Employed <50% of the last 24 months	A = employed 12 months or more B = employed less than 12 months	N
2.5	Good work assessment	0 = yes; 1 = no	Y
3.1	Drug/alcohol use causes disruption at work, school or home	A = no; B = yes	N
3.2	Drug/alcohol use when physically hazardous	A = no; B = yes	N
3.3	Legal problems related to use	A = no; B = yes	N
3.4	Continued use despite social problems	A = no; B = yes	N
3.5	Current alcohol problems	0 = no; 1 = yes	Y
3.6	Current drug problems	0 = no; 1 = yes	Y
4.1	Marital status	0 = married; 1 = not married	Y
4.2	Lives with spouse and/or children	A = no; B = yes	N
4.3	Lack of family support	A = support present; B = no support	N
4.4	Unstable family situation	0 = no; 1 = yes	Y
4.5	Companion status	A = good support; B = occasional association with negative peers; C = more than occasional association with negative peer; D = no friends	N
4.6	Lacks positive pro-social support	0 = no; 1 = yes	Y
5.1	Harbors antisocial attitude/values	A = no; B = yes	N
5.2	Attitude toward supervision and change	0 = motivated; 1 = not motivated	Y
6.1	Housing status	A = 1 address in last 12 months B = > 1 address last 12 months; no permanent address	N
6.2	Risk of criminal influence at home	A = no risks at home; B = risks at home	N
6.3	Financial stressors	A = adequate income to manage debts; concrete financial plans; B = no plan in place; expenses exceed income	N
6.4	Pro-social recreation	A = engages in pro-social activities B = has no interests; does not; or recreation presents criminal risk	N

Note. PCRA = Post-Conviction Risk Assessment; HS = high school; GED = general education development; PT/FT = part-time or full-time employment.

This work was performed under Contract No. GS-10F-0086K awarded by the Administrative Office of the U.S. Courts, Office of Justice Programs, U.S. Department of Justice. Points of view in this document are those of the authors and do not necessarily represent the official position of the Administrative Office of the U.S. Courts. The authors are responsible for any errors in the article.

Notes

Jeremy Luallen, PhD, is a senior associate at Abt Associates and a full-time faculty member at Tufts University. He has more than 10 years of experience conducting quantitative and qualitative analysis of justice topics, with his most recent work focusing on issues in recidivism, prison growth, health and corrections, and risk assessment modeling.

Sharmini Radakrishnan, PhD, is a senior analyst at Abt Associates. She is an economist with expertise in health policy and behavioral health. Her most recent work focuses on prescription drug misuse, prisoner recidivism, impacts of health reform, and risk assessment modeling.

William Rhodes, PhD, is an economist specializing in program evaluation and quantitative analysis. He is now retired, though he continues work as the coprincipal investigator for the Bureau of Justice Statistics’ (BJS) National Corrections Reporting Program.

References

Andrews

D. A.

Bonta

Hoge

R. D.

(1990). Classification for effective rehabilitation: Rediscovering psychology. Criminal Justice and Behavior, 17, 19-52.

Andrews

D. A.

Zinger

Hoge

R. D.

Bonta

Gendreau

Cullen

F. T.

(1990). Does correctional treatment work? A clinically relevant and psychologically informed meta-analysis. Criminology, 28, 369-404.

Berk

R. A.

Bleich

(2013). Statistical procedures for forecasting criminal behavior. Criminology & Public Policy, 12, 513-544.

Bonta

Andrews

D. A.

(2007). Risk-need-responsivity model for offender assessment and rehabilitation (User Report 2007-06). Ottawa, Ontario: Public Safety Canada.

Burgess

E. W.

(1928). Factors determining success or failure on parole. In Bruce

Burgess

Harno

(Eds.), The workings of the indeterminate-sentence law and the parole system in Illinois (pp. 221-234). Chicago, IL: University of Chicago Press.

Cohen

Lowenkamp

C. T.

VanBenschoten

S. W.

(2015). Does change in risk matter? Examining whether changes in offender risk characteristics influences recidivism outcomes. Under review.

Dawes

R. M.

Faust

Meehl

P. E.

(1989). Clinical versus actuarial judgment. Science, 243, 1668-1674.

Drew

C. A.

Wiersma

Huettmann

(2011). Predictive species and habitat modeling in landscape ecology. New York, NY: Springer.

Freund

Schapire

R. E.

(1997). A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55, 119-139.

10.

Gendreau

Little

Goggin

(1996). A meta-analysis of the predictors of adult offender recidivism: What works! Criminology, 34, 575-608.

11.

Gottfredson

S. D.

Moriarty

L. J.

(2006). Clinical versus actuarial judgments in criminal justice decisions: Should one replace the other? Federal Probation, 70, 15-20.

12.

Hanley

J. A.

McNeil

B. J.

(1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29-36.

13.

Johnson

J. L.

Lowenkamp

C. T.

VanBenschoten

S. W.

Robinson

C. R.

(2011). Construction and validation of the federal Post Conviction Risk Assessment (PCRA). Federal Probation, 75, 16-29.

14.

Lowenkamp

C. T.

Holsinger

A. M.

Brusman-Lovins

Latessa

E. J.

(2004). Assessing the inter-rater agreement of the Level of Service Inventory Revised. Federal Probation, 68, 34-38.

15.

Lowenkamp

C. T.

Holsinger

A. M.

Cohen

T. H.

(2015). PCRA revisited: Testing the validity of the federal Post Conviction Risk Assessment (PCRA). Psychological Services, 12, 149-157.

16.

Lowenkamp

C. T.

Johnson

J. L.

Holsinger

A. M.

VanBenschoten

S. W.

Robinson

C. R.

(2013). The federal Post Conviction Risk Assessment (PCRA): A construction and validation study. Psychological Services, 10, 87-96.

17.

Lowenkamp

C. T.

Latessa

E. J.

Holsinger

A. M.

(2004). Empirical evidence on the importance of training and experience in using the Level of Service Inventory–Revised. Topics in Community Corrections, 49-53.

18.

Mossman

(2013). Evaluating risk assessments using Receiver operating characteristic analysis: Rationale, advantages, insights, and limitations. Behavioral Sciences & the Law, 31, 23-39.

19.

Schwalbe

C. S.

(2008). A meta-analysis of juvenile justice risk assessment instruments: Predictive validity by gender. Criminal Justice and Behavior, 35, 1367-1381.

20.

Yang

Wong

S. C.

Coid

(2010). The efficacy of violence prediction: A meta-analytic comparison of nine risk assessment tools. Psychological Bulletin, 136, 740-767.

21.

Zara

Farrington

D. P.

(2015). Criminal recidivism: Explanation, prediction and prevention. London, England: Routledge.