The Influence of Risk Assessment Instrument Scores on Evaluators’ Risk Opinions and Sexual Offender Containment Recommendations

Abstract

In Colorado, evaluators conducting sex offender risk assessments are required to assess 17 risk factors specified by the state’s Sex Offender Management Board (SOMB), in addition to scoring actuarial risk assessment instruments. This study examined the association between instrument scores, the 17 SOMB risk factors, and evaluator opinions concerning risk and need for containment in 302 Colorado cases. Evaluators’ ratings of risk indicated by noninstrument factors were often higher than their ratings of risk indicated by instrument results, but only their ratings of noninstrument factors were independently predictive of containment recommendations. Several of the most influential noninstrument factors (e.g., denial, treatment motivation) have been described by researchers as potentially misleading because they are not predictive of future offending. Findings highlight the need for more studies examining the validity of what risk assessment evaluators actually do, as opposed to what researchers think they should do.

Keywords

sex offender actuarial risk assessment recidivism clinical judgment

Introduction

Evaluators conducting risk assessments collect and review a substantial amount of information. It is common practice for evaluators to interview the offender for several hours and review all available criminal, correctional, law enforcement, and treatment records (see Jackson & Hess, 2007; Viljoen, McLachlan, & Vincent, 2010). Offender records may be hundreds or even thousands of pages long, and interviews cover a broad range of topics during risk assessment interviews. Many evaluators also administer one or more risk assessment instruments. But how do evaluators use this vast array of information to come to a conclusion about the offender’s level of risk? Do they consistently favor some types of information over others?

In many risk assessment contexts, results from standardized risk assessment instruments are the strongest known predictors of future violence, outperforming unstructured decision-making approaches (Guy, 2008; Hanson & Morton-Bourgon, 2009). Although this has been a generally consistent finding across studies, clinicians rarely base their opinions on instrument results alone (Vrieze & Grove, 2009). In a survey of 109 evaluators who had used the Static-99R (Helmus, Thronton, Hanson, & Babchishin, 2012) in sex offender risk assessment cases, only one evaluator reported basing his/her risk opinion entirely on the basis of the Static-99R results (Chevalier, Boccaccini, Murrie, & Varela, 2015). Instead, most reported that the Static-99R results had either some (49%) or a lot (42%) of influence on their risk decisions.

When evaluators are required to use specific measures, some report scoring them diligently and using the results for decision making, but just as many report scoring them as a formality and basing their decisions on other factors (Miller & Maloney, 2013). In one recent field study, researchers found that probation officers overrode risk-level ratings in 74% of cases involving sexual offenders, concluding that the offender’s level of risk differed from that indicated by the instrument score (Schmidt, Sinclair, & Thomasdόttir, 2016). In each of these cases, they used the override to increase the offender’s risk-level classification. Another recent field study found that evaluators used risk instrument overrides in 33% of sex offender cases, with 57% of these overrides being used to increase risk and 43% used to decrease risk (Storey, Watt, Jackson, & Hart, 2012). On the Level of Service/Case Management Inventory (LS/CMI; Andrews, Bonta, & Wormith, 2010), an instrument with a built-in override option, evaluators overrode risk scores to increase risk more often than to decrease risk (14.9% increase vs. 1.6% decrease), especially for sexual offenders (33.5% increase vs. 1.6% decrease). These overrides lead to a decrease in predictive validity (Storey et al., 2012; Wormith, Hogg, & Guzzo, 2012).

Although the override rates varied across these studies, it appears to be common practice for evaluators to override risk instrument results in some cases, and that evaluators tend to use overrides to increase as opposed to decrease the offenders’ level of risk. These studies did not address evaluators’ reasons for using overrides, but in the context of risk assessment, overrides to increase risk may be a product of evaluators protecting themselves from liability (Guy, Nelson, Fusco-Morin, & Vincent, 2014). The pattern of overrides in these studies suggest that evaluators are more concerned about failing to identify truly dangerous offenders (false negatives) than identifying nondangerous offenders as dangerous (false positives).

It also appears to be common practice for evaluators to use more than one risk assessment instrument in each case. Evaluators administer as many as 15 different risk assessment instruments to the same offender in risk assessment cases, with an average of about four instruments per case (Neal & Grisso, 2014). The overall picture of contemporary risk assessment practice seems to be one in which evaluators administer multiple instruments, collect a significant amount of additional information through interviews and records reviews, and attempt to integrate all or parts of this information to come to a single conclusion about risk. This integrated approach raises questions about the type or types of information that have the strongest influence on evaluator opinions, and whether those factors are those with the most empirical support.

Predictors of Evaluator Opinions in Risk Assessment Cases

Few studies have examined the association between risk assessment instrument scores and evaluators’ opinions, especially when considered alongside other offender and case characteristics. In an early study, insanity acquittees’ scores on the Violence Risk Appraisal Guide (VRAG; Quinsey, Harris, Rice, & Cormier, 1998) were not significantly associated (r = −.06) with clinicians’ security level placement recommendations (Hilton & Simmons, 2001). Instead, evaluators’ recommendations were associated with the acquittees’ institutional behavior, medication use, preindex criminal history, and physical attractiveness. In a recent follow-up, VRAG and Psychopathy Checklist–Revised (PCL-R; Hare, 2003) scores were both significantly correlated (r = .33 to .47) with clinician and multidisciplinary clinical team recommendations to detain (as opposed to transfer) forensic patients, suggesting a possible increase in the association between instrument scores and evaluator opinions over time (Hilton, Simpson, & Ham, 2016).

Other more recent studies have also reported a stronger association between risk instrument scores and evaluator opinions. In the most comprehensive study, researchers coded 27 different offender, offense, and mental health variables from sexually violent predator (SVP) evaluation reports to identify factors that were predictive of evaluators’ civil commitment recommendations (Levenson & Morin, 2006). The evaluators scored offenders on the Static-99 (Hanson & Thornton, 2000), Minnesota Sexual Offender Screening Tool–Revised (MnSOST-R; Epperson et al., 1998), and PCL-R. Scores on each of the three instruments were predictive of commitment recommendations, even after controlling for the other offense, offender, and mental health variables. But several noninstrument variables also predicted commitment recommendations, including paraphilia diagnoses, victim age, and offender race.

The belief that someone scoring in the high-risk range on multiple risk assessment instruments is at a higher risk for reoffending than someone scoring in the high-risk range on fewer measures does not appear to be limited to SVP evaluations. In a large study of parole suitability decisions, offenders’ scores on the Historical, Clinical, Risk Management-20 (HCR-20; Webster, Douglas, Eaves, & Hart, 1997), LS/CMI, and PCL-R were each independently predictive of evaluators’ opinions of overall risk (Guy, Kusaj, Packer, & Douglas, 2015). The finding that multiple instrument scores predicted evaluator opinions and recommendations is interesting when considered alongside research suggesting that combining results from different measures does not lead to more accurate predictions than those based on the results of the best performing measure alone (Seto, 2005).

Together, findings from these three evaluator opinion studies provide only a limited view of the role of risk instrument scores in evaluator decision making. Only one study examined evaluators’ opinions of risk (Guy et al., 2015), the criterion that risk measures are designed to address. But that study focused on only instrument scores, as opposed to other factors that might predict risk opinions. The other studies examined evaluators’ placement or SVP commitment recommendations, which may have rightfully been influenced by factors other than risk instrument results (Hilton & Simmons, 2001; Levenson & Morin, 2006). For example, SVP commitment statutes require evaluators to consider whether the offender has a mental or behavioral abnormality, often conceptualized as a diagnosis (e.g., paraphilia; DeMatteo, Murphy, Galloway, & Krauss, 2015). Although it is important to study placement and commitment recommendations—because these are the types of referral questions evaluators are required to address—it is also important to study evaluator opinions of future risk. It seems likely that evaluators come to a conclusion about risk before coming to a conclusion about the referral question (e.g., commitment, placement). The field would benefit from studies that provide an examination of both risk opinions and referral question recommendations, which would allow for a more comprehensive picture of the role of risk instrument scores in evaluator decision making.

Risk and Placement Recommendations in Colorado Psychosexual Evaluations

In Colorado, each offender convicted of a sexual offense must undergo a psychosexual evaluation (sometimes referred to as a sex offense–specific evaluation) to provide the court with information that will be used for sentencing decisions. These evaluations provide a useful context for studying the role of risk assessment instrument scores in evaluator decision making. The Colorado Sex Offender Management Board (SOMB) was created pursuant to legislation passed by the Colorado General Assembly in 1992 (Section 16-11.7-101 through Section 16-11.7-107 C.R.S) to establish standards and guidelines for assessment, treatment, and behavioral monitoring of sexual offenders. Section 2.000 of the Colorado SOMB Standards and Guidelines outlines the standards for mental health sex offense–specific evaluations (SOMB, 2011, p. 17). These standards require evaluators to use risk assessment instruments for these evaluations, “to ensure the most accurate prediction of risk for sexual offenders” (SOMB, 2011, p. 23). The SOMB does not require specific measures, but does provide a list of possible measures, including the Static-99R, VRAG, and Sex Offender Risk Appraisal Guide (SORAG; Quinsey et al., 1998, 2006). The SOMB also cautions evaluators against overreliance on risk measure scores: “Evaluators shall incorporate all available information when making a determination regarding risk and not rely solely on risk assessment instruments. Risk assessment instruments will not identify all risk factors” (SOMB, 2011, p. 32).

The SOMB standards state an evaluator “shall consider” a list of additional factors for estimating an offender’s level of risk (see Table 1; SOMB, 2011, p. 25). These factors include some of the most well-known predictors of future sexual offending, but also some factors with questionable validity. For example, the SOMB asks evaluators to consider sexual deviance, prior offending, and antisocial traits, which are among the strongest predictors of future sexual offending (Hanson & Morton-Bourgon, 2005). But other factors on this list, such as denial and motivation for treatment, have been described by researchers as “potentially misleading” risk factors because they are not predictive of future offending (Hanson & Morton-Bourgon, 2005, p. 1158). For example, the association between denial and sexual recidivism was only d = .02 in the most recent meta-analytic review (Hanson & Morton-Bourgon, 2005). Although some more recent studies have found at least partial support for denial as a predictor of recidivism (e.g., Nuñes et al., 2007), others have found that offenders who were more likely to deny their level of risk were actually less likely to reoffend than other offenders (e.g., Harkins, Beech, & Goodwill, 2010).

Table 1:

The 17 SOMB Risk Factors and Proposed Sources of Data

Risk factor	Methods/sources of data
1. Offense history and victim choice	Number of offenses/length of time offending, number of victims, male, female, or dual gender choice of victims, type of offenses and escalation pattern, age/vulnerability of target victims, violence, sadism, or physical harm in offending, age of onset of deviant arousal/behaviors, nature and extent of coercion and manipulation to gain victim compliance during offense and regarding nondisclosure, offender’s intended outcome and desired response from victim.
2. Sexual deviance and arousal patterns	Frequency of deviant fantasies, frequency of masturbation to deviant fantasies, assessment of response to fantasy content and level of deviance, frequency of masturbation to nonabusive fantasies, arousal to violence or sadism, presence of sexual dysfunction, use of pornography/seeking sexualized atmospheres, results of phallometric measures, practicing responsible sexual behavior, connects sexuality with caring relationship
3. Developmental markers	Competency, deficits, resilience, organicity
4. Substance abuse	Alcohol use/abuse pattern, duration, treatment, other drug (legal or illegal) use/abuse pattern, duration, treatment, connection between substance abuse and offenses
5. Criminal history	Extent of documented/undocumented criminal history, type/number of criminal offenses, violence history, ritualistic and/or bizarre bases for offenses, history of childhood, or adolescent delinquency
6. Treatment history	Success/failure of prior sex offense–specific treatment, success/failure of prior nonsex offense–specific treatment (may be psychotherapy or pharmacological treatment), attitude about prior treatment
7. Social support system	Degree of functional social skills, presence/absence of social relationships, type and quality of relationships, presence of dysfunctional relationships, relationships supporting denial or minimization of offending, problems and stresses within support system relationships
8. Level of denial	Level of denial vs. omission about referral offense, level of denial vs. omission about past offenses, admission of undocumented offenses, disclosure of detail not on record and degree of consistency between self-reports and victim statements
9. Accountability for offense	Degree of personal responsibility for offenses assumed, degree of disowning behaviors, degree of cognitive distortions to justify the offenses, assumes responsibility for the after effects of offense on the victim
10. Cooperation during the evaluation	Overall attitude in evaluation process, willingness to divulge information, actively participates in interview, presence or absence of passive-aggressive or covert resistance.
11. Level of empathy	Level of general victim empathy, empathy for own victims, expressions of awareness and authentic regret regarding abusive traumatic and/or harmful nature of behavior to victim(s) and others, range and congruence of affective expression, expressions of guilt regarding victim harm, responds in a prosocial manner to social interaction
12. Lifestyle characteristics and past behaviors	Degree of antisocial behavior (victimizing, control seeking, exploits others, criminal thinking), degree of narcissistic behavior (grandiose, egocentric, demanding, inconsiderate), degree of borderline behavior (impulsive, erratic, markedly moody, possessive, unstable relationships, etc.), degree of schizoidal behavior (avoidant, flat affect, withdrawn, lacking social skills), attachment style, degree of sexualization of relationships
13. Psychopathology	Psychotic episodes, frequency and lethality of suicidal ideation, personality disorder (e.g., psychopathic traits), affective disorder, obsessive/compulsive disorder, PTSD symptoms, other concurrent psychiatric diagnosis
14. Understanding of behaviors and motivation to change	Understanding of deviant cycle, understands triggers and cues, demonstrates motivation to avoid and interrupt cycle, demonstrates ability to avoid and interrupt cycle, recognizes thinking errors, actively corrects thinking errors as they arise, has replacement behaviors, controls inappropriate sexual behavior
15. Motivation for treatment	Over concern with prison/legal consequences, superficial motivations, presents facade vs. genuine, authentic presentation, level of commitment to stop own offending, willingness to complete any needed treatment/recovery tasks
16. Self-worth	Base of self-worth, ways to get self-worth, self-esteem, level of confidence, lacks sense of inferiority, ability to appropriately cope with failures
17. Defensiveness	Level of defensiveness, projects blame, displacement of anger, irrational beliefs, criminal thinking distortions

Note. SOMB = Sex Offender Management Board; PTSD = posttraumatic stress disorder.

Despite these findings, denial continues to be included as a risk factor on structured risk assessment measures, including the Sexual Violence Risk-20 (SVR-20; Boer, Hart, Kropp, & Webster, 1997) and the Risk for Sexual Violence Protocol (Hart et al., 2003). Although global scores on these measures are associated with sexual recidivism, few studies have examined the predictive validity of individual items. For the SVR-20, the most thorough item validity study found that all but two of the items were predictive of reoffending, with one of the nonpredictive items being extreme minimization or denial (area under the curve [AUC] = .49; Rettenberger, Boer, & Eher, 2011).

In several instances, it is difficult to determine whether or not there is empirical support for a factor listed and defined by the SOMB. For example, the Offense History and Victim Choice factor asks evaluators to consider the number of prior offenses and victims, which are predictive of future offending, but also victim sex and harm, which are not (Hanson & Morton-Bourgon, 2005). For the psychopathology factor, the SOMB asks evaluators to consider a broad array of symptoms, ranging from psychosis to suicidality to personality disorders. There are some documented associations between personality traits and future offending, such as the moderate association between scores on the PCL-R and sexual offending (Hawes, Boccaccini, & Murrie, 2013), but research does not suggest a clear association between other types of psychopathology and sexual recidivism. For example, offenders with paraphilia diagnoses have been found to be more likely to recidivate than other offenders in some studies (Kingston, Olver, Harris, Wong, & Bradford, 2015), but not others (Harris, Boccaccini, & Rice, in press; Moulden, Firestone, Kingston, & Bradford, 2009). Another study suggests a more nuanced pattern of association between diagnostic disorders and recidivism. Specifically, offenders who had been diagnosed with paraphilic disorders were no more likely to recidivate than other sex offenders, but those with both a personality disorder and paraphilia diagnosis were more likely to recidivate than any other group of offenders (Abracen & Looman, 2006).

Current Study

In the current study, we examine the association between risk assessment results, evaluator risk opinions, and evaluator containment/placement recommendations (i.e., prison, probation, or release) among 302 sexual offenders undergoing psychosexual evaluations in Colorado.

Colorado evaluators are required to consider the risk factors outlined by the SOMB (Table 1), but they are not required to provide the type of factor-by-factor risk ratings used by the evaluators in this study. Although focusing on evaluators from one agency may limit the generalizability of our study findings, the agency-wide practice of separately rating each risk factor provides a unique opportunity to examine evaluator decision making in sex offender risk assessment cases. Because the evaluators rated each offender on each risk factor, we do not have to make subjective decisions about evaluator-implied risk from lengthy written statements or omit cases when the evaluator did not write about a specific risk factor. To our knowledge, this is the first comprehensive field study of evaluator decision making in which evaluators’ perceptions of risk factors are not based on a researcher-derived subjective coding process.

Method

Evaluation Sample

We coded 302 de-identified sex offender psychosexual evaluation reports conducted by clinicians (N = 7) working for the same forensic assessment practice. The evaluations described in these reports were completed between 2008 and 2013 and represent all of the evaluations completed by the agency evaluators during this timeframe. To provide some context for these numbers, the agency conducted 52 psychosexual evaluations in 2008. Every individual convicted of a sexual offense is required to undergo a psychosexual evaluation in Colorado. There were 932 sexual offense convictions in the state of Colorado in 2008 (Office of Research & Statistics, 2011). Thus, the evaluations in this study likely represent approximately 6% of the psychosexual evaluations conducted in Colorado. There are several other agencies in Colorado that provide evaluation services, and evaluations may also be conducted by private practice evaluators who have been certified in accordance to the standards and guidelines of the SOMB.

Offenders

The offenders ranged in age from 17 to 68 years (M = 35.55, SD = 12.92). Most of the offenders were male (n = 296, 98.0%).¹ Only 21.8% (n = 66) of the reports provided information about the offender’s race and/or ethnicity. Data from these reports indicated that the most common race/ethnic groups represented in the sample were Caucasian (n = 31, 46.9%), African American (n = 24, 36.3%), and Hispanic (n = 8, 12.1%) defendants.

Among the offenders in the sample, the most common sexual offense was Failure to Register as a Sex Offender (n = 54, 17.8%), followed by Attempted Sexual Assault of a Child (n = 43, 14.2%), Sexual Assault of a Child (n = 34, 11.3%), and Sexual Contact—No Consent (n = 19, 6.3%). In total, there were 18 types of offenses represented in the sample. Many of the offenders in the sample had recent nonsexual charges, but a record of past sexual offenses (n = 77, 25.5%). Of these offenders and offenders charged with Failure to Register (n = 171, 56.6%), the most common past sexual offenses were Sexual Assault of a Child (n = 43, 25.1%), followed by Rape (n = 22, 12.9%) and Attempted Sexual Assault of a Child (n = 16, 9.4%).

Evaluators

The seven evaluators (five female, two male) were master’s level clinicians certified as Full Operating Evaluators by the Colorado SOMB. Certification requires licensure (i.e., psychology, social work), 80 hr of sex offender specific training, completion of 40 sex offense–specific evaluations over the past 5 years, and registration as a sex offender treatment provider (which requires 1,000 hr of clinical experience). Six of the evaluators were MA-level psychologists and one was an MA-level social worker. The agency employing these seven clinicians focused exclusively on sex offender risk assessments. Most evaluations occurred at local jails, although offenders who were not in custody were evaluated at the agency office. Nearly all of the evaluations were court ordered.

Measures

SOMB Risk Factors

Evaluators rated offenders on each of the 17 SOMB risk factors outlined in Table 1. The evaluators provided a separate risk-level rating for each of the 17 factors using a five-category scale: 1 = Low, 2 = Low-Moderate, 3 = Moderate, 4 = Moderate-High, and 5 = High. There are no scoring rules for assigning risk-level ratings for these 17 factors. The ratings are based on clinical judgment.

Static-99

The Static-99 is a 10-item measure designed to measure likelihood of sexual recidivism (Hanson & Thornton, 2000). Static-99 items are based on static, historical factors about the offender and his/her offenses, such as the number of prior offenses, offender’s age, and victim characteristics. The developers of the Static-99 selected items on the basis of empirical relationships with recidivism. Evaluators report Static-99 scores as falling into one of four categories of risk: Low (scores 0 to 1), Low-Moderate (scores 2 and 3), Moderate-High (scores 4 and 5), and High (scores 6 and above). In reports available for this study, evaluators reported the offender’s risk category, but did not report a score. Table 2 provides descriptive statistics for Static-99 and other risk assessment instruments.

Table 2:

Descriptive Statistics for Clinicians Ratings of SOMB Risk Factors and Actuarial Measure Results

Factor	Risk level					n	M	SD
Factor	High	Moderate/high	Moderate	Moderate/low	Low	n	M	SD
SOMB factors
Level of empathy	52.0%	11.3%	23.2%	2.0%	11.6%	302	2.90	1.37
Lifestyle/past behavior	45.7%	15.6%	22.5%	6.6%	9.6%	302	2.81	1.33
Psychopathology	44.7%	15.9%	25.8%	5.3%	8.3%	302	2.83	1.28
Motivation for treatment	41.7%	11.6%	26.8%	5.6%	14.2%	302	2.61	1.43
Criminal history	41.4%	11.9%	21.9%	8.9%	15.9%	302	2.54	1.49
Understanding/motivation to change	37.4%	17.2%	35.1%	4.0%	6.3%	302	2.76	1.18
Accountability for offense	36.4%	18.2%	24.2%	6.6%	14.2%	301	2.56	1.40
Substance abuse	33.8%	13.6%	30.1%	7.0%	15.6%	302	2.43	1.41
Offense history/victim choice	26.5%	14.2%	40.1%	15.6%	3.6%	302	2.44	1.14
Level of denial	21.5%	43.7%	8.9%	3.3%	17.5%	287	2.51	1.37
Sexual deviance/arousal patterns	17.5%	3.3%	7.6%	4.5%	63.2%	291	1.04	1.59
Prior treatment history	16.6%	8.6%	39.7%	7.0%	28.1%	302	1.79	1.38
Social support system	11.6%	9.9%	42.1%	13.6%	22.8%	302	1.74	1.24
Developmental markers	6.0%	5.3%	23.8%	18.9%	45.7%	301	1.06	1.20
Defensiveness	7.9%	35.8%	34.4%	23.3%	9.6%	302	2.20	1.07
Self-worth	3.3%	0.7%	15.6%	2.6%	77.8%	302	0.49	1.00
Cooperation during evaluation	1.7%	2.6%	38.1%	12.9%	44.7%	302	1.04	1.04
Actuarial results
Static-99	15.2%	30.8%	0.3%^a	40.7%	7.9%	287	2.05	1.31
SORAG	13.2%	16.6%	38.1%	10.9%	19.9%	298	1.91	1.27
VRAG	6.3%	5.6%	57.9%	21.2%	7.6%	298	1.82	0.90
Summary risk ratings
SOMB summary rating	15.9%	27.8%	41.4%	12.9%	2.0%	302	2.42	0.97
Actuarial summary rating	7.3%	28.8%	36.8%	19.2%	6.6%	298	2.11	1.02

Note. SOMB = Sex Offender Management Board; SORAG = Sex Offender Risk Appraisal Guide; VRAG = Violence Risk Appraisal Guide.

VRAG

The VRAG is a 12-item measure designed to predict the likelihood of violent recidivism (Quinsey et al., 1998). VRAG items are based on static and historical factors such as alcohol-related problems, marital status, and victim injury, as well as the results of testing for psychopathy (e.g., PCL-R). Evaluators reported VRAG scores as falling into one of nine risk categories: Category 1 (scores below −22), Category 2 (scores −21 to −15), Category 3 (scores −14 to −8), Category 4 (scores −7 to −1), Category 5 (scores 0 to 6), Category 6 (scores 7 to 13), Category 7 (scores 14 to 20), Category 8 (scores 21 to 27), and Category 9 (scores above 28). They did not report raw VRAG scores.

According to the VRAG manual, Categories 1 to 3 are classified as Low, Categories 4 to 6 are classified as Medium, and Categories 7 to 9 as High. The evaluators in this study classified VRAG scores using the same five-category rating system that they used for all SOMB factors: 1 = Low, 2 = Low-Moderate, 3 = Moderate, 4 = Moderate-High, 5 = High. Evaluators described scores of 3 and 4 as Low-Moderate, and scores of 6 and 7 as Moderate-High.

SORAG

The SORAG, a modification from the VRAG, is a 14-item measure designed to predict the likelihood of sexual recidivism (Quinsey et al., 1998). The SORAG shares 10 items with the VRAG including static and historical factors, as well as a PCL-R score, similar to the VRAG, but also includes sex offense–specific items such as whether the victim(s) are all female and under the age of 14 years, and phallometric test results regarding sexual deviancy. Evaluators reported SORAG scores as falling into one of nine risk categories: Category 1 (scores below −10), Category 2 (scores −9 to −4), Category 3 (scores −3 to 2), Category 4 (scores 3 to 8), Category 5 (scores 9 to 14), Category 6 (scores 15 to 19), Category 7 (scores 20 to 24), Category 8 (scores 25 to 30), and Category 9 (scores above 31). They did not report raw SORAG scores. As with the VRAG, the evaluators classified SORAG scores using the same five-category rating system (1 = Low, 5 = High) that they used for all SOMB factors.

PCL-R

The PCL-R is a 20-item, clinician-scored measure of psychopathic traits (Hare, 2003). PCL-R scores are moderate predictors of sexual and violent recidivism among sexual offenders (Hawes et al., 2013). The SOMB recommends using the PCL-R as a measure of psychopathology (see Table 1), as opposed to actuarial risk, but evaluators use PCL-R results when scoring the VRAG and SORAG. We used PCL-R results in supplemental analyses aimed at exploring the association between evaluators’ psychopathology ratings and risk. Evaluators reported PCL-R results as percentiles, which we converted to raw scores. The mean PCL-R score among the 300 offenders with a score was 16.63 (SD = 7.28), which falls in the 28th percentile for adult male offenders.

Summary Risk Ratings

Evaluators’ reports included two summary risk ratings: One based on the evaluator’s opinion of the level of risk indicated by actuarial assessment results (Static-99, VRAG, SORAG) and one based on the evaluator’s opinion of the level of risk indicated by the 17 SOMB factors. The evaluators reported summary risk ratings using the five-category risk scale: Low, Low-Moderate, Moderate, Moderate-High, and High (see Table 2).

Containment Recommendation

Evaluators concluded their reports with a recommendation for placement/containment. These recommendations can fall into three categories ranging from least to most restrictive: no containment, community corrections, and department of corrections. Nearly all evaluations (n = 301; 99.7%) recommended some form of containment, and recommendations were for either community corrections (n = 161; 53.3%) or department of corrections (n = 140; 46.4%). We expected evaluators’ containment recommendations to be more strongly influenced by the SOMB ratings, which allow room for clinical judgment and may provide justification for overrides, than the actuarial risk ratings.

Procedure

Ethical approval was granted by the Institutional Review Board of Sam Houston State University. The agency provided electronic copies of de-identified SOMB evaluation reports to the first author. Each report was written by 1 of 7 evaluators working for the same private agency. In each report, the evaluator reported the results of three risk assessment instruments (SORAG, VRAG, Static-99) and classified the result of each instrument on the following five-category risk scale: Low, Low-Moderate, Moderate, Moderate-High, High. The evaluator also rated the defendant on each of the 17 SOMB risk factors (Table 1), using the same five-category risk scale. The evaluator then used the five-category risk scale to provide two summary risk ratings: a summary rating for the level of risk indicated by the actuarial results and a separate summary rating of the level of risk indicated by the 17 SOMB factors. After making the two summary risk ratings, the evaluator made a containment recommendation. The containment recommendations fall into one of three categories, ranging from least restrictive to most restrictive: no containment, community corrections, and department of corrections.

We coded the evaluators’ reports to examine which of the 17 SOMB risk factors best predicted their final summary ratings of SOMB risk and which actuarial instrument results best predicted their final summary ratings of actuarial risk. We then used the two sets of summary risk ratings to examine which type of summary risk rating (SOMB or actuarial) best predicted their containment recommendations.

Two clinical psychology doctoral students with MA degrees in clinical/forensic psychology read and coded de-identified copies of the electronic reports. The coders recorded the risk-level rating for each of the 17 SOMB factors, the three risk instruments, and the two summary risk ratings. They also listed the offender’s current conviction, prior offenses, diagnoses, PCL-R percentile score, and the evaluator’s placement recommendation. Both raters coded the first 151 reports (rater agreement = 100% for all variables). The remaining reports (n = 151) were coded by only one evaluator.

Results

The five-category ordinal rating scale that evaluators used to make SOMB factor, actuarial score, and summary risk ratings (1 = low risk, 5 = high risk) presents a challenge for data analysis in this study, especially because each evaluator made 22 different ordinal ratings (17 SOMB factors, three instrument results, two risk ratings). Although these ratings represent ordered categories, analyzing them as categorical variables in correlation and regression analyses quickly becomes unwieldy (e.g., 68 predictors in a model examining the association between the 17 SOMB ratings and risk). The categorical approach requires four separate variables (e.g., dummy coded) to represent a single rating item. Without follow-up analyses, results for these sets of coded variables provide only limited information about the pattern effects across all of the ordinal categories. Thus, we opted to treat the rating items as continuous variables in correlation and regression analyses, while providing separate descriptive statistics for these ratings as both categorical and continuous variables (see Table 2). To ensure that this approach did not lead to spurious findings, we examined the pattern of correlations between study variables using both Pearson and Kendall’s tau-b correlations (available from the first author) and found that they led to the same substantive conclusions.

SOMB Risk Factor and Actuarial Risk Instrument Ratings

Table 2 provides descriptive statistics for evaluators’ ratings of the 17 SOMB risk factors and actuarial instrument results. Recall that the evaluator provided a separate risk-level rating (Low, Low/Moderate, Moderate, Moderate/High, High) for each individual SOMB risk factor and for each actuarial instrument result. The ratings in Table 2 provide information about the perceived level of risk indicated by the offender’s level of the trait, as opposed to simply the offender’s level of the trait. For example, the first cell in Table 2 indicates that evaluators believed that 52% of offenders had a level of empathy suggesting high risk, not that 52% of offenders had a high level of empathy.

For eight of the 17 SOMB factors, evaluators were more likely to assign a high-risk rating than any other rating (see Table 2). These eight factors were also the most likely to be rated as high risk overall and included Empathy (52.0%), Lifestyle Characteristics and Motivation to Change (45.7%), Psychopathology (44.7%), Motivation for Treatment (41.7%), and Criminal History (41.4%). Conversely, evaluators most frequently gave low-risk ratings on four factors, including Self-Worth (77.8%), Sexual Deviance and Arousal Patterns (63%), Developmental Markers (45.7%), and Cooperation During the Evaluation (44.7%).

Actuarial instrument results rarely fell into either the high-risk or low-risk categories, with most falling in one of the moderate-risk categories. For example, evaluators reported high-risk results for only 6.3% of the sample on the VRAG, 13.2% on the SORAG, and 15.2% of the Static-99. The most common risk categories for actuarial instruments were moderate for the VRAG (57.9%) and SORAG (38.1%), and moderate/low for the Static-99 (40.7%).

Summary Risk Ratings

The most common summary risk rating was moderate risk, for both the SOMB factors (41.4%) and actuarial instrument results (36.8%; see Table 2). Agreement between the SOMB and actuarial risk summary ratings was fair to poor (Kappa = .21, 95% confidence interval [CI] [.13, .28]; see Table 3). The two sets of ratings were identical for only 126 cases (42.3%). The SOMB rating was higher than the actuarial rating in 124 cases (41.6%), but the actuarial rating was higher than the SOMB rating in only 48 cases (16.1%; Table 3). In other words, evaluators were significantly more likely to conclude the SOMB factors indicated higher risk than the actuarial instrument results, χ²(16, N = 298) = 146.86, p < .001, Cramer’s v = .35, 95% CI [.31, .43]. If we treat these two risk ratings as continuous variables, evaluators assigned significantly higher SOMB risk ratings (M = 2.43) than actuarial ratings, M = 2.11; t(297) = 6.20, p < .001, d = .31, 95% CI [.24, .48], a small but statistically significant effect.

Table 3:

Agreement Between SOMB and Actuarial Risk Ratings

Summary risk rating (SOMB)	Summary risk rating (actuarial)
Summary risk rating (SOMB)	Low	Low/moderate	Moderate	Moderate/high	High
Low	2	2	1	0	0
Low/moderate	11	17	9	1	1
Moderate	5	30	63	25	1
Moderate/high	2	8	32	32	8
High	0	1	6	29	12
Kappa						.21**
SE						.04

Note. N = 298. SOMB = Sex Offender Management Board. The bold-faced numbers indicate instances of agreement between SOMB risk ratings and actuarial risk ratings. Numbers above the diagonal are instances in which the actuarial risk rating indicated a higher level of risk than the SOMB risk rating. Numbers below the diagonal are instances in which the SOMB risk rating indicated a higher level of risk than the actuarial risk rating.

There were only 12 cases in which both the SOMB and actuarial results were described as indicating high risk. There were 36 cases in which evaluators concluded that the SOMB factors indicated high risk even though the actuarial results did not, but there were only 10 cases in which evaluators concluded that the actuarial results indicated high risk even though the SOMB factors did not. This is a large, and statistically significant difference for high-risk ratings, χ²(1, N = 302) = 26.52, p < .001, odds ratio = 8.00, 95% CI [3.22, 19.86]. In most of these cases, the disagreement was due to the actuarial results indicating moderate/high risk and the SOMB factors indicating high risk (see Table 3).

Association between SOMB Risk Factors and Perceived Risk

Table 4 lists Pearson correlations among the evaluator-rated SOMB risk factors and perceived risk. Many of the SOMB factors were significantly correlated with one another, with most correlations falling in the .20 to .40 range (Table 4). There were several moderate to large correlations (e.g., r > .60), such as those between Level of Denial and Accountability for Offense (r = .84), Lifestyle/Past Behavior and Criminal Behavior (r = .71), and Empathy and Accountability for the Offense (r = .65), but many correlations fell in the .00 to .15 range (see Table 4). Each of the 17 SOMB factors was positively correlated with evaluators’ SOMB summary risk ratings (r = .05 to .57), with 16 of the 17 correlations being large enough to reach statistical significance. Most of these correlations were small to moderate in size, suggesting no single factor had an overwhelming impact on evaluators’ risk opinions. The largest correlations with the SOMB summary risk rating were for Criminal History (r = .57), Psychopathology (r = .57), Understanding of Behavior and Motivation to Change (r = .56), and Empathy (r = .55).

Table 4:

Correlations Between SOMB Risk Factors and Summary Risk Ratings

Factor/rating	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18
SOMB factors
1. Offense history
2. Sex deviance/arousal	.13*
3. Developmental mark	.08	−.02
4. Substance abuse	.19**	.04	.22**
5. Criminal history	.26**	.08	.25**	.47**
6. Treatment history	.24**	.18*	.25**	.41**	.41**
7. Social support system	.21**	.07	.27**	.22**	.31**	.30**
8. Level of denial	.13*	.00	.09	−.01	.10	.06	.18*
9. Accountability	.22**	.01	.14*	−.03	.12*	.06	.18**	.84**
10. Cooperation	.13*	.06	.06	−.13	.15*	.14*	.25**	.36**	.40**
11. Level of empathy	.24**	.14*	.03	.01	.20**	.17*	.25**	.55**	.65**	.34**
12. Lifestyle/past behavior	.35**	.12	.34**	.50**	.71**	.35**	.37**	.12*	.16*	.09	.23**
13. Psychopathology	.29**	.06	.25**	.25**	.40**	.31**	.28**	.26**	.27**	.16*	.31**	.52**
14. Understanding	.20**	.10	.12	.03	.23**	.23**	.14*	.51**	.57**	.30**	.60**	.24**	.37**
15. Treatment motivation	.13*	.08	.10	−.08	.16*	.17*	.11	.47**	.50**	.30**	.54**	.15*	.31**	.61**
16. Self-worth	.01	.07	.24**	−.02	−.10	.09	.26**	.03	.01	−.02	−.09	.02	.04	−.03	−.09
17. Defensiveness	.17*	.11	.04	.01	.17*	.15*	.15*	.46**	.50**	.51**	.48**	.14*	.35**	.49**	.52**	−.10
Summary risk ratings
18. SOMB risk rating	.45**	.19*	.30**	.31**	.57**	.43**	.44**	.44**	.50**	.30**	.55**	.54**	.57**	.56**	.47**	.05	.48**
19. Actuarial risk rating	.30**	.12*	.33**	.39**	.65**	.39**	.27**	.14*	.15*	.15*	.21**	.61**	.54**	.22**	.16*	−.02	.18**	.59**

Note. SOMB = Sex Offender Management Board.

p < .05. **p < .001.

We used a multilevel regression to provide a more comprehensive picture of the association between SOMB risk factors and perceived risk. We used multilevel regression because the data for this study are nested, with offenders (N = 302) nested under evaluators (N = 7). Multilevel regression accounts for the variance in risk ratings attributable to the specific evaluator, such as the fact that some evaluators tend to assign higher risk ratings than others (see Boccaccini, Murrie, Rufino, & Gardner, 2014). In the current study, evaluator differences accounted for 8% of the variance in SOMB summarized risk ratings and 21% of the variance in summarized actuarial risk ratings.

To control for nested outcomes due to evaluator differences, we created a new variable that ranked each case for each evaluator and controlled for this variable’s effect as a random effect in the model (see Heck, Thomas, & Tabata, 2010). We then entered the 17 SOMB risk factor ratings into a single model (as fixed factors) to examine the factors that were the most predictive of evaluators’ summary risk ratings. Evaluator differences accounted for 8% of the variance of the SOMB summary risk ratings. After accounting for this evaluator variance, nine of the 17 SOMB factors were statistically significant predictors of SOMB summary risk ratings (see Table 4). The strongest predictors of SOMB summary ratings were Criminal History (b = .18; p < .001, 95% CI [.12, .25]), Offense History and Victim Choice (b = .14, p < .001, 95% CI [.07, .20]), and Psychopathology (b = .11; p < .001; 95% CI [.05, .18]).

Because evaluators used the PCL-R to assist them in making psychopathology ratings, we were curious about the extent to which PCL-R ratings might explain the association between the SOMB psychopathology rating and risk. PCL-R ratings were significantly correlated with psychopathology (r = .54, p < .001) and SOMB risk (r = .63, p < .001).² We used PCL-R scores and psychopathology ratings in a multilevel regression model predicting SOMB risk ratings. Although PCL-R scores were predictive of risk ratings in this model (b = .06, 95% CI [.05, .07], SE = .01, p < .001), psychopathology ratings continued to predict risk ratings (b = .24, 95% CI [.17, .32], SE = .04, p < .001), indicating that the effect for psychopathology ratings was not solely attributable to PCL-R scores.

Association between Actuarial Instrument Results and Perceived Risk

Evaluators’ ratings of the level of risk indicated by the actuarial instrument results were correlated with one another, with the strongest correlations between the VRAG and SORAG (r = .66, p < .001) and SORAG and Static-99 (r = .56, p < .001). The correlation between the VRAG and Static-99 was somewhat smaller (r = .32, p < .001). Each actuarial instrument rating was significantly correlated with evaluators’ actuarial summary risk rating: r = .72 for the Static-99, r = .63 for the VRAG, and r = .81 for the SORAG (ps < .001).

We used multilevel regression to examine whether each instrument result made a unique contribution to evaluators’ summary risk ratings or whether one instrument seemed to be more influential than the others. The regression model revealed that each instrument result was a significant predictor of summary risk ratings, with the largest effects for the two scales specifically designed to predict reoffending among sexual offenders: SORAG (b = .35, p < .001, 95% CI [.28,.42]), Static-99 (b = .31, p < .001, 95% CI [.26,.36]).

These regression findings suggest that evaluators consider the pattern of findings across all risk instruments when coming to a conclusion about actuarial risk. Because each instrument result was independently predictive of summary risk ratings, it seemed possible that evaluators were using something akin to a counting approach for risk measure integration, concluding that offenders with high scores on more instruments had a higher risk of reoffending than offenders with high scores on fewer instruments. To examine whether this might be the case, we used a one-way ANOVA to compare the mean actuarial risk rating for offenders who scored in the moderate-high or high-risk range on all three instruments (n = 23), two instruments (n = 54), one instrument (n = 82), or no instrument (n = 139). There was a large and statistically significant effect, F(3, 294) = 110.75, p < .001, η²_partial = .53, with moderate to large differences for each increase in the number of moderate-high or high-risk scores. For example, summary risk ratings were higher for those with three moderate-high or high scores than two (Cohen’s d = .51, 95% CI [.01, 1.00]), two moderate-high or high scores than one (d = 1.45, 95% CI [1.07, 1.84]), and one moderate-high or high score than none (d = 1.19, 95% CI [.89, 1.48]).

Which Type of Risk Rating Best Predicts Evaluators’ Containment Recommendations?

Evaluators’ SOMB and actuarial summary risk ratings were strongly correlated with one another (r = .59, p < .001). We used a series of generalized multilevel models to examine which type of summary risk rating best predicted evaluators’ containment recommendations. We used only SOMB summary risk ratings as a predictor in the first model, only actuarial summary risk ratings in the second, and both SOMB and actuarial summary risk ratings in the third. We coded the containment recommendation so that a value of one indicated a more secure placement (department of corrections) and a value of zero indicated a less secure placement (community corrections).

When examined in separate models, each type of summary risk rating was predictive of containment recommendations, with higher risk ratings indicating a higher likelihood of a more secure containment recommendation (Table 5). But when we entered both types of summary risk ratings into the model, only the SOMB risk ratings were predictive of containment recommendations.

Table 5:

Multilevel Regression Results for Models Examining Factors That Predict Summary Risk Ratings and Containment Recommendations

Model/factor	Estimate (b) & 95% CI	SE	t value	Wald Z
Predictors of summary risk ratings
SOMB factors predicting summary risk
Criminal history	.18** [.12, .25]	.03	5.54
Offense history and victim choice	.14** [.07, .20]	.03	4.14
Defensiveness	.12* [.03, .20]	.04	1.80
Psychopathology	.11** [.05, .18]	.03	3.46
Level of empathy	.10* [.03, .17]	.03	2.77
Understanding of behaviors/change motivation	.10* [.02, .18]	.04	2.45
Social support system	.09* [.02, .15]	.03	2.76
Developmental markers	.06* [.00, .13]	.03	2.08
Self-worth	.06 [−.02, .13]	.04	1.50
Prior treatment history	.05 [−.01, .10]	.03	1.61
Cooperation during the evaluation	−.05 [−.13, .03]	.04	−1.30
Accountability for offense	.05 [−.04, .15]	.08	1.08
Sexual deviance and arousal patterns	.04* [.00, .09]	.02	2.05
Motivation for treatment	.04 [−.02, .10]	.03	1.30
Level of denial	.04 [−.05, .12]	.04	0.83
Substance abuse	.03 [−.03, .08]	.03	0.91
Lifestyle characteristics and past behavior	−.01 [−.08, .07]	.04	−0.16
Residual	.29 [.24, .34]	.03		11.31**
Instrument results predicting summary risk (Actuarial)
SORAG	.35** [.29, .42]	.04	10.49
VRAG	.23** [.15, .31]	.04	5.58
Static-99	.31** [.26, .36]	.03	12.19
Residual	.22 [.19, .26]	.02		10.24**
Summary risk ratings as predictors of containment recommendations
Model 1
SOMB summary risk rating	.57** [.47, .67]	.05	11.32
Residual	.70 [.60, .82]	.06		12.23**
Model 2
Actuarial summary risk rating	.38** [.28, .48]	.05	7.24
Residual	.85 [.72, .99]	.07		12.17**
Model 3
SOMB summary risk rating	.50** [.38, .63]	.06	8.07
Actuarial summary risk rating	.10 [−.02, .21]	.06	1.62
Residual	.69 [.57, .85]	.07		9.74**

Note. N = 298. CI = confidence interval; SOMB = Sex Offender Management Board; SORAG = Sex Offender Risk Appraisal Guide; VRAG = Violence Risk Appraisal Guide.

p < .05. **p < .001.

Discussion

Overall, our findings suggest that risk instrument results mattered to the Colorado SOMB evaluators when coming to conclusions about offenders but that other factors mattered more. Evaluators’ ratings of the level of risk indicated by actuarial instrument results were predictive of their containment recommendations when examined alone but were no longer predictive once we included their ratings of the level of risk indicated by noninstrument factors (i.e., SOMB risk). Only their SOMB risk ratings were independently predictive of containment recommendations.

The extent to which these findings indicate appropriate evaluator decision making is complicated. On one hand, these findings suggest that evaluators are failing to take full advantage of the benefits in predictive validity that come with using a well-studied actuarial risk assessment instrument as it is intended to be used. Several studies have shown that adjusting or overriding risk instrument results leads to a decrease in predictive validity, especially when many of the overrides are used to increase the level of risk (see Hanson & Morton-Bourgon, 2009; Schmidt et al., 2016; Storey et al., 2012). There was indirect evidence of evaluators using overrides in this study. Evaluators’ SOMB risk ratings indicated a higher level of risk than their actuarial risk ratings in 41.6% of cases, and it was only their SOMB risk ratings that were predictive of containment recommendations in our final regression model. This pattern seems to suggest that evaluators often used SOMB ratings to override instrument results when making containment recommendations. Because we did not have access to recidivism data in this study, we were not able to assess the impact of overrides on accuracy. Although prior override research suggests that overrides are associated with attenuated predictive effects, the effect on predictive validity in the Colorado SOMB context is unknown.

On the other hand, the evaluators’ practices were consistent with the Colorado SOMB risk assessment guidelines, which require evaluators to consider each of the 17 SOMB factors (Table 1) as part of their evaluations. The SOMB specifically instructs evaluators to “incorporate all available information when making a determination regarding risk and not rely solely on risk assessment instruments” (SOMB, 2011, p. 32). Thus, the extent to which evaluators’ ultimate opinions reflected their beliefs about the factors most strongly associated with risk, those dictated by Colorado policy, or some combination of both is unclear. It could be that the SOMB evaluators in this study felt compelled to rely on SOMB factors, even those with questionable validity. Policy mandated risk assessment practices almost certainly lead to more consistency across evaluators and evaluations, but policy makers sometimes adopt promising procedures or measures without adequate research support for use in their system (see Boccaccini, Murrie, Caperton, & Hawes, 2009, for an example). Policies intended to improve decision making may ultimately force evaluators to rely on outdated science and procedures. Ideally, states would not mandate procedures in the absence of local field validity studies. Unfortunately, truly prospective field validity studies take years to complete and are just now becoming available to evaluators in some jurisdictions (see Boccaccini et al., 2009; Boccaccini, Rice, Helmus, Murrie, & Harris, in press). Because policies are slow to change, evaluators must look to the emerging research literature to guide their practices and work within their local systems to facilitate change when research findings clearly contradict current policy.

Evaluators’ SOMB and actuarial risk ratings were moderately to strongly correlated (r = .59), likely due to overlap between the list of SOMB factors and risk instrument items (e.g., criminal history, victim type, sexual deviance). If the SOMB and actuarial risk ratings are influenced by similar factors, it could be that both are predictive of future offending. But closer inspection of the associations between SOMB factors and risk ratings (Table 5) suggest that it is unlikely that SOMB risk ratings would outperform actuarial risk ratings. Researchers have described several of the factors that were most predictive of SOMB risk ratings, such as lack of empathy and poor motivation for treatment as “potentially misleading” because of their poor association with future offending (Hanson & Morton-Bourgon, 2005, p. 1158). Although evaluators’ SOMB risk ratings were also associated with known predictors of sexual recidivism (offense history, sexual deviance), it seems likely that opinions based on both predictive and nonpredictive factors will be less accurate than those based on predictive factors alone. Using a nonpredictive factor to make a prediction will always lead to an increase in error, even if it is used alongside other predictive factors. If a factor is truly unrelated to the outcome, it should have no influence on the prediction. If a nonpredictive factor is used alongside a truly predictive factor, it will just “muddy the water” and lead to a weaker overall predictive effect. Unfortunately, we did not have permission to obtain recidivism data for this study and have no way to compare the validity of the SOMB and actuarial risk ratings.

Gaps between Research and Practice

Several of our findings highlight persistent gaps between risk assessment research and risk assessment practice. Despite the consistent finding that decisions based on unadjusted actuarial risk instrument measure scores outperform those based on overrides or adjustments, most clinicians are unlikely to base their conclusions on instrument scores alone (Chevalier et al., 2015; Vrieze & Grove, 2009) and many use overrides (Schmidt et al., 2016). Researchers consistently find that overrides and adjustments do not work, but overrides remain common. At this point, it does not seem that more research documenting stronger effects for unadjusted scores is going reduce the tendency of clinicians to make adjustments.

It seems that the field would benefit from more sophisticated studies of the adjustment and override process, such as those that identify the most common reasons and methods for making adjustments. Indeed, proponents of actuarial decision making acknowledge that there are special circumstances that necessitate deviating from model results. Perhaps most well known is Meehl’s (1957) classic broken leg scenario, in which a broken leg might render meaningless the results of an equation predicting movie-going. But Meehl also warned that we should “beware of overdoing the broken leg analogy” (p. 270) and to use them only when “the psychological situation is as clear as a broken leg” (p. 273). It seems likely that adjustments are useful in some circumstances but harmful in others. Findings from studies comparing the validity of specific types of adjustments (i.e., adjustments based on specific factors) could better help clinicians understand why they should or should not use them. Existing studies simply compare the validity of adjusted and unadjusted results, lumping all adjustments into a single category. Studies that identify and test specific types of adjustments may help clinicians better understand why their adjustment practices are or are not helpful.

Clinicians’ tendency to override actuarial instrument scores in risk assessment may be one reason why many have adopted structured professional judgment (SPJ) measures, which allow for clinician judgment when coming to conclusions about risk. One possible solution to the problem with overrides is to replace actuarial measures with SPJ measures. Although broad reviews of the risk assessment literature have reported similar effects for SPJ measures and the types of actuarial measures used in this study (see, e.g., Yang, Wong, & Coid, 2010), meta-analyses of sex offender specific studies suggest stronger effects for actuarial measures (d = .67) than SPJ measures (d = .46; Hanson & Morton-Bourgon, 2009). Moreover, actuarial measures, especially the Static-99R, continue to be the most widely used measures in sex offender risk assessment (Chevalier et al., 2015; Neal & Grisso, 2014), suggesting that the field cannot overlook the adjustment issue because of the availability of SPJ measures.

Another noticeable gap between research and practice concerns the practice of using multiple risk assessment measures for decision making. Although it is common practice for researchers to use multiple measures in sex offender risk assessment cases (Chevalier et al., 2015; Neal & Grisso, 2014), there are no empirically based guidelines for integrating scores across measures to come to a single or summary conclusion about risk (Seto, 2005; Vrieze & Grove, 2010). In the current study, evaluators appeared to use a counting approach for risk measure integration. They concluded that offenders with moderate-high or high scores across more measures were at a higher risk for reoffending than those with scores in this range on fewer measures. Although it seems to make sense that more high-risk scores indicates a higher level of risk than fewer high-risk scores, researchers have had a difficult time finding measure combinations that are associated with an increase in predictive validity. Although some studies have found improved prediction with multiple measures (Lehmann et al., 2013), most others have not (Boccaccini et al., 2009; Looman, Morphett, & Abracen, 2013; Seto, 2005).

One factor that makes it difficult to integrate scores across risk measures is that their scores tend to be positively correlated (r = .32 to .66 in this study). Many of these measures include identical (i.e., VRAG and SORAG) or similar (Static-99 and SORAG) items. Using a counting approach to risk measure integration treats each score as an independent source of information about risk, which is not the case. The counting approach leads evaluators to give more weight to factors that are on both measures. Clinicians nevertheless continue to use multiple measures despite research suggesting that there is no benefit to using more than one measure. The field would clearly benefit from more studies aimed at identifying the best methods for integrating scores across risk measures.

Limitations

The Colorado SOMB reports we used for this study provided a unique opportunity to study the associations between risk factors, instrument scores, risk opinions, and containment recommendations. The Colorado evaluators in this study used a systematic, ordinal rating process to quantify risk across more than 20 variables. Although this process prevented us from having to make inferences about the evaluators’ opinions, we know of no other state or jurisdiction that uses a similar process, which limits the generalizability of our findings. In addition, the practice of providing separate ordinal-level risk ratings for each SOMB risk factors may be unique to this specific agency. Although all Colorado evaluators are required to consider these factors, they are not required to provide a separate rating for each factor. There may be something about the process of formally rating each risk factor that makes it more salient in the decision-making process. This type of rating process may encourage evaluators to consider factors (e.g., empathy) that they know are not empirically supported predictors of reoffending. One limitation of the ordinal rating system is that there are no rules of guidance for assigning ratings. Instead, the evaluators consider the risk factor (or combination of risk factors) and then make a clinical judgment as to the level of risk indicated by the factor. This process may leave too much room for subjectivity in scoring, which opens the door for idiosyncratic evaluator differences in interpretation practices (Boccaccini et al., 2014).

A related limitation is that we had access to reports from only seven evaluators, from one agency. Our best estimate is that these evaluations represent between 5% and 10% of the evaluations conducted in the state. We used mixed-model regression to statistically account for evaluator-specific variance in our analyses, but the small number of evaluators necessarily limits the generalizability of study findings.

Other limitations include the absence of raw risk measure scores and risk measure item scores. Our analyses of the VRAG, SORAG, and Static-99 results were based on evaluators’ interpretation of the results as falling into one of five ordinal risk categories, as opposed to the actual risk measure scores. This categorization process prevented us from examining the full range of possible scores on the measures. Because any type of score categorization leads to range restriction and is often accompanied by attenuated validity coefficients (Howard, 2017), our findings for risk measure results may have been stronger if we had access to raw scores as opposed risk categories. Nevertheless, clinicians often provide categorical interpretations of risk measure results, even when noncategorical options are available (Chevalier et al., 2015). Thus, our findings for risk measure categories may better map on to actual practice than analyses for raw scores because the evaluators reported categories, not scores. For the PCL-R, evaluators reported percentile values, which we were able to convert to raw scores, but were not able to examine effects for the PCL-R factors and facets, which vary in their associations with recidivism among sexual offenders (Hawes et al., 2013).

Finally, we did not have access to offender outcomes for this study. Our interpretations of the evaluators’ decision-making processes are based on predictive validity findings from other studies. Although many of those predictive validity findings are consistent across studies (e.g., attenuated effects for overrides), the extent to which they apply to Colorado SOMB evaluations is unknown.

Conclusion

Our results show that the inclusion of actuarial instruments in risk assessment does not always mean the instruments are influential in evaluator decision making. Evaluators’ final containment recommendations were more strongly influenced by noninstrument factors, including some that are known predictors of recidivism (prior offending) and some that are not (empathy, treatment motivation). In many instances, the evaluators concluded that these noninstrument factors indicated a higher level of risk than instrument results. Overall, these findings seem to reflect a continuing divide between risk assessment research and risk assessment practice and highlight the need for more research that examines the validity of what risk assessment evaluators actually do, as opposed to what researchers think they should do.

Footnotes

Notes

Katherine E. McCallum, MA, is a doctoral student in the clinical psychology program at Sam Houston State University. Her research focuses on risk assessment, sexual offenders, and psychopathy.

Marcus T. Boccaccini, PhD, is a professor of psychology at Sam Houston State University, where he serves as the Associate Director of Clinical Training for the department’s clinical psychology doctoral program. His research program focuses on field reliability and field validity in forensic assessment.

Claire N. Bryson, MA, is a doctoral student in the clinical psychology program at Sam Houston State University. Her research focuses on forensic assessment and suicide risk assessment.

References

Abracen

Looman

(2006). Evaluation of civil commitment criteria in a high risk sample of sexual offenders. Journal of Sexual Offender Civil Commitment: Science and the Law, 1, 124-140.

Andrews

D. A.

Bonta

Wormith

J. S.

(2010). The Level of Service (LS) assessment of adults and older adolescents. In Otto

R. K.

Douglas

K. S.

(Eds.), Handbook of violence risk assessment (pp. 199-225). New York, NY: Routledge.

Boccaccini

M. T.

Murrie

D. C.

Caperton

J. D.

Hawes

S. W.

(2009). Field validity of the Static-99 and MnSOST-R among sexual offenders evaluated for civil commitment as sexually violent predators. Psychology, Public Policy, and Law, 15, 278-314. doi:10.1037/a0017232

Boccaccini

M. T.

Murrie

D. C.

Rufino

K. A.

Gardner

B. O.

(2014). Evaluator differences in PCL-R factor and facet level scoring. Law and Human Behavior, 38, 337-345. doi:10.1037/lhb0000069

Boccaccini

M. T.

Rice

A. K.

Helmus

L. M.

Murrie

D. C.

Harris

P. B.

(in press). Field validity of Static-99/R scores in a statewide sample of 34,687 convicted sexual offenders. Psychological Assessment.

Boer

D. P.

Hart

S. D.

Kropp

P. R.

Webster

C. D.

(1997). Manual for the Sexual Violence Risk-20: Professional guidelines for assessing risk of sexual violence. Vancouver, British Columbia, Canada: Mental Health, Law, and Policy Institute.

Chevalier

Boccaccini

M. T.

Murrie

D. C.

Varela

J. G.

(2015). Static-99R reporting practices in sexually violent predator cases: Does norm selection reflect adversarial allegiance? Law and Human Behavior, 39, 209-218. doi:10.1037/lhb0000114

Colorado Sex Offender Management Board. (2011). Standards and guidelines for the assessment, evaluation, treatment and behavioral monitoring of adult sex offenders. Denver: Colorado Department of Public Safety, Division of Criminal Justice, Office of Domestic Violence and Sex Offender Management.

DeMatteo

Murphy

Galloway

Krauss

D. A.

(2015). A national survey of United States sexually violent person legislation: Policy, procedures, and practice. International Journal of Forensic Mental Health, 14, 245-266. doi:10.1080/14999013.2015.1110847

10.

Epperson

D. L.

Kaul

J. D.

Goldman

Hout

Hesselton

Alexander

(1998). Minnesota Sex Offender Screening Tool–Revised (MnSOST-R). St. Paul, MN: Department of Corrections.

11.

Guy

L. S.

(2008). Performance indicators of the structured professional judgment approach for assessing risk for violence to others: A meta-analytic survey (Unpublished doctoral dissertation). Simon Fraser University, Burnaby, British Columbia, Canada.

12.

Guy

L. S.

Kusaj

Packer

I. K.

Douglas

K. S.

(2015). Influence of the HCR-20, LS/CMI, and PCL-R on decisions about parole suitability among lifers. Law and Human Behavior, 39, 232-243. doi:10.1037/lhb0000111

13.

Guy

L. S.

Nelson

R. J.

Fusco-Morin

S. L.

Vincent

G. M.

(2014). What do juvenile probation officers think of using the SAVRY and YLS/CMI for case management, and do they use the instruments properly? International Journal of Forensic Mental Health, 13, 227-241. doi:10.1080/14999013.2014.939789

14.

Hanson

R. K.

Morton-Bourgon

K. E.

(2005). The characteristics of persistent sexual offenders: A meta-analysis of recidivism studies. Journal of Consulting and Clinical Psychology, 73, 1154-1163. doi:10.1037/0022-006X.73.6.1154

15.

Hanson

R. K.

Morton-Bourgon

K. E.

(2009). The accuracy of recidivism risk assessments for sexual offenders: A meta-analysis of 118 prediction studies. Psychological Assessment, 12, 1-21. doi:10.1037/a0014421

16.

Hanson

R. K.

Thornton

(2000). Improving risk assessment for sex offenders: A comparison of three actuarial scales. Law and Human Behavior, 24, 119-136.

17.

Hare

R. D.

(2003). The Hare Psychopathy Checklist–Revised (2nd ed.). Toronto, Ontario, Canada: Multi-Health Systems.

18.

Harkins

Beech

A. R.

Goodwill

A. M.

(2010). Examining the influence of denial, motivation, and risk on sexual recidivism. Sexual Abuse, 22, 78-94. doi:10.1177/1079063209358106

19.

Harris

P. B.

Boccaccini

M. T.

Rice

A. K.

(in press). Field measures of psychopathy and sexual deviance as predictors of recidivism among sexual offenders. Psychological Assessment.

20.

Hart

S. D.

Kropp

P. R.

Laws

D. R.

Klaver

Logan

Watt

K. A.

(2003). The Risk for Sexual Violence Protocol (RSVP): Structured professional guidelines for assessing risk of sexual violence. Burnaby, British Columbia, Canada: Simon Fraser University Mental Health, Law, and Policy Institute.

21.

Hawes

S. W.

Boccaccini

M. T.

Murrie

D. C.

(2013). Psychopathy and the combination of psychopathy and sexual deviance as predictors of sexual recidivism: Meta-analytic findings using the Psychopathy Checklist–Revised. Psychological Assessment, 25, 233-243. doi:10.1037/a0030391

22.

Heck

R. H.

Thomas

S. L.

Tabata

L. N.

(2010). Multilevel and longitudinal modeling with IBM SPSS (Quantitative Methodology Series). New York, NY: Routledge.

23.

Helmus

Thronton

Hanson

R. K.

Babchishin

K. M.

(2012). Improving the predictive accuracy of Static-99 and Static-2002 with older sex offenders: Revised age weights. Sexual Abuse: A Journal of Research and Treatment, 24, 64-101. doi:10.1177/1079063211409951

24.

Hilton

N. Z.

Simmons

J. L.

(2001). The influence of actuarial risk assessment in clinical judgments and tribunal decisions about mentally disordered offenders in maximum security. Law and Human Behavior, 25, 393-408.

25.

Hilton

N. Z.

Simpson

A. I.

Ham

(2016). The increasing influence of risk assessment on forensic patient review board decisions. Psychological Services, 13, 223-231. doi:10.1037/ser0000068

26.

Howard

P. D.

(2017). The effect of sample heterogeneity and risk categorization on area under the curve (AUC) predictive validity metrics. Criminal Justice and Behavior, 44, 103-120. doi:10.1177/0093854816678899

27.

Jackson

R. L.

Hess

(2007). Evaluation for civil commitment of sex offenders: A survey of experts. Sexual Abuse: A Journal of Research and Treatment, 19, 425-448.

28.

Kingston

D. A.

Olver

M. E.

Harris

Wong

S. C.

Bradford

J. M.

(2015). The relationship between mental disorder and recidivism in sexual offenders. International Journal of Forensic Mental Health, 14, 10-22. doi:10.1080/14999.2014.974088

29.

Lehmann

R. J. B.

Hanson

R. K.

Bachishin

K. M.

Gallasch-Nemitz

Biedermann

Dahle

(2013). Interpreting multiple risk scales for sex offenders: Evidence for averaging. Psychological Assessment, 25, 1019-1024. doi:10.1037/a0033098

30.

Levenson

J. S.

Morin

J. W.

(2006). Factors predicting selection of sexual violent predators for civil commitment. International Journal of Offender Therapy and Comparative Criminology, 50, 609-629. doi:10.1177/0306624x06287644

31.

Looman

Morphett

N. A. C.

Abracen

(2013). Does consideration of psychopathy and sexual deviance add predictive validity to the Static-99R? International Journal of Offender Therapy and Comparative Criminology, 57, 939-965. doi:10.1177/0306624X12444839

32.

Meehl

P. E.

(1957). When shall we use our heads instead of the formula? Journal of Counseling Psychology, 4, 268-273. doi:10.1037/h0047554

33.

Miller

Maloney

(2013) Practitioner compliance with risk/needs assessment tools: A theoretical and empirical assessment. Criminal Justice and Behavior, 40, 716-736. doi:10.1177/0093854812468883

34.

Moulden

J. M.

Firestone

Kingston

Bradford

(2009). Recidivism in pedophiles: An investigation using different diagnostic methods. The Journal of Forensic Psychiatry & Psychology, 20, 680-701. doi:10.1080/14789940903174055

35.

Neal

T. M. S.

Grisso

(2014). Assessment practices and expert judgment methods in forensic psychology and psychiatry: An international snapshot. Criminal Justice and Behavior, 41, 1406-1421. doi:10.1177/0093854814548449

36.

Nuñes

K. L.

Hanson

R. K.

Firestone

Moulden

H. M.

Greenberg

D. M.

Bradford

J. M.

(2007). Denial predicts recidivism for some offenders. Sexual Abuse: A Journal of Research and Treatment, 19, 91-105.

37.

Office of Resarch and Statistics. (2011). Crime and justice in Colorado: 2008-2010. Denver, CO: Division of Criminal Justice.

38.

Quinsey

V. L.

Harris

G. T.

Rice

M. E.

Cormier

(1998). Violent offenders: Appraising and managing risk. Washington, DC: American Psychological Association.

39.

Quinsey

V. L.

Harris

G. T.

Rice

M. E.

Cormier

C. A.

(2006). Violent offenders: Appraising and managing risk (2nd ed.). Washington, DC: American Psychological Association. doi:10.1037/11367-000

40.

Rettenberger

Boer

D. P.

Eher

(2011). The predictive accuracy of risk factors in the Sexual Violence Risk-20 (SVR-20). Criminal Justice and Behavior, 38, 1009-1027. doi:10.1177/0093854811416908

41.

Schmidt

F. S.

Sinclair

S. M.

Thomasdόttir

(2016). Predictive validity of the Youth Level of Service/Case Management Inventory with youth who have committed sexual and non-sexual offenses. Criminal Justice and Behavior, 43, 413-430. doi:10.1177/0093854815603389

42.

Seto

M. C.

(2005). Is more better? Combining actuarial risk scales to predict recidivism among adult sex offenders. Psychological Assessment, 17, 156-167. doi:10.1037/10403590.17.2.156

43.

Storey

J. E.

Watt

K. A.

Jackson

K. J.

Hart

S. D.

(2012). Utilization and implications of the Static-99 in practice. Sexual Abuse: Journal of Research and Treatment, 24, 289-302. doi:10.1177/1079063211423943

44.

Viljoen

J. L.

McLachlan

Vincent

G. M.

(2010). Assessing violence risk and psychopathy in juvenile and adult offenders: A survey of clinical practices. Assessment, 17, 377-395. doi:10.1177/1073191109359587

45.

Vrieze

S. I.

Grove

W. M.

(2009). Survey on the use of clinical and mechanical prediction methods in clinical psychology. Professional Psychology: Research and Practice, 40, 525-531. doi:10.1037/a0014693

46.

Vrieze

S. I.

Grove

W. M.

(2010). Multidimensional assessment of criminal recidivism: Problems, pitfalls, and proposed solutions. Psychological Assessment, 22, 382-395. doi:10.1037/a0019228

47.

Webster

C. D.

Douglas

K. S.

Eaves

Hart

S. D.

(1997). HCR-20: Assessing the Risk of Violence, Version 2. Burnaby, Canada: Simon Fraser University and Forensic Psychiatric Services Commission of British Columbia.

48.

Wormith

J. S.

Hogg

Guzzo

(2012). The predictive validity of a general risk-needs assessment inventory on sexual offender recidivism and exploration of the professional override. Criminal Justice and Behavior, 39, 1511-1538. doi:10.1177/009385481255741

49.

Yang

Wong

C. P.

Coid

(2010). The efficacy of violence prediction: A meta-analytic comparison of nine risk assessment tools. Psychological Bulletin, 136, 740-767. doi:10.1037/a00204