Comment on “The Influence of Risk Assessment Instrument Scores on Evaluators’ Risk Opinions and Sexual Offender Containment Recommendations”

Point 1

From the beginning, there is a significant error in the language McCallum et al. (2017) use to describe the processes of the SOMB and the Adult Standards and Guidelines. The abstract purports that the 17 risk factors identified by the SOMB are required to be assessed by evaluators. This is, in fact, not true. McCallum and colleagues (2017, p. 1216) also note,

The SOMB standards state an evaluator “shall consider” a list of additional factors for estimating an offender’s level of risk. (see Table 1; Colorado SOMB, 2011, p. 25)

However, page 25 of the Adult Standards and Guidelines for the Assessment, Evaluation, Treatment, and Behavioral Monitoring of Adult Sex Offenders (Colorado SOMB, 2011) does not include such a mandate that requires evaluators to consider the list of 17 risk factors, and it is not clear where the authors are finding this mandate. In fact, in the list of tools that are either required or optional to assess risk, the list of 17 risk factors does not appear. The only instruction regarding the use of the list of 17 risk factors comes in the Appendix (A) of the Standards (p. 136):

Risk assessment refers to an evaluation of the client’s overall risk of sexual re-offense. Risk assessments are typically done as part of the evaluation but should reoccur regularly throughout treatment and post-treatment if legal supervision continues.

The following factors should be reviewed in estimating a client’s level of risk . . . (see Colorado SOMB, 2011, pp. 136-138, emphasis added)

It is suggested that individuals read the preface recommendations for correct interpretation, as standards and guidelines are separate and distinct. For example, in the introduction to the Adult Standards and Guidelines (Colorado SOMB, 2011, p. 3), it notes,

In the body of the document, standards are denoted by the use of the terms “shall”; guidelines are distinguished by the term “should.”

As a result, it is unclear why the authors presented the 17 risk factors as a required part of the assessment when they are not. There is a significant difference between a guideline to review a list of 17 factors as part of an evaluation of risk, as compared with mandating the evaluation of each individual risk factor. Therefore, the conclusion that the SOMB developed and required a risk assessment process to drive “containment recommendations” must be considered suspect.

Point 2

The second major concern with the manuscript was the scoring schema assigned to the 17 factors. There was no scoring system despite the fact that the McCallum et al. (2017) article included a scoring for each item, and it should be noted that not all risk items should be considered equal or be assigned equal weight. The list was never intended to be a risk assessment tool per se, and there was no expectation in terms of treating each item equally or the weight provided to each item by the evaluator. Therefore, a straight scoring system of the 17 items, which is then averaged, may not account for those factors that are of greater or lesser concern.

The authors write, “Colorado evaluators are required to consider the risk factors outlined by the SOMB (Table 1), but they are not required to provide the type of factor-by-factor risk ratings used by the evaluators in this study” (McCallum et al., 2017, p. 1219).

With the advent of the actuarial risk assessment tools for adult sex offenders, this list became less useful, and in fact, a certain percentage of Colorado evaluators no longer used the list at all. This should be noted as a significant measurement flaw in this manuscript, as these self-imposed ratings are not endorsed by the SOMB. Therefore, they should not be labeled as “SOMB risk ratings” as the authors have on page 17 and elsewhere in the manuscript. This is misrepresenting the role of the SOMB. Furthermore, the arbitrary way the evaluators assigned scores is absolutely not endorsed by the SOMB.

The Colorado SOMB (2011) Adult Standards and Guidelines clearly state,

To ensure the most accurate prediction of risk for sex offenders, the following evaluation modalities are all required in performing a sex offense-specific evaluation:

Use of instruments that have specific relevance to evaluating sex offenders

Use of instrument with demonstrated reliability and validity . . .

Use of at least one validated risk assessment instrument that was normed on the population most similar to the offender being evaluated. (p. 23)

As the authors correctly assert, the SOMB does add a qualifier to its requirements in the Standards and Guidelines in the following discussion point:

. . . Evaluators shall incorporate all available information when making a determination regarding risk and not rely solely on risk assessment instruments. Risk assessment instruments will not identify all risk factors . . . (Colorado SOMB, 2011, p. 31)

However, nothing in this discussion point mandates the use of any specific risk factors to override risk assessment instrument scoring. This qualifier notwithstanding, as the field evolved, so too did the Adult Standards and Guidelines, which is a segue to the third point.

Point 3

The list of 17 factors remained as an artifact of an earlier era in risk assessment, despite there being little emphasis placed on them. In addition, and as rightly pointed out by McCallum and colleagues (2017), a number of the identified risk factors have subsequently been identified in the research as not empirically supported. This inclusion continued until March 17, 2017 (Colorado SOMB, 2017), when the SOMB removed the 17 factors altogether based on the belief that they were no longer relevant to the current risk assessment process. Note that this occurred prior to the publication of the article. This removal caused not even a ripple of concern in the provider community, suggesting that the list was not considered to be of much prominence in the Adult Standards and Guidelines. If the authors had checked with the SOMB prior to publication of this article, they would have been briefed about these changes. In addition, the SOMB maintains an online version of its Adult Standards and Guidelines that contains all changes to date. It is not clear why the authors chose to use the 2011 print version of the Adult Standards and Guidelines when a more contemporary version exists on the SOMB website.

Point 4

There is another possible confounding variable for the findings from McCallum et al. (2017). The Colorado Adult Standards and Guidelines (Colorado SOMB, 2011) required that all sex offenders in severe denial of the sex offense be recommended for a certain level of containment (e.g., Department of Corrections) regardless of risk assessment score. Therefore, some of the “override” that occurred may have been based on that Standard rather than on the results of the 17 factors. While some may equally challenge the policy decision of the SOMB to preclude those offenders in denial from being eligible for community-based services based on the fact that denial is not a known risk factor (Hanson & Morton-Bourgon, 2005), and in fact the SOMB itself deleted this requirement in 2014 for the same reason, the SOMB’s statutory mandate also includes the requirement to ensure the physical and psychological safety of victims (C.R.S. § 16-11.7-103 (4) (a)). The issue of an offender in denial living in the community was seen as contrary to that statutory mandate. As a result, it is possible that the containment decisions offered by the evaluators in the study were being guided by that Standard and not the 17 factors. This would be relatively easy for the authors to check in the study, if the issue of severe denial and a subsequent Department of Corrections recommendation were quantified. However, given the authors’ lack of collaboration with the SOMB, this issue was not addressed.

Point 5

In addition, it is important for the reader to understand the context of the recent work by the SOMB in Colorado. The SOMB has embarked on a significant initiative over the past 4 years to train all relevant stakeholders including evaluators on risk assessment instruments, including the Vermont Assessment of Sex Offender Risk–2 (VASOR-2) static risk assessment instrument (McGrath, Hoke, & Lasher, 2013) and the Sex Offender Treatment Intervention and Progress Scale (SOTIPS) dynamic risk assessment instrument (McGrath, Cumming, & Lasher, 2013). In fact, the SOMB obtained financial support from the Office of Sex Offender Sentencing, Monitoring, Apprehending, Registering, and Tracking (SMART Office) in 2013 to provide eight trainings for approximately 600 participants over the course of a year by Dr. Robert McGrath and Ms. Georgia Cumming, two of the developers of the instruments. This training initiative also included a train-the-trainer component to sustain this training on these instruments in the future, which the SOMB has done. While the SOMB ultimately decided not to mandate the use of any one risk assessment tool and allows its evaluators to use their instrument of choice, the State Probation Department has mandated the use of both the VASOR-2 and SOTIPS for all sex offender probationers and the Department of Corrections similarly mandated the use of the SOTIPS.

As a result, the sex offender management system in Colorado is necessarily and inevitably reliant upon and informed by the use of actuarial risk assessment instruments. While any use of the 17 factors by evaluators in Colorado may have continued beyond the early years of the Adult Standards and Guidelines, as clearly was done by the one agency providing evaluation services in Colorado noted by McCallum and colleagues (2017), this was not SOMB mandated and may not be indicative of current or recent practice in the state. To make broad conclusions about a state system based on the work of one agency seems overreaching and of questionable generalizability, and it is possible that the one agency in question is an outlier in the state in terms of assessing risk. In fact, this could be easily assessed by surveying other providers within the state on their use, or nonuse of the 17 factors, to clarify the findings. As such, this study seems to be more a case study of one agency than a broad-based analysis of state-level policy. In fact, McCallum et al. (2017) address this limitation in the manuscript (see page 7).

The SOMB strives, and in fact is statutorily mandated, to develop and revise the Adult Standards and Guidelines consistent with “a comprehensive review of evidence-based practices, (and) the effectiveness of the evaluation, identification, and treatment policies and procedures for adult sex offenders . . .” (C.R.S. § 16-11.7-103 (4) (e)). To suggest that the SOMB is doing anything less than this without objective proof recklessly suggests the SOMB is in violation of its legislative mandate.

Conclusion

While we concede the focus of the article and discussion are of value to the professional community, within its proper context and limitations, and the issue of overriding of risk scores is a genuine concern to the field for a variety of reasons (e.g., unnecessary use of resources on lower risk offenders, evaluator concern for perceived liability leads them to default to a higher risk score and intervention, etc.), McCallum et al. (2017) should have been more careful in their assessment and review. In particular, the authors should have reached out to the SOMB to ensure the accuracy of their information. To do anything less casts an unnecessary shadow on their conclusions at best, and calls into question the integrity of the research at worst.

Contrary to the conclusion of McCallum et al. (2017), the comparison of an actuarial risk assessment instrument tool with a list of 17 factors (never designed to be a risk assessment tool) that was operationalized by one treatment agency in a certain way presents a number of problems. The scoring system used by the agency has not been validated, and to our knowledge, there has been no attempt to develop a uniform approach to defining and scoring the items. Therefore, the comparison with an actuarial risk assessment instrument process, which has been subject to these parameters, appears premature and the conclusions drawn are less meaningful.

In addition, and even more importantly, the SOMB neither compelled the evaluators to rely on the SOMB factors, nor did it mandate evaluators rely on outdated science and procedures. The SOMB takes pride in providing up-to-date, research-supported practices for its providers and would never intentionally do otherwise, even in the interests of supporting its containment approach or enhanced consistency. To impugn the reputation of the SOMB due to a flawed methodology and policy interpretation is not consistent with the tenets of good research practice.