Abstract
Hecker raises a number of important considerations in interpreting the review of literature and empirical research presented in Fanniff and Letourneau’s article “Another Piece of the Puzzle: Psychometric Properties of the J-SOAP-II.” Despite these considerations and the publication of new research in the interim, the current authors continue to urge caution in the interpretation of Juvenile Sex Offender Assessment Protocol-II (J-SOAP-II) scores in disposition evaluations, largely due to the serious consequences that youth face in this context.
Keywords
Hecker (2014) raised important concerns about our (Fanniff & Letourneau, 2012) review of the literature and our empirical analyses as pertained to Scale 1 of the Juvenile Sex Offender Assessment Protocol-II (J-SOAP-II). Recognizing that we are all striving to prevent sexual abuse and provide for the best responses to youth who have sexually offended and their communities, we appreciate this opportunity to respond. While we acknowledge the legitimacy of Hecker’s points, we stand by our original conclusion that evaluators should be cautious regarding the use of J-SOAP-II Scale 1 in the context of disposition evaluations. The state of the science regarding existing risk assessment measures for juveniles adjudicated for sexual offenses (JSOs) remains inadequate in the face of the high stakes faced at disposition by this population, including extended periods of incarceration and/or secure treatment placement, post-incarceration civil commitment, sex offender registration and public notification, and other collateral consequences such as residency restrictions (Letourneau & Levenson, 2010).
Response to Key Points
Reliability
First, regarding the internal consistency of Scale 1, we acknowledge that there is limited research, that two of the four studies reporting this statistic found acceptable alpha coefficients, and that the translation of the measure in a third study may have negatively impacted consistency. As we originally noted, internal consistency is not the only or even the most important indicator of reliability. This is especially true for scales not intended to measure a single coherent construct, as Hecker suggested is true for Scale 1. We would simply note that presentation of each J-SOAP-II scale as representing a “domain” (Prentky & Righthand, 2003, p. 2) and previous reports of alpha coefficients and factor analysis by the test developers (Prentky, Harris, Frizell, & Righthand, 2000; Righthand et al., 2005) suggested that Scale 1 was intended to capture a coherent underlying construct.
Predictive Validity
Hecker’s point regarding the use of the measure in the intended manner (i.e., with adolescents who have committed hands-on sexual offenses) is well taken. J-SOAP-II’s developers state they do not endorse its use with children under age 12 (despite having published research on its use with children as young as 3 years; Prentky et al., 2010). Based on the inclusion of younger participants and those without hands-on offenses, the results of Aebi, Plattner, Steinhausen, and Bessler (2011) should indeed be interpreted cautiously.
Hecker also suggested that our review of predictive validity research was overly pessimistic. For example, we counted as less relevant relationships between Scale 1 and sexual acting out during treatment than the lack of relationships between Scale 1 and post-treatment recidivism (Viljoen et al., 2008). This emphasis was consistent with our review’s focus on the use of the J-SOAP-II in disposition evaluations. We also discounted one study’s reported area under the curve (AUC) of .72 due to a confidence interval that included .50 (i.e., chance prediction; Powers-Sawyer & Miner, 2009).
Findings unavailable at the time of our review are more encouraging, including the results of Chu, Ng, Fong, and Teoh (2012) and a meta-analysis that reported a moderate but significant AUC of .61 for Scale 1 and sexual recidivism (Viljoen, Mordell, & Beneteau, 2012). We remain of the opinion that existing evidence remains insufficient for supporting the use of Scale 1 in disposition evaluations. In particular, we are concerned that much of the predictive accuracy for Scale 1 and similar measures derives from their ability to accurately identify low-risk individuals (negative predictive value) but not high-risk individuals (positive predictive value). In a comprehensive evaluation of recidivism risk instruments, Fazel and colleagues found a positive predictive value of just .25 and a negative predictive value of .93 for adult-focused sexual recidivism instruments (Fazel, Singh, Doll, & Grann, 2012). Moreover, these authors argue that AUCs are often interpreted with more optimism than warranted, given instances of low positive predictive validity (Fazel et al., 2012; Singh, Grann, & Fazel, 2011). Worling and colleagues make similar points regarding the Estimate of Risk of Adolescent Sexual Offense Recidivism (ERASOR) (Worling, Bookalam, & Littlejohn, 2012). Fazel and colleagues note, and we concur, that tools with high negative predictive validity and moderate AUCs are beneficial in some contexts, including when used to “weed out” low risk individuals and for treatment planning purposes; however, a higher level of accuracy is demanded when test scores are used to inform decisions that curtail freedom or burden youth with long-term and potentially harmful regulations.
Fanniff and Letourneau Findings
Hecker also raised concerns regarding one of our outcome measures, a composite of two scales from the Adolescent Clinical Sexual Behavior Inventory (ACSBI; Friedrich, Lysne, Sim, & Shamos, 2004). As we noted in our article, the most relevant outcome measure for evaluating the predictive validity of Scale 1 is sexual recidivism, and the inability to examine sexual recidivism constituted a significant limitation of our study. However, we disagree that our combination of two ACSBI scales was “novel” given findings that suggest these scales contribute to a single factor (Wherry, Berres, Sim, & Friedrich, 2009). Furthermore, we maintain that it is reasonable to posit that high scores on this composite scale might serve as an indicator of recidivism risk though this remains untested, and consequently, the interpretation of our findings—specifically, a failure to find a significant relationship between baseline Scale 1 scores and follow-up ACSBI scores—remains debatable.
Conclusion
Hecker raised reasonable concerns regarding our study that along with our own acknowledgment of study limitations suggest cautious interpretation of our findings. Nevertheless, we assert that extant findings are too heterogeneous and support is too limited to endorse the use of Scale 1 in high-stakes dispositive assessments. Rather, we remind readers of Vitacco, Caldwell, Ryba, Malesky, and Kurus’s (2009) practice recommendations: to emphasize the fluid nature of adolescent development, sexuality, and risk; the low base rate of recidivism; the success of evidence-based treatment; and the importance of social context. To this we would add emphasis of findings that question the ability of sexual risk assessment measures to accurately identify high-risk individuals.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.