The Forensic Outpatient Risk Assessment and Evaluation (FORE): Examining Interrater Reliability and Predictive Validity in Dutch Forensic Outpatients

Abstract

This study evaluates the interrater reliability and predictive validity of a relatively new risk assessment tool designed for use in forensic outpatient populations, the Forensic Outpatient Risk Assessment and Evaluation (FORE). Data were collected from two samples: the first comprising 80 outpatients to assess interrater reliability between therapist–student and student–student rater pairs, and the second consisting of N = 1,268 patients for predictive validity analysis. Interrater reliability was generally good for static factors. The dynamic factors displayed more variability, with intraclass correlation coefficient values ranging from poor to good. The results demonstrated a good predictive validity for general recidivism (area under the curve = .72). The overall calibration was good (expected/observed [E/O] index = .98), although the lower predicted risk levels were overestimated and the higher levels underestimated. The overall findings suggest that FORE is a useful tool for forensic risk assessment, though further research into the psychometric qualities of the FORE is required.

Keywords

risk assessment forensic outpatients interrater reliability predictive validity recidivism

Get full access to this article

View all access options for this article.

References

Andrews

D. A.

Bonta

(1997). The Level of Service Inventory–Revised (LSI-R) manual. Multi-Health Systems.

Andrews

D. A.

Bonta

Wormith

J. S.

(2004). The Level of Service/Case Management Inventory (LS/CMI) manual. Multi-Health Systems.

Andrews

D. A.

Bonta

Wormith

J. S.

(2011). The Risk-Need-Responsivity (RNR) model: Does adding the Good Lives model contribute to effective crime prevention? Criminal Justice and Behavior, 38(7), 735–755. https://doi.org/10.1177/0093854811406356

Arifin

W. N.

(2020). Sample size calculator. GitHub Pages. http://wnarifin.github.io

Bogaerts

Spreen

Ter Horst

Gerlsma

(2018). Predictive validity of the HKT-R risk assessment for two and five-year recidivism in a cohort of Dutch forensic psychiatric patients. International Journal of Offender Therapy and Comparative Criminology, 62(8), 2259–2270. https://doi.org/10.1177/0306624x17717128

Bonta

Andrews

D. A.

(2024). The psychology of criminal conduct (7th ed.). Routledge. https://doi.org/10.4324/9781003292128

Bujang

M. A.

Baharum

(2017). A simplified guide to determination of sample size requirements for estimating the value of intraclass correlation coefficient: A review. Archives of Orofacial Sciences, 12(1), 1–11.

Canton

W. J.

Van der Veer

Van Panhuis

Verheul

Van der Brink

(2004). De betrouwbaarheid van risicotaxatie in de pro Justitia rapportage. Een onderzoek met behulp van de HKT-30. [Reliability of risk assessment with the HKT-30 in forensic psychiatric evaluation reports.]. Tijdschrift voor Psychiatrie, 46(8), 537–542.

Denault

Leclerc

Talwar

(2023). The use of nonverbal communication when assessing witness credibility: A view from the bench. Psychiatry, Psychology, and Law, 31(1), 97–120. https://doi.org/10.1080/13218719.2023.2175068

10.

De Ruiter

. (2009). B-Safer manual Dutch translation. Maastricht University.

11.

De Vet

H. C. W.

Terwee

C. B.

Mokkink

L. B.

Knol

D. L

. (2011). Measurements in medicine: A practical guide (1st ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511996214

12.

De Vogel

de Ruiter

. (2004). Differences between clinicians and researchers in assessing risk of violence in forensic psychiatric patients. Journal of Forensic Psychiatry and Psychology, 15(1), 145–164. https://doi.org/10.1080/14788940410001655916

13.

Douglas

K. S.

James

R. P.

Ogloff

J. D.

Hart

S. D.

(2003). Evaluation of a model of violence risk assessment among forensic psychiatric patients. Psychiatric Services, 54(10), 1372–1379. https://doi.org/10.1176/appi.ps.54.10.1372

14.

Drieschner

Tollenaar

(2021). Cahier 2021-18 recidive tijdens forensische zorgtrajecten 2013-2017 [Report recidivism during forensic treatment]. Wetenschappelijk Onderzoek en Data Centrum.

15.

Eisenberg

M. J.

Hutten

J. C.

Van der Veeken

F. C. A.

Bouman

Y. H. A.

Bogaerts

Van Horn

J. E.

(2020). Validatie FARE. Een multi-center studie naar de interbeoordelaarsbetrouwbaarheid, convergente validiteit, veranderbaarheid en de predictieve validiteit van de Forensische Ambulante Risico Evaluatie (FARE) [FARE validation: A multi-center study of the inter-rater reliability, convergent validity, changeability, and predictive validity of the Forensic Ambulatory Risk Assessment (FARE); Report validation of the FORE]. Kwaliteit Forensische Zorg.

16.

Fazel

Burghart

Fanshawe

Gil

S. D.

Monahan

(2022). The predictive performance of criminal risk assessment tools used at sentencing: Systematic review of validation studies. Journal of Criminal Justice, 81, 101902. https://doi.org/10.1016/j.jcrimjus.2022.101902

17.

Hanson

R. K.

(2017). Assessing the calibration of actuarial risk scales: A primer on the E/O index. Criminal Justice and Behavior, 44(1), 26–39. https://doi.org/10.1177/0093854816683956

18.

Hanson

R. K.

Morton-Bourgon

K. E.

(2009). The accuracy of recidivism risk assessments for sexual offenders: A meta-analysis of 118 prediction studies. Psychological Assessment, 21(1), 1–21. https://doi.org/10.1037/a0014421

19.

Harrell

F. E.

Jr. Califf

Pryor

Lee

Rosati

(1982). Evaluating the yield of medical tests. Journal of the American Medical Association, 247(18), 2543–2546. https://doi.org/10.1001/jama.1982.03320430047030

20.

Harrell

F. E.

Jr. Lee

K. L.

Mark

D. B.

(1996). Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine, 15(4), 361–387. https://doi.org/10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4

21.

Helmus

L. M.

Babchishin

(2017). Primer on risk assessment and the statistics used to evaluate its accuracy. Criminal Justice and Behavior, 40(1), 8–25. https://doi.org/10.1177/0093854816678898

22.

Hildebrand

Hesper

B. L.

Spreen

Nijman

(2005). De waarde van gestructureerde risicotaxatie en van de diagnose psychopathie. Een onderzoek naar de betrouwbaarheid en predictieve validiteit van de HCR-20, HKT-30 en PCL-R [The value of structured risk assessment and the psychopathy diagnosis. A study of the reliability and predictive validity of the HCR-20, HKT-30 and PCL-R.]. Expertisecentrum Forensische Psychiatrie.

23.

Hox

(2010). Multilevel analysis: Techniques and applications (2nd ed.). Routledge Academic.

24.

Johnson

S. M.

Bolstad

O. D.

(1973). Methodological issues in naturalistic observation: Some problems and solutions for field research. In Hamerlynck

L. A.

Handy

L. C.

Mash

E. G.

(Eds.), New developments in behavioral research: Theory, methods, and applications (pp. 167–188). Lawrence Erlbaum.

25.

Kazdin

A. E.

(1977). Artifact, bias, and complexity of assessment: The ABCs of reliability. Journal of Applied Behavior Analysis, 10, 141–150. https://doi.org/10.1901/jaba.1977.10-141

26.

Klinger

Ross

Bulla

(2020). Forensic outpatient variables that may help to prevent further detention. Frontiers in Psychiatry, 11, 42. https://doi.org/10.3389/fpsyt.2020.00042

27.

Koo

T. K.

M. Y.

(2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012

28.

Lam

K. ‘t

Lancel

Hildebrand

(2009). Manual of Short-term Assessment of Risk and Treatability (START). GGZ Drenthe.

29.

Lammers

Philipse

(2003). Manual of the Level of Service Inventory-revised (LSI-R) Dutch translation. Oldenkotte.

30.

Little

R. J. A.

(1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83(404), 1198–1202. https://doi.org/10.1080/01621459.1988.10478722

31.

(2021). Sample size formulas for estimating areas under the receiver operating characteristic curves with precision and assurance. Electronic Thesis and Dissertation Repository, 8045. https://ir.lib.uwo.ca/etd/8045

32.

Nickerson

(1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175

33.

Pepe

M. S.

(2003). The statistical evaluation of medical tests for classification and prediction. Oxford University Press.

34.

Philipse

de Ruiter

Hildebrand

Bouman

(2000). HCR-20. Beoordelen van het risico van gewelddadig gedrag. Versie 2 [HCR-20. Assessing the risk of violence, Version 2]. Prof. Mr. W. P. J. Pompestichting/Dr. Henri van der Hoeven Stichting.

35.

Powis

Randhawa-Horne

Elliott

(2019). Inter-rater reliability of the Extremism Risk Guidelines 22+ (ERG 22+). Ministry of Justice Analytical Series. https://doi.org/10.13140/RG.2.2.19651.68640

36.

Rice

M. E.

Harris

G. T.

(2005). Comparing effect sizes in follow-up studies: ROC area, Cohen’s d, and r. Law and Human Behavior, 29(5), 615–620. https://doi.org/10.1007/s10979-005-6832-7

37.

Rocque

Plummer-Beale

(2014). In the eye of the beholder? An examination of the inter-rater reliability of the LSI-R and YLS/CMI in a correctional agency. Journal of Criminal Justice, 42(6), 568–578. https://doi.org/10.1016/j.jcrimjus.2014.09.011

38.

Schuringa

Spreen

Bogaerts

(2014). Inter-rater and test-retest reliability, internal consistency and factorial structure of the Dutch Instrument for Forensic Treatment Evaluation (IFTE). Journal of Forensic Psychology Practice, 14(2), 127–144. https://doi.org/10.1080/15228932.2014.897536

39.

Shrout

P. E.

Fleiss

J. L.

(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420

40.

Singh

J. P.

(2013). Predictive validity performance indicators in violence risk assessment: A methodological primer. Behavioral Sciences and the Law, 31(1), 8–22. https://doi.org/10.1002/bsl.2052

41.

Singh

J. P.

Desmarais

S. L.

Hurducas

Thomas

S. D. M.

Vitacco

M. J.

De Ruiter

Guy

L. S.

Pham

T. H.

(2014). International perspectives on the practical application of violence risk assessment: A global survey of 44 countries. International Journal of Forensic Mental Health, 13(3), 193–206. https://doi.org/10.1080/14999013.2014.922141

42.

Smid

W. J.

Koch

Van den Berg

J. W.

(2014). Static-99R Scorehandleiding (herziene uitgave 2014) [Dutch translation and adaptation of Static-99R manual]. De Forensische Zorgspecialisten.

43.

Smits

van der Ark

L. A.

Conijn

J. M.

(2018). Measurement versus prediction in the construction of patient-reported outcome questionnaires: Can we have our cake and eat it? Quality of Life Research, 27(7), 1673–1682. https://doi.org/10.1007/s11136-017-1720-4

44.

Troquete

N. A. C.

Brink

R. H. S.

van den Beintema

Mulder

T. W. D. P.

van Schoevers

R. A.

Wiersma

(2015). Predictive validity of the Short-Term Assessment of Risk and Treatability for violent behavior in outpatient forensic psychiatric patients. Psychological Assessment, 27(2), 377–391. https://doi.org/10.1037/a0038270

45.

Van Beek

Doncker

de Ruiter

. (2001). Static-99: Inschatten van het risico van seksueel geweld dadige recidive bij volwassen seksuele delinquenten [Static-99: Assessment of the risk of sexually violent recidivism in adult sex offenders]. Forum Educatief.

46.

Van Horn

Bogaerts

Eisenberg

Van der Put

Dekker

Van den Hanenberg

Bouman

. (2016). Kernset K-factoren voor het ambulante forensische veld: Een multi-method onderzoek naar risico- en beschermende factoren in relatie tot algemene recidive, geweldsrecidive en seksuele recidive [Report core set forensic clinical risk factors for forensic outpatients: A multi-method study of risk and protective factors in relation to general, violent and sexual recidivism]. Kwaliteit Forensische Zorg.

47.

Van Horn

Eisenberg

Bouman

Van den Hanenberg

Bogaerts

. (2016). Handleiding Forensisch Ambulante Risico Evaluatie FARE [Manual Forensic Outpatient Risk Evaluation FORE]. Kwaliteit Forensische Zorg.

48.

Van Horn

Eisenberg

Bouman

Van den Hanenberg

Van der Put

Bogaerts

. (2016). Eindverslag pilot ‘praktische bruikbaarheid Forensisch Ambulante Risico Evaluatie-FARE [Report pilot usability of the Forensic Outpatient Risk Evaluation FORE]. Kwaliteit Forensische Zorg.

49.

Van Horn

Eisenberg

Uzieblo

. (2016). Risicotaxatie in de Nederlandse ambulante forensische geestelijke gezondheidszorg [The importance of risk assessment in the forensic outpatient mental health care]. Tijdschrift Voor Psychiatrie, 58(8), 583–592.

50.

Van Horn

J. E.

Wilpert

Bos

M. G. N.

Eisenberg

Mulder

. (2009). WaagSchaal jeugd: de psychometrische kwaliteit van een gestructureerd klinisch risicotaxatie-instrument voor de ambulante forensische psychiatrie [WaagSchaal youth: Psychometric qualities of a risk assessment instrument for the forensic outpatient psychiatry]. Panopticon, 30(2), 23–34.

51.

Ward

Stewart

(2003). Criminogenic needs and human needs: A theoretical model. Psychology, Crime & Law, 9(2), 125–143. https://doi.org/10.1080/1068316031000116247

52.

Webster

C. D.

Douglas

K. S.

Eaves

Hart

S. D.

(1997). HCR-20: Assessing risk for violence (Version 2). Mental Health, Law, and Policy Institute.

53.

Wertz

Schobel

Schiltz

Rettenberger

(2022). A comparison of the predictive accuracy of structured and unstructured risk assessment methods for the prediction of recidivism in individuals convicted of sexual and violent offense. Psychological Assessment, 35(2), 152–164. https://doi.org/10.1037/pas0001192

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.04 MB