Abstract
The need for domestic violence training has increased with the development of evidence-based risk assessment tools, which must be scored correctly for valid application. Emerging research indicates that training in domestic violence risk assessment can increase scoring accuracy, but despite the increasing popularity of electronic training, it is not yet known whether it can be an effective method of risk assessment training. In the present study, 87 assessors from various professions had training in the Ontario Domestic Assault Risk Assessment either face-to-face or using an electronic training program. The two conditions were equally effective, as measured by performance on a post-training skill acquisition test. Completion rates were 100% for face-to-face and 86% for electronic training, an improvement over a previously evaluated manual-only condition. The estimated per-trainee cost of electronic training was one third that of face-to-face training and expected to decrease. More rigorous evaluations of electronic training for risk assessment are recommended.
Recent years have seen an increase in the development of domestic violence risk assessment instruments, with several now showing evidence of predictive validity in meta-analyses (e.g., Hanson, Helmus, & Bourgon, 2007; Messing & Thaller, 2013). Actuarial instruments, in particular, permit assessors and policy makers to apportion treatment and other resources according to offenders’ risk of recidivism, consistent with the established “risk-needs-responsivity” (RNR) principles of offender intervention (e.g., Andrews, Bonta, & Wormith, 2011; Bonta et al., 2011). Meanwhile, the need for domestic violence training has increased with the development of these instruments, which must be scored correctly for valid application. Studies have shown that face-to-face training in violence risk assessment can increase mental health clinicians’ and criminal justice professionals’ use of risk assessment instruments, improve the quality of information they document, and raise their self-rated competency in using the instrument (e.g., McNiel et al., 2008; Reynolds & Miles, 2009). Limited availability of training, however, can be a barrier to evidence-based practice (Dimeff et al., 2009). Electronic delivery is a popular alternative to face-to-face training (e.g., Bondarouk & Ruël, 2010). It can be an economical option for overcoming organizational and geographical constraints (Bartley & Golek, 2004), and meta-analytic research suggests that it can be more effective than face-to-face training (Means, Toyama, Murphy, Bakia, & Jones, 2010). To date, however, no research has evaluated its costs and benefits to risk assessment training. The present article evaluates an electronic training program for one domestic violence risk assessment, the Ontario Domestic Assault Risk Assessment (ODARA), in comparison with face-to-face training.
The ODARA is an actuarial assessment designed for use by police and other front-line risk assessors (Hilton et al., 2004; Hilton, Harris, & Rice, 2010). Its 13 empirically selected, dichotomous items pertain to the offender’s criminal history, the most recent domestic assault, and the victim’s circumstances. Total scores range from 0 to 13 and are divided into seven categories of increasing risk of domestic violence recidivism. The categories refer to an actuarial table indicating the percentage of men with the same scores who committed a new domestic assault in follow-up studies. Higher scores are associated with greater likelihood of recidivism, with the number and severity of subsequent assaults, and with shorter time until the first new assault. The ODARA’s predictive validity has been demonstrated in a variety of samples selected from the community and custodial institutions (e.g., Hilton, Harris, Popham, & Lang, 2010; Hilton, Harris, & Rice, 2010; Hilton, Popham, Lang, & Harris, 2014; Rettenberger & Eher, 2013) and meta-analyses indicate that it outperforms other risk assessment instruments (Hanson et al., 2007; Messing & Thaller, 2013). To maintain this validity in practice, however, assessors must score the instrument accurately.
Hilton, Harris, Rice, Eke, & Lowe-Wetmore (2007) reported that assessors scored the ODARA more accurately from videos of victim interviews after 1 day of face-to-face training than before the training, and training led to greater scoring accuracy than practice alone. Similarly, after an 8-day face-to-face training course in the structured professional judgment model of risk assessment, criminal justice professionals identified more violence risk factors in written descriptions of domestic violence cases and gave more correct ratings of overall risk (Storey, Gibas, Reeves, & Hart, 2011). These professionals also reported feeling more confident and competent to conduct risk assessments after training, but self-reports were not related to performance, indicating the need to test actual performance. Despite the increasing popularity of electronic training, it is not yet known whether it can provide the same benefits as face-to-face training in domestic violence risk assessment.
The Present Study
We compared the effect of face-to-face and electronic training on trainees’ ability to score the ODARA (Hilton, Harris, & Rice, 2010). Initially, we offered manual-only training as a response to the cost and scheduling challenges of face-to-face delivery, but records from 126 manual-only trainees showed a completion rate of only 68%. In response to the increasing interest in alternative methods of training, we then created an electronic program using the content of face-to-face training, including a skills acquisition test. We conducted a quasi-experimental group comparison design using a standard outcome measure across both conditions.
Method
The study was approved by the authors’ institutional research ethics board. Participants provided written consent for their information and scores to be used for evaluation purposes. Electronic training participants also provided consumer feedback on the program.
Participants
The 87 participants included 45 assessors who completed training through ODARA 101 in a pre-release evaluation phase in 2011. The 42 other participants had attended face-to-face training in 2010-2011 and took the same skill test as that included in the electronic program. Participants included counselors (27), police officers (14), shelter directors and administrators (14), and mental health workers (10), as well as probation officers, nurses, social workers, shelter staff, and court-based personnel (<10 each).
Training
Face-to-face training was led by a male and a female co-author of the ODARA and consisted of approximately 5 hr of instruction in the ODARA’s empirical construction, validation studies, and item scoring criteria (following Hilton et al., 2007). Case videos were used for scoring practice with feedback, and participants had the opportunity to ask questions. Trainees completed the skill test at the training site immediately following this instruction.
“ODARA 101: The Electronic Training Program” was developed with the support of short-term funding, using Adobe Captivate 5.5 software. The content normally provided in face-to-face training was portrayed visually using animated slides and videos, and orally using professional voice-over recordings. Practice case videos were included, with feedback on the correct scoring only being revealed when selected by the learner. There was no frequently asked questions section or opportunity to have individual questions answered. Participants used ODARA 101 on a computer disk, prior to finalization of the program for web-based delivery.
Measures
Skill acquisition was measured by an assessment of ODARA scoring accuracy. Each participant scored the same 10 written, simulated case descriptions that ranged from two to five pages in length. Five cases were presented in the form of police reports and five as victim interview transcripts. The cases were different from the ones reported by Hilton et al. (2007). The inter-rater reliability between each trainee’s scores and the correct scores was measured using a two-way mixed model, single measures, absolute agreement, intra-class correlation (ICC) of ODARA category across all 10 cases, on SPSS 12.0, equivalent to ICC 3,1 (Shrout & Fleiss, 1979; Yaffee, 1998). This model was selected because each individual trainee rated all 10 cases, the individual trainee is the only rater of interest for each test, and the unit of analysis is the individual ODARA score per case rather than the mean of all the scores. Average measures and consistency models, not reported here, increased estimated reliability by a mean of .03 and .01 points, respectively. Completing the test and achieving an ICC of .750 was considered a success.
Electronic training participants gave feedback on the program’s production quality, instructional content, and user friendliness. Each aspect was assessed by three items rated on a five-point scale labeled from 1 (true) to 5 (false; for example, “It looked professional,” “The spoken content was easy to understand,” “It was clear how to make it work”). Open-ended comments were also solicited.
Results
The mean ICC for tests completed at face-to-face training was .917 (SD = 0.64), 95% CI = [0.890, 0.929], and for tests completed via electronic training, .917 (SD = 0.63), 95% CI = [0.896, 0.939]. Thus there was no statistical difference between the training delivery methods. All of the face-to-face participants completed the test and 98% passed on their first attempt, with 86% completion (all passing on the first attempt) for electronic training, and the difference between proportions Z = −2.52, p = .012.
Feedback was provided by 29 electronic training participants. The mean rating for professional quality items was 1.49 (SD = 7.78); for program content, M = 1.24 (SD = 0.57); and for user friendliness, M = 2.16 (SD = 1.01). Participants commented that they valued being able to take the training at their own pace and when convenient; however, some noted that they would have appreciated the “dynamic” of live training or having their questions answered.
Discussion
In this evaluation of domestic assault risk assessment training delivery alternatives, no difference was found in the levels of skill acquisition between participants who completed face-to-face and electronic training. Using a test of risk assessment scoring skill, we found that all electronic training participants who completed the test and all but one face-to-face participant were successful in passing the test. This finding represents the first demonstration that electronic training can be an effective alternative to face-to-face training in domestic violence risk assessment. Feedback on the newly created electronic program was generally positive; however, electronic training participants were less likely than face-to-face training participants to complete the training. This finding indicates that risk assessment training evaluation should be based on intended trainees rather than only completers, and future studies should attend to reasons for non-completion of risk assessment training.
Whether electronic training is a viable alternative to face-to-face training depends not only on effectiveness but also on the relative cost. The costs in Canadian dollars of developing and evaluating ODARA 101 were estimated at $33,000 including project staff time, equipment, and educational and visual media consultations. In comparison, the cost of the last face-to-face training session conducted in 2011 for 60 participants was estimated at $18,300 including room rental, equipment, meals, and participant travel and overnight accommodation. The additional cost of developing training content and producing case videos for scoring practice is constant because these resources were used in both face-to-face and electronic training. The management of the online training portal and processing of tests and certificates now requires approximately 10 min of training staff time per individual, compared with 20 hr of staff time to prepare and teach one face-to-face training session in the past (20 min per individual in a session with 60 participants). In addition, electronic training takes 4 to 6 hr per participant compared with 7 to 8 hr at face-to-face training, reducing the personnel cost of having staff in training. If the average staff salary is $25 per hour, the estimated cost of face-to-face training for a class of 60 participants is $500 per trainee. 1 To date, 881 assessors have been trained through ODARA 101, making the estimated cost of electronic training $168 per trainee 2 and decreasing to a potential minimum cost of just over $130. This cost reduction can be obtained without sacrificing any scoring proficiency among people who complete the training.
Limitations and Future Research
The non-random assignment method means that the apparent success of electronic training in this study cannot necessarily be attributed to the condition itself, as differences between the groups were not controlled for. For example, all face-to-face training participants had the benefit of a full working day set aside for the training and the test, whereas some electronic training participants were not afforded this time and some faced technical glitches with the DVD version of the program. Random assignment to conditions is a critical component of training evaluation and is still needed in the field of domestic assault risk assessment.
We did not measure retention of learning over time. Follow-up research to examine maintenance of risk assessment scoring accuracy could help determine whether re-certification would be valuable for risk assessors. Future evaluations should also examine reasons for failure to complete electronic training, which could inform improvements to electronic delivery. Studies of individual differences between people who choose to obtain risk assessment training through either face-to-face or electronic training, or matching on individual differences or professional sector (e.g., policing, victim support) within a random controlled design, could help investigate whether certain training conditions are more effective for different groups of assessors.
Conclusion
Accurate and reliable scoring of domestic assault risk assessment instruments is essential for valid risk assessment practice, and effective training can improve scoring accuracy. The present study is the first in the field of domestic violence risk assessment to compare the effectiveness of electronic and face-to-face training delivery. We demonstrated that a cost-effective training program can be created, despite the challenges typical of short-term funding for domestic violence education. We obtained preliminary but promising evidence that electronic training provided the same benefit as face-to-face training in terms of success on an objective measure of skill acquisition, at a lower cost. More rigorous evaluation methods and further investigation of training non-completion are warranted.
Footnotes
Acknowledgements
We would like to thank the ODARA 101 project partners, the agencies and individuals taking part in this evaluation, and Trent Maracle and Sonja Dey for their assistance. Jill Atkinson, Grant Harris, Marnie Rice, and Chelsea Turan provided helpful comments on earlier versions of this article.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The electronic training program project was made possible by a grant from the Ontario Ministry of Community Safety and Correctional Services.