Evaluating the Effectiveness of Online Training for a Comprehensive Violence Against Women Program: A Pilot Study

Abstract

Evaluating violence against women (VAW) training is essential to moving the field forward with proven approaches that can improve service provision for survivors of violence. Given existing resource constraints involved in VAW work, online training represents an economical and flexible option; however, existing evaluations of online programs in the VAW field are scant and face a variety of limitations. This study aimed to fill this gap by using a pre-/posttest design, comparison group, and mixed-method analysis to assess the effectiveness and value of an online training program. The program was intended to provide foundational knowledge in feminist antiviolence principles and values to a range of individuals working with survivors of intimate partner violence (IPV). Program participants (N = 108) included volunteers, students, and professionals from various sectors, allowing for the application of the results to a broader field of VAW support services. This is important as individuals who work with IPV survivors may do so in a range of settings outside of the shelter context. Results demonstrate the potential for online VAW training to improve participants’ knowledge of and attitudes about VAW, which can positively inform their work with survivors. Qualitative responses provide further insight into course impact and highlight positive and negative aspects of the course. Although preliminary, these results provide justification for continued development and evaluation of online VAW training programs.

Keywords

online training violence against women evaluation

Violence against women (VAW) has been designated an “urgent public health priority” by the World Health Organization (WHO; Garcia-Moreno & Watts, 2011, p. 1) due to its cumulative impacts on morbidity and mortality as well as significant social and economic costs (Varcoe et al., 2011; Wathen, 2012; Zhang, Hoddenbagh, McDonald, & Scrim, 2012). VAW is often perpetrated by an intimate partner, which is commonly referred to as intimate partner violence (IPV), with estimates that this type of violence affects 30% of women worldwide (WHO, 2013). With increasing recognition of the magnitude of VAW, governments worldwide are investing in primary-, secondary-, and tertiary-level prevention. This includes not only interventions to prevent violence but also action for improving services for women who experience it (Garcia-Moreno & Watts, 2011).

One way to improve services for woman-identified survivors of IPV is to develop comprehensive training programs for individuals who work with them (e.g., shelter workers, counselors, health and social service professionals, students, and volunteers). Although the development of training programs is growing, organizations often have little additional funding available beyond what is needed to develop and/or implement the training. As a result, many initiatives are not formally evaluated (Broll, Crooks, Baker, & Straatman, 2012). Furthermore, prospective or current students, staff or volunteers participating in training run by organizations providing services to IPV survivors and their prospective or current students, staff, or volunteers can face challenges in accessing and completing training, such as time constraints, geographical location, competing work/family demands, and schedule conflicts. Due to these challenges, an appealing training option might be to move programs online. Online training is an economical alternative to traditional face-to-face formats and has become increasingly popular in many industries (Long, DuBois, & Faley, 2008). In addition to its cost effectiveness, online programs are easy to update, can be offered more frequently than face-to-face training sessions, are more accessible regardless of location, can accommodate larger numbers of participants, are convenient and flexible, are learner centered/directed, and support long-term sustainability (Etherington & Baker, 2016).

There are, however, limited existing evaluations of VAW training programs, especially those delivered online. Available research has demonstrated that online VAW training has the capacity to improve risk assessment skills for a variety of professionals (Hilton & Ham, 2015), physicians’ abilities to manage patients experiencing violence (Harris, Kutob, Surprenant, Maiuro, & Delate, 2002; Short, Surprenant, & Harris, 2006), and nurses’ knowledge of sexual assault forensic examinations (Patterson & Resko, 2015). However, these studies primarily rely on participants’ subjective ratings of their experiences rather than objective evaluation measures and lack experimental design elements such as random assignment of participants and use of a comparison group. Programs that have been evaluated also tend to focus on specific issues (e.g., hospital screenings, risk assessment) instead of foundational knowledge grounded in feminist antioppression and intersectional principles and values. This aspect of training is of value to all individuals involved in VAW work as it provides a shared foundation comprised of principles that have been at the center of the feminist antiviolence movement since its inception (e.g., intersectional/antiracism/antioppression framework; Bograd, 1999; Crenshaw, 1989; Springtide Resources, 2008). These principles provide a conceptual framework that can be applied in different settings and to different types of service provision. Although profession-/student-/volunteer-specific training in procedures and approaches to specific aspects of work is necessary, it is important that everyone who works with VAW survivors has the foundational tools to assess individual situations, needs, and options when providing support services to survivors.

Although face-to-face VAW training programs have generally been found to be effective (e.g., Devine, Spencer, Eldridge, Norman, & Feder, 2012; Ramsay et al., 2012; Schoening, Greenwood, McNichols, Heermann, & Agrawal, 2004; Thompson et al., 2000; Zaher, Keogh, & Ratnapalan, 2014), it is unknown whether online training can enhance the knowledge and skills of VAW workers. Working with survivors requires an ability to work with diverse women, use culturally appropriate language and service approaches, apply strength-based models that are unique to individuals, and adapt service provision to survivors’ immediate needs and long-term goals (Phillips, Lyon, Fabri, & Warshaw, 2015). These skills may be difficult to impart without in-person opportunities for practical experience and individualized feedback. Certainly, face-to-face training has advantages over online formats in its ability to provide personal interaction, immediate feedback and answers to questions, and “live” emotional support (Etherington & Baker, 2016). It has also been found to be suitable for introducing new, unfamiliar, or sensitive topics and provide an opportunity to practice skills learning (e.g., through role-playing; Etherington & Baker, 2016). Still, it can be hypothesized that VAW training might benefit from online formats given emerging studies demonstrating the effectiveness of online training for professionals working in other areas of related violence, such as child abuse and child exposure to IPV (e.g., Heck, Saunders, & Smith, 2015; Sargent, McDonald, Vu, & Jouriles, 2016).

The Present Study

This pilot study examines the efficacy of an online VAW training course using a comparison group design to determine improvements in knowledge, attitudes, and application of learning. Pre- and posttest questionnaires were developed based on identified core competencies for VAW training initiatives (Broll et al., 2012) and broader evaluation metrics designed to assess actual and perceived learning (Kirkpatrick, 1996; Thackwray, 1997). This study, therefore, contributes to the VAW and training literature through evaluating an online training program with measurements based on core competencies representing key aspects of VAW work as identified by VAW stakeholders and those who develop and deliver VAW training (Centre for Research & Education on Violence Against Women & Children [CREVAWC], 2011).

Evaluated Training Program: Ontario Association of Interval & Transition Houses (OAITH) Foundations of VAW Online Training Course

This study evaluated an online training course developed by OAITH, a recognized provincial coalition in Canada founded by women’s shelter advocates in 1977. Membership includes first stage emergency shelters for abused women and their children, second-stage housing programs, and community-based women’s service organizations. OAITH works with member agencies to educate and promote change in all areas that abused women and their children identify as important to their freedom from violence. OAITH developed and piloted an online training course for individuals working or planning to work with survivors of IPV, including students, volunteers, and professionals. The course, titled “Foundations of VAW Online Training Course” (Foundations) first ran in 2013, with an updated version released in 2015. This study uses the most recent version of the course.

Description of Course

The Foundations of VAW Online Training Course is a 4-week course, covering a wide range of topics, which are important for building a strong VAW practice based on feminist antioppression and intersectional principles and values. Course topics include intersectional feminism and VAW, portfolio development, ethical decision making, record-keeping, self-reflective practice, feminist analysis of mental health issues, harm reduction, and risk assessment and safety planning (see Appendix A for a detailed description of course content). Strategies for applying feminist principles to working with IPV survivors are included with each topic area. The course supports the development of a portfolio for VAW counselors and advocates, and incorporates moderated discussions with peers. Course objectives are listed below:

Support frontline staff to develop a professional portfolio that can be used in professional practice settings.

Enhance participants’ knowledge of feminist intersectional theory and practice.

Develop a feminist intersectional analysis of documentation, risk assessment, harm reduction, and mental health to inform participants’ practice.

Encourage participants to engage in self-reflective practice.

Facilitate networking and relationship building among learners.

Method

Recruitment

The 2016 Foundations course was promoted via email to all shelters in Ontario as well as partner organizations (e.g., Ontario Coalition of Rape Crisis Centres, Aboriginal Shelters of Ontario). Materials promoting the course were also made available at provincial events in the fall of 2015. The course was promoted to student and adult learner populations using social media (Twitter and Facebook) and the OAITH website. In addition, individuals who had signed up for a waitlist on the OAITH website since March 2014 were contacted once registration for the course opened.

To participate in the course, individuals had to create an account in OAITH’s online learning portal and self-enroll in the course. Enrollment for both sessions (January and February) opened at the same time, giving individuals the option as to which course session they would enroll in. A waitlist was also generated for the February offering of the course, serving as a comparison group for the January participants.

Participants

A total of 187 participants registered for the online training program across the January (N = 85) and February (N = 102) offerings of the course. The January and February cycles will be referred to as Course 1 and Course 2, respectively. There were 79 participants in total who did not complete the course (35 in January, 44 in February). These individuals were excluded from the evaluation analyses, resulting in a final sample size of 108 participants (50 in Course 1, 58 in Course 2). Importantly, participants who dropped out of the course did not significantly differ from those participants who remained enrolled on any of the demographic characteristics assessed. The completion rate (58%) is consistent with and generally higher than average rates for external, nonmandatory online training and evaluation surveys (Archer, 2008; Long, Dubois, & Faley, 2009). In fact, completion rates for voluntary online training courses tend to hover around 20% (Long et al., 2009). Participants for both courses did not significantly differ on demographic characteristics assessed (Table 1). Most participants were 25 to 34 years of age (52%), had 3 or less years of experience in the VAW sector (60%), had a postsecondary degree (66%), and had previous VAW training or learning experiences (90%). However, a range of age groups, education levels, and experience levels were present among participants. For example, age groups ranged from 17 to 24 years (18%) to 55 to 64 years (2%). Experience levels ranged from school placements/practicums or volunteer experience only (12%) to more than 10 years of experience (12%). On average, participants spent 2.5 hr per week on the course.

Table 1.

Participant Demographic Characteristics.

r	Course 1 (n = 50; %)	Course 2 (n = 58; %)
Age group
17-24	18	14
25-34	52	38
35-44	14	14
45-54	14	22
55-64	2	12
Time worked in VAW sector
<1 year	30	24
1-3 years	30	28
4-6 years	12	14
7-10 years	4	7
>10 years	12	10
School placements/practicums or volunteer experience only	12	17
Highest education level
Secondary school	0	0
Some postsecondary (college/university)	16	14
Postsecondary (college/university) degree	66	64
Graduate degree	18	22
Previous VAW training/learning experience
Yes	90	84
No	10	16

Note. VAW = violence against women.

Measures

Core competencies

Pre- and posttest survey questions were developed based on a series of core competencies for domestic violence training programs (see Broll et al., 2012). These competencies cover such aspects of VAW knowledge and skills as the characteristics and impacts of violence (e.g., prevalence, dynamics of abusive relationships, trauma), intervention responses (e.g., disclosure, risk management), reflective practice (e.g., self-care, personal values, compassion fatigue), professional roles in the workplace (e.g., confidentiality, safety), and collaboration within and between organizations (e.g., making referrals). The competencies were developed based on a comprehensive review of the literature on VAW training initiatives and the role of evaluations in VAW training programs, as well as a knowledge exchange workshop hosted by the CREVAWC in 2011. Core competencies are an important addition to VAW training evaluations in that they extend evaluation beyond participants’ reactions or perceived learning levels (Broll et al., 2012) and can be used to understand the impact of training programs on the real-world experiences of participants (Kahan, 2008; United Nations Development Fund for Women [UNIFEM], 2010). These competencies were used to inform the survey measures detailed below (see Appendix B for a detailed list of competencies and corresponding knowledge and skills).

Survey measures¹

Participant responses to survey questions were scored so that higher scores indicate (a) greater knowledge, (b) more favorable attitudes, (c) more successful application, and (d) higher self-ratings of comfort, knowledge, and abilities. Knowledge measures included 15 multiple-choice and true/false questions where participants had to correctly identify the answer on such topics as oppression, intersectional analysis, self-reflective practice, documentation, survivors with mental health conditions, risk, and harm reduction. Correct responses were assigned a value of 1 and summed (maximum score = 15). Attitude measures involved participants indicating their level of agreement with 15 statements such as “Women who stay in abusive relationships do not consider the safety of their children” and “Many survivors of violence use drugs for a variety of reasons.” Level of agreement was rated on a scale of 0 (strongly disagree) to 4 (strongly agree). Individual items were recoded for analysis so that “4” always indicated more favorable attitudes. Participants could, therefore, receive a maximum score of 60. Application questions involved eight scenarios of violence where participants had to indicate whether the situation involved IPV or not. Correct answers were assigned a value of 1, for a total possible score of 8.

In Course 1, participants were asked to subjectively rate their comfort, abilities, and knowledge prior to and after completing the course. Example questions include rating “comfort level in providing services to/working with clients who have experienced violence,” “ability to create a safe environment for disclosing abuse,” “ability to apply a feminist intersectional approach,” and “knowledge of resource materials that can be provided to women who disclose abuse.” There were 13 questions in total, rated on a scale of 0 (very low) to 4 (very high). Responses were summed for a total possible score of 52.

In Course 2, an additional posttest question was asked where participants rated how much they felt their comfort, abilities, and knowledge was enhanced as a result of the training course. This question involved a scale ranging from –3 (greatly diminished) to +3 (greatly enhanced). The aim of this question was to capture participants’ perceptions as to whether they perceived the course to have improved their comfort, abilities, and knowledge. Responses were recoded as 0 (diminished, –1 to –3), 1 (no change, 0), and 2 (enhanced, +1 to +3) and are reported descriptively only as this question was asked after training was completed.

General evaluation metrics

Additional survey questions were designed to obtain more information on broader evaluation metrics: actual (i.e., reaction level) and perceived (i.e., learning level) knowledge gained, attitude change, and skills acquired, based on a widely accepted evaluation approach (Kirkpatrick, 1996; Thackwray, 1997). This approach has also been found to be useful in identifying information for developing, evaluating, and revising training programs (Gramckow, Nady Sigmon, Gaboury, & Burt, 1997). In fact, Kirkpatrick’s (1996) evaluation approach is widely accepted and credited with “revolutionizing the evaluation of training programs” (Broll et al., 2012; Thackwray, 1997, p. 8). These questions asked participants to indicate their level of agreement with a series of statements on the training course (e.g., applicability of topics, whether training would make a difference, whether skills taught would be useful) and its impact on their skills to work with VAW survivors, their knowledge on VAW, and their attitudes toward IPV.

Qualitative Component

In addition to the quantitative evaluation, participants were asked to provide qualitative responses on what they felt was the most helpful aspect of the training, how their practice would change as a result, and what changes could be made to improve the course overall. These responses were considered complementary to quantitative results and helped to provide useful information to the training organization for future course development. In addition, it is important to understand whether online training is appealing and valued by individuals in the VAW sector beyond its ability to increase knowledge, given limited research in this area. Qualitative measures added an additional layer of information about participants’ perceptions of the usefulness of the course.

Analytic Strategy

Paired t tests were conducted to determine whether pretest mean scores differed significantly from those at posttest for the entire sample (N = 108). ANCOVA and ordinary least squares (OLS) regression models were used to account for participant demographic characteristics (age, time worked in VAW sector, education, previous training). All quantitative analyses were conducted using STATA 14. Qualitative analyses were conducted using NVivo 11. Responses were coded line by line to reveal larger themes, which were then verified by a second coder.

Waitlist comparison group

In this analysis, the waitlist group which had not yet completed the course (Course 2, N = 58) is treated as the comparison group and the group that finished the course (Course 1, N = 50) is treated as the intervention group. The posttest scores of the intervention group were compared with the pretest scores of the waitlisted group to determine whether apparent increases could be attributed to the online training course using two-sample t tests. Participants in Course 1 also did not significantly differ from participants in Course 2 on any measured characteristic or pretest score (e.g., knowledge, attitudes, application), meaning the two groups are comparable. The exception was for subjective rating of comfort, abilities, and knowledge. The comparison group had a higher pretest rating on this subjective measure than the intervention group. Therefore, the quasi-experimental design could not be used to further assess changes in this measure beyond those that occurred within each group. It was, however, used for the objective knowledge and attitude measures.

Results: Assessment of Learning

Full Sample Pre-/Posttests

Participants demonstrated an improvement in knowledge and attitudes, with all changes significant at p < .001 (Table 2). From pre- to posttest, the mean knowledge score improved from 11.16 to 13.76, and the mean attitudes score improved from 50.01 to 52.01. There was no significant change in application scores prior to (6.48) and after (6.71) completing the course.

Table 2.

Knowledge and Attitudes Test Scores: Full Sample and Comparison Group Condition.

	Knowledge		Attitudes
	Mean Score (SD)	95% CI	Mean Score (SD)	95% CI
Full sample (n = 108)
Pretest	11.16 (1.78)	[10.81, 11.50]	50.01 (4.44)	[49.13, 50.88]
Posttest	13.76*** (1.59)	[13.45, 14.07]	52.01*** (4.51)	[51.12, 52.89]
Comparison group condition
Comparison group (n = 58)	11.22 (1.54)	[10.81, 11.63]	49.52 (4.27)	[48.39, 50.64]
Intervention group (n = 50)	13.7*** (1.60)	[13.24, 14.15]	52.5*** (4.64)	[51.17, 53.82]

Note. CI = confidence interval. ***p < 0.001.

Comparison Group Condition

There was a significant improvement in scores between the comparison group and intervention group on the measures of knowledge and attitudes (p < .001, Table 2). The mean knowledge score was 11.22 in the comparison group and 13.7 in the intervention group. The mean attitudes score was 49.52 in the comparison group and 52.5 in the intervention group. Once again, application scores were not meaningful.

Subjective Rating of Comfort, Abilities, and Knowledge (Course 1 Only; N = 50)

A significant improvement in participants’ ratings of their comfort, abilities, and knowledge in Course 1 was observed from pre- to posttest (p < .001, Table 3). The mean score increased from 34.40 prior to the training to 40.14 after training completion.

Table 3.

Course 1 Pre-/Posttest Subjective Ratings: Comfort, Abilities, Knowledge.

	Comfort, Abilities, Knowledge
	M Score (SD)	95% CI
Pretest	34.40 (7.78)	[32.18, 26.61]
Posttest	40.14*** (5.43)	[38.59, 41.68]

Note. CI = confidence interval. ***p < 0.001.

Subjective Rating of Enhancement (Course 2 Only; N = 58)

Participants’ reports of the degree of enhancement in their comfort, abilities, and knowledge in VAW work demonstrate a high level of perceived improvement (Table 4). For most areas (e.g., overall knowledge, ability to support harm reduction, understanding of impact of violence on health), more than 90% of participants reported enhancement. More than 80% of participants felt the training course enhanced their abilities to create a safe disclosure environment, to keep records and documentation in a manner consistent with feminist antioppression principles, and to identify indicators of violence.

Table 4.

Enhancement in Comfort, Abilities, and Knowledge as a Result of Training (N = 58).

Item	% Reporting Enhanced
Comfort level in providing services to/working with clients who have experienced violence	91
Ability to apply a feminist intersectional approach in practice	97
Ability to identify indicators of violence	88
Comfort level in making effective referrals for clients who have experienced violence	90
Ability to keep records and documentation in a manner consistent with feminist antioppression work	86
Understanding of the ways in which violence affects women’s health	93
Ability to engage in and/or support harm reduction efforts for clients who use drugs	95
Ability to create a safe environment for disclosing abuse	81
Knowledge of resource materials that can be provided to women who disclose abuse	90
Ability to confidently carry out a risk assessment	91
Understanding of the professional’s role in the VAW sector	91
Understanding of what it means to be self-reflective in practice	91
Overall knowledge of VAW	97

Note. VAW = violence against women.

General Evaluation Metrics: Reaction, Learning, and Performance (Course 2 Only; N = 58)

Participants report perceived changes in skills, attitudes, and knowledge because of participating in the training course (Table 5). Most participants (>90%) reported changes in these areas. Notably, 100% of participants reported that the training would make a difference in the way they would work with survivors of violence.

Table 5.

Changes in Skills, Attitudes, Knowledge as a Result of Training and Training Effectiveness (N = 58).

Item	% Reporting “Agree”
Skills in working with women who have experienced violence were enhanced as a result of this training.	93
Attitudes on woman abuse were reinforced or changed in ways that support woman-centered practice as a result of this training.	90
Knowledge on VAW increased as a result of this training.	93
Skills learned as a result of this training useful in daily work.	97
The topics covered in the training applicable to practice/service setting.	95
Better equipped to support clients who have experienced violence as a result of this training.	93
This training will make a difference in the way job is done.	100

Note. VAW = violence against women.

Results: Reflections on the Course and Subjective Value

Participants’ responses to the question “What was particularly helpful about the training?” revealed eight main themes, listed in order of significance: harm reduction (28%), discussion forums (23%), general content (14%), relevance (12%), structure of course (9%), risk and safety planning (6%), format (6%), and resources provided (2%). Quotes representative of the range of comments provided by participants are included below, with the first quote reflecting the general content theme and the second quote representing the discussion forums theme.

I found that the training talked a little bit about everything. It was very holistic in terms of the VAW sector and VAW workers.

I really found the forums beneficial. The posts are reflective and insightful. They made me think about my perspective and challenged me, which is the whole point.

Participants were also asked what they would recommend changing about the training. Approximately one third of participants (31%) indicated that no changes to the training were necessary, whereas 23% recommended including more practical application examples, such as safety planning with diverse clients and applying feminist theory to practice. See Table 6 for more suggested changes and examples.

Table 6.

Recommended Changes to Training Course.

Suggested Change	% Participants	Example(s)
No changes	31%	NA
More practical application examples	23%	• Case studies • Safety planning with diverse clients • Examples of best practices • Practical application of feminist theory • Harm reduction tools
More videos/visuals	15%	• Interactive readings, videos, visuals, or diagrams for all units
Resolve technical issues	13%	• Audio difficult to hear in some modules • Some modules difficult to maneuver • Typos
Improve course design	12%	• Provide all information at once • Allow to complete at own pace • Incorporate audio for all modules—accessibility issues for some (e.g., English not always second language; have narrator speak slowly) • Consistent length among unit • More collaborative learning • Make discussion/writing nonmandatory
More time to complete	6%	• Longer time frame for completion as some units “quite heavy”

With regard to additional training that would be helpful, a large proportion of participants (46%) indicated that additional training on risk and safety planning skills would be helpful, followed by counseling skills (18%) and an intersectionality focus (11%). Examples of additional risk and safety planning skills mentioned by participants included training on more assessment tools and handling high-risk situations. Counseling skills that participants named for addition to the course included trauma-focused training, disclosure training, and feminist counseling skills. Other identified areas for course enhancement included addiction and mental health, compassion fatigue/vicarious trauma, advocacy, and intervention with abusive partners.

Last, participants were asked how their practice would change as a result of the training. Many participants indicated that their practice would now reflect their increased understanding of harm reduction and enhanced ability to respond (25%). This change in understanding and approach is best reflected by one participant, who commented,

My argument against keeping women who are actively using drugs in the shelter has been completely diminished as a result of this training.

Participants also identified an enhanced capacity to support women (19%), such as using an intersectionality framework, being more comfortable and/or confident in working with women, and possessing greater knowledge of issues relevant to VAW work. Other themes included enhanced risk and safety planning skills, improved documentation practices, engaging in self-care, training development, improved collaboration, and the ability to better identify the continuum of violence. Changing use of language was another theme, with one participant commenting,

I will continue to work with my agency to change the language we use and to seek the opinions of the women that we work with and support as they are the most valuable resource that we have.

Discussion

The results of this pilot study show that online training programs may be a promising format for VAW training. Foremost, the course appealed to participants of various educational, age, and VAW experience backgrounds, and learning was demonstrated despite this diversity. This is critical as VAW work, including work with IPV survivors, occurs in a range of settings. Participants demonstrated increased knowledge both objectively through pre-/posttest questions and subjectively through self-reported learning. Demonstrated learning was also achieved in line with the core competencies for VAW training (Broll et al., 2012), given participants’ increased understanding of such items as the characteristics and consequences of violence, mental health and addiction, risk management, and documentation practices. Thus, online training appears capable of increasing knowledge on a wide range of relevant and complex topics.

Participants’ perceived gains and objective measures of increased knowledge were also attained through demonstrated attitude improvement (i.e., attitudes more reflective of feminist antioppression and intersectional principles, assessed by level of agreement with a series of statements on the nuances of VAW work) and perceived attitude change. Attitudinal scores reflecting an understanding of the dynamics of VAW and the broader societal context in which it occurs increased following completion of the training course. Both quantitatively and qualitatively, participants reported increased confidence in skills and their understanding of violence and its impacts. These outcomes are again reflective of the core competencies, and provide further support for the capacity of online training formats to adequately convey many of the complexities involved in VAW work.

The significant improvement in knowledge and attitudes scores in the intervention group (i.e., group who had completed the training) compared with the waitlist comparison group (i.e., group that had not yet completed the training) provides further support to the above findings. The fact that the observed improvement in knowledge and attitudes among the intervention group is similar to that which was observed in the overall pre-/posttest analyses strengthens conclusions of training effectiveness and evaluation measures. In addition, the training course appears to be of importance given that most (90%) participants indicated that they had previous VAW training and still demonstrated learning. The training may, therefore, be a valuable complement to existing initiatives available elsewhere. The objective assessment of learning and comparison group condition utilized in this study provide more rigorous support for online VAW training, overcoming some of the limitations of previous evaluations.

Significantly, 100% of participants reported that this training will make a difference in the way they do their jobs. Anticipated changes to practice as a result of the training included a greater understanding of and enhanced approach to harm reduction, an enhanced capacity to support women, improvement in risk and safety planning abilities, improved documentation practices, more informed use of language, engagement in self-care, promoting training development, improved collaboration among coworkers and between organizations, and the ability to recognize violence as a continuum. Participants also noted that they “enjoyed the material” and “learned quite a bit.” At the subjective level, then, participants appear to value what they learned in the online format.

Limitations and Future Research

This study is not without its limitations. Specifically, the time and resource constraints of the training organization inhibited the use of a randomized control group, which would have concretely determined whether results were truly the result of the training course or a result of external factors (e.g., other educational opportunities, increased work experience). Nevertheless, a quasi-experimental design revealed a change in scores. This evaluation included a waitlist comparison group that allowed for the results of the Foundations intervention group to be compared with the results of the group that was waiting to take the course the following month (Course 2). The use of a waitlist or “intent to train” comparison group has been utilized in other studies (e.g., Heck et al., 2015) and is an improvement over existing evaluations with no comparison or control group at all. Still, an experimental design that randomly assigns participants to intervention and control groups is recommended for future evaluations to determine with greater confidence that positive gains are the result of the Foundations course rather than confounding factors. The findings of the present study suggest only that the course appears to promote enhanced capacity for VAW practice, but randomly assigned groups would solidify this observation and enhance the generalizability of the findings. The use of a randomized control group, in addition to a follow-up survey of participants several months after the course was completed, would speak to both desired gains in, and retention of, knowledge. It is currently unknown how the training will apply to job performance, and if it translates to the workplace, how long these effects will last. It is, therefore, important for future research in this area to work with VAW organizations to overcome barriers to enhanced training evaluation.

Related to the observed findings, participant application scores did not improve. This may have occurred for two reasons. First, additional application questions may need to be tested and refined, which was beyond the scope of this preliminary study. Second, online training may not be the most suitable format for teaching or testing application of knowledge and skills learned. As research on learning modalities suggests, many skills of relevance to VAW work (e.g., counseling, responding to disclosures) may be best learned in a setting where they can be practiced through such activities as group skill-building exercises, motivational interviewing, and role-playing (Bennett-Levy & Perry, 2009; Danis, 2016; Rasmussen, Hughes, & Murray, 2008). Future research should investigate question design for testing application of VAW knowledge and skills in an online format and should compare online with face-to-face application results.

Challenges also exist related to sample size and retention of participants. Although sensitivity analyses reveal that results were not biased due to attrition and sample size provided adequate power for analyses conducted, an increased sample size with greater retention would strengthen the reliability, generalizability, and validity of results. Nevertheless, this study represents an important step forward in the evaluation of VAW training in an online format. Future work should examine recruitment and retention strategies relevant to the VAW field. This may be challenging given many participants who did not complete the course indicated time constraints as the primary factor influencing their decision, and tended to drop out at Week 2. In addition, retention of knowledge could not be assessed in this study, and it would be worthwhile for future courses to design a follow-up component.

Conclusion

Evaluating VAW training is essential to moving the field forward with knowledge on proven approaches that can guide future work. Given existing organizational limitations within the VAW sector, online training represents an economical and flexible method. Although face-to-face training may be necessary for specific skills (e.g., counseling), online trainings are suitable for the wide distribution of training to people who may encounter VAW in their work and to provide key baseline training for those entering more survivor-intense occupations (e.g., shelter workers). Existing evaluations of online VAW training, however, are scant and face a variety of limitations. This study aimed to fill this gap through utilizing a pre-/posttest design, comparison group, and mixed-method analysis to assess the effectiveness and value of an online VAW program. Results demonstrate the potential for online VAW training to improve participants’ knowledge and attitudes regarding violence, which can positively inform their practice. It appears that this type of training fills a need for VAW knowledge and skills that participants had not obtained elsewhere. Although preliminary, these results provide justification for continued work on developing and evaluating online VAW programs.

Footnotes

Appendix A

Appendix B

Core Competencies for Domestic Violence Training Programs.

Core Competency	Knowledge and Skills
Recognition	1. What is violence? (a) Characteristics (b) Prevalence (gender analysis) (c) Dynamics of abusive relationships: What is not abuse? (what is a healthy relationship? What does conflict look like in a healthy relationship?) (d) Understanding violence from a broader context (Anti-racist/Anti-oppression—human rights—decolonization framework—social determinants model)
Recognition	2. What are the impacts? (a) On the woman experiencing or having experienced violence? (b) On her children and family (c) Health affects (d) Other relationships (e) On vulnerable populations (f) Understanding trauma (g) Intersectional impacts of mental health, addictions, criminalization (h) Systems that the woman has to interface with
Response	3. Interventions (a) Having the conversation so trust is built (b) Creating safe environments for disclosure (c) Disclosure response (d) What else is going on in her life that complicates intervention? (e) Risk management/threat assessment (f) Safety plans (g) Unintended consequences (h) Supports available (i) Resource materials to provide
Risk reduction: Reporting	4. Professional role and practice in the workplace (a) Workplace policy (b) Roles/responses and mandate (c) Workplace program (d) Documentation, confidentiality, and information sharing (e) Worker’s safety (f) Understanding strengths and limitations of the professional frameworks within one
Risk reduction: Refer	5. Professional role and practice outside the workplace (a) Roles/responses and mandate (b) Understanding strengths and limitations of the professional frameworks (c) Making effective referrals
Risk reduction: Refer	6. Inter-/intraprofessional collaboration (a) Developing an integrated approach to intervention (b) Confidentiality (information sharing) (c) Mapping the network (d) Strengthening professional relationships and networks
Personal and professional development	7. Self-reflection (reflective practices) (a) Personal values, attitudes, and beliefs specific to the professional/worker/learner (b) Engagement with self-care (c) Compassion fatigue (d) Commitment to continuous learning

Source. Brol, Crooks, Baker, and Straatman (2012).

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: To avoid bias, the authors associated with OAITH were not involved in data analysis, but contributed to other aspects of the manuscript.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Author Biographies

Cole Etherington is a clinical research associate at The Ottawa Hospital Research Institute (Ottawa, Ontario, Canada) and a research consultant with the Centre for Research and Education on Violence Against Women and Children at Western University (London, Ontario, Canada). He is an applied sociologist committed to improving population health through bringing a social lens to health research.

Linda Baker is the learning director at the Centre for Research and Education on Violence Against Women and Children at Western University in London, Ontario, Canada. She is a clinical psychologist and currently leads provincial and national knowledge mobilization initiatives on violence against women and children, including trauma-informed health promotion for survivors of child maltreatment and intimate partner violence.

Marlene Ham, BSW, is the provincial coordinator with the Ontario Association of Interval and Transition Houses. She is experienced in using community-placed approaches to support and drive change. She has spent the last 15 years working in violence against women and social justice oganizations in the areas of support, social transformation, training and education.

Denise Glasbeek, MLIS, has more than a decade of experience working in grassroots and community organizations as a librarian and information professional, including providing services in education and online information delivery.

References

Archer

(2008). Response rates to expect from web-based surveys and what to do about it. Journal of Extension, 46(3), 1.

Bennett-Levy

Perry

(2009). The promise of online cognitive behavioural therapy training for rural and remote mental health professionals. Australasian Psychiatry, 17(Suppl. 1), S121-S124.

Bograd

(1999). Strengthening domestic violence theories: Intersections of race, class, sexual orientation, and gender. Journal of Marital & Family Therapy, 25, 275-289. doi:10.1111/j.1752-0606.1999.tb00248.x

Broll

Crooks

Baker

Straatman

A.-L.

(2012). Evaluating violence against women training initiatives: A primer. London, Ontario, Canada: Centre for Research & Education on Violence Against Women & Children.

Centre for Research & Education on Violence Against Women & Children. (2011). Knowledge exchange workshop on domestic violence training, June 16-17, 2011: Report of the meeting. London, Ontario, Canada: Author.

Crenshaw

(1989). Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. The University of Chicago Legal Forum, 140, 139-168.

Danis

F. S.

(2016). Teaching domestic violence online: A step forward or a step backward? Violence Against Women, 22, 1476-1483.

Devine

Spencer

Eldridge

Norman

Feder

(2012). Cost-effectiveness of Identification and Referral to Improve Safety (IRIS), a domestic violence training and support programme for primary care: A modelling study based on a randomised controlled trial. BMJ Open, 2, e001008-e001008.

Etherington

Baker

(2016). Gender-based violence training programs: Online, face-to-face and blended formats. London, Ontario, Canada: Centre for Research & Education on Violence Against Women & Children.

10.

Garcia-Moreno

Watts

(2011). Violence against women: An urgent public health priority. Bulletin of the World Health Organization, 89, 2.

11.

Gramckow

H. P.

Nady Sigmon

Gaboury

M. T.

Burt

M. R.

(1997). Measuring the impact of training. In Burt

M. R.

Harrell

A. V.

Newmark

L. C.

Aron

L. Y.

Jacobs

L. K.

(Eds.), Evaluation guidebook for projects funded by S.T.O.P. formula grants under the Violence Against Women Act (pp. 201-220). Washington, DC: Urban Institute.

12.

Harris

J. M.

Kutob

R. M.

Surprenant

Z. J.

Maiuro

R. D.

Delate

T. A.

(2002). Can internet-based education improve physician confidence in dealing with domestic violence? Family Medicine, 34, 287-292.

13.

Heck

N. C.

Saunders

B. E.

Smith

D. W.

(2015). Web-based training for an evidence-supported treatment: Training completion and knowledge acquisition in a global sample of learners. Child Maltreatment, 20, 183-192.

14.

Hilton

N. Z.

Ham

(2015). Cost-effectiveness of electronic training in domestic violence risk assessment: ODARA 101. Journal of Interpersonal Violence, 30, 1065-1073.

15.

Kahan

(2008). Review of evaluation frameworks. Regina, SK: Saskatchewan Ministry of Education.

16.

Kirkpatrick

D. L.

(1996). Evaluating training programs: The four levels. San Francisco, CA: Berrett-Koehler Publishers.

17.

Long

L. K.

DuBois

C. Z.

Faley

R. H.

(2008). Online training: The value of capturing trainee reactions. Journal of Workplace Learning, 20, 21-37.

18.

Long

L. K.

DuBois

C. Z.

Faley

R. H.

(2009). A case study analysis of factors that influence attrition rates in voluntary online training programs. International Journal on E-learning, 8, 347-359.

19.

Patterson

Resko

S. M.

(2015). Is online learning a viable training option for teaching sexual assault forensic examiners? Journal of Forensic Nursing, 11, 181-189.

20.

Phillips

Lyon

Fabri

Warshaw

(2015). Promising practices and model programs: Trauma-informed approaches to working with survivors of domestic and sexual violence and other trauma. Retrieved from http://www.nationalcenterdvtraumamh.org/wp-content/uploads/2016/01/NCDVTMH_PromisingPracticesReport_2015.pdf

21.

Ramsay

Rutterford

Gregory

Dunne

Eldridge

Sharp

Feder

(2012). Domestic violence: Knowledge, attitudes, and clinical practice of selected UK primary healthcare clinicians. British Journal of General Practice, 62 (602), e647-e655.

22.

Rasmussen

L. A.

Hughes

M. J.

Murray

C. A.

(2008). Applying motivational interviewing in a domestic violence shelter: A pilot study evaluating the training of shelter staff. Journal of Aggression, Maltreatment & Trauma, 17, 296-317.

23.

Sargent

K. S.

McDonald

N. L.

Jouriles

E. N.

(2016). Evaluating an online program to help children exposed to domestic violence: Results of two randomized controlled trials. Journal of Family Violence, 31, 647-654. doi:10.1007/s10896-016-9800-8

24.

Schoening

A. M.

Greenwood

J. L.

McNichols

J. A.

Heermann

J. A.

Agrawal

(2004). Effect of an intimate partner violence educational program on the attitudes of nurses. Journal of Obstetric, Gynecologic & Neonatal Nursing, 33, 572-579.

25.

Short

L. M.

Surprenant

Z. J.

Harris

J. M.

(2006). A community-based trial of an online intimate partner violence CME program. American Journal of Preventative Medicine, 30, 181-185.

26.

Springtide Resources. (2008). An integrated anti-oppression framework for reviewing and developing policy: A toolkit for community service organizations. Toronto, Ontario, Canada. Retrieved from http://www.oaith.ca/assets/files/Publications/Intersectionality/integrated-tool-for-policy.pdf

27.

Thackwray

(1997). Effective evaluation of training and development in higher education. London, England: Kogan Page.

28.

Thompson

R. S.

Rivara

Thompson

D. C.

Barlow

W. E.

Sugg

N. K.

Maiuro

R. D.

Rubanowice

M. D.

(2000). Identification and management of domestic violence: A randomized trial. American Journal of Preventive Medicine, 19, 253-263.

29.

United Nations Development Fund for Women. (2010). Monitoring, evaluation and knowledge management framework. New York, NY: Author.

30.

Varcoe

Hankivsky

Ford-Gilboe

Wuest

Wilk

Hammerton

Campbell

(2011). Attributing selected costs to intimate partner violence in a sample of women who have left abusive partners: A social determinants of health approach. Canadian Public Policy, 37, 359-380.

31.

Wathen

(2012). Health impacts of violent victimization on women and their children. Ottawa, Ontario, Canada: Department of Justice Canada.

32.

World Health Organization. (2013). Global and regional estimates of violence against women: Prevalence and health effects of intimate partner violence and non-partner sexual violence. Geneva, Switzerland: Author.

33.

Zaher

Keogh

Ratnapalan

(2014). Effect of domestic violence training: Systematic review of randomized controlled trials. Canadian Family Physician, 60, 618-624.

34.

Zhang

Hoddenbagh

McDonald

Scrim

(2012). An estimation of the economic impact of spousal violence in Canada, 2009. Ottawa, ON, Canada: Department of Justice Canada.

Evaluating the Effectiveness of Online Training for a Comprehensive Violence Against Women Program: A Pilot Study

Abstract

Keywords

The Present Study

Evaluated Training Program: Ontario Association of Interval & Transition Houses (OAITH) Foundations of VAW Online Training Course

Description of Course

Method

Recruitment

Participants

Measures

Core competencies

Survey measures 1

General evaluation metrics

Qualitative Component

Analytic Strategy

Waitlist comparison group

Results: Assessment of Learning

Full Sample Pre-/Posttests

Comparison Group Condition

Subjective Rating of Comfort, Abilities, and Knowledge (Course 1 Only; N = 50)

Subjective Rating of Enhancement (Course 2 Only; N = 58)

General Evaluation Metrics: Reaction, Learning, and Performance (Course 2 Only; N = 58)

Results: Reflections on the Course and Subjective Value

Discussion

Limitations and Future Research

Conclusion

Footnotes

Appendix A

Appendix B

Declaration of Conflicting Interests

Funding

Notes

Author Biographies

References

Survey measures¹