Impact of Virtual Training Environments on the Acquisition and Transfer of Knowledge

Abstract

Virtual training environments are appropriate to train complex tasks that require collaboration and interaction among the members of a team, especially if training in reality is not possible, too expensive or too dangerous. The field study reported in this paper compared three training conditions (virtual condition, standard condition, and control condition). The participants were police officers who were being trained in the communication between ground forces and a helicopter crew during an operation. This task (like many other tasks of the police, fire brigade and emergency services) is of high complexity and has no single “correct” solution, is based on specialization of tasks within a team, requires intensive communication among team members, and consists of situations in which human beings are in danger. Learning outcomes and knowledge transfer were measured as dependent variables. The results validate that virtual training was as efficient as standard training with regard to knowledge acquisition, and it was even more efficient with regard to knowledge transfer. With regard to the perceived value of the training, the participants judged standard training to be better than virtual training (except for training satisfaction, where no difference was found between standard and virtual training). These results indicate that virtual training is an effective tool for training in complex tasks that require collaboration and cannot fully be trained for in reality.

Introduction

Virtual training environments (VTE) are an appropriate training method if training in reality would be too costly or dangerous¹ or when the use of VTE is the only option for training.² Virtual training has been shown to be successful not only when using head-mounted displays or cave automatic virtual environments, where projectors are directed at three or more walls of a cube to create immersive virtual environments, but also when using desktop applications.^3–8 There are several empirical studies that demonstrate the impact of training in VTE on specific learning outcomes, such as motor tasks,¹ spatial relations,⁹ route learning⁶ or reactions to emergency situations.¹⁰ But VTE could be also used in more complex scenarios, where individuals not only have to be trained in how to react, but also how to collaborate in teams. The team members have to react in an adequate way under “rapidly evolving and changing conditions, severe time compression, and high degree of ambiguity and uncertainty”.^11(p18) They need to know the equipment they use to communicate and interact, need to realize what their task is and how to accomplish it, and need to make decisions. They must also recognize what their role is in a team context and thus must know what their teammates' knowledge and skills are.¹¹ Salas and coworkers¹² define team tasks by the following characteristics: “multiple sources of information, task interdependencies, coordination among members, common and valued goals, specialized member roles and responsibilities, task-relevant knowledge, intensive communication, and adaptive strategies to help respond to change”^{(pp1052–1053)}. All team members need certain knowledge, skills, and abilities to solve the common task.¹³

There is a lack of empirical investigation into the effectiveness of virtual training for complex and collaborative tasks, despite the variety of tools and concepts for the use of VTE. The questions are whether trainees can gain the required knowledge through virtual training and whether the training technology is efficient.¹⁴ There are three reasons for the lack of empirical evidence: It is often too dangerous and too costly to observe virtually trained behavior in a real task. The situations for which virtual training is used are often very rare and extraordinary, so the effects on behavior and performance cannot be directly measured. Further, it is often impossible to provide for nontreatment control groups¹⁴ in the field. Even for nonvirtual training of complex, collaborative tasks, up to now only a small number of field studies have been conducted.¹³

We confronted this gap and designed a study to measure the impact of VTE on the acquisition and transfer of knowledge. We assessed if virtual training can result in at least the same outcomes as standard face-to-face training, and if it can have better results than a non-treatment control group. Our study is based on the model of Kirkpatrick¹⁵ that identifies four levels of training to be evaluated: reaction, learning, behavior, and results. Reaction as a training outcome is seen as the satisfaction of participants with the training, and this outcome can be measured by self-reports. These reactions of the trainees predict transfer of knowledge and impact of training on job performance.¹⁶ A positive attitude toward the training and high motivation as well as useful training content¹⁴ all make transfer more probable. Learning outcomes are defined as the extent to which participants improve their knowledge or skills. Behavioral outcomes are changes in the behavior of participants because of a training program. Results as a training outcome refer to organizational and personnel changes, such as lower costs, higher efficiency, or better work climate. We focused on the reactions to the training, learning outcomes, and knowledge transfer. We did not measure results, or organizational outcomes, as they are mainly long-term outcomes that cannot be directly assessed after a training session or as part of an experimental design. These three aspects we chose to focus on are also in line with the dimensions of teamwork as defined by Salas and Cannon-Bowers¹⁷: cognition (learning outcomes), skills (knowledge transfer), and attitudes (reactions to the training).

The participants of the study were police officers who trained in operations where they had to interact with a helicopter crew. During these operations they were confronted with a lot of stressors (like the noise produced by the helicopter) and had to handle a variety of information from different sources. Such a highly complex and fast scenario requires team coordination under extreme conditions. We compared a virtual condition with a standard training condition and a control condition. If the virtual training condition is effective participants should acquire at least as much knowledge as participants in the standard condition. Because the virtual training provides training situations that are more complex than the situations provided by the standard training, the knowledge transfer to other complex situation should be enhanced. Virtual training provides different possibilities to vary the complexity of the task, and it provides guidance and feedback depending on the current state of each trainee. This reduces intrinsic cognitive load but induces germane cognitive load, and thus should support knowledge transfer.¹⁸ This argumentation is consistent with the transfer paradox¹⁹: Instructional strategies (e.g., trainees repeatedly practicing similar tasks) that support learning mastery for complex tasks do not necessarily support knowledge transfer. The training of very different scenarios in a VTE might lead to easier transfer, as trainees are already forced to transfer their knowledge to different scenarios and conditions in the VTE, and furthermore are forced to transfer their knowledge to the VTE. Another plus is that the virtual training scenarios could be discussed using a replay function of the VTE, making it possible to swap perspectives. These activities could not be implemented in the standard training. So we expect that with regard to knowledge transfer, participants of the virtual training should outperform participants of the other conditions. The participants' attitudes, on the other hand, remain an open question: would the participants in the virtual condition accept the virtual training as valuable and show motivational outcomes similar to participants in the standard condition?

Methods

Design

The study followed a between-subject design. The subjects were assigned to three conditions: virtual training, standard training, or control group. The participants trained in groups of eight persons. In the virtual condition the training conformed to an innovative training-concept including the use of the VTE Virtual Police²⁰ that was developed in cooperation with the Police of a federal state in Germany to fulfill their training requirements. In the standard condition the training was based on universally accepted standard training, that is, performing short training sequences in reality with a trainer while simulating the communication with the helicopter. Both training conditions started with a theoretical introduction to relevant strategies and rules. The training also included analyzing prior operations where such critical events had occurred. In the virtual condition, the trainees then performed several short training trials within the VTE to train for communication with the helicopter crew. In groups of up to eight people the participants had to carry out an operation. All trainees are represented by avatars and they perceive the training through their individual perspective. They used the mouse and keyboard of their computers to navigate through the virtual environment. Using headsets they could talk to each other if they were next to each other in the virtual environment or radio with other team members and the helicopter pilot and other colleagues. After each trial the replay function of the VTE was used to help participants reflect on the performance of the team and indicate possible problems or failures. The VTE allows showing the trainees every possible situation of the training from all points of view, to focus on critical situations and communication failures or identify occurring training needs. In the control condition, the participants received a written handout and conducted a group task that was not related to the training goals. This is the normal procedure to inform police officers about how to react in certain situations when training in reality is not possible. As dependent variables we measured reactions to the training (motivational outcomes and perceived value of training), learning outcomes, and knowledge transfer. Dependent variables were measured after the training.

Dependent variables

Reactions to the training—motivational outcomes

We measured acceptance of the training (3-item scale²¹), satisfaction with the training (2 items developed by the authors, sample item: “I am satisfied with the training”), and relevance of the training to their daily work (6 items developed by the authors, sample item “The training had a strong connection to my daily work.”).

Reactions to the training—perceived value of training

We measured subjective learning success (4-item scale²⁰), subjective assurance for future helicopter operations (4 items developed by the authors, sample item: “The training makes me feel well prepared for helicopter operations”), and realism of training (4 items developed by the authors, sample item: “I did not feel like being in a training”). All items were measured using a 7-point Likert scale with endpoints labeled “don't agree at all” and “agree completely”.

Learning outcomes

We implemented a pre- and post-test design to measure learning outcomes and used a knowledge test that consisted of 20 multiple choice questions about the defined learning goals. The items were constructed together with the police trainers. All items were pretested with a sample of 22 police officers (five female, age M=37, range 22–51). Four items were correctly answered by more than 90 percent of the sample and were thus revised to make them more difficult. We found that officers who had already taken part in a helicopter operation (r=0.46, p=0.03) or had already flown with a helicopter (r=0.56, p=0.01) had more knowledge than those who did not. This indicated that the test measured knowledge relevant for interactions with a helicopter crew.

Knowledge transfer

The knowledge transfer test contained 11 short videos of the real world showing ground forces that interacted with a helicopter crew in a varietyof different situations. The videos stopped at certain points of critical situations and the participants were asked to describe how they would behave (cf. the method suggested by Kirkpatrick¹⁵). The videos were presented on mobile computers. The participants had to write their answers directly into a text field after viewing each single video. These videos were constructed together with the police trainers and were based on typical and critical operation scenarios. All training goals were included and were also pretested with a sample of 22 police officers (same as reported below). All answers of the participants were rated according to knowledge, reactions, awareness of dangerous situations, and communication. The results indicate whether the participants could translate their knowledge into correct reactions to critical events and would handle a situation in an appropriate way.

Hypothesis

H1: If virtual training is at least as effective as the standard training, we expect that virtual and standard training groups will not differ in their reactions to the training.

H2: The virtual training group should acquire as much knowledge as the standard training group. Both groups should outperform the control group.

H3: With regard to knowledge transfer, the virtual training group should outperform the standard training group because the virtual training provides a variety of complex situations that enable more knowledge transfer than the limited standard trainings.

Participants

Forty-seven police officers participated in the study, 29 males and 18 females, between the ages of 19 and 34, mean age 23 years (SD=3.34). Participants voluntarily participated without any payment. Subjects were assigned to the three conditions by balancing work experience and gender to make certain that the three conditions were comparable with regard to competence, skills, and knowledge. The virtual and standard training conditions consisted of 16 persons each; the control condition consisted of 15 persons.

Results

Reactions to the training

Concerning the reactions to the training (H1) we compared virtual and standard condition with analysis of variances (ANOVAs) (we applied an alpha level of α=0.20 as null hypotheses are tested). Table 1 summarizes the results.

Table 1.

Motivational Outcomes and Perceived Value of Training for the Standard and the Virtual Condition

	Standard condition	Virtual condition
Motivational outcomes
Acceptance^a	6.15 (0.44)	5.56 (0.88)
Training satisfaction	6.06 (0.91)	5.72 (1.03)
Relevance^a	5.13 (1.13)	4.10 (1.05)
Perceived value of training
Subjective learning success^a	6.14 (0.62)	5.48 (0.83)
Assurance^a	6.06 (0.67)	5.59 (1.01)
Realism of training^b	4.19 (1.12)	3.44 (1.13)

Standard condition significantly higher than virtual.

Standard condition marginally higher than virtual.

For all scales Cronbach's alpha was between α=0.55 and α=0.86. Acceptance of the training was higher in the standard condition (M=6.15, SD=0.44) than in the virtual training condition [M=5.56, SD=0.88, F(1, 46)=5.68, p=0.02, part. η²=0.16]. There was no difference in satisfaction between the standard condition (M=6.06, SD=0.91) and the virtual condition [M=5.72, SD=1.03, F(1, 46)=1.00, p=0.33, part. η²=0.03]. Relevance of the training was higher in the standard condition (M=5.13, SD=1.31) than in the virtual condition [M=4.10, SD=1.05, F(1, 46)=7.05, p=0.01, part. η²=0.19]. Subjective learning success was higher in the standard condition (M=6.14, SD=0.62) than in the virtual condition [M=5.48, SD=0.83, F(1, 46)=6.44, p=0.02, part. η²=0.18]. Subjective assurance was higher in the standard condition (M=6.06, SD=0.67) than in the virtual condition [M=5.59, SD=1.01, F(1, 46)=2.41, p=0.13, part. η²=0.07]. Realism of training was marginally higher in the standard condition (M=4.19, SD=1.12) than in the virtual condition [M=3.44, SD=1.13, F(1, 46)=3.56, p=0.07, part. η²=0.11]. H1 can only be confirmed for satisfaction with the training. Acceptance and perceived relevance were higher in the standard condition, making a transfer of training to new situations more probable for these participants. Further, the standard condition showed better results for subjective learning success, subjective assurance, and realism of training.

Learning outcomes

A repeated measures analysis of variance with learning outcomes (pre- and post-test) as repeated measure and condition (virtual, standard, and control) as second factor, revealed a significant main effect for learning outcomes [F(1, 46)=334.07, p<0.001, part. η²=0.88]. Participants in all conditions made higher scores in the post-test than in the pre-test (Fig. 1). The analysis of variance also revealed interaction of knowledge acquisition with condition [F(2, 46)=17.55, p<0.001, part. η²=0.44]. A Bonferroni adjusted post hoc test showed no difference in the pre-test, but significant differences between the conditions [F(2, 46)=6.99, p=0.002, part. η²=0.24]. Participants acquired more knowledge in the standard training condition (M=7.13, SD=2.47) and the virtual training condition (M=6.94; SD=1.73), than in the control condition (M=3.07, SD=2.15, p<0.01). This confirms H2.

FIG. 1.

Results for learning outcomes (pre- and post-test) for the three conditions condition (virtual, standard, and control).

Knowledge transfer

We compared the three conditions with an ANOVA. Condition was found to be a significant main effect [F(2, 46)=6.03, p=0.005, part. η²=0.22]. Participants in the virtual condition scored higher than participants in the other conditions (Fig. 2). A Bonferroni adjusted post hoc test showed higher scores for the virtual condition (M=5.17, SD=1.21) than the standard condition (M=3.98, SD=1.33, p=0.02) and the control condition (M=3.85, SD=0.94, p=0.01). H3 can thus be confirmed. Only the group that trained virtually showed better results in knowledge transfer than the control group. Knowledge transfer significantly correlates with the post-test score for learning outcome (r=0.34, p=0.02), but not significantly with the pre-test score (r=0.19, p=0.19).

FIG. 2.

Results for knowledge transfer for the three conditions condition (virtual, standard, and control).

Conclusion

In our study we compared virtual training with a standard training and a control group concerning motivational outcomes and perceived value of training, learning outcomes, and knowledge transfer. Concerning motivational outcomes and perceived value of training we found that the standard training was judged to be better than the virtual training, except for training satisfaction (no difference between virtual and standard condition). These results could be used to improve perception of the virtual training by addressing usability issues and to integrate different training tools and methods into the virtual environment. In knowledge acquisition all three groups show higher scores in the post-test than in the pre-test. But the two training groups show higher scores in the post-test than the control group. Thus, we conclude that the effect of knowledge acquisition resulted from the trainings, not just from the second test measurement. In knowledge transfer the virtual training group shows the highest scores and we did not find any differences between the standard training and the control group. The virtual training might thus have a special advantage in supporting knowledge transfer. Compared to a real helicopter operation the virtual training was the most realistic training and offered the most complex training situation. This led to good results in the video test for the virtual training group.

Our study indicates that virtual training is valid for providing training in complex, collaborative tasks, training that cannot be fully accomplished in reality. We could show that virtual training is valid for gaining knowledge, and that knowledge transfer from a VTE to another task is possible. For complex tasks that require continuous communication among members of a diverse team, virtual training is as efficient as standard training with regard to knowledge acquisition, and even more efficient with regard to knowledge transfer.

There is some empirical work on the transfer of knowledge acquired in VTE to real world tasks.^1,22,23 Additional research comparing virtual and nonvirtual conditions has been done on individual tasks like route learning.^24,25 But we discovered that there is a lack of research comparing virtual training to standard training when the focus is on more complex and team-based tasks like communication and interaction within difficult operations. Our study thus focuses on more complex and team-based tasks like communication and interaction within difficult operations. The results demonstrate how important it is to examine the impact of virtual training and to directly compare different training methods, not only measure transfer from virtual training to real situations. Our experiment provides the first empirical evidence that virtual training is a valid method of supporting coordination and communication of teams performing complex tasks. Future research should concentrate on the transfer of virtual training to tasks in reality, especially when the tasks are combined with stress and pressure. In addition, future work should regard team performance and measure the developed knowledge, skills, and attitudes on a team level.^13,26

Footnotes

Author Disclosure Statement

No competing financial interests exist.

References

Rose

, Attree

, Brooks

et al. Training in virtual environments: transfer to real world tasks and equivalence to real task training. Ergonomics, 2000; 43:494–511.

Romano

, Brna

. Presence and reflection in training: support for learning to improve quality decision-making skills under time limitations. CyberPsychology and Behavior, 2001; 4:265–277.

Harrison

, Derwent

, Enticknap

et al. The role of virtual reality technology in the assessment and training of inexperienced powered wheelchair users. Disability and Rehabilitation, 2002; 24:599–606.

Heinrichs

WLR

, Youngblood

, Harter

et al. Simulation for team training and assessment: case studies of online training with virtual worlds. World Journal of Surgery, 2008; 32:161–170.

Jennings

, Collins

. Virtual or virtually U: educational institutions in second life. International Journal of Social Sciences, 2007; 2:180–186.

Lloyd

, Persaud

, Powell

. Equivalence of real-world and virtual-reality route learning: a pilot study. CyberPsychology and Behavior, 2009; 12:423–427.

Rafi

, Samsudin

. Practising mental rotation using interactive Desktop Mental Rotation Trainer (iDeMRT) British Journal of Educational Technology, 2009; 40:889–900.

Warburton

. Second life in higher education: assessing the potential for and the barriers to deploying virtual worlds in learning and teaching. British Journal of Educational Technology, 2009; 40:414–426.

Arthur

, Hancock

, Chrysler

. The perception of spatial layout in real and virtual worlds. Ergonomics, 1997; 40:69–77.

10.

Gamberini

, Cottone

, Spagnolli

et al. Responding to a fire emergency in a virtual environment: different patterns of action for different situations. Ergonomics, 2003; 46:842–858.

11.

Cannon-Bowers

, Salas

. 1998. Individual and team decision making under stress: theoretical underpinnings. Cannon-Bowers

, Salas

. Making decisions under stress: implications for individual and team training. Washington, D.C.: American Psychological Association, 17–38.

12.

Paris

, Salas

, Cannon-Bowers

. Teamwork in multi-person systems: a review and analysis. Ergonomics, 2000; 43:1052–1075.

13.

Delise

, Allen Gorman

, Brooks

et al. The effects of team training on team outcomes: a meta-analysis. Performance Improvement Quarterly, 2010; 22:53–80.

14.

Salas

, Milham

, Bowers

. Training evaluation in the military: misconceptions, opportunities, and challenges. Military Psychology, 2003; 15:3–16.

15.

Kirkpatrick

. 1976. Evaluation of training. Craig

. Training and development handbook. New York: McGraw-Hill, 301–319.

16.

Alliger

, Tannenbaum

, Bennett

et al. A meta-analysis of the relations among training criteria. Personnel Psychology, 1997; 50:341–358.

17.

Salas

, Cannon-Bowers

. 2000. The anatomy of team training. Tobias

, Fletcher

. Training and retraining: a handbook for businesses, industry, government and military. Farmington Hills, MI: Macmillan Reference, 312–335.

18.

Van Merriënboer

JJG

, Kester

, Paas

. Teaching complex rather than simple tasks: balancing intrinsic and germane load to enhance transfer of learning. Applied Cognitive Psychology, 2006; 3:343–352.

19.

Van Merriënboer

JJG

, De Croock

MBM

, Jelsma

. The transfer paradox: effects of contextual interference on retention and transfer performance of a complex cognitive skill. Perceptual and Motor Skills, 1997; 84:784–786.

20.

Moskaliuk

, Bertram

, Cress

. 2011. Training in virtual training environments: connecting theory to practice. Spada

, Stahl

, Miyake

et al. Connecting Computer-Supported Collaborative Learning to Policy and Practice: CSCL2011 Conference Proceeding, 1 Hong Kong, International Society of the Learning Sciences, 192–199.

21.

Kopp

, Dvorak

, Mandl

. 2003. Evaluation des Einsatzes von Neuen Medien im Projekt ‘Geoinformation - Neue Medien für die Einführung eines neuen Querschnittfachs (Evaluation of new media usage within the project “geo-information—using new media to implement a cross-section subject”) München: Universität, Department Psychologie, Institut für Pädagogische Psychologie.

22.

Koh

, von Wiegand

, Garnett

et al. Use of virtual environments for acquiring configurational knowledge about specific spaces. I. preliminary experiment. Presence: Teleoperators and Virtual Environments, 1999; 8:632–656.

23.

Ruddle

, Payne

, Jones

. Navigating buildings in “desk-top” virtual environments: experimental investigations using extended navigational experience. Journal of Experimental Psychology: Applied, 1997; 3:143–159.

24.

Barlow

. Acquisition of route knowledge through navigation in a virtual environment. International Journal of Cognitive Ergonomics, 2001; 5:279–295.

25.

Koenig

, Crucian

, Dalrymple-Alford

et al. Assessing navigation in real and virtual environments: a validation study. International Journal on Disability and Human Development, 2011; 10:325–330.

26.

Goldstein

, Ford

. 2002. Training in organizations: needs assessment, development, and evaluation. Belmont: Wadsworth Thomson Learning.