Efficacy and Usability of a Virtual Simulation Training System for Health and Safety Hazards Encountered by Healthcare Workers

Abstract

Objective:

This study evaluated the efficacy, usability, usefulness, and desirability (UUD) of a Home Healthcare Interactive Virtual Simulation Training System (HH-VSTS) designed to train home healthcare workers (HHWs) and healthcare students to identify and respond to health and safety hazards in client homes.

Materials and Methods:

Participants were randomly assigned to either the HH-VSTS training group or to the paper-based training group. The HH-VSTS group completed three HH-VSTS Training Modules on a laptop/desktop computer. The training modules addressed hazard identification, hazard rationale, and hazard response to electric/fire/burn, slip/trip/lift, and environmental hazards. The paper-based training group reviewed identical information in a written hard-copy format. Both groups completed an HH-VSTS Assessment module. Participants completed demographic/background and UUD questionnaires, and in-system metrics measured their performance on hazard identification, rationale, and response.

Results:

Participants (n = 74) were HHWs and students in health profession programs. There were no significant differences in participants' ability to correctly identify hazards, rationale, or how to address them. Participants identified over 90% of hazards, although fewer participants were able to correctly identify what makes an item a hazard or how to manage it. For those in the HH-VSTS group, over 83% found the HH-VSTS easy to use, over 94% agreed the HH-VSTS was useful, and over 80% liked it.

Conclusion:

The HH-VSTS provided and engaging, efficacious training that was as effective as a typical paper-based training. In addition, the HH-VSTS is usable by a variety of end users, regardless of computer or gaming experience.

Introduction

Home healthcare workers (HHWs) experience unique challenges to their health and safety, performing physically demanding work in a relatively uncontrolled, unpredictable environment.^1–3 Although HHWs are at high risk for injury and illness, delivery of HHW health and safety training is hampered by the nature of their work, including community-based care, travel time between client homes, and lengthy work hours.⁴ Existing training approaches are often selectively focused, not comprehensive, and/or are provided sporadically by homecare agencies,^5–10 and conventional training may not engage learners in immersive, interactive training experiences that effectively facilitate knowledge retention and transfer to HHW practice.¹

Advances in technology provide multiple opportunities for enhanced approaches to HHW health and safety training. Gaming technologies and virtual simulation in particular, can be used to create engaging, effective training programs. Compared with traditional training approaches, these “Serious Games” for training and education^11–13 may better support knowledge transfer to the healthcare environment based on the ability to create fully immersive, real-life situations and imagery.^14,15

We developed the Home Healthcare Virtual Simulation Training System (HH-VSTS) for HHWs using an immersive, participatory, user-centered design approach, a usability best practice¹⁶ that engaged HHWs from multiple professions with program designers and investigators in an iterative and highly interactive process. The HH-VSTS (https://homehealthcaresafety.osu.edu) was developed as a Serious Game to help HHWs identify and respond to health and safety hazards they encounter in client homes.^1–3

The HH-VSTS, built with the Unity™ gaming engine, uses an interactive virtual environment representing a basic two-story house. It is delivered using a laptop/desktop computer in which the user navigates the virtual environment using a standard keyboard and mouse. The system includes an overview, walk-through video, and tutorial to orient players to the simulation and to home safety assessment (Fig. 1). The system contains three training modules and an assessment module that address: (1) electrical/fire/burn, (2) slip/trip/lift, and (3) environmental hazards often found in client homes (Table 1). Each module includes a client case, providing clues about potential hazards the HHW may encounter. Other features include a flashlight and magnifier to examine conditions and hazards more closely, and teleportation from room to room. Assets such as fire alarms or water faucets can be tested as working/not working. Participants move through the house and select assets in the virtual environment that they think may be hazardous. Upon selection, a dialog box appears on the screen. If the selection is not a hazard, a brief justification is provided and the HHW is returned to the game. If the selection is a hazard, a brief justification, rationale for the hazard, and potential solutions are provided. Participants can see how many of the hazards they have found in each room and within the home. The assessment module includes a performance score, including the number of correct answers to test questions.

FIG. 1.

Scenes from the HH-VSTS electrical, fire, burn hazard training module (a), slip, trip, and lift hazard training module (b), environmental hazard training module (c), and assessment module (d). HH-VSTS, Home Healthcare Virtual Simulation Training System.

Table 1.

Examples of Hazards Found in the Virtual Simulation Training System by Training Module

Training module	Hazard examples
Electric/fire/burn	Frayed electric cord
	Lit cigarette with oxygen flowing
	Overloaded power strip
	Space heater on with newspapers stacked nearby
	Water heater set too high
Slip/trip/lift	Area rugs
	Water on floor
	Low bed
	Clutter on stairs
	Snow/ice on front steps
Environmental	Biohazard (blood) in carpet
	Roaches on kitchen walls
	Mold on shower curtain
	Needles sticking out of trash bags
	Bedside commode that has not been emptied

A comprehensive efficacy assessment, including assessment module outcome data, and user perceptions of technical usability, accessibility, and program usefulness was completed.^17,18 Of key interest was whether prior experience using computers or playing computer games would impact the overall effectiveness and usability of the HH-VSTS. The study aims were to: 1.

Evaluate the efficacy of the HH-VSTS in preparing HHWs and students enrolled in healthcare programs to identify and respond to hazards in client homes and determine if there were differences based on study group assignment (HH-VSTS group vs. paper-based training group) and individual characteristics.

Assess the perceived usability, usefulness, and desirability (UUD) of the HH-VSTS among HHWs and students enrolled in healthcare programs.

Determine the impact of computer use and gaming experience on perceived UUD of the HH-VSTS.

Materials and Methods

The efficacy and UUD of the HH-VSTS were tested using a block randomized design. Participants were assigned to either the HH-VSTS group who completed the HH-VSTS training modules, or to the paper-based training group. There were 75 participants: 39 in the HH-VSTS group and 36 in the paper-based group. Participants were at least 18 years old, either HHW or a student in a health profession education program (nurse, occupational therapy, and physical therapy), and passed a prescreening assessment to detect susceptibility to simulation sickness. The study was approved by the University Institutional Review Boards. All participants signed informed consent documents and received a $50.00 incentive before the training.

Measures

The validated Modified Home Healthcare Worker (M-HHCW) questionnaire⁴ was used to collect data on demographics, home healthcare experience, and level of experience with computers and computer gaming. Participants answered questions about weekly computer use (On average, how many hours a week do you use a computer?) and computer gaming experience (How would you rate your experience with computer gaming?). Computer use was coded as follows: low use (0–3 hours per week), moderate to high use (>3 hours per week), and student use (students were assumed to use computers at least 3 hours per week in their academic programs). Computer gaming experience was categorized into two levels based on participant self-report: no or limited experience, and moderate or a lot of experience.

The UUD questionnaire assessed usability (ease of use), usefulness (value and applicability), and desirability (appeal or interest). Participants rated the extent to which they agreed with statements about HH-VSTS UUD using a five-point ordinal scale (1 = strongly disagree; 5 = strongly agree). Participants were also provided the opportunity to provide comments regarding the VSTS.

The HH-VSTS assessment module yielded objective data on participant performance. The assessment module includes 27 hazards: 9 from each of the three hazard categories, as well as selectable nonhazards. For identified hazards, participants responded to the multiple choice questions, “What makes this a hazard?” and, “What to do about the hazard?”

Procedures

Group training sessions were scheduled at community home healthcare agencies and university computer laboratories. The type of training (HH-VSTS group vs. paper-based training) was randomly assigned to each training time using a block randomization schedule. All participants were assigned to the group training time that worked for their work or class schedule. For example, group 3 was randomly predetermined to be a HH-VSTS group; therefore, the participants who attended the group 3 training time completed the HH-VSTS training. There were 22 training groups with between 1 and 9 participants per group.

Once informed consent was obtained, all participants completed the hard-copy M-HHCW. Participants attending during a HH-VSTS training time then completed the HH-VSTS overview, walk-through, tutorial, the three HH-VSTS training modules, and the HH-VSTS assessment module, and the UUD assessment. After completing the M-HHCW, participants attending during a paper-based training time read a hard-copy, textual version of the HH-VSTS, which contained the same information about hazards, rationale, and hazard management. The paper-based group then completed the HH-VSTS tutorial, assessment module, and the UUD assessment.

Data management and analysis

The percentage of correct responses to the assessment module dialog box queries were compared between the HH-VSTS and paper-based training groups, as well as by participant type, age category (≦29 years old/30+ years old), years of home healthcare experiences (≦5 years/≧6 years), gaming experience (yes/no), and comfort with computer technology. Comparisons were made using analysis of variance or chi-square analyses, depending on the level of the data, with adjustment of the alpha level through a Bonferroni procedure. Chi-square analysis was used to assess between-group differences in effects of computer use levels and gaming experience.

Results

Efficacy testing

Seventy-four participants were included in efficacy testing: 38 in the HH-VSTS group and 36 in the paper-based group. Most participants were female (86.5%), white (81.1%), and healthcare professional students (52.7%). About 38% were home healthcare aides/homemakers. Their average age was 34.1 years (standard deviation [SD] = 14). The average length of time in home healthcare for the 34 currently employed participants was 8.25 years (SD = 8.1). There were no significant between-group differences for demographic variables.

The two groups identified approximately 24 hazards in the assessment module. We compared percentage of (1) hazards correctly identified as a hazard (hit), (2) nonhazards correctly identified as a nonhazard (correct rejection), (3) hazards incorrectly identified as nonhazards (false positive), and (4) nonhazards incorrectly identified as a hazard (miss) between groups (Table 2). The HH-VSTS group had an average of 4.7 (SD = 3.2) correct rejections of nonhazards, versus an average of 10.8 correct rejections for the paper-based training group (t = 5.9, P < 0.001). Independent of training mode, students significantly more often correctly responded to the “What makes this a hazard” compared with HHWs (90.1% vs. 84.7%, P < 0.001). The HH-VSTS and paper-based training groups were similar in correct identification of hazards 98.1% and 96.4% of the time (hit rate), respectively (P = 0.07; Table 2). There were no significant differences in identification of hazards based on participant type (HHWs vs. students), age group, years in home healthcare, computer gaming experience, or comfort with computer technology. Likewise, there were no significant differences between the VSTS and the paper-based training groups regarding the correct responses to the “What makes this a hazard” or the “What to do about the hazard” items (86.2% vs. 89.0%, P = 0.21; 81.8% vs. 86.9%, P = 0.15, respectively).

Table 2.

Hazard Identification Responses

	Hazard state
HH-VSTS group (n = 38)	Hazard	Nonhazard
Identified as a hazard	98.1%	52.4%
Identified as a hazard	Hit	False positive
Identified as a non-hazard	1.9%	47.6%
Identified as a non-hazard	Miss	Correct rejection
Paper-based training group (n = 36)
Identified as a hazard	96.4%	39.5%
Identified as a hazard	Hit	False positive
Identified as a nonhazard	3.6%	60.5%
Identified as a nonhazard	Miss	Correct rejection

Note: Differences between the HH-VSTS and paper-based training groups regarding hazard identification considered only the hazards selected by participants. For example, if a hazard such as “water on the floor in the kitchen” was not selected as a possible hazard, it was not considered in any analysis. This allowed us to determine the accuracy of participants' ability to indicate if an item they identified in the home was/was not a hazard. Thus, each participant had a unique number of hazards identified. The percentage correctly identified was based on the number of potential hazards identified. Similarly, only the “What makes this a hazard?” and “What to do about the hazard?” possible response options for hazards identified were considered. If a participant did not identify the “water on the floor” as a hazard, they were not able to respond to the “What makes this a hazard?” and “What to do” dialog boxes, therefore, each participant also had a unique score for the “What makes this a hazard” and “What to do” items, and the unit of analysis was the percentage correct/incorrect based on the total possible correct responses.

HH-VSTS, Home Healthcare Virtual Simulation Training System.

Exploration of correct responses by the three hazard types revealed that the HH-VSTS and paper-based training groups were able to identify over 90% of the hazards across hazard types (Fig. 2). Within each training module, significantly more participants could correctly identify hazards than correctly identify what makes an item a hazard. For fire, electric, and burn hazards, significantly fewer participants correctly identified the appropriate “What to do” response as compared with both hazard identification and hazard rationale. In addition, participants in both groups also correctly identified the slip/trip/lift “What makes this a hazard” and “What to do” items more often than the other two hazard types (electric/fire/burn; environmental). Overall, understanding of what makes an item a hazard and what to do about the hazard was best in the slip/trip/lift training module.

FIG. 2.

Percent correct responses to the three types of questions within each module. Dotted lines between bars indicate where there are statistically significant differences (P < 0.05) between question types within each module. Brackets between bars indicate where there are statistically significant differences (P < 0.05) between training modules for each question type.

UUD testing

Data from 26 HHWs and 13 students were provided for the UUD analyses. Most participants were female (87.1%) and had an average of 14 (SD = 10.6) years of experience working in home healthcare. Approximately 67% were white and 33% percent African American/black. Overall, participants reported that the HH-VSTS was usable (Table 3) and useful (Fig. 3). Some common challenges observed by the investigators included difficulty using the mouse to look around the environment and using the “W,” “A,” “S,” and “D” keys or arrow keys to move forward, back, or side-to-side. Participants agreed/strongly agreed that the HH-VSTS is helpful for identifying all three categories of hazards, that information about “What to Do” was useful, and that the HH-VSTS would help to keep workers safe. Only 11% agreed with the statement that the content was too simple. As for desirability, 81% of participants agreed/strongly agreed that they liked the HH-VSTS and 84% indicated their coworkers would like it. No participants agreed with the statement that the HH-VSTS was boring. Over two-thirds agreed/strongly agreed that they would like this type of training at their place of work.

FIG. 3.

Level of agreement with usefulness statements.

Table 3.

Home Healthcare Virtual Simulation Training System Usability (n = 39)

Usability indicators	Percentage that agree or strongly agree (n)
Usability of hardware
VSTS is easy to use	83.8 (31/37)
Information boxes easy to click through	97.3 (36/37)
Keyboard controls easy to use	75.7 (28/37)
Easy to move through house	64.9 (24/37)
Easy to click on hazards	63.9 (23/36)
Usability of program
Coworkers would learn quickly	82.9 (29/35)
Hard-to-see hazards	16.7 (6/36)
Training took too long	16.2 (6/37)
Training too complicated	8.1 (3/37)
Coworkers would need help using VSTS	3.5 (3/37)

Effects of computer gaming and computer use experience on usability

Most of the usability indicators were consistent across participants' weekly computer use (Table 4) and gaming experience (Table 5). However, although not statistically significant, 38% of those who reported low computer use found it easy to move through the virtual home compared with 67% of those who reported moderate–high computer use, and 85% of those who identified as college students. Chi-square analyses of usability by gaming experience revealed only one statistically significant between-groups difference: those with limited gaming experience were less likely to report that it was easy to move through the virtual home compared with those with moderate to a lot of experience (χ²₍₁₎ = 4.03, P = 0.045)—47% of individuals with limited or no computer gaming experience agreed/strongly agreed with the statement that it was easy to move through the house as compared with 84% of the participants with moderate or a lot of gaming experience.

Table 4.

Percentage of Participants That “Agree” or “Strongly Agree” with Virtual Simulation Training System Usability Statements by Participant Weekly Computer Use

Usability indicators	Low use (0–2 hours per week) % (n)	Moderate–high use (>3 hours per week) % (n)	College student use, % (n)
Usability of hardware
Information boxes easy to click through	100.0 (8/8)	100.0 (15/15)	92.3 (12/13)
VSTS is easy to use	87.5 (7/8)	80.0 (12/15)	92.3 (12/13)
Easy to click on hazards	75.0 (6/8)	60.0 (9/15)	58.3 (7/12)
Keyboard controls easy to use	50.0 (4/8)	80.0 (12/15)	92.3 (12/13)
Easy to move through house	37.5 (3/8)	66.7 (10/15)	84.6 (11/13)
Usability of training
Coworkers would learn quickly	71.4 (5/7)	80.0 (12/15)	100.0 (12/12)
Hard-to-see hazards	28.6 (2/7)	9.1 (1/11)	23.1 (3/13)
Training too complicated	16.7 (2/8)	0.0 (0/15)	0.0 (0/12)
Training took too long	12.5 (1/8)	13.3 (2/15)	15.4 (2/13)
Coworkers need help	12.5 (1/8)	33.3 (5/15)	46.2 (6/13)

Table 5.

Percentage of Participants That “Agree” or “Strongly Agree” with Virtual Simulation Training System Usability Statements by Participant Gaming Experience

Usability indicator	Limited experience, % (n)	Moderate—a lot of experience, % (n)
Usability of hardware
Information boxes easy to click through	94.1 (16/17)	100.0 (19/19)
VSTS easy to use	82.4 (14/17)	89.5 (17/19)
Keyboard controls easy to use	70.6 (12/17)	84.2 (16/19)
Easy to click on hazards	68.8 (11/16)	57.9 (11/19)
Easy to move through house^*	47.1 (8/17)	84.2 (16/19)
Usability of training
Coworkers would learn quickly	81.3 (13/16)	88.9 (16/18)
Coworkers need help	35.3 (6/17)	31.6 (6/19)
Training took too long	23.5 (4/17)	5.3 (1/19)
Hard-to-see hazards	18.8 (3/16)	15.8 (3/19)
Training too complicated	5.9 (1/17)	5.3 (1/19)

P < 0.05.

Discussion

The overall goal of this study was to evaluate the efficacy, and perceived UUD of the HH-VSTS, as well as to examine the impact of computer use and gaming experience on UUD, in a sample of HHWs and health profession students. The overall results support the efficacy, and UUD of the HH-VSTS.

HH-VSTS efficacy

Accuracy of detection of hazards was high in both study groups (HH-VSTS and paper-based training), supporting the efficacy of the training, but with a linear decline in correct responses for rationale for hazards and hazard response approaches, respectively. This result makes sense because identification of hazards is a more basic skill, whereas a more sophisticated understanding is required to articulate rationale for what makes a hazard hazardous, and what to do to effectively respond to the hazard. A single exposure to the HH-VSTS may not be fully sufficient for achieving 100% proficiency with hazard rationale and response strategies. Consistent with this interpretation, participants in both study groups also correctly identified the slip/trip/lift “What makes this a hazard” and “What to do” items more often than the other two hazard types (electric/fire/burn; environmental). This may reflect more emphasis on these hazards in general HHW training and/or that they are the more common hazards in client homes (e.g., throw rugs). An implication of these results is that specific graduated training may need to occur for more subtle or complex hazards that may not be as intuitively obvious as to why these are hazardous and/or what to do about the hazards. For example, while most participants may readily identify an object in the footpath as a slip/trip hazard based on everyday experience with this type of hazard, it may not be as obvious (without training) why a space heater that is plugged into a power strip instead of a wall outlet is hazardous. Likewise, it may be more obvious that a moveable object in the path of travel should be moved from the footpath, but less obvious how mold in the bathroom should be managed. Additional practice (exposure) to the HH-VSTS and some enhanced features (e.g., role playing, hints for more subtle hazards) may support these more complex training needs.

HH-VSTS usability, usefulness, and desirability

The HH-VSTS was perceived as usable, useful, and desirable, but with some room for additional enhancements. While almost 50% of those with limited computer gaming experience agreed that it was easy to move through the simulated house, additional research is needed to understand why some individuals were able to learn virtual navigation more readily than others. Research on usability issues experienced by users attempting to navigate virtual environments has identified important components of navigation, including orientation, wayfinding, and traveling.¹⁹ To improve navigation, design of virtual environments can integrate aids such as maps, landmarks, speed control, mode of travel (walking, flying, or teleporting), and appropriate camera, or viewpoint control.^20,21 These options should be carefully vetted by targeted end users before final decisions are made regarding system integration because they have the potential to impede system usability.^20,21 For example, in the current study, two modes of travel were available: walking and teleporting. For some, walking through the virtual environment was difficult; and most readily adopted teleporting as a preferred and more efficient mode of travel between rooms.

Contrary to expectations, there was a lack of statistically reliable differences between subgroups of participants who used a computer regularly and those who did not, or between those with computer gaming experience and those without. There was one exception: participants with moderate to “a lot” of computer gaming experience were more likely to report that it was easy to move through the virtual environment. Given that individuals with gaming experience typically demonstrate superior navigation skills, including notable familiarity with the virtual environment and keyboard/mouse controls,²² this was not unexpected. Learning to use the W, A, S, and D keys or the arrow keys to walk through the virtual home and using the mouse to adjust direction of travel and to look around a room are not necessarily intuitive means of navigating the virtual environment and require some practice before this navigation becomes easy, indicating a learning curve.²³ The HH-VSTS includes a navigation training module, but a single interaction with the navigation training module may not be sufficient for achieving proficiency. There were some trends in the data that suggest those with low computer use and gaming experience had more usability issues, particularly with keyboard use, finding hazards, and length of time it took to complete the training, which can lead to added frustration with the experience.²⁴

Limitations and future research

Options to improve the limitations of the HH-VSTS in terms of accessibility and ease of use include, but are not restricted to, the addition of avatars, audio-narrated training information, and a more natural and immersive virtual reality environment allowing for more natural movements and intuitive environmental interactions. Some participants experienced difficulty seeing some of the hazards in the virtual environment, which could also be improved by these suggested enhancements of the HH-VSTS, as well as by incorporating search aids, such as hints, to assist with hazard location. Efficiency of the HH-VSTS also remains a priority and reducing the overall length of time required to complete the training could be accomplished through these types of HH-VSTS refinements.

This study was limited by the lack of standardized video recording of participants completing the HH-VSTS training and assessment modules. Additional qualitative and observational data and feedback from end users would be helpful to inform proposed refinements of the HH-VSTS, consistent with participatory design best practices. For example, further details could be gathered to develop more effective tutorials (e.g., practicing navigation). This could be done through video recordings and probing specific details about how participants use their computers (e.g., video watching, budgeting spreadsheets, e-mail) or what kinds of videogames, if any, they play (e.g., smartphone games, videogame consoles, or keyboard input computer games). Also, concepts, such as computer anxiety,²² spatial literacy,²³ computer self-efficacy, and distress associated with technology use, may be important to measure and address to more individually tailor technology for virtual simulation training experiences.²⁴ Another limitation was the lack of follow-up data to determine the effectiveness of the training on HHW health and safety outcomes. Future research will include effectiveness outcomes.

Footnotes

Acknowledgments

This original research was supported by the National Institute for Occupational Safety and Health (NIOSH) of the Centers for Disease Control and Prevention under award number R01OH010425. The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of the NIOSH.

Author Disclosure Statement

No competing financial interests exist.

References

Darragh

, Lavender

, Polivka

, et al. Gaming simulation as health and safety training for home health care workers. Clini Sim Nurs, 2016; 12:328–335.

Polivka

, Wills

, Darragh

, et al. Environmental health and safety hazards experienced by home health care providers: A room-by-room analysis. Workplace Health Saf, 2015; 63:512–522.

Wills

, Polivka

, Darragh

, et al. “Making Do” decisions: How home healthcare personnel manage their exposure to home hazards. West J Nurs Res, 2016; 38:411–426.

Gershon

RRM

, Canton

, Raveis

, et al. Household-related hazardous conditions with implications for patient safety in the home health care sector. J Pat Saf, 2008; 4:227–234.

Schoenfisch

, Lipscomb

, Phillips

. Safety of union home care aides in Washington State. Amer J Ind Med, 2017; 60:798–810.

Hittle

, Agbonifo

, Suarez

, et al. Complexity of occupational exposures for home health-care workers: Nurses vs. home health aides. J Nurs Manag, 2016; 24:1071–1079.

Capewell

, Brown

, Rockefeller

. Evaluation of a tool kit for safe patient handling in the home care setting. Am J Safe Patient Handl Mov, 2011; 1:22–29.

Hamadi

, Probst

, Khan

, et al. Home-based direct care workers: Their reported injuries and perceived training knowledge. Workplace Health Saf, 2016; 64:249–261.

McCaughey

, McGhan

, Kim

, et al. Workforce implications of injury among home health workers: Evidence from the National Home Health Aide Survey. Gerontologist, 2012; 52:493–505.

10.

Larsson

, Karlqvist

, Westerberg

, Gard

. Perceptions of health and risk management among home care workers in Sweden. Phys Ther Rev, 2013; 18:336–343.

11.

Zyda

From visual simulation to virtual reality to games. Computer, 2005; 38:25–32.

12.

Dekker

, Williams

. The use of user-centered participatory design in serious games for anxiety and depression. Games Health J, 2017; 6:327–333.

13.

Michael

, Chen

. Serious Games: Games That Educate, Train, and Inform. Boston, MA: Muska & Lipman/Premier-Trade; 2005.

14.

Dawe

, Pena

, Windsor

, et al. Systematic review of skills transfer after surgical simulation‐based training. Br J Surg, 2014; 101:1063–1076.

15.

Goolsby

MJ.

The role of computer‐assisted simulation in nurse practitioner education. J Am Acad Nurse Pract, 2001; 13:90–96.

16.

Khoo

, Merritt

, Cheok

. Designing physical and social intergenerational family entertainment. Interact Comput, 2008; 21:76–87.

17.

Verkuyl

, Atack

, Mastrilli

, Romaniuk

. Virtual gaming to develop students' pediatric nursing skills: A usability test. Nurse Educ Today, 2016; 46:81–85.

18.

Nahm

, Preece

, Resnick

, Mills

. Usability of health Web sites for older adults: A preliminary study. Computers Nurs, 2004; 22:326–334.

19.

Sebok

, Nystad

, Helgar

. Navigation in desktop virtual environments: An evaluation and recommendations for supporting usability. Virtual Real, 2004; 8:26–40.

20.

Burigat

, Chittaro

. Navigation in 3D virtual environments: Effects of user experience and location-pointing navigation aids. Int J Hum Comput Stud, 2007; 65:945–958.

21.

Sayers

Desktop virtual environments: A study of navigation and age. Interact Comput, 2004; 16:939–956.

22.

Murias

, Kwok

, Castillejo

, Liu

, Iaria

. The effects of video game use on performance in a virtual navigation task. Comput Human Behav, 2016; 58:398–406.

23.

McClymont

, Shuralyov

, Stuerzlinger

Comparison of 3D navigation interfaces. Paper presented at Virtual Environments Human-Computer Interfaces and Measurement Systems (VECIMS), 2011 IEEE International Conference on Virtual Environmental, Human-Computer Interfaces and Measurement System Proceedings. September 19-21, 2011, Ottawa, ON, CA.

24.

Santos

, Dias

, Pimentel

, et al. Head-mounted display versus desktop for 3D navigation in virtual reality: A user study. Multimed Tools Appl, 2009; 41:161.