Using Virtual Reality to Compare Design Alternatives Using Subjective and Objective Evaluation Methods

Abstract

Objective:

This study sought to develop a method that supports a more evidence-based approach to evaluating multiple design options in virtual reality (VR), combining subjective insights gathered using traditional approaches and objective feedback gathered using the VR platform. Additionally, this study sought to understand how objective data garnered from the VR platform could be used to compliment traditional evaluation strategies.

Background:

VR can be a viable research platform for supporting evidence-based design practices. Prior studies have predominately utilized experiential user feedback. While able to provide valuable subjective insights, these approaches are less effective in making objective comparisons between multiple designs alternatives.

Method:

A repeated measures study was conducted with nursing faculty. User feedback was captured through surveys, interviews, and the VR platform.

Results:

The survey, interview, and the objective VR data converged in terms of identifying the highest performing design option. Survey data showed that Room 2 performed best in terms of perceived physical access to supplies, unobstructed movement, and availability of space to accommodate additional equipment. VR data showed that participants in Room 2 had significantly higher visibility to both patient and care partners throughout their simulated interaction.

Conclusion:

Simulation-based evaluations in VR that use a combination of users’ subjective insights and objective data obtained from VR can be an effective tool for helping designers evaluate multiple design options. The use of scenario-based simulations provided a structured and clinically relevant approach to comparing three preoperative rooms, supporting a more robust assessment of users’ physical response to a simulated healthcare environment.

Keywords

virtual reality simulation-based evaluation ambulatory surgical environments prototyping healthcare design

Designing healthcare environments can be especially challenging, as they are dynamic, complex, technology-intensive environments that need to support the sometimes conflicting needs of different stakeholders including patients, care partners, and clinicians (Yousefli, Nasiri, & Moselhi, 2017). To ensure these environments support the delivery of high-quality care, it is crucial to understand how proposed designs may support the integration of new technologies, organizational processes, and users’ tasks into the built environment (Carayon et al., 2006; Holden et al., 2013). Once constructed, these buildings can potentially be around for decades, necessitating that any potential safety and quality concerns be proactively addressed during the design process rather than postconstruction.

The healthcare process often involves choosing between multiple design alternatives. Physical mock-ups have traditionally been used to garner subjective feedback from end users during the design process to support design decision-making with multiple stakeholders. However, the use of physical mock-ups to compare multiple design options, especially in early design stages, can be cost prohibitive in an era of cost containment where healthcare organizations are being challenged to do more with less (Litvak & Bisognano, 2011). In contrast, the use of virtual reality (VR) mock-ups not only provides a platform for collaboration between multiple stakeholders, but it potentially affords a more cost-effective means for visualizing, experiencing, evaluating, and analyzing multiple architectural design options preoccupancy (Bullinger, Bauer, Wenzel, & Blach, 2010; Gopinath & Messner, 2004; Heydarian et al., 2015; Majumdar, Fischer, & Schwegler, 2006; Maldovan, Messner, & Faddoul, 2006). At the present time, VR technology can support a highly immersive experience for users that simulates a high degree of visual and behavioral realism (Kuliga, Thrash, Dalton, & Hölscher, 2015), providing an opportunity to conduct experiments in virtual environments (VE) with increased ecological validity (Kort, Ijsselsteijn, Kooijman, & Schuurmans, 2003; Rebelo, Noriega, Duarte, & Soares, 2012). Prior research comparing users’ appraisals of corresponding virtual and real environments suggests that users behavioral, cognitive, and experiential responses in virtual and real environments are analogous (Heydarian et al., 2016; Kuliga, Thrash, Dalton, & Hölscher, 2015).

Prior research considering the efficacy of using VR for evaluating the design of healthcare environments suggests that VR can be leveraged to help develop a more user-centered healthcare environment for staff and patients. One study using VR found through qualitative approaches that multiple stakeholders were able to identify issues in a VE for the redesign of a pharmacy related to potential issues with privacy, discrepancies in electrical outlet locations, layout of equipment and workstations, and alterations to casework (Leicht, Abdelkarim, & Messner, 2010). Another study comparing nurse and patients’ perception of patient rooms in both virtual and real environments found that participants were able to evaluate design characteristic pertaining to color, windows, surface materials, potential disturbances, placement of furniture and other items, quality of performing nursing tasks, and aesthetics in the VE (Wahlström et al., 2010). However, they also found that participants expressed limitations in evaluating enabling/inhibiting characteristics of furniture, room size, lighting, and movement within the virtual space (Wahlström et al., 2010).

Findings from these studies suggest that VR can be a viable research platform for supporting evidence-based design practices. However, prior studies have predominately utilized VR to build consensus between multiple stakeholders and make alterations to a single design option, or compare virtual and real environments. Additionally, these studies have largely used survey and interview evaluations to garner experiential user feedback. While able to provide valuable subjective insights, these traditional methods are less effective in making objective comparisons between multiple designs alternatives.

The Systems Engineering Initiative for Patient Safety framework in healthcare suggests that effective task performance is a result of the dynamic interaction between the built environment, people, tasks, and technology (Carayon et al., 2006; Holden et al., 2013). Rubelo and colleagues (2012) suggest that both subjective and objective measures are needed to provide a holistic evaluation of a user’s experience in VR. VR provides a unique opportunity to objectively evaluate interactivity between user and environment through sensors such as motion and eye tracking that capture users real-time position and interaction with the environment. To facilitate this type of data collection strategy, Rubelo and colleagues (2012) suggest a framework that includes the use of compelling, task-based scenarios that require user interactions with objects in the VE. While scenario-based evaluations requiring user interactions with objects and the environment have been used effectively in evaluating physical mock-ups (Bayramzadeh et al., 2018), few studies have used this approach to proactively compare the performance of multiple design alternatives using VR.

VR provides a unique opportunity to objectively evaluate interactivity between user and environment through sensors such as motion and eye tracking that capture users real-time position and interaction with the environment .

Study Objectives

The purpose of this article is 2-fold. First, to develop a method that supports a more evidence-based approach to evaluating multiple design options in VR, combining subjective insights gathered using traditional approaches and objective feedback gathered using the VR platform. The second goal was to understand how the subjective insights gathered using traditional approaches agree with or contradict the objective feedback gained from the VR platform.

This study was part of a multiyear project focused on developing workspaces in ambulatory surgical environments. In-depth case studies of two ambulatory surgery centers (Joseph, Wingler, & Zamani, 2017) found that key issues related to (1) visibility between nursing staff, patient, and their care partners; (2) privacy; (3) accessibility to patient and supplies; (4) flexibility of the space to support ease of movement of people and equipment while flexing between functions; and (5) the integration of the computer workstation into the room in regard to location, mobility, and functionality was critical for supporting patient, care partners, and staff needs (Joseph et al., 2017). This work served as the foundation for the development of an assessment tool using a macro ergonomic framework to evaluate preoperative and postoperative workspaces in these environments (Wingler, Joseph, & Joshi, 2017). A graduate architecture studio project was then conducted where teams of students developed three different preoperative room designs based on design guidelines generated from the prior work outlined above, as well as evidence-based design literature, and industry best practices. Students were also asked to use the ergonomic assessment tool to evaluate their design options to ensure all key issues were addressed in their final design. The three distinct preoperative rooms generated from this design studio were used as the basis for this study.

Method

Study Design

A repeated measures study within three virtual environments (Devlin, 2017) was conducted with nursing faculty from a major Southeastern University. Nursing faculty evaluated the three distinct virtual preoperative rooms to determine which room best supported nursing work performance. A multimethod approach consisting of a survey, semistructured interviews, and objective data pertaining to user motion and head gaze captured through tracking systems built into the VR platform was utilized to capture user feedback for the following design characteristics that were identified as crucial through prior work: visibility, privacy, accessibility, flexibility, and aesthetic quality. Approval for this study was received from the university’s institutional review board.

Setting and Study Participants

The experiment was set up in a common study area within the School of Nursing. The HTC Vive headset was used to display the three virtual preoperative rooms. A 15 ft. × 15 ft. section of the room was allocated to the VR setup. Each preoperative room was able to completely fit within this space, which allowed participants to freely move throughout the room via walking, just as they would in the real environment. Participants were able to interact with the environment using two handheld controllers, and their physical position in the room was tracked using an HTC Vive tracker attached to their hip. Figures 1 and 2 show the room configuration and equipment setup for the study.

Figure 1.

Room configuration while conducting the study.

Figure 2.

Equipment setup during the study.

All nursing faculty (n = 30) at the university were invited to participate in the study. A total of 21 nursing faculty (male = 2, female = 19) participated in the experiment. Participants ranged in age (34–65 years) and clinical experience (7–41 years), with varying levels of VR experience based upon a self-reported 5-point scale rated from novice to expert (novice = 13, somewhat novice = 1, moderately familiar = 3, somewhat expert = 0, and expert = 1).

VE Development

For this study, a VE of each preoperative room was designed to afford the most natural movement, ambiance, and manipulation of components within the work system. To develop the VE, Revit models were first constructed for each preoperative room. The Revit models were exported to Unity3D, where surface textures and finishes unique to each design option were applied. Interactive capabilities were then added to the computer workstations, movable equipment, and furniture in each virtual preoperative room using Unity3D to simulate manufacturer usability specifications (see Figure 3).

Figure 3.

Floor plans and virtual environments for three distinct preoperative rooms.

Scenario Development

To evaluate the design features in each preoperative room, simulation-based evaluation framework from the Health Quality Council of Alberta (HQCA, 2016) was used to develop a standard preoperative preparation sequence for a 30-year-old male patient with a baseline medical condition of successful postcongenital heart structure repair during childhood who was having a surgical repair of a right torn rotator cuff. For this scenario, preoperative preparation required IV meds and hydration, as well as the use of oxygen. The scenario was developed in collaboration with a nursing faculty member to include the most common activities performed during the preoperative phase of surgery and was comprised of six primary tasks and their associated steps (see Table 1).

Table 1.

Scenario tasks and associated steps.

Task	Steps
Introductions (nurse, patient, and care partner)	1. Sanitize hands 2. Greet patient and care partner 3. Adjust workstation
Patient interview	1. Log into EMR 2. Review chart 3. Interview patient
Patient assessment (vital sign: blood pressure)	1. Locate blood pressure cuff 2. Apply cuff 3. Take reading 4. Check monitor 5. Record reading
Patient preparation (insert IV and apply oxygen)	1. Put on gloves 2. Locate IV supplies 3. Insert IV 4. Check IV line and rate 5. Apply oxygen 6. Remove gloves 7. Record IV procedure and oxygen application
Surgical site preparation	1. Verify surgical site in EMR and with patient 2. Locate marker 3. Mark site 4. Return marker to workstation
Transport	1. Prepare IV, oxygen, and telemetry 2. Move patient out of room

Note. EMR = electronic medical record; IV = intravenous.

As it was not possible to fully simulate all portions of these tasks due to the use of VEs, metaphors were developed that captured the intent of the action without requiring all of the component tasks to be performed. For example, participants’ hands were not visible. When putting on gloves, participants grabbed a pair of gloves from the virtual dispenser and then attached the gloves to a floating indicator near the hips, which then changed color to indicate that they were now wearing gloves. The color changed again when the gloves were removed and disposed of. Participants received an explanation of how each of these metaphors operated when instructed to perform each task.

Data Collection Process

For this study, multiple methods were utilized to garner participant’s subjective and objective evaluations of three preoperative rooms with varying types of computer workstations integrated into their work system. Figure 4 illustrates the study protocol, outlining the discrete steps in the data collection process along with their associated data collection strategies (see Figure 4). Upon arrival to the experiment location, participants were given an opportunity to read and sign the consent. Participants were then given a unique, four-digit identification number and were assisted by the investigators in placing on the VR head-mounted display (HMD) and hip tracker. The HMD and hip tracker remained on the participant until all three virtual preoperative rooms were evaluated and were removed prior to conducting the interview.

Figure 4.

Study protocol for the data collection process.

Prior to starting the virtual preoperative room evaluations, participants were first placed into a virtual test room containing the components from the virtual preoperative work system that would be used to enact the scenario (chair, IV bag, gloves, syringe, blood pressure cuff, oxygen mask, and marker). Participants were then prompted by the investigator to familiarize themselves with the components in the room and given an opportunity to practice moving the components using the handheld VR controllers. Once participants were familiar with the components and functionality of the VR platform, the evaluation of the preoperative rooms started.

In each virtual preoperative room, participants were asked to complete the same scenario and its associated series of tasks and then rate the room through a uniform survey that was verbally administered, resulting in three separate evaluation sessions in the VE. Participants were encouraged to talk aloud while performing the scenario, describing the barriers and facilitators encountered while performing the simulation. To mitigate the effect of increased familiarity with the VR platform over the course of the experiment (Pals, Steg, Donjte, Siero, & van der Zee, 2014), the order in which the preoperative rooms were presented to participants was randomized.

Following evaluation in each of the three virtual preoperative rooms, participants were removed from the VR gear. Participants then completed a survey regarding the fidelity of the VR platform and provided demographic information pertaining to their age, height, years of clinical experience, years of surgical clinical experience, and their level of VR experience. Further insight into participants’ perceptions of the preoperative rooms and the VR platform were then captured through a semistructured interview. Each study session lasted approximately 30 min.

Survey

An online, environmental survey was developed based on prior research (Joseph et al., 2017; Wingler et al., 2017) using the Qualtrics platform and was administered verbally by an investigator who entered the participants’ responses into the online platform via a laptop computer after each environmental assessment. The survey contained nine closed questions broken into the following four subsections: visibility, privacy, accessibility, and flexibility. The closed questions asked the participant to rate how well each preoperative room supported nurse work performance based on a 5-point Likert-type scale, with 1 being not well at all and 5 being extremely well. Following the closed questions, four open-ended questions provided qualitative insights into facilitators and barriers in the environment, additional features that should be added to the environment to provide further support for work performance and aesthetic quality. The survey was piloted with two school of nursing faculty to ensure accuracy and validity of the questions.

Movement through the virtual rooms

Participant physical position and visual orientation in the virtual preoperative rooms during the simulation were captured using the VR platform. Physical position was captured as a stream of data marking the location of the participant on an x, y coordinate as they moved through the VE. Visual orientation of the HMD was captured as a stream of data in the form of a vector indicating the direction of head rotation at any given moment within the VE.

Observation logs

To capture spontaneous qualitative commentary during the simulation and provide context to the findings, one investigator took notes in an observation log. The observation log served as a systematic way of capturing participant’s perceptions of the three preoperative environments as they were experiencing them.

Interviews

Semistructured interviews were conducted with the participants using photoelicitation to determine their preferred room, identify any desired features they would like to see added to the room, and gain insight into their VR experience. During the interviews, participants were encouraged to write or draw on the 8½ × 11 formatted images of each virtual preoperative room evaluated to highlight any features they preferred, needed to be added, or reconfigured. Interviews lasted between 10 and 15 min, were audio recorded, and transcribed verbatim for analysis.

Analysis

To analyze the survey data, a one-way repeated measure analysis of variance (ANOVA) with a within-subjects factor was used to analyze variations among three preoperative rooms with regard to visibility, privacy, accessibility, and flexibility. IBM SPSS Statistics 24 was used to conduct the survey analyses. R 3.4.2, R Studio 1.1.383, and the ez package were used to analyze the quantitative data collected using the VR platform.

A five-step process was used to analyze the qualitative data from the interviews, survey open-ended questions, and spontaneous commentary. First, the interview transcripts were sorted by the respondents preferred room. Next, the qualitative data from the interview, open-ended survey questions, and the observation logs from each room evaluated were compiled for each respondent by room. Then, content analysis was performed on data from all sources for each room to determine whether a design feature either supported (facilitator) or inhibited (barrier) nurse perceived work performance (Hsieh & Shannon, 2005). The facilitators and barriers were then categorized by their associated design criteria. Finally, content analysis was conducted using principles from grounded theory that incorporate using participants’ own words and phrases to inform the coding process (Corbin, Strauss, & Strauss, 2014) and explain why a design feature was perceived to either support or inhibit nurse work performance.

Findings

The approach of utilizing both subjective and objective measures garnered from the VR platform helped in the comparison of the performance of the three different design alternatives as well as in identifying features that explained why one option was preferred to the others. The three data sources used in this study—survey, interview, and VR data—all provided extremely valuable and complementary data.

Survey Data

The survey analysis revealed significant difference for four measures across the three rooms. Significant differences were found for all three accessibility questions and for one of two flexibility questions. No significant differences were found for the privacy question or for the three visibility questions.

Perception of physical access to patient care supplies was found to be significantly different among the three preoperative designs, Wilks’s λ = .515, F(2, 19) = 8.944^b, p < .002, η² = .485. Follow-up post hoc comparisons were conducted. A first paired samples analysis indicated that there was a significant difference between the scores for Room 2 (M = 4.57, SD = 0.75) and Room 1 (M = 3.76, SD = 1.14), p < .008. There was also a significant difference between Room 1 and Room 2 (M = 3.24, SD = 1.338), p < .003. This suggests that Room 2 provided better physical access to patient care supplies compared to Rooms 1 and 3.

There were also significant differences among the three design alternatives with regard to perception of physical access to all sides of patient, Wilks’s λ = .687, F(2, 19) = 4.321^b, p < .028, η² = .313. Post hoc comparisons showed a significant difference between Room 3 (M = 4.00, SD = 1.05) and Room 1 (M = 3.29, SD = 1.10), p < .037. This suggests that perception of physical access to all sides of patient was better in Room 3 to Room 1

Perception of experiencing unobstructed movement during patient care activities was significantly different among the three design alternatives, Wilks’s λ = .532, F(2, 19) = 8.356^b, p < .002, η² = .468. Room 1 (M = 2.81, SD = 0.98) was significantly different from Room 2 (M = 4.00, SD = 0.95), p < .004 and Room 3 (M = 3.71, SD = 1.007), p < .003. Thus, perceived unobstructed movement during patient care activities was lower in Room 1 compared to Rooms 2 and 3.

The perception of availability of space to accommodate flow of additional equipment was significantly different across the three design alternatives Wilks’s λ = .538, F(2, 19) = 8.144^b, p < .003, η² = .462. Post hoc analyses indicated that Room 1 (M = 3.05, SD = 1.02) was significantly different from Room 2 (M = 4.14, SD = .910), p < .004 and Room 3 (M = 3.95, SD = 0.87), p < .005. This means that perception of availability of space to accommodate flow of additional equipment was lower in Room 1compared to Rooms 2 and 3. No significant difference between other measures of visibility, privacy, accessibility, and flexibility across three preoperative rooms was found. All results are provided in Table 2.

Table 2.

Comparison of visibility, privacy, accessibility, and flexibility across the three preoperative rooms.

Design Characteristics	Room 1			Room 2			Room 3			F	P
	M (SD)			M (SD)			M (SD)
	M	SD	95% CI	M	SD	95% CI	M	SD	95%CI
Direct visibility to patient and family simultaneously from computer workstation	4.5	0.93	[4.1, 4.9]	4.4	0.97	[3.9, 4.8]	4.3	0.96	[3.9, 4.7]	.26	1
Visual access to team members outside the room	4.1	1.20	[3.6, 4.7]	4.5	0.70	[4.2, 4.8]	4.2	1.00	[3.7, 4.6]	1.10	3.610
Visual access to vital monitor from the computer workstation	4.4	0.86	[3.9, 4.7]	4.6	0.93	[4.2, 5.0]	4.6	0.67	[4.1, 4.9]	0.66	0.528
Visual privacy for patients while performing patient care activities	2.7	1.37	[2.07, 3.2]	2.9	1.37	[2.3, 3.5]	2.7	1.38	[2.0, 3.3]	0.45	0.643
Physical access to patient care supplies	3.8	1.14	[3.2, 4.3]	4.57	0.75	[4.23, 4.9]	3.24	1.34	[2.63, 3.85]	8.94	0.002*
Physical access to all sides of patient	3.3	1.10	[2.8, 3.8]	4.1	0.94	[3.67, 4.5]	4	1.05	[3.5, 4.5]	4.32	0.028*
Unobstructed movement during patient care activities	2.8	0.98	[2.4, 3.3]	4.0	0.95	[3.6, 4.4]	3.7	1.01	[3.3, 4.2]	8.36	0.002*
Adjustability of the computer workstation to support patient care activities	4.4	0.81	[4.1, 4.8]	4.2	1.09	[3.7, 4.7]	4.6	0.98	[4.1, 5.0]	0.55	0.584
Availability of space to accommodate flow of additional equipment	3.1	1.02	[2.6, 3.5]	4.1	0.91	[3.7, 4.6]	4.0	0.86	[4.0, 4.4]	8.14	0.003*

Note. CI = confidence interval.

Qualitative Data

Of the 21 participants, only 1 participant selected Room 1 as their preferred room. Twelve participants selected Room 2 as their preferred room, while 8 reported preference for Room 3. The analysis of the interview data was useful in understanding and interpreting participant preferences for the preoperative rooms. Table 3 illustrates the barriers and facilitators associated with the different design alternatives related to visibility, privacy, accessibility, flexibility, and aesthetic quality identified through the qualitative analysis.

Table 3.

Design features identified as facilitators and barriers to supporting nurse work performance and example passages.

Design Criteria	Facilitator/Barrier	Example Passage From Qualitative Data
Visibility	Monitor orientation	“…Open view of patient, view of care partner was nice.” (P21, R2) “Computer station made my back completely oriented toward the family member. Not a good view of patient.” (P8, R3) “Having my back to the patient while working with the computer.” (P13. R1)
Privacy	Privacy shield	“Shades could be pulled down for privacy [from window].” (P3, R2) “A curtain or a confidentiality shield…an extra shield is needed at the door opening.” (P6, R2) “Openness to corridor through glass is nice…curtains are needed for privacy.” (P9, R1) “I like the window to the outdoors, depends on what floor the patient is on for how it affects privacy. There needs to be some sort of curtains for privacy when the nurses are looking [from corridor]” (P19, R3)
Accessibility	Room shape	“It was diagonal. I loved it…easy access to all sides of patient, required less travel to accomplish tasks.” (P16, R2) “The shape of the room. The way the room is physically located at an angle…flow…there was more space to get around.” (P18, R2) “Going to opposite side of patient was difficult because of the narrow room.” (P19, R3)
	Colocation of supplies	“Everything in one area (workstation, supplies, IV fluid, drawers)…Everything was on the same side. Workstation was right next to the patient.” (P11, R3) “The workstation had everything on it. Everything was collocated. Things did not get in the way of each other…Moving around was easy. Drawers at the computer were good.” (P4, R2) “A lot of back and forth between patient and work spaces [computer workstation and supplies]…cumbersome.” (P11, R1)
	Supply cabinet ergonomics	“Drawers were difficult, had to move the chairs and trashcan…hit head on keyboard trying to bend down and get somethings [supplies].” (P4, P3) “More visibility of drawers, and they are too low. Behind the workstation is awkward (P1, R3)” “Drawers are pretty low. That was awkward for me.” (P20, R1)
Flexibility	Workstation mobility	“…mobility of computer, clear floor under computer, uncluttered room…easy for moving patient and equipment in and out.” (P10, R2)
Flexibility	Room size	“Great…it’s the best of the 3 scenario rooms! It is good size. Extra room at foot of bed.” (P17, R3) “Felt like this room was smaller. Not enough space for equipment.” (P12, R1)
Aesthetic quality	View to outside	“I like the bigger window, the colors, the circular vs. square.” (P3, R2) “Nice view outside, so the patient at least has a couple of options. And it feels more unlike a typical hospital room.” (P7, R2) “Really pretty…Loved the windows.” (P13, R3)
	Color and texture	“Really liked the look of this room…love the scene on the ceiling.” (P16, R2) “The colors are nice, seems like a calming relaxing room.” (P16, R3) “Can I merge these rooms together (Room 3 and Room 2)? Room 2 was the happiest.” (P20, R3) “If you put the Room 2 ceiling in Room 3 it would be the best one yet!” (P11, R3) It’s boring…bland, but that’s typical. Not that I agree with it (P16, R1)
	Amenities	“Very home-like…TV/entertainment is good.” (P17, R2) “I liked the artwork and contrast between wood and wall.” (P20, R3) I liked the patient’s entertainment screen…I did not see that until the end…liked the bright colored painting (P3, R3) “The TV for their [patient] viewing is at their level, so that they [patient] can see without strain (P21, R2)

Note. R1 = Room 1; R2 = Room 2; R3 = Room 3; P = participant.

VR Data

Of the five design characteristics (visibility, privacy, accessibility, flexibility, and aesthetic quality), visibility was the easiest to assess objectively using VR data. To objectively measure visibility, the proportion of time when the patient and the care partner were visible to the participant was calculated for each room. Two one-way repeated-measures ANOVA was conducted to determine whether the patient’s and care partner’s visibility differed significantly between the three rooms. Significant differences were observed in the visibility of the patient, F(2, 30) = 30.140, p < .001, η² = .510, and the care partner, F(2, 30) = 117.755, p < .001, η² = .807, between the three rooms. Post hoc analysis revealed that the patient was significantly more visible in Room 2 (M = 89.79%, SD = 2.63%) than Room 3 (M = 72.01%, SD = 5.00%, p < .001) or Room 1 (M = 77.37%%, SD = 4.28%, p < .001). The care partner was also significantly more visible in Room 2 (M = 67.96%, SD = 5.05%) than Room 3 (M = 24.95%, SD = 4.84%, p < .001) or Room 1 (M = 32.41%, SD = 5.54%, p < .001). Thus, patient and care partner visibility was better in Room 2 compared to the other two rooms. No significant differences were found between Room 3 and Room 1 (see Figure 5).

Figure 5.

Portion of time patient and care partner were visible to participant in each room.

Discussion

Simulation-based evaluations in VR can potentially provide a more cost-effective and engaging way for users to realistically experience future work environments and compare different design alternatives. This study breaks new ground by using simulation-based VR evaluations, and combining data from traditional survey and interview techniques with objective data obtained directly from the VR platform to compare the performance of different design alternatives.

This study breaks new ground by using simulation-based VR evaluations, and combining data from traditional survey and interview techniques with objective data obtained directly from the VR platform to compare the performance of different design alternatives .

By using interactive scenario-based simulations in VR, this study allowed users to actively interact with their work system (tools, technology, and built environment) in order to be able to rate its performance. The clinical tasks integrated into the scenario facilitated interactions between the participant and objects in the rooms such as the computer workstation, storage units, furniture, trash cans, and medical supplies, as well as between the participant and people in the room (patient and care partner). The scenario also required the participants to move around the room to complete different tasks within the workflow in a sequential order. As a result, both the objective and subjective data reflect realistic responses to tasks that might be performed in such a setting. This simulation-based approach is different from a more experiential approach where the participant might move around freely in the space, interacting with different components, though not in any particular order or with the intent of completing specific tasks. The former approach has the advantage of helping users understand how a space as part of a dynamic work system may support or inhibit their work practices within the context of simulated workflows.

The structured simulation-based evaluation approach allowed for comparison of design alternatives, and the use of objective and subjective evaluation techniques helped in selecting a design option that performed best on multiple counts. This approach and some of the metrics developed in this study have immediate applicability for other healthcare design projects. Given the increasing trend in architecture firms toward using VR for visualization of healthcare spaces, there is an extensive opportunity to utilize some of these objective and subjective metrics to support a more rigorous approach to decision-making during the design process. The survey data and the VR data allowed for numerical comparison of perceived and objective performance outcomes between the three preoperative rooms. The interview data allowed for a deeper understanding and interpretation of the findings. Additionally, the interview data provided directions for modifications to the preferred room (Room 2) for future application and testing. For example, the interview data highlighted the benefit to collocating supplies within a movable computer workstation as experienced with the boom in Room 2. Further testing of the preferred room examined the implications of using a workstation on wheels with integrated supply storage versus the proposed boom to identify the facilitators and barriers associated with each approach to technology integration.

While the survey data showed no difference in perceptions of visibility to patient and care partner between the three rooms, the VR data clearly showed that participants had a view of the patient and care partner for a higher proportion of time in Room 2 compared to the other rooms. The concerns voiced by the participants during interviews around turning their back to the care partners while in Rooms 1 and 3 and having obstructed visibility to the patient in Room 1 complement the objective data obtained from VR around visibility. Several studies have shown that direct visibility between staff and patient and staff and care partners is critical from the perspective of improving communication and improving patient satisfaction (Gharaveis, Hamilton, Pati, & Shepley, 2017). This metric could be effectively used to objectively compare different design options during the design process.

The survey findings suggested that there was no significant difference between the rooms in terms of perceived privacy for the patient. However, during the interviews, the participants indicated that the windows were a factor that impacted patient privacy. The glass walls to the corridors and the large windows to the outside in two of the rooms were noted as being problematic from the perspective of privacy. No proxy metrics related to privacy could be easily developed using VR.

There was a significant difference between the three rooms on all three survey questions related to accessibility. Insights obtained from the interviews supported this finding, suggested that the reason for the increased accessibility in Room 2 was the shape, which was perceived by participants as affording a greater degree of accessibility to all sides of the patient due to increased space around the bed to easily accommodate the flow of equipment and people. While no proxy measures related to accessibility were developed through the VR data, there may be an opportunity to develop additional VR measures related to access (e.g., number of bumps between participants and objects) since VR allows the collection of fine-grained spatial data.

Findings from the qualitative data suggest that room size could be a factor in the slight increase in perceived flexibility between Room 3 over Room 2, and Room 1 being perceived as significantly less flexible in the environmental survey. However, interestingly, Rooms 1 and 2 are similar in size. Further analysis revealed that the angle of the room contributed to participants’ perception of increased size for Room 2. The issue of flexibility seemed best addressed through surveys and interviews and a proxy measure may be hard to develop in VR to compare flexibility of different design options.

Participants were not asked to numerically rate aesthetic quality in the environmental survey. Based on the analysis of the interview data and open-ended survey responses, it was clear that participants did not like Room 1. There was a preference for Room 2, though some participants liked the aesthetics of Room 3. Future research may want to include a numerical rating of aesthetics in the survey to further understand the degree to which aesthetic quality contributes to users’ perceptions when making an environmental appraisal of perceived work performance. However, it was clear that the VR supported participants’ visualization of the aesthetic quality in the preoperative rooms. It may be possible in future studies to use eye tracking to understand specific features or characteristics of design options that are preferred aesthetically by users.

This study suggests that VR mock-ups can be effectively used with healthcare professionals to obtain feedback about the built environment that can be compared across design alternatives, especially if the tasks performed are relatively simple. For more complex tasks—for example, a surgeon simulating a surgery—a high-fidelity scenario-based virtual simulation may be extremely difficult at this time.

Limitations

The study participants were nursing faculty from a university rather than practicing perioperative nurses. Only a few of the participants had prior experience in surgical environments. However, the types of tasks that were included in the scenario did not require a high degree of specialty expertise and could be effectively performed with anyone with a background in nursing. The other study limitation is a small sample size of 21 participants. However, this is typical of many similar VR evaluation studies. While the hip tracker was able to capture spatial data related to movement, the data that were most easily accessible through the VR platform were distance traveled in the room. However, that information is not extremely useful at the scale of the patient room and is also not a sufficient proxy for accessibility, which is important in this environment. Further research is needed to understand how to more effectively use spatial data obtained from VR to calculate metrics related to micro movements between the user and the environment within the work system, resulting in interactions such as bumps, which may better help to characterize the level of accessibility to components in the work system such as patient care supplies.

Conclusions

The scenario-based evaluation provided a structured and clinically relevant approach to facilitating the experience of different preoperative rooms, such that subjective and objective responses could be effectively compared. The objective VR data were extremely valuable and allowed for numerical comparison of visibility between design alternatives. In the future, there may be an opportunity to build this and other metrics into the platform to provide some sort of score at the end of a simulation. This would provide immediate feedback to the team regarding the objective performance of different design options, creating an impetus for strategically identifying why a certain performance metric may be significantly increased or decreased as compared to other design alternatives. However, this study also shows that VR may not be able to completely address all design performance issues such as those related to privacy or aesthetics. Also, VR may not be able to clearly identify why one option performs better than another. There is a need to use traditional survey and interview methods in conjunction with VR data to make well-informed design decisions. These objective and subjective evaluation methods used in conjunction with the experience of the VR environment have the potential to engage end users in evaluating multiple design options that are still on the drawing board early in the design process, as well as support an evidence-based decision-making process.

Implications for Practice

The combination of subjective data collected through traditional evaluation strategies and objective data collected through the VR platform was complimentary and helped in providing a more robust comparison of the three preoperative rooms.

The use of scenario-based evaluations allowed users to actively interact with their work system, providing a more clinically relevant and systematic comparison of multiple design options.

VR data were extremely valuable and allowed for statistical comparison of design alternatives.

Data gathered through the VR platform may not be able to completely address all design performance characteristics, such as those related to privacy and aesthetics, or clearly identify why one design option performs better than another.

Supplemental Material

Supplemental Material, sj-pdf-1-her-10.1177_1937586719851266 - Using Virtual Reality to Compare Design Alternatives Using Subjective and Objective Evaluation Methods

Supplemental Material, sj-pdf-1-her-10.1177_1937586719851266 for Using Virtual Reality to Compare Design Alternatives Using Subjective and Objective Evaluation Methods by Deborah Wingler, Anjali Joseph, Sara Bayramzadeh and Andrew Robb in HERD: Health Environments Research & Design Journal

Footnotes

Acknowledgments

We are grateful to the College of Nursing at Clemson University, especially Dr. Susan O’Hara and Dr. Kathleen Valentine, Director of the School of Nursing, for their support in the execution of this study. We would like to thank James Dominic for his work in the development of the VR models, as well as Byron Edwards and the Spring 2017 Architecture + Health Design Studio for the three preoperative room designs. We would also like to thank the graduate students who contributed to the data collection process including James Dominic, Rutali Joshi, Herminia Machry, Roxana Jafarifiroozabadi, Michelle Eichinger, Lansing Dodd, Hannah Schultz, and Wenz Tuttle.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by a gift from Haworth through the Watt Family Innovation Center at Clemson University.

ORCID iD

Deborah Wingler, PhD, MDS-HHE, EDAC

Anjali Joseph, PhD, EDAC

Sara Bayramzadeh, PhD, MArch

References

Bayramzadeh

Joseph

Allison

Shultz

Abernathy

Group

R. O. S

. (2018). Using an integrative mock-up simulation approach for evidence-based evaluation of operating room design prototypes. Applied Ergonomics, 70, 288–299.

Bullinger

H.-J.

Bauer

Wenzel

Blach

(2010). Towards user centred design (UCD) in architecture based on immersive virtual environments. Computers in Industry, 61, 372–379.

Carayon

Hundt

A. S.

Karsh

B. T.

Gurses

A. P.

Alvarado

C. J.

Smith

Brennan

P. F.

(2006). Work system design for patient safety: The SEIPS model. BMJ Quality & Safety, 15, i50–i58.

Corbin

Strauss

A. L.

(2014). Basics of qualitative research. Thousand Oaks, CA: Sage.

Devlin

A. S.

(2017). The research experience: Planning, conducting, and reporting research. Thousand Oaks, CA: SAGE.

Gharaveis

Hamilton

D. K.

Pati

Shepley

(2017). The impact of visibility on teamwork, collaborative communication, and security in emergency departments: An exploratory study. Health Environments Research & Design Journal. doi:10.1177/1937586717735290

Gopinath

Messner

J. I.

(2004). Applying immersive virtual facility prototyping in the AEC industry. Paper presented at the Proceedings of the Conference on Construction Applications of Virtual Reality (CONVR 2004) (pp. 70–86). Lisbon, Portugal.

Health Quality Council of Alberta (HQCA). (2016). Simulation-based Mock-up Evaluation Framework. Calgary, Alberta, Canada: Health Quality Council of Alberta (HQCA). Retrieved from www.hqca.ca/humanfactors

Heydarian

Carneiro

J. P.

Gerber

Becerik-Gerber

Hayes

Wood

(2015). Immersive virtual environments versus physical built environments: A benchmarking study for building design and user-built environment explorations. Automation in Construction, 54, 116–126.

10.

Holden

R. J.

Carayon

Gurses

A. P.

Hoonakker

Hundt

A. S.

Ozok

A. A.

Rivera-Rodriguez

A. J.

(2013). SEIPS 2.0: A human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics, 56, 1669–1686.

11.

Hsieh

H.-F.

Shannon

S. E.

(2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15, 1277–1288.

12.

Joseph

Wingler

Zamani

(2017). Balancing the human touch with the need for integrating technology in ambulatory surgical environments: Barriers and facilitators to nursing work and care team interactions. Journal of Interior Design, 42, 39–65.

13.

Kort

Y. A. D.

Ijsselsteijn

W. A.

Kooijman

Schuurmans

(2003). Virtual laboratories: Comparability of real and virtual environments for environmental psychology. Presence: Teleoperators & Virtual Environments, 12, 360–373.

14.

Kuliga

S. F.

Thrash

Dalton

R. C.

Hölscher

(2015). Virtual reality as an empirical research tool—Exploring user experience in a real building and a corresponding virtual model. Computers, Environment and Urban Systems, 54, 363–375.

15.

Leicht

R. M.

Abdelkarim

Messner

J. I

. (2010). Gaining end user involvement through virtual reality mock-ups: A medical facility case study. Paper presented at the Proceedings of the CIBW, Cairo, Egypt.

16.

Litvak

Bisognano

(2011). More patients, less payment: Increasing hospital efficiency in the aftermath of health reform. Health Affairs, 30, 76–80.

17.

Majumdar

Fischer

M. A.

Schwegler

B. R

. (2006). Conceptual design review with a virtual reality mock-up model. Paper presented at the Joint International Conference on Computing and Decision Making in Civil and Building Engineering (pp. 2902–2911). Montreal, Canada.

18.

Maldovan

K. D.

Messner

J. I.

Faddoul

. (2006). Framework for reviewing mockups in an immersive environment. CONVR, 2006, 6th. Orlando, FL.

19.

Pals

Steg

Dontje

Siero

F. W.

van Der Zee

K. I.

(2014). Physical features, coherence and positive outcomes of person–environment interactions: A virtual reality study. Journal of Environmental Psychology, 40, 108–116.

20.

Rebelo

Noriega

Duarte

Soares

(2012). Using virtual reality to assess user experience. Human Factors, 54, 964–982.

21.

Wahlström

Aittala

Kotilainen

Yli-Karhu

Porkka

Nykänen

(2010). CAVE for collaborative patient room design: Analysis with end-user opinion contrasting method. Virtual Reality, 14, 197–211.

22.

Wingler

Joseph

Joshi

. (2017). An ergonomic evaluation of preoperative and postoperative workspaces in ambulatory surgery centers. In Duffy

Lightner

N., N.

, N. (Eds.), Advances in Human Factors and Ergonomics in Healthcare and Medical Devices. AHFE 2017. Advances in Intelligent Systems and Computing (vol. 590). Cham: Springer.

23.

Yousefli

Nasiri

Moselhi

(2017). Healthcare facilities maintenance management: A literature review. Journal of Facilities Management, 15, 352–375.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.32 MB