Human Factors Extended Reality Showcase

Demonstration 1: Demonstration-Based Training with the XR Vest: Improving Immersion, Engagement, and Usability. Presenters: Allison Bayro, Bryan Havens and Heejin Jeong

Demonstration-based training (DBT) is a popular and effective training method that utilizes observation and imitation (Grossman et al., 2013; Rosen et al., 2010). However, as jobs become more complex, DBT using physical equipment can become limited in mobility (Heyes, 2001). XR can provide a solution by allowing for safe and mobile DBT even in simulations of hazardous conditions. The limitations of using XR head-mounted displays (HMDs) arise with a large group of trainees and a minimal number of HMDs. One solution is to train the group one at a time, which can become costly. Another solution is to have the trainer run the simulation in XR and wear a screen for the trainees to observe, allowing the trainees to watch the trainer's movements while providing context to the actions. As a result, we developed an XR Vest that offers trainees a dual-view training experience. The vest worn by the trainer provides a first-person perspective through an integrated tablet screen and a live third-person view of the trainer's motor movements.

In demonstrating the XR Vest, we will first provide a brief overview of the problem it solves. Traditional DBT is minimally engaging and potentially hazardous, leading to low motivation and knowledge retention. Then we will highlight the unique features of the XR Vest, including the combination of a first-person perspective on an integrated tablet and a third-person perspective of the trainer's motor movements. Next, we will demonstrate the XR Vest in action, showcasing how it improves training immersion, safety, and engagement compared to traditional DBT modalities. To further emphasize the benefits of the XR Vest, we will share the results of our study that showed decreased cognitive load and improved system usability compared to other DBT methods. Finally, we will provide a hands-on demonstration for those interested in trying it out for themselves.

Demonstration 2: AR Visualization for Environmental Awareness Training. Presenters: Kelly Hale, Andrew Raij, Eric Collins, Pooja Bovard, Todd Nelling, Connor Diehl, Andrew Uhmeyer, Brian Alligood

Training operators in effective and efficient techniques across advanced diagnostics of environmental threats is challenging – particularly when detecting, collecting, processing, analyzing, and disseminating information related to radiation sources. Safety is a top priority, and training is often limited to small radiation button sources at the unit level, and joint training opportunities are still limited under OSHA regulations to protect the safety and exposure of students. Having an opportunity to utilize AR to place virtual sources within an environment and leverage virtual sensor readings relevant to planted sources provides a training opportunity that (1) visualizes the invisible source propagation based on sensor readings, (2) provides a digital footprint of the search area, and (3) provides exposure to high-risk training scenarios while maintaining student safety.

At this year’s XR showcase, we will demonstrate an AR-based training application designed to enhance situation awareness in locating and identifying environmental threats. The current application leverages the Android Tactical Assault Kit (ATAK) platform, CBRN Plug-In, and HoloLens technology to provide an AR prototype for visualizing heat crumbs within a training environment. The closed loop system can be used with virtually placed sources and/or live sources with an associated sensing technology. Information related to sensor readings are provided in a persistent dashboard, allowing students to maintain eyes on target instead of having to look down at the sensor (virtual or live). A ‘sand table’ or ‘minimap’ is available to review a third-person 3D mesh of the surrounding environment. A natural gesture interaction interface is incorporated into the HoloLens to provide quick access to create scenarios, adjust settings, and run sessions. Future iterations will incorporate performance feedback post-session for evaluation. The advantage of communication between the students’ HoloLens systems and ATAK allows instructors on the network to monitor student progress through the environment by visually tracking heat crumbs on a 2D overhead map on their android device. During the demonstration, session attendees will be able to interact with the AR application and its interface and features.

Demonstration 3: VR-Based On-Person Screening Trainer. Presenter: Aaron Jones

The Transportation Security Administration (TSA) currently uses several On-Person Screening (OPS) systems to clear passengers of threats as they enter the checkpoint, including Advanced Imaging Technology (AITs). Current AIT systems employ a Representative Human Figure avatar to provide a visual alarm indicator of the location of unexpected items. Transportation Security Officers (TSOs) interpret the location of a bounding box placed on the 2D avatar indicating the region of concern (i.e., the “referent”) to guide their pat-down of the much larger 3D passenger (i.e., the “map”). Previous work discovered that TSOs’ ability to find the location of an alarm is influenced by how the body of the passenger is represented by the avatar. The effects of the shape of the avatar are further complicated because different screening systems vary substantially in the presentation of threat information. To address the growing need to study human performance for this task, a VR methodology was developed to collect data on TSO search performance in a simulated AIT screening scenario.

The VR application presents users with a simulated alarm reading from either a traditional or experimental OPS system and asks participants to indicate where on a virtual passenger the alarm was located and then to pat-down the virtual passengers per their standard operating procedure. The VR approach was hypothesized to provide a more accurate evaluation of TSO search behavior than previously employed methods. The VR application provides detailed control over the location of alarms, the physical attributes of passengers that may influence TSO search (e.g., body size, fitting of clothes), and search area metrics (e.g., search area size, location, duration) in a quantifiable, replicable manner and could be applied to other training/testing domains at TSA or for other applications.

For the XR showcase, participants will wear a VIVE Focus 3 headset and use the standard controller to conduct mock pat-downs on simulated passengers at a checkpoint. Participants will view AIT alarm results on one of two interfaces and identify the estimated alarm location and intended pat-down area on the passenger by reaching out and “touching” the simulated passenger.

Demonstration 4: VR-based Infection Prevention and Control Training Application. Presenter: Rachel Regina

The Emory Healthcare Human Factors Lab will demonstrate a VR application for assessing and improving infection prevention and control (IPC) competency among registered nurses. The VR application includes a series of standardized adult inpatient care simulations of increasing difficulty. In each simulation, the user is given a set of tasks (e.g., wound care, stool specimen collection) to complete. Certain components of each task are virtually contaminated, providing the opportunity for cross- contamination if IPC practices (e.g., hand hygiene) are not performed or performed incorrectly. The VR-based scenarios also include common barriers to IPC practices that impose a cognitive burden on nurses (e.g., time pressure, interruptions, and a cluttered work environment). After a scenario has been completed, users receive an assessment score and feedback for improvement.

The VR application is an adaptation of standardized healthcare simulations designed to understand the cognitive processes that underlie successful and unsuccessful IPC practices and to develop methodologies for assessing and improving IPC competencies. These simulations were conducted at the Emory Center for Experiential Learning (ExCEL) Simulation Center, a dedicated simulation facility that utilizes lifelike mannequins, advanced audio/video equipment, and a team of researchers and simulation facilitators. While effective, this approach limits simulation-based research, competency assessment, and training to certain healthcare populations and settings. The VR application is designed to make simulation methods more accessible, such as at facilities without a dedicated simulation center.

Demonstration 5: Virtual Reality Aviation Illusion Trainer. Presenter: Bob Thomas

Training general aviation pilots in how to recognize and react to visual illusions and spatial disorientation while flying is difficult and generally requires large, expensive equipment. In some cases, the training can also be dangerous or nearly impossible to recreate, resulting in the inability to train pilots in physical aircraft. Adequate training in recognizing and counteracting visual illusions and spatial disorientation is often relegated to videos for many pilot training programs.

We have developed the Virtual Reality Aviation Illusion Trainer (VRAIT) as a research testbed to address this need. VRAIT software uses a VR experience to present users with examples of different aviation visual illusions pilots can experience in flight. VRAIT provides flight training programs with a safe, affordable way to expose pilots to dangerous flight conditions (i.e., visual illusions), teach them how to recognize they are seeing visual illusions and/or experiencing spatial disorientation, and train them on how to recover from a potentially dangerous situation. At the XR showcase, we will demonstrate the software’s capabilities and discuss current and future research projects involving VR/XR and spatial disorientation in pilot training.

Demonstration 6: Experience Design Tool. Presenters: Kevin Owens and Benjamin Goldberg

Simulation and XR tools and methods are being leveraged more than ever to provide immersive, interactive learning experiences for professions that operate under volatile, uncertain, complex, and ambiguous conditions. While these technologies replicate real-world task environments and enable the applied practice of knowledge, skill, and behavior requirements (i.e., competency), they often lack robust methods to objectively measure performance and learning over time. From this perspective, a data strategy is needed that translates multi-modal information captured from an XR training environment in context-rich measures of performance/ effectiveness and combines measures longitudinally to model competency acquisition over time. In support of this objective, the US Army Synthetic Training Environment Experiential Learning for Readiness (STEEL-R) science and technology project is conducting learning engineering on a data approach that leverages existing standards and open-source solutions to combine performance across several XR learning events to drive a persistent modeling strategy (Vermisso et al., 2019). To enable this, we identified a necessity to establish standards and workflows for calibrating a training scenario to measure the knowledge and skill components organizations care about.

This demonstration will illustrate an example of STEEL-R’s extensible Experience Design tool (XDT). The function of the XDT is to guide the development of XR scenarios to support competency-based experiential learning (Goldberg et al., 2021). The tools and methods are applied to the exercise design process to create elements called experience events, which are associated with discrete and measurable tasks within an XR scenario, and coordinate the stimuli, measures, and standards and criteria for measuring a targeted person or team’s performance. Experience events also stipulate the best sources of data to both support the measures of targeted task performance and to facilitate a more effective form of feedback that will enable later reflection and the actual experiential learning process to occur. These events are embedded in a larger Learning Experience Object that are reusable, human/machine- readable products that inform training plans for experiential learning. The current demonstration of the XDT prototype highlights a use case in the US Army maneuver occupational domain.

Demonstration 7: Expanding Spatial Awareness with Augmented Reality Display for Exoskeleton Users. Presenters: Mark Hollister, Hsiang-Wen Hsing, and Nathan Lau

Wearable exoskeletons aim to enhance human performance in industrial settings by combining the capabilities of machines and humans. However, due to restrictions in maneuverability and the need for increased clearance, supporting accurate spatial awareness for exoskeleton users is a significant need. We have developed a Rearview Visualization System (RVS) using an AR heads-up display and computer vision to improve the spatial awareness of exoskeleton users in warehouse or manufacturing environments to reduce the likelihood of collision with nearby pedestrians (Kulkarni et al., 2020). The RVS employs an optical camera mounted at the back of the user and MonoLoco computer vision algorithm to estimate the distance of nearby pedestrians. The pedestrian locations are presented to the user with different visualization designs on an AR headset, including a plain rear camera view, overhead radar, and ring radar. RVS expands the user’s effective field-of-view, enabling them to detect and avoid pedestrians outside their peripheral vision.

We will demonstrate the AR system and its rear-view visualization. Participants will use a HoloLens AR headset outfitted with two rear optical cameras and wear a backpack computer system. The AR displays will present users with different viewpoints that aim to enhance spatial awareness of their surroundings.

Demonstration 8: Scale Worlds. Presenters: Linfeng Wu, Karen Chen

The Next Generation Science Standards show that “Scale, Proportion, and Quantity” is a crosscutting concept across various science disciplines and can aid students in making connections across topics to construct a more robust understanding of science. Yet, research has shown that students tend to have a “compressed” conception of the size of entities, failing to realize that there are huge relative size differences among entities too small to see. VR technology is one approach to facilitate the conceptualization of size and scale enabled by the apparent “shrinking” or “growing” of learners through visual illusions of animation. It can also support embodied cognition, where the mind requires acknowledgment of its inextricable relationship to a “physical body that interacts with the world”.

The XR application we propose to demonstrate is called Scale Worlds, which is a virtual learning environment encompassing scientific entities of a wide range of sizes (e.g., COVID-19 virus 10^-7m versus Earth 10⁷m). The development of Scale Worlds was motivated by students’ common misconception of size and scale and the need to help students better understand the sizes of scientifically relevant entities. Scale Worlds is experienced as an immersive virtual environment through an HMD. During this experience, users can apparently “shrink” or “grow” as they travel to different “scale worlds,” each with a scientific entity at a size removed from others in tenfold increments. At each stage, they can simultaneously view five entities of various sizes. For example, in the “Ant World”, a learner sees a human that is of the same size, two entities (ovum, pollen) that are 10 and 100 times smaller, and two entities (acorn, bird) that are 10 and 100 times larger in the fog at a distance. A user interface in the form of a panel with numeric information is placed in front of the entities. A user points at the exponent on the numeric panel and then clicks a button to interact with the numbers and “scales” into another scale world. A series of animations accompany the transferring into another scale world to deliver the visual illusion of shrinking or growing.

Session attendees will be able to experience the VR-based simulation and its educational benefits. Scale Worlds is the centerpiece of an ongoing, multidisciplinary project with our team of collaborators in human factors engineering, design, and education.

Demonstration 9: Visualizing Farming Data in a 3D Environment: New Methods to Interrogate Agricultural Data for Fresh Insights. Presenters: Luke Sturgeon, Angelica Jasper

Data analysis touches all aspects of farming decisions, allowing users to monitor how crops respond to modern farming practices, new farm management decisions, and unique environmental events throughout the growing season. Currently, farmers use a mixture of data and heuristics to make these decisions, sourcing information anywhere from cloud-based data collected during previous passes of the farm cycle (John Deere OpsCenter) to what they’ve always done (what they were taught growing up). This leads to possibly inaccurate decisions made through a time-consuming and frustrating process. Synthesizing this information into one place in an easy-to-understand platform could improve farmers’ quality of life.

In this session, we will demonstrate a virtual environment-based application we are developing that converts farming- operations data into a virtual farm environment. The information presented in the environment is interactive and aims to improve farmers’ decision-making and awareness of the impact of farming practices on farm production outcomes. Users can explore novel connections between unique data streams to discover emergent information that users can implement to improve their farms. By taking the data analysis process from spreadsheets to an immersive environment, this process may improve farmers' ability to understand the impacts of farming practices and management decisions on farm production rates.

Demonstration 10: Virtual Online Formative Usability Testing Platform. Presenters: Janell S. Joyner, Monifa Vaughn-Cooke, PhD

Early-stage usability testing allows valuable feedback to be derived early in the design process and is commonly performed with low-fidelity physical models (e.g., made of paper or cardboard). Advancements in 3D modeling using computer-aided design (CAD) software allow designers the ability to create visuals of the final product, which could be used to assess usability. However, programming and tracking of human-product interactions required for usability testing are limited in CAD models.

To address these limitations, an online gaming platform was created to perform usability testing of CAD models in a virtual environment. The virtual online platform integrates product training and instructions for use along with 3D interactive functionality to track object interactions (button pressing, component positional changes, etc.). The platform also includes a template to integrate post-experiment subjective feedback to assess performance on critical interaction tasks.

For this session, we will demonstrate how the platform can be used to evaluate the usability of an Automatic External Defibrillator. A laptop with the online platform website loaded will be brought to the demonstration along with an external monitor for expanded viewing. Participants will have the ability to walk through the entire process starting with a brief training session and then followed by product interaction. The platform is intended to be used outside the direct supervision of an experimenter; therefore, direct engagement is not necessary. A facilitator will be onsite to answer questions on the simulation design and the associated research study.

Demonstration 11: Haptics in VR. Presenter: Nuela Chidubem Enebechi

Task performance and feedback of task performance are considered important fields in the world of Human-Computer Interaction (HCI). With the emergence of advanced technologies such as VR, it is important to understand how individuals can utilize this tool for productive task performance. There are three main senses typically engaged in a VR setting (Visual, Audio, and Tactile- haptics). Several researchers are currently exploring how human senses contribute to a positive immersive experience for users (Ghosh et al., 2018). However, there is a gap in the literature on how the frequency and intensity of haptic sensation through VR controllers affect users during task performance.

During this session, we will showcase a VR testbed application we created using Unreal engine. This application enables us to explore and gain insights into the effects and robustness of variations in haptic feedback delivered through VR controllers during task performance. In this demonstration, participants will have the opportunity to solve one or more 6x6 puzzles in the VR application. As they complete the task, they will experience different levels of haptic feedback, including variations in frequency and intensity, provided by the VR controllers. Our current focus is on utilizing this application to examine the optimum amount of haptic feedback that can be seamlessly integrated into immersive environments, ultimately enhancing the interaction between humans and computers.

Demonstration 12: Enhancing Art Gallery Visitors’ Experiences with AR Technology. Presenters: Abhraneil Dam, Yeaji Lee, Arsh Siddiqui, Wallace Santos Lages, Myounghoon Jeon

AR technologies offer the ability to increase engagement and immersion in informal learning environments by presenting users with additional visual and auditory information that aims to enhance visitors' experiences. For this session, we will demonstrate an AR-based application we are designing to enhance art gallery visitors’ experiences. The application combines visual and auditory information to augment the experience of viewing paintings in an art gallery. Our expectations are that by providing additional cues to complement the painting, visitors will engage with the paintings more, understand the artworks at a deeper level, discuss various aspects of the paintings more, and be more willing to return to art galleries in the future.

For the purposes of our demonstration, we will use an AR headset through which participants will view several paintings. The AR-based application will present users with auditory augmentations produced through the sonification of the visual image of the painting. This is done by extracting the major colors from the painting, which are then input to a Jython algorithm previously developed for a related study (Nadri et al., 2022). The algorithm produces auditory files, which are further enhanced by a musical artist based on the individual painting. The visual augmentations contain two formats, static and dynamic. Static augmentations essentially extrapolate the painting out of the canvas space, and dynamic augmentation involves layering moving elements within and outside of the painting. Users will have the ability to view all paintings at the same time, and as they approach a painting, the multimodal augmentations will begin to play.