The Use of Augmented Reality and Virtual Reality in Ergonomic Applications for Education,Aviation,and Maintenance

Method of Review

The literature review was conducted based on an autoethnographic practitioner perspective from human factors engineers working in or in the periphery of the military. The authors have experience using AR/VR applications in each of their respective fields: education, aircraft crew stations, and maintenance. Each search criterion (e.g., peer-reviewed journals, databases searched, bounding years, results, articles examined, articles rejected, articles saved, important information summaries, and relevance to the research questions) was recorded in a spreadsheet. Key word search combinations are listed in Table 1. Data/articles collected were examined for relevant information related to AR/VR applications that address ergonomic concerns (see research questions outlined in Table 1). Initial examination of articles began with the abstract, and further review was conducted based on the following two criteria: (1) Is the article relevant to the research questions (regarding education, pilot crew station, and maintenance AR/VR use outlined in Table 1)? (2) Does the article either include experimental data that show benefit of an AR/VR application to an ergonomic use or provide a literature review of other AR/VR topics where summaries of experimental data were discussed? This work was not intended to be a consolidation of all literature findings on the topic but merely a prospectus of themed articles showing how the capability of AR/VR uses can be spread across three seemingly unrelated industries that all value ergonomics considerations.

OPEN IN VIEWER

Table 1.

Key Word Search Combinations

Research Question	Key Words
How are AR/VR applications being effectively used to address ergonomic issues in or by education (e.g., training techniques)?	AR/VR human factors training AR/VR education training AR/VR ergonomics training college AR/VR ergonomics training AR/VR college ergonomics AR/VR human factors college training
How are AR/VR applications being effectively used to address ergonomic issues in aircraft crew stations (e.g., mental workload)?	AR/VR cockpit AR/VR aviation AR obstacle detection AR flight decks Synthetic vision cockpit Smart window cockpit Transparent cockpit aircraft
How are AR/VR applications being effectively used in maintenance (e.g., reduction in errors)?	AR/VR maintenance applications AR/VR remote assistance AR/VR error reduction AR/VR inspection

Note. AR/VR = augmented reality and virtual reality.

Selected source information was summarized to provide an overarching concept of AR/VR use within the specified industry and to illustrate how ergonomic issues are being addressed using AR/VR. The systematic review utilized database searches (i.e., EBSCO, Google Scholar, and Professional Journals) based on the key word list related to each research question. Abstracts were reviewed for each article, until irrelevant or repeating information was presented in the search results. Irrelevant articles typically only contained key word matches, but not in the context of the research question. Articles with abstracts that provided potential information for the research questions were cataloged in a spreadsheet and thoroughly reviewed for more information.

While numerous search results were returned, many results only made brief references to the specific subject matter of interest or only included a citation with the relevant key words. These articles were rejected for inclusion in this literature review due to the minimal amount of specific information relevant to the topics. Thirty-eight articles were initially cataloged based on abstract reviews, and 30 articles were used to address the proposed research topics. The remaining eight articles did not provide unique research that was applicable to the research questions. Additional articles identified through cross-referencing or recommendations in the review process were used to provide either supporting information to the research questions or as information related to general concepts of AR/VR. However, the use of and potential issues with AR/VR in industry is such a broad topic that making this article an exhaustive literature review is not realistic in scope; therefore, the authors chose to highlight research breadth of AR/VR applications across the industries encompassed within their own practitioner engineering and ergonomic expertise.

AR/VR Ergonomic Applications in Education

Don Quixote, the titular character of Miguel de Cervantes Saavedra’s most famous novel, quipped, “Thou hast seen nothing yet.” He could be easily speaking of the future possibilities of AR/VR within education. While the literature results reveal limited human factors education applications of AR/VR, the material alludes to vast applications of AR/VR within the realm of training and education. The articles provide insight into considerations of both the AR/VR design features as well as the users of the technology to generate a successful application. Most of the selected articles related to educational applications related to medical training for patient recovery, while a few articles related to specific on-the-job training or applications in a K–12 educational environment.

The medical environment training articles focused on patient performance in specific areas such as gait improvement, balance, or upper limb movement. Prior research developed a total of six low-cost, nonimmersive VR scenarios to augment balance training for people with multiple sclerosis. In developing the VR scenarios, the researchers gained feedback from both the patients and the therapists using a participatory design framework to determine what information needed to be included in the scenario and the specifications for the therapy exercises. The study suggests that VR training helps in successful patient rehabilitation because of the ability to support high repetitions and varieties of tasks (Khalil et al., 2019). Research to investigate the use of VR training for physical therapy for upper limb motor function after a stroke did not find a significant difference between VR therapy and conventional therapy, which suggests that VR therapy could potentially be used if the number of patients increases to a point where conventional therapy personnel or financial resources cannot support the entire workload. These researchers suggest that VR therapy may be better for those who are not severely impaired by an injury (Schuster-Amft et al., 2018). These types of therapy can be conducted in a facility environment with immersive surroundings (i.e., large screens) through VR headsets. Patients can utilize the VE to accomplish tasks or interact with virtual objects to encourage use of a specific rehabilitation technique. VR scenarios can also be recorded and analyzed to track patient progress while even finding their way into telehealth communications.

The use of VEs for safety training primarily concentrated on the areas of risk identification as compared with risk evaluation, risk response planning, and risk monitoring and control. Using a VE to improve construction and engineering safety and training resulted in an improved ability for users to recognize hazards while also proving to be an engaging method for learning. The researchers found that previous construction and engineering safety training VE research focused little on addressing human factors issues such as cognitive distraction, which was found to often contribute to accidents (Kassem et al., 2017).

In training research related to analyzing three different virtual training environments for wildfire firefighting, researchers determined that users favored the Oculus rift head-mounted display (HMD) for immersion. There was no statistical significance in situational awareness ratings between the HMD and the 270° cylindrical projection system (SimPit). However, these two environments were statistically significant with higher situation awareness when compared with using the high-definition TV environment (Clifford et al., 2018).

The research associated specifically with the traditional educational settings provided specific considerations related to the use of AR for both students and teachers. AR applications that have a good design can create positive and productive educational settings that require a low cognitive load to students (Küçük et al., 2014). Researchers investigated the use of AR to support traditional K–12 curriculum materials. They suggested three guidelines when considering using AR for educational purposes, which include using both quantitative and qualitative metrics to gain an overview of the effects and situation of use in the educational environment, considering the long-term impact of the technology on the student’s learning, and considering the level of involvement and demands placed on the teacher in order to use the technology (Da Silva et al., 2016).

There are also concerns related to the use of VR technologies in education that could manifest within other industries as well. Research in the pros and cons of simulator use in medical training found that lower fidelity simulators did not fully mimic human systems, may foster negative learning and shortcuts to cheat the simulation, encourage cavalier behavior, and create other technical difficulties related to remote information technology support (Krishnan et al., 2017).

At the time of this writing, the world is facing a pandemic due to COVID-19 (coronavirus disease 2019). AR/VR applications in both telemedicine and education could potentially see an increase in use and influx of resources to assist in evaluating patients at a distance and providing instruction and training on treatments like rehabilitation exercises. Additionally, as states expand online learning opportunities, AR/VR could play a major role in student education from kindergarten to graduate school. A pilot study by Chen et al. (2019) demonstrates that if students were to move from a PowerPoint lecture-driven model to a more interactive AR model, their learning outcomes may be comparable but their test taking self-efficacy could increase implying a greater interest in the learning experience. Likewise, long-term separation from schools could encourage the use of AR/VR-related technologies in households for pursuing education.

AR/VR Ergonomic Applications in Aircraft Crew Station Design

Beyond having “the need for speed,” Maverick had the need for AR. The 1986 movie Top Gun introduced many movie goers to their first vision of true AR in the form of the head-up-displays of the F-14 Tomcat. Aviation continues to rely heavily on AR/VR during flight, training, and system design.

Several ergonomic- and human factors–related issues are consistently present in crew station-related applications. When evaluating and developing crew station applications, practitioners must be aware of ergonomic constructs such as mental workload, situation awareness, information management, effective training, and general usability (Salas et al., 2010). The incorporation of AR/VR capabilities can help address these issues in the aircraft environment.

The development of VR flight simulators provides researchers with a method to evaluate ergonomic issues related to both usability and training. A VR flight simulator allows for rapid development of flight deck or cockpit designs, simulated out-the-window views, and pilot interaction with minimal hardware requirements (Oberhauser et al., 2016). These VR prototypes allow researchers to collect pilot feedback, perform cognitive walkthroughs related to task performance, and evaluate general interactions with control and display layouts as well as other novel systems that may not be easily integrated into rigid simulators. Additionally, VR simulators provide a method to address some levels of flight training by providing a virtual representation of the cockpit that reduces the need for advanced hardware simulators (Oberhauser et al., 2018). Previous researchers outlined a six-step methodology to design VR training simulators that required considering and identifying which actual system components to present in the VR environment as well as understanding the requirements for the training (Yuviler-Gavish et al., 2013).

Interactive virtual cockpits and flight decks are also in development (Figure 1). VR concepts can be implemented to allow pilots to manage information and usability by adjusting cockpit displays (e.g., size, location, information) in real-time within the VE (Comerford & Johnson, 2007). For example, pilots could view and manipulate a virtual map that allows for interaction in 3D space, independent of traditional display sizes.

OPEN IN VIEWER

Figure 1.

ForeFlight’s synthetic vision (ipadpilotnews.com).

Researchers are investigating the use of AR/VR applications for aircraft operations in degraded visual environments to prevent spatial disorientation, increase situational awareness, and reduce the likelihood of accidents (Prinzel & Kramer, 2009). AR technologies augment cockpit displays during flight phases (e.g., taxi, enroute, approach) with symbols, guidance, and obstacle representations to guide pilots through difficult visual environments (Foyle et al., 2005). VR and synthetic vision systems are being used as solutions to degraded visual flight environments by replacing or augmenting the displays with situational awareness cues and synthetic terrain maps for reducing spatial disorientation (Figure 2). HMDs utilize VR and camera systems to allow pilots to “look through” the environment and cockpit to observe the synthetic environment without environmental or structural occlusions (Doehler et al., 2015; Prinzel et al., 2006).

OPEN IN VIEWER

Figure 2.

FlyQ augmented reality (ipadpilotnews.com).

Similar research is being conducted in the AR/VR domains for the use of underwater vehicles and human divers. The underwater environment faces challenges similar to aviation when considering degraded visual environments. Advanced sensors and data fusion are used to provide a clear picture of the environment to equipment users and divers in harsh underwater conditions. High-resolution sensor images can be used as virtual representations of terrain to enhance user situational awareness by providing detailed range and feature data for underwater operations (Tremori et al., 2019). Virtual and augmented information can be displayed through a diver’s mask or user control station, utilizing the same concepts as an aircraft HMD or crew station display.

Reductions in pilot workload can also be attributed to AR applications. The use of AR technology to provide pilots information related to tasks or emergency procedures by either overlaying the critical areas of the cockpit or by providing direct instruction can improve response times and reduce the complexity of performing tasks during high workload instances (Tran et al., 2018).

AR overlays are also being used to display areas of interest and environmental elements through applications such as the Seattle Avionics FlyQ Insight App (Figure 3). The use of these applications can allow for the overlay of information such as routes, airfields, waypoints, or restricted operating zones. Pilots can use these applications to assist in projecting the AR entities into the environment, rather than on a 2D map. These AR projections could potentially improve situational awareness and reduce heads-down time in the cockpit. Future work will likely need to focus on integration to windscreens or HMDs to reduce the need for an external device (i.e., tablet or phone) to project the imagery.

OPEN IN VIEWER

Figure 3.

FlyQ augmented reality with moving map (ipadpilotnews.com).

Several challenges still exist in the implementation of AR/VR in crew station design. Utilizing AR/VR technologies in live flight requires adherence to strict performance and safety standards. System reliability and fidelity are critical for in-flight use. Designers are also identifying the usability issues of working with complex AR/VR environments and hardware (e.g., latency, accuracy, and modal interactions) that must be resolved prior to flight-ready implementations (Ernst et al., 2016).

AR/VR Ergonomic Applications in Maintenance

In the 2008 movie Iron Man, audiences are introduced to the ultimate maintenance manual, J.A.R.V.I.S. (Just A Rather Very Intelligent System). Not only does J.A.R.V.I.S. have artificial intelligence that allows him to interact with and respond to Tony Stark but J.A.R.V.I.S. also provides an AR display that assists Tony’s efforts to quickly diagnose and fix the Iron Man suit. While practical holographic projects remain fantasy, the increased prevalence of handheld data devices such as tablets and smartphones provide a method to enhance the functionality of paper manuals through AR (Manuri et al., 2019). The interactive manuals provide step-by-step tutorials with easily comprehended visuals aligned to the system requiring maintenance (Henderson et al., 2011). Additionally, AR provides less experienced repairmen access to the knowledge of more experienced maintainers (Jayaweera et al., 2017).

From Star Wars to Minority Report and James Bond to Ender’s Game, Hollywood has promised the benefits of AR. But just because an idea looks great on the big screen does not mean that it translates well to real-life situations. Ensuring the utility of AR in a maintenance environment requires scientific rigor. One study compared the effectiveness of instructions supplied through AR presented on a handheld tablet versus an annotated picture of the workpiece. The study found a significant decrease in task completion time and number of errors for the AR arrangement. Furthermore, a posttask NASA TLX (NASA Task Load Index) showed a preference toward the AR device (Polvi et al., 2018). A second study explored the use of AR overlays to assist maintenance personnel in tracing faulty pipes through a nine-story building eventually leading to the correct shutoff valve. The augmented overlay allowed novices to successfully complete the task (Diao & Shih, 2019). In addition, a study conducted with U.S. Marine Corps maintainers found that AR instructions projected through an HMD improved task completion time while reducing head and neck movements, leading to possible reductions in musculoskeletal strain (Henderson et al., 2011). However, AR devices can present a hindrance to job performance when used by an expert. A group of European researchers found that the task completion timed suffered when experts introduced assistive technology into a task that was “too easy” (Aschenbrenner et al., 2019).

“You can lead a horse to water . . .” Likewise, providing the maintainer with the proper task instruction does not guarantee a correctly completed task. Since maintenance tasks are dependent on completing all the steps in the proper sequence, ensuring proper task completion represents a high priority. A team of researchers developed a context-aware checklist. The checklist determines the portion of the equipment that the operator is engaged with and provides a checklist that the user must confirm completed before the next step is revealed (Perla et al., 2017). A second team enlisted computer vision to compare the finished state of reality with the programmed finished state in a virtual scene. If the two scenes match, then the maintainer is prompted to complete the next task in the sequence (Manuri et al., 2019). While both approaches show promise for enhancing maintainer performance, several technical improvements, such as system component recognition fidelity, system response time, and unexpected input error handling, are being addressed prior to conducting user performance testing.

While using the maintenance manual should allow a person to fix most issues that arise, sometimes the maintainer needs to “phone a friend.” In cases where a help desk discussion with a subject matter expert is needed, AR provides the opportunity to improve communication. A group of scientists demonstrated a collaborative AR system using simple video capture. The operator’s HMD captures a visual reference of the scene. This image was projected on the experts’ workstation. The expert physically gestured on the projected image. The gestures were captured by a second video camera and projected as overlays in the operator’s HMD (Zenati-Henda et al., 2014). Alternatively, a second group of researchers placed the expert in a VR environment and allowed the expert to communicate through placement of interactive iconology into the operator’s HMD or through digitized hands projected in the HMD (De Pace et al., 2019).

Although AR technology has come a long way since Ivan Sutherland invented the HMD (Billinghurst & Kato, 2002), technological obstacles prevent the widespread adoption of AR in maintenance scenarios. The introduction of Google Glass and the Microsoft Hololens demonstrate that industry is working on the information delivery system. However, the greater hindrance stems from content creation. Maintenance manuals are typically exhaustive tomes filled with myriad diagrams and detailed written task descriptions. The paper versions or even digital versions require immense efforts to convert to AR and often do not translate well to an AR environment. German researchers have approached the content issue by developing a framework to transfer the written manuals directly into the augmented environment by associating a digital version of the written text with a particular AR interaction point. The concept is to input as much content as possible followed by the effort to redesign the content specifically for AR (Engelke et al., 2015). On the other hand, Japanese researchers are developing a methodology of generating the 3D content directly in the AR environment. This approach allows for instructional content that maximizes the capabilities of AR visuals from the beginning (Plopski et al., 2018). Finally, Singaporean scientists have developed a method that focuses on the context of the application by linking the manual data to the scene spatially (Zhu et al., 2015).

To be clear, the scope of the capabilities of AR are quite broad – as these maintenance examples allude. Some AR systems are considered a limited view in that they utilize a small screen, such as a smartphone (i.e., Web or mobile AR). These systems still augment the environment by providing additional context when the phone is held over specific parts of the environment (Qiao et al., 2019). The user is not fully immersed, but the added context or instructional information is still available as needed (Obermair et al., 2020). HMD systems such as the wearable Google Glass and related products do fully immerse the wearer such that users still can view the real world, but additional information or visuals are layered on top of the real environment, again for that added context that can be task specific or informational (Patibandla et al., 2020). There are noted benefits and issues with both ends of the AR spectrum in that the limited view not only allows the user to only engage AR when the situation warrants but also requires the user to often hold a device and to know where to aim the device – and that AR information layering is even present somewhere in the environment. The fully immersive HMD solutions allow the user to remain hands free and to stay immersed in the real world without necessarily requiring preexisting knowledge that contextual information exists in the environment. However, staying immersed in AR has its disadvantages from having to use a facial wearable to the potential for overloading the user with information that may be considered unnecessary for any user above a novice or trainee.

AR has garnered much of the attention for maintenance given its ability to place the user in the workspace versus fully inside a simulation, the authors would like to note that VR is also used in maintenance for training purposes. AR enables the user to learn on the job by providing those additional pieces of overlayed information while doing the task; VR provides training scenarios for jobs that may be too difficult (e.g., marine communication equipment; Bingchan et al., 2018), too dangerous (e.g., nuclear fusion environments; Rastogi & Srivastava, 2019), or too costly to make a mistake (e.g., aviation; Eschen et al., 2018) to learn on-site. Another good use for VR in the maintenance space is during facility development. For example, the development of a fusion plant proves to be both challenging and dangerous; therefore, VR was studied to preemptively understand maintenance and assembly challenges during facility design (Louison et al., 2017).

Commonalities Across Industries

The review of AR/VR applications across industries reveals commonalities of technology uses related to assisting users to perform tasks in both training and live scenarios. When new equipment designs or training are being introduced, AR/VR incorporation can be a valuable tool to refine the user experience, interface, and training environment. The literature review found that various industries have successfully used AR/VR to address ergonomic concerns related to workload, situational awareness, information processing, training, and usability. Industries that are concerned with ergonomic issues related to product designs, environments, or training should evaluate the tradeoffs and potential benefits of using AR/VR systems to enhance overall user performance. A common theme found throughout the literature includes determining the best visualizations and information to include within the AR/VR medium. When determining an AR/VR solution, identifying the goals of the application is an important first step. When full sensory information is required, AR/VR applications would be unable to accommodate users. When partial sensory information and a realistic contextual environment is required, AR technologies provide a platform to maintain external sensory information in the environment, while still taking advantage of interactions with virtual objects (Steffen et al., 2019). Finally, VR applications remove the majority of environmental context, but provide the most flexibility for the development of creative and novel environments (Steffen et al., 2019).

Future Applications

As AR/VR technologies and methods are refined, the use among industries should increase. The lower cost of implementing AR/VR techniques for training and user assistance has the potential to greatly influence user efficiency, skill, and safety. For example, applications in high-risk environments (e.g., medical, aviation) can help provide valuable training and guidance to reduce user errors and promote good technique. However, poor implementations of AR/VR could hinder progress in both user learning and efficient job completion. Future research could investigate other areas of AR/VR implementation within the associated industries to reduce user risk. Additionally, future research could examine additional methods to improve communication or multiple user AR/VR interaction in high-risk environments. Future research should focus on refining the user experience with AR/VR technologies to ensure that user compatibility (e.g., AR/VR sickness, poor gesture mapping, user comfort) does not hinder the immersion experience. Standardizing the AR/VR experience across industries and products can help reduce the volatility in product designs and improve the overall user experience (Ritsos et al., 2011). Successfully adapting AR/VR technologies in any workplace (either those discussed herein or the many environments not covered in this prospectus) depends heavily on positive user experiences. Early research has found that workers see the benefits of AR/VR applications in industry use, but still have concerns related to the ergonomics, safety, and full integration to daily tasks (Aromaa & Kaasinen, 2018). Recent research has found that a number of factors (e.g., hardware displays, demographics, and user experiences) can contribute to discomfort (e.g., VR sickness) for users in AR/VR environments (Chang et al., 2020). Continued improvement of the AR/VR user experience will be vital for user acceptance. As standards continue to develop and user applications extend beyond the laboratory, more information and best practices will be available for successful technology integrations.

Dr. Bruce Knerr, noted Army Research Institute psychologist, mentioned in a prior interview that since VEs develop and transmit a lot of information, it is important to understand how to share that information in a way that is beneficial and usable (Hamilton & Holmquist, 2005). Future research could investigate methods to effectively translate immense amounts of maintenance manual data into AR/VR. Users must be careful when adopting AR/VR applications to ensure that their use is beneficial and does not negatively affect performance. Research is ongoing in the VR domain to identify the effects of high- and low-fidelity immersion experiences and their contribution to meeting learning objectives (Makransky et al., 2019). Identifying the appropriate levels of immersion is an important consideration to ensure users are not distracted by extraneous information in the VE. When developing VR training, design guidelines have been proposed across industries for interface creation, including the consideration of virtual avatars, simple graphics, explanatory peripheral cues (both visually and tangible), and user considerations when making the graphics and instructions (Sethumadhavan, 2013). Research has been conducted across industries (e.g., maintenance, education, and flight) to study effective uses of AR/VR training methods within industries (Gavish et al., 2011; O’Neil & Andrews, 2000; Osterlund & Lawrence, 2012; Vergara et al., 2017). Each industry application is unique; however, many researchers agree that using AR/VR applications can allow for familiarization of interactive use cases and product representations for initial and sustainment training through cost-effective visualization (O’Neil & Andrews, 2000). Adopters should consider all aspects of AR/VR hardware and software requirements to ensure that the available solutions meet the intended goals of the application. In conclusion, the autoethnographic frame prospectus taken by practitioners working directly in or within the periphery of the military who focus on three distinctly different research areas was intended to demonstrate how AR/VR can universally aid in ergonomic improvements.