Virtual Environments for Training: Human Factors Limitations,Challenges,and Opportunities

Ensuring Rapid Employment of Effective Virtual Environments For Training

James Pharmer, Ph.D.

The world has clearly become enamored with virtual reality systems in environments such as gaming, performance support, operational system design, and training devices. Many of those systems are highly effective at supporting the tasks for which they were developed. However, the novelty, flashiness, and apparent inexpensiveness of virtual environments has sometimes led to less than optimal applications in the training domain. Obviously, the most effective medium for training some tasks is not always a virtual environment. For example, while there may be virtual applications for training basic reading and math skills, years of effectively training these skills by other means attests to the fact that these applications are not essential elements to training those skills to proficiency.

The United States military has embraced the instructional system design process as the best approach for developing effective training systems. A number of instructions (e.g., Chief of Naval Operations Instruction 1500.76D) and handbooks (e.g., Department of Defense Handbook series 29612) exist to guide instructional systems designers through the processes of 1) examining the current training environment and situation, 2) developing requirements to identify the tasks being trained, as well as the performance standards, and 3) determining knowledge, skill competencies, and associated levels of proficiency required. Through this front end analysis process, instructional system designers can allocate specific training media to the training requirements as defined. This allocation process depends heavily on understanding the results of the scientific studies that have empirically determined what media are most effective at training certain knowledge and skills and in what contexts. The process also relies on an understanding of the cost associated with applying these training media into the current training environment and situation.

Unfortunately, as a burgeoning technology, applications of virtual training are outpacing the empirical research necessary to understand how best to apply them and the development of the instructions and guidance necessary to direct instructional designers toward the optimal employment of virtual environments. Moreover, the high demand for rapid deployment of low-cost training solutions has constrained the time available to thoroughly research, understand, and promulgate information on when virtual environments should be applied to a training system and when it should not. Further, data are lacking on the training effectiveness of these systems once deployed.

Overcoming this dilemma will require comprehensive solutions from the training and human performance community, the instructional systems design community, the training systems acquisition community, and the military operational community. More rapid methods for executing and transitioning human performance and training research findings, better methods for communicating these finding to the instructional design community, more rapid and efficient methods for conducting instructional analyses, faster methods of delivering effective advanced training capabilities, and better feedback mechanisms on the effectiveness of these systems from the operational community are all parts of the solution. The purpose of this discussion is to identify some of the barriers that must be addressed by each of these communities to best employ effective virtual environment training systems to the fleet and to offer solutions to these challenges.

Optimizing Extended Reality Impact in an Overall Training Strategy

Benjamin Goldberg, Ph.D.

How does Extended Reality (XR) fit into a larger training strategy and how can we optimize its impact on overall skill acquisition? In practice, XR can support longitudinal skill acquisition by providing learners with opportunities to engage in immersive, interactive, and personalized learning experiences that mimic real-world tasks under real-world conditions. This can be particularly beneficial for acquiring complex skills that requires a lot of sets and reps to build an automated performance response (Fenwick, 2000). This is of relevance to training communities that operate under unsafe and hard to replicate conditions, such as military operations, aviation, maintenance, medical and surgical procedures, language learning, etc. XR can provide learners with a safe and controlled environment where they can practice and experiment without the risk of real-life consequences. But how do we know a specific instance of training is actually influencing knowledge and skill development, and how can we objectively analyze a sequence of experiences to better predict skill proficiency and competency over time.

An emerging discipline centered on this very problem statement is learning engineering (Goodell & Kolodner, 2022). Learning engineering seeks to apply engineering principles and methods to the design and development of future learning environments by directly leveraging data, analytics, and immersive technology. From a technology standpoint, this involves studying the utility of XR tools and methods within a broader learning ecosystem, and establishing empirically based principles and heuristics on how XR interaction best supports organizational learning objectives. This involves not only designing functional scenarios and interactions that mimic real operational settings to drive a behavioral response, but also instrumenting those experiences with data-driven techniques to monitor performance, evaluate task outcomes, and to integrate that learning experience into a persistently oriented skill acquisition curve. Establishing these techniques can support further advanced adaptive functions that utilize performance metrics to manage coaching, scaffolding and task complexity.

To achieve this vision, a data strategy is required to guide the design and implementation of scenarios that adhere to learning engineering principles. A strategy must be adopted around standards and requirements used to develop training solutions that plug into a larger training strategy. In this instance, a training environment must be engineered to perform the following functions: (1) capture multi-modal data generated during the interaction of a scenario; (2) build context-rich measures from that multi-modal data that provides insights on process and procedure; (3) creating assessment functions on those context-specific measures that are used to infer performance (i.e., grading someone as below-standard, at-standard, or above-standard on a task step or outcome); (4) establishing coaching and pedagogical services that act on assessment state information to personalize interaction; and (5) using data standards and models to capture assessment data across time and environments to measure longitudinal performance trends against a set of competencies (Hernandez et al., 2022).

With these capabilities in mind, the data strategy must emphasize two critical functions. This involves a structured scenario creation process grounded in experiential learning theory and experience design. This will enforce a scenario to be deconstructed into its context dependent task and sub-task parts, and then populating a task model with the metrics and assessments that will be monitored when a task is active and being executed. This will enable an evidence-centered approach to performance modeling, with careful attention to the task triggers and injects that will drive a learner’s response, both at the individual and team level. The second function is enforcing system interoperability through a standard data reporting specification (Walcutt & Schatz, 2019). This will allow the persistent storage of performance data across multiple training resources, and to build mathematical reasoning on how to infer skill progression when a learner engages with several environments across a designated period of time. There are existing standards to facilitate this second function, like the eXperience API (xAPI), which is an open-source specification for learning technology that enables the tracking of learning experiences across multiple platforms, systems, and devices. However, how xAPI supports experiential learning and evidence capture requires research, especially when considering learning data from different sources, such as simulations, games, and social learning platforms (Hernandez et al., 2022).

While the effectiveness of XR tools and methods to support modern training and education needs continues to improve, there are still open questions on best practice use and their impact on long-term skill acquisition and sustainment goals. With a data strategy to collect context rich evidence across numerous training experiences, big data and AI techniques can assist in developing policies and agents that accelerate skill acquisition once we know what tools and training approaches actually make a difference.

Artificial Intelligence to Enhance Virtual Environments Training Fidelity

Brian Stensrud, Ph.D.

Virtual environments offer a range of benefits for both trainers and trainees. These simulations provide a safe and controlled environment where trainees can learn and practice skills that can be applied to a variety of applications, from pilot training to medical simulations. However, virtual environments are limited in complex training environments for a variety of reasons.

One of the primary limitations of virtual environments for training is the lack of robust supporting and opposing roles necessary to support the student. Without individual role players who can participate simultaneously, these roles are either omitted entirely or emulated by simple scripts and behavior models. As a result, trainees may not receive a realistic and engaging experience that prepares them for complex real-world scenarios. Another limitation of virtual environments for training is the difficulty of generating unique scenarios that cover the desired training objectives. As a result, trainers may over-rely on a few "known-good" scenarios. This approach can be limiting and may not fully prepare trainees for the complexities of the real world. Virtual training scenarios are also typically very linear and brittle, unable to adapt to meet the needs of the student in real-time. This lack of adaptability can be a significant challenge in complex training environments where trainees may encounter unexpected or unusual situations. Without the ability to adapt and adjust, trainees may struggle to develop the skills they need to be successful in these environments.

Artificial intelligence (AI) has the potential to address each of these limitations, resulting in a significantly more effective and meaningful experience for both students and instructors. Intelligent, interactive virtual AI role players can be integrated to serve as teammates, opponents, and other supporting functions in scenarios. By providing realistic and engaging experiences, trainees can develop the skills they need to be successful in complex training environments without the need for significant human support. Additionally, AI-powered generation tools can automatically generate any number of training scenarios given relevant input constraints, such as specific training requirements. This can enable trainers to create more unique and engaging scenarios that cover a wide range of training objectives, without relying on a few "known-good" scenarios. This can help trainees to prepare for a wider range of scenarios and challenges they may face in their future careers. Finally, integrated performance analysis can drive intelligent adaptation and personalization of training scenarios, allowing the training system to meet students where they are in their training. By analyzing student performance, an embedded AI capability can dynamically adjust the scenarios to better suit their needs and provide a more personalized and effective training experience.

To fully realize the potential of artificial intelligence in virtual environments for training, there is a need for significant investment in both AI technologies and the virtual environments themselves. Note that AI is not a singular topic or technology, but represents a broad spectrum of paradigms and algorithms that must be specifically designed and instantiated towards a specific purpose. As such, significant investment in research and development is needed to ensure that the appropriate AI systems and algorithms are properly designed and integrated to support effective training. One key challenge we have observed is the difficulty of acquiring the necessary knowledge to drive the decision-making of these AI systems. Hand-encoded knowledge is too expensive and time-consuming to develop for many applications, while auto-generated knowledge is prone to errors and mistrust. This challenge requires a focus on developing techniques and algorithms for acquiring (or auto-generating) and representing knowledge that can be effectively utilized by the AI systems. A related challenge is the development and extension of existing virtual environments that they are compatible with AI systems. As an example, virtual environments must be able to support the activity of interactive agents, have dynamic object manipulation, support reasoning and planning, and provide real-time performance metrics.

High-fidelity environments and next-generation VR devices are not sufficient to support robust, effective virtual training for our Warfighters. We additionally need to invest in research and development to support the effective integration of knowledge-rich AI systems into these environments, for the reasons cited above. This investment will require collaboration between AI researchers, domain experts, and virtual environment developers to ensure that the AI systems are effectively designed and integrated to support effective training. Additionally, the development of integrated AI systems must be supported by continued investment in the virtual environments themselves, to ensure that the environments are robust and effective for supporting integrated AI capabilities.

Using Reinforcement and Imitation Learning to Train Intelligent Agents in Virtual Worlds

Joe Mercado, Ph.D.

Effective use of Machine Learning (ML) to train Intelligent Agents (IA) that can be used to train humans in a virtual environment is highly dependent on exposing those intelligent agents to realistic situations so we can develop adequate ML benchmarks for these tasks.

However, as these grand challenges are “solved,” new challenges materialize, requiring the creation of new environments, which is often time-intensive and requires specialized domain knowledge. Furthermore, developing hand-crafted IAs in these environments is equally challenging, time-consuming, and often intractable, resulting in resources spent with sub-optimal results.

ML enables developers to create IAs that learn their behavior from data instead of hard-coding behaviors based on domain-specific expertise. ML is used in many domains, such as computer vision, gaming, and military tactics/decision-making. For example, ML can be applied to military tactics in several interesting ways, with the most promising approach is developing software to have agency via a simulation. For example, when examining a military strategy game, such as Starcraft II, an input, such as a game screen or some state the agent has access to, is given to an agent and mapped to an output, such as the best decision for the agent to make. In that case, we can build an ML pipeline that can train an intelligent agent to take those inputs and learn a mapping function (producing the correct decision) to “win” the game.

The two most prominent methods to teach agents are Reinforcement Learning (RL) and Imitation Learning (IL). In RL, we train an agent to generate a policy or set of instructions by taking in observations and performing actions. This policy is then optimized to maximize the cumulative reward that the agent receives while taking action in an environment. IL uses demonstrations that are recorded observation/action pairs, formally called the expert trajectory, to train the agent. IL is most useful when the reward function is difficult to define by hand or when it is simpler to show the agent the appropriate behavior. This panel will discuss the benefits of using ML to train IAs, highlighting use cases for ML-Agents throughout the DoD.

Considerations for User-Centered Design Research with Augmented Reality Training Applications

Barbara Chaparro, Ph.D.

The use of Augmented Reality (AR) technology in training has been growing as more devices and software development tools become available, affordable, and easier to use. The appeal of AR over VR training is that users can visualize digital assets in real-world environments, thus maintaining situation awareness of their physical surroundings and workplace. Training AR use cases include remote assistance, simulation, maintenance tasks, and step-by-step guidance which have been used in many industry domains and government settings.

Typical user-centered design process of a training application involves assessing user needs and iterative design, prototyping and testing with representative target audiences. For AR training, needs analysis must consider the applicability of AR devices to a diverse set of users to insure an inclusive solution. In addition, the range of environments (lighting, temperature, noise) in which it will be used must be considered. Finally, needs analysis must assess if AR technology is appropriate for the training task at hand. It is unfortunately common for organizations to seek a one-technology-fits-all solution without considering the unique needs of the training exercise and goals.

Prototyping AR solutions, especially in low-fidelity, can be difficult due to the lack of easy-to-use no-code tools. Sketching in 3D is more difficult than 2D and not a widely shared skill among designers. Successful solutions may use multiple tools and creativity for early prototype testing. VR may be a possible testbed to demonstrate concepts early in the design phase or alternatively, simple slide deck images superimposed with digital assets.

Usability assessment of training applications is critical first step before conducting research to demonstrate its effectiveness. Neglecting to do this presents the problem of running a confounded study in which the failure to find an effect of AR training could be attributed to either training ineffectiveness and/or usability issues. This is especially important in AR applications in which the device, application, and environment all contribute to the overall training experience.

There are two primary methods of usability evaluation in user-centered design – heuristic evaluation and usability testing. Heuristic evaluation is an expert-driven methodology that assesses a product based on key heuristics, or best practice guidelines. While many heuristics have been developed for computer software, mobile device interfaces, video games, and other digital interfaces, few guidelines exist for augmented reality interfaces, as the technology is still in its infancy stages. As a result, there is a lack of consistency in the design of interfaces across devices and even within a single device. Recently, a validated AR heuristic checklist was developed to assess AR devices (HMD and mobile) and applications (Derby and Chaparro, 2022). This checklist allows a researcher to evaluate an AR device based on ease of set-up and use, comfort to the user, and maintainability over time. It also provides heuristics related to an AR application’s ease of learning and recall, interaction methods, collaboration effectiveness, and integration of the virtual and real-world environment. This heuristic checklist provides guidance to many stakeholders interested in AR adoption including developers, practitioners, and purchasing decision-makers.

A second method of assessment of an AR training experience is usability testing, which involves observation of actual users interacting with AR devices and applications. Usability testing with AR technology presents unique challenges to the researcher not experienced with traditional computer-based or mobile interfaces. For example, testing with a HMD requires device training and calibration, availability of sufficient physical testing space, enhanced think-aloud protocol, screen sharing between the participant and the researcher, and testing across platforms, as appropriate. All of these adaptations require extra time and resources.

Following proper user-centered design principles is critical to the development of effective AR training solutions. We need to recognize and embrace the challenges and complexities of establishing user needs, iterative design and testing across a range of devices and persistent technological advances.

Virtual Environment Training in Aviation

Meredith Carroll, Ph.D.

Aviation has a long history of utilizing virtual environments, typically referred to as simulators, in training. The industry continues to utilize environments ranging from flight training devices (FTDs) that replicate the aircraft instruments and controls without visual representation of the external environment, to medium fidelity environments which also represent the visuals outside the windscreen with either non motion or 3-axis motion, to extremely high full-motion simulators which closely replicate the experience on an actual aircraft. The aviation industry is one of the few industries that allows experience in a virtual environment to count towards certification hours in the aircraft. Pilots-in-training typically encounter this full range of environments throughout the course of their training, typically starting with low-fidelity FTDs to learn basic procedural skills, to higher fidelity simulators to practice assessing situations and making decisions, to skill consolidation in full motion simulators. There has been a range of research studies examining which levels of fidelity are appropriate for various skills. In fact, the Office of Naval Research had a large program, the VIRtual Training and Environments (VIRTE) program, partially devoted towards answering these questions of how and when to utilize various levels of fidelity. The research findings are varied, but some studies point to the need for only functional fidelity to train procedural skills, and physical fidelity only becoming important when it is necessary to train spatial tasks or tasks which are very physical in nature, and that very high-fidelity environments are best used during the skill consolidation phase as they can become distracting (Champney et al. 2017).

Historically, high fidelity came at a very high cost and therefore it was utilized sparingly and only when necessary. One byproduct of the prolific gaming industry is virtual reality (VR) headsets, such as the Oculus Quest, that are both inexpensive and easy to use, making high fidelity a much more accessible option. Another byproduct is fairly accurate models of a range of aircraft and environments that allow someone with an inexpensive, commercial-off-the-shelf flight simulation package such as X-Plane or Microsoft Flight Sim to practice flying in various small aircraft, large airliners and even military aircraft, in and out of airports and locations around the world. These advancements have resulted in emerging training uses for virtual environments in aviation. As a result of this, there are a range of new and different ways that training in aviation is being re-imagined. For example, VR headsets are being used to train maintenance technicians on physical tasks that require spatial knowledge and skills, and there is research to suggest there is training benefit (Carmody and Carroll, 2022). Research is also being conducted to examine the effectiveness of using VR headsets to induce spatial disorientation to train spatial disorientation recognition and recovery skills (Satter, 2023). The high quality and accuracy of the simulation models have also facilitated the use of these models for familiarization training. For instance, a pilot could utilize a virtual version of the aircraft panel to start to prepare for transition to another aircraft. This combined with VR can facilitate the development of proprioceptive skills associated with scanning for various pieces of information around the flight deck and in the overhead panel. Accurate models of airports and runways can be used to familiarize pilots with airport runway configurations and markings and how these look under various lighting conditions. Students can “walk” the runways, giving them a first person view and practice following instructions provided by ATC.

There is currently limited empirical research into the effectiveness of these new instructional approaches. However, there is a great deal of anecdotal evidence from instructors regarding learning gains. As virtual environment training in aviation continues to evolve, there is a need to continue to evaluate effectiveness, as well as ensure that training is occurring at the appropriate fidelity levels. Empirical research should examine questions such as at what point in the training cycle, and for what skills, is VR-based training most effective?