Augmented Reality Improves Procedural Work on an International Space Station Science Instrument

Abstract

Objective:

The purpose of the current study was to determine whether an augmented reality instruction method would result in faster task completion times, lower mental workload, and fewer errors for simple tasks in an operational setting.

Background:

Prior research on procedural work that directly compared augmented reality instructions to traditional instruction methods (e.g., paper) showed that augmented reality instructions can enhance procedural work, but this was not true for simple tasks in an operational setting.

Method:

Participants completed simple procedural tasks on spaceflight hardware using an augmented reality instruction method and a paper instruction method.

Results:

Our results showed that the augmented reality instruction method resulted in faster task completion times and lower levels of mental and temporal demand compared with paper instructions. When participants used the augmented reality instruction method before the paper instruction method, there was a transfer of training that improved a subsequent procedure using the paper instruction method.

Conclusion:

An off-the-shelf augmented reality head-mounted display (HoloLens) can enhance procedural work for simple tasks in an operational setting.

Application:

The ability of augmented reality to enhance procedural work for simple tasks in an operational setting can help in reducing costs and mitigating risks that could ultimately lead to accidents and critical failures.

Keywords

augmented reality procedural work spaceflight

Introduction

Immersive technology, or mixed reality (MR) environments, can be traced back to Baum’s (1901) short story titled The Master Key, in which he depicts a pair of glasses that augments the vision of the wearer to display the inherent character of other people. Later, in an essay titled “The Ultimate Display,” Sutherland (1965) describes visual displays that can make solid objects transparent, and he would later go on to invent the first immersive head-mounted display (HMD; Sutherland, 1968). As shown in Figure 1, the reality-virtuality continuum by Milgram, Takemura, Utsumi, and Kishino (1994) denotes that MR environments exist between two opposing extremes of completely real and completely virtual environments. In these MR environments, real-world objects and virtual objects are presented together in the same display. Perhaps the most popular example of an MR environment is that of augmented reality (AR), in which virtual objects are overlaid onto a real-world environment (Milgram & Colquhoun, 1999).

Figure 1.

The reality-virtuality continuum.

Procedural Work

One area of research where AR has received considerable attention involves procedural work, in which an established sequence of activities is performed to accomplish a particular outcome. Procedural tasks are common for installation, assembly, and maintenance work where regulations often dictate that procedural instructions are presented on a paper medium (Neumann & Majoros, 1998; Ong, Yuan, & Nee, 2008). These instructions are often complex and contain large portions of redundant information in multiple forms such as descriptive text, photos, or diagrams (Hou, Wang, Bernold, & Love, 2013; Okamoto & Nishihara, 2016; Ong et al., 2008). As a result, operators devote a significant amount of time to studying paper instructions (Henderson & Feiner, 2009). In some cases, operators may spend up to 45% of their work-shift searching and reading procedural instructions (Ott, 1995).

Operator costs

Procedural instructions are often physically separated from the equipment they accompany, which inherently divides the work into two distinct spaces: information and physical (Neumann & Majoros, 1998). The information space is primarily cognitive and requires the operator to search, read, comprehend, interpret, and translate instructions to the physical space. The physical space is primarily kinesthetic and requires the operator to retrieve information to inspect, align, adjust, and manipulate the equipment. Because they are detached, there is often a large gap between the information space and the physical space, which creates what Kim and Dey (2009) refer to as cognitive distance. Both the information space and the physical space individually require attentional and cognitive resources, but considered together, the added cognitive distance imposes additional demands on operators because of the need to integrate information between the two spaces.

To integrate information, operators must switch their attention between the two spaces. Such voluntary switching of attention is associated with increased demands on working memory (Arrington & Logan, 2004; Monsell, 2003). Moreover, research showed that moving the head to visually scan separate spaces increases the effort of information access (Wickens, Helleberg, Goh, Xu, & Horrey, 2001).

AR benefits

Augmented reality has the potential to mitigate the cost of cognitive distance by enhancing the operator’s perception of and interaction with the physical space by directly overlaying the information space onto the environment. Such superimposing of virtual objects onto the physical space creates an immersive experience that provides operators with an intuitive way to integrate and access information (Ong et al., 2008). By integrating the information space and the physical space, operators are not required to make additional attention shifts, and valuable time can be saved if operators are continuously attending to a single space (Yeh, Wickens, & Seagull, 1998). Using paper instructions, operators are required to attend to the information space, interpret the intended action to be performed, shift their attention to the physical space, visually scan for the task location, and then translate the information from the information space to a physical action in the physical space. Using AR, instructions take the form of virtual text or diagrams that are superimposed onto the physical space.

Augmented reality cues can be used to draw the operator’s attention to essential information in the physical space. Placing a virtual indicator such as a bounding box around the target location provides immediate information to the operator about where the task is performed in the physical space. These cues eliminate the need for operators to shift their attention between the information and physical spaces and can eliminate the need to scan the physical space. When the physical space is sufficiently large such that a full view of the task is impossible, AR cues can be provided to orient the operator to the task location(s). For example, if the operator is not currently looking at a task location, a virtual object such as an arrow can be placed in the operator’s current field of view to guide him or her to the task location. Such cueing effects in AR have been shown to reduce visual scanning (Biocca, Tang, Owen, & Xiao, 2006; Bonanni, Lee, & Selker, 2005).

AR procedures

There has been a concerted effort to investigate how AR might enhance procedural work (for surveys, see Ong et al., 2008; Palmarini, Erkoyuncu, Roy, & Torabmostaedi, 2018; Wang, Ong, & Nee, 2016). We focused on the subset of the literature that compared AR instructions to traditional instructions (e.g., paper), reported quantitative results, and used HMDs. We limited our discussion to HMDs because they present visual information naturally and free the hands of the operator to perform manual tasks.

In one such study, participants performed a procedural task assembling toy blocks (Tang, Owen, Biocca, & Mou, 2003). Task performance and mental workload were measured across four between-subjects conditions of print instruction, computer assisted instruction (CAI) shown on a nearby LCD monitor (CAI-LCD), CAI shown statically in an HMD (CAI-HMD), and AR. Task performance was defined as time to completion and error. Mental workload was measured using the National Aeronautics and Space Administration Task Load Index (NASA-TLX; Hart & Staveland, 1988). Results showed that AR instructions resulted in faster completion times and fewer errors compared with print instructions but not compared with CAI-LCD or CAI-HMD. Additionally, results of the NASA-TLX showed that the AR instructions resulted in the lowest mental workload.

Wiedenmaier, Oehme, Schmidt, and Luczak (2003) investigated whether AR assembly instructions could reduce completion time for assembling a car door compared with paper instructions or a tutorial from an expert. Participants were randomly assigned to a condition and completed assembly tasks such as mounting a window regulator, wiring cables, and attaching wire clips to the inner door panel. Results showed that overall task completion time was fastest for the expert tutorial condition, followed by the AR condition and then the paper instruction condition. When looking at task complexity, results showed that AR can reduce task completion time compared with paper instructions but only when the task was complex (e.g., mounting a window regulator). For tasks that were simple and repetitive, there was no significant difference between the AR and the paper instruction conditions (e.g., attaching wire clips).

Another study evaluated whether AR instructions resulted in better performance compared with traditional instructions for simple routine maintenance tasks (e.g., flip a switch, remove a bolt, connect a cable) inside an armored vehicle turret (Henderson & Feiner, 2009). A baseline condition was created by adapting images and text from a technical manual and was presented on an LCD monitor. To control for any effects of wearing the HMD in the AR condition, participants also experienced a third condition (head-up display; HUD) where they wore the same HMD as the AR condition but instead saw the same content as the LCD condition; AR cues were not present. Results showed that AR resulted in faster task localization compared with the LCD and HUD conditions. That is, participants visually located the next task area quicker using AR instructions. However, there were no differences in task completion times among the three conditions. The authors attributed this to the simplicity of the task, stating that once the task was initiated, participants did not require information provided by the displays.

Henderson and Feiner (2011) also conducted a comparison between AR instructions and traditional instructions for a more complex task of assembling a motor vehicle combustion chamber. A baseline condition was created from printed materials by presenting images on an LCD screen. The rationale was that this format resembles technical manuals used by the U.S. Department of Defense. Results showed that AR instructions resulted in faster task completion times and fewer errors and was rated by participants as more intuitive compared with traditional instructions.

In summary, previous studies examined whether using AR instructions can improve performance for procedural work. Task completion times were faster when participants used AR instructions compared with paper instructions; this was true for assembling toy blocks (Tang et al., 2003), assembling a motor vehicle combustion chamber (Henderson & Feiner, 2011), and complex steps in assembling a car door (Wiedenmaier et al., 2003). However, this was not true for simple tasks (e.g., attaching wire clips) in assembling a car door (Wiedenmaier et al., 2003) or simple tasks (e.g., flipping switches) inside an armored vehicle turret (Henderson & Feiner, 2009). Moreover, AR instructions resulted in lower levels of mental workload compared with paper instructions (Tang et al., 2003).

Finally, it should be noted that large-scale solutions are still unavailable because they often require expensive and custom hardware (Sanna, Manuri, Lamberti, Paravati, & Pezzolla, 2015). In addition, AR solutions for procedural work have faced a variety of ergonomic issues such as the bulk and weight of the HMD, displays that required secondary computing power, or tripping hazards caused by tethered devices (Baird, 1999; Starner et al., 1997).

Spaceflight Operations

Procedural work in spaceflight operations presents many challenges that would benefit from AR. Between the years 2010 and 2017, the direct cost of NASA spaceflight accidents was nearly $500 million (NASA Annual Mishap Reports, n.d.). Moreover, half of NASA accidents from 1996 to 2005 were caused by human error (Chandler, 2007), and a significant proportion of these accidents resulted from incorrect procedure execution (Barshi & Dempsey, 2016). These accidents can result in damage to or destruction of public or private property, mission failure, loss of public confidence, or loss of human life. Improving procedural work for NASA spaceflight operations can help prevent accidents and reduce the associated costs that result from human error.

On both the ground and the International Space Station (ISS), standardized procedures provide the necessary instructions to perform assembly, service, and maintenance tasks on flight instruments. These instructions are complex and contain written text, diagrams, and pictures. They are also detached from the science equipment, which creates cognitive distance for operators by requiring them to shift their attention from procedural instructions to the task area. This attentional shifting can lead to increased visual scanning and increased workload. For these reasons, AR has the potential to improve performance for procedural work in NASA spaceflight operations.

Current Study

The purpose of the current study was to determine whether an AR instruction method would result in faster task completion times, lower mental workload, and fewer errors for simple tasks in an operational setting using a spaceflight science instrument. We displayed AR instructions using the Microsoft HoloLens (“Microsoft,” n.d.), which is an off-the-shelf mixed-reality HMD. The HoloLens is less susceptible to the same ergonomic issues as previous AR systems because it is a self-contained unit that does not require physical wires or external processing. Also, because the device is available for purchase from Microsoft, researchers and practitioners are not required to develop an expensive and custom HMD in-house. For these reasons, we believe the HoloLens is one of the best candidates available to examine AR in procedural work.

Because AR integrates the information and physical spaces, we hypothesized that participants would be significantly faster to complete the task, report significantly lower mental workload, and make significantly fewer errors when using AR instructions compared with paper instructions.

Method

Participants

Twenty employees from the NASA Jet Propulsion Laboratory (7 female) participated in the study. Ages ranged from 22 to 52 years (M = 31.55, SD = 8.59). All participants reported normal or corrected visual acuity and normal motor control and were naïve to the experimental hypotheses. This research complied with the American Psychological Association Code of Ethics and was approved by the Institutional Review Board at the California Institute of Technology. Informed consent was obtained from each participant.

Apparatus and Display

The study was conducted using a physical mockup of the Cold Atom Laboratory (CAL) Science Instrument (The Coolest Spot, n.d.). CAL was designed to study ultra-cold quantum gases in the microgravity environment of the ISS. We created a notional procedure based on the installation and maintenance procedures for this instrument. During the procedure, participants were tasked with searching for named cables to make a connection to a port on the device (mate) or disconnect the cable from a port on the device (demate) using a paper instruction method and an AR instruction method.

Augmented reality cues were generated using in-house software programmed with Unity3D (Unity, n.d.) and were displayed using the Microsoft HoloLens. The HoloLens is a wireless, self-contained HMD that has an integrated Intel 1 GHz CPU/GPU, 2 GB of RAM, and a holographic processing unit with 1 GB of RAM. Displays were presented at 30 frames per second in 1268 × 720 pixel resolution for each eye with an estimated total field of view of 30° × 17.5°.

Design

There were 30 total trials from three critical areas on the instrument: CPU, electrical filter, and science module. Half of the trials were randomly assigned as mate trials and the other half as demate trials. Next, we created two sets of trials to ensure that participants did not complete the same trials in each instruction method. We randomly selected half of the trials for each set with the following constraints: an equal number of trials from each of the three areas, an equal number of connection types, and an equal number of mate and demate trials. Finally, we randomized the order of the 15 trials in each set. Participants were randomly assigned to one of two instruction method orders that determined which instruction method participants used first. Instruction method order and trial order were counterbalanced, resulting in four unique run orders.

Instruction Methods

The baseline paper instruction method was based on an actual procedural paper instruction for CAL. As shown in Figure 2, the baseline paper instructions were presented as a checklist with the minimal task-relevant information required to carry out the procedure: cable name, task type, port name, and area in which the port was located. In this way, we created a simplified version of the traditional procedural instructions that would allow us to make a direct experimental comparison to the AR instruction method that displayed the same information, albeit in a different presentation medium. During the experimental task, this paper checklist was attached to a clipboard, and participants were permitted to hold the clipboard or place it on a table within reach. Participants were required to indicate that a trial was complete by marking the box in the completed column before moving on to the next trial.

Figure 2.

Example paper instruction method that participants used to perform the task.

In the AR instruction method, participants viewed a virtual checklist that was fixed in space at the center of the instrument and did not obstruct critical task areas. Similar to the paper checklist, the cable name, task type, and port name were displayed. Analogous to the task area column on the paper checklist, three types of AR cues were provided to assist the participant in locating the task area (Figure 3). First, we displayed a virtual bounding box (only the corners were shown) around the target port that was intended to provide a quick reference point for where the task occurred; the bounding box was automatically reoriented toward the HoloLens’s gaze direction in real time. Second, we provided a virtual nametag with the name of the target port, which was slightly offset vertically above the port. Third, when the target port was not in the user’s field of view, a small virtual chevron appeared in the display that directed participants’ attention to the target port. Participants traversed the checklist by issuing voice commands to the HoloLens device. A checkmark was placed in the completed column after the voice command was issued to move to the next trial.

Figure 3.

Example augmented reality instruction method that participants used to perform the task. The left panel displays the virtual checklist affixed to the physical device. The right panel displays a target port highlighted with the virtual nametag and attention directors. Also note the physical labels that are spatially bound to the physical location of the cable.

Procedure

Prior to testing, participants completed a short demographic questionnaire that included general questions about their previous experience with AR and VR technology. Afterward, participants were given information about the science instrument used for the experiment and a general overview of the task.

Participants completed one block of 15 unique trials using each instruction method. Prior to completing each block of trials, participants were given specific instructions about the instruction method and performed three practice trials to become familiar with the task and the instruction method (including both mate and demate trials and covering all three task locations). They were allowed to ask questions and repeat the practice trials until they felt comfortable performing the task. Participants were instructed that they would not be told where the cables were located and that they would need to find them to complete the task. Every cable that was used in the experiment was clearly labeled and matched the name that was given to participants in the checklist. Additionally, participants were instructed to complete the task as quickly and accurately as possible. Before performing the task with the AR instruction method, participants completed the HoloLens calibration, which ensured that the HoloLens display was calibrated optimally for each individual participant (i.e., by calculating their interpupillary distance and providing visual feedback to ensure the participant could see the entire display); they were also trained on the voice commands.

Immediately after each block, participants completed paper versions of the NASA-TLX and the System Usability Scale (SUS; Brooke, 1996) to assess subjective workload and usability of the instruction method. After both blocks were completed, participants filled out a short questionnaire designed to elicit feedback about their perceived pros and cons of each instruction method and were asked which instruction method they preferred.

Individual trial completion times were recorded by the experimenter using a stopwatch, and a trial was considered to be complete when the participant had checked the completed box in the paper instruction method or finished issuing the voice command in the AR instruction method. Attempts to perform a trial using an incorrect cable or target port were considered errors. The experiment was completed in approximately 90 min.

Results

Data Preprocessing

Seven out of the 20 participants experienced a software malfunction with the voice commands in which they were taken to a previous trial that they had already completed and required them to issue the voice command several more times to reach the current trial. In total, this malfunction occurred for 5% (18/355) of the voice commands issued. These instances were not “misses” by the voice recognition software, but they did result in unexpected functionality. We suspect that this can be attributed to the procedural software and feasibly remedied in a future version. As we did not expect this malfunction to occur and because our intent was to compare the paper instruction method to a fully functional AR instruction method, we assumed a “best-case” level of technology and subtracted the extra time required to navigate through steps that had already been completed.

Trial Completion Time

Trial completion times (in seconds) were subjected to a 2 (task type: mate, demate) × 2 (trial order: A, B) × 2 (instruction method: AR, paper) × 2 (instruction method order: AR first, paper first) mixed analysis of variance (ANOVA) with instruction method order as a between-groups factor. Results showed that demate trials (M = 25.47, SD = 8.32) were significantly faster than mate trials (M = 46.33, SD = 10.98), as indicated by a main effect of task type, F(1, 18) = 212.34, p < .001, η_p² = .92. This result was not surprising because participants were required to visually search for and locate cables for mate trials without any search assistance, whereas demate trials, having the cables already attached to the port, included either an approximate port location (paper) or virtual attention directors directly indicating the target port (AR). Participants were also significantly faster when using the AR instruction method (M = 32.12, SD = 12.29) compared with the paper instruction method (M = 39.68, SD = 15.25), as indicated by a main effect of instruction method, F(1, 18) = 24.45, p < .001, η_p² = .58. However, this main effect will be interpreted in light of the two-way interaction that follows. No other main effects were significant.

There was a significant two-way interaction between instruction method and instruction method order, F(1, 18) = 19.68, p < .001, η_p² = .52. Follow-up tests were performed using separate one-way ANOVAs and Tukey’s honestly significant difference (HSD) tests to determine the effect of instruction method at each level of instruction method order. As displayed in Figure 4, mean trial completion time was significantly faster for the AR instruction method compared with the paper instruction method but only when participants completed the paper instruction method first. In other words, when participants performed the task using the paper instruction method first, they were significantly faster in the AR instruction method compared with the paper instruction method. When participants performed the task using the AR instruction method first, there was no significant difference in mean trial completion time between the two instruction methods. These results suggest that using an AR instruction method first resulted in a transfer of training that improved subsequent procedure execution using a paper instruction method.

Figure 4.

The effect of instruction method on mean trial completion time (seconds) for each instruction method order. Error bars represent ± 1 standard error of the mean.

Errors

We performed a paired-samples t test on the number of errors committed to determine if there was a significant difference between the two instruction methods. Results showed the number of errors committed were not significantly different, t(19) = −1.710, p = .104, d = 0.38. In general, participants committed few errors when using both the AR instruction method (four total) and the paper instruction method (eight total). We attribute this to the relative simplicity of the task.

NASA-TLX

We omitted the weighting procedure of the NASA-TLX to obtain Raw TLX (RTLX) scores (Byers, Bittner, & Hill, 1989; Hendy, Hamilton, & Landry, 1993; Nygren, 1991) from each of the six subscales that could range from 0 (low workload) to 100 (high workload). To determine differences in workload between the two instruction methods, we performed a paired-samples t test on the RTLX scores for each of the individual subscales (Hart, 2006). As shown in Figure 5, mental workload was significantly lower for the AR instruction method, t(19) = −2.996, p = .007, d = 0.67, and temporal workload was significantly lower for the AR instruction method, t(19) = −2.511, p = .021, d = 0.56. These results are consistent with prior research that showed AR instructions can result in lower mental workload (Tang et al., 2003).

Figure 5.

The effect of instruction method on mean NASA-RTLX score for each of the six subscales. Error bars represent ± 1 standard error of the mean.

SUS

We calculated the results of the SUS to obtain an overall score that could range from 0 (not usable) to 100 (usable). We performed a paired-samples t test on the SUS scores to determine if there was a difference in usability between the two instruction methods. Results showed that there was no significant difference between the AR instruction method (M = 81.50, SD = 13.39) and the paper instruction method (M = 78.00, SD = 13.27), t(19) = 1.294, p = .211, d = 0.29. Although we did expect that the AR instruction method would be rated higher in usability, this finding is not surprising because of the simplicity of the paper instruction method.

Questionnaires

In the posttest questionnaire, participants reported three limitations about using the HoloLens. The first limitation is related to the weight of the device; the HoloLens weighs about 1.3 lbs. In the current study, 5 out of 20 participants reported that the device felt heavy. This is important because participants wore the HoloLens, on average, for no more than 12 to 13 min (about 4 to 5 min to calibrate, 8 min to complete the task). This may present challenges for operators that are performing work for extended periods of time or using the device multiple times throughout the workday.

The second limitation is related to the device display. Seven out of 20 participants reported that they were unable to see the entire checklist (most likely due to the display’s small field of view) or the entire instrument (most likely due to the HMD obstructing the participant’s vision). This may present challenges for operators that are required to be in close physical proximity to the task or when the task requires a larger field of view.

The third limitation is related to a conflict of visual perception. Seven out of 20 participants reported that there were times when they struggled to shift their visual focus between the AR cues and the physical instrument or reported slight eye strain. For example, participants focused on the bounding box to locate the target port but then doublechecked that the location was correct by viewing the physical label on the device. Unfortunately, the HoloLens’s display has a fixed optical focal distance at a depth of 2 m (Windows Mixed Reality, 2018), even though the virtual cues can be placed at arbitrary depth. This vergence-accommodation conflict is a well-known phenomenon for AR (and HMDs in general; Drascic & Milgram, 1996) and has been shown to cause visual fatigue (Hoffman, Girshick, Akeley, & Banks, 2008). Moreover, this may present challenges for operators who are performing procedural work. For example, if operators are using AR cues or models to align or install physical hardware, switching visual focus between the virtual object and the physical may increase visual fatigue, which in turn may decrease performance. However, despite these limitations, 14 out of 20 participants reported that they preferred using the AR instruction method.

Discussion

The purpose of the current study was to determine whether an AR instruction method would result in faster task completion times, lower mental workload, and fewer errors for simple tasks in an operational setting using a spaceflight science instrument compared with a paper instruction method. Our results showed that the AR instruction method resulted in faster task completion times (32.12 s) compared with paper (39.68 s). More importantly, when participants used the AR instruction method before the paper instruction method, there was a transfer of training that improved a subsequent procedure using the paper instruction method. In addition, participants reported significantly lower levels of mental and temporal demand when using the AR instruction method. There was no significant difference between the number of errors for the two instruction methods.

Task Complexity

An interesting finding is that we observed these results even though our task was relatively simple (mating and demating cables). This is not consistent with the results observed by Henderson and Feiner (2009) for simple routine maintenance tasks inside an armored vehicle turret or for simple tasks (e.g., attaching wire clips) in assembling a car door (Wiedenmaier et al., 2003). One potential reason for this difference could be related to the hardware used in the current study (HoloLens). It is possible that current AR technologies offer better possibilities with respect to visual perception and user interaction compared with AR technologies used in prior studies.

Another possible reason for this difference could be due to prior training. Whereas Henderson and Feiner (2009) recruited service members that were familiar with the task in their study, we recruited participants that had no prior exposure to the task or the instrument. Wiedenmaier and colleagues (2003) recruited novice participants, but they trained them on a different assembly task to familiarize them with the AR instructions. More research is required to determine whether AR can enhance procedural work for operators that are performing familiar or routine tasks.

Limitations

Our results indicate that an AR instruction method using off-the-shelf hardware (HoloLens) can enhance procedural work, but using the device has limitations. In particular, participants reported that the device was heavy, presented a limited field of view, and caused slight eye strain because they had to switch their visual focus between the virtual and physical objects. As optical see-through AR technology advances, we expect that devices will become much lighter and be capable of displaying a larger field of view. However, a complete redesign of the optical display technology would be required to combat the vergence-accommodation conflict reported by participants in the current study. Although various solutions have been proposed to address this problem (Kramida, 2016; Matsuda, Fix, & Lanman, 2017), it may still be several years before an actual pragmatic solution is available for off-the-shelf use. So long as there is a mismatch between AR displays’ focal distance and the distance of real or virtual content, the vergence-accommodation conflict will continue to be a limitation of AR. In this study, for AR attention director cues, we found that using a virtual bounding box instead of an overlaying virtual red target helped alleviate the vergence-accommodation conflict because a bounding box could be easily understood while being more in the periphery of the user’s view.

In addition, several factors may have contributed to the transfer of training from the AR instruction method to the paper instruction method: voice recognition, the attention director, the virtual bounding box, target port highlighting, and reduced visual scanning by presenting a virtual checklist. However, our study was not designed to independently assess each of these factors, and future research should focus on which of these factors contribute to the difference in performance we observed.

Practical Implications

Our results have important practical implications. Procedural work in NASA spaceflight operations involves complex science instruments, and the documentation to assemble, install, repair, or maintain these instruments is complex and time-consuming. Because we simplified the paper instruction method for the current experiment, we would expect greater performance benefits for the AR instruction method compared with using the full, original paper instruction method. Moreover, based on the relative simplicity of the task, we would expect greater performance benefits for tasks that are more complex.

We also observed a transfer of training when participants used the AR instruction method first. This would be helpful in training new engineers or technicians who are unfamiliar with spaceflight science instruments. In a similar vein, our results showed enhanced performance for novice participants that were unfamiliar with the procedure and the science instrument, apart from a short training session. This finding is particularly relevant for procedures performed aboard the International Space Station (ISS). Science instruments on the ISS may require periodic maintenance or need to be repaired if components are damaged. In these cases, it is up to the astronauts aboard the ISS to carry out maintenance or repair procedures on these instruments. Although astronauts may receive an initial briefing about these instruments before departure, there is a significant lag time of 18 months (or longer) between the training they receive on the ground and the time in which they might use the training while aboard the ISS (Lengyel & Newman, 2014). It is unlikely that they are able to recall the specifics of a procedure after such a considerable time lag. For this reason, an AR instruction method could effectively bootstrap the astronaut back into the procedure to improve performance (just-in-time training; Dempsey & Barshi, 2017; Foale et al., 2005). If procedure execution fails as a result of human error, not only will critical science information be lost, but the time and money required to develop, produce, and send the instrument to the ISS will be lost as well. AR instruction methods will also be valuable for future deep space manned missions where time delays due to the distance traveled will make real-time remote assistance from Earth impossible or difficult at best.

Our results suggest that procedural work in NASA spaceflight operations will benefit from using off-the-shelf AR technology to display procedural information. Spaceflight accidents that result from incorrect procedure execution can result in damage or destruction of property, mission failure, loss of public confidence, or loss of human life. Using AR to improve procedural work for NASA spaceflight operations can help prevent accidents and reduce the associated costs that result from human error.

Key Points

An augmented reality instruction method for a procedural task on a NASA spaceflight science instrument resulted in faster completion times compared with a paper instruction method.

Participants reported lower mental and temporal demand when using an AR instruction method compared with a paper instruction method.

When participants used an augmented reality instruction method first, there was a transfer of training that improved a subsequent procedure using a paper instruction method.

Footnotes

Acknowledgements

This research was conducted at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration. The authors would like to acknowledge Jim Kellogg for his assistance with the CAL mockup used in the study.

ORCID iD

Adam M. Braly

Adam M. Braly is currently a student at Rice University in the Department of Psychological Sciences. He received his MA in psychology from the University of Central Oklahoma in 2015 and his MA in experimental psychology from Texas Tech University in 2017.

Benjamin Nuernberger is an immersive user interface developer at the Jet Propulsion Laboratory. He completed his PhD in computer science from University of California Santa Barbara in 2017.

So Young Kim is a senior user experience lead at the Jet Propulsion Laboratory. She completed her PhD in aerospace engineering from Georgia Institute of Technology in 2011.

References

Arrington

C. M.

Logan

G. D.

(2004). The cost of a voluntary task switch. Psychological Science, 15, 610–615. doi:10.1111/j.0956-7976.2004.00728.x

Baird

K. M.

(1999). Evaluating the effectiveness of augmented reality and wearable computing for a manufacturing assembly task. Unpublished master’s thesis. Virginia Polytechnic Institute and State University, Blacksburg.

Barshi

Dempsey

D. L.

(2016). Evidence report: Risk of performance errors due to training deficiencies. Retrieved from https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20160003868.pdf

Baum

L. F.

(1901). The master key: An electrical fairy tale founded upon the mysteries of electricity and the optimism of its devotees. It was written for boys, but others may read it. Indianapolis, IN: Bowen-Merrill Company.

Biocca

Tang

Owen

Xiao

. (2006). Attention funnel: Omnidirectional 3D cursor for mobile augmented reality platforms. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1115-1122). New York, NY: ACM.

Bonanni

Lee

C. H.

Selker

(2005). Attention-based design of augmented reality interfaces. In Proceedings of CHI’05 Extended Abstracts on Human Factors in Computing Systems (pp. 1228-1231). New York, NY: ACM.

Brooke

(1996). SUS: A “quick and dirty” usability scale. In Jordan

P. W.

Thomas

Weerdmeester

B. A.

McClelland

I. L.

(Eds.), Usability evaluation in industry (pp. 189–194). London: Taylor and Francis.

Byers

J. C.

Bittner

A. C.

Jr. Hill

S. G.

(1989). Traditional and Raw Task Load Index (TLX) correlations: Are paired comparisons necessary? In Mita

(Ed.), Advances in industrial ergonomics and safety I (pp. 481–485). Philadelphia, PA: Taylor and Francis.

Chandler

(2007). Learning from NASA mishaps: What separates success from failure? Report presented to the Project Management Challenge. Retrieved from http://origins.sese.asu.edu/ses405/Failure%20Reports/Chandler_Faith_LearningFromFailures.pdf

10.

The coolest spot in the universe . (n.d.). Retrieved from https://coldatomlab.jpl.nasa.gov/

11.

Dempsey

D. L.

Barshi

(2017). ISS training best practices and lessons learned. Poster presented at the Human Research Program Investigators’ Workshop, Galveston, TX.

12.

Drascic

Milgram

(1996). Perceptual issues in augmented reality. Proceedings of the Stereoscopic Displays and Virtual Reality Systems III, 2653, 123–135. doi:10.1117/12.237425

13.

Foale

C. M.

Kaleri

A. Y.

Sargsyan

A. E.

Hamilton

D. R.

Melton

S. L.

Martin

D. S.

Dulchavsky

S. A.

(2005). Diagnostic instrumentation aboard ISS: Just-in-time training for non-physician crewmembers. Aviation, Space, and Environmental Medicine, 76, 594–598.

14.

Hart

S. G.

(2006). NASA-Task Load Index (NASA-TLX); 20 years later. In Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting (pp. 904–908). Santa Monica, CA: Human Factors and Ergonomics Society.

15.

Hart

S. G.

Staveland

L. E.

(1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, 52, 139–183. doi:10.1016/S0166-4115(08)62386-9

16.

Henderson

S. J.

Feiner

(2009). Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret. In Proceedings of the 8th IEEE International Symposium on Mixed and Augmented Reality (pp. 135–144). New York, NY: IEEE.

17.

Henderson

S. J.

Feiner

(2011). Augmented reality in the psychomotor phase of a procedural task. In Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality (pp. 191–200). New York, NY: IEEE.

18.

Hendy

K. C.

Hamilton

K. M.

Landry

L. M.

(1993). Measuring subjective workload: When is one scale better than many? Human Factors, 35, 579–601. doi:10.1177/001872089303500401

19.

Hoffman

D. M.

Girshick

A. R.

Akeley

Banks

M. S.

(2008). Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. Journal of Vision, 8, 33.1–33.30. doi:10.1167/8.3.33

20.

Hou

Wang

Bernold

Love

P. E.

(2013). Using animated augmented reality to cognitively guide assembly. Journal of Computing in Civil Engineering, 27, 439–451. doi:10.1061/(ASCE)CP.1943-5487.0000184

21.

Kim

Dey

A. K.

(2009). Simulated augmented reality windshield display as a cognitive mapping aid for elder driver navigation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 133-142). New York, NY: ACM.

22.

Kramida

(2016). Resolving the vergence-accommodation conflict in head-mounted displays. IEEE Transactions on Visualization and Computer Graphics, 22, 1912–1931. doi:10.1109/TVCG.2015.2473855

23.

Lengyel

D. M.

Newman

J. S.

(2014). International Space Station lessons learned for space exploration. Retrieved from https://www.nasa.gov/sites/default/files/atoms/files/iss_lessons_learned.pdf

24.

Matsuda

Fix

Lanman

(2017). Focal surface displays. ACM Transactions on Graphics (TOG), 36, 86.1–86.14. doi:10.1145/3072959.3073590

25.

Microsoft Hololens. (n.d.). Retrieved from https://www.microsoft.com/en-us/hololens/

26.

Milgram

Colquhoun

(1999). A taxonomy of real and virtual world display integration. In Ohta

Tamura

(Eds.), Mixed reality: Merging real and virtual worlds (pp. 1–26). Berlin: Springer-Verlag.

27.

Milgram

Takemura

Utsumi

Kishino

(1994). Augmented reality: A class of displays on the reality-virtuality continuum. Proceedings of Telemanipulator and Telepresence Technologies, 2351, 282–293. doi:10.1117/12.197321

28.

Monsell

(2003). Task switching. Trends in Cognitive Sciences, 7, 134–140.

29.

NASA annual mishap reports . (n.d.). Retrieved from https://nsc.nasa.gov/resources/annual-reports?tagFilter=mishap-investigation

30.

Neumann

Majoros

(1998). Cognitive, performance, and systems issues for augmented reality applications in manufacturing and maintenance. In Proceedings of the IEEE 1998 Virtual Reality Annual International Symposium (pp. 4–11). New York, NY: IEEE.

31.

Nygren

T. E.

(1991). Psychometric properties of subjective workload measurement techniques: Implications for their use in the assessment of perceived mental workload. Human Factors, 33, 17–33. doi:10.1177/001872089103300102

32.

Okamoto

Nishihara

(2016). Assembly assisted by augmented reality (A³R). In Bi

Kapoor

Bhatia

(Eds.), Intelligent systems and applications. Studies in computational intelligence (pp. 281–300). Cham: Springer.

33.

Ong

S. K.

Yuan

M. L.

Nee

A. Y. C.

(2008). Augmented reality applications in manufacturing: A survey. International Journal of Production Research, 46, 2707–2742.

34.

Ott

(1995). Maintenance executives seek greater efficiency. Aviation Week & Space Technology, 142(20), 43–44.

35.

Palmarini

Erkoyuncu

J. A.

Roy

Torabmostaedi

(2018). A systematic review of augmented reality applications in maintenance. Robotics and Computer-Integrated Manufacturing, 49, 215–228. doi:10.1016/j.rcim.2017.06.002

36.

Sanna

Manuri

Lamberti

Paravati

Pezzolla

(2015). Using handheld devices to support augmented reality-based maintenance and assembly tasks. In Proceedings of the IEEE International Conference on Consumer Electronics (pp. 178–179). New York, NY: IEEE.

37.

Starner

Mann

Rhodes

Levine

Healey

Kirsch

Picard

Pentland

(1997). Augmented reality through wearable computing. Presence: Teleoperators & Virtual Environments, 6, 386–398. doi:10.1162/pres.1997.6.4.386

38.

Sutherland

I. E.

(1965). The ultimate display. Proceedings of the International Federation of Information Processing, 65, 506–508.

39.

Sutherland

I. E.

(1968). A head-mounted three dimensional display. Proceedings of The American Federation of Information Processing Studies, 68, 757–764.

40.

Tang

Owen

Biocca

Mou

(2003). Comparative effectiveness of augmented reality in object assembly. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 73-80). New York, NY: ACM.

41.

Unity . (n.d.). Retrieved from https://unity3d.com/

42.

Wang

Ong

S. K.

Nee

A. Y. C.

(2016). A comprehensive survey of augmented reality assembly research. Advances in Manufacturing, 4, 1–22. doi:10.1007/s40436-015-0131-4

43.

Wickens

C. D.

Helleberg

Goh

Horrey

W. J.

(2001). Pilot task management: Testing an attentional expected value model of visual scanning (University of Illinois Institute of Aviation Technical Report). Savoy, IL: Aviation Research Lab.

44.

Wiedenmaier

Oehme

Schmidt

Luczak

(2003). Augmented reality (AR) for assembly processes design and experimental evaluation. International Journal of Human-Computer Interaction, 16, 497–514. doi:10.1207/S15327590IJHC1603_7

45.

Windows mixed reality | interaction design | comfort . (2018). Retrieved from https://docs.microsoft.com/en-us/windows/mixed-reality/comfort#vergence-accommodation-conflict/

46.

Yeh

Wickens

C. D.

Seagull

F. J.

(1998). Effects of frame of reference and viewing condition on attentional issues with helmet mounted displays (University of Illinois Institute of Aviation Technical Report). Savoy, IL: Aviation Research Lab.