Collaboration in the Cockpit

Artificial Wingman and the Autopilot

Perhaps aircraft accidents involving spatial disorientation can be avoided through the introduction of context-aware computing when the autopilot is not active. An examination of current tools for maintaining attitude awareness in the cockpit reveals potential opportunities.

Why did our realization that night require the insight of the wingman, an external resource whose task did not require monitoring attitude? What resources were available in the lead aircraft that could have prevented this near mishap? One possible resource is the autopilot, but autopilots are not typically used in formation because of the possibility of uncommanded disengagement. As a result, our aircraft provided no protection against Type I SD. Why doesn’t aircraft instrumentation technology provide more extensive flight crew protection? Why can’t aircraft instrumentation actively monitor the flight environment, decipher when the aircraft is potentially deviating from the context of specific flight conditions, and advise the flight crew?

In recent articles, Geiselman and colleagues discussed the need for context-aware logic to improve safety in flight deck automation and particularly in the autopilot operator interface (Geiselman, Johnson, & Buck, 2013; Geiselman, Johnson, Buck, & Patrick, 2013). However, as the foregoing story illustrates, the utility of a context-aware partner in the cockpit extends beyond the autopilot. That is, the technology present in today’s fly-by-wire aircraft control systems can determine in real time what aircraft attitudes are appropriate and will be appropriate in the context of the present flight environment. This technology can also be applied to interact with the pilot in real time regarding unanticipated flight deviations, independent of whether the autopilot is engaged.

The Traditional Attitude Indicator

Sperry-Rand Corporation developed the traditional attitude indicator in 1929 (Bassett & Sperry, 1934), and that device has dramatically improved flight safety over the past nine decades. It has also been at the center of significant controversy within the flight community. Originally designed to provide the pilot with an artificial horizon, this instrument depicts both the horizon and a normally horizontal portion of the aircraft. The controversy revolves around the point of view provided by the instrument. Should the moving part of the instrument be fixed to the horizon or fixed to the aircraft? This decision influences the naturalness of the control movement necessary to compensate for non-straight-and-level flight.

Colonel James Doolittle argued that the pilot and the aircraft function as one and the pilot’s main frame of reference is the aircraft. Doolittle believed that because the aircraft never moves with respect to the pilot, it makes no sense for the aircraft in the display to move; thus, the instrument should employ a moving horizon (Hasbrook & Rasmussen, 1973). An alternative argument was that the best displays employ movement in a manner that accurately and intuitively represents the movement in reality. Therefore, given that the pilot knows that moving the controls moves the aircraft, the aircraft within the display should also move. This controversy eventually led to the development of the principle of the moving part (Roscoe, 1968).

Although a number of research studies have been conducted in response to this controversy, the results do not decisively favor one approach over the other. Furthermore, the results often vary as a function of pilot training and whether the experiment is conducted in a ground-based simulator or in flight. As a result, the attitude instruments flown in the United States arguably violate the principle of the moving part, a deficiency that has been partly overcome through extensive training that is difficult to reverse. Therefore, this instrument has undergone little change since its development, except that its image is generally depicted graphically in the glass cockpits of today rather than existing as a physical device that is viewed directly by the pilot.

An example of this instrument display is shown in Figure 1. The yellow dot in the center represents the nose of the aircraft, and the thin white line, which separates the blue and brown areas, indicates the horizon. The blue area represents the sky, and the brown area the ground. The display is intended to show pilots exactly what they would see if looking out of a hole in the nose of the aircraft. In this example, the aircraft is banked to the left, with the nose pointed below the horizon by about 5°.

OPEN IN VIEWER

Figure 1.

Sperry-style attitude indicator depicting nose down, 30° left bank.

Besides violating the principle of the moving part, other known design issues exist with the traditional attitude indicator. For example, the pilot must fixate on the display when determining aircraft attitude. This behavior differs from visual flight, in which the horizon can easily be determined from peripheral vision. As a result, the need for foveal fixation when using this instrument – whereby the pilot is forced to focus his or her visual attention on the instrument – likely reduces the frequency with which orientation is ascertained and causes increased conflict with other central vision tasks (Weinstein & Wickens, 1992).

Altitude Indicator Redesign

Led by a desire to spare others the anguish of barely saving an aircraft deep in a valley – or incurring a less desirable outcome – we embarked on a reexamination of the traditional attitude indicator. The redesign originally included two independent goals relevant to this article:

We also redesigned the graphics in an attempt to avoid the moving-part controversy, but this element is not of significant consequence for the current discussion.

Figure 2 shows images of the initial graphical design. The aircraft being flown is banked left at 30°, as indicated by the aircraft symbol near the top of the display, which is banked to the left of the center of the display. The gray wedge augments the image of this aircraft, providing a larger visual cue. Because the aircraft’s current contextual assessment shows that it should be cruising straight and level, the command text Bank Right indicates that the pilot needs to bank right to return to straight-and-level flight. This command would be augmented with an aural indicator if the bank angle was outside the expected bounds for the current flight context. This verbal command would also state, “Bank right,” to inform the pilot that a control action was required without requiring him or her to look at the display.

OPEN IN VIEWER

Figure 2.

Attitude stabilization display depicting nose down, 30° left bank.

The pilot could directly select the cruise phase of flight to provide contextual information to the system; alternatively, the system could derive this information from the flight plan or recent pilot behavior. In this example, the pitch is shown by the “pitch status pointer” (green chevron in the center of the screen), which indicates that the nose of the aircraft is level.

Standardized Communication

Before we discuss this redesign further, it is instructive to return to our story or, more specifically, to the root cause of this near mishap. In evaluating the root cause, one can see that communication between the two pilots in the lead aircraft was prone to error. If they had been able simply to see each other, they would have realized immediately that they were both aborting the act of flying in favor of a noncritical secondary task. This visual cue was removed during the aircraft’s design. Designers chose to put one pilot behind the other in the interest of aerodynamics, giving the fighter the speed necessary to survive in battle. To compensate for this loss, a set of standard procedures was developed that required redundancy of information transfer (i.e., requiring the receiver to restate the information received) to ensure the proper transfer of information between the crew members. Unfortunately, we did not follow these standard procedures in this example, further contributing to the near mishap.

In the design of context-aware systems, this two-way communication of state information is also desirable, but it is difficult to achieve. The system can utilize cameras or other sensors to observe the pilots and determine general state information, such as their locus of attention, alertness, and perhaps a measure of their certainty. However, it is more challenging for humans to observe the computer to determine similar state information.

Is it important to understand the cues the computer is using to effect an autonomous decision or to understand the certainty of the computer’s analysis? Furthermore, if this information is important and one cannot directly ascertain it, how can one formalize communication protocols to ensure the transfer of similar information? One potential approach is to assign the computer anthropomorphic attributes, providing a representation of the computer as a fallible teammate. In this solution, the computer would express certainty through humanlike attributes, including tone of voice, speed of speech rhythms, or graphical symbols.

However, given that early attempts at anthropomorphic interfaces have illustrated their challenges, other approaches should be considered. These approaches include standardizing or formalizing two-way communication protocols between the human and computer to provide efficient, verified communication in time-constrained environments. This interaction might have redundant elements, similar to the standardized protocol for aircraft handover between pilots.

The example also illustrates the need for the system to understand longer-term goals and the short-term behaviors that are consistent with these goals. My wingman, in this example, understood that we were heading home and under no immediate threat. Therefore, the departure from straight-and-level flight was unexpected, prompting a life-saving radio call. Similarly, there is a need for context-aware systems to not only be informed of the long-term goals but also to determine the unlikely events that warrant communication regarding system status.

In one implementation of the redesigned display, a pilot might be required to manually select a phase of flight to indicate his or her intentions. Early feedback from pilots indicated that this overt action might be acceptable during normal operations (e.g., when indicating that they are entering straight-and-level flight after takeoff). Under other circumstances (for example, when suddenly entering turbulence), neither manually indicating a change in status nor regularly hearing the audible warning as the aircraft rocks from side to side within the turbulence was desired, as both increase operator workload that is already likely to be high.

So how can systems be designed so they are able to make these contextual switches on their own with high reliability, or at least without significantly adding to operator workload? In the instrument design, it is important to anticipate these circumstances and consider them within the context-aware logic to readily update the instrument when additional circumstances are discovered.

Additionally, the tone of this communication is important. Because of the lack of easy two-way communication between the operator and the system, many systems communicate unexpected events through auditory warnings or alerts. These warnings are often repeated incessantly until disabled through a manual switch or by altering the system state. As a result, operators ignore (often subconsciously) many of the audible warnings to the extent possible, which adds to the potential danger and operator stress during stressful events.

During interpersonal communication, people understand that unless operating from a point of urgency and certainty, questions from a team member are often received more readily than commands or warnings. Furthermore, operators understand intuitively that during times of high workload, a response requires additional mental capacity and that the pure repetition of a warning is not necessarily warranted by the lack of an overt response from the receiver. One must then ask: How should context-aware systems be designed to enable more relaxed two-way communication while ensuring transfer of critical information?

Returning to the design of the attitude display, our initial discussions and demonstrations with experienced pilots indicated an acceptance of applying contextual information to aid him or her in detecting the onset of spatial disorientation, which permits early compensation before the onset of a significant issue. However, many challenges exist to elegantly respond to broader contextual information and permit the system to gain this contextual information without requiring explicit cueing from the pilot.

Avoiding Loss of Control

Our second design concept provided verbal commands to aid pilot decision making. Authors of previous research have explored cues, such as spatial 3-D audio, to indicate the horizon or departures from the horizontal position (Endsley, Rosiles, Zhang, & Macedo, 1996). However, this research indicated that the pilot requires time to process this information and to determine appropriate control actions to resume a desired attitude. The use of verbal commands was originally envisioned as a means to decrease the time required for this analysis by providing commands, such as “Stick right,” to the pilot, which he or she could then follow to aid recovery. However, the time required to verbalize the command and allow the pilot to process it and then react can exceed 1 s. With the aircraft in a fast roll, this may be long enough for the system state to change before the system issues the command and the pilot responds. Therefore, the primary goal of the system must be to alert the pilot early, before loss of control, providing time to respond to the commands.

However, if a condition were to occur whereby this early alert was not successful and the aircraft was undergoing rapid maneuvers, it would become necessary to anticipate the aircraft’s likely orientation, which would provide the pilot time to respond. Otherwise, there will be circumstances under which the pilot will execute a control action late, perhaps at a time when the opposing control action would be appropriate.

Returning to the discussion of context-aware systems, these verbal commands may be appropriate in many situations, and we continue to assess their utility. However, if circumstances occur in which the pilot has lost control of the aircraft and rapid maneuvering is required for recovery, the delay imposed by the human operator can make such commands impractical.

Thus, one arrives at the point at which human limitations must be recognized. As Paul Fitts and colleagues (1951) pointed out more than half a century ago, systems should be designed to leverage human judgment. However, hardware–software systems have certain abilities that people do not possess (Fitts et al., 1951). Today, when systems have the ability to sense and assess conditions much more rapidly than humans can, as well as provide general aircraft control, one must ask whether there is a time when the aircraft itself should assume that the pilot’s limitations have been exceeded and attempt its own recovery.

Initial systems toward this end are being deployed. For example, the U.S. Air Force has explored the automatic ground collision avoidance system for fighter-attack aircraft. This system is designed to automatically recover the aircraft when it is within 1.5 s of approaching the point of no return preceding a ground collision (Trimble, 2009). It might be argued, though, that helping the pilot to detect loss of orientation knowledge early might eliminate the need for the machine to assume control from the operator.

Furthermore, not all significant mishaps occur with the aircraft encountering the ground. For example, on January 31, 2013, an Air Force pilot became spatially disoriented during night flight. Perceiving an impending crash, and unable to determine his orientation, the pilot ejected from the aircraft (Svan, 2013). Unfortunately, because of the aircraft’s speed and orientation, together with other factors, the pilot suffered fatal head and neck injuries during ejection. Would verbal recovery commands have aided the pilot’s understanding and given him time to stabilize his orientation? The answer is unclear. However, it is clear that there are circumstances under which displays, which require foveal fixation and a consequent decision in a time-critical and highly stressful environment, will not be sufficient to permit recovery.

Members of the human factors/ergonomics community will be challenged by the progress of autonomous and context-aware systems to define new interaction models. The development of this knowledge will need to keep pace with computing and sensing technology as they advance at a high rate of speed. With luck, this technology will race toward a critical level of capability before another young pilot finds himself or herself racing toward terra firma, wondering, “Lead, where are you going?”