Abstract
This discussion reflects on cognitive engineering considerations highlighted by a recent design project developing recommendations for database, functional, and interface design requirements for a General Aviation (GA) performance data repository in order to improve aviation safety. Data collection systems can now record hundreds of key flight parameters, such as altitude, air speed, route (latitude/longitude), pitch, roll, oil pressure, power, and so forth (Higgins, Clachar, & Hennselek, 2013). In addition, other parameters may need to be manually entered by the pilot or inferred (e.g., on what runway it landed).
This data repository is intended to improve aviation safety in two ways. First, it could provide analysts with access to aggregated data to more thoroughly evaluate GA safety. Second—the focus of this paper—it could provide individual GA pilots, chief pilots, and flight instructors with feedback on an individual’s performance coupled with data on how this pilot compares with other GA pilots. In particular, such individuals would want to know when an important event occurs, such as an exceeding some established norm or limit.
To set the stage for this design project, members of the design team included specialists in cognitive systems engineering, software engineering, design and operation of the aviation system, GA piloting and instruction, and industrial design. The assumption is that these pilots would commonly access the repository using tablet-sized devices via a web browser, thus limiting screen real estate and the underlying architecture. It is assumed that the users will be web fluent but not necessarily power users, that many of them will be occasional users, and that they will be familiar with the aviation terminology used in the system.
Cognitive Engineering Design Consideration. It is important to frame the goal of such a design task as first helping the user to focus attention on a potentially important event (in this case, an exceedance) and then providing integrated displays that help the user to construct a deeper understanding of the context in which an exceedance occurred. Although it is tempting to focus the system design on the metrics and calculations underlying the detection of exceedances, the user needs to view each exceedance in its context.
Once the design goal was framed, a very iterative design process was applied, designing “from the outside in” by first developing and evaluating storyboards that specified how the interactions should appear from a user perspective. These storyboards provided an explicit definition of the desired interface and associated interactions and an implicit definition of the necessary underlying functionality. Cognitive walkthroughs (Lewis & Wharton, 1997; Smith, Stone, & Spencer, 2006) were then used to predict users’ interactions with this design.
After refinements based on these walkthroughs, a functional prototype was developed that enabled more dynamic cognitive walkthroughs. This made it possible to easily explore a larger range of flights using actual data and also forced the design team to confront initially underspecified functional requirements.
This prototype led to a design defined by several tabs. The Search Builder Tab (see Figure 1) serves as the Home Page and supports three different classes of users (single pilots, flight operators managing or instructing multiple pilots, and system analysts). It supports user interaction with the underlying data repository via forcing functions (options in dropdown menus mirroring repository data categories) and feedback to help users who enter something that the system does not recognize.
Search Builder Tab (for creating a search query).
The Results Tab (see Figure 2) shows a broad summary of search results. The “Exceedances” section uses criteria for indicating that a flight is approaching or has crossed some threshold. The section labeled Detailed Reports: Stabilized Approach provides feedback about how that pilot is performing relative to the other pilots in the database for the glideslope (10th and 90th percentile) and categorizes approach airspeed and vertical speed relative to commonly accepted limits.
Results Tab (summary of performance for key metrics).
Underlying the design of the Results Tab are several important considerations:
The design needs to focus the pilot’s attention on those flights that merit deeper consideration and then make it easy to navigate to the relevant additional details.
The functionality needs to perform adequately in terms of hits and misses for flights it labels as exceedances.
If the underlying database and criteria do not characterize the appropriate aspects of performance, significant events may be missed.
Cognitive Engineering Design Considerations. Like other problem-solving tasks, design is susceptible to cognitive narrowing. In the development of such storyboards and prototypes, the risk of cognitive narrowing then arises because the apparent next step in such a linear design approach (from start to finish) is to display the parameter of interest (such as roll angle) as a function of time for a flight of interest and to indicate when the exceedance occurred (i.e., to just show a single graph of roll for that flight, as illustrated in the graph on the left in the red box in Figure 3).
Pop-up window: Detailed results for a single flight.
The cognitive walkthroughs with the prototype helped avoid this narrowing. For expert pilots and instructors, a typical response in viewing this single graph for roll was: “I can’t tell if this exceedance is really an example of poor performance or not. I need more information about the flight.” In response to such input, one designer commented: “This was a real eye opener. It was very easy to fixate on the exceedances as the end result to show the pilot. That’s not enough to understand what was happening for a given flight.” (An interesting question is whether this suggests that we should design not only from the outside in but also from the endpoint—the final display viewed by the user—back to reduce designer fixation on a much too limited and simplistic display.)
The Detailed Flight Data Display, selected from the hyperlinks on Results tab, creates a pop-up window for flights with particular attributes (e.g., Figure 3 shows a display for the one flight for which Roll = High). The pilot can then select among these flights to see details. (In this illustration, there is only one flight, so its details appear by default.) Based on the insights from our cognitive walkthroughs, this display was revised to present all of a flight’s data in a consistent order to help the pilot to gain a more holistic sense of the context. This decision involved defining a consistent format to avoid the increased navigational complexity and user effort associated with a display customized to different exceedances. Critical information in the pop-up about each flight is shown in the same order and format, with a consistent scheme for highlighting important phenomena:
A red or yellow rectangle is drawn around any flight parameter with a peak value in the Caution (yellow) or High/Low (red) range.
A sliding timeline is provided, with an associated text box showing the corresponding values of the critical parameters for the position of the marker on the timeline. (The slider moves across all the timelines simultaneously.)
The display also provides a bar chart to show the pilot how this flight compares to flights by other pilots as well as to his/her own past flights, with the performance for this particular flight coded green (OK), yellow (Caution), or red (High/Low).
Cognitive Engineering Design Consideration. It is important to keep the real goal of this information retrieval system in mind: to influence pilots to fly more safely in the future. It is not enough to provide the pilot with information about his/her flights: The information needs to be presented in an impactful manner. Specifically, the third feature noted above helps portray the level of risk associated with this flight. For example, the strong indication in Figure 3 (red bar on the bar chart) highlights that the degree of roll was well beyond that of all other flights by all other pilots. Further, this design supports involvement of experts, such as flight instructors or flying club managers, helping them to focus on flights of concern and to better understand what occurred.
In short, the overall theme of this discussion has been that cognitive systems engineering has continuously emphasized the importance of driving a design effort forward based on the real goal, not some technological substitute. In the process of design, it is easy to get sidetracked and translate the goal of supporting decisions (such as influencing a pilot to fly more safely) into the goal of providing him/her with access to information. Cognitive engineering instead emphasizes the need to go beyond providing simple information access and to craft integrated displays that support the user in applying his/her knowledge to more deeply understand the context of the data and to more accurately assess the implications. With this, the challenge still remains for cognitive engineering to more deeply explore the question of how designs can successfully motivate a user to improve future performance.
Footnotes
Acknowledgements
This work was sponsored by the Federal Aviation Administration under Project 5 of the FAA Center of Excellence for General Aviation, PEGASAS (Partnership to Enhance General Aviation Safety Accessibility and Sustainability), via Grant 12-C-GA-OSU to The Ohio State University. The authors are grateful to FAA project technical monitor Mr. Michael Vu and to project partners at Purdue University and the Georgia Institute of Technology. The information in this research does not constitute FAA Flight Standards or FAA Aircraft Certification policy.
Dr. Philip J. Smith is a professor and chair in the Department of Integrated Systems Engineering at The Ohio State University. He teaches courses in the areas of cognitive systems engineering, the design of distributed work systems, collaborative decision making, artificial intelligence, human–automation interaction, the design of cooperative problem-solving systems, and the design of intelligent tutoring systems. His research focuses on issues concerned with the design of systems to support information retrieval, continuous adaptive planning, training, and fault diagnosis.
Dustin Johnson is a software engineer in Columbus, Ohio. His software experience includes user interface, web application, and embedded systems programming. He has developed software in the fields of aviation, manufacturing, and academic research, including substantial work on the design of airport surface management systems and postoperations analysis tools for the National Aviation System.
Shawn Pruchnicki is a faculty member at The Ohio State University associated with the Center for Aviation Studies and teaches cognitive engineering, aviation safety, human factors, and accident investigation courses. He has a master’s degree from Embry Riddle and is finishing a PhD in Industrial and Systems Engineering at The Ohio State University.
Josh Schimmel graduated in May 2015 with a bachelor’s degree in Industrial and Systems Engineering and is currently completing his first year as a graduate student, all at the Ohio State University. He is a research assistant at the university’s Center for Aviation Studies and hopes for a career improving the nation’s air transportation system. In addition to his research role, he works at Delta Air Lines and Ohio State Athletics and is past president of the AAAE (American Associations of Airport Executives) student chapter at The Ohio State.
Amy Spencer has a master’s degree with a focus on industrial design, human computer interaction, and cognitive systems engineering. She has extensive experience in the design and evaluation of new tools and procedures to support collaborative decision making in the National Airspace System (NAS) and human automation interaction on the flight deck. This includes work on advances in airport surface management, integrated arrival and departure flow management, and the integration of unmanned aerial systems into the NAS.
Seth Young, PhD, AAE, Certified Flight Instructor, is the McConnell Chair of Aviation and Director of The Ohio State University’s Center for Aviation Studies at The Ohio State University and an associate professor of civil environmental and geodetic engineering. He holds a PhD in civil and environmental engineering/transportation and an MS in industrial engineering/operations research and holds an instrument-rated commercial airplane and seaplane pilot’s license and certified flight instructor certificate.