Abstract
Objective:
The objective is to provide a review of ecological interface design (EID), to illustrate its value to human factors/ergonomics, and to identify areas for future research and development.
Background:
EID uses mature interface technologies to provide decision making and problem solving support. A variety of theoretical concepts and analytical tools have been developed to meet the associated challenges. EID provides support that is simultaneously grounded in the practical realities of a work domain and tailored to human capabilities and limitations.
Method:
EID’s theoretical foundation is discussed briefly. Concrete examples of ecological and traditional interfaces are provided. Different categories of work domains are described, as well as the associated implications for interface design. A targeted literature review is conducted and the experimental outcomes are summarized. A representative evaluation is discussed, and interpretations of performance are provided.
Results:
The evidence reveals that EID has been remarkably successful in significantly improving performance for work domains with constraints that are law driven (e.g., process control). In contrast, work domains that are intent-driven (e.g., information retrieval) have, by and large, been ignored. Also, few studies have addressed nonvisual displays.
Conclusion:
EID has not yet realized its potential to improve safety and efficiency across the entire continuum of work domains.
Application:
EID provides a single integrated framework that is (a) sufficiently comprehensive to deal with complicated work domains and (b) capable of producing innovative support that will generalize to actual work settings.
Keywords
Introduction
Cognitive systems engineering (CSE) provides an overarching framework for the analysis, design, and evaluation of complex systems (Rasmussen, Pejtersen, & Goodstein, 1994; Vicente, 1999; Woods & Roth, 1988). Ecological interface design (EID) is an extension of the CSE framework that focuses on using mature interface technologies to develop effective decision-making and problem-solving support. A fundamental goal of EID is to transform the type of behavior that is needed for workers to do their jobs: from activities that require the worker to draw upon limited capacity cognitive resources to activities that allow powerful perception and action capabilities to be leveraged. A key process in achieving this goal is to understand the abstract, complex, semantic structure of a work domain and then design graphical displays that make it both concrete (i.e., easy to see) and meaningful (i.e., easy to interpret) to the worker.
EID was first described by Rasmussen and Vicente (Rasmussen & Vicente, 1989, 1990; Vicente & Rasmussen, 1990, 1992). A great deal of progress has been made in the ensuing 30 years. Here we summarize the evolution of EID, provide concrete examples of ecological and alternative interfaces, summarize its effectiveness in improving performance, and outline areas for future research.
Theoretical Overview of EID
An overview of the CSE/EID approach is illustrated in Figure 1. This approach views human computer-interaction as a complex, closed-loop dynamical system (Bennett & Flach, 2011). It is a triadic, semiotic approach: The interface is the medium (a “virtual” ecology) that stands between the work domain (situations) and the human (awareness). Each of these primary system components contributes a set of constraints (i.e., affordances, attunement, specificity); the CSE analytical tools (e.g., abstraction and aggregation hierarchies, decision ladder) are used to understand these constraints. Ultimately, the mutual interaction of these three system components (that is, the extent that they are well matched; semantic mapping) determines how well the overall system will perform (Bennett & Flach, 1992; Woods 1991). The principles of EID (i.e., direct perception, direct manipulation, visual momentum) guide interface development and reduce the number of design-evaluate iterations required.
Theoretical overview of ecological interface design: Primary system components (i.e., work domain, interface, human), critical relationships between them (e.g., semantic mapping), and design principles (direct perception, direct manipulation, visual momentum). (Adapted with permission from Bennett & Flach, 2011.)
A detailed description of the EID approach is beyond the scope of the present article. We refer the interested reader to the four original articles by Rasmussen and Vicente mentioned earlier and the following core literature that has expanded the theoretical foundations of EID since its advent in 1989. This includes a number of books (Bennett & Flach, 2011; Burns & Hajdukiewicz, 2004; Flach, Hancock, Caird, & Vicente, 1995; Hancock, Flach, Caird, & Vicente, 1995; Rasmussen et al., 1994; Vicente, 1999) and reviews (Bennett, Flach, Edman, Holt, & Lee, 2015; Borst, Flach, & Ellerbroek, 2015; Burns, 2013; Flach, 2017; McIlroy & Stanton, 2015; Rasmussen, 1999; Read, Salmon, & Lenné, 2012; Vicente, 1996, 2002).
Direct Perception
Here we will focus on perhaps the core design principle of EID: direct perception. This term was first applied in the context of interface design by Flach and Vicente (1989) as a way to link the insights from Gibson’s theories of direct perception (Gibson, 1966, 1979) to interface design. Gibson believed that meaningful aspects of an ecology (specifically, its affordances) could be perceived directly, without the need to be mentally “inferred, deduced, or calculated.” (Flach & Vicente, 1989, p. 19). Ultimately the goal is to transform decision making and problem solving from a cognitive task (using limited capacity resources) to a perceptual one (leveraging powerful human capabilities). This involves three sets of mappings (see Figure 1): content, form, and semantic mapping. Each set of mappings will be explored further.
Content mapping
Gibson (1979, p. 2) placed a premium on understanding the ecology: “First, the environment must be described, since what there is to be perceived has to be stipulated before one can even talk about perceiving it.” Similarly, EID differentiates itself from other approaches to interface design by its emphasis on understanding the ecology for work (i.e., the work domain).
Content mapping is the extent to which the interface contains all of the work domain information that is necessary for effective system control by the user. A high-quality mapping is achieved through the application of CSE analytical tools (i.e., abstraction and aggregation hierarchies). A work domain analysis is conducted with them to develop a model of the work domain semantics (e.g., goals, functions, physical resources). The end product is a description of the collective possibilities for action: its affordances (Gibson, 1979, also referred to as relational invariants or behavioral shaping constraints).
Form mapping
Form mapping is the extent to which the visual properties produced by the display can be easily picked up and interpreted properly by the worker. Rasmussen and Vicente (1989, p. 532) originally described form mapping in very general terms: “According to Gibsonian theory, these invariants in the interface should allow operators to perceive the system’s affordances directly.” We (Bennett & Flach, 1992) used insights from the visual attention and form perception literatures to develop more precise principles that govern the quality of form mapping. Emergent features are higher order perceptual properties (e.g., the area and overall shape of a rectangle) that arise from the interaction of lower level graphical elements (e.g., the height and width of the rectangle). Some additional examples of emergent features include colinearity, equality, closure, area, angle, horizontal extent, vertical extent, and good form.
Semantic mapping
Furthermore, there is a need for content and form mappings to be mutually compatible. Our review and analysis of the early laboratory research on visual displays revealed the principle of semantic mapping (see Figure 1, Bennett & Flach, 1992, p. 530): If the display produces highly salient emergent features and these emergent features directly reflect the critical data relationships and inherent constraints in the domain, then improved performance is likely to follow.
One fundamental characteristic that often differentiates ecological displays from those designed using alternative approaches is a failure in content mapping: a lack of functional information in the interface. For example, in a seminal article, Vicente and Rasmussen (1992, p. 603) describe their ecological interface “as the physical/functional (P + F) interface” and an alternative “based on a more traditional format … as the physical (P) interface.” Note that there are any number of alternative approaches to display design in the literature (e.g., psychophysical, aesthetic, visual attention, naturalistic decision making, problem solving, Bennett & Flach, 2011). We will use the term alternative to refer to all displays that were not designed using the EID framework.
Concrete Examples
These abstract principles of design will be demonstrated through concrete examples drawn from our research program in military command and control (Bennett, Posey, & Shattuck, 2008; Hall, Shattuck, & Bennett, 2012). Our work domain analyses revealed that the single most important piece of functional information for an Army battalion commander is force ratio (Bennett et al., 2008, p. 352): “the relative amount of combat power that exists between two opposing forces at any point in time.” Force ratio incorporates both tangible (e.g., number of tanks) and intangible (e.g., morale) factors; it ebbs (resources expended or losses incurred) and flows (reinforcements) during a tactical engagement. It is a system property that must be monitored constantly; it is a barometer of success that informs all tactical decisions: whether to initiate, continue, alter, or abort a mission. We (Bennett et al., 2008) worked with Army personnel to develop a practical estimate of force ratio: a numerical ratio of friendly and enemy military force based on force estimates for individual tanks and armored personnel carriers.
Ecological Interface
We then developed ecological displays (Bennett et al., 2008) to represent force ratio and its value over time (see Figure 2). Calculated levels of force for friendly and enemy units are represented by bar graphs (labeled “Friendly Force” and “Enemy Force”) in the force ratio display (right side of Figure 2). The critical functional property of force ratio is redundantly represented by the relative size of the bar graphs (the friendly force bar graph is approximately three times the length of the enemy force bar graph) and the force ratio reflecting line (this line connects the two bar graphs and intersects the appropriate trend display at the appropriate point on its axis). Trend lines inside the force ratio trend display are used to represent actual and planned values of force ratio over time (see labeled trend lines on the upper left side of Figure 2).
Ecological displays for the functional properties of force (friendly and enemy) and force ratio over time (planned and actual). (Adapted with permission from Bennett, Posey, & Shattuck, 2008.)
A representative example of dynamic behaviors for these displays are illustrated in Figure 3. At the beginning of the engagement (see Figure 3a) the enemy forces enjoy a 2.5 to 1 advantage; the balance gradually tips toward friendly forces (Figures 3b, 3c), who have a 4 to 1 advantage by the end of the engagement (Figure 3d). In summary, the critical functional property of force ratio is calculated and represented in the ecological interface; it is there to be perceived directly and does not have to be derived by the commander through an analytical process.
Force ratio and force ratio trend displays over time: Enemy forces have an initial advantage of 2.5 to 1 (a); over time this turns into a 4 to 1 advantage for friendly forces (d). (Adapted with permission from Bennett, Posey, & Shattuck, 2008.)
Army Interface
An alternative method of presenting these combat resources is provided by the Army’s Force XXI Battle Command Brigade and Below (FBCB2) interface. This was being phased into use at the beginning of our research program (2000), was fielded shortly thereafter in Iraq and Afghanistan, and is still in use today. We closely replicated its visual appearance, data presentation, and interaction conventions. Combat resources (e.g., tanks, armored personnel carriers) are represented as squares in a matrix (see Figure 4a); these squares are color coded to provide categorical estimates of current levels (i.e., green, amber, red, and black for decreasing resources). Exact values could be obtained for each battalion company (A, B, C, D) by clicking on one of the buttons located to the left of the matrix (e.g., D/1–22): A window appears with associated labels and digital values (see Figure 4b). No explicit representation of the abstract functional property of force ratio is provided in this interface; it must be estimated analytically from related low-level data.
An experimental version of the Army’s The Force XXI Battle Command Brigade and Below (FBCB2) interface. (Adapted with permission from Talcott, Bennett, Martinez, Shattuck, & Stansifer, 2007.)
Quality of Mappings
These two interfaces will now be analyzed in terms of the mappings described earlier.
Content mapping
The quality of content mapping is very high for the ecological display. The functional information in the display includes friendly and enemy force levels, actual force ratio, planned force ratio, and the value of both force ratios over time. In contrast, the quality of content mapping is very low for the alternative interface. The display contains only the raw, low-level data that would need to be gathered and integrated to estimate force ratio.
Form mapping
The ecological displays are analogical in nature: The dynamic geometrical forms provide a one-to-one relationship between a change in the domain property and a visual change in the display. The emergent features produced by these displays include horizontal extent from a common baseline (each bar graph), relative horizontal extent (each bar graph relative to each other), orientation (reflecting line), and vertical extent (force ratio marker relative to baseline of trend display; difference between actual and planned force ratio). These emergent features are salient; therefore, the quality of form mapping is very high for the ecological displays.
In contrast, the quality of form mapping for the alternative interface is particularly low. The categorical and alphanumeric representations do not support direct perception of functional information (i.e., force and force ratio); the relationships between underlying meaning in the work domain and the visual properties of the representations are arbitrary and must be mentally calculated. In addition, related information (e.g., low-level data for each company) is spread out across separate display nodes, forcing navigation and adding additional memory load.
Semantic mapping
A primary consideration in semantic mapping is to match the structure of the information (and its relative importance) in the work domain to the structure (and visual salience) of the representations in the interface. The domain property here is a ratio, which essentially consists of two values and the quantitative relationship between them. The bar graphs in the ecological display provide salient emergent features that represent each of the two relevant values (i.e., friendly and enemy force) directly. The relationship between these two values (i.e., force ratio) is directly specified by a very salient higher-order emergent feature (i.e., the relative lengths of the two bar graphs). This relationship is further specified by yet another and higher order emergent feature: the orientation of the force reflecting line. Thus, the ecological interface provides an effective semantic mapping that enables direct perception of critical work domain properties. In contrast, the semantic mappings of the alternative interface do not. In the following section we summarize how these types of differences in design between alternative and ecological interfaces have translated into performance differences.
Analysis and Synthesis of the EID Literature
Categories of Work Domains
EID was initially applied to two work domains with very different types of behavioral constraints. The first was thermal hydraulics (e.g., Vicente & Rasmussen, 1990) where unfolding system events (i.e., the course of events over time that produce changes in system state) are governed by the physical laws of nature. There is an element of causality in this type of “law-driven” work domain. For example, the rate of fluid flowing through a pipe can be accurately predicted when the critical variables (e.g., pressure, pipe diameter, viscosity, etc.) are known. Conversely, the needs and preferences of an individual are secondary to the laws of nature (e.g., a need or intent to change the rate of flow in a pipe is not sufficient; the valve must be turned).
The second was fiction retrieval (e.g., Pejtersen, 1992; Rasmussen et al., 1994), where the unfolding events (i.e., various activities required to find a book of fiction) depend upon an individual’s needs, goals, practices, and intentions. Library patrons are free to choose any book at their discretion and leisure. Thus, in this type of “intent-driven” work domain, the needs and preferences of an individual will play a primary role in determining how events unfold; as a result, the course of events may be very difficult to predict (even for the individual performing the work). Conversely, the laws of nature and causality play a minor role.
These two work domains are drawn from opposite ends of a continuum of behavioral constraints for systems. In the middle of this continuum, there are work domains where both law-driven and intent-driven constraints are important determinants of system behavior (i.e., work domains with “mixed” behavioral constraints). These distinctions have important implications for interface design (e.g., Bennett & Flach, 2011). Therefore, we use these three categories to organize and integrate the EID literature in the following section.
Application of EID Across Work Domains
We conducted a literature search for EID by submitting the exact phrase “ecological interface design” to the Google Scholar search engine; 3,340 results were obtained. We identified a subset of those publications where the analytical tools and principles of EID were explicitly applied and implications for design were made clear (see also McIlroy & Stanton, 2015; Read et al., 2012; Vicente, 1996, 2002). This subset of literature has been included as supplementary material (available with the manuscript on the HF website; organized to support the outcomes presented in Figures 5 and 6).
The number and type of publications found in a literature search for ecological interface design. The publications are organized by the behavioral shaping constraints of the underlying work domain.
A summary of the overall outcome of each journal article (see supplementary materials) that empirically compared ecological and alternative interfaces using objective performance measures. The overall outcome is with reference to the EID display (e.g., a positive outcome is one in which the majority of significant differences indicated that EID produced superior performance).
We placed each publication in one of three categories according to the nature of underlying work domain constraints: primarily intent-driven, primarily law-driven, or a mixture of both. The total number of publications falling in each category is illustrated in Figure 5 as segmented bar graphs. The shading distinguishes between journal articles (shaded fill) and other types of publications (no fill); the cross hatching distinguishes between evaluation (cross hatching) and design only (no cross hatching).
Figure 5 indicates that the original application of EID to work domains at both ends of the spectrum is a trend that did not continue over time. EID has been applied only sporadically to intent-driven work domains (i.e., the left bar graph in Figure 5). The vast majority of subsequent work has focused on mixed and law-driven work domains (i.e., middle and right columns in Figure 5). Note also that the primary focus of virtually every publication appearing in the mixed category is on the law-driven constraints, not the intent-driven constraints.
EID has been applied to an extremely wide variety of work domains in the mixed and law-driven categories. We identified specific application areas with a high volume of research activity. We list these application areas here (for each work domain category and ordered from higher to lower volume) along with a particularly representative study for each application area: For the law-driven category, these application areas are thermal hydraulic (e.g., Vicente & Rasmussen, 1990), power distribution (e.g., Tran, Hilliard, & Jamieson, 2017), petrochemical (e.g., Jamieson & Vicente, 2001), and pasteurization (e.g., Reising & Sanderson, 2002). For the mixed category, these application areas are aviation (e.g., Amelink, Mulder, van Paassen, & Flach, 2005), transportation (e.g., Seppelt & Lee, 2007), military (e.g., Bennett et al., 2008), medicine (e.g., Effken, Kim, & Shaw, 1997), and network management (e.g., Burns, Kuo, & Ng, 2003).
Empirical Evaluation
Overall pattern of results
In this section, we summarize the outcome of empirical evaluations. Only journal articles which included (a) direct comparisons between ecological and alternative interfaces and (b) objective performance measures were considered. We classified the overall outcome of each individual article meeting these criteria into one of three categories: positive (the majority of significant differences between EID and alternative displays indicated that EID produced superior performance), negative (the majority of significant differences indicated that the alternative displays produced superior performance), or neutral (no significant differences or an equitable distribution between displays). The overall patterns are illustrated in Figure 6 (the number of evaluations appearing here is less than those in Figure 5 because not all evaluations met the criteria listed above).
The results indicate that EID has proven to be remarkably effective in significantly improving performance relative to alternative interfaces. Overall, approximately four out of five evaluations reported in journals have obtained significant performance advantages favoring ecological interfaces. Approximately one in five evaluations reported neutral results. Only one study reported significant performance advantages for an alternative interface: Jungk, Thull, Hoeft, and Rau (1999). However, a redesign (Jungk, Thull, Hoeft, & Rau, 2000) of the original ecological interface (involving an increase from four parameters to 35) produced significant performance advantages relative to the alternative interface.
Representative example
Our evaluation (Hall et al., 2012) of the ecological and alternative interfaces described earlier (e.g., Figures 2–4 and associated text) provides a representative example of an ecological evaluation. Experienced Army officers completed three different types of simulated, realistic, tactical engagements. They executed a variety of experimental tasks representative of those in the real world (e.g., situation assessment reports, commander’s critical information requirements). Six of the seven dependent measures revealed significant differences in performance; all six favored the ecological interface. The performance benefits were substantial: on average, responses were twice as fast, twice as accurate, and completed with half the subjective workload. This outcome was obtained despite the fact that these officers had extensive training and combat experience with the alternative interface, but extremely limited exposure to the ecological interface. We will now provide a generic interpretation of the improved performance for ecological interfaces.
Poor performance with alternative interfaces
The consistent performance decrements for alternative interfaces occur for a variety of reasons. Critical domain information (usually functional information) is often not present (a failure in content mapping). Information is often represented categorically or alphanumerically instead of analogically; it is often spread across multiple windows or displays (a failure in form mapping). Even when analogical representations do exist, the visual properties of the emergent features are often poorly mapped to the semantic structure of the work domain (a failure in semantic mapping, e.g., Holt, Bennett, & Flach, 2015). As a result, these interfaces often require extensive analytical cognitive processing (e.g., memory and integration) and excessive navigational requirements. We (Hall et al., 2012) referred to this class of interface as GGUIs (gratuitously graphical user interfaces) because powerful graphic capabilities are essentially squandered.
Effective performance with ecological interfaces
In contrast, EID strives to replace cumbersome, analytical reasoning with a more direct coupling between perception and affordances, thereby leveraging the powerful perceptual motor capabilities of the worker. The CSE analytical tools provide templates for identifying the “deep” semantic structure of work domains (i.e., the affordances), required decisions, and related information. An effective semantic mapping is achieved by choosing visual properties (i.e., emergent features produced by analogical displays) that (1) directly reflect these underlying domain semantics and (2) are easily picked up by the observer. As a result, the worker will be able to see and act directly upon the work domain. Decision making will be supported through effective cues for action: There will be a one-to-one mapping between the affordances in the work domain and salient perceptual patterns produced by the displays. Problem solving will be enhanced because the analogical representations function as an explicit model of the process being controlled: A continuous graphical explanation of underlying affordances that will support exploration and hypothesis testing. See the original EID articles by Rasmussen and Vicente and the core EID literature described in the theoretical overview section for more complete descriptions of these and similar points.
Discussion
At this point, we believe that the utility of the EID approach has been well established for the design of visual, analogical displays which represent law-driven work domain constraints. Although EID can provide general guidance about how to frame design questions (e.g., work domain analyses) and clear design objectives (e.g., direct perception), interface design remains a creative act. Thus, the fundamental challenge here will be designing specific visual forms that achieve appropriate semantic mappings for new work domains. There are at least two additional challenges.
One major challenge for EID is to design displays for perceptual modalities other than vision. Initial progress has been made. For example, McIlroy (e.g., 2016) developed and evaluated a vibrotactile display for energy-efficient driving. Watson and Sanderson (e.g., 2007) designed ecological auditory displays for anesthesia monitoring. Giang et al. (2010) provide an overview of how EID might be applied to auditory and tactile displays. Collectively, there has been some clarification of how the EID approach can be modified to meet the associated challenges.
Perhaps the most daunting challenge for EID lies in the design of ecological interfaces for intent-driven constraints. These work domains (e.g., information retrieval, operating systems, browsers, e-mail, spreadsheets, word publishing, graphics programs, consumer devices, etc.) form an important part of the fabric of our everyday work life. Pejtersen’s seminal work (e.g., Rasmussen et al., 1994) demonstrated very clearly how EID could be successfully applied to this category of interfaces. However, as Figures 2 and 3 indicate, the evidence suggests that this area has been largely ignored by the EID community. Some notable exceptions in the EID literature include Burns and Proulx (2002), Xu, Dainoff and Mark (1999) and Huynh, Dawe, and Ayanian (2015).
EID for intent-driven work domains requires completely different techniques (see Bennett & Flach, 2011; Rasmussen et al., 1994 for detailed discussions). The affordances of the work domain are typically represented through visual metaphors (e.g., desktop icons and higher order metaphors), rather than analog displays. These metaphors must tap into existing knowledge from other, more familiar work domains (which also must have systematic parallels with the target work domain). Supporting (and switching between) alternative strategies is a much higher design priority relative to law-driven work domains.
The design of metaphors for these types of domains will have to satisfy the same semantic mapping principle that applies to analogical displays. However, designers have less control over the structure of metaphors than they have over the structure of visual analogs. The challenge is to find metaphors that are systematically aligned with the properties of the target domain without any misleading entailments. Applying EID to meet the associated challenges is a critical need.
Conclusions
The advantage of the EID approach lies in both its theoretical orientation and its practicality. The theoretical foundations for EID have a rich history reflecting the intuitions of early Functionalist psychology, Gestalt psychology, ecological psychology, and more contemporary work in applied cognitive psychology (Flach, 2017; Flach & Voorhorst, 2016). In this brief treatment we have barely scratched the surface of EID’s complexity (e.g., conceptual distinctions, analytical tools, design principles, and evaluative settings). EID is a robust approach, offering a single, integrated framework that is sufficiently comprehensive to deal with complicated work domains and to produce innovations that will generalize to actual work settings.
Key Points
Ecological interface design (EID) is an approach used to develop decision-making and problem-solving support.
This support is both grounded in the practical realities of a work domain and tailored to human capabilities/limitations.
A brief summary of EID’s theoretical foundation, concrete examples, and categories of work domains are outlined; a literature review is conducted.
The evidence reveals that EID has been remarkably successful in improving performance for some work domains but has not yet realized its potential for others.
EID is a comprehensive framework for analysis, design, and evaluation, which is capable of producing innovative support.
Supplemental Material
ATF_EID_Bennett_supplemental_references – Supplemental material for Ecological Interface Design: Thirty-Plus Years of Refinement, Progress, and Potential
Supplemental material, ATF_EID_Bennett_supplemental_references for Ecological Interface Design: Thirty-Plus Years of Refinement, Progress, and Potential by Kevin B. Bennett and John Flach in Human Factors: The Journal of Human Factors and Ergonomics Society
Footnotes
Supplementary Material
The online supplementary material is available with the manuscript on the HF website.
Kevin B. Bennett is a professor in the Department of Psychology at Wright State University. He received his PhD in 1984 from the Catholic University of America in applied-experimental psychology.
John Flach is a Senior Cognitive Systems Engineer at Mile Two LLC. He received his PhD degree in human experimental psychology from the Ohio State University in 1984.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
