Supporting Interruption Management and Multimodal Interface Design

Abstract

Objective:

The aim of this study was to integrate empirical data showing the effects of interrupting task modality on the performance of an ongoing visual-manual task and the interrupting task itself. The goal is to support interruption management and the design of multimodal interfaces.

Background:

Multimodal interfaces have been proposed as a promising means to support interruption management. To ensure the effectiveness of this approach, their design needs to be based on an analysis of empirical data concerning the effectiveness of individual and redundant channels of information presentation.

Method:

Three meta-analyses were conducted to contrast performance on an ongoing visual task and interrupting tasks as a function of interrupting task modality (auditory vs. tactile, auditory vs. visual, and single modality vs. redundant auditory-visual). In total, 68 studies were included and six moderator variables were considered.

Results:

The main findings from the meta-analyses are that response times are faster for tactile interrupting tasks in case of low-urgency messages. Accuracy is higher with tactile interrupting tasks for low-complexity signals but higher with auditory interrupting tasks for high-complexity signals. Redundant auditory-visual combinations are preferable for communication tasks during high workload and with a small visual angle of separation.

Conclusion:

The three meta-analyses contribute to the knowledge base in multimodal information processing and design. They highlight the importance of moderator variables in predicting the effects of interruption task modality on ongoing and interrupting task performance.

Applications:

The findings from this research will help inform the design of multimodal interfaces in data-rich, event-driven domains.

Keywords

meta-analysis multimodal interfaces interface design guidelines time sharing interruption management multiple resources auditory tactile redundancy

Introduction

Operators in a wide range of complex, event-driven domains, such as process control, aviation, and medicine, experience considerable attentional demands. They are required to monitor the performance of an ever-increasing number of automated systems, often resulting in data overload in the visual channel. In many cases, they also need to cope with a growing number of tasks and responsibilities. These new tasks and technologies bring with them an increased risk of interruptions of ongoing tasks and associated performance costs. The effective management of interruptions requires timely detection, accurate interpretation, and appropriate integration of interruptions while performing an ongoing task. A promising means that addresses both the challenge of data overload and the need for effective interruption management is a multimodal interface that distributes information across vision, audition, and touch (e.g., Oviatt, 2003; Sarter, 2002).

The benefit of employing multiple modalities for task and information presentation was first suggested by early research on time sharing (Navon & Gopher, 1979). This research gave rise to the multiple resource theory (MRT; Wickens, 1980, 2002, 2008), which posits that people have the ability to multitask by drawing from separate limited mental resources associated with four dimensions: processing stage, processing code, response type, and modality. With respect to the latter dimension, MRT predicts that multiple tasks and more information can be processed simultaneously if they are distributed across multiple sensory channels.

Traditionally, most information has been presented in visual form. However, the development of new tactile and auditory display technologies in the past two decades has made it possible to use nonvisual channels also. Some studies have confirmed the expected benefits of employing these modalities for interrupting tasks and messages. For example, Sklar and Sarter (1999) found that pilots on a simulated modern flight deck detected unexpected events more reliably with tactile signals than with visual signals yet performed no worse at their ongoing visual task. However, other studies have highlighted drawbacks and limitations of using or combining nonvisual sensory channels. For instance, if an interrupting task is presented in the auditory modality, it may inappropriately draw the operator’s attention away from the ongoing task (Banbury, Macken, Tremblay, & Jones, 2001; Wickens, Dixon, & Seppelt, 2005). The use of redundant modality combinations, which has traditionally been considered beneficial, can result in competition for attentional resources when the same message is presented and processed simultaneously in more than one sensory channel (Wickens, Prinet, Hutchins, Sarter, & Sebok, 2011).

These mixed findings present a challenge for designers of multimodal interfaces and motivated the present research. The aim here is to compare task performance in the context of interruption management, which Latorella (1996, 1998, 1999) defines as “the detection, interpretation, and integration of interruptions within ongoing task performance” (Latorella, 1996, p. 21). In the context of the reported meta-analyses, the ongoing task is a continuous visual task that is potentially disrupted by an interrupting task in a different modality. A prototypical scenario is a driver performing the visual-manual ongoing tasks of lane keeping and hazard monitoring who is periodically interrupted by a message from some in-vehicle device, such as a pedestrian crossing warning system.

To help designers determine which modality to use for presenting such warnings or any other potentially interrupting signal, the three meta-analyses conducted as part of this work integrated findings from numerous dual-task paradigm studies, that is, studies involving an ongoing visual-manual task and an interrupting task. Performance on both the ongoing and interrupting tasks was examined as a function of interrupting task modality. Specifically, the meta-analyses reviewed studies that compared tactile interruptions with auditory ones, visual interruptions with auditory ones, and redundant auditory-visual interruptions with auditory or visual ones.

Meta-analyses were employed because they provide numerous advantages, including seeing the “landscape” of a research domain, keeping statistical significance in perspective, minimizing wasted data, becoming intimate with the data summarized, asking focused research questions, and identifying moderator variables (Rosenthal & DiMatteo, 2001). Naturally, there are some costs of the meta-analysis technique, particularly related to the studies that are not selected for inclusion and the subjective coding of moderator variables. We discuss these potential limitations at the end of the article and note the importance of using meta-analyses and experimental results in a complementary fashion. Importantly, the analyses reported in this article employ a new meta-analytic technique, the ratio score, for synthesizing quantitative data across empirical studies (Wickens, Hollands, Banbury, & Parasuraman, 2012). Their outcomes contribute to a better understanding of multimodal information processing and help inform the design of multimodal interfaces in support of interruption management in a variety of workplaces.

Theoretical Background and Hypotheses

Prior to our research, two meta-analyses had compared information processing in the auditory and tactile modality (Burke et al., 2006; Elliott, Coovert, & Redden, 2009). However, the studies included in this work did not necessarily employ an interruption management paradigm. Thus, these meta-analyses did not provide a basis for making predictions about the relative effectiveness of audition versus touch for supporting multitasking.

Regarding the auditory and visual comparison, the original version of MRT predicts better performance if the ongoing and interrupting tasks are presented in different modalities. Specifically, with an ongoing visual task, better performance is expected if the interruption is auditory rather than visual because of less interference and competition for attentional resources (Wickens, 1980). At the same time, the opposite outcome would be expected because of auditory preemption; this term refers to the fact that, given the intrinsically more salient and disruptive nature of the auditory modality, an auditory signal is more likely than a visual one to capture and draw attention away from an ongoing visual task (Wickens & Liu, 1988; Wickens, Hutchins, Carolan, & Cummings, 2012).

With respect to auditory-visual redundancy, whereby the same information is presented in both channels, very few meta-analyses have surveyed the performance costs and benefits of redundant versus single-modality presentation for an interrupting and an ongoing task. The existing data offer no consistent conclusions (e.g., Wickens et al., 2011; Wickens & Gosney, 2003). Redundancy may result in increased accuracy; however, the dual information-processing load of reading and listening imposed by redundancy can delay the time to process information and therefore reduce efficiency. Furthermore, the added processing requirements of redundant information could result in a performance cost on the ongoing task.

To summarize, based on a review of existing work on multimodal task performance to date, the following hypotheses were formulated:

People will perform an interrupting task better when the two tasks are presented in different modalities.

With regard to performance on the ongoing visual task, no strong hypothesis can be offered, as the effect of interrupting task modality will ultimately depend on the relative strength of two offsetting factors: resource competition and auditory preemption. Our data will show which of these factors has a stronger impact on performance.

Presenting people with redundant auditory-visual interrupting tasks will lead to more accurate, but slower, performance for the interrupting task compared with the presentation of information using a single modality.

Redundancy is also expected to degrade people’s performance on the ongoing task.

Analytical Method

To test these hypotheses, three meta-analyses were conducted. Rosenthal and DiMatteo (2001) define a meta-analysis as

a methodology for (1) systematically examining a body of research and carefully formulating hypotheses, (2) conducting an exhaustive search and establishing inclusion/exclusion criteria for articles, (3) recording and statistically synthesizing the combined data and effect sizes from these studies, (4) searching for moderator variables to explain effects of interest, (5) and reporting results. (p. 62)

In the following sections, we briefly describe the method developed for the purpose of each of our meta-analyses, which closely parallels the aforementioned five steps of a typical meta-analysis but employs a new measure, ratio scores, to contrast performance for different task modalities and modality combinations (Wickens, Hutchins, et al., 2012).

Step 1: Formulating Hypotheses and Examining Available Literature

The four hypotheses we sought to examine are presented in the previous section. The general framework we adopted is that of interruption management (Steelman-Allen, McCarley & Wickens, 2011; Trafton & Monk, 2007), whereby a continuous ongoing task is potentially disrupted by an interrupting task. We conducted a literature search using an iterative three-tiered approach. First, key terms (vision or visual; audition or auditory; touch, haptics, vibrotactile, or tactile; redundant or redundancy; modality; multimodal; cross-modal) were searched in Google Scholar, a number of applied journals, and other types of publications. Some examples include Army Research Laboratory Technical Reports, Ergonomics, Human Factors, IEEE Transactions on Haptics, the International Journal of Aviation Psychology, the International Journal of Human–Computer Studies, Naval Postgraduate School Technical Reports, the Proceedings of HCI International, the Proceedings of the Human Factors and Ergonomics Society, and Transportation Research Record: Journal of the Transportation Research Board.

Second, publications that were referenced in the articles from Step 1 were reviewed. Third, the tables of contents of those publications found in Step 1 and Step 2 were examined for additional relevant articles that might not have been captured in the keyword search. Overall, of the 150 journal articles, conference proceedings, and dissertations that were identified as being of potential interest, 68 (45%) were ultimately used. They were published between 1983 and 2012. Next, criteria were established to determine which of the publications should be included in the meta-analyses.

Step 2: Establishing Inclusion Criteria

The meta-analyses compare information presentation in three modalities within multitask paradigms: auditory, tactile, and visual. Redundant information presentation was also examined but was restricted to auditory-visual redundancy (A+V) because of the scarcity of data on redundant tactile modality pairings. To be included, studies needed at a minimum to involve examination of how the interruption of an ongoing visual task by another task in the same or different modality affected performance on the interrupting task. Ongoing task performance was considered if available. For example, a study for the auditory-tactile meta-analysis might report the average response time and response accuracy to an auditory and tactile warning indicating the presence of a pedestrian while a driver performs the visual driving task. Although a considerable number of studies address modality differences within the interrupting task–ongoing task paradigm, the comparisons of interest were not always all performed within a single study. Therefore, we decided to conduct three separate meta-analyses with the modality comparisons shown in Table 1, all with ongoing visual tasks.

Table 1:

Overview of Modalities Compared in Each Meta-Analysis

Meta-Analysis	Interrupting Task Modality Comparison		Ongoing Task Modality
1. Auditory-tactile	Auditory	Tactile	Visual
2. Auditory-visual	Auditory	Visual
3. Redundant auditory + visual (A+V)	Auditory or visual^a	Redundant A+V

In most studies in which there was a redundant A+V comparison, the interrupting task was auditory; only a few employed a visual interrupting task.

In the auditory-tactile (A-T) meta-analysis, we examined studies comparing the performance effects of presenting interrupting tasks in two modalities that can be used to offload vision: audition and touch. For example, Smith, Clegg, Heggestad, and Hopp-Levine (2009) compared the use of the auditory and tactile modality for alerting and orienting attention to an interrupting gauge reading task while participants were performing the ongoing visual task of identifying whether an aircraft was hostile. Note that a visual-tactile meta-analysis was not conducted because very few studies address this comparison.

In the auditory-visual (A-V) meta-analysis, we examined a larger, historically older population of studies that compare the performance effects of visual and auditory interruptions of an ongoing visual task (e.g., Wickens, 1980; Wickens & Liu, 1988). For example, Hurwitz and Wheatley (2002) examined how the appearance of the target letter P during a visual or auditory letter monitoring task affected an ongoing visual driving task. In this analysis, the authors consider a critical variable that is not present in the A-T meta-analysis, namely, the visual angle of separation (VAS) between the ongoing visual task and the location of delivery for the visual interrupting tasks.

In the redundant (A+V) meta-analysis, we focused on redundant auditory-visual interrupting task delivery, in contrast to auditory-only or visual-only presentations. For example, Haas, Hill, Stachowiak, and Fields (2009) compared the effects of visual and auditory-visual warnings in the context of a visual robotic planning task. This third meta-analysis included many of the same articles as the A-V meta-analysis because most of the A+V redundancy studies also contained single-modality auditory and/or visual control conditions. In addition, the redundancy meta-analysis again addressed the VAS, as defined here by the separation between the ongoing task and the visual source of information in the redundant A+V delivery.

The studies included in each meta-analysis can be found in Tables 3, 4, and 5 in the Results section. They are also denoted with * for the A-T meta-analysis, # for A-V, and + for A+V in the References. Note that the in-text citations to studies included in the meta-analyses are not preceded by any distinction.

Step 3: Statistically Synthesizing the Combined Data With Ratio Scores

Typically, meta-analyses concerned with differences between two or more treatment conditions rely on the d′ or Hedge’s g measure, whereby the effect size of each study is the difference in means divided by the pooled standard error (Rosenthal & DiMatteo, 2001). An important shortcoming of the effect size score is that it is an ambiguous measure, affected not only by raw effect size (e.g., percentage difference in means) but also by sample size (N) and variance. Furthermore, not all studies report data that allow extraction of an effect size statistic (d′ or Hedge’s g) for the between-modality comparisons of interest. Therefore, we decided to employ a different measure for representing contrasts: ratio scores (Wickens, Hutchins, et al., 2012). The performance effect of interrupting task modality in each study was compared for the interrupting task itself and the ongoing task, if available. For example, in the A-V meta-analysis, the ratio would be $\frac{v i s u a l p e r f o r m a n c e}{a u d i t o r y p e r f o r m a n c e}$ , whether comparing interrupting task performance, or ongoing task performance when interrupted by either a visual or auditory interrupting task. Performance was typically assessed as response time to or accuracy for the interrupting or ongoing task. Ratio measures were employed for each experimental condition.

In our ratio calculations, “performance” was always converted to a metric such that a larger number indicated “better” performance, meaning faster and/or more accurate. Ratios greater than 1.0 corresponded to better performance for the nonauditory modalities (visual or tactile) or represented a redundancy advantage. For measures such as response time or error rate, whereby lower values indicate better performance, the ratio was calculated as such: $\frac{A}{T}, \frac{A}{V},$ and $\frac{A}{A + V} .$ If the measure was accuracy, whereby higher values indicate better performance, then the ratios were inverted as such: $\frac{A + V}{V}$ and $\frac{A}{T}$ . The performance cost or benefit of one modality versus another is directly interpretable from the ratio statistic. For example, a 1.5 ratio is 50% greater than 1.0, indicating a 50% performance benefit, whereas a 0.75 ratio is 25% less than 1.0, indicating a 25% performance cost. Table 2 shows how to interpret the performance gains for each meta-analysis.

Table 2:

Direction of Performance Gain for Each Meta-Analysis

	Ratio Performance Gain for Each Modality
Meta-Analysis	Ratio Less Than 1.0	Ratio Greater Than 1.0
1. Auditory-tactile	Auditory gain	Tactile gain
2. Auditory-visual	Auditory gain	Visual gain
3. Redundant auditory + visual (A+V)	Auditory or visual gain^a	Redundant A+V gain

In most studies in which there was a redundant A+V comparison, the interrupting task was auditory; only a few employed a visual interrupting task.

Ratio scores have a number of benefits compared with traditional meta-analytical approaches. For example, since they are based purely on mean performance differences between two conditions within a study, ratio scores allow researchers to include the results of studies that did not report effect sizes or did not provide data for calculating effect sizes. However, the ratio score method is not without limitations. When raw ratios are defined as the basic data point within the meta-analysis, then traditional statistical comparisons, such as t tests, lose statistical power if the number of studies involved in that comparison is small. In addition, as ratios are averaged, large ratios from a single study will contribute disproportionally even if the two means defining the ratio did not differ significantly. Finally, ratios may create positively skewed distributions, making it important for researchers to carefully examine their data prior to analysis. Note that a recent systematic comparison of both the traditional effect size measures and ratio scores approaches revealed a high degree of consistency between the two measures in a meta-analysis of training strategies (Wickens, Hutchins, et al., 2012).

Step 4: Searching for Moderator Variables

An additional goal of the analysis was to both identify and determine the impact of possible moderator variables, which are defined as variables that affect the relationship between two other variables, in this case, interrupting task modality and performance on the ongoing and interrupting tasks. The moderator variables were suggested by recurring themes across studies, such as workload manipulations, and by earlier research suggesting that factors such as workload, urgency, and complexity play an important role in interruption handling (e.g., Hameed, Ferris, Jayaraman, & Sarter, 2009). The specific moderators for each meta-analysis are discussed in the Results section.

Step 5: Reporting Results

Given that ratios were generated from individual studies, or from multiple conditions within a study (e.g., ratios under both low and high workload or for both ongoing task and interrupting task), the condition’s ratio itself could be treated as a single data point in the meta-analysis. For example, one source might provide one performance ratio only, whereas another source may yield several ratios reflecting multiple performance measures. For example, Straughn, Gray, and Tan (2009) compared compatible and incompatible auditory and tactile pedestrian crossing warnings wherein compatible refers to whether the “warning comes from the direction of the obstacle to be avoided” (p.1). Since response times were reported for both compatibility conditions, two response time ratios resulted for the A-T meta-analysis. These data points across studies could then be subjected to statistical analysis, in the same manner that individual participant observations are analyzed with conventional statistical tests, for example, ANOVA or a t test, to see whether there is a significant difference between modalities with different levels of a moderator variable.

Also, the nature of ratio scores allowed the mean value of a set of ratios to be compared to 1.0 to see whether one modality was significantly better or worse than the other, that is, whether 1.0 lies outside of the 95% or 90% confidence interval around the mean. For the t tests and ANOVAs that were conducted as part of this study, p values less than .10 were classified as marginally significant, and p values less than .05 were considered significant.

The following sections describe, for each meta-analysis, the number of studies and moderator variables that were included, the number and meaning of the ratios, and the results for each comparison.

A-T Meta-Analysis

A-T Method

The $\frac{A}{V}$ ratio represents an auditory performance gain for ratios less than 1.0 and a tactile performance gain for ratios greater than 1.0. Overall, 25 studies were identified, which generated 42 interrupting task response time ratios, 24 interrupting task accuracy ratios, and 12 ongoing task performance ratios (see Lu, Wickens, Sarter, & Sebok, 2011, for more details). Table 3 shows the studies and ratios used for the A-T meta-analysis.

Table 3:

Auditory-Tactile Meta-Analysis Studies, Ratio Values, and Moderator Variable Classifications

Study	Ongoing Task Description	Interrupting Task Description	Ratio Value			Workload	Decision Complexity	Urgency	Processing Code	Other Notes
			RT	Acc.	OT
Bliss, Liebman, & Brill, 2010	Navigate along route	Identifying targets	1.04	0.95			Low	High	Categorical
Bliss, Liebman, & Brill, 2010	Navigate along route	Identifying targets	1.27	0.94			Low	High	Categorical
Calhoun, Draper, Guilfoos, & Ruff, 2005	Flying	Attending to alerts	1.05		1.00		Low	Low	Categorical
Calhoun, Fontejon, Draper, Ruff, & Guilfoos, 2004	Flying, radio task	Warning response data entry task	1.00				Low	Low	Categorical
Ferris & Sarter, 2008	Command convoy, radio task	Attending to cues	0.84				High	High	Spatial	Ipsilateral
Ferris & Sarter, 2008	Command convoy, radio task	Attending to cues	1.05				High	High	Spatial	Contralateral
Ferris, Penfold, Hameed, & Sarter, 2006	Driving	Steer car away from danger	0.79	0.77		Low	High	High	Spatial	Valid cue
Ferris, Penfold, Hameed, & Sarter, 2006	Driving	Steer car away from danger	0.78	0.77		High	High	High	Spatial	Invalid cue
Fitch, Kiefer, Hankey, & Kleiner, 2007	Driving	Detect crash direction	1.15	2.69			High	High	Spatial
Glumm, Kehring, & White, 2006	Recalling info	Target location	0.97	1.06			High	High	Spatial
Ho & Spence, 2009	Driving	Avoiding collisions	0.92				Low	High	Categorical
Ho, Reed, & Spence, 2007	Driving, listening to radio	Detecting deceleration	0.89				Low	Low	Categorical
Ho, Hong, & Spence, 2006	Driving, cognitive task	Avoiding collisions	1.01	0.51	1.00		High	Low	Spatial	Valid cue
Ho, Hong, & Spence, 2006	Driving, cognitive task	Avoiding collisions	0.81	0.64			High	Low	Spatial	Invalid cue
Ho, Hong, & Spence, 2005	Cognitive task	Avoiding collisions	1.12	1.01	1.00		High	Low	Spatial	Front cue
Ho, Hong, & Spence, 2005	Cognitive task	Avoiding collisions	1.06	1.03			High	Low	Spatial	Back cue
Krausman, Elliott, & Pettitt, 2005	“Soldiering”	Message cuing	0.79				Low	Low	Categorical
Larkin, 1983	Flying	Attending to alerts	1.46				Low	Low	Categorical
Lee, McGehee, Brown, & Marshall, 2006	Driving	Avoiding collisions	0.89				Low	Low	Categorical
Merat & Jamson, 2008	Driving	Detection task	1.09				Low	Low	Categorical
	Driving, counting cues		1.07
	Driving, counting cues		1.16
Mohebbi, Gray, & Tan, 2009	Engaging in conversation	Avoiding collisions	1.02	1.19		—	Low	Low	Categorical	No conversation
			1.19	1.51		Low				Simple conversation
			1.08	1.32		High				Complex conversation
Mortimer, 2006	Shadowing sentences	Target task	1.01		1.00	Low	High	Low	Spatial
			1.05		1.00	High
			1.26			Low
			1.29			High
Oskarsson, Eriksson, & Carlander, 2012	Driving	Threat response	1.20				Low	High	Categorical
Savick, Elliott, Zubal, & Stachowiak, 2006	Game navigation	Robotic navigation	1.19	1.02			High	Low	Categorical
Scott & Gray, 2008	Driving	Attending to headway alerting	1.14	1.47		Low	Low	Low	Categorical	5-s time to crash
Scott & Gray, 2008	Driving	Attending to headway alerting	1.09	1.47		High	Low	Low	Categorical	3-s time to crash
Smith, Clegg, Heggestad, & Hopp-Levine, 2009	Aircraft task	Gauge task	1.04	0.99	0.99		Low	Low	Categorical
Smith, Clegg, Heggestad, & Hopp-Levine, 2009	Aircraft task	Gauge task	0.92	1.00	1.00		High	Low	Spatial
Stanley, 2006	Driving, memorizing material	Avoiding collisions	1.34	1.00	1.06		High	Low	Categorical
Stanley, 2006	Driving, memorizing material	Avoiding collisions	1.28	0.36	1.14		High	Low	Categorical
Straughn, Gray, & Tan, 2009	Driving	Attending to pedestrian warning	1.19	1.30			High	Low	Spatial	Compatible
Straughn, Gray, & Tan, 2009	Driving	Attending to pedestrian warning	1.23	1.21			High	Low	Spatial	Incompatible
Tilak et al., 2008	Driving	Attending to cues	0.97	0.62		Low	High	High	Spatial	Valid cue
Tilak et al., 2008	Driving	Attending to cues	0.98	0.62		High	High	High	Spatial	Invalid cue
Wickens, Small, Andre, Bagnall, & Brenaman, 2008	Flying	Unusual attitude recovery command	1.00				High	High	Spatial
Wickens, Small, Andre, Bagnall, & Brenaman, 2008	Flying	Unusual attitude recovery command	1.00				High	High	Spatial

Note. RT = interrupting task response time; Acc. = interrupting task accuracy; OT = ongoing task performance.

In addition to the main modality effects, the possible impact of the following moderator variables was analyzed:

Ongoing task workload (high vs. low). Low and high workload were extracted from the individual studies in which the factor was specifically manipulated within the experiment.

Interrupting task decision complexity (level of uncertainty within the signal). For low-complexity interruptions, such as a general warning, the interrupting task simply informs the operator of the occurrence of an event (zero bits of information). For high-complexity interruptions, the interrupting task requires some choice of action, such as turning left or right, and informs the operator of a set of possible events (e.g., more than than zero bits of information).

Interrupting task urgency (alarm vs. notification). An alarm requires an immediate response to a critical task or event, whereas a notification informs the participant of a task or event that can be postponed. For example, an alarm can be a warning of an impending collision in an automobile or aircraft, and an example of a notification is the need to check tire pressure when feasible. Thus, an alarm was classified as high urgency and a notification as low urgency.

Interrupting task processing code (spatial vs. categorical). The former “relates to spatial relationships between stimulus components such as left-right,” whereas categorical information “refers to the extracted information [that] has symbolic meaning or refers to identity within a category” (Ferris & Sarter, 2010).

A-T Results

Interrupting task performance.

The mean ratio score for interrupting task response time was 1.06 (n = 42 ratios; range = 0.78 to 1.46). This value is significantly greater than 1.0 (α = .05), indicating that averaged across all conditions, tactile interruptions are responded to 6% faster than auditory interruptions. Of the total 42 ratios, 28 (67%) showed this response time advantage for tactile versus auditory interruption tasks. The analysis of interrupting task accuracy data yielded a mean ratio of 1.06 (n = 24 ratios; range = 0.36 to 2.69), indicating a marginally significant tactile advantage (α = .10).

Ongoing task performance

All studies included in this meta-analysis employed an ongoing task, but only seven of them reported ongoing task performance data; and these generated nine ratios. Smith et al. (2009) and Stanley (2006) both generated two ongoing task ratios (see Table 3 for the ongoing task ratios from each study). Since there were not many ongoing task ratios, response time and accuracy were both considered and pooled together. The mean ongoing task performance ratio for these studies was 1.02 (range = 0.99 to 1.14), and this value was not significantly greater than 1.0 (α = .10).

Ongoing task workload

Workload was varied in only 4 of the 25 studies. A pairwise t test showed that the interrupting task response time ratios for low and high workload (ratio = 1.11 and ratio = 1.10, respectively) did not differ significantly from each other, t(8) = 0.16, p = .88. The effect of workload on interrupting task accuracy was not examined in this meta-analysis because only 1 of the 4 studies that varied workload reported interrupting task accuracy data for each modality-workload combination (Mohebbi, Gray, & Tan, 2009).

Interrupting task decision complexity (level of uncertainty within the signal)

There was no significant difference for response time between high and low complexity (ratio = 1.06 and ratio = 1.07, respectively), t(39) = 0.36, p = .72. This equivalence also holds true when comparing only those studies that reported within-experiment differences between modalities. However, when we excluded the one outlier ratio that was more than three standard deviations from the mean, for the studies that reported interrupting task accuracy (n = 12 studies), there was a significant difference in accuracy favoring the tactile modality at low complexity (ratio = 1.14) and the auditory modality at high complexity (ratio = 0.86), t(21) = 2.57, p = .02.

Interrupting task urgency

We found that 16 studies focusing on the presentation of low-urgency signals (notifications) produced a tactile advantage for response time (ratio = 1.09), whereas the 9 studies examining high-urgency signals (alarms) showed neither a tactile or auditory advantage (ratio = 1.00). This difference between alarms and notifications ratios in response time was significant, t(23) = 1.99, p = .05). Further evidence of the difference is provided by the following. Of the notification studies, 14 reported significant within-experiment response time differences between modalities, and 12 of these studies showed a significant tactile advantage (86%). Regarding interrupting task urgency and accuracy, there was no significant difference between alarms (ratio = 1.17) and notifications (ratio = 1.04), t(5) = 0.42, p = .69.

Processing code (spatial vs. categorical)

Of the 40 ratios in this analysis, 22 were classified as spatial (ratio = 1.06) and the remaining 18 were classified as categorical (ratio = 1.07). For response time, the difference between the mean interrupting task ratios was not significant, t(39) = 0.36, p = .72. Again, when we excluded the one outlier ratio that was more than three standard deviations from the mean, there was a significant difference between spatial and categorical cues (0.88 and 1.20, respectively), t(17) = 2.24, p = .04, for accuracy, as spatial cues were more accurate with audition and categorical cues with touch.

A-V Meta-Analysis

A-V Method

In this analysis, the $\frac{A}{V}$ ratio represents an auditory advantage for ratios that are less than 1.0 and a visual advantage for ratios greater than 1.0. There were 29 studies included in this meta-analysis. Overall, the 29 studies generated 46 interrupting task response time ratios, 22 interrupting task accuracy ratios, 5 ongoing task response time ratios, and 33 ongoing task performance ratios. Table 4 shows the studies and ratios used for the A-V meta-analysis.

Table 4:

Auditory-Visual Meta-Analysis Studies, Ratio Values, and Moderator Variable Classifications

Study	Ongoing Task Description	Interrupting Task Description	Ratio Value			Decision Complexity	Urgency	Processing Code	Visual Angle of Separation	Interrupting Task Permanence
			RT	Acc.	OT
Bliss, Liebman, & Brill, 2010	Navigating along route	Identifying targets	1.35		0.94 0.92	Low	High	Categorical	15	Transient
Bronkhorst, Veltman, & van Breda, 1996	Following aircraft	Finding target	0.92			High	Low	Spatial	0	Transient
Chen, Zeng, & Kao, 2005	Following vehicle	Responding to cues	2.57			High	High	Categorical	0	Permanent
Chen, Zeng, & Kao, 2005	Following vehicle	Responding to cues	1.64			High	High	Categorical	0	Permanent
Colcombe & Wickens, 2006	Tracking task	Answering questions	0.93	0.64	1.00	Lo/Hi	High	Spatial	15	Transient
			0.88		0.96
			1.05	0.98	1.04
					0.98
Dingus, McGehee, & Hankey, 1997	Driving	Noticing warnings		1.05		Low	Lo/Hi	Spatial	15	Permanent
Dixon, Wickens, & Chang, 2005	Flying unmanned aerial vehicles	Monitoring for system failures	0.5			High	High	Categorical	5
Dixon, Wickens, & Chang, 2005	Flying unmanned aerial vehicles	Monitoring for system failures	0.36			High	High	Categorical	5
Ferris & Sarter, 2008	Command convoy, radio task	Attending to cues	0.97				High	Categorical	20
Finomore, Popik, Castle, & Dallman, 2010	Military monitoring task	Monitoring for critical phases	0.36	1.2, 1.06		High	High	Spatial/categorical
Finomore, Popik, Castle, & Dallman, 2010	Military monitoring task	Monitoring for critical phases	0.40	1.45, 1.20		High	High	Spatial/categorical
Gish, Staplin, Stewart, & Perel, 1999	Driving, navigation, avoiding targets	Attending to guidance info	0.93	1.11	0.91	Low	Low	Spatial	5/20	Transient
Gish, Staplin, Stewart, & Perel, 1999	Driving, navigation, avoiding targets	Attending to guidance info	0.89	0.77	0.89	Low	Low	Spatial	5/20	Transient
Glumm, Kehring, & White, 2006	Target location	Recalling information	1.33		1.08	High	High	Spatial	15	Transient
Glumm, Kehring, & White, 2006	Target location	Recalling information	1.01		1.12	High	High	Spatial	15	Transient
Helleberg & Wickens, 2003	Flying	Air traffic control task	1.26		1.28	High	Low	Spatial	13	Transient
			3.30		1.10
					1.12
Ho, Hong, & Spence, 2005	Driving, cognitive task	Avoiding collision	0.70	0.99		Low	High	Spatial	15	Transient
Ho, Nikolic, Waters, & Sarter, 2004	Air traffic control task	Noticing cues	1.05	0.99		High	Low	Categorical	5	Permanent
Hurwitz & Wheatley, 2002	Driving	Noticing letter p			0.84	Low	High	Spatial	25	Transient
Hurwitz & Wheatley, 2002	Driving	Noticing letter p			0.92	Low	High	Spatial	25	Transient
Iani & Wickens, 2007	Flying	Weather changes	0.33			High	Low		0	Permanent
Krausman, Elliott, & Pettitt, 2005	Platoon leader task	Responding to messages	0.48			High	High	Categorical	15	Permanent
Larkin, 1983	Flying	Attending to alerts	0.47			Low	Low	Categorical	0	Transient
Latorella, 1998	Air traffic control direction	Flight Management System task	0.80			Low	Low	Spatial
Latorella, 1998	Air traffic control direction	Flight Management System task	0.81			Low	Low	Spatial
Liu, 2001	Driving, navigation	Button task	0.83	0.80	0.31	Lo/Hi	High	Spatial	6	Transient
Liu, 2001	Driving, navigation	Button task	0.64	0.64	3.45	Lo/Hi	High	Spatial	6	Transient
Merat & Jamson, 2008	Driving	Detection task			0.93	Low	Low	Spatial	25	Transient
Merat & Jamson, 2008	Driving	Detection task			0.90	Low	Low	Spatial	25	Transient
Mollenhauer, Lee, Cho, Hulse, & Dingus, 1994	Driving	Attending to road signs	1.06	0.94		Low	Low	Categorical	6	Transient
Ratwani, Andrews, Sousk, & Trafton, 2008	Sea vessel task	Three additional problems		1.03	0.80	High	High	Categorical	0	Transient
Scott & Gray, 2008	Driving	Attending to headway warning	0.93	0.77		Low	Low	Categorical	10	Permanent
Scott & Gray, 2008	Driving	Attending to headway warning	0.83	0.77		Low	Low	Categorical	10	Permanent
Srinivasan, 1997	Driving, noticing shape changes	Attending to guidance info			1.01	Low	Low	Spatial	0/15	Transient /permanent
Srinivasan, 1997	Driving, noticing shape changes	Attending to guidance info			1.01	Low	Low	Spatial	0/15	Transient /permanent
Srinivasan, Yang, Jovanis, Kitamura, & Anwar, 1994	Driving	Attending to guidance info	1.02			Low	Low	Spatial	0/20	Transient
Srinivasan, Yang, Jovanis, Kitamura, & Anwar, 1994	Driving	Attending to guidance info	0.87			Low	Low	Spatial	0/20	Transient
Walker, Alicandri, Sedney, & Roberts 1991	Driving	Attending to guidance info	1.05	0.87		Low	Low	Spatial	15	Transient
			1.05	0.43
			1.11	0.92
			0.60	0.94
			1.35
Whitmire, Morgan, Oron-Gilad, & Hancock, 2010	Driving	Attending to warning messages			1.09	Low	Low	Spatial	15	Transient
Whitmire, Morgan, Oron-Gilad, & Hancock, 2010	Driving	Attending to warning messages			3.30	Low	Low	Spatial	15	Transient
Wickens, Dixon, & Seppelt, 2002, 2005	Vehicle tracking	Digit entry task	0.50		1.00	High	High	Categorical	0	Transient
			0.50		1.00
			0.44		0.85
			0.42		0.70
			0.34		1.54
Wickens, Sandry, & Vidulich, 1983	Tracking task	Responding to series of letters	1.01	1.10, 1.01		High	High		15	Transient
		Responding to series of letters	1.30	1.23
	Flying	Target localization	1.00	1.03

Note. RT = interrupting task response time; Acc. = interrupting task accuracy; OT = ongoing task performance.

The same three moderator variables, aside from workload, employed in the A-T analysis were examined for this meta-analysis in addition to the following two:

Auditory permanence (permanent vs. transient). We examined differences between a relatively permanent (e.g., a repeated tone) versus a highly transient tone.

VAS. The angle of separation is measured by the number of degrees between the ongoing task’s center of focus and the interrupting task’s visual display. It was hypothesized that the larger this angle, the greater the visual cost (lower ratio) because of the increased scan required between the interrupting task and ongoing task. If this angle was not directly reported in the article, we estimated it from the geometry of the separation between information sources on the screen (e.g., 20 cm) and the typical seating distance from the screen in most experimental settings (i.e., approximately 60 cm). However, in nearly all studies, this information either was reported or could be estimated from figures depicting the experimental setup.

A-V Results

Interrupting task performance

The overall mean response time generated from 46 ratios was 0.88 (ratio range = 0.34 to 2.57). This value is significantly less than 1.0 (α = .05), showing a clear auditory advantage. The analysis of interrupting task accuracy data resulted in a mean ratio of 1.01 (n = 24 ratios; range = 0.33 to 3.30). This ratio was not significantly different from 1.0 (α = .10).

Ongoing task performance

Since there were only five response time ratios with respect to ongoing task performance, response time and accuracy ratios were pooled together to calculate the mean ongoing task ratio. The overall mean ratio was 1.13 (n = 33 ratios; ratio range = 0.31 to 3.54) was not significantly different from 1.0 (α = .10), indicating that the ongoing task was unaffected by interrupting task modality.

Interrupting task decision complexity (level of uncertainty within the signal)

There was a marginally significant difference between the 16 low-complexity ratios (ratio = 1.03) and the 30 high-complexity ratios (ratio = 0.82) with regard to response time, t(44) = 1.78, p = .08: While one performs a visual ongoing task, complex auditory events are processed faster than complex visual ones. There was no effect of signal complexity with regard to accuracy (low-complexity ratio = 0.86, n = 8; high-complexity ratio = 0.96, n = 15), t(21) = 0.91, p = .37.

Interrupting task urgency

The 13 interrupting task notification ratios and 33 interrupting task alarm ratios showed no significant difference for response time, t(44) = 0.31, p = .76, or accuracy, t(18) = 1.00, p = .33.

Processing code (spatial vs categorical)

We classified 38 ratios in this analysis as spatial (ratio = 0.91) and another 30 ratios as categorical (ratio = 0.82). Processing code had a marginally significant effect on response time such that categorical cues were best presented with audition (ratio = 0.72), whereas for spatial cues, the auditory benefit was diminished although still marginally significant (ratio = 0.89), t(29) = 1.77, p = .09. Regarding accuracy, no significant difference between modalities was observed for spatial (ratio = 0.91) or categorical cues (ratio = 1.01), t(12) = 0.67, p = .52.

Auditory interrupting task permanence

The permanence of the auditory interrupting task had a marginally significant effect on the ratio. If it was fairly permanent (e.g., a repeated tone; mean ratio = 0.75), auditory interrupting task performance was better than if it was transient (mean ratio = 0.95), t(7.28) = 1.93, p = .09. For the transient signal, the ratio was not significantly less than 1.0, and thus there was no auditory benefit.

VAS

We performed a regression analysis on the combined response time and accuracy ratios against the VAS. The slope was not significantly different from zero, indicating that visual interrupting task performance costs did not increase with eccentricity (r = 0.13, p < .22).

Redundancy (A+V) Meta-Analysis

A+V Method

In the redundancy meta-analysis, we examined 31 redundant A+V versus single-modality (auditory or visual) studies. Table 5 shows the studies and ratios for the A+V meta-analysis. The best of the two single-modality conditions was used in all cases. This single-modality baseline definition was chosen, because only with such a baseline can we assure that human information processing is truly exploiting redundancy and not just filtering the poorer of the two single modalities (see Wickens & Gosney, 2003). A redundancy ratio greater than 1.0 indicated a redundancy gain and one less than 1.0 indicated a redundancy cost. The three moderator variables that were included in this analysis were ongoing task workload, interrupting task type (communications, alert, or spatial), and VAS.

Table 5:

Redundancy (Auditory Plus Visual) Meta-Analysis Studies, Ratio Values, and Moderator Variable Classifications

Study	Ongoing Task Description	Interrupting Task Description	Ratio Value			Workload	Urgency (Interrupting Task Type)	Visual Angle of Separation	Other Notes
			RT	Acc.	OT
Almen, 2002	Driving	Reading numbers, obstacle detection	1.08			High	Alerts or warnings		Obstacle 1
Almen, 2002	Driving	Reading numbers, obstacle detection	0.96			High	Alerts or warnings		Obstacle 2
Belz, Robinson, & Casali, 1999	Driving	Collision avoidance alert	0.99			High	Alerts or warnings	15
Bouis, Voss, Geiser, & Haller, 1979	Driving	Responding to alerts	0.50			High	Alerts or warnings	20
Bronkhorst, Veltman, & van Breda, 1996	Flying	Target detection	1.09			High	Location cuing	0
Dingus, McGehee, & Hankey, 1997	Driving	Respond to braking events	1.03			High	Alerts or warnings	15
Dowell & Shmueli, 2008		Responding to questions		0.98		Low
Dowell & Shmueli, 2008		Responding to questions		0.94		High
Haas, Hill, Stachowiak, & Fields, 2009	Robotic planning task	Responding to signals	1.12			Low	Alerts or warnings	35
Helleberg & Wickens, 2003	Flying	Attending to air traffic control clearance info	0.75		0.89	High	Communication task	14.4
Helleberg & Wickens, 2003	Flying	Attending to air traffic control clearance info				High	Communication task	14.4
Ho, Reed, & Spence, 2007	Driving	Responding to critical events	1.09			High	Alerts or warnings
Kaber, Wright, & Sheik-Nainer, 2006		Mine disposal task	1.05
Kramer, Cassavaugh, Horrey, Becic, & Mayhugh, 2007	Driving	Respond to warning	1.16	2.44		High	Alerts or warnings	6	Forward collisions
Kramer, Cassavaugh, Horrey, Becic, & Mayhugh, 2007	Driving	Respond to warning	1.16	2.00		High	Alerts or warnings		Side collisions
Lancaster & Casali, 2008	Flying	Air traffic control messages	1.05			High	Communication task	15
Lancaster & Casali, 2008	Flying	Air traffic control messages	0.76			High	Communication task	15
Lee & Spence, 2008	Driving	Mobile phone task	1.18		1.05	High	Communication task	35
Lewandowki, Hursh, & Kobus, 1985		Braking events	0.91			Low			Matching words
Lewandowki, Hursh, & Kobus, 1985		Braking events	0.83			High			Nonmatching words
Liu, 2001	Driving	Route guidance, push button task	1.04	3.03		Low	Alerts or warnings	6	Low complexity
Liu, 2001	Driving	Route guidance, push button task	1.18	2.44		High	Alerts or warnings	6	High complexity
Maltz & Shinar, 2004	Driving	Responding to alerts	1.00			Low			Danger zone
Maltz & Shinar, 2004		Responding to alerts	1.03						Safe zone
Mortimer, 2006	Targeting task	Shadowing sentences	1.08			Low	Location cuing		Experiment 1
Mortimer, 2006	Targeting task	Shadowing sentences	1.08			High	Alerts or warnings		Experiment 2
Navarro, Mars, & Hoc, 2007	Driving	Responding to lane deviations			0.78	High	Alerts or warnings		Alert in bends
Navarro, Mars, & Hoc, 2007	Driving	Responding to lane deviations			1.23	Low	Alerts or warnings		Alert in straight roads
Oskarsson, Eriksson, Lif, P., Lindahl, & Hedström, 2008	Attending to radio calls	Attending to threat popups	0.98	1.15	0.93	High	Location cuing	90
Perry, Stevens, Wiggins, & Howell, 2007	Performing math tasks	Identifying warnings	0.89	1.02		Low	Alerts or warnings	0	Complex math
Perry, Stevens, Wiggins, & Howell, 2007	Performing math tasks	Identifying warnings	0.94	1.00			Alerts or warnings		Simple math
Santamaria & Quesada, 2007	Driving	Attending to messages	1.28			Low	Alerts or warnings	15
Seagull, Wickens, & Loeb, 2001	Tracking task	Monitoring for parameter changes	1.04		0.95	Low	Alerts or warnings	0.5
Seppelt & Wickens, 2003	Driving	Attending to messages	0.75		1.00	High	Alerts or warnings	23.9
Seppelt & Wickens, 2003	Driving	Attending to messages	0.54		0.96	High	Alerts or warnings	23.9
Sklar & Sarter, 1999	Flying	Monitoring for mode changes	0.92	1.00		High	Communication task	15
Stanley, 2006	Driving	Responding to warning	0.80	0.94		Low	Alerts or warnings
Stanley, 2006	Driving	Responding to warning				High	Alerts or warnings
Steelman-Allen et al., 2010	Flying	Air traffic control messages	0.57	1.00	1.00	High	Communication task	15/35	Left visual field
Steelman-Allen et al., 2010	Flying	Air traffic control messages	0.64		0.88	High	Communication task	15/35	Right visual field
Stone, Bisantz, Llinas, & Paquet, 2009	Land detection	Operating robot	1.01
Van Erp & Van Veen, 2004		Peripheral detection task	1.10		0.99	Low	Location cuing	45	High-gain steering
Van Erp & Van Veen, 2004		Peripheral detection task	1.10		1.01	High	Location cuing	45	Low-gain steering
Van Erp & Van Veen, 2001		Peripheral detection task	0.99	0.95	0.97	High	Location cuing	45
Wickens, Goh, Helleberg, Horrey, & Talleur, 2003	Flying	Air traffic control instructions	0.67	10.00	1.00, 0.94	Low	Communication task	14.4	Lateral error
Wickens, Goh, Helleberg, Horrey, & Talleur, 2003	Flying	Air traffic control instructions			1.05, 0.78	High	Communication task	14.4	Vertical error
Wickens & Gosney, 2003	Tracking	Digit recall task	0.46		0.83	Low	Communication task	32.5	7-digit recall task
			0.44		0.91				4-digit recall task
			0.77		0.85				7-digit recall task
			0.75		0.86				4-digit recall task

Note. RT = interrupting task response time; Acc. = interrupting task accuracy; OT = ongoing task performance.

A+V Results

Interrupting task performance

The overall ratio for the interrupting task generated by 49 ratios was 0.97, which was not significantly less than 1.0 (α = .10). This indicates that redundant A+V presentation was on average as good as, but not better than, the best of the single-modality conditions, which usually was audition. However, this effect was qualified by a number of moderator variables as described later, some of which produce a true redundancy gain, and some actually illustrate a redundancy cost relative to the single-modality auditory condition.

There was a significant redundancy gain for accuracy (ratio = 1.34, α = .05), but a significant redundancy cost for response time (ratio = 0.83, α = .05). This large difference was important but was not surprising in that for most systems, redundancy helps guarantee security (by providing more ways for the information to be noticed) but at the expense of efficiency. To the extent that humans do not process visual and auditory information entirely in parallel (Wickens, 2002), this added cost of dual-channel processing will lead to a response time penalty. This penalty also manifests as time away from the ongoing task, which may explain the overall 7% redundancy cost for the ongoing task described next.

Ongoing task performance

For the ongoing task, the ratio was 0.93 calculated from 48 ratios, which was significantly less than 1.0 (α = .05), indicating that on average, there was a small redundancy cost to the ongoing task.

Ongoing task workload

When data were pooled across both response time and accuracy, ongoing task workload affected the redundancy gain for interrupting task communications information (e.g., data link) but not for other types of tasks, F(47) = 4.99, p < .01. Specifically, for interrupting communications tasks, there was a significant redundancy gain under high ongoing task workload (ratio = 1.67) but not low workload (ratio = 0.71; not significantly less than 1.0). In contrast, for the other two task types, workload did not alter the redundancy effect.

Interrupting task type

For response time, there was a significant interrupting task type interaction, F(2) = 4.20, p = .02, with a marginal significant redundancy cost for communication tasks, such as text–voice data link messages (ratio = 0.85). However, there was a marginal redundancy gain for alerting tasks (ratio = 1.06) and a significant gain for spatial tasks (ratio = 1.06). The accuracy ratio did not differ for task type.

VAS

The separation between the primary visual display of the ongoing task and the visual component of the interruption task affects the redundancy gain. As shown in Figure 1, when these channels are close together but not overlaid, there was a clear redundancy gain at 5° (ratio = 1.56). However, when the visual channels are more separated, this effect regresses through 1.0, and a regression analysis yields a significant slope (r = .29; p < .01). Note, however, this slope does not include the points at 0 VAS because they are qualitatively different and involve clutter from the overlay of multiple displays (Horrey & Wickens, 2004). The analysis reveals that at wide visual angles, there was a redundancy cost. Because the best single task modality in these experiments was always auditory, such a regression essentially means that the visual component of the redundant A+V interrupting task information was either ignored or processed at a cost when it was widely separated from the ongoing task.

Figure 1.

Ratio of the redundancy condition to the best single task modality as a function of visual angle of separation (ratio values greater than 1.0 equals a redundancy gain).

Summary of Findings across All Meta-Analyses

Figure 2 provides a summary of the significant findings across the three meta-analyses. The findings are discussed in further detail in the following section.

Figure 2.

Summary of marginally significant and significant findings across the meta-analyses in order of appearance: Auditory-tactile results are the in the top white section, auditory-visual results are in the shaded gray section, and redundant auditory-visual results are in the bottom white section. Arrows indicate modality gain direction for each analysis.

Discussion

Operators in complex, data-rich domains experience visual data overload and an increased need for effective interruption management. Multimodal interfaces, which combine visual, auditory, and tactile information presentation, have been proposed as a promising means to address those challenges with processing multiple tasks or sources of information simultaneously. However, not enough is known about the relative benefits and shortcomings of employing and combining the various channels. To help fill this gap, three meta-analyses were conducted concerning the effects of interrupting task modality, for example, visual, auditory, tactile, or redundant auditory-visual, on the performance of an ongoing visual-manual task and the interrupting task itself. The impact of several moderator variables, such as workload and complexity, was examined as well.

Primary Hypotheses

Four primary hypotheses were proposed. First, on the basis of the strong predictions of MRT, we proposed that in the context of visual ongoing tasks, a nonvisual interrupting task would be processed more effectively than a visual one. This effect was directly confirmed in the A-V meta-analysis, but it was also indirectly confirmed in the A-T analysis, in which tactile processing was found to be, on average, even more efficient than auditory processing when it was imposed during an ongoing visual task. Hence, by extrapolation, we would expect a tactile interrupting task to be processed better than a visual interrupting task, even though there were insufficient head-to-head comparisons of these two modalities to avail a meta-analysis (but see Sklar & Sarter, 1999). The finding of nonvisual superiority of the interrupting task is also fully consistent with auditory preemption theory. Finally, we note that the cost to the visual interrupting task can be only partially explained by the peripheral effects of visual scanning, because a larger VAS did not significantly increase this cost.

Our second hypothesis, regarding the ongoing task, was weaker but still confirmed. That is, for the ongoing task, we proposed that the auditory benefits of separate resources would be offset by the auditory costs of preemption. Such weakening was clearly confirmed. Ongoing task performance was found to differ not significantly or at all (ratio = 1.02) between interrupting task modalities. This “absence of effect” was demonstrated experimentally by Wickens and Liu (1988) and Wickens et al. (2005), but the meta-analysis provided the added statistical power to confirm the null hypothesis.

Thus, on balance, the findings related to the first two hypotheses provide further evidence that when used appropriately, auditory signals can support multitasking in a nonintrusive manner (Kramer, 1994) and lead to a net gain in performance across multiple tasks.

The third hypothesis, concerning redundancy, was also confirmed. A statistically significant 34% redundancy gain was found for accuracy, and a 13% (ratio = 0.87) redundancy loss was observed for response time. The former reflects the increased security resulting from processing the same information in multiple channels, and the latter indicates the increased time that that processing requires compared with the single modality, which was almost always an auditory display.

This finding of a speed–accuracy trade-off with redundancy leads us to the fourth hypothesis: the predicted time cost of processing a visual interrupting task when it is coupled with an auditory display. Scanning such visual information presumably caused the small but significant 7% cost in ongoing task performance.

Finally, although not offered as a specific hypothesis, the A-T comparison yielded important findings as well. In particular, the A-T analysis showed that in many cases, participants responded even faster to tactile interrupting tasks than to auditory ones. This finding appears to contradict earlier recommendations not to use tactile cues alone when response time is critical (e.g., Hale & Stanney, 2004). However, a closer look at the data indicates that this contradiction is resolved when moderator variables are considered. The tactile advantage to response time vanished in the case of urgent interruptions, and the auditory presentation resulted in more accurate responding for more-complex signals. Our study confirms the findings from a study conducted by Wogalter, Conzola, and Smith-Jackson (2002), which demonstrated that audition is a powerful means of getting attention and effective in producing an alerting reaction. Second, it confirms the findings from the Chang, O’Modhrain, Jacob, Gunther, and Ishii (2002) study, that without considerable training, it is difficult for users to interpret complex tactile signals or tactons. This difficulty is reflected not only in response time but also in terms of accuracy, which was found to be lower for tactile signals with high signal complexity.

Additional Moderator Variables

VAS

The lack of effect of VAS was unexpected. However, before we truly conclude that VAS does not matter, we note that (a) the data points were highly variable, so it is not appropriate to confirm the null hypothesis; (b) the function beyond 15° does show visible increase in visual costs; and (c) when the three studies that varied VAS within the experiment were examined, all three showed a consistent and significant monotonic decrease in interrupting task performance with increasing VAS (e.g., Wickens, Dixon, & Seppelt, 2002).

Task type

The effect of task type was complex and complicated by the fact that task type could not be coded in the same way across all three meta-analyses. For example, there were no studies involving communications tasks in the tactile modality that affected the A-T meta-analysis. However, certain effects did emerge. When the auditory and tactile modalities were compared, the auditory interrupting task had better accuracy than the tactile ones for spatial tasks, hence confirming the fluency of this modality for conveying spatial information (Begault & Pittman, 1996). This auditory spatial advantage was also confirmed by the fact there was no loss in accuracy. However, when the two modalities were contrasted for categorical tasks, the tactile modality yielded better accuracy, with no loss in speed.

In partial contrast, when the auditory and visual interrupting tasks were compared, the auditory modality now emerged superior for categorical tasks with respect to response time but with no modality difference for spatial tasks for both response and accuracy. The auditory speed advantage for categorical tasks possibly reflects the natural or compatible mapping between sound and language (Wickens, Sandry, & Vidulich, 1983), as many such tasks involved simple linguistic processing. But this visual advantage disappears when spatial tasks are used, often naturally mapping to the inherent spatial property of the visual system.

Finally, in the redundancy meta-analysis, we found that redundancy slowed the processing of communications task information relative to the single modality, which was almost always auditory. The slower response times may be attributed to the added time required to read the visual text component of the redundant information. In contrast, the relatively less complex, shorter messages of information inherent in the alerts and spatial tasks were not slowed by the added visual task and even resulted in a faster response time.

Applied Implications

Overall, the findings from the three meta-analyses highlight the fact that, rather than focusing on overall performance differences between modalities and modality combinations, it is critical to consider the effects of moderator variables when developing recommendations for the design of future multimodal and possibly adaptive interfaces. For example, redundant information presentation is beneficial for communication tasks only in case of high workload, and even then, only when accuracy is most important. In low-workload conditions, redundancy leads to improved performance only for alerting and spatial tasks. A+V redundancy is recommended only when the VAS between the visual ongoing task and the visual aspect of the interrupting task is small. Tactile messages lead to improved performance compared with audition for low-complexity and low-urgency messages. Thus, the sense of touch should be reserved for simple notifications. In contrast, the auditory channel results in better performance when a message is complex and urgent, suggesting the use of this channel for alarms and alerts. Also with regard to accuracy, the tactile modality is recommended for categorical tasks, whereas audition is recommended for spatial tasks. The importance of moderator variables strongly suggests a need for adaptive interface designs (Sarter, 2007; Scerbo, 1996; Trumbly, Arnett, & Johnson, 1994) whereby the nature of information presentation is varied depending on context in an effort to optimize information processing performance.

Limitations and Future Directions

There are a number of limitations to the approach taken for this study. We begin by describing those related to the meta-analytic approach in general. First, ideally, a meta-analysis should be analogous to a factorial experimental design, whereby moderator variables in the meta-analysis correspond to factors in the design and interactions between moderator variables can be examined in the same way as interactions between factors. However, in reality, this is rarely the case. Unlike in an experiment in which all cells are equally populated, in the meta-analysis, we are at the mercy of the population of studies available. At best, overlapping sets of studies will include examination of each moderator variable of interest.

Second, even when there are multiple studies involving a particular moderator variable, it is possible that two or more levels of the variable may be confounded with another variable. For example, suppose that all or most studies of high complexity involved communications tasks, and all or most of low-complexity studies involved alerting tasks. It would be difficult to establish the extent to which any difference was attributable to task type or to task complexity.

Third, as Simmons, Nelson, and Simonsohn (2011) have noted, there are multiple sources of bias created by the “researcher degrees of freedom,” or the biases that are associated with the decisions that researchers have to make when collecting and analyzing data. These biases can come in two forms:

We can be biased on how we coded moderator variables of the studies included and, indeed, what moderator variables we chose to identify in the first place (our justification was articulated in the Introduction).

The number of potential studies included will be biased downward by what is referred to the “file drawer problem” (Rosenthal, 1979), which refers to the fact that a number of valid studies in a given area of research may be conducted but never reported, in part because of a bias toward reporting the presence rather than the absence of effects.

Fourth, we note that our meta-analysis employed the less conventional ratio analysis, as opposed to the effect size analysis, and the potential limitations of the former were previously described.

These limitations notwithstanding, we believe the current results are important because of the following:

They provide confirmation to effects reflected in other studies only by single experiments, thus reinforcing the validity of those prior findings that there is an advantage of modality separation for the interrupting task (Wickens et al., 1983).

They identify some new effects revealed by the “collective wisdom” of the meta-analysis in integrating multiple studies, primarily, the effects of urgency and complexity moderator variables on the relative benefits of tactile and auditory modalities for speed and accuracy.

They provide suggestions for important new directions of research, particularly in the area of redundancy effects, where the relatively low statistical power from few studies has left intriguing questions to be resolved regarding the circumstances of redundancy gain and loss. For example, given that the meta-analyses appeared to reveal a balance of effects of the MRT and auditory preemption theory on the performance of the ongoing task, further research is invited to establish the moderating variables that may tip this balance one way or the other. Also, since the studies included focused mostly on visual ongoing tasks, authors of future work can compare the different interrupting task modalities in the context of an ongoing auditory or tactile task. Finally, the meta-analyses reveal that workload is a moderator variable that researchers should consider in the design of future studies; only a limited number of experiments that were included in our analyses incorporated this variable.

Key Points

Significant differences between auditory and tactile interrupting tasks were observed as a function of two moderator variables: complexity and urgency. Accuracy was higher for tactile tasks in case of low-complexity signals; in contrast, high-complexity signals resulted in higher accuracy in the auditory modality. Faster responses were seen for low-urgency messages in the tactile modality, and there was no difference between the two modalities for high-urgency messages.

Audition, rather than vision, should be used for spatial and nonurgent tasks when accuracy is the primary concern and for categorical tasks when the issue of importance is response time.

Redundant auditory-visual combination should be used for communication tasks under high workload, for alerting and tracking tasks in low workload, and when there is a small visual angle of separation.

Footnotes

Acknowledgements

We would like to thank Kara Latorella (technical monitor) at NASA Langley Research Center and Durand Begault and Kenneth Goodrich for their direction, support, and guidance on this research. This work was performed for NASA Langley and the Integrated Intelligent Flight Deck program, under Contract NNX09AQ34A.

Sara A. Lu is a PhD candidate in the Department of Industrial and Operations Engineering, Center for Ergonomics, at the University of Michigan. She received her MSE in industrial and operations engineering from the University of Michigan in 2011.

Christopher D. Wickens is a senior scientist at Alion Science and Technology, Micro Analysis and Design Operations, in Boulder, Colorado, and professor emeritus at the University of Illinois at Urbana-Champaign. He received his PhD in psychology from the University of Michigan in 1974.

Julie C. Prinet is a PhD precandidate in the Depart-ment of Industrial and Operations Engineering, Center for Ergonomics, at the University of Michigan. She received her MS in engineering from the École Centrale de Nantes in 2011.

Shaun D. Hutchins is a senior human factors engineer at Alion Science and Technology in Boulder, Colorado. He is also a PhD precandidate in the School of Education at Colorado State University. He received his MA in experimental psychology with a minor in experimental statistics from New Mexico State University in 2007.

Nadine Sarter is a professor in the Department of Industrial and Operations Engineering, Center for Ergonomics, at the University of Michigan. She received her PhD in industrial and systems engineering from The Ohio State University in 1994.

Angelia Sebok is a principal human factors engineer/program manager at Alion Science and Technology in Boulder, Colorado. She earned her MS in industrial and systems engineering from Virginia Tech in 1991.

References

+Almen

(2002). Comparing audio and tactile inputs as driver attention control. In Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting (pp. 1777–1781). Santa Monica, CA: Human Factors and Ergonomics Society.

Banbury

S. P.

Macken

W. J.

Tremblay

Jones

D. M.

(2001). Auditory distraction and short-term memory: Phenomena and practical implications. Human Factors, 43, 12–29.

Begault

D. R.

Pittman

M. T.

(1996). Three-dimensional audio versus head-down traffic alert and collision avoidance system displays. International Journal of Aviation Psychology, 6, 79–93.

+Belz

S. M.

Robinson

G. S.

Casali

(1999). A new class of auditory warning signals for complex systems: Auditory icons. Human Factors, 41, 608–618.

*#Bliss

Liebman

Brill

(2010). Alert modality and behavioral compliance during virtual combat. In Proceedings of the Human Factors and Ergonomics Society 54th Annual Meeting (pp. 2373–2377). Santa Monica, CA: Human Factors and Ergonomics Society.

+Bouis

Voss

Geiser

Haller

(1979). Visual vs. auditory displays for different tasks of a car driver. In Proceedings of the Human Factors and Ergonomics Society 23rd Annual Meeting (pp. 35–39). Santa Monica, CA: Human Factors and Ergonomics Society.

#+Bronkhorst

A. W.

Veltman

J. A.

van Breda

(1996). Application of a three-dimensional auditory display in a flight task. Human Factors, 38, 23–33.

Burke

J. L.

Prewett

M. S.

Gray

A. A.

Yang

Stilson

Coovert

M. D.

Elliot

L. R.

Redden

(2006). Comparing the effects of visual-auditory and visual-tactile feedback on user performance: A meta-analysis. In Proceedings of the 8th International Conference on Multimodal Interfaces (pp. 108–117). New York, NY: ACM Press.

*Calhoun

G. L.

Draper

M. H.

Guilfoos

B. J.

Ruff

H. A.

(2005). Tactile and aural alerts in high auditory load UAV control environments. In Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting (pp. 145–149). Santa Monica, CA: Human Factors and Ergonomics Society.

10.

*Calhoun

Fontejon

Draper

Ruff

Guilfoos

(2004). Tactile versus aural redundant alert cues for UAV control applications. In Proceedings of the Human Factors and Ergonomics Society 48th Annual Meeting (pp. 137–141). Santa Monica, CA: Human Factors and Ergonomics Society.

11.

Chang

O’Modhrain

Jacob

Gunther

Ishii

(2002). ComTouch: Design of a vibrotactile communication device. In Proceedings of the Conference on Designing Interactive System (pp. 312–320). New York, NY: ACM Press.

12.

#Chen

W. H.

Zeng

J. J.

Kao

K. C.

(2005, June). Effect of auditory intersection collision avoidance warnings on driving behaviors in different distracted driving conditions. Paper presented at the 19th International Technical Conference on the Enhanced Safety of Vehicles, Washington, DC.

13.

#Colcombe

Wickens

C. D.

(2006). Cockpit display of traffic information automated conflict alerting: Parameters to maximize effectiveness and minimize disruption in multi-task environments (Tech. Rep. AHFD-05-22/NASA-05-9). Moffett Field, CA: NASA Ames Research Center.

14.

#+Dingus

T. A.

McGehee

D. V.

Hankey

J. M.

(1997). Human factors field evaluation of automotive headway maintenance/collision warning devices. Human Factors, 39, 216–229.

15.

#Dixon

S. R.

Wickens

C. D.

Chang

(2005). Mission control of multiple unmanned aerial vehicles: A workload analysis. Human Factors, 47, 479–487.

16.

+Dowell

Shmueli

(2008). Blending speech output and visual text in the multimodal interface. Human Factors, 50, 782–788.

17.

Elliott

Coovert

Redden

(2009). Overview of meta-analyses investigating vibrotactile versus visual display options. In Proceedings of 13th International Conference of Human-Computer Interaction (pp. 435–443). Berlin, Germany: Springer-Verlag.

18.

*Ferris

Penfold

Hameed

Sarter

(2006). The implications of crossmodal links in attention for the design of multimodal interfaces: A driving simulation study. In Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting (pp. 406–409). Santa Monica, CA: Human Factors and Ergonomics Society.

19.

*#Ferris

Sarter

(2008). Cross-modal links among vision, audition, and touch in complex environments. Human Factors, 50, 17–26.

20.

Ferris

Sarter

(2010). When content matters: The roles of processing code in tactile display design. IEEE Transactions on Haptics, 3, 199–210.

21.

#Finomore

Popik

Castle

Dallman

(2010). Effects of a network-centric multi-modal communication tool on a communication monitoring task. In Proceedings of the Human Factors and Ergonomics Society 54th Annual Meeting (pp. 2125–2129). Santa Monica, CA: Human Factors and Ergonomics Society.

22.

*Fitch

Kiefer

Hankey

Kleiner

(2007). Toward developing an approach for alerting drivers to the direction of a crash threat. Human Factors, 49, 710–720.

23.

#Gish

K. W.

Staplin

Stewart

Perel

(1999). Sensory and cognitive factors affecting automotive head-up display effectiveness. Transportation Research Record, 1694, 10–19.

24.

*#Glumm

Kehring

White

(2006). Effects of tactile, visual, and auditory cues about threat location on target acquisition and attention to visual and auditory communications (ARL-TR-3863). Adelphi, MD: U.S. Army Research Laboratory.

25.

+Haas

Hill

Stachowiak

Fields

(2009). Designing and evaluating a multimodal interface for soldier-swarm interaction. In Proceedings of the Human Factors and Ergonomics Society 53rd Annual Meeting (pp. 259–263). Santa Monica, CA: Human Factors and Ergonomics Society.

26.

Hale

Stanney

(2004). Deriving haptic design guidelines from human physiological, psychophysical, and neurological foundations. IEEE Computer Graphics and Applications, 24(2), 33–39.

27.

Hameed

Ferris

Jayaraman

Sarter

(2009). Using informative peripheral visual and tactile cues to support task and interruption management. Human Factors, 51, 126–135.

28.

#+Helleberg

Wickens

C. D.

(2003). Effects of data-link modality and display redundancy on pilot performance: An attentional perspective. International Journal of Aviation Psychology, 13(3), 9–21.

29.

*#Ho

Hong

Spence

(2005). Warning signals go multisensory. In Proceedings of the 11th International Conference on Human-Computer Interaction: Vol. 9. Advances in virtual environments technology: Musings on design, evaluation, and applications (pp. 22–27). Mahwah, NJ: Lawrence Erlbaum.

30.

*Ho

Hong

Spence

(2006). The differential effect of vibrotactile and auditory cues on visual spatial attention. Ergonomics, 49, 724–738.

31.

*+Ho

Reed

Spence

(2007). Multisensory in-car warning signals for collision avoidance. Human Factors, 49, 1107–1114.

32.

*Ho

Spence

(2009). Using peripersonal warning signals to orient a driver’s gaze. Human Factors, 51, 539–556.

33.

#Ho

C. Y.

Nikolic

Waters

Sarter

(2004). Not now! Supporting interruption management by indicating the modality and urgency of pending tasks. Human Factors, 46, 399–409.

34.

Horrey

W. J.

Wickens

C. D.

(2004). Driving and side task performance: The effects of display clutter, separation, and modality. Human Factors, 46, 611–624.

35.

#Hurwitz

J. B.

Wheatley

D. J.

(2002). Using driver performance measures to estimate workload. In Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting (pp. 1804–1808). Santa Monica, CA: Human Factors and Ergonomics Society.

36.

#Iani

Wickens

C. D.

(2007). Factors affecting task management in aviation. Human Factors, 49, 16–24.

37.

+Kaber

D. B.

Wright

M. C.

Sheik-Nainer

M. A.

(2006). Investigation of multi-modal interface features for adaptive automation of a human-robot system. International Journal of Human-Computer Studies, 64, 527–540.

38.

+Kramer

A. F.

Cassavaugh

Horrey

W. J.

Becic

Mayhugh

J. L.

(2007). Influence of age and proximity warning devices on collision avoidance in simulated driving. Human Factors, 49, 935–949.

39.

Kramer

(1994). An introduction to auditory displays. In Kramer

(Ed.), Auditory display: Soniﬁcation, audiﬁcation, and auditory interfaces (pp. 1–77). Reading, MA: Addison-Wesley.

40.

*#Krausman

Elliott

Pettitt

(2005). Effects of visual, auditory, and tactile alerts on platoon leader performance and decision making (ARL-TR-3633). Adelphi, MD: U.S. Army Research Laboratory.

41.

+Lancaster

J. A.

Casali

J. G.

(2008). Investigating pilot performance using mixed-modality simulated data link. Human Factors, 50, 183–193.

42.

*#Larkin

R. J.

(1983). A comparison of audio, visual, and tactile warning devices in a simulated flight environment (Master’s thesis). Naval Postgraduate School, Monterey, CA.

43.

Latorella

K. A.

(1996). Investigating interruptions: Implications for flight deck performance (Unpublished doctoral dissertation). State University of New York at Buffalo.

44.

#Latorella

K. A.

(1998). Effects of modality on interrupted flight deck performance: Implications for data link. In Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting (pp. 87–91). Santa Monica, CA: Human Factors and Ergonomics Society.

45.

Latorella

K. A.

(1999). Investigating interruptions: Implications for flightdeck performance (Tech. Rep. NASA/TM-1999-209707). Hampton, VA: NASA Langley Research Center.

46.

*Lee

McGehee

Brown

Marshall

(2006). Effects of adaptive cruise control and alert modality on driver performance. Transportation Research Record, 1980, 49–56.

47.

+Lee

Spence

(2008). Assessing the benefits of multimodal feedback on dual-task performance under demanding conditions. In Proceedings of the 22nd British HCI Group Annual Conference: People and Computers XXII. Culture, Creativity, Interaction (pp. 185–192). Swinton, UK: British Computer Society.

48.

+Lewandowski

Hursh

Kobus

D. A.

(1985). Multimodal versus unimodal information processing of words (Naval Submarine Medical Research Laboratory Report No. 1056). Groton, CT: Naval Submarine Medical Research Laboratory.

49.

#+Liu

Y. C.

(2001). Comparative study of the effects of auditory, visual and multimodality displays on drivers performance in advanced traveler information systems. Ergonomics, 44, 425–442.

50.

S. A.

Wickens

C. D.

Sarter

N. B.

Sebok

(2011). Informing the design of multimodal displays: A meta-analysis of empirical studies comparing auditory and tactile interruptions. In Proceedings of the Human Factors and Ergonomics Society 55th Annual Meeting (pp. 1170–1174). Santa Monica, CA: Human Factors and Ergonomics Society.

51.

+Maltz

Shinar

(2004). Imperfect in-vehicle collision avoidance warning systems can aid drivers. Human Factors, 46, 357–366.

52.

*#Merat

Jamson

(2008). The effect of stimulus modality on signal detection: Implications for assessing the safety of in-vehicle technology. Human Factors, 50, 145–158.

53.

*Mohebbi

Gray

Tan

(2009). Driver reaction time to tactile and auditory rear-end collision warnings while talking on a cell phone. Human Factors, 51, 102–110.

54.

#Mollenhauer

M. A.

Lee

Cho

Hulse

M. C.

Dingus

T. A.

(1994). The effects of sensory modality and information priority on in-vehicle signing and information systems. In Proceedings of the Human Factors and Ergonomics Society 38th Annual Meeting (pp. 1072–1076). Santa Monica, CA: Human Factors and Ergonomics Society.

55.

*+Mortimer

(2006). Affects of task difficulty on target guidance using auditory and tactile cues. In Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting (pp. 2109–2113). Santa Monica, CA: Human Factors and Ergonomics Society.

56.

+Navarro

Mars

Hoc

J.-M.

(2007). Lateral control assistance for car drivers: A comparison of motor priming and warning systems. Human Factors, 49, 950–960.

57.

Navon

Gopher

(1979). On the economy of the human processing systems. Psychological Review, 86, 254–255.

58.

*Oskarsson

P.-A

Eriksson

Carlander

(2012). Enhanced perception and performance by multimodal threat cueing in simulated combat vehicle. Human Factors, 54, 122–137.

59.

+Oskarsson

P.-A.

Eriksson

Lif

Lindahl

Hedström

(2008). Multimodal threat cueing in simulated combat vehicle. In Proceedings of the Human Factors and Ergonomics Society 52nd Annual Meeting (pp. 1287–1291). Santa Monica, CA: Human Factors and Ergonomics Society.

60.

Oviatt

(2003). Multimodal interfaces. In Jacko

Sears

(Eds.), The human–computer interaction handbook: Fundamentals, evolving technologies and emerging applications (pp. 286–304). Mahwah, NJ: Lawrence Erlbaum.

61.

+Perry

N. C.

Stevens

C. J.

Wiggins

M. W.

Howell

C. E.

(2007). Cough once for danger: Icons versus abstract warnings as informative alerts in civil aviation. Human Factors, 49, 1061–1071.

62.

#Ratwani

R. M.

Andrews

A. E.

Sousk

J. D.

Trafton

J. G.

(2008). The effect of interruption modality on primary task resumption. In Proceedings of the Human Factors and Ergonomics Society 52nd Annual Meeting (pp. 393–397). Santa Monica, CA: Human Factors and Ergonomics Society.

63.

Rosenthal

(1979). The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638–641.

64.

Rosenthal

DiMatteo

M. R.

(2001). Meta-analysis: Recent developments in quantitative methods for literature review. Annual Review of Psychology, 52, 59–82.

65.

+Santamaria

Quesada

(2007). Technical status report: Task No. HMI-08-02-001. Experimentation support and data analysis. Adelphi, MD: Robotics Collaborative Technical Alliance.

66.

Sarter

N. B.

(2002). Multimodal information presentation in support of human-automation communication and coordination. In Salas

(Ed.), Advances in human performance and cognitive engineering research (pp. 13–36). New York, NY: JAI Press.

67.

Sarter

N. B.

(2007). Coping with complexity through adaptive interface design. Human–Computer Interaction, 4552, 493–498.

68.

*Savick

Elliott

Zubal

Stachowiak

(2006). The effect of audio and tactile cues on soldier decision making and navigation in complex simulation scenarios (ARL-TR-4413). Adelphi, MD: U.S. Army Research Laboratory.

69.

Scerbo

M. W.

(1996). Theoretical perspectives on adaptive automation. In Parasuruman

Mouloua

(Eds.), Automation and human performance: Theory and applications (Human Factors in Transportation) (pp. 37–63). Mahwah, NJ: Lawrence Erlbaum.

70.

*#Scott

Gray

(2008). A comparison of tactile, visual, and auditory warnings for rear-end collision prevention in simulated driving. Human Factors, 50, 264–275.

71.

+Seagull

Wickens

Loeb

(2001). When less is more? Attention and workload in auditory, visual, and redundant patient monitoring conditions. In Proceedings of the Human Factors and Ergonomics Society 45th Annual Meeting (pp. 1395–1399). Santa Monica, CA: Human Factors and Ergonomics Society.

72.

+Seppelt

Wickens

C. D.

(2003). In-vehicle tasks: Effects of modality, driving relevance, and redundancy (Tech. Rep. AHFD-03-16/GM-03-2). Savoy: University of Illinois, Aviation Human Factors Division.

73.

Simmons

J. P.

Nelson

L. D.

Simonsohn

(2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.

74.

+Sklar

A. E.

Sarter

N. B.

(1999). Good vibrations: Tactile feedback in support of attention allocation and human-automation coordination in event-driven domains. Human Factors, 41, 543–552.

75.

*Smith

Clegg

Heggestad

Hopp-Levine

(2009). Interruption management: A comparison of auditory and tactile cues for both alerting and orienting. International Journal of Human–Computer Studies, 67, 777–786.

76.

#Srinivasan

(1997). Effect of selected in-vehicle route guidance systems on driver reaction times. Human Factors, 39, 200–215.

77.

#Srinivasan

Yang

C.-Z.

Jovanis

P. P.

Kitamura

Anwar

(1994). Simulation study of driving performance with selected route guidance systems. Transportation Research, 2(2), 73–90.

78.

*+Stanley

(2006). Haptic and auditory cues for lane departures warnings. In Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting (pp. 2405–2408). Santa Monica, CA: Human Factors and Ergonomics Society.

79.

Steelman-Allen

McCarley

Wickens

C. D.

(2011). Modeling the control of attention in visual work spaces. Human Factors, 53, 142–153.

80.

+Steelman-Allen

Talleur

Carbonari

Yamani

McCarley

Nunes

(2010). Effects of data link display format and position on flight performance. Paper presented at the Human Factors and Ergonomics Society 54th Annual Meeting, San Francisco, CA.

81.

+Stone

Bisantz

Llinas

Paquet

(2009). Augmented multisensory interface design (amid): A human-centric approach to unisensory and multisensory augmented reality design. Journal of Cognitive Engineering and Decision Making, 3, 362–388.

82.

*Straughn

Gray

Tan

(2009). To go or not to go: Stimulus-response compatibility for tactile and auditory pedestrian collision warnings. IEEE Transactions on Haptics, 2, 111–117.

83.

*Tilak

Xholi

Schowalter

Ferris

Hameed

Sarter

(2008). Crossmodal links in attention in the driving environment: The roles of cueing modality, signal timing, and workload. In Proceedings of the Human Factors and Ergonomics Society 52th Annual Meeting (pp. 1815–1819). Santa Monica, CA: Human Factors and Ergonomics Society.

84.

Trafton

Monk

(2007). Task interruptions. Reviews of Human Factors and Ergonomics, 3, 111–126.

85.

Trumbly

J. E.

Arnett

K. P.

Johnson

P. C.

(1994). Productivity gains via an adaptive user interfaces. Journal of Human–Computer Studies, 40, 63–81.

86.

+Van Erp

van Veen

(2001). Vibro-tactile information presentation in automobiles. In Proceedings of Eurohaptics (pp. 99–104). Paris, France: Eurohaptics Society.

87.

+Van Erp

van Veen

(2004). Vibrotactile in-vehicle navigation system. Transportation Research Part F: Traffic Psychology and Behaviour, 7, 247–256.

88.

#Walker

Alicandri

Sedney

Roberts

(1991). In-vehicle navigation devices: Effects on the safety of driver performance. In Proceedings of Vehicle Navigation and Information Systems Conference (pp. 499–525). New York, NY: IEEE.

89.

#Whitmire

Morgan

J. F.

Oron-Gilad

Hancock

P. A.

(2010). The effect of in-vehicle warning systems on speed compliance in work zones. In Proceedings of the Human Factors and Ergonomics Society 54th Annual Meeting (pp. 2023–2027). Santa Monica, CA: Human Factors and Ergonomics Society.

90.

Wickens

C. D.

(1980). The structure of attentional resources. In Nickerson

R. S.

(Ed.), Attention and performance VIII (pp. 239–258). Hillsdale, NJ: Lawrence Erlbaum.

91.

Wickens

C. D.

(2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3, 159–177.

92.

Wickens

C. D.

(2008). Multiple resources and mental workload. Human Factors, 50, 449–455.

93.

#Wickens

C. D.

Dixon

Seppelt

(2002). In-vehicle displays and control task interferences: The effects of display location and modality (Tech. Rep. AFHD-02-7/NASA-02-5/GM-02-1). Savoy: University of Illinois, Aviation Research Lab.

94.

#Wickens

C. D.

Dixon

S. R.

Seppelt

(2005). Auditory preemption versus multiple resources: Who wins in interruption management? In Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting (pp. 463–466). Santa Monica, CA: Human Factors and Ergonomics Society.

95.

+Wickens

Goh

Helleberg

Horrey

Talleur

(2003). Attentional models of multitask pilot performance using advanced display technology. Human Factors, 45, 360–380.

96.

+Wickens

C. D.

Gosney

(2003). Redundancy, modality, and priority in dual task interference. Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting (pp. 1590–1594). Santa Monica, CA: Human Factors and Ergonomics Society.

97.

Wickens

C. D.

Hollands

J. G.

Banbury

Parasuraman

(2012). Engineering psychology and human performance (4th ed.). New York, NY: Prentice Hall.

98.

Wickens

C. D.

Hutchins

Carolan

Cummings

(2012). Effectiveness of part task training and increasing difficulty training strategies: A meta-analysis approach. Human Factors. Advance online publication.

99.

Wickens

C. D.

Liu

(1988). Codes and modalities in multiple resources: A success and a qualification. Human Factors, 30, 599–616.

100.

Wickens

C. D.

Prinet

Hutchins

Sarter

N. B.

Sebok

(2011). Auditory-visual redundancy in vehicle control interruptions: Two meta-analyses. In Proceedings of the Human Factors and Ergonomics Society 55th Annual Meeting (pp. 1155–1159). Santa Monica, CA: Human Factors and Ergonomics Society.

101.

#Wickens

C. D.

Sandry

D. L.

Vidulich

(1983). Compatibility and resource competition between modalities of input, central processing, and output. Human Factors, 25, 227–248.

102.

*Wickens

Small

Andre

Bagnall

Brenaman

(2008). Multisensory enhancement of command displays for unusual attitude recovery. International Journal of Aviation Psychology, 18, 255–267.

103.

Wogalter

Conzola

Smith-Jackson

(2002). Research-based guidelines for warning design and evaluation. Applied Ergonomics, 33, 219–230.