Auditory Graphs Are Not the “Killer App” of Sonification,But They Work

The Elusive Killer App

In a sociotechnical analysis of sonification research, Supper (2012a) noted the search for a killer app as a recurrent theme in the discourse of the auditory display community. Sonification researchers, it seems, will view the field as underachieving its potential and playing second fiddle (sometimes rather literally) to visual displays until a breakthrough in sonic information design ushers in a new era of popularity and ubiquity for auditory displays.

The anticipation of such an imminent breakthrough was palpable, according to accounts of the early years of the ICAD conference, established in 1992. This enthusiasm was apparent, for example, in the first chapter of Kramer’s (1994) seminal introduction to the field of auditory display, an edited volume of the first ICAD conference proceedings. By the time I began attending the annual ICAD conference in 2005, optimism had begun to give way to frustration – the big break that would reveal the utility of audio to the ears of the world still had not come.

At the 23rd ICAD in 2017, Carla Scaletti, an attendee at the inaugural ICAD conference in 1992, gave a keynote presentation titled “Why Sonification Is a Joke,” because the field had become the butt of an actual joke (Scaletti, 2017). Researchers at CERN had released a 2016 April Fool’s video spoofing overly enthusiastic sonification researchers. The video claimed that sonifying data from the Large Hadron Collider revealed a hidden marvel: Wagner’s Ride of the Valkyries (Jarlett, 2016). Twenty-five years after the first ICAD, there was no killer app, and sonification was laughable.

Scaletti’s address took an optimistic turn, however, as she noted that the field has matured and developed even in the absence of the holy grail of a killer app. Examples of successful empirical and professional advances include an ever-expanding knowledge base, a host of new audio tools for producing sonifications, a number of well-established research labs, annual meetings dedicated to sonification research, and increasing willingness of funding agencies to support sonification research.

Further, I would argue that the auditory display community already has created multiple compelling cases of successful sonic information design, even if we have yet to see a sonification application rise to mainstream popularity.

Auditory Graphs: A Sonification Success Story

As an example, consider the auditory graph: a humble, aurally plain auditory display whose usefulness is supported by a rich research history. In the mid-1980s, Mansur, Blattner, and Joy (1985) published an empirical report on “sound graphs,” auditory displays that translated visual Cartesian line graphs to sound. Moving from left to right on the visual x-axis, the sounds unfolded in time. Changes in auditory frequency indicated changes in values on the graphical y-axis. Mansur et al. viewed their initial evaluation as a promising development for people with visual impairments. The auditory graphs conveyed information nearly as effectively as tactile graphics.

In the early 1990s, Flowers and colleagues undertook a research program (summarized in Flowers, 2005) that provided convincing evidence that auditory graphs were comparable – and sometimes superior – to visual graphics for conveying characteristics of a data set, including central tendency, range, shape, slope, and linearity. In a representative study, Flowers, Buhman, and Turnage (1997) presented statistically savvy undergraduates with both auditory and visual representations of scatterplots of bivariate data sets. Participants’ task was to estimate the Pearson r correlation depicted in both the auditory and visual versions of the scatterplots. The correlation between their estimates formed from visual and auditory representations of the same data was r = .97. In other words, the information conveyed by auditory graphs was identical to that conveyed by visual graphs.

The auditory graphs literature expanded to include design guidelines (Brown, Brewster, Ramloll, Burton, & Riedel, 2003) and theoretical models (Nees & Walker, 2007). Tools were developed to make auditory graphs (Walker & Cothran, 2003). Researchers deployed auditory graphs in mainstream classrooms (Hetzler & Tardiff, 2007) and classrooms for students with visual impairments (Davison, 2012) with mostly encouraging results.

What Counts as Success?

Given the promising and relatively thorough science behind auditory graphs, why didn’t the auditory display community embrace auditory graphs as an example of a killer app of sonification? One potential answer lies in an epistemological rift that has divided the auditory display community on the issue of what constitutes good evidence of a successful sonification (see Supper, 2012b).

On one side are those who emphasize careful evaluation and user testing of prototypes. On the other are those who believe the value of an auditory display can be judged by an expert listener. The former camp values data from controlled studies, whereas the latter camp takes more of a Gestaltist approach – the evidence of value is apparent in the auditory percept without need for further justification. Supper (2012b) labeled the user-testing enthusiasts “Correlation Coefficients” and their counterparts “Trained Ears.” She also suggested that the division fell at least partly along fault lines between researchers with backgrounds in science versus those with expertise in art.

Anecdotally, during the question-and-answer portion of my presentation of a paper on auditory graphs at the 2006 ICAD conference, an audience member – presumably a Trained Ear – asked me something to the effect of “Why do you still study auditory graphs? Who is still interested in simple tone graphs?” The interdisciplinary divide strikes again. Auditory graphs were uninteresting to a Trained Ear because of their aural simplicity. Yet their importance seemed obvious to me, a Correlation Coefficient, because they potentially work to fulfill a role in a system effectively.

The exchange revealed what may be another important difference in perspectives on successful examples of sonification. To a Trained Ear, the killer app will be a new and vital listening experience. To a Correlation Coefficient, a simple, effective sound is still an effective sound for accomplishing the goals of the system.

A Human Factors Perspective

As a field, human factors is defined broadly by evidence-based approaches to improving human interactions with systems. As such, a human factors approach to sonic information design would not define a successful sonification with respect to ubiquity or aural novelty. Instead, audio is a tool to be implemented when audio is most appropriate for accomplishing the goals of a given system. An effective auditory display is one that facilitates human interactions with technologies and information.

This is not to say that sonic information design should exclude complex, novel, and/or pleasing sounds; those qualities may be important for some applications of audio. But a human factors approach to sonic information design acknowledges that a simple auditory display may be effective in some contexts of use. An even more inconvenient truth (for auditory enthusiasts, at least) is that audio might be wholly inappropriate in some application scenarios (see Edworthy, 1998). And what if music already is the killer app of audio? Where does that leave sonic information design?

Perhaps the arrival of a mature and respectable field of sonification need not be marked by some yet-to-be-discovered, novel application of audio or by the implementation of audio solutions for every possible design problem. In keeping with a human factors approach, perhaps audio, rather than being an end in itself, is a tool to be implemented when evidence has shown that audio is the most appropriate display modality for a given application. By this metric, auditory graphs are just one example of successful, evidence-based sonic information design. Strong cases also could be made for the efficacy and success of other examples, including auditory menus (e.g., Walker et al., 2013) and sonifications for monitoring patients under anesthesia (e.g., Watson & Sanderson, 2004).

These proof-of-concept use cases are accompanied by other hallmarks of an established area of inquiry. For example, Watson and Sanderson (2007) provided a framework for evaluating the appropriateness of auditory displays, and Nees and Walker (2011) outlined an iterative process to validate the design of auditory displays. Examined holistically, the field of auditory display already has established itself as a respectable source of evidence-based solutions through sonic information design.

Obstacles Remain

And still, it remains true that few (if any) commercial or widely used systems implement auditory graphs or other empirically supported examples of successful sonification, even in contexts for which audio information displays would be highly appropriate. The impulse behind the search for the killer app of sonification (and the accompanying disappointment in the underappreciation of audio) can help to illuminate some notable barriers that have prevented wider adoption of auditory information display in design.

Though surprisingly little research has examined why audio is not used more frequently in user interfaces, a survey conducted by Frauenberger, Stockman, and Bourguet (2007) provided some insights. Lack of knowledge regarding best practices for auditory displays may preclude consideration of using audio in interfaces. Better translation and dissemination of the base of knowledge accumulated by ICAD and related communities could help designers who are unfamiliar with audio to gain a foothold in sonic information design.

Another barrier is the lack of sustainable tools for prototyping and implementation of audio in systems. The sonification literature abounds with descriptions of one-off tools and methods for producing sonifications, but these tend to be characterized by ad hoc capabilities, short-lived compatibility (with operating system updates, etc.), and little or no technical support. For those without a background in audio, the barriers to entry for auditory display remain relatively high, even when audio might offer a useful design solution.

Whether reducing these barriers would result in ubiquitous sonification remains to be seen (and heard). But wider dissemination of the auditory display knowledge base could promote better general awareness of the scenarios for which audio has a strong, evidence-based record of usefulness. And sustainable and well-supported tools perhaps could increase the prevalence of auditory displays.

Regarding auditory graphs, for example, one could imagine sonifications embedded alongside visual graphs presented online and in electronic textbooks as a standard option for all users, including people with visual impairments who access the information using screen readers. Accessibility is one obvious area where more widespread use of well-designed auditory displays could prove beneficial – perhaps in dramatic ways – for some users. Because of this potential to enhance human interactions with systems, I remain convinced that sonification is not, in fact, a joke.