Abstract
Visualization has shown its ability to produce powerful tools for analyzing, understanding, and communicating data, as well as making it accessible for several different tasks and purposes. A large community of casual and professional users is increasingly consuming and producing both interactive and static visualizations. Yet, while interactive visualizations move from research into practice at an increasing rate, it still remains an important challenge to find appropriate methods to evaluate them according to various metrics such as utility, usability, and beyond. The community has responded by developing, extending, and reappropriating special approaches and metrics for evaluation at all stages of the development life cycle that address specific needs in visualization.
This need is reflected, for example, in the increasing number of papers on visualization evaluation at many visualization venues. The BELIV (“Beyond Time and Errors: Novel Evaluation Methods For Visualization”) workshops (http://www.beliv.org/) provide a dedicated forum for this type of research with the possibility to discuss visualization evaluation and to spread the word on alternative and novel evaluation methods and methodologies in our community. BELIV started in 2006 as a bi-annual event and BELIV 2014 was the fifth instance of the workshop series. It took place on 10 November 2014, as a one-day workshop at the IEEE VIS Conference 2014 in Paris, France. For an overview of all BELIV proceedings, see their online versions in the Association for Computing Machinery (ACM) digital library:
BELIV 2006: https://dx-doi-org.web.bisu.edu.cn/10.1145/1168149
BELIV 2008: https://dx-doi-org.web.bisu.edu.cn/10.1145/1377966
BELIV 2010: https://dx-doi-org.web.bisu.edu.cn/10.1145/2110192
BELIV 2012: https://dx-doi-org.web.bisu.edu.cn/10.1145/2442576
BELIV 2014: https://dx-doi-org.web.bisu.edu.cn/10.1145/2669557
The position and research papers submitted to BELIV 2014 were selected according to their novelty, quality, and relevance for visualization evaluation. We were inclusive in our selection and accepted 23 papers with an acceptance rate of 76.6%. The best reviewed articles of this selection have been invited to submit an extended paper to this special issue, leading to five accepted articles in this special issue that discuss a variety of concerns and methodologies dedicated to evaluating visualization and visual analytics systems and processes. These are summarized below:
“Task” is an overloaded term that has been used to refer to a wide range of activities from low-level analytical action to high-level domain-specific analyses. In Rind et al.’s “Task Cube: A Three-Dimensional Conceptual Space of User Tasks in Visualization Design and Evaluation (pp. XX–YY),” the authors question the use of the term task and conduct a literature survey to extract different concepts of what a task means in visualization (evaluation). Based on this survey, they define a three-dimensional conceptual space of user tasks in visualization to assist future visualization researchers in formulating their research contributions.
Problem characterization is as essential as task characterization for designing successful visual analysis tools. However, how to properly implement such characterizations is a large open research problem in the visualization community. Toward a deeper understanding of this challenge, Winters et al. offer a conceptual model that is centered around the question of “what visualization works for whom and in which situation” in their paper “A Conceptual Model for Characterizing the Problem (pp. XX–YY).” The model helps to guide the process of problem characterization.
To shed more light on data explorers dealing with complex information visualizations in real-world scenarios, new methodologies and models are needed which overcome existing explanatory gaps. Smuc’s paper, “Just the Other Side of the Coin? From Error to Insight (pp. XX–YY),” introduces a novel model to analyze users’ errors and insights. Derived from Rasmussen’s model on different levels of cognitive processing, Smuc’s model integrates explorers’ skills, schemes, and knowledge (skill–rule–knowledge model) in the context of visual analytics.
Similarly, Reda et al. investigated how to better understand the behavior of users during analysis. In their paper “Modeling and Evaluating User Behavior in Exploratory Visual Analysis (pp. XX–YY),” the authors discuss how to deduce and subsequently describe and model human’s mental, interaction, and computational processes from various data sources such as observations and log files. The model focuses on describing transitions from one state to the next which helps to ultimately understand how behavior and insight formation may be related.
In terms of methodologies, Kurzhals et al. survey the visualization literature for evaluation approaches that are based on eye tracking in “Eye Tracking Evaluation of Visual (pp. XX–YY),” focusing specifically on visual analytics. Based on this survey, they extract common evaluation goals for such evaluation approaches and provide an overview of the state of the art of eye tracking data analysis techniques. Based on this discussion, the authors propose directions of future research for visualization evaluation methodologies.
Of course, organizing an event such as BELIV is not possible without the help of many people. We specifically want to thank all the authors, reviewers, and workshop participants who helped to make BELIV 2014 a productive and pleasant event and especially those reviewers who helped out in the last minute. We also would like to thank our invited speaker, Pierre Dragicevic, for agreeing to present an interesting and thought-provoking talk. We also express our gratitude, in particular, to the people who initiated BELIV in 2006 and who helped to organize it in the past. We also thank all organizers of IEEE VIS 2014 who agreed to accept BELIV into the program quite early in the process to give us enough time for advertizing, collecting submissions, and running the review process. Last, but not least, we would like to thank the people at ACM who have been hosting the online version of our workshop series since 2006—a service to the community that we much appreciate. The upcoming instance of the BELIV workshop series will happen this year on Monday, October 24, 2016, and will again be co-located with IEEE VIS, this time in Baltimore, Maryland, USA.