Figurative frames: A critical vocabulary for images in information visualization

Figurative visualization

An image or element of a visualization is figurative when the shape of the representation is recognizably similar to the shape of the represented object. Interpreting a figurative visualization requires the reader to recognize the depicted shape using prior knowledge. Figurative visualizations are open to misinterpretation, but also lend themselves to more natural interpretations. When describing simple line drawings, readers are more likely to see physical objects, providing a metaphorical description for an image only when they cannot think of a matching physical object. ³⁹ Furthermore, recognition of objects in an image is faster and more intuitive than interpreting an abstract visualization.^30,39

Figurative elements are included within information visualization for two main reasons: to enhance the comprehension of or engagement with the information encoded through data visualization, or because the information is inherently visual and cannot be communicated through abstract visualization techniques alone.

Figurative elements have been empirically shown to enhance a reader’s ability to remember abstractly encoded information.^5–9 More specifically, the memorability effect appears to apply to the information involved in a dual figurative–abstract encoding. ³² Different figurative variations on a bar chart were compared: pictographs (bars made up of icons where each icon represents a fixed quantity of the represented object), figuratively shaped bars whose height was stretched to the quantitative value, charts with figurative labels on the axis, and charts with a figurative background (similar to Figure 2(b)). Values of abstract attributes were better remembered (in the very short term) when they were encoded as part of an element which also had a figurative attribute (pictographs and stretched figurative bars) than when a figurative element appeared in the background of the visualization. ³² A plausible explanation is that the level at which figurative images are placed in the visualization focuses the reader’s attention and aids retention of the associated level of information – topic for background images, values for elements and explanations for integrated visualization.

OPEN IN VIEWER

Figure 2.

Alternative hybrid visualizations of the same dataset showing UV-B radiation under different cloud conditions. (a) a purely abstract representation (bar chart) of the data; (b) a hybrid visualization using the same abstract representation technique as A, but with a figurative background; (c) figurative images on each bar describe the cloud conditions; (d) figurative images describing the cloud conditions are attached to bar ends; (e) each panel shows the different cloud conditions, with the number of rays reaching the person conveying relative UV-B radiation; (f) a figurative panel describes the cloud conditions, with the quantitative information conveyed by an adjacent bar chart; data source: Schoonmaker. ⁴⁰ Bottom right: high-level design space classifying the different hybrid visualizations according to their integration and information density (see later sections).

The second argument in favour of figurative elements relates to their expressiveness. The resemblance between the representation shape and the object shape allows the visualization to describe the geometry, topology or aesthetics of the object. However, figurative visualization relies on the reader’s recognition of correspondence between an object and its representation, and so is susceptible to misinterpretation. ³⁵ A reader may be unfamiliar with the object, and so fail to recognize its representation. Furthermore, two people can look at the same line drawing and recognize it as a representation of two different objects and each express certainty about their perceptions. ³⁹ Additional ambiguity arises in the recognition of concepts ⁴¹ rather than specific instances of objects. A drawing of ‘a bird’ inevitably must have a shape that is closer to one species than another. To understand the representation, the reader must not only recognize the shape, but also infer the level of abstraction. A reader may recognize the shape but read the representation as a specific type of bird (e.g. an eagle) rather than the abstract concept. Similarly, a reader may infer an abstract concept when a specific instance is intended.

There are inherent limits in what can be represented figuratively. By definition, figurative visualizations show the shape of the object they represent – there is a fundamental correspondence between the drawn shape and the object shape. Hence, to be represented figuratively, an object must have some characteristic shape, or be depicted using a recognizable shape, which includes abstract, socially constructed or imagined objects, such as a low pressure front, country border and airspace no fly zone. Similarly, a concept design for an imagined object can be figuratively represented, even if it is unrealistic, such as a hoverboard, jetpack and unicorn. Figurative elements may be created any number of ways, including hand drawn on paper or screen, vector based, three-dimensional modelling, photography or other imaging.

Data visualization

A data visualization is a set of visual marks (points, glyphs, lines or areas) whose attributes (position, size, shape, colour, etc.) are determined by (mapped from) data values. ³⁶ To interpret a data visualization, the reader needs to read the key or axes to determine the meaning of visual attributes, which may be completely unrelated to the prior meaning of that attribute in the reader’s experience.

Data visualization has a limited range of expression, as it can only be used to show relationships between data values and categories; ³⁵ however, it also has distinct advantages compared to figurative visualization. Data visualization can represent abstract properties and can be used to place relevant variables in the same frame of reference. ³⁷ Salient and accurate perceptual variables (e.g. position, length) can be used to prioritize information important to the problem at hand. ⁴² Additionally, since attributes of visual marks set through an abstract mapping depend entirely on data values, their relationships can be compared in isolation from other extraneous information. ³

Data visualizations are easy to create and replicate across diverse datasets. Techniques for data visualization have been developed and documented in surveys^43–47 and grammars,^35,36 providing resources for the reuse and redesign of visualizations.

While less ambiguous than figurative visualizations, data visualization can also be misinterpreted. Through axes and keys, a data visualization explicitly describes the information encoded in the visualization, in contrast to figurative visualization. However, readers can misinterpret or fail to understand an unfamiliar data visualization because they fail to correctly identify the visualization mapping, for example, not understanding what lines connecting points represent, or whether attributes such as colour are meaningful. ⁴⁸ A survey of visitors to science museums found low levels of key visual literacy measures including interpretation of ‘simple’ data visualizations such as network representations. ⁴⁹ When readers lack the visualization literacy required to confidently interpret a data visualization, they may misread a visualization to match their pre-existing beliefs – for example, reporting that the visualization shows a trend opposite to the actual depicted trend. ³⁰ By avoiding representational ambiguity at all costs, data visualizations risk misunderstandings due to the reader not being able to keep track of or correctly interpret abstract mappings.

Data ‘metaphors’

While there is little evidence that including images in an information visualization negatively affects the comprehension of abstract information, there is also no reason to think that recognizable images should be used to communicate abstract information. Chernoff faces, which are data visualizations designed to look like a human face, aim to take advantage of human perceptual sensitivity to viewing faces, but are slower and more error prone to interpret compared to other data visualizations. ⁵⁰ The strengths and weaknesses of figurative visualizations depend on the two defining characteristics of recognition and resemblance. A Chernoff face has a recognizable shape, but the shape has no resemblance to the object it is representing (abstract data), and so is a data visualization, not a hybrid visualization. ⁵¹

Hybrid visualization examples

A hybrid visualization combines abstract and figurative representation; aspects of the visualization have an interpretation based on resemblance, or on an abstract encoding, or both. A survey of data visualizations and infographics showed that images can be used in three roles: as backgrounds for abstract data, icons to label data or as content (images whose primary purpose was to describe the shape or composition of an object). ¹⁴ The following example considers the potential impact of the three image roles on the information communicated by a hybrid visualization.

A range of different information visualizations can be used to explain the effect of cloud cover on the amount of UltraViolet-B (UV-B) radiation reaching a person and their consequent sunburn risk ⁴⁰ (see Figure 2). For example, a bar chart (Figure 2(a)) shows the radiation dose for each condition through the abstract height attribute of bars. The data visualization can show how UV-B quantities vary as cloud conditions change, but does not explain what each condition involves. Instead, the visualization assumes the reader has a pre-existing understanding of the conditions (and the concept of UV-B radiation) which can be evoked by the axis labels. Adding a background image to the chart area (e.g. Figure 2(b)) can describe the overall topic (outside, sunshine), but does not differentiate between the different conditions. When the figurative images are used as labels for specific abstract elements (Figure 2(c) and (d)) however, the visualization provides a visual description of each cloud condition in addition to the comparison of their relative UV-B radiation values. The figurative bar fill in Figure 2(c) and figurative icons at the end of each bar in Figure 2(d) are independent of the abstractly meaningful bar height. Both the previous uses of figurative images at the global (Figure 2(a)) or element level (Figure 2(c) and (d)) can be described as ‘embellished’ data visualization, since without the figurative meanings the visualization is still a complete data visualization. The abstract and figurative interpretations can take place without reference to each other. At the most extreme end of separation between figurative and abstract representation, diagrams and data visualization can be used as separate visualization views (Figure 2(f)), with one view describing the different cloud conditions, and another comparing the associated quantitative UV-B values. In contrast, the figurative and abstract interpretations can be interconnected (Figure 2(e)). Here, the visualization uses an abstract ordered horizontal axis to set out the cloud conditions at different positions on the page, but each condition is depicted figuratively. The position attribute of line elements in the visualization has dual meaning. The lines have a figurative interpretation as rays from the sun which interact with clouds, but also have an abstract meaning as one-tenth of a unit of UV-B radiation received by the human recipient. The interconnected visualization shows a causal explanation for why the UV-B values arise as a result of the cloud conditions which is missing from the unconnected version.

Existing approaches to data visualization suitability provide a lens to evaluate each of the hybrid visualization examples as a vehicle for comparing UV-B values. The perceptual judgements (height, position or stack length) used for quantitative comparison can be identified in each case and compared in terms of accuracy. ⁴²

Missing from existing theory is a means of evaluating the success of the images in providing descriptive information and connecting description and quantitative information. The UV-B examples suggest that hybrid visualizations can vary in terms of level of description detail provided by figurative images and in their ability to explain the connection between descriptive and quantitative information, or merely establish an association. The shift from background to content images (Figure 2(b) vs (f)) provides increasingly detailed description, while the switch from background images to images-as-labels (Figure 2(b) vs (c)) changes the relationship between the two types of information conveyed in a hybrid visualization, a difference echoed in the comparison between the most connected and most separate examples (Figure 2(e) vs (f)).

Geospatial visualization

One domain which has established theory and practice for using hybrid visualization is geospatial visualization. Figurative images are used in the form of map images which provide a geographic and topographic description of space, and also as map icons showing the locations of landmarks or other objects. Abstract visualization in geospatial displays shows properties of locations, paths or regions. Both the figurative and abstract components in geospatial visualization are recognized as providing valuable information. Abstract marks reveal patterns of geographic distributions in the data, ³⁵ while the underlying map image provides knowledge of how the depicted locations relate to each other – 10 minutes of studying a map can provide better knowledge of the spatial layout of locations than living in an area for 10 years. ⁵² Recognizable map icons (and recognizable colours) aid map interpretation, although they can, of course, be misinterpreted.^35,53

Several principles have been developed for geospatial visualization which guide map image design and integration with data. One key factor is the choice of an appropriate projection used to display the three-dimensional (3D) world in a two-dimensional (2D) image. Projection appropriateness depends on the match between the distances, angles or areas the projection preserves, and the purpose of the map. A second principle is based on the concept of ‘generalization’, the idea that a map must choose a particular scale at which it is accurate, and omit some level of detail – for example, smoothing edge contours of coastlines.^54,55 Good generalization allows the reader to see relevant features clearly while not providing a misleading impression of shape or content.^54,55 Techniques have also developed around the integration of figurative and abstract representation for geospatial applications – for instance cartograms which distort the map based on some data value, or using per capita operations to pre-process data to avoid simply revealing the underlying populations. The motivation behind techniques such as cartograms is to strike a balance between providing a clear view of patterns in the abstract data and a rich, detailed view of the geospatial features which may cause those patterns. ³⁵

The approach of geospatial visualization offers a template for how to understand hybrid visualization more generally. The figurative frame concept presented in the next section provides a model which can generalize the geospatial case and classify the range and dimensions of variation seen in the UV-B examples.

Visualization aims

Visualizations are used to capture, communicate or analyse information; to enable the discovery of new information, verification of existing beliefs, or to provide enjoyment. ⁵⁶ Depending on a visualization’s aims, the topic, variable meanings, variable values or causal relationships between variables may be more important to emphasize; engagement and memorability may be more or less important. It may be desirable to present results and hypothesized causes together or in isolation, depending on the audience and purpose of the visualization. The aim of this article is not to advocate for one form of hybrid visualization in favour of others or in favour of purely abstract visualization, but instead to enable classification and analysis of distinct types of information visualization, and to identify appropriate measures of suitability for each type.

Reading cues

In contrast with geospatial visualization, where there is a single conventional coordinate system, and standard feature types (state boundaries, surface types), figurative visualizations more generally can have a wide range of frames, which the reader may never have encountered previously. Successful communication of information through a figurative image depends on the reader recognizing the frame. Although parts of the frame can be explicitly described through captions, labels and guidelines, many conventional cues also typically exist which the reader can use to infer the frame.

As with communicating an abstract mapping, labels and guidelines play an important role in explicitly explaining a figurative frame. Gridlines and coordinate labels (as seen in the geospatial visualizations in Figure 3) can be used to show how a quantitative coordinate system projects onto an image. Similarly outlines or point markers can show a set-based coordinate system (see Figure 4, top and middle, centre column). Labels and outlines can also be used to define and draw the reader’s attention to features of frame. The overall visualization title and description additionally provides cues as to the type of frame used. Labels for visualizations can also explicitly indicate the precision of the coordinate system, using terms such as ‘to-scale’, ‘sketch’ and ‘artist’s conception’. Information accompanying a visualization similarly suggests parts of the image are frame features - the title of the UV-B visualizations (Figure 2) suggests that the sun, clouds and recipient are features of the frame in Figure 2(e), but that the sunglasses on the sun are not in Figure 2(b). Ambiguous images with explanatory captions are recalled more easily and recalled as being more like an image closer to the explanation. ⁶²

OPEN IN VIEWER

Figure 3.

Alternative projections preserve different aspects of an object (the Earth) to create figurative visualizations which encode different information. Sources: Mercator and Lambert projections by Jecowa, distributed under a CC-BY-SA-3.0 license, and Mysid, after a USGS image.

OPEN IN VIEWER

Figure 4.

Frames with different coordinate systems or projections underpin different combinations of abstract and figurative visualization. Different figurative frames (left) can be used to combine abstract and figurative visualizations in integrated representations (right column, created from artificial data) to address specific questions. Photo credit (bottom image): T. Neville.

The style and design choices of the image, as well as the surrounding context, also offer cues for identifying the frame. The level of realism in an image forms an implicit claim of accuracy with realistic images suggesting a quantitative rather than set-based coordinate system, and more detail indicating more quantitative accuracy and a higher number of features. Similarly, the absence of detail or a ‘sketchy’ appearance implies a more generalized representation (see analysis of sketchiness as a data visualization variable ⁶³ ). In fields such as zoology which make frequent use of figurative images as a research tool, the interpretive guidance provided by less detailed figurative media such as illustration over photography is well established; illustration ‘can emphasize only the most important information about a subject, leaving out distracting clutter’ (p. 165). ⁶⁴ Context can also be informative – an image of a bird with its wings stretched next to another of the same bird with wings folded suggests different frame features than an image of a bird next to a lion. In the former, the wings and associated differences in the bird’s posture are almost certain to be features of the frame; the same inference is not implied in the latter case.

A third mechanism by which the frame of an image can be inferred is through the type and position of associated data visualization marks. Consider example visualizations of gameplay in baseball (see Figure 1). When an image of a baseball pitch is used to plot the progress of hitters around the bases before being caught or tagged out (Figure 1(b)), all of the data points are positioned along the lines between each of the four bases. The distribution of the points suggests that quantitative scale in the image is only important for the one-dimensional line around the bases; the scale of the remainder of the field is unimportant. The reader can infer a coordinate system consisting of the interval [0, 360ft], with an additional element representing the surrounds. Similarly, the features which are relevant for interpreting the data (in addition to the overall identity feature) are the bases and running line ([first base (90), second base (180), third base (270), home base (360), running line: [0, 360]]). By comparison, in a plot showing where the baseball is hit instead (Figure 1(a)), the 2D positions of line-ends are relevant, implying that scale matters throughout the image, and a 2D quantitative coordinate system is being used in the frame, with origin at the originating point of the line markers. A third example of a baseball hybrid visualization shows throws between fielders in a game of baseball; it makes use of icons in place of network nodes (Figure 1(c)). There is at most one link between each pair of fielders, and aesthetically similar links connect to images at different points (torso, head etc.), implying that the associated data concerns each represented player as a unit object, whose component parts are not important information content. Thus, the reader can infer that each image uses a single point frame with only the identity feature.

One measure of figurative visualization success that can immediately be identified through the frame model is the reader’s ability to discern the frame from the visualization and surrounding information (title, labels, captions, etc.). ‘Frame identifiability’ offers a complementary measure to data visualization performance measures such as accuracy and error rate, capturing the ability of a figurative image to clearly convey the intended content.

Frames and types of hybrid visualization

Frames can be used to define two dimensions differentiating types of hybrid visualization: level of frame density and level of integration (Figure 2 shows the dimensions with the UV-B examples plotted).

The first dimension, frame density, is defined as the average number of features and preserved judgement types (distances, angles etc.) per figurative image in the visualization. It is one measure of the amount of information being conveyed by the figurative images in a hybrid visualization. The overall balance between descriptive and abstract information provided by a hybrid visualization can be gauged by comparing the frame density and the number of data dimensions. A visualization with high frame density and only a few dimensions of data is mostly descriptive, while low frame density and many data dimensions indicate a mostly abstract visualization. Visualizations with simple frames and only one or two data dimensions are balanced but simple in terms of their overall information content, while visualizations with complex frames and many data dimensions are balanced and information-rich overall.

The second dimension, integration, describes the extent to which marks in the visualization correspond both to frame coordinates and abstract data values. Integration can occur in two ways – first when data are attached to frame coordinates or features to show information about an object or scene (as in the baseball field examples in Figure 1(a) and (b)), and second, when frames are positioned based on abstract data mappings (as in the baseball throws between players example in Figure 1(c)). The following two sections explore in detail how the different strategies of integration apply to both simple and complex frames, and the types of visualization problem to which each is suited. Two different integration strategies are identified: ‘frames as spaces’ and ‘layouts of frames’. Either strategy can be used to generate visualizations covering the same region of the hybrid visualization dimension grid (medium to highly integrated visualizations with frames of any density). The frames as spaces strategy have two subtypes – attached marks and abstract attributes; layouts of frames have one subtype – abstract layouts. Additionally, there is a subtype – combined layouts – which uses both the frames as spaces and layouts of frames strategy. These different approaches to integration are explored in turn in the following sections.

Anchoring data

When abstract elements are embedded within a figurative visualization, the data are ‘anchored’ to the frame. Specifically, an attribute of the data (the anchoring property) is identified with the coordinates of the frame. For example, in the baseball visualizations in Figure 1, hit location is anchored to the baseball field frame by mapping hit x and hit y to coordinates x and y of the frame. Data can be anchored to any feature defined in the geometry of the frame structure, including points, lines, parts, segments or networks. The anchored data can take the form of a single value (in addition to the anchoring value), or a tuple of values, and can be any data type – nominal, ordinal, interval or ratio. Data can also be anchored at different scales: to the object as a whole, to precise locations, to locations within a range or grid cell or to components of the frame. Frame density determines the number of different places where data can be anchored to the visualization – the more coordinates a frame has, the more points available for data values.

A key measure of visualization quality for embedded abstract elements is that the level of detail in the coordinate system should match the type and range of the anchoring data property. To show data corresponding to specific locations on the represented object, the frame needs to preserve the spatial relations of the object. Similarly, if the figurative visualization is distance preserving, but the data are imprecisely located on the object, the resulting hybrid visualization is only as accurate as the data (see Figure 5). A mismatch between the type and resolution of data and the frame coordinate system creates a misleading or imprecise visualization, implying that data values relate to more specific locations than can be substantiated (detailed frame, less detailed data), or providing less information than is available (detailed data, less detailed frame). If data locations on an object or in a scene are quantitative, a scale image and quantitative (e.g. Cartesian or polar) coordinate system should be used. If data specifies information about object parts, then either a set coordinate system should be used, or parts should be specified as features of the frame (useful in the case where some data are about parts, and others is about quantified locations).

OPEN IN VIEWER

Figure 5.

Matching figurative structures and abstract anchoring in touch interactions of a human with a robot (artificial data). (a) The figurative visualization is distance preserving (i.e. drawn to scale), and interaction markers are anchored to points on the robot, with a dodge operation used to reposition overlapping points (high frame density, high integration). (b) Distances are not preserved, and so interactions cannot be anchored to points on the robot. Instead, interactions are anchored to robot parts (indicated by dashed outlines), and the positions of the point markers only show which part of the robot was touched, not the precise location (medium frame density, high integration).

Attaching marks to a figurative visualization

One technique for integrating abstract elements within a figurative visualization is the superimposition of additional markers – points, lines or areas – at the data anchors (e.g. Figure 1(a) and (b)). In this form, the figurative element acts both to show the object and as a space for the abstract markers, providing context to the marker values.

Operations which work on marks in a data visualization can also be applied to abstract markers anchored to a figurative visualization and interpreted in the same way. For example, a ‘dodge’ operation which separates overlapping points in a data visualization ³⁶ can also be applied to separate points anchored to the same location on a figurative shape (see Figure 5(a)).

Abstract attributes: setting attributes of a figurative visualization with data

The second technique for integrating abstract data within a figurative visualization is to make use of visual attributes not set as part of the figurative image. Figurative images may or may not include shading, texture, colour or transparency (opacity). When any or all of these attributes are not used in the figurative element, they may be defined based on data values anchored to the frame.

Each visual attribute (colour, shading, texture etc.) is applied in the same way. For example, the scale and geometry of the data anchoring and frame structure will determine how colour is applied to the figurative shape, while the data values and choice of colour scale will set the colour value. If data are anchored to frame features, the whole feature is coloured by the data value and colour scale (e.g. Figure 6(a)). If, on the other hand, data are anchored to coordinate regions (or points, or lines) on the frame surface, each region is coloured independently of its neighbours (e.g. Figure 6(b)). Colour scales may be categorical or quantitative depending on the type of data.

OPEN IN VIEWER

Figure 6.

Abstract shading of a figurative visualization showing the number of touch interactions on a robot (artificial data). (a) The visualization on the left uses a frame with a partitioned structure, showing number of touches (normalized based on surface area) on different functional parts of a robot (medium frame density, high integration). (b) The visualization on the right uses a quantized structure, showing the number of touches within a grid cell (high frame density, high integration).

Interpretation

As an integration strategy, data embedded in a figurative element show how values of some abstract property are distributed on or in the figuratively represented object or scene, relative to the descriptive information shown by the figurative frame features. It allows the reader to make the same types of queries as in data visualization: to judge individual values, compare sets of values, or to summarize the distribution of values across the object (see Brehmer et al. ⁵⁶ for data visualization tasks and queries).

The distinction between abstract elements superimposed on an image and images with abstract attributes is based on whether the abstract marks are perceived as part of or separate to the image. The perception of part or separate is in turn determined by Gestalt principles such as continuity. ⁶⁶ Visualizations which define figurative attributes suggest that the abstract information is about the object (or scene), while visualizations which attach markers show information on an object or in a scene.

Measures of visualization quality

Abstract marks embedded in a figurative image are subject to the same measures of quality as marks in data visualization, as well as some additional measures relevant to hybrid visualization. The visual attributes (shape, size, shade, texture or hue) used to denote data values have different judgement accuracies as well as strengths for conveying ordered or non-ranked categorical values.^35,42 Additionally, a good embedding of data in a figurative image will clearly distinguish the abstract marks or attributes from the background image.

Labels: simple frames

Figurative images are often used to label points, bars or lines in information visualization projects. ¹⁴ In Wilkinson’s grammar of graphics, labelling is achieved through an ‘.image’ function which sets the shape of a point in the final stages of the construction process. ³⁶

Within the figurative frame model, figurative images used as labels have the simplest type of frame: a single coordinate element and the identity feature. The image visually identifies the object (through the reader’s recognition of the identity feature), while other attributes of each image – such as size or position – provide abstract information. Any abstract layout which maps data values to a pair of coordinates can be used with simple-frame images, including bar charts (e.g. Figure 2(d)), scatterplots, network layouts (e.g. Figure 1(c)), pictographs ³² and more. A reader can use their prior knowledge of the represented objects to find relationships in or explanations for an observed pattern in the abstract visual attributes (for instance needle-leafed plants clustering together on a graph of tree climate vs deciduousness). However, the visualization does not provide any new descriptive knowledge.

The suitability of image labels for a particular problem depends on a number of factors. Images need to accurately reflect the data they are labelling and need to be large enough within the visualization to be recognized, without obscuring each other or visual attributes (e.g. position values). Additionally, any attributes used in the image – shape, colour, texture – cannot be used to encode other data in the abstract mapping. Small, simple images are required to label large datasets, while complex images can be used for visualizing a smaller number of data points.

Abstract layouts of more complex frames

Images with more complicated frames can also be arranged in an abstract layout, that is, their position, size or other attributes defined using an abstract mapping based on data values. The abstract information conveyed in such a visualization is the same as in the images-as-labels case, but additional descriptive information is conveyed through the frame features. Features which are similar (or different) across multiple objects can be used to recognize relationships in the data, even if they are not part of the reader’s prior knowledge (e.g. thin, straight wing shape can be correlated to long flight-times in a plot of the length of time different birds can stay airborne). Abstract layouts of images create hybrid visualizations which allow the reader to compare, find relationships between, and summarize both properties and descriptions of a collection of objects.

Combined layouts

The hybrid visualizations described thus far include abstract marks on a figurative space (previous section), and images in an abstract arrangement (subsections above). However, it is also possible for hybrid visualizations to use a combination of both strategies.

When non-trivial frames are arranged in an abstract layout, additional data can be anchored to each of those frames and visualized through the addition of abstract marks or by modifying ‘free’ attributes of the image. The result is a nesting of the ‘layouts of frames’ and ‘frame as space’ strategies. A nested strategy allows the reader to compare properties of objects and properties of or related to their component features. However, the limitations on the size of images and number of represented objects relative to the display are even more pronounced than in the images-as-labels case. Images need to be large enough to allow the reader to distinguish both the image frame and the anchored marks or attributes.

A second type of combination is when abstract visualization techniques are used within the frame itself. A frame is a figurative layout: it arranges features according to their actual location in the represented object or scene. As such, an abstract property can be used to adjust or modify the figurative arrangement. When a frame is used as a space, the anchored marks or modified attributes are distinguishable from the image and have no figurative interpretation (i.e. are not part of the visual description of the object). In contrast, in a modified frame, features convey both descriptive and abstract information.

An example (see Figure 7) is provided by the panels of the highly integrated, complex framed UV-B visualization described previously (Figure 2(e)). Through a figurative layout, cloud features are positioned between the sun and the UV-B recipient against a background showing ground and sky. The figurative positioning of sun, clouds and recipient also defines areas in which UV-B ray frames can be placed to show recognizable interaction paths. An abstract mapping determines the number of UV-B ray features in the frame based on the quantity of radiation reaching the recipient (13 rays, where each ray represents 0.1 unit of radiation). The position of the rays in the final visualization is determined by an abstract stacking operation (as in a pictograph) constrained by limits imposed by the figurative layout. Additionally, the panels are positioned in order by amount of cloud coverage, using the width of the sky feature in the background frame (an abstract layout).

OPEN IN VIEWER

Figure 7.

Combined figurative and abstract layouts applied to produce one panel of the integrated UV-B visualization shown in Figure 2.

Modified frames are more integrated than either frames as spaces or layouts of frames – the abstract data are represented as an intrinsic part of the visual description. Integration can be a strength (it explains data) or a weakness (it conflates data and explanation), depending on the visualization aim and audience. Both the figurative and abstract interpretations need to be clear in a modified frame hybrid visualization, constraining the range of possible designs; different data values (e.g. a UV-B index of 1.3 vs 0.8) need to be distinguishable, but marks also need to be recognized as features of the frame (i.e. part of the object description).

Three hybrid representations

All three visualizations concern bird species identification and comparison and focus on one set of hard-to-differentiate bird species: the cattle egret, little egret and intermediate egret, all of which are found in the mid- and north-east of Australia. ⁶⁷ Three alternative hybrid visualizations (Figures 9–11) show different information about the egrets and have different levels of integration and frame density (Figure 8).

OPEN IN VIEWER

Figure 8.

Integration and frame density of the three hybrid visualization designs (Figures 9–11).

The first design uses an images-as-labels strategy as part of a timeline visualization comparing the breeding seasons for each egret species (Figure 9). Each figurative image has a simple single coordinate frame with only the identity feature. In the abstract component of the visualization, saturation gradient represents the sets of months when the birds are likely to be breeding, when they may breed if environmental conditions are suitable, and when breeding is highly unlikely. ⁶⁷

OPEN IN VIEWER

Figure 9.

A timeline visualization allows comparison of the egrets’ breeding seasons.

A layout of frames approach is also used in the second hybrid design, however images have slightly more complex frames – each image shows at least one additional feature (the marking) as well as the identity feature (the bird part with that marking) (Figure 10). A tiling mapping is used to arrange the images according to bird location, and colour and pattern are applied as in a node-ring visualization ⁶⁸ to show markings linked by their presence within a single species and plumage type. The stack of rings around each image indicates how the represented marking is shared across different species and seasons.

OPEN IN VIEWER

Figure 10.

A node-ring based tiling of identifying marks emphasizes the commonality of different features.

The third hybrid visualization considered in the case study combines the frame-as-space and layout-of-frames approach. At the top level, it uses a faceting or small multiples approach,^36,69 in which all the information about a particular species is contained within a subset of the visualization. Within each facet, the visualization deploys the frame-as-space approach, with a radiating lollipop-shaped marker attached to the edge of any feature of the egret which changes plumage during breeding season. Each marker contains a second image (creating a layout-of-frames on the frame-as-space), showing the breeding plumage. The main images have frames of medium feature density – multiple features, but only a set-based rather than a quantitative coordinate system. The secondary images have a lower feature density, showing only one feature more than the identity feature. Integrated into each high-level facet is an abstract circular timeline showing breeding season; however, the timelines are only minimally integrated with the frame-as-space representation (shared colour, but separate position).

Evaluating figurative frames and images

Visualization has the potential to aid in different challenges related to egret identification. Bird species are an important marker of biodiversity, and organizations are increasingly taking a ‘citizen scientist’ approach to collecting bird distribution data, recruiting the public to report bird sightings and participate in bird count surveys. ⁷⁰ The quality of crowdsourcing approaches depends on accurate bird identification, which can be difficult for non-experts. Novice observers would benefit from a visual explanation of the normal and breeding plumage of each bird, as well as the times when each bird is likely to be breeding. For those training or giving advice to observers, on the other hand, it is important to identify the most easily spotted distinguishing characteristics to instruct novices to look for, since a bird may only be seen briefly. Visualization could also aid in planning the observation studies at appropriate times to compare breeding and non-breeding behaviour and habitat.

A suitable hybrid visualization will address the problem, as well as meeting general measures of data and figurative visualization quality (see Table 2). The suitability for a problem will depend on the type of hybrid visualization, compared to the required balance of description and abstract information. Visualization quality includes the choice of data visualization variables, the frame precision and the readability of the frame.

OPEN IN VIEWER

Table 2.

Hybrid visualization suitability measures, and relevant paper sections or references.

Measures	Section/references
Appropriate balance of descriptive and abstract information	Frames
Use of frame-as-space technique if problem requires comparison within an object or scene	Frames as spaces
Use of layout-of-frames technique if problem requires comparison across multiple objects or scenes	Layouts of frames
Sufficient frame precision and features	Frames
Appropriate choice of variables and layout for abstract components (existing data visualization suitability)	See Zuk et al. 71 and Mackinlay 72
Readability of the figurative frame	Frames

Of the three example visualization problems, descriptive information is most important for the novice identification problem, and least important for observation planning. The novice identification problem requires a balance of abstract and descriptive information: abstract visualization is needed to compare the uniqueness of different markings, while descriptive information shows how easily each marking can be spotted. The combined layout (Figure 11) is the most suited to novice identification since it provides the most descriptive information (i.e. has the highest frame density). In contrast, the minimal frames in the labelled timelines visualization (Figure 9) are suited to observation planning, since it focuses on the abstract breeding season data. The frames showing each marking in detail, in a layout which facilitates comparison across species and breeding season, makes the node-ring visualization (Figure 10) suited to the distinguishing characteristics problem.

OPEN IN VIEWER

Figure 11.

Egret identification showing plumage in normal versus breeding seasons. The visualization is suited to assisting novices in discriminating between similar species.

Having matched visualizations to purposes, there remains the question of whether each visualization is a good visualization (as opposed to simply the best of the three presented options). One measure of hybrid visualization quality is whether the precision of the coordinate system and the choice of frame features are sufficient. The level of descriptive information provided by the labelled timelines visualization and node-ring visualization were already assessed as suitable for their associated problems; for the combined layout, it is possible that a more realistic image would provide better guidance for novice observers. However, individual birds within a species vary in terms of their size and markings and can look different in motion in the wild, so a more realistic image would not offer a more accurate description of the species in general. Furthermore, additional features such as wing shape would not provide additional cues for distinguishing between the different egrets. User evaluations could be conducted to check the performance of realistic but less representative images as a means of instructing novices.

One interesting observation is that more complicated hybrid visualizations (i.e. those with higher frame density) are not necessarily more suited to expert rather than novice audiences. Readers who are highly familiar with an object may be able to recognize part of that object from a representation just of that part, drawn with low accuracy, whereas a less familiar reader may need to see the part represented more accurately within a more recognizable context. Working geneticists are less likely than biology students to include the details of shape and dimension in chromosome diagrams they generate to solve problems. ⁷³ Furthermore, highly detailed illustrations can often be perceived as simpler than more minimalist representations. ³⁹

Another quality consideration is the choice of variables and layouts for the abstract components of a hybrid visualization. Abstract visualization quality is discussed extensively elsewhere, ^3,30,71,72 so will not be addressed in detail here. One factor worth noting is that abstract visualization concerns can interact with the suitability of particular figurative frames, as exemplified by the hybrid node-ring visualization (Figure 9). The number of rings around each marking image shows how many of the egrets have that marking. Discounting the two leftmost ‘bird shape’ markings, the use of a common image size means that the overall size of each node corresponds to the prevalence of that marking – the reader can determine prevalence through an area judgement rather than judging line thickness. The cost of allowing an area judgement is that images cannot use frames with a common scale coordinate system and projection – the markings are not to-scale relative to each other. The ‘bird shape’ markings (left) are involved in the alternative end of a similar trade-off: if the images were reduced to the same size as the other marking images, the frame features (notably the head and neck shapes) would be difficult to identify, but an area judgement could be used instead of line thickness.

Finally, quality needs to take into account the frame readability. Too little is known about the interpretation of figurative visualization to predict the ease of reading a particular frame. However, it is possible to identify the different reading cues provided to help the reader recognize the frames of each visualization. In the labelled timeline, size provides a contextual cue that a minimal frame is in use – the images are too small to make out detailed features clearly. Text labels for each timeline provide a name for the identity feature and help the reader ascertain the level of abstraction of the images (species, not individual specimen). In the hybrid node-ring visualization, highlighting techniques are used to emphasize the frame features. The head angle and neck of the ‘bird shape’ nodes are outlined with dotted lines to signal their importance in the image (i.e. to show they are frame features). In the remaining nodes, a line drawing style is used, with only the distinguishing marking coloured. The colour scheme indicates the sole non-identity feature for each image. The colour scheme also allows the image to represent a more abstract concept – an egret, of any of the three species, provided it has the relevant marking. The abstraction level can be inferred from the associated abstract marks – for the yellow ‘lores colour’ (2nd row, leftmost) node to apply to intermediate, cattle and little egrets, the image must show a concept more general than any one of the species individually. The combined layout visualization also uses highlighting and contextual cues, as well as annotations and guides to communicate the figurative frame. The egrets’ neck and head shape features are highlighted by adding lines as in the node-ring visualization, explained through a note in the visualization guide. However, the primary means of frame explanation is comparison with surrounding images. Each anchored image (inside the lollipop markers) suggests a corresponding feature in all of the main egret frames. The features of the anchored images (breeding markings) are suggested by the focal point of the marker and reinforced by differences with neighbouring anchored images and the corresponding part on the main non-breeding plumage image. Annotations indicate the concepts (including abstraction level) shown in the images. Identifying the reading cues in each visualization provides something for user evaluations to test, as well as supplying alternative strategies in case test readers struggle to comprehend the visualization.

The ‘chart junk’ debate

As noted in the introduction, there has been debate within the information visualization community about the value of images in information visualizations. Early influential thinkers in the field have been critical either of figurative visualization in general ³⁵ or of specific styles of figurative embellishment (labelled ‘chart junk’ ² ). However, their views have not been supported by evidence.^5–9

We contend that the distinction between different kinds of images is important, and that frames provide a way to understand the relevant dimensions. Chart junk images have often been used to explore the effects of images on visualization interpretation.^5–7,18 However, the term ‘chart junk’ was originally presented alongside exemplars of visualization which made use of figurative illustrations. ² Moreover, empirical evidence suggests that while readers will more easily remember the overall content of a visualization if it contains relevant images, if the images are irrelevant, the user may struggle to absorb the message of the visualization. ⁸ In practice, images in visualization can be classified into three different information-bearing roles – providing background context, showing content and labelling points. ¹⁴

The figurative frame model makes the information content of a figurative element explicit, and thus provides a means to distinguish between informative and irrelevant images, and between images in different information-bearing roles. Irrelevant images are figurative elements whose frame features have no connection to the topic or other data in the visualization. It is also possible to identify the difference between background supporting images and content images, using the frame model. Background images occupy the same space as a data visualization, but the data are not anchored to the image frame. Content images have a non-trivial frame (with relevant features) and can form a meaningful background space, or sit within or alongside abstract visual elements. The figurative frame model enables the recognition that images can play many roles, not simply embellishment.

Comprehension and ambiguity

In addition to objections against embellishment and chart junk images, figurative representation has been criticized on the basis that it is ambiguous. ³⁵ While frames do not alleviate ambiguity, they do shed light on how visualizations can be misinterpreted and help place figurative visualization interpretation challenges in the broader context of issues with visualization comprehension.

The figurative frame model differentiates between the designer’s model of the object (the frame) and what is presented to the reader (the figurative element), revealing prerequisites for understanding. In particular, figurative visualization depends not only on the designer and reader sharing a visual sign system (in order for the reader to recognize the frame from the image) but also on common concepts and concept boundaries (in order to infer the correct level of abstraction).

Misunderstanding is not only a problem for figurative visualization: data visualizations can also fail to impart the intended understanding to their readers.^48,49 The risk of misinterpretation and its mitigation should be seen as an inevitable aspect of communication, rather than something that can be avoided by restricting the range of representation types.

A third argument in favour of figurative elements used in this article (and well recognized in practice) is their expressiveness. When comprehension is measured for a given visualization, the assessment is based on what the visualization can show. Yet for some problems, the information a designer wants to communicate is not able to be captured in data visualization. Figurative and abstract elements have unique communicative capabilities, and thus, the information visualization community needs methods and layouts for creating representations of both types. A designer may not know which aspects of a visual sign system are known a priori by the readers. Strategically placed annotations and other reading cues can assist in recognizing the frame of a figurative representation, thereby bootstrapping readers’ interpretative abilities and hence reaching a shared understanding through the visualization itself. Measuring readers’ comprehension of descriptive information from a hybrid visualization (i.e. frame recognition), and how different types of reading cues assist in comprehension, is an important future challenge for hybrid visualization.

Tools and techniques for hybrid visualization

Understanding the range of hybrid visualization designs provides a foundation for assessing the capabilities of tools used to construct such visualizations, and for the development of specific hybrid visualization techniques.

The classification of hybrid techniques can be used to evaluate the support for hybrid visualizations in existing visualization tools. Tools vary from accessible (e.g. excel, Tableau) to advanced (e.g. d3js, ggplot, p5js) and have the capacity to support different types of hybrid visualization. The capacity of a tool to implement particular types of hybrid visualization depends on its ability to satisfy three criteria:

Images with minimal frames can be implemented in any tool which satisfies criteria 1 (i.e. most tools including Excel, Tableau and scripting languages). To use a figurative image as a space (as shown in Figure 5), the size and proportions of the background figurative image need to match the dimensions of the abstract space. Tools satisfying criteria 2 (e.g. Tableau) simplify the calculation of an appropriate projection mapping. For example, if an image of a baseball field is attached in Tableau, the annotations function can be used to find the coordinates of the home plate within the image. The retrieved coordinates can then be used to apply a transformation (i.e. projection mapping) to hit location data so that when plotted graphically, markers are accurately located relative to the baseball field image. Figurative images can be used as spaces even in tools when criteria 2 is not satisfied, since projection mappings can be calculated manually based on measurements of the image, but tools reduce the effort required. Satisfying criteria 3 (e.g. the Data-Driven Guides tool ²¹ ) enables the implementation of frames-as-space visualizations where attributes are modified based on data (e.g. Figure 6) or combined layouts (e.g. Figures 2(e) and 11). For example, in d3js, Scalable Vector Graphics (SVG) paths can be used to draw figurative shapes whose attributes can be set and manipulated through properties added to the paths (ids, CSS classes, data). The capacity of tools to meet criteria 3 is not always obvious. For example, Tableau allows the user to upload and use figurative shapes described by coordinate paths by deliberately mislabelling an array of coordinates as a custom polygon map with longitude (x) and latitude (y) values.

The hybrid visualization types described here (e.g. frames-as-spaces) are high-level layouts or design strategies. One of the main advantages of data visualization is that more specific layouts (e.g. bar chart or ‘parallel coordinates plot’) are articulated independently of a specific tool or dataset, and so can be reused across different applications. Future efforts in understanding hybrid visualization could include the development of specific hybrid layouts useful for particular kinds of problem. For example, the hybrid layout used for the egret novice identification problem (Figure 11) could be generalized to a ‘time-varying object comparison’ technique and applied to different datasets. It could be used to compare the timing of actions for different coffee-making devices (Figure 12). Articulating such techniques makes it simpler to create visualizations and also provides a foundation for more detailed study of information visualization involving figurative elements. To create a novel hybrid visualization, a designer has to create or source figurative images of the subject, choose a figurative frame and create a layout integrating abstract data with the frame structure coordinates. Each choice is an open-ended design problem. In contrast, an existing technique guides each design step and also provides the designer with an advanced view of what the finished visualization will show. A technique requires a particular type of figurative frame and image fidelity, just as data visualization layouts require specific data types. The time-varying object comparison technique requires a ‘parts-of-the-whole’ frame which identifies changing parts of the object, and images detailed enough to distinguish these parts. A technique showing abstract data anchored to locations on an object (e.g. Figure 5(a)) would require both a quantitative coordinate frame and scale-preserving image. The more existing visualization strategies are codified, and the more their frame and image requirements are made explicit, the easier is the task of visualization designers in choosing appropriate strategies. Articulating figurative information visualization techniques also aids the scientific study of visualization. The ability to apply visualization techniques consistently to multiple datasets and to apply alternative techniques to the same datasets is a key enabler for controlled experiments measuring the performance of specific types of hybrid visualization.

OPEN IN VIEWER

Figure 12.

Application of the technique from the combined layout egret visualization to compare action and timing for different coffee-making methods.

User-centred studies based on the figurative frame model are required to refine the typology and measures proposed here and provide more specific heuristics for hybrid visualization design. Future work could include testing the ability of visualization creators to construct figurative images which convey specified frames, as well as learning from and formalizing successful strategies for communicating frame identity, precision and features. The figurative frame model also enables the articulation of specific research questions for evaluating hybrid visualization techniques. For example, studies of the effect of different mark types and styles on perceived figurative image precision would aid the development of more readable hybrid techniques. Similarly, the performance of different strategies for highlighting frame features (e.g. outlining, grid overlays, description in accompanying text, aesthetic style) and interference between such highlighting strategies and abstract data marks would allow the identification of compatible (or incompatible) combinations of abstract and figurative representations.