Abstract
Based on a larger practice-based research project in digital writing, this article examines how the materiality of digital media contributes to a layered metaphor that delivers meaning, reflects on the cognitive processes (the writer’s and the reader’s) of navigation and generates a dynamic narrative structure through multimodality, unnatural narration and user interaction. Many writers and artists engage with their chosen medium through an instinctive understanding of the materials at hand, gained through experience; the explicit study of a medium’s materiality is not always required for artistic success, however, that may be judged. This article offers insights into the creative process of creating digital, multimodal fiction, based on a practice-based research project designed to explore the effects of digital media on author and text, and argues that digital media have a significant effect on the outcome of the artefact itself. Awareness of these effects, their variations according to hardware and software, and the affordances of these various materials offer the digital writer greater insight and capability to craft his/her texts for the desired metaphorical meaning.
Keywords
Introduction
The materiality of fiction narratives is, ironically, a rather intangible concept, particularly as the notion of materiality traditionally relates to specifically tangible tools of creation, such as the painter’s brush or the sculptor’s clay. In presenting her theory of the technotext, 1 however, Hayles calls for media-specific analysis in the literary arts, one that includes an examination of materiality, accounting for ‘how the work mobilizes its resources as a physical artifact’ in terms of both physical manipulation and conceptual frameworks 2 (2002: 33). Hayles argues that it is the conjunction of the physical embodiment of technotexts (whether semi-tangible in digital form, or as fully physical as a book) with their embedded verbal signifiers that constructs both plurimodal meaning and an implicit construct of the user/reader (2002: 130–131). This practice-based research article seeks to examine the dynamic on the other side of technotexts: that of the creator and the text. Specifically, this article explores how the materiality of digital media contributes to a layered metaphor that delivers meaning, reflects on the cognitive processes (the writer’s and the reader’s) of navigation and generates a dynamic narrative structure through user interaction.
This materiality is thus not physical, and unlike other narrative media that unfold in a single material layer (such as ordered text in a novel or the sequential images in film or comics), digital media afford multiple layers of materiality in multiple possible orders. Thibodeau categorizes digital material objects as physical inscriptions, logical processes and conceptual objects (2002: n.p.); Kirschenbaum draws upon these categorizations in his approach to mechanisms in digital media (2008). These texts, among others, establish an extensive framework for the study of digital materiality in the overall digital humanities, addressing questions of transmitting, preserving and archiving digital objects. This article, however, focuses on the digital medium as artistic tool and composition medium, the processes of inscribing a conceptual object through logical processes into digital form; as such, it also draws upon Drucker’s more phenomenological model of materiality (1994), examining the narrative effects that result from the cognitive processes and procedures involved in composing narratives through digital materials.
The expansion of the narrative space in both temporal and spatial dimensions calls for a requisite expansion of composition strategies in the writer’s cognitive spaces. Flower and Hayes’ (1984) Multiple Representation Thesis poses the notion that even when writing prose, the author’s ideation is multimodal, inspired by images, sounds, interactions and associations, as well as by language; the act of prose writing is a process of translating these cognitive pieces into a single textual layer of ordered language. The construction of digital texts, however, enables the author a more direct translation of these multiple modes into images, sounds and words, while requiring him/her to layer them in a collage of digital materials constructed in multiple spaces. The digital text that emerges is thus a mosaic of different digital materials, each one bringing its own narrative and cognitive effects for both writer and reader and resulting in a layered multiplicity of meanings (Lemke, 1998).
The argument for media-specific analysis for these layered texts is important in both post-textual analysis and practice-based analysis. The materiality of a storytelling medium such as film is a fairly straightforward notion to grasp, because many of the tools and artefacts of the medium are physically graspable: cameras, celluloid, reels, scissors, props, lenses, filters, lights and so on. The materiality of digital artefacts, however, lies only superficially in the haptic hardware of screens, keyboards and mice; the materiality of modes, navigation and interaction must also be explored for their effects on metaphor and meaning. Serge Bouchardon and Davin Heckman identify three levels of materiality in digital literary works: the figure of a semiotic form, the grasp required to physically interact with the work and the memory of the work – its whole compiled from the parts of code, hardware and user/reader experience that form meaning in cognitive spaces (2012: n.p.). This memory ‘relies entirely on the materiality of the trace, the immediacy of the recording, the visibility of the image’ (Nora in Pence, 2002: 346). Without consideration of these material aspects of digital works, ‘we have little hope of forging a robust and nuanced account of how literature is changing under the impact of information technologies’ (Hayles, 2002: 19). More importantly, without a similarly robust and nuanced understanding of how these technologies affect creative cognition and the resulting artefact, digital storytellers may be hard-pressed to craft works that create these levels of metaphor and meaning through the interplay of apparatus and text.
Often such an understanding is not a conscious process; many writers and artists engage with their chosen medium through tacit knowledge (Nonaka and Takeuchi, 1995): An instinctive understanding of the materials at hand, gained through exposure to others’ works and through their own experiences. In other words, the explicit study of the materiality of a medium is not always required for artistic success, however, that may be judged. As this article will demonstrate, however, digital media have a significant effect on the outcome of the artefact itself; awareness of these effects, their variations according to hardware and software and the affordances of these various materials offers the digital writer greater insight and capability to craft his/her texts for the desired meaning.
Methodology
Practice-based research
While practice-related research has always been present to some extent in the arts and humanities, in recent years, artistic practice has developed into a major focus of research activity, both as process and product, and several recent texts (Brophy, 2009; Carter, 2004; Gray and Malins, 2004; Macleod and Holdridge, 2006; McNiff, 1998, 2013; Smith and Dean, 2009; Sullivan, 2010) as well as discourse in various disciplines have made a strong case for its validity as a method of studying art and the practice of art. Practice-related research ‘[involves] the identification of research questions and problems, but the research methods, contexts and outputs then involve a significant focus on creative practice’ (Sullivan, 2009: 48). The outcomes of such research are intended to develop the individual practice and the practice of the field, to build theory related to the practice in order to gain new knowledge or insight (Niedderer and Roworth-Stokes, 2007: 10; Sullivan, 2009: 48). Research in which the creative artefact is the basis of the contribution to knowledge about the nature of creative practice is defined as practice-based research (Candy, 2006). This method is applied to original investigations seeking new knowledge through practice and its outcomes and forms the foundation of the method used for this article. This method is detailed more thoroughly in Skains 2016a.
Graeme Sullivan’s (2009) model identifies a framework of four key areas in which a practice-based research methodology is applicable and appropriate: theoretical, conceptual, dialectical and contextual. The research communicated herein represents Sullivan’s second category, conceptual, wherein ‘artists give form to thoughts in creating artefacts that become part of the research process’ (2009: 50); in my work, I am interested how constructing narratives in different media affects me as a writer and the structures of the stories that result.
Observation and analysis
The notions of practice-based research discussed above serve as an overarching methodology, within which specific methods of observation and analysis must be applied. This section offers a brief overview of the implementation of these practices and measures; the details of my particular method, including the inherent affordances and limitations, are more thoroughly outlined elsewhere (Skains, 2017b).
Ethnomethodology
Reflective analysis is probably the method most frequently applied by practitioners to their creative projects; however, dependent as it is upon memory, and conducted after the creative act rather than during (or as close to as possible), reflection is an unfortunately fallible method (Edmonds et al., 2005). Thus, I argue that practice-based research in creative writing calls for the employment of a self-directed form of ethnomethodology during the composition of the texts, in the form of a research log (noting insights, process, difficulties) and draft materials and revision notes (which could later be analysed as in situ utterances). Ethnomethodologists observe their subjects’ speech and activities within a given context in order to make these actions ‘visibly-rational-and-reportable-for-all-practical purposes’ (Garfinkel, 1967: vii). Deborah Brandt argues for just such a practice of ethnomethodology for writers (1992), noting that ‘[s]ense-making in writing entails more than producing a coherent and appropriate text; fundamentally, writers must also make continual sense to themselves of what they are doing’ (1992: 324). The process of this continual sense-making is expressed in notes, journal entries and comments on revised drafts: observable paratexts to the composition.
While I acknowledge the limitations of self-observation and reflection, this methodology also attempts to mitigate these limitations by stipulating that the practitioner-researcher (A) approaches the creative activity from a clearly defined research question; (B) observes his/her activities in situ, but interpret these observation records (creative notes, drafts, research logs) after a time period that allows for a distanced perspective and (C) supplements these observations of process with media-specific analysis of the creative artefacts themselves (discussed below). Combination of methodological approaches provides a more robust approach to examination of creative practice than reflection or post-textual analysis provides on their own.
Media-specific analysis
While the post-textual analysis methods may vary according to the art, genre, practice and/or research question at hand, I am particularly interested in fictional narratives and digital writing. Narratology theory provides the broad foundation for critical approaches to digital writing through: transmedia narratology (Ryan, 2006), cognitive narratology (Herman, 2007) and unnatural narration (Alber et al., 2010, 2012; Alber and Heinze, 2011; Bell and Alber, 2012; Richardson, 2006). Transmedia narratology offers insights into the techniques and structures a text utilizes across and within media. Cognitive narratology enables yet another approach to understanding the process of composition. Finally, theories of unnatural narration contextualize digital works, which remain largely outside of natural narration and convention, within the larger literary domain.
Within the overarching theoretical framework of narratology, the base for examination of the creative artefacts for meaning-making lies in Hayles’s (2002) media-specific analysis (MSA), which facilitates analysis of the materiality of the multimodal texts, and how that materiality shapes the resulting narrative. This MSA includes semiotic analysis of visual grammar and design (Kress and van Leeuwen, 2006), of hyperstructures such as navigation and interactivity (Bouchardon and Heckman, 2012; Ryan, 2006) and of source code (Marino, 2006; Montfort, 2003, 2011). This approach is applicable not only to a digital work as displayed, in order to examine the effects of digital media upon the works themselves, but also source code, in order to discuss aspects of process and composition.
As both proponents and opponents of practice-related research have noted, it is impossible to entirely separate authorial intention in the creation of a work from their post-textual analysis in this process of observation and analysis. One necessarily informs the other: Awareness of semiotic modes and critical theory on narrative structures guides and shapes the creative process; likewise, knowledge of the creative decisions made for particular audience affect influences the textual analysis. Thus, the post-textual analysis in this practice-based method carries with it the limitations of the blinkers created by authorial intention and desire as well as the advantageous expansion of insight into that particular work given through the creator’s self-awareness and critical awareness.
The materiality of technotexts
The following sections examine the materiality of technotexts, exploring how the material aspects of multimodality, navigation and interaction influence the literary artefact in terms of structure and meaning. Each section presents examples of how the materiality of texts affects narrative meaning and examines the elements of the author’s own digital fiction storyworld, Færwhile 3 (Skains, 2013), as well as other contextual technotexts, that demonstrate these effects.
Multimodality
Digital texts are frequently multimodal, creating meaning through text, image, sound and movement. While these modes may be used to illustrate one another, as when an image is used to illustrate an article, or merely to provide a pleasingly aesthetic textscape, most multimodal works create meaning through the interplay of the modes used: Meanings in multimedia are not fixed and additive (the word meaning plus the picture meaning), but multiplicative (word meaning modified by an image context, image meaning modified by textual context), making a whole far greater than the simple sum of its parts. (Lemke, 1998: 312)
An example of this multiplicative meaning, or ‘pluricode’ (Saemmer, 2012), can be found in Campbell’s (2009) 4 ‘Consensus Trance, Part 1’, the first chapter of his multimodal and multimedia work Nightingale’s Playground. The narrator in this Flash story is driven by an inner conflict, a desire to discover what of his memory is real, and what is merely delusion. Campbell uses text, image, interactivity, light, colour, movement and sound to express this inner conflict. The story begins in the ‘bedsit’ sequence, toned in browns and greys to reflect the sour, depressed mood of the narrator, the peeling wallpaper and stained mattress of the barely furnished room reflecting his dire circumstances. The room is poorly lit, sunlight from the one window unable to reveal the hidden shadows. This searching quality is reflected thus in the mise en scène of the sequence, as well as the action the reader must carry out to reveal the lexias: Mouseovers of the entire screen reveal four segments of text, the narrator describing his circumstances, his search in both physical form and mental memory of a high school friend he is no longer sure exists outside his own mind. The modes in this sequence coalesce into a ‘coherent coupling’ (Saemmer, 2012), as the meanings of each (colour, lighting, text, image) combine to denote a coherent whole, shaping the narrator’s shadowed and fragmented memory.
Multiple modes can also be used in ‘de-coherent couplings’ (Saemmer, 2012), in which the meanings of each mode seem to contradict one another, perhaps leading to a third meaning. Scott’s (1982) film Blade Runner presents an apparently straightforward character, script-wise, in Rick Deckard (Harrison Ford), the blade runner tasked with hunting and killing four rogue ‘replicants’ (androids) who have illegally escaped their duties on extraterrestrial colonies to return to Earth in an attempt to extend their own short lives. The replicants, in both script and visual elements of the film, continually pose the question ‘What does it mean to be human?’ Roy Batty (Rutger Hauer) demonstrates this central theme through the text, the script, in his actions to find his creator, his drive to extend his own life, stressing his own humanity in statements such as ‘We’re not computers, Sebastian. We’re physical’ and through his final speech that encapsulates his memories, his life. The visuals offer a cohesive coupling of this question, through their repeated use of eye and animal imagery (Skains and Chambers, n.d.).
The character of Deckard, however, presents an example of de-coherent coupling. The scripted dialogue presents Deckard as a hardened blade runner, never questioning his own status as human. The visuals, however, offer a contradictory meaning: Deckard is frequently associated with the colour green, committing to neither the blue associated with the mechanical replicants, nor the yellow representing natural life. Further, Deckard is linked through his own dream imagery and Gaff’s (Edward James Olmos) origami animals to the figure of the unicorn, which provides more contradiction around the question of his own humanity. The visual of the dream unicorn questions Deckard’s status as human; he is the only character represented by a mythological creature (Burt, 2002: 74). Similarly, the article unicorn Gaff leaves for Deckard in the final sequence conflicts with the notion of Deckard as fully human: How can Gaff know the contents of Deckard’s dreams unless they are the programmed memories of a replicant? ‘The controversial unicorn [image] perhaps reflects Deckard’s hidden replicant desire to become something mythical, something that no longer exists in his word: truly alive’ (Skains and Chambers, n.d.).
An examination of ‘chapter 1’, the first chapter in Færwhile, reveals both coherent and de-coherent coupling, even within the opening frames, an intentional effect created through the interplay of the different semiotic modes contained in the text. The background image is an image I chose to signify summer and warmth, togetherness and family: a beach scene with bright skies and families playing in the sand. The text, however, tells the tale of a brother and sister who, orphaned, are separated by the foster system, each lost to one another; the narrative content of the text against the beach background offers a de-coherent coupling. The tonal quality of the image also offers a coherent coupling with the narrative content: I adjusted the colours for overexposure, turning the bright sky into a flat white, the shadows in the foreground black and cold, signalling a harsh, almost alien environment. Given the narrative content, this would indeed be an alien environment to the brother and sister in question, who have never experienced the comfort and apparent normalcy of a simple day at the beach amid family and friends. Similarly, the verbal style of the text offers yet another de-coherent coupling. The text uses a lilting, storybook voice, beginning with ‘Once upon a time’, which signals a fable with a comforting ending; this comfort is quickly belied by the narrative and visual shift into the deepest of the shadows on the screen. Combined, these two modes – the visual and the written text – offer layers of meaning in this sequence that neither offer alone, opening a story whose ontological level is about two lost siblings seeking one another in an expression of love and family and whose metaphorical level reveals the manipulations and machinations of external and internal powers that, in the end, leave everyone unfound.
It is only through consideration of the full complexity of the multimodality of these texts that a full realization of their meaning can be reached. Similar to Hayles’ note on the recursive quality of varying media, so do multiple modes within one work ‘engage in a recursive dynamic’ (2002: 30) to reflect and refract meaning through various layers, levels and angles of multimodal fictional narratives.
Navigation
Navigation in texts provides yet another of these layers, forming a significant ‘part of the work’s signifying structure’ (Hayles, 2005: 91), offering a mechanism for ‘active manipulation of features on the level of discourse and presentation’ (Drucker, 2008: 121). Narratives, whether fiction or nonfiction, natural or unnatural, share a set of structures that define them: spatial elements such as setting and objects, temporal elements indicating a sequence of related and often causal events, intelligent agents who take actions based on these events and a sense of closure or denouement (Chatman, 1978; Ryan, 2004, 2006). While some elements of narratives can be experimented with, rearranged and remediated, the cognitive action of the audience pieces the narrative puzzle together to construct this recognizable shape (Douglas, 1992).
In presenting these puzzle pieces of structure to the audience, narratives can employ unicursal navigation, a singular pathway through the arc of narrative events leading to climax and closure – as typically offered in the novel – or multicursal, offering multiple paths through, as offered in hypertexts (Aarseth, 1997; Hayles, 2001). The technology of the printed page, bound into an ordered codex – the materiality of the book – largely dictates a unicursal navigation of the narrative within, as the reader engages in the ingrained action of reading from left to right (in Western cultures), top to bottom, front to back. Some texts, digital antecedents or ‘cybertext[s] in antiquity’ (Aarseth, 1997: 9), attempt to disrupt this expected unicursality by unbinding the codex and shuffling the pages (Saporta’s (1962) Composition No. 1, Roman), directing the reader to pages or chapters ‘out of order’ (Cortázar’s (1966) Hopscotch) or deviating from the norms of narrative structure (Calvino’s (1981) If on a Winter’s Night a Traveler).
The materiality of digital media, however, readily affords multicursal navigation. Persson (1998) identifies four types of digital navigation: the spatial (up-down, left-right) navigation popular in graphics-based games; the social navigation present in discussion forums and social media sites denoting how much and what type of activity is occurring; the semantic navigation connecting objects in the digital environment through ‘some semantic connection like similar, alike, more/less general, associated’ (Persson, 1998: 191, emphasis original) and the navigation inherent in narrative structure. The navigational possibilities in digital media are thus expanded to a significant degree, whereas unicursal narratives normally employ one method of navigation (narrative structure, as described above), digital media afford many different combinations of navigation within a single work. Digital fiction most commonly affords spatial, semantic and narrative navigation; fictions engaging in social media tools such as forums and blogs also engage social navigation. Joyce’s (1987) afternoon: a story offers a multicursal path through the narrative, as the reader navigates through the segments of the hypertext through semantic links, digging deeper into the narrative structure even as repeated lexias reveal meaning through their very repetition. Short’s (2006) Bronze, an interactive fiction (IF) adaptation of the ‘Beauty and the Beast’ fable, offers spatial navigation through the Beast’s palace, the reader navigating the narrative structure by exploring the rooms and objects afforded by the IF. These exploratory, nonlinear forms of navigation afford narrative pathways unique to each read-through of the work, with repeated readings offering an additive effect as different perspectives and event orders congeal to reveal multiple layers of action and causation in the narrative structure.
Both afternoon and Bronze are examples of ‘wayfinding’ navigation (Benyon and Höök in Persson, 1998: 192) – the reader/navigator has a clear quest to discover what has happened on the afternoon of the accident in the case of the former and a quest to save the Beast in the latter. ‘Exploration’ navigation, the reader/user exploring a text with no clear goal or to get an overview, can also occur in hyperfiction (questing to reveal all lexias), IF (visiting spaces and examining objects that contribute to storyworld but not necessarily narrative) and most obviously games such as online role-playing games or virtual worlds that provide a significant level of spatial navigation. These strategies neither add to nor subtract from the foundational structures of narrative; they merely offer different approaches for the reader to seek out and collect the varying pieces of the narrative structure. Some narrative media, such as exploratory virtual worlds, emphasize spatial elements such as setting and objects while encouraging the user/player to provide the remaining elements of intelligent agents and events, offering pleasure through mimesis, fantasy and interactive activity (Douglas and Hargadon, 2000). Others, such as novels and films, provide all the narrative structures, thus relieving the reader of the burden of contribution, offering cathartic pleasure through an emotional connection to the characters, suspenseful development of conflict for those characters and a sequential resolution of that conflict (Aristotle, 1968; Hiltunen, 2002). Digital fiction as a medium affords a continuum of navigational strategies for the creator to select, depending upon the intended effect on and affect for the reader.
While Færwhile’s ‘chapter 1’ seeks to engage the reader in the familiar straightforward narrative structure navigation in its role as the introductory piece, I chose to employ more interactive strategies in later chapters. ‘chapter 4’ and ‘chapter 3’ offer more complex navigation strategies through spatial exploration and semantic associations. The navigation paradigm in ‘chapter 4’ mirrors the narrative’s events: the characters are wandering separately across a landscape, eventually merging together. The imagery in ‘chapter 4’ is that of a map; the reader must navigate the space of the map and seek out clickable areas (semantic links) that reveal storybook chunks of narrative related to those areas. The more the story they find, the closer they progress to the castle, where all the characters eventually converge and this sequence concludes. The piece progresses in sections; each section is exploratory, enabling multicursal pathways within, but the sections themselves progress unicursally. Exploration through spatial navigation is possible in small areas, in other words, but overall the reader is manipulated down a unicursal narrative pathway toward the denouement of the segment. The technotext thus provides a navigational mirror of the narrator’s ontological manipulation: just as the Trickster has manipulated the characters down their various paths (while still allowing for deviances along the way), so too does the navigation in ‘chapter 4’ manipulate the reader through the narrative, reflecting the underlying metaphor of the tale.
‘Chapter 3’s’ navigational structure is similar to Bronze’s, with opportunities for exploration through the spaces of the storyworld, but also a wayfinding structure in that the player-character is navigating Lilly through the world toward the goal of finding her brother. The possible pathways of IF are by nature multicursal and unicursal simultaneously: The player-character’s choices move them through the narrative in many possible pathways, but the overarching goal of the IF is to ‘win’ or ‘succeed’ by achieving a successful traversal (Montfort, 2011). This again, much like in ‘chapter 4’, requires manipulation of the player-character toward actions along a unicursal path, enabling a ‘successful’ ending to the narrative, in which the player-character as Lilly escapes this world with Hal as her companion. As the reader explores and discovers, and occasionally gets lost, so too does Lilly; the exploratory nature of IF enhances the narrative metaphor of the lost little girl, navigating strange places with strange expectations.
Through devices such as the hyperlink, spatial movement (whether text- or graphics-based), semantic feedback in various modes and even emerging social tools such as integration of social media into narrative spaces, digital media afford a staggering degree of possibilities – to the writer and the reader both – for the recombination of narrative. Drucker, in her examination of the navigational effects of graphic devices, argues that the cognitive processes in a reader that piece together narrative existents and events into a coherent story (regardless of order or form) function not only because of the content of the text but also because of how it is ordered and presented (2008). ‘Depending on how the designer chooses to organize the [digital] environment, it will give rise to different types of experiences in the user/player/reader/navigator’ (Persson, 1998: 191) and possibly to previously unimagined structures of narrative as well.
In the writing of the Færwhile texts, these cognitive processes in the writer also came into play. The branching game structure, navigated through reader input and the Inform7 parser, played a significant role in the realization of the final text. Composing in Inform7, perhaps moreso than any other medium employed in the Færwhile project, necessitated a firm divide between the text-as-composed (the source code, written in Inform7’s specific programming language) and the text-as-played (the game script that results from the reader-player’s commands). Jenny Weight, based on her own digital composition practice, described this as an effect of the text-as-apparatus: ‘As author mutates into programmer, texts transform into a range of possibilities and circumstances – it may be better to conceive of texts in the text-as-apparatus as environments rather than as traditional narratives’ (2006: 434). Writing with a plan for interactivity opens up multiple possible paths the character could take, forming multiple ‘potential narratives’ (Montfort, 2003: 14); not only must the reader-player navigate the story in its final form, so too must the author navigate the text and its multicursal pathways in its very construction. This navigation took me into many unconsidered pathways for the narrative (as well as closing some off, due to constraints of the medium), significantly altering the narrative-as-planned. 5
Interaction
Digital interfaces afford various levels and ways for the reader/user to interact with the text. These interactions typically arise from the physical gestures (typing, scrolling, mousing over, clicking, tapping) necessary to use most digital devices. Their effects within the digital environment, however, can range from a simple click to advance to the next section of story, to highly engaged interactions such as those requiring gameplay or typed commands. ‘The reader’s physical as well as cognitive encounters with a text as much form the basis of the text as the words and links provided by the author’ (Nack, 2009: 15–16); the text is realized through the physical and cognitive interaction between reader and apparatus.
Bouchardon and Heckman’s notion of a ‘figure of manipulation’ in digital works provides a rhetorical model for evaluating how gestures of navigation and manipulation (mousing, keying, etc.) add elements of metaphor, metonym and synecdoche to the text ‘based on the user’s interaction with the interface’ (2012: n.p.). The capability for the user to interact with digital elements and by doing so discover more text than is initially apparent on the screen adds significant depth to the digital work. ‘When interactive text is manipulated by the reader, the linguistic sign is again coupled to an iconic sign: a sequence of gestural manipulations performed for a purpose’ (Saemmer, 2012: 8).
These gestures become what Saemmer terms ‘semiotic units of manipulation’ (2012: 8), as certain manipulations become associated with particular meanings. In Alan Bigelow’s interactive self-portrait Because You Asked (2006), several figures of manipulation are at work. The reader must click on icons to reveal lexias (which are presented in both text and audio), a simple gesture calling forth the next segment of the piece. With each lexia, however, a segment of the artist’s portrait is revealed. In the final sequence, a mouseover of the revealed portrait erases the image wherever the cursor moves. The mouseover gesture signifies erasure, wiping out a fleeting image. Heckman describes the significance of this interaction: Rather than the act of interacting via a purely technological interface, ‘Because You Asked’ implies that reader involvement takes place at a more fundamentally human level, that of curiosity, imagination, and consciousness, suggesting, perhaps, that we see ourselves as much as anything else in the things that we look for. (2009: n.p.)
These questions arise in the final chapter because they developed over the course of writing Færwhile. The deeper I delved into multicursal story structures, navigation into branching narratives and composition on several levels of code, prose and screen image, the fuzzier the notions of narrator, narrative power and authorship became. The numerous instances of unnatural narration (Alber, 2011; Alber et al., 2010), in the forms of metalepsis, multiple narrators and direct address, emerging over the course of chapters composed in various textual machines (source code, prose, image manipulation, image borrowing, etc.) led me to question my own text and to revel in the loss of power from author to character. An inherent function of the technotext’s materiality, per Hayles’ (2002) discussion, is the questions it raises in the reader’s experience about the actual meaning of the text; in my practice-based research, I found that the materials of technotext construction raise these same questions for the author as well and necessitate a deeper engagement in order to approach and present meaning for the eventual reader-player.
Whereas in the novel, one strives for the physical materiality of the text (paragraph breaks, page turns) to fade away as the reader immerses in the narrative, the gestures and manipulations involved in interacting with digital texts can add yet another layer of meaning, metaphor and theme to the narrative. To the extent the user enters the imaginative world of this environment and is structured by her interactions with it, she also becomes a simulation, an informational pattern circulating through the global network that counts as the computational version of human community. (Hayles, 2002: 49)
Conclusion
Materiality is a significant contributor to the cognitive processes of both creating and experiencing digital fiction. The materiality of digital media, and of specific media platforms such as Flash or Inform7, implies certain affordances and limitations unique to these forms. As such, the writer’s approach to narrative, as well as the structure and shape of the narrative itself, adjusts and transforms in order to engage fully with the new media.
Hayles notes that ‘electronic authors are normally involved in every aspect of the production process, which includes the appearance of the interface, the linking mechanism, animation, audio files, and image generation and placement’ (2001: 23). While some authors collaborate with digital designers and programmers, I chose to undertake this more embodied approach in order to gain a more thorough insight into how the work’s materiality influences author and narrative. Digital media offer the capability to produce multiple modes equally, thanks to the underlying programming that transforms code into image as easily as it does text or audio. Based on this foundation, the electronic author can transduce these modes: shifting semiotic material across modes, layering meaning through multimodality, navigation and interaction.
As this article demonstrates, extensive and nuanced knowledge of how these modes affect and transduce meaning is required to make full use of them in creating digital narratives (Kress, 2003). The layering of multiple communication modes within a single text produces a multiplicative meaning (Lemke, 1998), as different elements interact to offer either coherent or de-coherent coupling (Saemmer, 2012) that shape the underlying metaphor of the narrative. Spatial, semantic, narrative and occasionally social navigation can be used to mimic actions of exploration, to provide associated links of meaning and to influence the reader to construct a path through a potential narrative, its metaphor structured in part through these navigational clicks and choices. Even the very action of entering commands, clicking on buttons and links, mechanically spinning the narrative wheel through the digital device is a choice that affects the reader’s experience of the narrative, and thus the communication and cognitive construction of the narrative’s metaphor and meaning. Digital media offer this dizzying array of narrative devices in addition to those that are familiar through reading and literary study, and thus the authorial choices for creation of narrative are significantly increased beyond the unicursal presentation of written language. The author of the technotext must appreciate and use all of the semiotic and cognitive capabilities of the apparatus at hand.
Flower and Hayes’ (1984) multiple representation thesis poses the notion that even when writing prose (with its monomodal, unicursive outcomes), the author’s ideation is multimodal, inspired by images, sounds, interactions and associations as well as by language; the act of prose writing is a process of translating and ordering these modes and ideas into ordered language. From a writer’s perspective, digital media afford a more direct transcription of the original concepts, as Alan Sondheim notes: ‘As far as writing is concerned – I don’t care whether or not I’m writing/sounding/visualizing; it’s all a mix, all developed cross-application, cross-platform, cross-technology, cross-output devices’ (2006: 376). Rather than constraining the ideas and possibilities to one unicursal narrative, digital media afford multiple possibilities to present in a single text, a ‘text-as-apparatus as environment rather than as [a] traditional narrative’ (Weight, 2006: 434).
This material mapping (Hayles, 2001: 31) transfers to the reader of these texts, as ‘the reader’s physical as well as cognitive encounters with the text as much form the basis of the text as the words and links provided by the author’ (Nack, 2009: 15–16). Gaudreault and Marion argue that the text’s fabula 7 is manifest not only in the syuzhet’s text but also the structure of the syuzhet (2004: n.p.); the materiality of the syuzhet’s medium not only informs but actually forms the text, ‘alter[ing] the conditions of reception’ (Ryan, 2009: 4). Kirschenbaum posits that the reader’s ‘forensic imagination’ is thus activated, as the ‘process collapses into product’ (2008: 253). Choices made in the navigation of the text, gestures carried out in order to explore the text and the multiplicative meanings triggered by the multimodal layers coalesce into a mental model of the narrative (Persson, 1998: 193), relying upon the same cognitive processes in the reader to construct the text as the writer engaged in creating it.