Crossing media boundaries

Abstract

It is necessary to continuously review the definition of the book moving from one bound by its material form to one determined by its function as a means of communication. The book’s social function as the high status vehicle for communicating new ideas and cultural expressions is being challenged by sophisticated systems of conveying meaning in other media. In this article, we report on two projects: electronic book (e-book) publication and reader forum for Nature Mage and the transmedia augmented reality (AR) fiction Sherwood Rise, which investigate these issues. Claudio Pires Franco’s work is based on the adaptation of a source work: Duncan Pile’s Nature Mage. The project aims to develop the book from e-book to a fan-produced enhanced digital book. Through this practice-based research, Franco investigates the definitions and classification of the e and i forms of the book and adaptation in new media; the role of the author in creative collaboration with readers through online forums; the extension of the story world through creative collaboration and reader participation while respecting and safeguarding creative properties. One remove from the traditional book, David Miller’s Sherwood Rise, research the user experience with AR to examine narrative problems and explore new storytelling aesthetics. These new media forms define the outer borders of the book system within which content is formed and moulded, and around which society is shaped.

Keywords

Activism adaptation audiences augmented reality authorship book history categorisation e-book e-reader film iBook media adaptation readers scriptwriting

Introduction

AR is poised to transform the storytelling of twenty-first-century journalism in perhaps the same fashion as photography of 150 years before. What may be needed today is a modern-day Walt Whitman or Samuel Morse who can see the potential in a new and novel technology whose potential is perhaps best captured in poetic verse. Where is this digital wordsmith, the journalist of augmented storytelling vision? (Pavlik and Bridges, 2013)

Jonathan Westin points out in an earlier article in Convergence, ‘As the concept of digital books – ebooks and ebook-readers – has become ubiquitous, and the book’s translation from an analogue format to a digital one is well under way, questions arise regarding how those cultural values which are negotiated around a physicality translate to a digital sphere’ (Westin, 2013: 8). In This is not the end of the book, Jean-Claude Carrière and Umberto Eco similarly discuss the book as a cultural concept but ‘do not linger on the physical format as a generator of “culture”’. They contend that ‘the concept of hypertextuality is bound to damage’ that ‘peculiar intimacy between the author and reader’ (Carrière and Eco, 2011: viii). Westin however points out that the cultural connotations of a format are just as important as the content for the ‘limitations of a given format are identified as actants enrolled by stakeholders in the translation process, and, as such, anchor the format to society’ (Westin, 2013: 5).

As a way into understanding the cultural connotations of the book, Alexis Weedon considered the values of metaphors used about them: the codex as ‘a hinge’ binding pages of fixed and locatable ideas; the cut-glass wine goblet that holds and displays its content to its best advantage; and the rose, a symbol of the intensity of the relationship between the reader and the text and how that is transferred to the reification of the object itself. She argues that the value system evident in these metaphors and which underpins the status of the book is changing. The book’s social function as the high-status vehicle for communicating new ideas and cultural expressions is being challenged by sophisticated systems of conveying meaning in other media. As an example, she shows how an individual book changes value during its existence in terms of price, inherent or cultural value, exchange value and use value (with global positioning system (GPS) navigation and satellite views of the earth, for example, the use value and inherent value of a map and an atlas have changed) and yet gains cultural and economic value through its different adaptations and media forms (e.g. the longevity and many adaptations of Sherlock Holmes). Weedon believes the concept of the book will cross into the digital era and that the digital book will seek to retain the essential characteristics of its predecessor including the intensity and depth of the relationship between reader and book. In this article, two projects investigate these issues: electronic book (e-book) publication and reader forum for Nature Mage and the transmedia augmented reality (AR) fiction Sherwood Rise.

Claudio Pires Franco’s work is based on the adaptation of a source work: Duncan Pile’s Nature Mage series is published in e-book and print format and the project aims to develop the book from e-book to a fan-produced enhanced digital book (Franco, 2013) (Figure 1). A ‘normal’ e-book and a printed book are very similar in terms of content – both contain text, and in the case of Nature Mage, the same text, and the same cover – although they offer different reading experiences. The project invites readers to contribute backstories, side stories and visual representations of the story universe to feed into the site and ultimately the iBook. The main narrative is still in the original books, but the fan website, the iBook, and social media such as YouTube are used to spread content and expansions to the story universe. The three aims of the project are to investigate (i) the definitions and classification of the e and i forms of the book and adaptation in new media; (ii) the role of the author in creative collaboration with readers through online forums; (iii) the extension of the story world through creative collaboration and reader participation while respecting and safeguarding creative properties.

Figure 1.

The Nature Mage fan site http://www.naturemage.com/.

David Miller’s Sherwood Rise is an AR story (Figure 2). The story is a free adaptation of the Robin Hood tale placed in the contemporary context of the global economic crisis. It is an experiment in multilinear, episodic storytelling, using the accessible technologies of open-source software and android tablet, Internet and email. The project investigates the application of available technology to multilinear storytelling and explores the affordances of AR narratives and the effect of AR on story construction and character.

Figure 2.

Sherwood Rise Project link: http://itsthetruth.org.

Problems arising from the classification of new media forms of the book

There are a wide range of new and emerging media forms of the book. Printed books can be easily remediated into digital media as e-books and read on electronic book reader (e-readers), or consoles as is the case with electronic arts’ (EA) flip books. But authors and publishers are going a lot further than that, reaching a point where it is not clear whether we are faced with a ‘book’, or simply a story told in some other new way. ‘Enhanced’ books, for example, are digital books that not only contain text and illustrations but also make more use of computational capacities, adding hyperlinks, game, video and other features. There are also ‘interactive books’, for example, picture books for children on the iPad, whereby the reader can tap into objects to hear sounds, watch animations, unravel the story further and so on. A further form is the iBook, which integrates multiple modes, from text to sound, to moving image, three-dimesional (3D) objects and even dynamic features (widgets) produced in HyperText Markup Language 5. In many ways, an iBook is closer to a multimedia website than to an e-book. But it is presented as an iBook, produced with a software programme called iBooks Author, and its visual representation remediates the format of a book (Bolter and Grusin, 1999). Even the iBooks menu on an iPad is a representation of a bookshelf where iBooks are exhibited; when you select an iBook, it pops out of the shelf, and as it gets bigger it is possible to see in some detail the edges and curves of the book cover. AR books add 3D content and sound to extend and enhance the text for learning (e.g. Carlton Kids series) or storytelling purposes as part of the merchandising of brands (e.g. Disney Cars, Moshi Monsters). These are based on paper forms of the book with AR content added on. There is clear evidence of a relationship between the digital book and older paper form, which designers wish to retain and signal. Similarly the status of book apps and AR books has been raised by adaptations of classics such as Robert Stevenson’s story ‘Strange Case of Dr Jekyll and Mr Hyde’, Frankenstein and TS Eliot’s ‘The Waste Land’ (Hügli and Kovacovsky, 2010; Morris, 2012; Eliot, 2011). Furthermore story app publishers like Les éditions volumiques, Somethin Else and Fake Press have explored other remediated forms: fold-out books, animated maps, video games, tablet board games and crossovers between paper and tablet publications (Dawkins and Mckean, 2011; FakePress, 2011; Les éditions volumiques, 2013). Thomas Leitch calls this a ‘postliterary phase’ of adaptation (Leitch, 2009: 278).

The high levels of media convergence that characterise the current marketplace make any attempt at a simple, neat taxonomisation of media texts difficult. The source book and e-book are indeed accessed through different media, but the e-book is arguably not much else than a remediation of the printed book, adding little to it other than the experience of reading using a different device. Cross-device functionality has been strategically introduced by brands, so, for example, the Kindle e-book and the iBook can be accessed using the same device, an iPad tablet or a Mac computer. Nowadays computers and the newest type of wi-fi-enabled tablets, smartphones or even smart TVs can be used to search and read e-books, websites and some types of enhanced books, all of which may be open simultaneously.

There have been attempts to classify the range of core and ‘ancillary’ (but very self-sufficient and independent) outputs and texts generated across different platforms, devices and media. Systematic classification of these emerging forms of the book by the International Federation of Library Associations and Institutions has sought to categorise books, artworks and media for retrieval and access. The Functional Requirements for Bibliographic Records (FRBR), originally developed in the 1990s, has an underpinning conceptual framework based on a relational database. FRBR is a conceptual entity relationship model that, at its top level, identifies a distinct intellectual and artistic creation: a work, expression, manifestation and item (IFLA, 1998). These relate to the artist’s work (Duncan Pile’s fictional work Nature Mage), its expression (the draft of the work and the redrafts of book three Nature’s Peril), its manifestation (its ebook form via Amazon) and a specific item (the specific ebook downloaded on someone’s device). The relationships describe the link between the entities: the draft is the realisation of the story, the e-book is the embodiment of the story, the item is one particular example such as the e-book download or the on-demand printed paperback (Bennett et al., 2003; Eden, 2006; Hickey et al., 2002). Significantly the FRBR distinguishes between the responsibility or custodianship of the different entities identifying who – person, family or corporate body – is the creator of the work, the realiser of the draft, the producer of the e-book form and the owner of the download. FRBR also recognises the derivative relationship between the work and different editions, versions, translations, adaptations and genre changes. For our purposes that is where its usefulness stops as the relationship of adaptation gives rise to another work or expression. Should Nature’s Peril give rise to an iBook drawing on the fan contributions on the reader community website, this would become a new work. Similarly the relationship of transformation gives rise to a dramatisation, novelisation, versification or screenplay – or iBook. While the relationship is stated, the new work is autonomous. For the theorists of adaptation the relationship between the works is more important than their status as separate expressions or works: Linda Hutcheon says adaptations are differentiated from ‘autonomous works’ because they are ‘deliberate, announced and extended revisitations of prior works’ (2006).

Linda Hutcheon suggests that different media – and ‘genres’ she adds – ‘represent various ways of engaging audience’. Some ‘are used to tell stories (for example, novels, short stories); others show them (for instance, all performance media); and still others allow us to interact physically and kinaesthetically with them (as in video games or theme park rides)’ (2006: xiv). Hutcheon’s model of three modes of engagement is to a large extent dependent on conceptualising different modes – and media – as clearly distinct and separate from each other, which nowadays is increasingly difficult to do. In digital media, all three modes can be found simultaneously or in alternation. For example, an artefact such as an iBook is capable of being programmed to tell (text), show (videos) and invite interactivity (puzzle game). However, the iBook adaptation of say, Nature’s Peril, is a ‘new media form’. It is not the first incarnation of the story on digital media (the e-book already exists), and it may still be read (or used) on the same device as the e-book (on a tablet), but it certainly offers a different multimedia and multimodal experience to the reader. The Nature Mage project carries the story from a medium that just tells (printed book, e-book), to one where it mixes all three modes: tell, show and interact (as reader chooses to engage with some parts). Besides these three modes, there is also a fourth one, which could be classed as the participatory mode: the project invites readers and artists to participate in the expansion of the story, thus creating an experience that takes it beyond reading a book.

Applying Murray’s digital media and Jenkin’s transmedia storytelling to the book

The rapid innovation of new media technologies presents challenges to defining the relationship between the book and its derivative or transformative works. Technology providers distinguish between a device such as an iPad or a television or personal computer and a platform, which is typically used on the media industries to define a group or set of devices that are seen as of the same type, or very similar. Television is considered a platform, as is a game console, a mobile (regardless of type of mobile or specific device) and print is yet another platform. However, modern devices fulfil overlapping functions and present access to the same content that formerly was reserved for a specific medium and associated device or technology. Frequently the only barrier to ‘universal’ access to digital content is the industrial strategy of placing content in silos and seeking exclusive access to content by specific devices: you would read a story in a printed book, watch a film at the cinema, later also on a television; you would listen to a radio station on a radio. As Henry Jenkins (2008) explains, nowadays it is hard to find a mobile phone that only makes calls. It is still called a ‘phone’ but for some users it is much more a portable music store, a website device or a games console, than a phone. Now you can read, listen, watch and even play the same text, on one platform, using one device. For Jenkins, the smartphone is one of the most suitable symbols of media convergence in our societies. In a similar vein, Janet H. Murray (2011: 8) points out that ‘the new media’ encompass a range of artefacts including the Internet, video games, documents, photographs, social sites. She argues that ‘new’ media are not necessarily the newest forms of media around, but what they have in common is their use of digital technologies, and she offers instead the term digital media. As a designer and creative practitioner herself, she argues that it is more productive to view digital artefacts as parts of a single ‘digital medium, the medium that is created by exploiting the representational power of the computer’. She adds an important point about convergence: formats ‘that we once thought of as fixed and separate, like spoken and written messages, books and games, movies and file cabinets, television and telephones are being deconstructed into their component parts and reconfigured for interactivity’ (2011: 9). As deconstruction, remixing and integration take place, we witness the appearance of more and more emerging combinations. These can perhaps be described as hybrid converging formats (Franco, 2013).

Franco’s use of the adapted text is intended to extend the story world, not simply reproduce it in another format. The Nature Mage project adds content that expands the book story (or world), across different media, including the website and social media where fans and artists spread and discuss derivative works based on the books. It is then arguably a cross-media adaptation. It creates a multimedia and multimodal work that, contrary to the adapted texts, can only be accessed through the use of digital media. Arguably the prerequisites for being a cross-media adaptation are met because several media are present: the adapted texts in two media (printed book and Kindle e-book accessible on an e-reader); the fan–artist website and linked social media on the Internet, accessed via any web-enabled device and an eventual iBook and/or printed version of the book enhanced with AR to allow readers, through their smartphones, access to the multimedia content also present on the website and in the digital book.

Taking this further, does this make Nature Mage a transmedia story? The term transmedia storytelling was coined to refer to specific types of artefacts, stories or universes that are spread indeed across media, but in particular ways, typically where parts of the story are written for different media. Transmedia is multimodal, multisited and encourages engagement with the story world in a range of different locations, often both physical and virtual. In the Nature Mage project, the main narrative would still be in the original books, but both the fan website, the iBook, and the social media such as YouTube that are used to spread content, would contain expansions to the story and universe. It makes the story ‘come to life’ visually, by creating illustrations of characters, creating backstories and branching narrative to fill gaps and explore specific characters or plots in more detail, adding paratexts such as author interviews, adding extra content based on original writing material that never made it to the final edited version of the book and other forms of content that expand the story. However, the definition of transmedia is a difficult one to pin down.

The market place employs terms such as transmedia, 360 media, cross-media and other related terms often without these ever being defined. The terms transmedia and cross-media are commonly used by producers, marketing professionals and the trade press within industrial contexts to refer to using more than one media of communication such as when an adverting campaign uses all forms of television, radio, newspapers and the Internet. This raises questions about the application of these terms within industry: does a transmedia project need to cross from one medium to another? And what constitutes a medium in this era of convergence? Is it enough to jump from one platform to another? Is there any point in considering devices? In fact, what is the difference between the uses of these terms? Similarly the term transmedia has been applied in such a varied manner, with its meaning so stretched, that it risks – as in Murray’s criticism of the term ‘new media’ – becoming too vague and of little value. On a website of a company that specialises in transmedia training, the author rolls other concepts into the term arguing it includes ‘immersive storytelling, deep media, persistent narratives and multi-platform storytelling’. The company prefers ‘to use the word “transmedia” because it is a neat shorthand to capture all of those concepts’. Similarly, in a report from the last day’s ‘think tank’ at the Power to the Pixel industry event in 2010, the authors state that they use the term ‘“cross-media” rather than “transmedia” to describe projects which take stories to audiences across a range of media platforms. The differences in the two terms are a minor semantic issue and the two terms are used interchangeably by advocates without any confusion’ (2011: 4). Although there have been some wonderful creative media artefacts coming out of Power to the Pixel, this oversimplification and the merging of the two concepts means that transmedia again seems to be anything that crosses media (or platforms), and nothing more than cross-media (Gubbins, 2010).

In academic terms however Marsha Kinder’s first use of the term transmedia in 1990 (Kinder, 1991) remains a classic definition. Transmedia intertextuality links television, movies and toys by brand. Kinder cites Teenage Mutant Ninja Turtles as an example of a ‘commercial supersystem of transmedia intertextuality’. According to Kinder, these systems engage with their audiences in ‘combined modes of spectatorship’, in order to ‘construct consumerist subjects who can more readily assimilate and accommodate whatever objects they encounter, including traditional modes of image production like cinema and new technological developments like interactive multimedia’ (1991: 4).

Today Jenkins’ definition is one of the most widely used both in academia and in industrial circles:

Transmedia storytelling represents a process where integral elements of a fiction get dispersed systematically across multiple delivery channels for the purpose of creating a unified and coordinated entertainment experience. Ideally, each medium makes it own unique contribution to the unfolding of the story. (Jenkins, 2011)

Jenkins cites the example of The Matrix franchise, which conveys ‘key bits of information’ through various artefacts: films, animated shorts, comic books and video games. He explains that ‘there is no one single source or ur-text where one can turn to gain all of the information needed to comprehend the Matrix universe’.

(Jenkins, 2011: 101). In his blog Transmedia 202, Jenkins revisits the term adding a list of reflections and considerations to help explain his meaning for it but also suggests that there is no transmedia ‘formula’ and that its very definition is shifting and evolving:

[W]e are still in a period of experimentation and innovation. New models are emerging through production practices and critical debates, and we need to be open to a broad array of variations of what transmedia means in relation to different projects. The more we expand the definition, the richer the range of options available to us can be. It doesn’t mean we expand transmedia to the point that anything and everything counts, but it means we need a definition sophisticated enough to deal with a range of very different examples. (Jenkins, Transmedia 202, 1 August 2011)

Arguably the most open definition is offered by the ‘T is for Transmedia’ project at the University of Annenburg, which states transmedia as ‘any combination of relationships that might exist between various texts (analogue or digital) that constitute a contemporary media entertainment experience’ (Reilly, 2013: 1). Focusing on the virtues of transmedia experiences for educational purposes, the project offers a constructivist definition: ‘Through immersive, interconnected and dynamic narratives, transmedia engages multiple literacies, including textual, media and visual literacies as well as multiple intelligences’ (Reilly, 2013: 4).

Transmedia publishing is often focused on children’s brands that cross fields of play from books, films and toys to digital media products such as video games, e- and ibooks, TV series on demand and websites. Reilly argues that transmedia can promote new approaches to literacy and reading as through cross-media play children learn to read multimedia texts more deeply. There is evidence too that the older age groups do this: Samantha Pearce’s work on reader–audience’s online comments on the film adaptation of Breaking Dawn part 2 shows how a depth of reading and intimate knowledge of the adapted text are reinforced through the film and subsequent online discussions. Participants displaying knowledge of the detail of the adapted text and showing evidence of multiple rereading are amongst the most active members of the forums. Such experiences lie alongside older forms of shared communal engagement such as the cinema exhibition of cult movies (e.g. midnight showings) and heightened audience reaction (from reports of members of the audience swooning at Valentino to later reports of audiences crying, screaming, swearing and not breathing in Breaking Dawn part 2) (Pearce, 2014). Reading a book or watching a film in a cinema is a deeply immersive experience. The question for many designers and critics is whether new transmedia forms add to or detract from this depth of engagement.¹

Transmedia storytelling in AR books

In an article for the publishing industry produced by Inglobe Technologies in 2011, the advantages of AR to book publishing included reducing ‘cognitive overload on the reader’, increasing the ‘quantity and types of information available in the same act of enjoyment’, improving the ‘performance of readers in learning and retaining information’, increasing ‘the level of “presence” and engagement experienced by readers’ and helping ‘to create immersive reading experiences’ (Inglobe, 2011: np). The potential of AR gained traction in the consumer market when Nintendo brought out their hand-held console 3DS. With the camera on their handheld console, game players ‘read’ the printed AR card’s computer-readable image linking the device and its on-screen 3D game to the location of the card, a technique that has also been used by book publishers (FIW, 2011). The publishing sectors who have been the early adopters of the technology are reference, education, manuals and travel guides and the revenue models reflect those in use with e-books (one-off fee or subscription), royalty on digital content received or using the game app model that offers a free or low-cost AR book within which are embedded transactions for additional content (Inglobe, 2011; Perry 2011, 2012).

Currently AR books (in contradistinction to Accelerated Reading books) bolt on the affordances of mobile technology. In the children’s or education market, this may be adding a ‘pop-up’ to the book read off the page by a webcam or mobile device. In the book iSolar System, for example, the book text is informative and a downloadable app is used to recognise the image and realise the planets in a 3D model of the solar system on an android or apple device (Carlton kids, 2013, see also Pile, 2012). As Tony Mullen said ‘The idea of AR is to mingle what is not really there with what is there as seamlessly as possible, and to present users with an enhanced, or augmented, display of the world around them’. This is done by ‘integrating virtual content and data with live, real-time media’ (Mullen, 2011:1). With the growth in mobile computing technology, AR development has incorporated location-based activities. Bruce Stirling, for example, created ‘Dead Drops’ on the Layar AR app with which you can use your iphone to locate ‘thumb-sized flash drives publicly hidden in cities around the world’ (Sterling, 2011). Recent changes in technology have been facilitating the development of AR. For example, the new generation of smartphones offers mobile access to digital 3D modelling. AR is rapidly becoming more widespread, accessible and available to a wider audience.

The capacity of AR to ‘add to’ the book is well known, but it can also disguise and hide narratives. In early AR books, the experience was centred on the novelty of the technology and the elements of surprise and wonder at the ‘magical’ illusion of viewing in 3D from multiple angles. It has therefore been used in fiction to signal the unexpected and as a mechanism to narrate mystery, confusion, altered/distorted or magical reality, and in non-fiction to depict floating objects, additional layers of information, 3D diagrams and make normally static into dynamic objects. The aesthetic of layered images can be playful or mysterious as they move around when viewed from different angles (Austen, 2010 and examples Aurasma, 2012; DiTerlizzi, 2010; FIW, 2011; Guinness Book World Records, 2013; Kunan, 2010; Loyola, 2012; Salariya, 2010).

Many advertising companies have seen the potential of AR: In September 2012, Nestle UK embedded GPS trackers in chocolate bars, and compared it to the Golden Ticket from Willy Wonka and the Chocolate Factory (Kiefaber, 2012). Others addressed their own campaign messages: Weetabix allowed the consumer to download the free Weetakid app where players gave energy to a virtual hero by feeding him Weetabix to fuel his run round the galaxy (beccacaddy, 14 October 2011); Macdonalds used an AR app in Australia to combat their image for unhealthy fast food (Lum, 2013); and Pringles developed an online football game in China using the (real) Pringles box as an integral part of the game (Pringles, 2010). Yet AR is often described as a ‘disruptive’ technology. According to Clayton Christensen, technological development becomes disruptive when it produces such a significant and swift advance in design that it unexpectedly changes the market. It does this first by creating an innovation that attracts more customers and then, as a result of the rise in size and overall value of the market, it enables a reduction in prices affecting existing businesses using the older technologies and trade practices cannot complete with. They are then driven out of the market (Christensen, 1997). This positions it as an innovation that will create new markets and a new value network, potentially disrupting existing ones and displacing earlier technology (see AugmentedRealityOrg, 2013).

Experiments have not yet provided evidence of this potential. Both advertising AR and alternative/additional content AR have been used in the periodical and book trade but have not caused market disruption (Gorkana Group, 2013). The Philadelphia Inquirer, for example, uses AR to duplicate the online material in the printed newspaper (Hodgkins, 2012). While the Tokyo Shimbun newspaper created an AR app that simplifies news for children (Steadman, 2013). Other newspapers and magazines have developed AR enhancements, for example, Hearst magazines teamed up with Zapper to create an AR front cover for InStyle, linking it to video coverage of catwalks (beccacaddy, 2011; HouseofHollandLtd, 2011). Book artists have also experimented with AR in various forms: In 2008, the artist Camille Scherrer developed her book Souvenirs du Monde des Montagnes, using a webcam to recognise the text and display overlays, and Yuri Suzuki created The Barcode Book (2010), which used barcodes to start audio playback (Scherrer et al., 2009). Siglio Press’s Between Page And Screen (Borsuk and Bouse, 2010) is a digital pop-up book with (only) quick response codes for the reader to hold up to a webcam. Fascinating and skilful as these are, they remain novelties in the book market.

It was within this context that Sherwood Rise was developed. At the time of the project, January 2012 onwards, AR had already been developing for many years and was widely used in industry, yet had been mainly limited to personal computer or desktop applications such as those mentioned above, generally using webcams. The aim of the project was not to deliver a commercial, market-driven or entertainment product but to push the boundaries of the storylines and explore the reader’s engagement with the characters. The project was about media literacy and how we read the news, but it also tested user–reader’s engagement with the story and characters, measuring how deeply they burrowed for the information. In Miller’s work, AR is the ‘x-ray specs’ revealing what lies beneath or under the surface of a news story. AR is used, quite literally, to decode the media. It becomes a gateway or interface between the traditional book and other digital content and offers the possibility of linking the logocentricism of the book to audio–visual and time-based media. Therefore, Sherwood Rise combines traditional multilinear storytelling demonstrated through its newspaper layout with polyphonic narratives accessed through AR. Aesthetic devices such as 3D imagery, comic strip storytelling, animations, characters’ blogs, radio button voting systems and sound (voice overs, ‘live’ sound and music) offered alternative views and additional information for the user–reader to access (Miller, 2012–13).

AR books require a different kind of screenwriting. In a notable reversal of the writer–developer order, Bruce Sterling wrote his fiction after his Layar AR app was created. Sherwood Rise, however, originated as a five-page story, with a further synopsis of backstories of the characters written by David Moorhead. Visualising the characters in drawings based on his descriptions, Miller contributed to the characterisation and separated out parts of the script, dividing the story into the narrative layers. Miller and Moorhead sought to give the story game-like properties drawing on the genre of puzzle and ‘If’ books. The story unfolds over 3 to 4 days, changing the AR book into a time-based narrative (Figure 3). The reader in Sherwood Rise becomes implicated in the story, receiving mail messages from the groups of characters.

Figure 3.

Structure of storylines over the 4 days.

Admitting he had no ‘experience in future media or augmented reality’, Moorhead’s approach was initially traditional:

A strong and distinctive beginning, middle and end would make the project adjustable. Writing for an unknown format was difficult and changes would have to be made, but the platform of ‘a story’ was important. (Miller, 2013)

Creating a tension between the static story (the printed book) and the dynamic story (the AR story) was important to keep the reader engaged across platforms. Moorhead’s ‘strong basic structure and key events’ made the story more easily adapted, but he wanted the reader to experience the mix of the viewpoints of the characters in the story and encourage engagement with the AR. Influenced by Ian Bogost’s Newsgames: Journalism and Play (2010), Miller chose the format of a series of newspaper editions, where the AR revealed the alternative stories behind the news, allowing the reader to gain further insight into the ‘truth’ behind the main narrative. The story was structured so that a different newspaper arrived each day for 3 days. The edition of the newspaper the reader received depended on how much they had actively supported or helped Robin, the protagonist.

The AR does three main things: it disrupts the usual linear flow of the printed book, introducing a game-like aspect, so that the AR becomes an integral part of the total reading experience; it acts as a truth decoder decrypting the packaged news or revealing the ‘invisible’ truth (or information); and it enables other voices to speak within the story giving counter narratives. The readers’ actions/choices in the Robin AR segments are recorded in a database that calculates a score value (Figure 4). For each day, there are three versions of the newspaper, and the version the reader gets depends on their current score in the database. Therefore, there are nine editions of the newspaper and 18 Junaio AR segments (Miller, 2012–13).

Miller and Moorhead observed the adaptation process of moving the five-page story to the structure of the AR book was complicated, and writing to the AR architecture might have been a preferable alternative. Much of the characterisation was lost but the backstories were suitable for development within the AR framework. The story in its newspaper and AR format was not immersive and those testers who were also gamers reported that the strong narrative and limited gaming options was dissatisfying. Others – who were readers, not gamers – preferred to work out the narrative from the newspaper. It was not a game, it was a story where readers interact with the technology, yet the genre expectations were so strong that this affected the readers/players levels of satisfaction. (Miller, 2012–13)

The project intended to investigate the user experience with AR to examine narrative problems and explore new storytelling aesthetics. What proved to be most successful was the aesthetics of the contested/augmented content. AR was an effective way of representing the power struggles in the contemporary Robin Hood story. The interface represented newspaper (left and right wing political views) and hacker content, utilising the disruptive power of the AR technology, emails and social media to depict activist politics, echoing much studied contemporary modes of political communication. The particular aesthetics of AR – telling graphical stories through 2D images on 3D planes, through mobile gestural interfaces (e.g. swipe, flip) – retained print’s user-led access. Miller’s graphics drew on the graphic novel genre and his slide viewer stories were similar to cartoon strips. Flip book animations added to the visual repertoire and provided political humour reminiscent of Punch. The newspaper aesthetic was derived from print traditions.

Figure 4.

Plan of choice modes.

Conclusion

‘The limitations of the format – the rules our interaction with the artefact relentlessly have to adhere to – anchor it to society’ (Westin, 2013: 5). For Weedon, the book is a dynamic system to commodify ideas and cultural expressions (2012). The system verifies the provenance and ownership of a work, ensures ownership rights, secures access and packages the work for readers. It also allows for the transference of ownership, stores the work, its information and ideas and more broadly caters for a social need. Sherwood Rise pushed the definition of the book raising questions central to the book as a system of publishing. The authorship of ideas was a collaborative process – it in fact drew on the skills not only of Miller as an artist/programmer and Moorhead as scriptwriter, but also of Tracey Bearton as layout artist, journalist subs and music technology students who created the audio tracks. While collaborative production in the arts is not unusual, the project-focused attention back on the central role of the story as the vehicle for the immersive experience and on the rights of all the creators contributing to that process. Books are enduring packages of ideas and stories, and so it is important their provenance can be verified and the content can be transmitted across time and space. Ownership is a key part of that process in the West and so it must be recorded and acknowledged. Similarly readers enjoy ownership and while you can ‘own’ a copy of a book how do you own a copy of a transmedia AR story? And what processes are needed for publishers to verify authorship and provenance and to exercise quality checks before publishing/exhibiting an AR transmedia work? While authorial intention is still very evident in the Sherwood Rise narrative, Miller’s work offers an architecture for writers and readers to explore the elements of the narrative rather than offering a particular reading. Indeed some of the reader–testers read all the alternatives, rereading (and replaying) for different storylines. As Boris Debackere observes in AR, ‘The virtual tool becomes a (re-)active actor in the creation of dynamic processes’; AR demands ‘different approaches to those from the era of mimicking media’ (2010).

E-book and iBook systems delivered though e-readers and tablets are less of a challenge to the book system. Publishers and software developers work together on combining the media in a package that is authored and delivered through distributors and sellers (iBook author to iPad via iTunes or authored via Amazon epublishing and delivered via the online Kindle shop). Social media is then viewed as a marketing opportunity. However, Claudio Pires Franco’s Nature Mage project demonstrates how inadequate this approach can be. Social media provides a new relationship between the author and the reader. Pearce’s work shows that authors, directors and scriptwriters retain their celebrity aura even when safely behind participatory forums. Yet, like the games industry’s crowdsourcing of beta testers (games that are launched prior to being finished, so that players can test them, find bugs and suggest improvements), authors can get in touch with their readers’ opinions and ideas, and interact with their creative visualisations of the story world. They can do this at a distance and yet with an immediacy and level of feedback other media cannot provide. As Sherry Turkle has pointed out, mediated relationships can point to a state of instability, a distancing and the desire for safety in our social contact (Turkle, 2011) The online relationship between the author and the reader crosses perceived social barriers. Conversations, author opinions, interactions with and between readers become part of the system. Bob Stein, (2013) with his social book and social reading experiments, and history of reading scholars such as Bob Owens, suggest that readers texts from marginalia to paratexts can become part of the work, assuming in some cases an equal value to the original words (Towheed and Owens, 2011). The conversation with readers also has the potential to influence and eventually alter current or future works in a much more public way that is also recorded – at least temporarily – on the online spaces where such conversations occur. Furthermore, some digital tools such as LeanPub take conversations with readers as feedback and use them to revisit and improve book texts. In this kind of platform, books are seen in a similar light to beta games. Nowadays these processes are open to potentially a large number of people and are often recorded and available online, allowing us to track the evolution of a work. These new media forms define the outer borders of the book system within which content is formed and moulded, and around which society is shaped. As our social interactions are changing with new communication technologies, so is the book, and we should view this as a positive change.

Footnotes

Acknowledgements

These projects are part of the UNESCO Chair-lead network supported by the University of Bedfordshire ‘Crossing Media Boundaries: Adaptations and New Media Forms of the Book’, which links researchers who are investigating how stories and ideas which have been transmitted through print are being adapted for communication through electronic media. Researchers are combining creative and analytical insights to demonstrate how the book is adapting to competition from alternative media platforms and seek to offer a vision of the future of the book grounded in differing cultural and technological contexts.

Note

References

AugmentedRealityOrg (2013) Augmented reality startup launchpad and competition at AWE 2013. Available at: http://www.youtube.com/watch?v=_hjTsr6Cqoo (accessed 23 October 2013).

Aurasma (2012) Demo: create your own 3D Dinosaur pop-up book with Aurasma. Available at: http://youtu.be/TS0c411XcX4 (accessed 23 October 2013).

Austen

(17 November 2010) ‘Storytelling 2.0: open your books to augmented reality’ culture lab: where books, arts and science collide. New Scientist. Available at: http://www.newscientist.com/blogs/culturelab/2010/11/storytelling-20-open-your-books-to-augmented-reality.html (accessed 23 October 2013).

beccacaddy (14 October 2011) Weetabix adds AR game weetakid to its boxes. Available at: http://www.shinyshiny.tv/2011/10/weetabix_adds_ar_game_weetakid_to_its_boxes.html. (accessed 23 October 2013).

beccacaddy (22 November 2011) Is augmented reality the future of publishing? Henry Holland launches app with Aurasma and InStyle. Available at: http://www.shinyshiny.tv/2011/11/is_augmented_reality_the_future_of_publishing.html (accessed 23 October 2013).

Bennett

Lavoie

O’Neill

(2003) The concept of a work in worldcat: an application of FRBR. Library Collections, Acquisitions, and Technical Services 27(1): 45–59.

Bogost

Ferrari

Schweizer

(2010) Newsgames: Journalism at Play. Cambridge, MA: London, MIT Press.

Bolter

Grusin

(1999). Remediation: Understanding New Media. Cambridge, MA: London, MIT Press.

Borsuk

Bouse

(2010) Between Page and Screen. Los Angeles, CA: Siglio.

10.

Carlton

Kids

(2013) iSolar System: An Augmented Reality Book. London, UK: Carlon Kids.

11.

Carrière

J-C

Eco

(2011) This is Not the End of the Book: A Conversation Curated by Jean-Philippe de Tonnac. London, UK: Harvill Secker.

12.

Christensen

(1997) The Innovator's Dilemma: When New Technologies Cause Great Firms to Fail. Boston, MA: Harvard Business School Press.

13.

Dawkins

Mckean

(2011) The magic of reality. Available at: http://www.magicofrealityapp.com/ (accessed 23 October 2013).

14.

Debackere

(2010) Augmented dreams: keynote. In: Symposium On Mixed and Augmented Reality - Arts, Media, and Humanities (ISMAR-AMH), 2010 IEEE International Seoul.

15.

DiTerlizzi

(2010) The search for WondLa. Available at: http://www.youtube.com/watch?v=Ih97liHkc14&feature=related. (accessed 23 October 2013).

16.

Eden

(2006) Application studies of FRBR. Library Technology Reports 42(6): 24–31.

17.

Eliot

(2011) The Waste Land for iPad. London: Faber and Touch Press.

18.

FakePress (2011) CoS – Consciousness of streams. transmediale11, Available at: http://cos.artisopensource.net/DNWeb/. (accessed 23 October 2013).

19.

FIW (2011) The future is wild living book. Available at: http://www.thefutureiswild.com/ and http://www.youtube.com/watch?v=Tx3NtPzd51M&feature=related. (accessed 23 October 2013).

20.

Franco

(2013) Books 3.0: cross-media adaptation and audience involvement. Available at: http://transmediabooks.wordpress.com (accessed 29 October 2013).

21.

Gorkana Group (7 May 2013) Is augmented reality the future for newspapers?’ Available at: http://www.gorkana.com/notes-from-basecamp/industry/is-augmented-reality-the-future-of-newspapers/ (accessed 23 October 2013).

22.

Gubbins, Michael with additional writing by Liz Rosenthal (2010) Power to the pixel think tank report. Available at: http://thepixelreport.org/wp-content/uploads/2011/03/ThinkTank2011hiqual.pdf (accessed 23 October 2013).

23.

Guinness Book World Records (2013) Guinness book world records 2013 book app. Available at: http://youtu.be/74RgjOfAc5k (accessed 23 October 2013).

24.

Hickey

O′Neill

Toves

(2002) Experiments with the IFLA functional requirements for bibliographic records (FRBR). D-Lib Magazine 8(9).

25.

Hodgkins

(4 April 2012) Augmented reality iPad app enhances Philadelphia Inquirer. Available at: http://www.tuaw.com/2012/04/04/augmented-reality-ipad-app-enhances-philadelphia-inquirer/ (accessed 23 October 2013).

26.

HouseofHollandLtd (15 November 2011) Our moving instyle cover. Available at: http://www.youtube.com/watch?v=P1S5Z14OoF4&feature=youtu.be. (accessed 23 October 2013).

27.

Hügli

Kovacovsky

(2010) Jekyll and Hyde augmented reality book. Available at: http://augmentedrealityoverview.blogspot.co.uk/2011/10/jekyll-and-hyde-augmented-reality-book.html (accessed 23 October 2013).

28.

Hutcheon

(2006) A Theory of Adaptation. London, UK: Routledge.

29.

IFLA (1998) Functional requirements for bibliographic records: final report. München, K.G. Saur.

30.

Inglobe Technologies

SRL

(2011) Augmented reality and the future of printing and publishing: opportunities and perspectives. Available at: http://www.inglobetechnologies.com/docs/whitepapers/AR_printing_whitepaper_en.pdf. (accessed 23 October 2013).

31.

Jenkins

(2008) Convergence Culture: Where Old and New Media Collide. New York; London: New York University Press.

32.

Jenkins

(2011) Transmedia 202. Available at: http://henryjenkins.org/2011/08/defining_transmedia_further_re.html (accessed 23 October 2013).

33.

Kiefaber

(24 September 2012) Nestlé Puts GPS trackers in candy bars for golden ticket style contest Willy Wonka would be proud. Available at: http://www.adweek.com/adfreak/nestl-puts-gps-trackers-candy-bars-golden-ticket-style-contest-143962 (accessed 23 October 2013).

34.

Kinder

(1991) Playing With Power in Movies, Television, and Video Games: From Muppet Babies to Teenage Mutant Ninja Turtles. Berkeley, Oxford: University of California Press.

35.

Kunan (2010) Augmented reality storybook. Available at: http://vimeo.com/10903553 (accessed 23 October 2013).

36.

Leitch

(2009) Film Adaptation and Its Discontents: From Gone with the Wind to The Passion of the Christ: From “Gone with the Wind” to “The Passion of the Christ”. Baltimore, MD: Johns Hopkins University Press.

37.

Les éditions volumiques (2013) Catalogue. Available at: http://volumique.com/v2/en/ (accessed 23 October 2013).

38.

Loyola

(2 April 2012) Marvel AR for iPhone and iPad. Available at: http://www.macworld.com/article/1166151/marvel_ar_for_iphone_and_ipad.html (accessed 23 October 2013).

39.

Lum

(17 January 2013) McDonalds AR app shows what’s inside your meal. Available at: http://www.creativeguerrillamarketing.com/augmented-reality/mcdonalds-augmented-reality-app-shows-whats-inside-your-meal/ (accessed 23 October 2013).

40.

Miller

(2012–13) Augmented reality and the future of the book. Available at: http://augmentedwonder.blogspot.co.uk (accessed 23 October 2013).

41.

Miller

(2013) Augmented reality and the future of the book, unpublished internal progress report. University of Bedfordshire.

42.

Morris

(2012) Frankenstein, for iPad and iPhone. Available at: https://itunes.apple.com/gb/app/frankenstein-for-ipad-iphone/id516047066?mt=8 (accessed 23 October 2013).

43.

Mullen

(2011) Prototyping Augmented Reality. Indianapolis, IN: Wiley.

44.

Murray

(2011) Inventing the Medium: Principles of Interaction Design as a Cultural Practice. Cambridge, MA: London, MIT Press.

45.

Pavlik

Bridges

(2013) The emergence of augmented reality (AR) as a storytelling medium in journalism. Journalism & Communication Monographs 15(1): 59.

46.

Pearce

(2014) Readers and Audiences on Online Forums: The Case of Twilight: Breaking Dawn Part 2. Luton: University of Bedfordshire.

47.

Perey

(2011) Standards for expanding AR with print. Available at: http://www.perey.com/ARStandards/PereyStandards_for_expanding_AR_with_Print.pdf (accessed 23 October 2013).

48.

Perey

(2012) Print and publishing and the future of augmented reality. Available at: http://www.perey.com/White_Paper_for_ARCHP&P_January_12.pdf (accessed 23 October 2013).

49.

Pile

(2012) Nature Mage. Amazon UK ebook.

50.

Pringles (June 2010) Pringles augmented reality football game using packaging. Available at: http://youtu.be/deUDtw_GHew (accessed 23 October 2013).

51.

Portal Entertainment (2010) The Craftsman. Available at: http://www.portalentertainment.co.uk (accessed 23 October 2013).

52.

Reilly

(2013) T is for transmedia: learning through transmedia play. Available at: http://www.slideshare.net/ebreilly1/transmedia-play-final-144 (accessed 23 October 2013).

53.

Salariya

(2010) Scribblers augmented reality book ‘What Lola Wants… Lola Gets’ London Book Fair. Available at: http://www.youtube.com/watch?v=pJcg-MCZnCQ (accessed 23 October 2013).

54.

Scherrer

Pilet

Lepetit

Fua

(2009) Souvenirs du monde des montagnes. Leonardo 42(4): 350–355.

55.

Steadman

(8 February 2013) Japanese newspaper’s augmented-reality app simplifies news for kids. Available at: http://www.wired.co.uk/news/archive/2013-02/08/tokyo-newspaper-augmented-reality-kids (accessed 23 October 2013).

56.

Stein

(2013) The future of the book. Available at: http://futureofthebook.org/blog/2013/03/ (accessed 23 October 2013).

57.

Sterling

(18 August 2011) Science fiction writer becomes augmented reality developer. Bruce Sterling has created his own augmented reality fiction - location based book. New Scientist. Available at: http://www.wired.com/beyond_the_beyond/2011/08/augmented-reality-science-fiction-writer-becomes-augmented-reality-developer/ (accessed 23 October 2013).

58.

Suzuki

(2010) Barcode book. Available at: http://yurisuzuki.com/works/barcode-book/ (accessed 23 October 2013).

59.

Towheed

Owens

(2011) The History of Reading: International Perspectives c.1500-1990. Basingstoke: Palgrave Macmillan.

60.

Turkle

(2011) Alone Together: Why We Expect More from Technology and Less from Each Other. New York, NY: Basic Books.

61.

Weedon

(2012) The book as a dynamic system for the commodification of ideas and cultural expressions. Primerjalna Književnost 35: 177–186.

62.

Westin

(2013) Loss of culture: new media forms and the translation from analogue to digital books. Convergence: The International Journal of Research into New Media Technologies 19(2): 129–140.

63.

Whitehead

(19 October 2009) Augmented reality turns breakfast cereal into a 3D games console. Available at: http://www.brandrepublic.com/news/946623/ (accessed 23 October 2013).