Abstract
This article is set within a technological paradigm shift that denotes a transition from hardware to software and that, by means of applications and operating systems, plays a central role in the sociocultural sphere, intervening in the creation, classification and distribution of cultural objects. The aim of this research is to delve into how the graphical user interface (GUI) integrates into contemporary audiovisual discourse, focusing on its potential as a spatial metaphor. To that end, a theoretical framework is built on the concept of space as a tool and its instrumentalization in the GUI throughout the years, with a view to design a taxonomy in relation to the different ways in which the GUI integrates into digital image composition. The article concludes that the ability to operate in the virtual and physical space, moving between the logic of the tool and its potential as spatial metaphor, provides the GUI with the necessary specificity to be considered a key cultural element for analysing new media’s visual identity.
Introduction
Mechanisms underlying the technological paradigm shift are, to a large extent, linked to automation processes. The enhancement of artificial intelligence systems associated with the qualitative leap of machine learning, together with the ubiquity and interconnection of the ever more efficient mobile devices, cameras, sensors, screens and solutions in the context of human–computer interaction (HCI), would be strengthening the communicating vessels between audiovisual creation and software, between creativity and algorithm. Consequently, applications intervene in almost all artwork construction, distribution and consumption processes (Weibel, 2005), and thus the digital nature and technology linked to such processes can be expected to have an impact on their formal configuration (Manovich, 2001).
Cinema, video games and the graphical user interface (GUI) share their digital nature as well as a common space within mass culture. Thus, they can be considered as algorithmic cultural objects inasmuch as, according to the conceptual scenario raised by Galloway (2006), cinema can be understood as a system-received message, video games as a result of an input/output dialogue between the user and the system and the GUI as a key element for the coding inherent to the previous processes to operate properly, placed in the intermediate space between software and user. Although the author refers to video games, this definition applies to the whole field of audiovisual creation.
Although the significance of the relationship between computing and cultural objects would resonate at a deeper level – habits and lifestyles – the impact of software on the cultural layer dates back to before the exponential growth of automation processes, to the very origins of computational thinking, when hardware was still used as a metaphor by the thinkers of the time. Shannon, Weiner, Von Nenmann, Turing have radically altered our image of our mental processes. In the place of the ever-changing cloud that we carried in our heads until the other day, the condensing and dispersal of which we attempted to understand by describing impalpable psychological states and shadowy landscapes of the soul – in the place of all this we now feel the rapid passage of signal on the intricate circuits that connect the relays, the diodes, the transistors with which our skulls are crammed. (Calvino, 2017: 8)
At present, when software has come to figure prominently in the development of social and labour relations, as well as to get integrated into almost all processes of creation and distribution of cultural objects, it seems appropriate to analyse the nature of a dialogue that has encouraged a relationship between the computer layer and the cultural layer (Manovich, 2001), has led to a changing relationship with the media (Bolter and Grusin, 1996) and has become part of the reconfiguration process of digitextuality (Everett and Caldwell, 2003).
Therefore, the present article considers new media to be those media capable of producing cultural objects whose nature and creation, distribution or consumption processes are influenced by the presence of software.
Finally, we are currently witnessing how the software flexibility and modularity for audiovisual creation, the increasing absorption of transmedia strategies by the audiences and the implementation of 5G networks could pave the way for the implementation of wireless systems in virtual reality (VR) solutions, which until now could not integrate data on user’s movements in the physical space. Even though these forms pivoted towards industrial applications in the mid-90s, driven by the Web’s influence, they have made a comeback once the technological ecosystem has proved able to respond not only to the user/viewer but also to offer the possibility of integrating creation processes from those same virtual spaces. 2 If we add to this the progressive implementation of the internet of things (IoT), the development of user-friendly tools as a key to democratizing the construction of machine learning systems (Matherson, 2019) and the tendency (through spatial computing) to gradually abandon the screen as the physical access to software (Pangilinan, Lukas and Mohan, 2019), it seems appropriate to focus on the HCI and, more broadly, on the GUI.
In short, the interest in how the technological logic is transferring to the cultural layer leads us to conduct the present study, focused on the evolution of space as an abstraction integrated into the GUI, with a view to observe how such metaphor influences New Media’s visual identity. This is what it means to say that an algorithm is a culture machine; it operates both within and beyond the reflexive barrier of effective computability, producing culture at a macro-social level at the same time as it produces cultural objects, processes, and experiences. (Finn, 2017: 34)
Methods
The physical space of a theatre in darkness became the theoretical, iconic and ideal reference for creators and researchers once the movie theatre’s spatial architecture got standardized, thus enabling a homogeneous perspective when addressing the relationship between viewers and audiovisual discourse, as well as the study of the mediation of technology, regarding both how fiction is constructed and how it is presented to, and interpreted by, the audience (Baudry, 1974: 40). Although the social, economic and, ultimately, cultural changes resulting from the ubiquity of screens and network software would have undermined for some time the standards that film-makers and critics once had in relation to the way in which we decode audiovisual discourses (Nickdel, 2015: 2), the study models would also be tending to interdisciplinary perspectives that do not only take into consideration changes in the audiovisual consumption’s hegemonic space predominant in the 20th century, but also those changes arising from the introduction of computing in our cultural life and the narratives that it may inspire (Finn, 2017: 15). In short, an analysis perspective in which ‘[…] cinema is considered holistically as technology, space, experience and form’ (Llinares and Arnold, 2015: 6).
It is expected that the implementation of the algorithm in the production processes of cultural objects will have an impact on their form, which is why it seems appropriate to address the relationships established among GUIs, applications and the cultural layer itself: The GUI implies the use of interactive visual solutions for parametrizing the algorithms that integrate the applications. The applications are involved in a scenario in which audiovisual image is digitally processed throughout all production stages, which entails the instrumentalization of the algorithms intervening in the creative process through software that runs on operating systems (OS). The cultural and social sphere in which applications play a role by means of the use of mobile devices, whose relevance has exceeded the principles of ubiquitous computing (Weiser, 1993) and which, in turn, serves as a source of inspiration for the creation of cultural objects.
This working paper aims to analyse how the GUI integrates into contemporary audiovisual discourse, taking into consideration its new status as a cultural object. In the same way as, according to Lefebvre (1991: 34), the analysis of the dialogue that we establish with the social space characterizes us as individuals, we will seek to characterize new media’s visual identity by observing the potential of the GUI as a spatial metaphor.
Thus, the creation of the digital cultural object and the nature and development of the GUI may not be operating as independent entities, but initiating a dialogic process through the use of creation applications and giving rise to a hybridization based on two fundamental principles, which constitute the main assumptions of this working paper: GUI’s design and audiovisual creation share a cultural layer defined by the ubiquitous algorithm and the screens for manipulating it, which favours the homogeneous design of the spatial metaphors incorporated into their discourses. Hybridization between the audiovisual formal discourse and the GUI can be categorized based on how the relationship between the logic of physical and digital space is established.
The present article therefore intends to analyse the impact of the GUI as the embodiment of the algorithm in contemporary audiovisual discourse, thus it seems useful to create a taxonomy of transfers between GUI and audiovisual discourse, which will allow us to demonstrate its consolidation as a visual and cultural element while facilitating its critical analysis.
To that end, we will build first a general theoretical framework on the instrumentalization of space as a creation tool, capable of sustaining the space on which the GUI operates.
Secondly, we will create a visual tool with the ability to visually link the development of the GUI and its impact on the formal configuration of the discourse, taking into account the dialogic processes in relation to the cultural layer and the computer layer.
3
It is a timeline designed through a web application
4
and developed taking into consideration the following criteria for selecting its elements. The timeline layer for applications and OS mainly addresses visual innovations introduced to the GUI, and not the relevance of the technology at issue. For instance, the first digital video effects system, DVM800/1, allowed to edit 3D images in a 3D space, but through external physical controls and without a GUI, so it is not included in the timeline. OS are included inasmuch as their libraries have had an impact on the GUI’s development of the applications that run on them. Inclusion criteria for applications and video games are based on the implementation of GUI relevant visual solutions, as well as on their number of users, inasmuch as the impact of their visual solutions is likely to trigger a greater transfer to discourse and have a stronger presence in the cultural layer. This is why, at a certain point, some software versions will be given higher relevance when capable of running on non-proprietary OS, such as Windows, Mac or Linux. Even though some video games that are considered pioneers, such as Tennis for Two (1958) or Spacewar! (1962), are not included since their visual design did not involve a GUI, our timeline does include computer-aided design (CAD) applications with similar features, such as Sketchpad (1963), because, in this case, the image itself served as a graphical interface. Other techniques that create more synergies with research in terms of the instrumentalization of the dialogic space between physical and virtual space, such as tracking, motion tracking, match move, modelling or creation in 3D virtual spaces, will also be taken into consideration.
The tool used for developing the timelines includes a 3D version, which offers improved functionality as a mood board, understood as a valid methodology for the design of a taxonomy, in line with what was suggested by Ferrara and Russo (2018). In this regard, a correct visual experience linked to the GUI’s development acquires greater significance, which is why, even though we have clearly specified the year corresponding to each milestone, their distribution over the months is only aimed at improving readability. Also to that same end, other key milestones in the configuration of the theoretical framework on which the GUI’s development as we know it today is built are not included. 5
Thirdly, and lastly, we will try to design a classification reflecting how the GUI integrates into the audiovisual discourse by referring to audiovisual works included in the timelines in order to illustrate a taxonomy composed of non-exclusive categories.
Therefore, this research falls within one of the three trends outlined by Broeckman (2019) regarding the analytical discourses arising from the new media or Postmedia, the one that addresses its technological nature, ‘digital as Postmedia’. Accordingly, we will focus on the digital nature of the works and on how their creation processes give rise to cultural objects that not only respond to a context of simulation and automation of the previous aesthetic manifestations, but also show specific features stemming from the very digital nature of their productive processes:
Faced with a quiet world, on a soothing plain, mankind can enjoy peace and repose. But in an imagined world, the sights of the plain often produce only the most commonplace effects. To restore their action to these sights, it is therefore necessary to supply a new image (Bachelard, 1975: 182).
Theoretical framework
In the early 1970s, GUI’s design was considered a minor area and its scarce development could be attributed to the fact that it was treated as a ‘traditional industrial design problem’ (Negroponte, 1995: 58) and not as a challenge regarding how to respond to the huge communicative potential and the sensory richness that we have as human beings: Our tools have a profound effect on how we think about problems (…). This applies both to the structure and organization of the code, as well as to the user interface that results from the code. (Myers et al., 2000: 14)
Also in that period, Ed Catmull and Fred Parke created the first 3D animated digital object, A Computer Animated Hand (1972), which later became an integrated GUI in Futureworld (Heffron, 1976), devoid of functionality as a tool but responding to art direction criteria, giving rise to what Mark Coleran called FUI (fantasy user interfaces) (UX Week, 2010, 7: 50). Even though in neither occasion was there any GUI involved in the process, both cases illustrate the establishment of a dialogic process triggering the synergy between technology and creativity that would boost its development.
Nowadays, the image modification process through an audiovisual creation application 7 is a simulation in most cases; the original media is never modified and work is carried out on previews of the effects of interactions performed thorough a GUI designed so that the effective computability associated with this simulation is invisible. The parametrization of these creative decisions is conducted through attractive visual metaphors, 8 inspired by a tendency to, from the moment that hardware made it possible, develop cinematically inspired visual solutions; transitions, or animations (Myers et al., 2000: 15).
In addition, in the same way that video game logic describes a scenario in which the environment itself can be a coherent fictional space while fulfilling interface functions ‘…as an informational and interactive environment’ (Jorgensen, 2013: 4), the digital image, when subjected to digital creation, editing and composition processes, also serves as an interface by becoming an element directly selectable and modifiable through, for instance, layer self-selection functions (upper left image), by combining with visual metaphors such as tracking points (upper right image), or as part of the composition map through its representation in a virtual 3D space (lower left image).
9
[…] both the interface and the game may become a test or a simulation. Both involve similar processes of subjectification, such that becoming the user of an interface is like becoming the player of a game. And both the interface and the game come into being through a threshold condition; to be the user of an interface or the player of a game is to inhabit and engage that threshold, to correspondingly be granted a kind of sacred status with respect to the interface or the game. (Hookway, 2014: 33)
Integration of visual metaphors into the creative process. Source: Developed by authors.
This gamification of audiovisual creation processes is integrated into a software logic culture according to which services operated through interfaces tend not to separate the game from the productive–creative process, which, taking into consideration the ubiquity of devices and screens, presents us a social and communicative scenario in which the distinction between simulation and reality is relegated to the scope of user subjectivity (Hookway 2014: 36–37). In addition, as noted by the author, ‘both play and work are increasingly defined by tasks performed in relation with machines and network’. At the same time, it is to be hoped that efficiency plays a decisive role at the heart of GUI’s development (Myers et al., 2000: 14).
The symbiosis between culture and computing graphical representation, which was already visible in the adaptation of the ‘window metaphor’ to television commercials at the turn of the century (Vered, 2002), 10 underwent an acceleration due to World Wide Web development transforming the computer into ‘a technology for communication more than for computation’ (Myers et al., 2000: 17) and transferring effective computing progress both to the social relations field and to the sophistication of cultural object creation. Throughout the 90s, applications transcended their nature, going from being relevant in the construction of cultural objects to being fundamental in their storage, indexing, distribution and access, so that ‘[…] all culture, past and present, is beginning to be filtered through a computer, with its particular human-computer interface’ (Manovich, 1997: 3).
Many studies have shown that the interface graphical format is close to our mental representation of problems (Clarisse and Chang, 1986), thus for any online or offline service, the graphical user interface is likely to have a quantitative impact on production processes (Galitz, 2007: 6), that is, on a time dimension. However, it is the qualitative approach and its intermediary role in creative processes that leads us to study the graphical user interface development in terms of its highly spatial and instrumental nature.
Space as a tool
Logicians draw circles that overlap or exclude each other, and all their rules immediately become clear. Philosophers, when confronted with outside and inside, think in terms of being and non-being. Thus profound metaphysics is rooted in an implicit geometry which – whether we will or no – confers spatiality upon thought; if a metaphysician could not draw, what would he think? (Bachelard, 1975: 185)
When the French philosopher published The Poetics of Space (1957) more than a century after the birth of the first algorithm and one decade after the release of the machine capable of hypothetically simulating its operational logic (Fuegi and Francis, 2003), the development of the conceptual framework that would constitute the HCI was still in the making. 11 However, the poetic dimension of his work provides perspectives on the instrumentalization of virtual space as part of a technology attached to the audiovisual creation process from the late 90s. In the final part, the author analyses how Balzac addresses the metaphysical dimension of the boy-genius Louis Lambert, focusing specifically on the phrase ‘he left space’, which the author considers less evocative than the one that Balzac had used in the first version of his work, ‘he made space withdraw before his advance’, ‘because it really was admirable, this power to make space withdraw, to put space, all space, outside, in order that meditating being might be free to think’ (Bachelard, 1975: 200). This abstraction and isolation of the spatial element is practical for the present study, since it allows us to isolate the study of space as a tool for the construction of cultural objects and to interpret Bachelard’s ‘meditating being’ as the creator who manipulates spatial metaphors as audiovisual creation tools.
In the same way as the approach to audiovisual creation in the new technological paradigm required an interdisciplinary methodological approach (Llinares and Arnold, 2015), and in line with what Lefebvre (1991: 412) pointed out regarding the need to avoid addressing the transcendence of the spatial dimension from a single discipline, we observe how the instrumentalization of space in the context of audiovisual creation draws on the narrative tradition from other languages, with which it establishes a permeable relationship.
One of the most common hybridizations in this regard occurs in comics. Thus, it is easily noticeable in the discourse of Neil Gaiman, whose comic and cinema characters in Neverwhere and Mirrormask are largely defined by the way in which they interact with space but also by how space is presented before them, since narrative spatial features are presented in accordance with a coherence adapted to the characters’ traits and not to a general logic for all of them. Therefore, while ‘[…] Richard was a navigator of space, Helena is both a creator and destroyer’ (Sommers and Eveleth, 2019: 141). This is how a dialogic relationship is established between narrative and space, which becomes a significant element of identity within his work, in terms of what Lefebvre (1991: 38) calls cohesiveness, referring not so much to a logical, unidirectional and structural relationship, but to a broader one, which in this context can be understood as narratively consistent.
Byrne (2012) explores this issue in depth, with a view to highlight the creation (and reception) space as a key element in the design of musical creation tools, which presupposes, after all, a two-way relationship ensuring an ever-changing cultural object, inasmuch as tools evolve in one direction and space and our perception of it evolve in another direction: In a sense, the space, the platform, and the software “makes” the art, the music, or whatever. After something succeeds, more venues of a similar size and shape are built to accommodate more production of the same. After a while the form of the work that predominates in these spaces is taken for granted – of course we mainly hear symphonies in symphony halls. (Byrne, 2017: 14)
It is easy to identify that instrumentalization of the space, that ‘put space outside’ (Bachelard, 1975: 200) in the story construction imagined by David Blair in Wax or the Discovery of Television Among the Bees (1993). From his use of image compositing, understood as vertical editing in the interface and spatial editing for Manovich (2001), to the use of the Web for developing a hypertextual narrative through a software called Storyspace, ‘[…] he connects his work with what he calls a ‘spatialized fiction’, made of fragments that ‘existed like connected places or many-exited plazas’’ (Willis, 2005: 42–43). That process of navigating the space to extract information is essentially the foundation of the mnemonic techniques used by Simonides of Ceos (556–468 BC) in his speeches, associating parts of his oration with objects and places in a temple, thus exploring the space to obtain information (Negroponte, 1995: 68).
At present, Walt Disney, making use of its economic potential, is committed to developing VR systems in which film-makers, gaffers and operators navigate by operating cameras and other virtual elements to obtain computer-generated (CGI) photographically realistic images (Rubin, 2019).
12
The physical film set is, in this case, an empty space taken up by mechanical solutions aimed at moving the sensors, VR headsets and joysticks that transmit orders and motion data to the virtual film set: ‘to layers of reality, meatspace motions capturing digital dailies’ (Rubin, 2019: 3). Regardless of the process innovation in terms of the instrumentalization of space, the film to which he refers has a traditional linear structure, predominant in mainstream cinema and based on temporal montage, in contrast to the spatial narrative tradition in European culture, in which events were shown at the same time
13
and it was the viewer who had to decode the visual proposal through his/her analytical observation and experience (Manovich, 2001: 321-322). According to the author, this culture would be reinventing itself through the construction of the digital image and through the impact of software, which is making it easier to manage the juxtaposition of images whose significance materializes in the relationship established by those elements in space, so that ‘[…] time becomes spatialized, distributed over the surface of the screen’ (Manovich, 2016: 40). Therefore, we consider space and its digital representations as an integral and essential element in the contemporary visual identity, in our internal cultural spirit, in our zeitgeist: The nineteenth century found its essential mythological resources in the second principle of thermodynamics. The present epoch will perhaps be above all the epoch of space. We are in the epoch of simultaneity: we are in the epoch of juxtaposition, the epoch of the near and far, of the side-by-side, of the dispersed. (Foucault and Miskowiec, 1986: 22) As a boundary condition it would be inherently active. While imperceptible in itself, it would be inferable according to its effects. It would be the site of both continuous contestation and the resolution of competing pressures. (Hookway, 2014: 59)
The tool as a metaphor
While for some creators the space of the cinema screen was an opportunity to explore visual fragmentation (Abel Gance 14 ), a spatial limitation to overcome (Jose Val del Omar 15 ), a window to the world (Víctor Erice 16 ), a reference point for the construction of an out-of-shot space (Vitorio de Sica 17 ) or an element in physical space (Doug Aitken 18 ), in the case of audiovisual creators with digital tools, their conceptualization naturally moves between the traditional representation of space and the use of the image as an interface, as a control panel (Manovich, 2001: 290). Furthermore, spatial limitations on the computer screen were the source and cause of the development of overlapping windows, so that they could cover the human field of vision, their attention span (Myers et al., 2000: 5) and, consequently, their creation process.
In the HCI context, the GUI can be observed as part of the space or area separating two entities: the humans and the code sequence that is the software (Lehni, 2008: 22), the algorithms implemented in almost any visual creation process and whose graphic expression would be interfering with its reception, all of which being able to naturally penetrate into the ‘impression of reality’ 19 that we get when we are dreaming and to which Metz and Guzzetti (1976) alluded to decode the way in which, seated in the darkness of the theatre, we gave in to the magic of cinema.
The GUI, as a non-physical interface, still maintains that quality of fluidity inasmuch as its integration into the cultural objects as well as into their creation systems and reception solutions allows it to fluctuate, establishing a relationship ranging from symbiosis and unity to the breakdown of internal aesthetic coherence. This issue becomes particularly evident when studying our relationship with video games, as the interface materializes in a space that goes from its coherent integration into the fictional space to the implementation of non-integrated graphical solutions that prioritize usability and playability (Jorgensen, 2013: 3).
Based on these two spatial logic trends, the GUI connects with other visual elements, moving from the traditional photographic coherence in composition to the breakdown or deliberate absence of integration, in which the space in the frame is constructed in a fragmented way through texts, buttons and the graphical representation of links that are part of the tools with which they have been created. Thus, ‘[…] the cinematic realism is being displaced from being its dominant mode to become only one option among many’ (Manovich, 2001: 308): While the Hollywood film industry strives to meld film stock and digital video, the tools used in digital film and sound work have created an undeniable sense of creative convergence: because the tools for creating music, designing video games and doing graphic design overlap with those for making films, they have affected and influenced each other and the impact is seen in many films created at the turn of the new century. (Willis, 2005: 45)
Screenshot of the timeline 3D view. Available at: http://bit.ly/2IeV8Ur.
Findings
In accordance with the reviewed studies, a tendency towards the implementation of cinematically inspired visual solutions was to be expected in GUI’s historical evolution. Similarly, it is evident that contemporary audiovisual discourse incorporates the GUI into proposals distant from science fiction, since software has come to influence the social and cultural ecosystem in which its discourses are developed or inspired. Thus, while most visual solutions linked to GUIs at the end of the century were the result of researches popularized in the mid-80s, almost 20 years after the digitalization of the audiovisual creation process those solutions do not seem to have varied substantially from the WIMP paradigm (window, icon, menu and pointing device). In this regard, it is interesting to examine how the digital object’s visual identity and the representation of the 3D virtual space were defined in the early 1970s in Halftone (Catmull and Park, 1972) by the same polygonal nature of the technical process necessary to accomplish that first 3D animation and modelling. Despite all this, there has been an evident and transcendent formal evolution of the visual metaphors associated with GUIs for visual creation, as well as an increased impact on the cultural layer, in a process favoured by the improved performance of tools, together with the development of the web and its language in terms of interactivity.
In this regard, we find that, as a result of improved spatial resolution in all areas: collection, CGI creation, transfer rates and online distribution, there has been a gradually increase in the number of elements and details in the GUIs (and FUIs) and, consequently, in their ability to hold a larger amount of controls and to add complexity to the visual metaphors accompanying real or fictional creation processes. At the same time, there is an emerging tendency, sometimes incorporated into the user experience, to simplify the interface or make it disappear through user options. This is particularly evident in the video game field, which generally tends towards a customizable disappearance of the GUI with a view to enhance the cinematic spatial experience, thus converging the three timeline trajectories: cinema, video game and applications.
Another convergence-related issue is the progressive use of black backgrounds in the applications and OS’ GUIs, which reflects the need to balance out the aforementioned complexity through a reduction of the visual weight of the background space by using a colour that acts as a metaphor for the absence of information, more than as a solid black colour; it is very much like an alpha channel, a transparency actually connecting with a progressive tendency of FUIS to levitate through mixed reality simulations, in which the background black is replaced by physical space. Thus, the GUI settles in the frontier between physical and digital space.
Next, we present our proposed GUI’s integration taxonomy, with which we aim to lay the foundations to study the shift from CGI technology to raw material for the construction of spatial metaphors.
GUI taxonomy in the cinematic image
The GUI as spatial editing layer. This is a literal integration of the GUI in the role of foreground or background. In both cases, the GUI integrates in a photographically realistic and coherent way into the perspective of the shot and the other composite elements and based on their position. Spatially, its behaviour responds to the logic of the GUIs from the applications that it simulates. In this regard, as a consequence of the ubiquity of mobile devices, the proliferation of visual proposals integrating messaging applications into more traditional audiovisual narratives, such as in Sherlock (Gatiss, 2010) or House of Cards (Willimon, 2013), is particularly relevant. It is also remarkable the integration of visual solutions that explore a symbiotic relationship with physical space from the user’s viewpoint in an augmented reality scenario, as in HYPER-REALITY (Matsuda, 2016).
The GUI as containers. The GUI acts as container for the audiovisual discourse and the access to it. Some prime examples are Noah (Cederberg and Woodman, 2013) and Searching (Chaganty, 2018), whose image is entirely designed through the GUI of the applications in which the images of the characters are integrated. The image establishes a dialogic relationship with the GUI, which places it in space through three different methods: ∘ Static: Detectable on video streaming platforms and their GUIs when the image does not cover the entire screen; or even when it does and we, as users, try to change the streaming settings; also through offline file playback applications’ GUIs or through OS’ preview options. It is interesting to note the connection between the proposed GUI in Keshavarz (2011) and the APCMX 600 system’s GUI included in the timeline of the present article. ∘ Dynamic: The GUI moves around the desktop space, emulating the existence of a user, who is actually the editor. Evil Twin (Dent de Cuir, 2012) and The Wilderness Downtown (Milk, 2020) are two paradigmatic examples of this method. ∘ Mixed: The GUI operates as a spatial simulation element to become a graphical continuity element used to structure the discourse: Live Cinema Documentary (Spark, 2009), Searching (Chaganty, 2018) or Define Beauty: K-Pop Men (Shin, 2019).
The GUI as a navigable space. In this scenario, the GUI becomes part of the discourse itself without losing its usability, while its interaction gives rise to a real-time generated audiovisual discourse. The spatial metaphor in the GUI responds to a double nature: the one, the tool logic, related to effective commutability, to the implementation of the algorithm; and the other, the visual metaphoric nature of its representation. An evident example is video game interface, whose GUI provides a logical integration on the aesthetic level, and which showed a clear interaction intention already in the early 1980s with Battlezone (Atari, 1980) and Punchout (Nintendo, 1983) and consolidated at the turn of the century with Mafia (Vochozka and Kudera, 2002). The most paradigmatic audiovisual piece in this regard included in the timeline is Versum (Barri, 2009).
The GUI as spatial metaphor. The GUI detaches itself from the usability logic, in part as it already did in science fiction cinema, but also freed from its credibility as a tool, to become a new visual entity stemming from computational thinking and its impact on the sociocultural sphere. Within this category, there are two types of visual proposals:
In the first one, the GUI and its logic have an impact on the creator by questioning the nature of his/her visual discourse and the spatial dialogue established by the elements in the shot. Some paradigmatic examples in this regard are Timecode (Figgis, 2000), whose proposal originated from the director’s user experience editing his previous film, when he noticed the dialectics between the images from the two Avid Media Composer interface monitors during the editing process (Bozak, 2008); Ballet Rotoscope (Sato, 2011), which starts from the spatial metaphors linked to tracking, 3D modelling and rotoscoping processes to establish a choreographic dialogue with the dancer’s movements in the physical space; or Define Beauty: Kpop Men (Shin, 2019), which, starting from a traditional documentary structure, naturally implements a catalogue of generic image manipulation processes (selection, scanning, computer vision) through applications’ spatial metaphors and whose presence reflects the transfers between the cultural and computer layer.
The second type of visual proposals include the GUI as a visual metaphor at the core of a discourse that refers to its own digital nature and contributes to building New Media’s visual identity through a spatial representation influenced by software. GUI’s implementation hybridizes with high-impact technologies in New Media’s visual identity since they include automations (self-generating image, or responsive to sound or spatial stimuli) and image interventions in the physical space. Thus, in Zero Days (Gibney, 2016), the display of the virus stems from an application generating geometries based on the position of a virtual camera that shows how the virus attacks the system. Synespecies (Merino and Droljc, 2018) uses algorithm-generated sounds leading into a 3D geometrical synthesis, an automated visual discourse in the context of digital art paradigms in which GUI plays a role both as inspiration and result. A similar, although much more literal example, is Unitxt univrs (Noto, 2011), inspired and generated by modular interfaces that respond to an object programming system. Finally, we would like to highlight two paradigmatic works: #DÉFILÉ (Tardy, 2013), which, starting from a giant web browser projected on a building, extends World Wide Web’s spatial metaphors through videomapping techniques. By doing so, #DÉFILÉ is transferring to the nature of its audiovisual discourse and its projection through spatial augmented reality the underlying dialogic tension (cultural/computer layer) with a proposal that moves between the intervention on the physical space of a historic building in the city of Prague and the projection of web browser windows to turn the volumetric object of the building into an interface simulation that does not recreate its usability, but inspires itself. The second one, Cold Stares (Manabe, 2015), challenges the physical space by digitally manipulating the scenic one occupied by its characters. Thus, by using a virtual 3D interface and drones for real-time tracking, it develops a discourse in which it delves into the aesthetic and philosophical possibilities linked to the overlapped space digitally constructed through GUI-inspired visual metaphors and a physical space that becomes just another reality representation layer.
Conclusions
The aforementioned convergence between the three timelines, in addition to the role played by the GUI in each of the categories described in the proposed taxonomy, lead us to think that GUI’s design and integration have transcended its instrumental nature and even its cultural significance to operate simultaneously on both levels, which is causing the homogenization of spatial metaphors regarding both the design of applications and the visual creations that integrate them.
Thus, in the same way that the abstraction of navigable 3D spaces as a tool is a key point in the evolution of audiovisual creation technology, it seems equally relevant that GUI has overcome its simulation nature by blending into the cultural layer; what originally was a set of visual metaphors for algorithm parametrization has become a key element within contemporary visual identity, since it takes shape through the spatial relationships occurring on the frontier between the physical and the virtual world; this is a bordering space, relevant to observe the mutations between both worlds, and which, by returning to the first meaning of the term ‘interface’, allows us to explore key issues regarding the challenges ahead of us as creators or consumers of cultural objects in a scanned and interconnected environment. The GUI as spatial metaphor questions the existence of the physical/virtual space frontier by constructing images in which the visual metaphors and the physical space get integrated by means of compositional features that go from being photographically realistic to digitally consistent and also favours the construction of a unique logic on the visual representation of cinematic space. The GUI as spatial metaphor embodies the dialogue between the computer and the cultural layer and represents algorithmic thinking through visual proposals tending to the self-referential literalness, the dialogue between simple and complex and the lack of temporal linearity.
The GUI tends to be considered as a CGI subtype, such as matte painting, 3D characters or particles, but has transcended its instrumental nature to become a key element when considering the role that software can play in the cultural sphere and, consequently, through the creative process as a method of communication and dialogue with the environment, to delve into the foundations from which to build our relationship with technology.
Spatial visual metaphors operating in the GUI have gone through three different forms of relationship with space: the physical space as inspiration (the window and the icon), the virtual space as creation interface (3D programmes’ view) and, finally, the transformation of spatial metaphors linked to virtual spaces into part of new media’s visual identity, while the GUI has become part of our social and cultural life through applications and OS.
The GUI’s ability to mutate through the different media opens up the possibility of observing its spatial quality just like Bachelard observed Louis Lambert, in its potential to show itself before us, as an independent body. In this manner, the GUI as a visual element stops responding to the tool/cultural object causal relationship so as to intervene in the physical space through pixels or light, in public and private spaces, revealing the presence of the algorithm and allowing us to call its prominence into question, while we integrate its codes more and more naturally.
In this context of dialogue with the algorithm that favours the spatial dimension of the experience, it will be necessary to focus on how we are going to address the self-generating contents associated with the IoT and with spatial computing to analyse the progress of the visual metaphors stemming from such dialogue; and if hardware implementation in our biological systems actually takes place, what are going to be the graphical solutions to communicate with them, that is, with ourselves.
