Playing with design flaws: Youth transgressive play and multimodal learning in VR ecologies

Abstract

This paper builds on prior work Ryokai and colleagues from 2022, examining youths’ design processes, perspective-taking, and collaboration with VR SandScape—a mixed-reality system combining a spatial augmented reality (SAR) sandbox with a virtual reality (VR) head-mounted display (HMD). The current case study examines how youths’ transgressive play—a form of play challenging established rules, norms, and expectations through creative manipulation of multiple meaning-making modes—expands the boundaries of possibility for multimodal learning within VR. By attuning to youths’ ingenuity and agency, we present three examples of youth co-designing new VR experiences through their play activity. Researchers presented youth with design challenges using VR SandScape. Analysis of video recordings, semi-structured interviews, and youth-created design artifacts revealed that transgressive play supported the development of multimodal interactions enabling youth to (1) learn by glitching sociotechnical systems; (2) leverage their bodies as semiotic resources to create VR user experiences; and (3) coordinate their embodied actions in which new narratives and roles were imagined and enacted. The findings highlight the potential of how transgressive play can transform top-down VR designs and learning objectives by centering youths’ ingenuity, agency, and co-design processes.

Keywords

Transgressive play multimodal practices Virtual reality embodied modes interaction,meaning making

Introduction

Virtual Reality (VR) researchers and designers are developing new ways to engage users by designing innovative experiences that reimagine how individuals interact and communicate (Bailey and Bailenson, 2017; Martin et al., 2022; Price et al., 2020; Ryokai et al., 2022). While there are many exciting developments in VR technologies, VR designs are predominantly informed by top-down approaches, where adults control the design process with little input from youth. As a result, they frequently overlook the creativity and agency that emerges when youth are given the freedom to “mess around” with these tools (Ito et al., 2013) and miss opportunities to capture the ingenuity that youth bring to sociotechnical activities. Research shows that youth resist passive media consumption by pushing the limits of intended uses, transforming technologies beyond the designers’ original intent (Barron and Gomez, 2014; Ito et al., 2013; Marsh, 2010). Examples include youth engaging with digital systems through practices such as tinkering, hacking, and glitching – digital activities that transform technological systems in ways that go beyond the designers’ intended uses. Following these lines of inquiry, this paper highlights the multimodal interactions and learning that emerged from youths’ transgressive play within a VR-mediated learning ecology. We draw on three cases from an out-of-school study that used VR SandScape to highlight the affordances and generative possibilities of youth’s transgressive play with VR technologies.

Our VR study paired middle school-aged youth, ages 13 to 14, into groups of two and engaged them in design challenges using VR SandScape, a researcher-designed system that combines a spatial augmented reality (SAR) physical sandbox with an Oculus Rift VR head-mounted display (HMD) (Ryokai et al., 2022). One youth took on the designer role, interacting with the SAR physical sandbox, while the other engaged with the VR head-mounted display (HMD) in the virtual environment. Actions performed in the sandbox were mirrored in the VR environment and the VR user’s body was represented by a dot on the sandbox (see Figure 1). From these pairings, we identified three cases that exemplify how the hybrid digital-physical VR environment creates the space for youth to challenge established rules, norms, and expectations in ways that lead to collaborative forms of multimodal learning centered on playful imagination. Through analysis of video recordings of youth’s VR design sessions, semi-structured interviews with youth, and youth-created design artifacts, we explore how these cases illustrate the potential of transgressive play to expand the boundaries of VR-mediated learning. As such, we pose the following questions:

1. What multimodal interactions emerge through youth’s transgressive play with VR technologies?

2. What are the affordances of transgressive play for supporting students’ design of new VR experiences?

Figure 1.

VR SandScape system social arrangement.

Transgressive Play

Drawing on Vygotsky’s (1978) sociocultural theory of play, which emphasizes its role as a leading activity for learning and development, our study explores how youth’s transgressive play can lead to new multimodal interactions within VR systems. Play creates the conditions for developing psychological processes, paving the way for a youth’s progression to higher levels of development and serving as a fundamental context for reorganizing performance (Griffin and Cole, 1984). Additionally, play serves as a social practice for creating and sharing meaning, learning, and engaging in cultural activities (Göncü et al., 1999; Sutton-Smith, 1997). Through play, youth leverage their ingenuity (McDermott and Raley, 2011) and imagination to take on new roles that can support the development of novel relationships. Ingenuity is a distributed phenomenon that emerges through interactions between materials, people, and unfolding environments rather than being an inherent individual trait McDermott and Raley (2011). In the VR SandScape, children’s transgressive play exemplifies ingenuity as they manipulate digital and physical elements, subvert implicit rules, and reconfigure social-spatial arrangements to create new forms of interaction and meaning. As McDermott and Raley state, “what a person does is ‘ingenious’ if it transforms those materials into something interesting, fun, or new” (McDermott and Raley, 2011: 389). Building on this idea, we focus on the transformative potential of children’s play, particularly how their subversive actions generate new possibilities. To better understand this, we drew on conceptualizations of transgressive play.

While play typically operates within established rules, goals, and boundaries, we build on scholarship that highlights the affordances of a particular form of play called transgressive play. Through transgressive play individuals intentionally subvert established norms and expectations. It encompasses actions that may be sometimes perceived as “offensive, harmful, speculative, uncomfortable, or otherwise problematic” (Bjørkelo and Jørgensen, 2018; Jørgensen and Karlsen, 2019). This form of play includes practices such as line-stepping, where individuals navigate and challenge socio-interactional boundaries to subvert participation norms (Gutiérrez et al., 2019), and glitching, wherein players intentionally subvert algorithms and system constraints to disrupt expected functions (Rivero and Gutiérrez, 2019). By engaging in transgressive play, individuals critically examine and redefine their participant within an environment, leading to innovative forms of interaction and understanding. Furthermore, through this activity, children in the VR SandScape transform social and technological boundaries into generative sites for exploration, expanding the affordances of the environment and opening new possibilities for engagement. Our framework builds on previous conceptualizations of transgressive play, focusing on its role within the VR SandScape with which the youth in our study engaged. In this context, transgressive play is characterized by the creative combination and manipulation of multiple modes of representation and interaction (e.g., visual, auditory, kinesthetic), leading to novel and unexpected learning experiences, design outcomes, and deeper understandings of the VR SandScape environment.

Given our interest in youths’ agency and ingenuity within the VR context we draw on notions of transformative agency defined as “clusters of volitional actions which break away from well-established constraining frames” and contribute to reshaping specific circumstances (Virkkunen, 2006: 1). These actions often begin as individual inquiries or initiatives but expand into collective efforts that create new possibilities for change (Sannino, 2015). We also incorporate relational agency, defined as the “capacity for working with others to strengthen purposeful responses to complex problems” (Edwards, 2011: 34). This perspective emphasizes collaboration across different knowledge systems, allowing individuals to co-construct solutions and develop shared understandings (Engeström and Sannino, 2017). By integrating these perspectives, we examine how transgressive play enables youth to navigate constraints, reimagine possibilities, and engage in collective meaning-making within VR environments. Specifically, we examined how youth leveraged and generated multimodal semiotic resources within rule-based and goal-directed VR environments through their transgressive play to subvert limitations of the system and design new experiences. These deviations from normative uses of VR technologies created generative possibilities for learning with these technologies that led us to pay increased attention to youths’ transgressive play in our study. We argue that this form of line-stepping, a generative form of transgressive play documented in our work (Gutiérrez et al., 2017), has the potential to reshape the design process of VR spaces while also showcasing youth agency and ingenuity within these contexts.

Multimodal Interactions With VR Technologies

Our theoretical lens is informed by a focus on multimodal meaning-making and interactions throughout design processes. Multimodality challenges traditional disciplinary divisions by recognizing that different means of communication—such as image with writing or speech with gesture—often appear together (Kress, 2000). VR environments are inherently well-suited for the study of multimodal learning as they afford complex interactions that combine different sensory inputs. For example, VR environments allow for the integration of visual, auditory, haptic, and even olfactory feedback, enhancing the overall learning experience of youth by providing richer, more immersive sensory experiences (Birt et al., 2018; Marsh and Yamada-Rice, 2018). Recent empirical studies have demonstrated that VR technologies also provide learners with tools to design multimodal texts featuring 3D elements, which support spatial reasoning and the development of embodied literacies (Jewitt et al., 2021; Mills et al., 2024). These affordances of VR as a tool for the development of multimodal literacies align with the robust multiliteracies framework proposed by the The New London Group (1996), which calls for a pedagogy that integrates multiple modes of meaning to transform and create new meaning.

VR technologies offer different affordances for multimodal communication, distinguished from previous forms of virtual technology with the advent of HMDs that have become prominent since 2013 (Jensen and Konradsen, 2018). VR systems allow users to interact with sophisticated three-dimensional environments using vision, sound, haptics, and touch, as well as head and body movements in fully immersive settings (Jewitt et al., 2023; Mills and Brown, 2022), which differs from being seated at a computer screen observing an avatar in a virtual world (Mills and Brown, 2022; Minogue and Jones, 2006). Recent empirical studies have highlighted the educational potential of these immersive activities, demonstrating that VR’s affordances enhance spatial reasoning, collaboration, and embodied learning (Pellas et al., 2017). For example, Pellas et al. (2017) found that immersive VR environments significantly improve the transfer of knowledge and skills to the real world.

As VR technology continues to grow in popularity, it is crucial to develop new pedagogies to leverage its full potential in promoting multimodal learning (Miguel-Alonso et al., 2023). The integration of various sensory modalities, including touch, in VR technologies (Jewitt et al., 2023) enables youth to engage in immersive experiences that can support cooperative interactions. One of the major advantages of VR in promoting multimodal learning is its ability to simulate experiences that are inaccessible in the real world, such as exploring historical settings or complex scientific phenomena. These immersive experiences allow learners to engage in story worlds and other scenarios that promote generative thinking and the transmediation of modes across media (Mills, 2022; Mills and Brown, 2022). Despite these advancements, researchers have yet to fully explore how youth agency and transgressive play can expand multimodal learning within VR contexts. While studies like those by Mills et al. (2024) and Jewitt et al. (2023) have begun to address the potential of VR for collaboration and embodiment in learning, there remains a gap in understanding how youth-driven, transgressive play with VR technologies can lead to innovative designs and expansive forms of multimodal engagement.

Methodology

Drawing on participatory research frameworks (Bang and Vossoughi, 2016), which recognize the social and cultural dimensions of learning, we conducted an ethnographic case study to explore how youth engage with VR technologies. Through this lens, we conducted participant observations and examined social interactions and meaning-making processes that emerged as students navigated the VR landscape. In collaboration with the classroom teacher, we designed learning objectives and design challenges that integrated the VR SandScape activities with students’ makerspace projects. In the makerspace, students were exploring geography and topography to design a functional train track replica. Similarly, in the VR SandScape, they engaged with concepts of topography and geology by creating mazes for their design challenges, allowing them to experiment with landscape manipulation in both virtual and physical environments. Through detailed observations of the VR sessions, we focused on students’ verbal communications, gestures, and interactions with the VR environment, which enabled us to examine how their multimodal interactions and instances of transgressive play reshaped engagement with the learning environment and expanded possibilities for meaning-making within VR. Following each session, we conducted semi-structured interviews with each student, providing further insights into their experiences.

VR SandScape is a mixed-reality environment developed by researchers to examine collaborative learning and embodied interaction within a digitally augmented physical space (Ryokai and Li, 2020). The system enables collaborative use by pairing two learners: one manipulates the physical sandbox, shaping the terrain with their hands, while the other, wearing an Oculus VR headset and controllers, interacts with the corresponding virtual landscape. A Kinect motion-sensing device captures the topography of the physical sandbox in real time, generating a digital overlay that is projected back onto the sandbox and rendered in the virtual environment. The system does not distinguish between the presence of human hands and sand terrain, so when the designer’s hand is present within the sandbox, the hand becomes a dynamically moving part of the terrain in VR. An external LCD monitor facilitates real-time observation of user interactions within the virtual environment by both users and researchers. The open-ended nature of the VR SandScape environment, coupled with its combination of physical and digital elements, provided ample opportunities for youth to engage in transgressive play and explore multimodal learning through their interactions with the system and each other.

Participants

It is important to note the age restrictions that come with current commercially available high-quality VR HMDs such as Oculus, HTC Vive: they are rated for users 13 years or older because HMDs are designed to be worn on adult-sized heads. While the HMDs come with adjustable straps, the display may appear blurry if the headset does not fit well. While brief exposure (e.g., under 20 minutes) to some blurriness should not be harmful, effects from long-term usage are currently unknown. For safety reasons, and in adherence to manufacturer’s guidelines, our study did not allow children under the age of 13 to participate in the HMD experience. Sixteen children, seven girls and nine boys, at an out of school program in a northern California middle school, volunteered to participate in our study with their parents’ permission. All sixteen children knew each other as classmates and formed eight pairs of design collaborators. However, this case study focused on 3 pairs who deviated from the design challenges so that we could examine their transgressive play.

Design Challenges

All eight pairs of children (total of 16 children) participated in three 15–30 minute design sessions, with each session spaced about 1 week apart. All eight pairs of children were asked to use the VR SandScape system to design a maze model out of sand that could be “built at full scale at their school for them and their schoolmates to experience.” We requested all children to design a “maze” (as opposed to an open ended landscape) because a maze is an object easily understood by children of this age (13–14 years). The objective of the maze was to create a navigable environment with a clear beginning and end, as well as certain elements such as dead-end and turns along the way, which afforded interesting design decisions for the children in our study.

Data Collection

The study employed a mixed-methods approach, drawing on both qualitative and quantitative data to examine youth engagement with transgressive play within a mixed-reality environment. Video recordings of participant interactions within the VR SandScape environment served as a primary data source, capturing a visual record of our research participants’ actions, gestures, verbal and non-verbal communication, and interactions, which were subsequently analyzed through micro-genetic and thematic coding to identify patterns and recurring themes. These recordings were supplemented by in-situ observations during VR SandScape sessions, where researchers documented participant interactions and communication through detailed field notes. To gain deeper insights into their experiences, semi-structured interviews were conducted with participants following each VR SandScape session.

Video recordings, field notes, and interview transcripts were then transcribed and cross-referenced to ensure accuracy. We employed micro-genetic analysis (Parnafes and DiSessa, 2013) to closely examine momentary shifts in participant activity, important for capturing the nuances of youth’s design processes and play within the VR environment. To ensure inter-coder reliability, all researchers independently coded a subset of transcripts. Subsequent discussions amongst the researchers were held to resolve discrepancies and refine the coding schema. This iterative process, guided by a collaboratively developed coding thesaurus, continued until data saturation (Glaser, 1978) was reached, indicating that no new insights were emerging from further coding. Open coding yielded 24 unique codes, which were then consolidated into fourteen final codes based on recurrence, conceptual overlap, and relevance to the research questions. The final codes were categorized into two primary themes: ‘Design Process’ and ‘Navigating Maze’.¹

Given our research interest in agency, ingenuity, and multimodal interaction, we paid particular attention to instances of “transgressive play,” again defined as playful acts that challenge or deviate from established rules and norms within the VR environment. A total of 70 such instances were identified, serving as a key focus for subsequent analyses, which drew upon thematic analysis to identify patterns and recurring themes within the data. Our analysis was informed by Goodwin’s (2000) theory of action, which attends to “the details of language use and the way in which the social, cultural, material and sequential structure of the environment” in which youth’s action occur in the co-construction of meaning. This lens allows us to examine how youth’s transgressive play within the VR SandScape environment is enacted not only through language, but also through their embodied interactions. In our study, the VR SandScape serves as the environment where pairs of participants “carry out courses of action in concert with each other” through both verbal and nonverbal means (Goodwin, 2000). Field notes and video recordings reveal how youth’s transgressive play unfolds through a multi-layered semiotic landscape. As such we attend to the language-use as the participants negotiate rules, propose ideas, express excitement, and narrate their experiences within the VR world. As Goodwin (2000) notes, “talk itself contains multiple sign systems with alternative properties,” and we examine how youth leverage these diverse linguistic resources to achieve their design goals. We also attend to embodied resources, documenting gestures, body movements, and physical interactions with the VR SandScape tools. We investigate how embodiment, in conjunction with their talk, index, change, or alter elements within the VR environment, thus shaping the meaning and trajectory of their play. Additionally, we consider how the “material” affordances of VR SandScape—its spatial configuration, the potential for glitches, and the interactive objects within it—become resources that youth creatively exploit to construct their playful design actions. Additionally, we examine the sequential unfolding of interactions, attending to how youth build upon and respond to each other’s actions, both verbally and nonverbally, to co-construct shared meanings and narratives.

Findings

This section presents the study’s findings, utilizing illustrative vignettes to offer deeper dives into the micro interactions and multimodal engagements that supported youth learning and collaboration throughout their co-design activities with the VR SandScape. We examined participant meaning-making, co-design choices, and playful interactions during their engagement with the VR SandScape, with a specific focus on how transgressive play supported the development of multimodal interactions and learning.

Vignette 1: Glitching VR interactions

We found evidence across our data to support our conjecture that transgressive play supports multimodal learning in VR ecologies. Our systematic analyses of video recordings documenting youth’s design activities revealed that transgressive play occasionally caused technological glitches within the VR SandScape, which, in turn, catalyzed opportunities for expansive multimodal learning as youth negotiated these unexpected disruptions. In the following, we present a salient example of coded instances of transgressive play with the following empirical vignette of two boys, Tom and Ike, simultaneously co-designing in a VR SandScape. We highlight the affordances of coordinated actions in multimodal environments that enabled Tom’s transgressive play to transform the VR experiences of his co-designer. The vignette highlights a particular instance of transgressive play that resulted in a youth-generated glitch within the system. We show how this glitching activity provided a rich and unexpected context for youth to develop new multimodal experiences between each other and the VR SandScape system. Although youths’ coordinated actions were initially constrained by the technological limitations of the VR software, we found that youth-created glitches afforded new modes and social interactions between the co-designers. The language-in-action described below illustrates how one youth participant, Tom, plays with the functionalities of the VR SandScape system, which typically does not support simultaneous interaction, by manipulating a dot on the sandbox that serves as an extension to Ike's avatar on the screen. The vignette opens with Tom creating his own term, “godding,” to describe his newly-found omnipotent ability to manipulate the VR user’s experiences and environment by tinkering with the dot on the SandScape.

1 Tom: ((points in sandbox)) See? Look. I’m just building. I’M GOD. I am just building.

2 Ike: Look, I’m going over the mountain! Look, I’m going over the mountain!

3 Tom: Really?

4 Ike: Yeah, that’s the other side.

5 Tom: Wait. GOD?

6 Tom: GOD KILLED YOU!

7 Tom: Hold on, where are you right now? Oh there?

8 Ike: I need to do…

9 Tom: There! I found you. I build a mountain over you! ((puts sand over Ike's dot))

10 Ike: All right. Uh…

11 Tom: There! Now you are on this. Just woke up!

12 Ike: I’m doing it, I’m doing it!

13 Tom: God! ((puts hand over Ike's dot in VR))

14 Ike: Move! Tom, move out of the way. I can’t see! ((Tom’s hand obstructs Ike's view of LCD screen))

[FN 4, 12/06/2020]

In this co-design interaction, Tom’s transgressive play with the dot on the SandScape, which serves as an extension to Ike's body, created the context for the development of unique multimodal experiences. Tom’s ability to manipulate the VR SandScape in ways that were unintended by the system’s designers demonstrates how the glitch becomes a tool for Tom to explore the boundaries of the sociotechnical systems mechanics, pushing the limits of what is possible within the virtual world. In this regard, Tom’s glitching activity created a new situation where both co-designers had to leverage multimodal sensibilities, using various modes of communication, to engage in joint-problem solving within the virtual environment. Tom’s assertion, “God killed you!” and the subsequent interactions where Tom manipulates Ike's dot on the sand, illustrate a form of line-stepping (Gutiérrez et al., 2017) — moments when youth consciously push against the boundaries of systemic constraints. As Gutiérrez et al. (2017) point out “by subtly identifying and testing a line, the line-stepper learns how and where lines are permeable and the available latitude in their enforcement” In this case, Tom’s engagement with the VR SandScape system’s boundaries is both transgressive and playful and allows Tom to assume a new omnipotent identity to reorient the objectives of the design activity that the researchers had set out for the youth. As the dialogue unfolds between Tom and Ike, they navigate the glitch-induced virtual world, leveraging multimodal resources to collaboratively problem-solve. As Ike tries to climb the “mountain,” created by Tom placing his hand next to Ike's dot, the interaction is mediated by verbal, visual, and spatial engagements. As such, the example shows how transgressive play created glitches that stretched the boundaries of the VR ecology, while supporting youth’s heterogenous modes of communication throughout their learning processes (Bezemer and Kress, 2014) (Figure 2).

Figure 2.

Tom and Ike co-designing in the sandbox.

Tom’s participation in the sand also exemplifies transgressive play as a form of agency in VR ecologies that repositions him as an “omnipotent” designer. As the youth’s interactions continued to unfold, Tom declared “I just put my hand in it. So, I just ‘godded’ you!” This exclamation undergirds the sense of design control and agency he experiences in the VR environment. In this work, virtual ecologies open up new opportunities for students to manipulate and transform their surroundings by drawing on semiotic resources such as spatial manipulation and gesture control (Lee, 2021). As he creatively assumes god-like powers and invents obstacles through the development of glitches that deviate from the intended design prompts, Tom’s actions create opportunities for learning through multimodal creativity and imagination within the virtual design. Below, we see Tom continuing to glitch the sociotechnical system with his hand such that he creates new interactions and forms of assistance for his peer.

44 I: Oh, it’s happening. It’s happening. ((T is moving I virtually using his hand in the sandbox))

45 T: I just put my hand in it. So, I just “godded” you!

46 T: There you go. I can see you right there.

47 I: I gotta go the other way [I changes direction of his avatar]

48 T: No, you are going the right way.

49 T: Turn! And go to the...

50 I: I don’t think I can get up there. [I is looking at the mountain on the screen]

51 I: Yo, I’m on the other side of the mountain. That’s great. I can see your hand.

52 T: Now Keep walking man.

[FN 4, 12/06/2020]

Tom’s use of the term “godded” to describe his actions within the virtual ecology captures the feeling of omnipotence and power he experiences through the virtual design process. As multimodal designers in VR, the youth can see their creations unfold in live-action and can experiment with different spatial arrangements and perspectives. For Tom, “godded” represents a new subject position enabling him to experiment with different perspectives and to design new VR experiences for his partner who is immersed in the virtual world. Through his transgressive play, Tom also begins to leverage new multimodal and material resources to develop new forms of assistance for his partner, as showcased by the following dialogue.

59 I: What are you doing?

60 ((T dumps sand over I’s position))

61 ((T is turning the box and I is holding the controllers))

62 T: I’m going to turn you.

63 T: Stop, stop, stop.

64 T: Oh, you are at the top of the mountain!

65 T: God! ((puts hand over I in VR))

66 I: No! Why did you god me?! ((laughs))

67 I: Now I’m falling ((laughs))

68 T: Yeah, you did. Keep walking.

69 I: Wait. Where am I?

70 T: Yeah, that’s a pretty steep fall I must admit.

71 I: Tom, you know what to do now. Tom do not god me. Stop it.

72 T: Godding you is so fun. ((giggles))

[FN 4, 12/06/2020]

Tom’s transgressive and playful manipulation within the virtual ecology, exemplified by him pouring sand on Ike's dot to trigger a glitch, demonstrates a deeper understanding of the boundaries of the sociotechnical system. Although originally, the interactions within this virtual system were triggered through his physical actions (i.e., the use of Tom’s hand), Tom introduces sand as a new materiality that glitches the system, affording an additional layer of interaction between the design partners. The glitches induced by Tom’s actions do more than just alter the environment; they afford new multimodal modes of design. In this case, we see how Tom is able to communicate with his partner and participate in an unexpected form of world-building by glitching the system with sand. This glitching activity creates a steep descent in the landscape (Line 70) that not only alters Ike's physical virtual position but also his perception of scale and dimension.

Vignette 2: Glitched bodies as VR experiences

This vignette continues our central point in this article – that transgressive play provides expansive opportunities to deepen knowledge of VR systems and co-design possibilities within these technological ecologies. In this vignette, we focus on how a youth’s transgressive play generated expansive possibilities for touch and the role of the body in VR design. We examine how one youth, Joe, in the course of his transgressive play experiments with his body to create new VR experiences for the youth co-designers. Our analytical focus is on a specific instance where Joe, as the SandScape designer, engaged in transgressive play by “flicking the dot” on the SandScape, unexpectedly causing the VR user, Zac, to fly off the screen. Although the design challenge assigned by the researchers involved using the VR SandScape technology to create a cliff that is “safe yet thrilling,” the youth’s transgressive play embodied by his flicking action laid the groundwork for novel VR experiences between the participants. The dialogue opens with Joe, the SandScape designer, trying to move Zac, the VR user, by using his fingers to manipulate the virtual dot on the SandScape. The dialogue then shows how the youth’s co-design process unfolds – a process marked by negotiation, uncertainty, and tensions (Bang and Vossoughi, 2016; Dorst, 2006; Kirshner, 2010; Vakil et al., 2016).

10 Z: Oh wait. No!

11 J: Can I take you in the other direction? ((Joe uses his fingers to pick up a virtual dot on the screen))

12 J: You are still in… You are still in… OK.

13 Z: Move your hand, move your hand.

14 Z: Oh… I’m gonna go… Alright Joe…

15 Z: We should make cliffs, too.

16 J: Yeah cliffs. ((nods))

17 Z: OK. We should start making cliffs.

[FN 3, 12/04/21]

As illustrated by the dialogue, during the initial phases of their co-design process, Zac poses a critical question: “Can I take you in the other direction?’ (line 11). This inquiry prompts Joe to broaden his design toolkit, leading him to experiment with using his fingers as VR design tools by interacting with the dot on the SandScape. When Zac covers the dot with his hand, he unintentionally triggers a glitch that prompts Zac to respond with “move your hand, move your hand” (Line 13). Observing the effects of his actions, Joe begins to engage in transgressive play, specifically by flicking the virtual dot and deviating from the original design challenge.

24 Z: Don’t do it, don’t do it, don’t do it. [telling J not to flick the “dot” virtually]

25 J: I won’t do it ((J mischievously smiles))

26 Z: You just did! ((smiles))

27 J: Look at that. OK, just put it there. Just put it on top of that. ((points at top of sand hill))

28 Z: Just put it back… You must have …

29 Z: Oh no. It’s on you Joe! [“dot” is on Joe’s hand/arm]

30 Z: I need to do that. Oh but, Mountain …. needs to be smaller than other…

31 J: There’s this one! There is this mountain.

[FN 3, 12/04/21]

Although Joe’s flicking initially creates a frustrating experience for his design partner, it also introduces new forms of interaction between the youth and the VR SandScape technology. Joe’s actions, detailed in line 29, illustrate how his transgressive play opened an opportunity for him to experiment with how he could leverage his body as a design tool within the VR SandScape environment. Specifically, Joe’s act of using his arm as a mountain reveals how he is leveraging multimodal resources, including his body, to work through the design challenge. We position his flicking activity as a form of transgressive play against expected forms of interaction, as well as a form of glitching that led to new possibilities within the VR SandScape. When youth enables the VR user to climb on his arm by creating a virtual “mountain” (line 31) with his arm, the co-designers generate new insights into how the body can be strategically utilized as an effective design tool to create new VR user experiences. These kinds of multimodal interactions enable youth to gain an intuitive and embodied understanding of the mechanisms that undergird the VR sociotechnical system and its functions (Figure 3).

Figure 3.

Joe using with arm to create a VR mountain.

Joe’s act of turning his arm into a mountain reveals the innovative potential of youth’s transgressive play to create new social landscapes, forms of interaction, and sensory experiences with and for youth in VR-mediated learning ecologies. In the example, we see how VR technologies can subvert physical constraints to extend the boundaries of how touch is perceived and experienced. In other words, Joe’s utilization of his arm as a design tool exemplifies how the materiality changes afforded by VR technologies affords innovative forms of touch and interactions beyond real-world constraints (Jewitt et al., 2021). It also highlights how VR technologies coupled with youth’s transgressive play can add a different dimension to how we perceive touch for expanded multimodal learning. In the VR SandScape, touch is mediated by various technologies, including sand, light, the VR oculus, the youth’s bodies, and the HDMI screen. As such, when Joe uses his arm to craft a mountain in the VR world for his collaborator, the youth engage with a new simulated interaction and sense of touch between each other – the youth climbing on the arm of his partner. This multimodal experience is central to the youths’ design process as it mediates a deeper, experiential understanding of the material available in the sociotechnical system and affords new interactions between the SandScape user and the designer.

Vignette 3: Transgressive play in joint storytelling

The third vignette illustrates how youth’s transgressive play supports the development of multimodal interactions that enable coordinated embodied actions, in which new narratives and roles are imagined and enacted. Finn and Gus, engaged in a maze-building task, navigate both physical and virtual spaces—Finn working with the sandbox and Gus controlling the virtual design. As Finn deviates from the original task by constructing a bridge, his playful transgression challenges the established parameters of the activity and introduces a new narrative direction. Rather than simply following the prescribed design, Finn’s creation of the bridge reconfigures the task, transforming the structure into a central narrative artifact that reshapes their collaborative story. When Finn later demolishes the bridge, this act causes a glitch, resulting in Gus’s avatar experiencing a virtual fall, which further intensifies the unfolding narrative. This unexpected disruption leads Gus to adopt a new role, that of an explorer navigating unfamiliar virtual terrain, as he adjusts to the altered landscape created by Finn’s transgressive actions. The following dialogue and actions capture this multimodal embodiment of Gus’s role as he continues to co-construct their immersive VR narrative.

39 G: Ok [G crosses the bridge while singing the Indiana Jones theme song] ((Moving his arms back and forth as if he is jogging))

40 F: ((giggles))

41 G: Yup! [proudly finishes crossing the bridge]

42 F: Yeah!

43 F: Well. You were supposed to fall off the bridge and I was going to make a staircase for you.

44 G: Oh! I’m gonna… I’m gonna go to… What!? ((F’s hand obstructs G’s virtual view)) The…?!

45 F: ((giggles))

46 G: Dude! What the heck was that?

47 F: ((giggles))

48 F: No! Go find the staircase!

49 G: Ahh, fine. Where is the staircase?

50 F: Find it next to the bridge!

[FN 1, 11/22/20]

Gus’s performative crossing of the bridge, accompanied by the Indiana Jones theme song and exaggerated jogging motions (line 39), embodies the adventurer role he creatively weaves into their shared narrative. However, Finn’s revelation of his initial intent, ‘You were supposed to fall off the bridge and I was going to make a staircase for you.’ (Line 43), reveals the unpredictable nature of transgressive play in this co-creative process. Finn’s use of a system glitch—where the physical proximity of his hand to the virtual avatar in the sandbox translates to a virtual ‘bump'—to enact the ‘bridge collapse’ (line 44) further shows how Finn’s transgressive play creates unexpected outcomes that spark new narrative directions. Gus’s surprised exclamation, ‘Dude! What the heck was that?’ (line 46), reveals a momentary disruption in the shared narrative as Finn’s transgressive act momentarily challenges Gus’s sense of control within the virtual environment. However, Finn’s lighthearted response (line 47) and subsequent redirection, ‘No! Go find the staircase!’ (line 48), demonstrate a negotiation of their agency in co-constructing the narrative (Figure 4).

54 G: Oh, it’s not a stair, Dude!

55 F: Well, that’s as good as it could be. ((laugh))

56 G: [singing Indiana Jones theme song] ((swinging arm to climb up))

57 F: OK! [as G finishes climbing up]

58 G: You may have the [inaudible] sensing!

59 F: That’s the path! ((points)) down the path.

60 G: Run! ((swinging arm as if he is running))

[FN 1, 11/22/20]

Figure 4.

Finn and Gus engaging in joint-storytelling through VR.

Gus’s playful challenge, ‘Oh, it’s not a stair, Dude!’ (line 54), coupled with Finn’s lighthearted acceptance, ‘Well, that’s as good as it could be’ (line 55), and their shared laughter, illustrate the negotiation in their collaborative storytelling process. Their exchange reveals flexibility and a willingness to adapt to unexpected outcomes within the virtual environment, as well as to include new modalities. Rather than adhering to set expectations, they incorporate imperfections in their design process and playful critiques into the evolving narrative. Gus’s use of the Indiana Jones theme song and accompanying gestures as he navigates the virtual terrain (line 56) transcends mere playfulness; it highlights the potential of multimodal interaction in joint storytelling. By blending multiple modes into the virtual experience, Gus deepens his own immersion and also shapes the shared narrative, inviting Finn into a more robust experience. Their embodiment exemplifies how youth leverage their agency within VR environments, moving beyond verbal communication to co-construct a multi-sensory narrative. It serves as a compelling illustration of how transgressive play in virtual reality fosters the development of rich, multimodal interactions and collaborative storytelling practices.

Discussion

This research paper highlighted the generative multimodal interactions that youth’s transgressive play engendered in a VR-mediated learning context. In this discussion section, we feel compelled to add a conceptual note about the “transgressive” nature of play in relation to VR technologies. We employed the concept of transgressiveness to describe children’s actions when they deviated from established norms and design tasks within this context. However, these “transgressive” acts, where children push on conventions, are routinely part of children’s creativity and ingenious play activity. In other words, the play activity of the children is transgressive insofar as it deviates from adult-imposed norms, rules, and standards. Drawing on McDermott and Raley’s (2011) notion of ingenuity as a distributed phenomenon that emerges through interactions with materials, people, and unfolding environments, children’s transgressive play is a form of ingenuity that transforms the constraints of the VR system into opportunities for novel forms of interaction and meaning-making. Through play, children are able to experiment with societal norms and expectations in ways that challenge normative usages of technology and corporations intend for us to do with them.

In our ethnographic case study, we observed how youths’ transgressive play generated novel multimodal interactions that enabled them to move beyond the sociotechnical constraints of the VR SandScape and the adult design challenges that were given to them. Through their play, youth co-constructed new ways of participating within the VR sociotechnical system that were not previously imagined by the designers or researchers in this study. This process aligns with Virkkunen’s (2006) concept of transformative agency, in which individuals take volitional actions that break away from well-established constraining frames. In addition to individual experimentation, the youth in our study leveraged relational agency, which emphasizes collaboration across knowledge systems and the ability to co-construct solutions (Edwards, 2011). Rather than engaging in transgressive play in isolation, they built on each other’s deviations, negotiated new possibilities, and worked collectively to reconfigure the designed affordances of the VR environment. This collaborative reconfiguration of the VR SandScape reflects what Engeström and Sannino (2017) describe as transformative agency in action. As such, we argue that educational contexts can benefit from embracing and leveraging youths’ transgressiveness, line-stepping (Gutiérrez et al., 2017), and boundary crossing, to reimagine and reorganize learning ecologies in innovative and consequential ways. By rearticulating and normalizing the transgressive nature of youth play when engaging with technologies, we open up new possibilities for understanding youth agency and exploring the affordances of new technologies. As such, a rearticulation and normalization of the transgressive nature of youth play when they engage with technologies can open up new ways of seeing and engaging with technologies that can be beneficial to the entire learning ecology.

The present study revealed a new aspect of the role of transgressive play in youth’s design processes within virtual reality. Youth’s transgressive play during VR SandScape design challenges generated multimodal interactions that enabled them to experiment with system glitches, use their bodies to create VR experiences, and engage in collaborative storytelling. By merging analog forms of sandbox play with VR, the SandScape technology enabled youth to leverage multiple modes and modalities that help them navigate complex sociotechnical entanglements in their everyday lives. Youth’s transgressive play produced unexpected visual distortions, altered spatial arrangements, and surreal representations of bodies in the virtual world. These outcomes set the conditions for new design processes and collaborative storytelling through coordinated action between the pairs of youth. Remarkably, the youth often embraced the glitches and distortions that emerged from their play, seeing them as creative opportunities rather than constraints. These technical imperfections challenged the youth’s preconceived notions of how the virtual world should behave, encouraging them to think outside the box and consider design possibilities they might not have otherwise explored.

Our study also underscores the value of enabling youth to engage in transgressive play with VR technologies. Allowing them to challenge prescribed design parameters and explore unconventional ways of engaging with technologies reveals youths’ spontaneous and creative manipulation of virtual environments, highlighting the imaginative possibilities that emerge when they are free to explore without rigid boundaries. This approach emphasizes the importance of prioritizing youth agency in VR design, centering their voices and perspectives. By empowering youth to play and explore in VR settings, they leverage multimodal resources, such as system glitches, that may not surface in more structured, adult-directed activities. Positioning youth as VR designers provides critical insights into the types of VR experiences that resonate with them and can inform the development of more engaging and relevant designs. This study suggests that embracing the uncertainty and unpredictability inherent in youth’s activity within VR can lead to more robust multimodal learning as well as innovative and inclusive VR design outcomes.

Conclusion

The study highlights the importance of co-designing VR technologies and experiences with youth. As the presence of VR in learning ecologies continues to grow, new challenges will emerge regarding how to implement these technologies in critical and equitable ways. Our study has implications for how educators and learning institutions may leverage the collective agency and transgressive play of youth to enhance multimodal learning experiences within these contexts. Although adults often have the best intentions for how to use these technologies with youth, it is the experimentation, ingenuity, and relationships that youth form with technologies that can inform how these tools become meaningful and impactful in their lives. Moreover, youth play and subversion of the intended uses of these technologies can reveal innovative approaches to integrating them into learning environments. The study underscores the need for educators to embrace the dynamic and often unpredictable ways in which youth engage with technology, offering deeper insights into the intersections of VR technologies and multimodal learning.

However, this study has limitations that should be considered in future research. First, the findings are based on a single case study within a larger study, which provides rich qualitative insights but limits the generalizability of the results to other contexts. Additionally, while our analysis highlights the role of youth agency and transgressive play in shaping VR interactions, the study does not systematically examine how different sociocultural or technological factors—such as prior experience with VR or differences in access to technology—may influence participation and meaning-making. Future research could explore how these factors shape youth engagement with VR across learning contexts.

Taking expansive approaches that leverage youth transgressive play and creativity in learning ecologies can also inform how we prepare youth to live through a rapidly changing literacy landscape that is widely adopting VR technologies (Mills et al., 2024). VR technologies are anticipated to continue to have a growing influence in our everyday socioeconomic lives. For example, Tilhou et al. (2020) emphasize that the latest-emerging industries increasingly depend on job activities mediated by VR technologies. Given that VR technologies and texts are also being widely adopted in higher education and K-12 settings across various nations (Tilhou et al., 2020), there is an increased need for learning ecologies that integrate VR systems into their curricula in equitable ways that draw on youth creativity, imagination, and agency. Our work highlights how embracing youth’s transgressive play within VR ecologies can help educators better understand how to support their learning and multimodal interactions that emerge in these contexts.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Sandra Jacobo

Note

Author biographies

Sandra Jacobo PhD in Learning Sciences and Human Development from the University of California, Berkeley. Her work focuses on youth media practices and co-constructed learning experiences across digital and physical environments.

Edward Rivero, Ph.D., is a Visiting Assistant Professor at Teachers College, Columbia University, in the Bilingual/Bicultural Education program. His research examines the cultural practices, digital literacies, and sociopolitical meaning-making that young people develop through their everyday media and play activities, with a particular emphasis on emergent bilingual youth. Dr. Rivero’s work explores both how technologies, including artificial intelligence, can be used to surveil and constrain student learning as well as how young people can leverage these same technologies to co-dream and co-construct al-ternative and more liberatory social futures.

Kris D. Gutiérrez is the Carol Liu Professor of Education and brings expertise in the learning sciences, literacy, educational policy, and qualitative and design-based approaches to inquiry. Gutiérrez is an elected member of the National Academy of Education and the American Academy of Arts and Sciences, the Board of Directors of the National Academy of Education and the International Society of the Learning Sciences, and past president of the American Educational Research Association.

Kimiko Ryokaiis an associate professor at the UC Berkeley School of Information and Berkeley Center for New Media. She studies representation, physicality, and interactivity of personal data and information systems, particularly in the context of mental health, social networks, and educational applications.

References

Bailey

Bailenson

(2017) Considering virtual reality in children’s lives. Journal of Children and Media 11(1): 107–113. DOI: 10.1080/17482798.2016.1268779.

Bang

Vossoughi

(2016) Participatory design research and educational justice: studying learning and relations within social change making. Cognition and Instruction 34(3): 173–193. DOI: 10.1080/07370008.2016.1181879.

Barron

Gomez

(2014) The Digital Youth Network: Cultivating Digital Media Citizenship in Urban Communities. Cambridge, MA: MIT Press.

Bezemer

Kress

(2014) Touch: a resource for making meaning. Australian Journal of Language and Literacy 37(2): 77–85.

Birt

Stromberga

Cowling

, et al. (2018) Mobile mixed reality for experiential learning and simulation in medical and health sciences education. Information 9(2): 31.

Bjørkelo

Jørgensen

(2018) The asylum seekers larp: the positive discomfort of transgressive realism. In: Proceedings of Nordic DiGRA 2018 Conference, Turin, Italy, 25–28 July 2018.

Dorst

(2006) Design problems and design paradoxes. Design Issues 22(3): 4–17.

Edwards

(2011) Building common knowledge at the boundaries between professional practices: relational agency and relational expertise in systems of distributed expertise. International Journal of Educational Research 50(1): 33–39. DOI: 10.1016/j.ijer.2011.04.007.

Engeström

Sannino

(2017) Studies of expansive learning: foundations, findings and future challenges. In: Introduction to Vygotsky. Oxfordshire, UK: Routledge, 100–146.

10.

Glaser

(1978) Theoretical Sensitivity. Oakland, CA: University of California.

11.

Göncu

Tuermer

Jain

, et al. (1999) Children's play as cultural activity. In: Goncu

(ed) Children’s Engagement in the World: Sociocultural Perspectives. Cambridge, UK: Cambridge University Press, 148–170.

12.

Goodwin

(2000) Action and embodiment within situated human interaction. Journal of Pragmatics 32(10): 1489–1522.

13.

Griffin

Cole

(1984) Current activity for the future: the Zo7ped. New Directions for Child and Adolescent Development 1984(23): 45–64.

14.

Gutiérrez

Cortes

Cortez

, et al. (2017) Replacing representation with imagination: finding ingenuity in everyday practices. Review of Research in Education 41(1): 1–17.

15.

Gutierrez

Higgs

Lizarraga

, et al. (2019) Learning as movement in social design-based experiments: play as a leading activity. Human Development 62(1-2): 66–82.

16.

Ito

Gutiérrez

Livingstone

, et al. (2013) Connected learning: an agenda for research and design. Irvine, CA: Digital Media and Learning Research Hub.

17.

Jensen

Konradsen

(2018) A review of the use of virtual reality head-mounted displays in education and training. Education and Information Technologies 23: 1515–1529.

18.

Jewitt

Chubinidze

Price

, et al. (2021) Making sense of digitally remediated touch in virtual reality experiences. Discourse, Context & Media 41: 100483.

19.

Jewitt

Barker

Golmohammadi

(2023) Feeling our way: methodological explorations on researching touch through uncertainty. International Journal of Social Research Methodology 26(5): 515–533.

20.

Jørgensen

Karlsen

(2019) Introduction: playful transgressions. In: Jørgensen

Karlsen

(eds) Transgression in Games and Play. Cambridge, Massachusetts: MIT Press.

21.

Kirshner

(2010) Productive tensions in youth participatory action research. Teachers College Record: The Voice of Scholarship in Education 112(13): 238–251.

22.

Kress

(2000) Multimodality: challenges to thinking about language. Tesol Quarterly 34(2): 337–340.

23.

Lee

(2021) Entangled: a multi-modal, multi-user interactive instrument in virtual 3D space using the smartphone for gesture control. In: Proceedings of the International Conference on New Interfaces for Musical Expression (NIME 2021). DOI: 10.21428/92fbeb44.eae7c23f.

24.

Marsh

(2010) Young children’s play in online virtual worlds. Journal of Early Childhood Research 8(1): 23–39.

25.

Marsh

Yamada-Rice

(2018) ‘Using augmented and virtual reality in the language arts curriculum’. Language Arts 96(1): 47–50.

26.

Martin

Malpica

Gutierrez

, et al. (2022) Multimodality in VR: a survey. ACM Computing Surveys 54(10s): 1–36.

27.

McDermott

Raley

(2011) Looking closely: toward. The Sage Handbook of Visual Research Methods. Thousand Oaks, CA: Sage, 372, 372–391.

28.

Miguel-Alonso

Rodriguez-Garcia

Checa

, et al. (2023) Countering the novelty effect: a tutorial for immersive virtual reality learning environments. Applied Sciences 13(1): 593.

29.

Mills

(2022) Potentials and challenges of extended reality technologies for language learning. Anglistik 33(1): 147–163.

30.

Mills

Brown

(2022) Immersive virtual reality (VR) for digital media making: transmediation is key. Learning, Media and Technology 47(2): 179–200.

31.

Mills

Brown

Funnell

(2024) Virtual reality games for 3D multimodal designing and knowledge across the curriculum. Australian Educational Researcher 51: 1–31.

32.

Minogue

Jones

(2006) Haptics in education: exploring an untapped sensory modality. Review of Educational Research 76(3): 317–348.

33.

New London Group (1996) A pedagogy of multiliteracies: Designing social futures. Harvard educational review 66(1): 60–92. DOI: 10.17763/haer.66.1.17370n67v22j160u.

34.

Parnafes

diSessa

(2013) Microgenetic learning analysis: a methodology for studying knowledge in transition. Human Development 56(1): 5–37.

35.

Pellas

Kazanidis

Konstantinou

, et al. (2017) Exploring the educational potential of three-dimensional multi-user virtual worlds for STEM education: a mixed-method systematic literature review. Education and Information Technologies 22(5): 2235–2279. DOI: 10.1007/s10639-016-9537-2.

36.

Price

Yissacnoutsou

Vezzoli

(2020) Making the body tangible: elementary geometry learning through VR. Digital Experiences in Mathematics Education 6(2): 213–232. DOI: 10.1007/s40751-020-00071-7.

37.

Rivero

Gutiérrez

(2019) “You know what’s glitching?”: Emergent digital literacies in video game play. The Routledge Handbook of Digital Literacies in Early Childhood. London: Routledge, 158–168.

38.

Ryokai

(2020) VR SandScape: working with multiple perspectives in a hybrid VR/SAR collaborative play space. In: Extended Abstracts of the 2020 annual symposium on computer-human interaction in play, Canada, 02–04 November 2020, 350–354.

39.

Ryokai

Jacobo

Rivero

, et al. (2022) Examining children's design processes, perspective-taking, and collaboration when using VR head-mounted displays. International Journal of Child-Computer Interaction 33: 100451. DOI: 10.1016/j.ijcci.2021.100451.

40.

Sannino

(2015) The principle of double stimulation: a path to volitional action. Learning, culture and social interaction 6: 1–15.

41.

Sutton-Smith

(1997) The Ambiguity of Play. Cambridge, MA: Harvard University Press.

42.

Tilhou

Taylor

Crompton

(2020) 3D virtual reality in K–12 education: A thematic systematic review. In: Yu

Ally

Tsinakos

(eds). Emerging technologies and pedagogies in the curriculum: Bridging human and machine—future education with intelligence. Singapore: Springer, 169–184. Available at: https://doi.org/10.1007/978-981-15-0618-5_10

43.

Vakil

McKinney de Royston

Suad Nasir

, et al. (2016) Rethinking race and power in design-based research: reflections from the field. Cognition and Instruction 34(3): 194–209.

44.

Virkkunen

(2006) Dilemmas in building shared transformative agency. Activités revue électronique 3(3–1): 43–66. DOI: 10.4000/activites.1850.

45.

Vygotsky

(1978) Mind in Society. Cambridge, MA: Harvard University Press. (Original work published 1934).