Augmented Reality in Science Laboratories: Investigating High School Students’ Navigation Patterns and Their Effects on Learning Performance

Abstract

Augmented reality (AR) has the potential to fundamentally transform science education by making learning of abstract science ideas tangible and engaging. However, little is known about how students interacted with AR technologies and how these interactions may affect learning performance in science laboratories. This study examined high school students’ navigation patterns and science learning with a mobile AR technology, developed by the research team, in laboratory settings. The AR technology allows students to conduct hands-on laboratory experiments and interactively explore various science phenomena covering biology, chemistry, and physics concepts. In this study, seventy ninth-grade students carried out science laboratory experiments in pairs to learn thermodynamics. Our cluster analysis identified two groups of students, which differed significantly in navigation length and breadth. The two groups demonstrated unique navigation patterns that revealed students’ various ways of observing, describing, exploring, and evaluating science phenomena. These navigation patterns were associated with learning performance as measured by scores on lab reports. The results suggested the need for providing access to multiple representations and different types of interactions with these representations to support effective science learning as well as designing representations and connections between representations to cultivate scientific reasoning skills and nuanced understanding of scientific processes.

Keywords

mobile AR science lab navigation patterns multiple representations scientific reasoning and processes

Introduction

Augmented Reality (AR) integrates physical and virtual objects into the same scenario and allows physical and virtual objects to run interactively in real-time (Azuma, 1997; Moro et al., 2021). Conceptually, AR is located between the physical and virtual environments on the reality-virtuality continuum (Milgram et al., 1995). One type of AR technology is an optical see-through system (e.g., AR headset) in which the physical environment is observed through a partially reflective glass and the virtual objects are displayed on the glass. Another type is a video see-through system (e.g., a mobile or tablet) in which a video camera provides the view of the physical world and the virtual objects are merged into a single video stream. In Martin et al.’s (2011) review of technology trends in education, the authors highlighted the trend of moving conventional AR towards mobile AR technologies (Kourouthanassis et al., 2015) for ubiquitous learning and seamless interaction between physical and virtual worlds.

AR has great potential to empower students to learn science concepts effectively. Literature has shown that AR technologies could cultivate students’ interests in science (Abdinejad et al., 2021; Ibáñez et al., 2014), help students to gain in-depth science knowledge and practice (Chiu et al., 2015; Turan & Atila, 2021), and facilitate transferring science knowledge across contexts (Chiang et al., 2014). Meanwhile, the literature pointed out that learning science with AR technology could support students in constructing sound scientific explanations and called for more rigorous investigations of the learning effect of AR technologies, in particular high-level cognitive outcomes such as capacities in developing scientific claims and understanding scientific processes (Radu, 2014).

Scholarly attention has turned to examine affordances of AR technology that could contribute to science learning opportunities. Researchers found that AR afforded interactions with multiple representations in both physical and virtual worlds (Zimmerman et al., 2016), which significantly lowers the barriers to effective science learning (Kamarainen et al., 2013). With AR technology, students could annotate real-world objects (e.g., annotating forces on a physical moving object; Sotiriou et al., 2006), learn science concepts in the rich context of a real environment (e.g., museum, campus, and garden; Tarng & Ou, 2012), and engage in science learning with embodied interaction (e.g., using hand to move and rotate digital models of internal organs; Blum et al., 2012). In Ibáñez and Delgado-Kloos’s (2018) review on AR technology for STEM education, the authors explained that AR technologies could aid the consumption of science ideas through presenting multiple representations, including representations that connect the physical and virtual world. Collectively, these studies demonstrate that flexible interactions with different representations positively affect learning outcomes.

However, patterns of students’ navigating multiple representations in AR remain unexplored and there is limited evidence that suggests how the navigations could lead to science learning opportunities (Akçayır & Akçayır, 2017). This study aims to address this gap by examining the navigation patterns and the relationship between navigation patterns and learning performance in science laboratory settings, which is one of the first in this context to our knowledge. In this paper, we explore student learning with a mobile AR technology, Infrared Explorer (described in the section of methodology), in science laboratories. Specifically, this study addresses the following research questions:

What kinds of navigation patterns exist when students conducted hands-on science experiments with the mobile AR technology?

How did the different navigation patterns, measured on the basis of log data, relate to students’ learning performance?

Theoretical and Empirical Background

Two theoretical frameworks were integrated to understand student learning in AR-based learning environments: embodied learning that occurs through interacting with physical objects and multimedia learning that involves interactions with different representations, such as visual and verbal information. In cognitive sciences, embodied learning is a pedagogical theory that highlights the coupling of body movements and cognitive activities in educational practice (Skulmowski & Rey, 2018). Cognitive processes are deeply rooted in sensorimotor processing and come from embodied interaction with a physical environment (Wilson, 2002). Previous studies have shown that both fine-grained and gross motor skills led to improved science learning as learners could encode tactile information along with the educational content (Lindgren et al., 2016). As an example, Fidan and Tuncel (2019) showed that the tactile characteristics of FenAR application contributed to high school students’ long-term retention of physics concepts (e.g., weight, mass, and gravity) in a problem-based learning environment. Extending embodied learning to acquire invisible science concepts, involving bodily movements and haptics can be an alternative method in science education that enhances the understanding of these foundational concepts.

Multimedia learning theory suggests that learning outcomes can be improved if students learn through more than one representation (Mayer, 2014; Moreno, 2006). For instance, Altmeyer et al. (2020) presented a novel tablet‐based AR application that supports conducting laboratory experiments to learn electricity in higher education. The authors compared cognitive load and learning gain in two conditions, an AR-supported and a matching non-AR learning environment. The study showed that when designed appropriately, multiple representations from AR applications would not add the cognitive load of interpreting representations and held the promise of supporting the acquisition of in-depth scientific knowledge. Furthermore, Mayer (2014) suggested that students would actively select and integrate different representations to make meaning of them and this meaning-making process could contribute to effective learning. Therefore, we expect that providing access to multiple representations in AR-based learning environments could promote students’ engagement and increase learning outcome.

To date, AR has been applied as an educational medium to benefit student learning. AR-based learning environment could improve students’ understanding of complex phenomena (Chiu et al., 2015; Turan & Atila, 2021), lengthen long-term memory knowledge retention (Chiang et al., 2014), and increase student motivation of learning (Akçayır et al., 2016). Experiences in AR bring three positive and unique attributes to improve learning outcomes. First, the physical interactivity provided by AR promotes embodied learning. Therefore, learners in AR can encode proprioceptive information along with the educational content. Second, AR can align educational content with physical items through spatial and temporal registration, so that spatial and temporal contiguity can be guaranteed (e.g., a virtual label with textual explanation shows up next to a physical object). According to multimedia learning theory, spatial and temporal contiguity can effectively reduce the cognitive load for the brain to process information from sensory channels. Third, AR interface and content are novel and motivational because 2D representations become lifelike 3D objects in the students’ own physical environments. With such an engaging experience, an AR educational environment enables easier processing of the delivered educational content (Mayer & Moreno, 2003), and promotes student exploration and creativity (Kaufmann & Dünser, 2007). Given these benefits brought by AR, we developed a video see-through mobile AR application to facilitate carrying out hands-on science laboratory experiments in high school classrooms.

Methodology

Participants and Learning Context

Seventy ninth-grade students (Female 37, Male 31, Other 2; White 47, Asian/Pacific Islander 12, Latinx 5, American Indian/Alaska Native 4; African American 3, Other 4) from four science classrooms of one suburban public high school in the Northeastern United States participated in this project. They were taught by a male teacher, Kevin (all names are pseudonyms). Kevin attended our professional development workshops before implementing the project and used course materials developed by the research team when implementing the project. The research team had a well-established partnership with Kevin and the school. We took the position of participant-observer (Spradley, 1980) in this study and conducted the main task of collecting data and giving students feedback when needed.

In this project, students conducted science experiments in pairs. Each pair was provided with one smartphone to run Infrared Explorer. Kevin first paired up students who did not submit consent forms and then paired up the rest of students based on how he used to pair them in other hands-on labs. Even though the setup reduced the individual data available for further analytics, pairing students in science labs did enhance collaborative learning experience (Shibley & Zimmaro, 2002). Specifically, they participated in a five-day curricular unit with one session (approximately one hour) per day. Kevin introduced the mobile AR technology at the beginning of the first day and then students conducted the following experiments to learn thermodynamics: thermal radiation, natural convection, forced convection, conduction, and latent heat (Table 1).

Table 1.

Guiding Questions for Students to Discuss, Illustration of Set Ups and Experimental Procedure in Lab Reports for Each Experiment.

Experiment	Guiding question	Experimental procedure
Radiation: catch invisible light!	How does heat transfer through thermal radiation?	1. Record a video of hot water jar facing the paper. Students should first put the jar 1” away from the paper and then move the jar 1” further away.2. Record a video of hot water jar alongside the paper. Students should first put the jar 1” away from the paper and then move the jar 1” further away.3. Conduct video analysis.
Natural convection: track invisible flow!	How can you distinguish between the effects of convection and radiation?	1. Record a video of paper with a hot water jar in the cutout.2. Record a video of paper above the jar in the horizontal position.3. Conduct video analysis.
Forced convection: blow heat away?	Do we always feel cooler when we turn on the fan?	1. Record a video of hot water jar alongside the paper while using the hand-held fan to blow air towards the jar.2. Conduct video analysis.
Conduction: two thumbs up!	Why do metals feel colder than wood?	1. Record a video of touching two rulers with thumbs.2. Conduct video analysis.
Latent heat experiment: paper on cup	How can thermal energy be generated without using a heat source?	1. Record a video of covering a cup with dry paper.2. Conduct video analysis.

When using the technology to conduct experiments, students went through the prediction-observation-explanation (POE) process (Coştu et al., 2012; Ebenezer & Erickson,1996). They were required to finish a lab report in each experiment. Guided by the report, they first answered a few questions about science phenomena based on prior knowledge (i.e., prediction phase), then conducted hands-on experiments to observe science phenomena (i.e., observation phase), and lastly, answered a few questions similar to those in the prediction phase to explain science phenomena (i.e., explanation phase). These questions were used to understand students’ prior knowledge about relevant science concepts. For example, in the first experiment, students conducted a radiation experiment to learn heat transfer. Before conducting the experiment, students answered two multiple-choice questions and explained their choices. The two questions were about comparing the temperature of the piece of paper when it was 1 inch and 2 inches away from the hot water jar in the scenarios of jar facing the paper and jar alongside the paper respectively (as shown in Table 1). After answering the questions, they moved to the observation phase and carried out two experiments: jar facing the paper and jar alongside the paper. In the observation phase, students used Infrared Explorer to explore the temperature of the paper when the jar was placed at different places. After conducting the experiment, they answered two questions that aim to assess students’ understanding of the concepts of thermal radiation. The questions were similar to the questions in the prediction phase. Specifically, students described the temperature changes when the paper was facing the jar and when the jar was alongside the paper. Other experiments followed a similar flow of POE activities.

AR Technology: Infrared Explorer

This study builds on a five-year design-based research project that explored affordances of a mobile AR technology, Infrared Explorer (Figure 1; Sung et al., 2021; Xie, 2011; Xie & Hazzard, 2011), for science learning. A wide range of science experiments that the technology can support are listed here: (website link omitted for blind review). In this project, we focused on the learning of physics concepts, in particular, thermodynamics.

Figure 1.

The Experiment Set Up of Infrared Explorer, a Mobile AR Application.

Infrared Explorer allows students to explore a thermal version of laboratory experiments using the FLIR ONE IR camera attached to an Android or iOS device (Figure 1). As easy to use as a conventional camera, the FLIR ONE is a high-throughput data acquisition instrument that collects thousands of temperature data points each time a picture is taken. To facilitate the investigation, exploration, and analysis of experiments, Infrared Explorer provides interactions with different representations, such as data graphs (Figure 2A), images (Figure 2B), videos (Figure 2C), and physical objects through editing thermometers (Figure 2D). The data graphs present temperature changes over time with time and temperature being the x-axis and y-axis respectively. The images illustrate temperature differences with contrasting colors (i.e., areas in purple have lower temperature than areas in red in default settings; users could change colors) and the videos show temperature differences over a particular period. Students can create galleries of data graphs, images, and videos as evidence to support their scientific claims. In addition, students can add thermometers in images and videos to read temperature of anywhere of interest and at any particular time point.

Figure 2.

Four types of representations in Infrared Explorer. (a) Data graph; (b) image; the thermal view shows up in default; students can switch between thermal view and real-world view and can also change color scheme; (c) video; students can record a video to show the thermal view over time; (d) students can interact with objects in the physical world, such as through adding a thermometer to read temperature.

Data Collection

We collected multiple sources of data to examine students’ science learning with Infrared Explorer and navigation patterns, including semi-structured interviews, log data, and lab reports.

Semi-structured interviews

At the end of the project, we conducted semi-structured interviews (Patton, 1990) with students to learn about their experiences of learning science with the mobile AR technology. The teacher, Kevin, selected 31 students to conduct the interview based on students’ classroom performance, English proficiency, and availability. These students were selected to cover both students with high and low achievement levels and students with high and low English proficiency. This stratified sampling (Patton, 1990) was utilized to select interviewees to ensure broad representativeness and applicability of results. This was individual -based interview. Each interview lasted around ten minutes. We asked about their attitudes and perceptions toward science learning with AR technology, such as “how did the technology, the smartphone, the camera, and the lab activities help you do science” and “which lab activity surprised you the most?”

Log data

The mobile AR application logged students’ processes of conducting hands-on experiments, including interactions with multiple representations. There are 41 types of interactions with different representations (Table 2): 9 interactions with data graphs (e.g., Scale data graphs to fit window), 10 interactions with images (e.g., Open palette to change the color legend of images), 14 interactions with videos (e.g., Start dragging slider to select a particular period of videos), and 8 interactions with physical worlds (e.g., adding thermometers to read temperatures of areas of interests). We collected log data from 35 pairs while data for one pair was missing due to technical issues and one pair did not submit consent forms. Thus, our analysis was based on log data for 33 pairs.

Table 2.

Overview of 41 Types of Interactions With Representations.

Representation	Interaction (number of interactions with representations)
Data graph	Open Analyze Menu, Graph T(t), Graph T(x), Pan Graph, Scale Graph, Fit Graph in Window, Export Graph as Image, Export Time Graph as CSV, No Graph (9)
Image	Open Image, Screenshot, Rotate Image, Blur, Brighten, Posterize, Open Palette, Close Image, Delete Image, Save Image (10)
Video	Create MyVideosActivity, Start MyVideosActivity, Pause MyVideosActivity, Stop MyVideosActivity, Resume MyVideosActivity, Destroy MyVideosActivity, Start Dragging Slider, Stop Dragging Slider, Open Videos, Delete Video, Start, Pause, Play, Stop (14)
Physical world	Add Thermometer, Move Thermometer, Remove Thermometer, Clear Thermometers, Add Thermoruler, Move Thermoruler, Remove Thermoruler, Clear Thermorulers (8)

Lab reports

We collected 35 lab reports from students as they worked in pairs to conduct experiments. In the lab report, students answered 11 questions related to the experiments in the explanation phase (Table 3). Their responses to these questions were used to evaluate students’ learning performance.

Table 3.

Questions That Students Answered in the Explanation Phase.

Experiment	Questions
Radiation: catch invisible light!	1. The paper warmed up when facing a jar of hot water and cooled down when facing a jar of cold water. Why was this phenomenon caused by radiation but not something else? 2. When the paper facing the jar was further away from the jar, did the paper absorb less thermal radiation? Why or why not?3. When the jar was alongside the paper, did the paper absorb thermal radiation as much as the case when the paper faced the jar? Why or why not?
Natural convection: track invisible flow!	1. For the cutout paper experiment, how can you distinguish the effects of heat transfer from a hot water jar to the paper through radiation and convection, respectively?
Forced convection: blow heat away?	1. Does the paper always cool off when we turn on the fan?2. How do you determine if the flow of thermal energy is facilitated by an external force? Explain.
Conduction: two thumbs up!	1. Does thermal energy diffuse at different rates in different materials?2. Why does the thumb on the metal ruler feel colder?
Latent heat: paper on cup	1. Why is the water temperature lower than the room temperature?2. Why does the area of dry paper covering the cup warm up?3. What mechanisms are responsible for the formation of the thermal pattern?

Data Analysis

This study involved four phases of data analysis. In the first phase of analysis, we openly coded semi-structured interviews (Patton, 1990) to understand students’ learning experiences in the project. The emerging themes showed that students perceived interacting with different representations (e.g., data graphs) as engaging and effective for science learning (Jiang et al, 2020). The emerging themes from the first phase of analysis guided our analysis of log data in the second phase of analysis.

In the second phase, we coded log data by focusing on interactions with different representations, including data graphs (D), images (I), videos (V), and objects in the physical world (P). The navigation pattern represented the sequence of interacting with different representations from the first to the last experiment. Considering navigation patterns in one experiment could be totally different from navigation patterns in another experiment and our focus was examining navigation patterns in this learning environment over time, we used the sequence of all experiments. As an example of sequence, in a scenario, a student first clicked video recording button, then added thermometers to measure temperature of objects, and then dragged the timeline of the video to see temperate changes over time. The navigation pattern of this scenario would be represented as VPV. Afterward, hierarchical clustering analysis (Li & Tsai, 2017) was performed in R studio using the TraMineR package to classify student pairs into different groups based on the navigation pattern. The clustering analysis generated two clusters of navigation behaviors (as described in the Results section). Drawing from gene pattern detection in the field of microbiology (Bekal et al., 2003), navigation length and breadth were used to differentiate these two clusters. Navigation length, measured with frequency, represents the number of representations students explored while navigation breadth, measured with standard deviation, shows the distribution of movements from one representation to another, regardless of the direction. In the aforementioned scenario (VPV), the navigation length is three (V, P, V) and the navigation breadth is 0.66 (DI = 0, DV = 0, DP = 0, IV = 0, IP = 0, VP = 2). We employed independent t-tests to compare mean differences of navigation length and breadth between these two groups.

Furthermore, to fully understand students’ nuanced interactions with different representations, we visualized navigation patterns using d3.js, a JavaScript library for producing dynamic and interactive data visualizations. Figure 3 illustrates an example of the visualization: each node represents one type of interaction; the color of nodes represents interactions with different representations; the arcs represent relationships between nodes with arcs above nodes showing direction from left to right and arcs below nodes showing direction from right to left; the weight of arcs indicates frequency. We compared and discussed the visualizations to gain an in-depth view of learning opportunities that the patterns reveal and characteristics of patterns for each group (Andrienko & Andrienko, 2013). We also reviewed interview data to find evidence for or against findings about learning opportunities for accuracy and used the transcripts to fill in any gaps. When reviewing the interview data, we were particularly interested in, from individual students’ perspectives, how and why they performed certain navigation patterns. In other words, interviews were used to show students' perspectives when presenting observations from log data.

Figure 3.

Juan and His Partner’s Navigation Pattern: Iterative Video Interaction. In this pattern, the weight of arcs within video interactions is greater than arcs within other representations, indicating the pair interacted frequently with videos.

In the third phase, we adopted Ruiz‐Primo and Shavelson’s (1996) framework to code lab reports. In particular, we coded student responses to explanation questions from two dimensions: giving rationale and describing processes. These two dimensions have been stressed as critical but challenging science practices (Chiu et al., 2015). In each dimension, we coded the responses using a 3-point scale (0-2). If students extensively or briefly described the reasons for their scientific claims, their answers were coded as 2 or 1 point respectively. If they did not explain the rationales, we coded the answers as 0 point. Likewise, we coded students’ responses as 2 if students clearly described dynamic changes in science phenomena, 1 if there were limited descriptions of dynamic changes, 0 if they did not describe dynamics changes. We discussed and resolved coding disagreements in weekly meetings. Learning performance was the sum of scores for the eleven questions (Table 3) from each experiment as these experiments are independent (Garribba et al., 2001).

In the last phase of data analysis, we conducted independent t-tests in SPSS (version 27) to identify differences in learning performance between the two groups. In addition, given these two groups displayed different navigation patterns, we discussed the relationship between navigation patterns and learning performance as measured by student responses in lab reports in weekly meetings and generated analytical memos (Lee et al., 2019) to describe the relationships. In this process, we also revisited interview data to understand students’ perspectives on their learning experiences. In the following section, we will present student navigation patterns, characteristics of navigation patterns that could contribute to learning opportunities, and the relationship between navigation patterns and learning performance.

Results

RQ1: What Kinds of Navigation Patterns Exist When Students Conducted Hands-on Science Experiments with the Mobile AR Technology?

The cluster analysis classified student pairs into two distinct groups, group 1 with 13 pairs and group 2 with 20 pairs. These two groups showed different patterns of navigating representations. Compared with group 1, group 2 had significantly more frequent movements (i.e., group 2 had a larger value in navigation length; t (31) = −7.09, p <.001) and a significantly more uneven distribution of movements between different representations (i.e., group 2 had a larger value in navigation breadth; t (31) = -7.48, p <.001; see Table 4). In other words, students in group 1 tended to focus on particular representations while students in group 2 were more likely to engage in moving from one particular representation to another. This does not necessarily mean that group 1 spent less time in conducting lab experiments as they might have in-depth exploration within certain representations.

Table 4.

Differences Between Groups in Terms of Navigation Length and Breadth.

	Group	Mean	SD	t
Navigation length	1	51	11.06	−7.09∗∗∗
	2	94.95	20.40
Navigation breadth	1	7.14	1.54	−7.48∗∗∗
	2	12.89	2.86

∗∗∗p < .001.

Specifically, in group 1, three out of thirteen pairs performed iterative video interactions. They engaged extensively in observing the thermal view over time and editing video recordings of the thermal view. The thermal view offered students a new perspective to investigate nuances and dynamic changes in science phenomena. As Juan described in the interview, “It (the thermal view) is more of a way to experience different things from a new angle or point-of-view.” Juan and his partner’s interaction (Figure 3) with the AR technology demonstrated frequent movements within the video representation (i.e., pattern of iterative video interactions). In Figure 3, clearly, the weight of arcs within video interactions is greater than other arcs. This indicates that they had frequent navigations within this representation. For instance, the arc from “start dragging slide” to “stop dragging slide” is large, showing that the pair dragged sliders to view specific video frames. They created approximately four videos in each experiment. In particular, they dragged the slider (i.e., representing the timeline of videos) to identify temperature changes in critical moments (e.g., pushing the paper 1” to the side after covering the cup with the dry paper for one minute) and recorded new videos when they could not observe obvious temperature changes over time. This pair represented those who had fewer (navigation length = 52) and less uneven distribution of movements (navigation breadth = 8.46).

Students in group 1 tended to perform video-physical-world interactions, navigation breadth of this group was smaller than group 2 though. These students not only had frequent navigations within video representation but also often interacted with objects in the physical world. They explored the thermal view while editing thermometers to capture temperature of different places in the physical world. Camille and her partner’s laboratory experience highlights this pattern (as shown in Figure 4). Figure 4 shows arcs from video representation to physical world representation and comparing with the pattern of iterative video interaction, there were more arcs in the physical world representation for this pattern. This pair frequently edited thermometers to collect evidence for lab reports. In the interview, Camille shared, “Probably the thermometers (help me most in doing science). Because I could just see how hot something is just by taking a picture of it (referring to screenshot of a video) and putting a thermometer on it.” This pair’s navigation pattern illustrates that interacting with videos was in the service of gaining the temperature needed as evidence collection for lab reports.

Figure 4.

Camille and Her Partner’s Navigation Pattern: Video-Physical-World Interaction. This pattern shows that in addition to frequent interactions with videos, the pair navigated between videos and objects in the physical world.

Group 2 had larger navigation breadth, which indicates that some students developed strategies to conduct experiments by focusing on moving from certain representations to another. For instance, Olivia, a student in group 2, performed iterative video-data-graph interactions, which entails flexible movement between video and data graph. The graph, showing temperature changes over time for one or multiple thermometers, served as a venue for scientific reasoning. Olivia stressed that video and graph helped her to learn science and further explained:

Because every single time that I moved the video and I see the thermometer (referring to the graph) go up and down. That was so, in my opinion it was like it was controlling how much it went up every single minute and second. Some changed and some stayed the same, and then it comes to me why some stayed the same, you know, with hot jar close to it.

Olivia’s interview response demonstrates that the video provided contexts of interpreting graph and the graph triggered scientific reasoning. She observed and reasoned about different patterns of changes based on the graph. Figure 5 represents this pair’s navigation pattern. Comparing with other patterns, there were more arcs from video representation to data graph. This pattern illustrated frequent and diverse navigations within the representation of data graph as well as flexible movements between video and data graph.

Figure 5.

Olivia and Her Partner’s Navigation Pattern: Iterative Video-Data-Graph Interaction. In this pattern, the pair navigated between videos and data graphs.

However, two students, Mary and Bryan, explained the challenge of interpreting the graph: “the graph was a little too complicated (Mary’s interview)” and “the one where you had to graph and put the temperatures that was really confusing for me. (Bryan’s interview)” This calls the attention that adding multiple layers of information might increase the cognitive load and suggests the need of helping students to develop data literacy.

As shown in these patterns of navigating representations, students not only frequently interact within one representation, but also moved across different representations. Furthermore, these patterns demonstrated various ways of observing, describing, exploring, and evaluating science phenomena. These cases, among other cases, demonstrate that providing access to multiple representations and offering different kinds of interactions had the potential of supporting effective science learning.

RQ2: How Did the Different Navigation Patterns, Measured on the Basis of Log Data, Relate to Students’ Learning Performance?

Students in group 2 performed better in describing reasons for scientific claims than those in group 1. Learning performance was measured by scoring lab reports from two dimensions: giving rationale and describing processes. As shown in Table 5, students in group 2 significantly got better scores in giving rationale than those in group 1 (t (31) = -2.27, p <.05). Such a significant difference might be related to group 2’s frequent movements between different representations. The flexible movement might indicate that these students understood and could use connections between different representations to reason about science phenomena. For example, Byron, a student in group 2, reflected in his interview:

Table 5.

Differences Between Groups in Terms of Learning Performance.

	Group	Mean	SD	t
Giving rationale	1	8	6.65	−2.27∗
	2	12.45	4.65
Describing processes	1	2	2.71	−0.49
	2	2.50	2.97

∗p < .05.

They (representations) help you understand better what exactly is going on at some point, and as time goes by. It (referring to the thermal view) helps you understand the things that you can't really see with your own eyes, so it really makes it easier to understand what's going on on a cellular level or a thermal level and over time with graph, to see the changes.

Byron mentioned that the thermal view presented a new angle to investigate science phenomena at particular time points and the data graph helped him to reason overall temperature changes over time. The data graph could help students to identify when and where temperature was increasing, decreasing, or constant while the thermal view could contextualize specific lab settings at that point. As shown in this case, students became aware of the connections between representations and developed strategies to leverage the connections to support scientific reasoning.

Students in group 1 and 2 performed similarly in describing dynamic changes in scientific phenomena. There was no significant performance difference in describing processes between the two groups, the mean of group 2 (M = 2.5) is larger than group 1 (M = 2) though. This result makes sense as the group was clustered on the basis of navigation length and breadth while describing processes might be more related to navigations within particular representations, such as videos. For instance, in group 1, Camille shared her experiences of frequent edits with video to explore temperatures and collect evidence for lab reports. In the lab report for Two Thumbs Up, she and her partner provided detailed descriptions of temperature changes, “once the thumbs pressing on the rules the temperature near the bottom started to rise. The metal ruler had a greater temperature. And once the thumbs moved away the temperature where the thumbs were, decreased a little.” This case demonstrated that students paid close attention to dynamic changes in scientific phenomena through describing videos. In addition, Table 5 shows that the mean score of describing processes was much lower than the mean score of giving rationale. The low score of describing processes was associated with the fact that students were not required to elaborate on dynamic changes when answering questions. This could also partially explain the no significant difference in describing processes between the two groups. This finding indicates that we should closely align the assessment of learning performances and the analysis of navigation patterns in order to make meaningful connections between these two.

Discussion and Implications

In this study, we examined student learning with a mobile AR technology in science laboratory settings. We first employed cluster analysis to categorize students based on log data. The log data captured sequences of representations that students interacted with. We identified two groups: group 1 interacted with fewer representations and had more even distribution of movements between representations than group 2. Afterward, we coded lab reports to assess learning performance and compared learning performance of those two groups. This work contributes to the understanding of different ways that students interacted with multiple representations in AR technology and the relationship between navigation patterns and learning performance.

Results of independent samples t-tests revealed insignificant difference between two groups’ performance in describing processes, but a significant difference in giving rationale. This finding indicates that frequent movements between representations might help students to reason why temperature changed in the experiments from scientific perspectives. Our further analysis of nuanced interactions (i.e., navigations within representations) showed that some students had in-depth exploration of certain representation, which the sequences of presentations could not capture. These students’ knowledge acquisition of describing processes might be related to nuanced interactions, instead of sequences of interactions with different representations. In addition, the score of describing processes was low as the questions did require students to explain dynamic changes in the experiments. This could also explain the insignificant difference in this dimension. Previous studies find conflicting results in the relationship between navigation patterns, generated from log data, and learning performance (Baker et al., 2020). For instance, Crossley et al. (2020) argued that click-stream patterns generated from interaction data in a game-based learning platform could predict math identity. In contrast, Yu, Pardos, and Scott (2019) examined the correlation between event-level course behaviors and course grades in online learning environments and found that student interactivity measures failed to predict grades. The conflicting results could be attributed to different ways of representing log data. This study stressed the need of representing fine-grained log data in ways that are meaningful for explaining learning performance when investigating the relationship between these two types of data sources.

This study provides a new understanding into students’ interaction with multiple representations in AR-based learning environments. As described by Zimmerman et al. (2016), “AR mobile technology supports interest-driven interactions between learners and the nature center setting (p. 103).” They argue that offering students the flexibility of selecting points of interest to interact with could contribute to active learning. In addition to designing flexible interactions, the literature highlighted that we should provide multiple representations to facilitate students to learn disciplinary knowledge through the interaction. This study contributes to the growing interest among researchers in designing interactions and multiple representations by demonstrating the patterns of students’ interaction with multiple representations. The patterns could help us to understand different ways that students used representations in science experiments and guide us to design scaffolds based on these patterns. However, the patterns we identified are closely related to the designed activity and the affordances of the technology, future research is needed to reveal students’ navigation patterns in other contexts and with different kinds of AR technologies.

Moreover, this study presents learning opportunities made available through different ways of interacting with multiple representations. Specifically, we found that iterative video interaction, iterative video-physical-world interaction, and iterative video-data-graph interaction might support understanding dynamic changes in scientific process, collecting evidence, and reasoning about science phenomena respectively. This finding supports the current understanding of the importance of presenting multiple representations to engage students in various science practices, such as planning and carrying out scientific investigations. In this study, students gained scientific perspectives of the physical world through observing, exploring, and analyzing the thermal view over time. The observation, exploration, and analysis are made available through multiple representations and the connections between representations. For instance, students can add thermometers on the thermal view and the graph presents temperature changes for the added thermometers. Connecting video, object in the physical world, and data graph in such way had the potential of supporting reasoning about science phenomena. These findings have implications for providing multiple representations and designing connections between representations for meaningful science learning.

In accordance with the literature, this study shows that AR promoted effective science learning. Most studies investigated the affective outcomes in AR-based science learning environments, including satisfaction, motivation, enjoyment, attitude, and engagement (e.g., Akçayır et al., 2016). There is a lack of understanding as to science learning with AR technologies in laboratory settings. This study presented new applications of a mobile AR technology that students utilized as an inquiry tool to conduct experiments and learn science concepts. In addition, the majority of studies focused on low-level cognitive outcomes, such as remembering facts and content (e.g., Cai et al., 2017) while limited research has been done examining high-level cognitive outcomes that demonstrate more complex cognitive processes involving creating, applying, analyzing, and evaluating information. Our study fills this gap and shows that AR afforded positive high-level cognitive outcomes, the ability of describing dynamic changes in scientific processes and in particular, reasoning about science phenomena. These two types of abilities have been stressed as critical but challenging cognitive skills in science experiments. It is important to continue an exploration of other high-level cognitive outcomes, such as interpreting data and drawing insightful conclusions. For future studies, it is also worth linking detailed responses from lab reports (in this study, we used aggregated scores in two dimensions) with navigation patterns to explain students' intention during scientific investigation and inquiry.

These findings support the current understanding of the positive effect of high-degree interactivity in science labs (e.g., Washington et al., 2019). The literature pointed out the high-degree interactivity improved the level of engagement. As highlighted in the literature, one common type of interaction was changing the configuration of a few parameters. Contributing to this line of research, this study illuminates that augmented interaction with physical objects (e.g., adding thermometers) had the potential of engaging students in free exploration of science phenomena and facilitate students in gaining different perspectives to evaluate scientific claims. In addition to interacting with laboratory settings in the physical world by adding thermometers, students used the AR application to analyze data in real-time. Both interactions had their own unique affordances for science learning. In this study, we focused on interactions with different representations. However, navigations within the same representation (e.g., taking a screenshot of an image and rotating an image) might provide different learning opportunities. More efforts should be devoted to this area. These findings have implications for designing different kinds of interactions for providing authentic science practices in science labs.

In conclusion, this study presents an example of how AR technologies can be easily used for conducting hands-on science experiments. Furthermore, the analysis of log data, interviews, and lab reports demonstrated that the technology held the promise of supporting engaging and effective science learning. However, more research is needed to extend and build on our findings about designing multiple representations and different types of interactions in mobile AR technologies for high school science labs.

Limitation

One limitation is the short duration of intervention time and only 31 students were involved in the interview due to time and resource constraints. The other limitation is that our analysis did not consider students’ age, gender, language proficiency, and ethnic backgrounds as it’s not directly related to the research questions. An in-depth study is needed to investigate how these variables might influence learning performance. Furthermore, this study investigates the learning effect of a particular type of AR technology, which affords limited types of interactions and representations. It would be beneficial to study student learning with other types of AR technologies in various settings. This study opens the door to needed conversations about supporting hands-on science experiments with mobile AR technologies in high school classrooms.

Footnotes

Data Accessibility Statements

The dataset generated and analyzed during the current study are not publicly available but are available from the author upon request.

Ethical Approval

All consent processes and forms for this study were approved by the Solutions Institutional Review Board (IRB) () prior to the study’s implementation. In addition, the analysis was performed using non-identifiable data.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Science Foundation under Grant #1712676.

ORCID iDs

Shiyan Jiang

Cansu Tatar

Author Biographies

Shiyan Jiang received her PhD in technology-enhanced STEM education from the University of Miami in 2018. She was a postdoc at Carnegie Mellon University and worked in designing an innovative writing feedback system. Currently, she is an assistant professor of Learning Design and Technology at North Carolina State University. She designs and studies technology-enhanced learning environments to facilitate the development of science identities.

Cansu Tatar is a doctoral student in the program of Learning Design and Technology at North Carolina State University. Her research focuses on designing innovative technology learning experiences for middle and high school students. She is also interested in mining educational data using learning analytics.

Xudong Huang received her PhD in Cognitive Science at the Institute for Intelligent Systems at the University of Memphis in 2016. Currently, she is a postdoctoral researcher with a focus on learning of engineering using computer-aided design tools such as Energy3D and science using AR tools such as Infrared Explorer. She has five years of experience evaluating the efficacy of intelligent systems on students’ learning of STEM majors in real-world scenarios.

Shannon H. Sung received her PhD in science education from the University of Georgia in 2014. She was a former professor at Spelman College. While at Spelman, she served as the PI or Co-PI of two NSF-funded projects and worked on an HMMI project for promoting interdisciplinary learning of science. Currently, she is a Learning Scientist at Institute for Future Intelligence (IFI). Her research focuses on the technology-enhanced learning environment, interdisciplinary STEM learning, and the assessment of cognitive learning processes.

Charles Xie holds a PhD in computational materials science from the University of Science and Technology Beijing. He is the founder of IFI. Before staring IFI, he was a scientist at Concord Consortium, MA, USA. His current research includes computational science, learning science, data mining, machine learning, artificial intelligence, computer‐aided design, scientific visualization, virtual reality, mixed reality, Internet of Things, molecular simulation, solar energy engineering, and infrared imaging. He has been developing and co‐developing Infrared Explorer, Energy2D, Energy3D, Molecular Workbench, and Quantum Workbench that have been used by countless students around the world.

References

Abdinejad

Talaie

Qorbani

H. S.

Dalili

(2021). Student perceptions using augmented reality and 3d visualization technologies in chemistry education. Journal of Science Education and Technology, 30(1), 87–96.

Akçayır

(2017). Advantages and challenges associated with augmented reality for education: A systematic review of the literature. Educational Research Review, 20, 1–11.

Akçayır

Pektaş

H. M.

Ocak

M. A.

(2016). Augmented reality in science laboratories: The effects of augmented reality on university students’ laboratory skills and attitudes toward science laboratories. Computers in Human Behavior, 57, 334–342.

Altmeyer

Kapp

Thees

Malone

Kuhn

Brünken

(2020). The use of augmented reality to foster conceptual knowledge acquisition in STEM laboratory courses—Theoretical background and empirical results. British Journal of Educational Technology, 51(3), 611–628.

Andrienko

(2013). Visual analytics of movement: An overview of methods, tools and procedures. Information Visualization, 12(1), 3–24.

Azuma

R. T.

(1997). A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 6(4), 355–385.

Baker

Park

Cung

Fischer

Rodriguez

Warschauer

Smyth

(2020). The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: Opening the black box of learning processes. International Journal of Educational Technology in Higher Education, 17(1), 1–24.

Bekal

Brousseau

Masson

Prefontaine

Fairbrother

Harel

(2003). Rapid identification of Escherichia coli pathotypes by virulence gene detection with DNA microarrays. Journal of Clinical Microbiology, 41(5), 2113–2125.

Blum

Kleeberger

Bichlmeier

Navab

(2012, March). Mirracle: An augmented reality magic mirror system for anatomy education. In 2012 IEEE virtual reality workshops (VRW) (pp. 115–116). IEEE.

10.

Cai

Chiang

F. K.

Sun

Lin

Lee

J. J.

(2017). Applications of augmented reality-based natural interactive learning in magnetic field instruction. Interactive Learning Environments, 25(6), 778–791.

11.

Chiang

T. H.

Yang

S. J.

Hwang

G. J.

(2014). An augmented reality-based mobile learning system to improve students’ learning achievements and motivations in natural science inquiry activities. Journal of Educational Technology & Society, 17(4), 352–365.

12.

Chiu

J. L.

DeJaegher

C. J.

Chao

(2015). The effects of augmented virtual science laboratories on middle school students’ understanding of gas properties. Computers & Education, 85, 59–73.

13.

Coştu

Ayas

Niaz

(2012). Investigating the effectiveness of a POE-based teaching activity on students’ understanding of condensation. Instructional Science, 40(1), 47–67.

14.

Crossley, S. A., Karumbaiah, S., Ocumpaugh, J., Labrum, M. J., & Baker, R. S. (2020). Predicting math identity through language and click-stream patterns in a blended learning mathematics program for elementary students. Journal of Learning Analytics, 7(1), 19–37.

15.

Ebenezer

J. V.

Erickson

G. L.

(1996). Chemistry students' conceptions of solubility: A phenomenography. Science Education, 80(2), 181–201.

16.

Fidan

Tuncel

(2019). Integrating augmented reality into problem based learning: The effects on learning achievement and attitude in physics education. Computers & Education, 142, 103635.

17.

Garribba

Micera

Panzanelli

Strinna-Erre

Stara

(2001). Distinguishing calcium carbonate from calcium sulfate dihydrate by instrumental methods. A set of laboratory experiments for analytical chemistry and spectroscopy. Journal of Chemical Education, 78(8), 1090.

18.

Ibáñez

M. B.

Delgado-Kloos

(2018). Augmented reality for STEM learning: A systematic review. Computers & Education, 123, 109–123.

19.

Ibáñez

M. B.

Di Serio

Á.

Villarán

Kloos

C. D.

(2014). Experimenting with electromagnetism using augmented reality: Impact on flow student experience and educational effectiveness. Computers & Education, 71, 1–13.

20.

Jiang

Huang

Xie

Sung

Yalcinkaya

(2020, June). Augmented scientific investigation support the exploration of invisible “fine details” in science via augmented reality. In Proceedings of the interaction design and children conference (pp. 349–354).

21.

Kamarainen

A. M.

Metcalf

Grotzer

Browne

Mazzuca

Tutwiler

M. S.

Dede

(2013). EcoMOBILE: Integrating augmented reality and probeware with environmental education field trips. Computers & Education, 68, 545–556.

22.

Kaufmann

Dünser

(2007, July). Summary of usability evaluations of an educational augmented reality application. In International conference on virtual reality (pp. 660–669). Springer.

23.

Kourouthanassis

P. E.

Boletsis

Lekakos

(2015). Demystifying the design of mobile augmented reality applications. Multimedia Tools and Applications, 74(3), 1045–1066.

24.

Lee

McAlister

Ehlert

Faber

Kajfez

Creamer

Kennedy

(2019, October). Enhancing research quality through analytical memo writing in a mixed methods grounded theory study implemented by a multi-institution research team. In 2019 IEEE frontiers in education conference (FIE) (pp. 1–7). IEEE.

25.

L. Y.

Tsai

C. C.

(2017). Accessing online learning material: Quantitative behavior patterns and their effects on motivation and learning performance. Computers & Education, 114, 286–297.

26.

Lindgren

Tscholl

Wang

Johnson

(2016). Enhancing learning and engagement through embodied interaction within a mixed reality simulation. Computers & Education, 95, 174–187.

27.

Martin

Diaz

Sancristobal

Gil

Castro

Peire

(2011). New technology trends in education: Seven years of forecasts and convergence. Computers & Education, 57(3), 1893–1906.

28.

Mayer, R. E. (2014). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 43–17). Cambridge University Press.

29.

Mayer

R. E.

Moreno

(2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43–52.

30.

Milgram

Takemura

Utsumi

Kishino

(1995, December). Augmented reality: A class of displays on the reality-virtuality continuum. In Telemanipulator and telepresence technologies (Vol. 2351, pp. 282–292). International Society for Optics and Photonics.

31.

Moreno

(2006). Does the modality principle hold for different media? A test of the method-affects-learning hypothesis. Journal of Computer Assisted Learning, 22(3), 149–158. https://doi.org/10.1111/j.1365-2729.2006.00170.x

32.

Moro

Phelps

Redmond

Stromberga

(2021). HoloLens and mobile augmented reality in medical and health science education: A randomised controlled trial. British Journal of Educational Technology, 52(2), 680–694.

33.

Patton

M. Q.

(1990). Qualitative evaluation and research methods (2nd ed). Sage.

34.

Radu

(2014). Augmented reality in education: A meta-review and cross-media analysis. Personal and Ubiquitous Computing, 18(6), 1533–1543.

35.

Ruiz‐Primo

M. A.

Shavelson

R. J.

(1996). Rhetoric and reality in science performance assessments: An update. Journal of Research in Science Teaching, 33(10), 1045–1063.

36.

Shibley

I. A.

Jr ,., & Zimmaro

D. M.

(2002). The influence of collaborative learning on student attitudes and performance in an introductory chemistry laboratory. Journal of Chemical Education, 79(6), 745.

37.

Skulmowski

Rey

G. D.

(2018). Embodied learning: Introducing a taxonomy based on bodily engagement and task integration. Cognitive Research: Principles and Implications, 3(1), 6.

38.

Sotiriou, S., Anastopoulou, S., Rosenfeld, S., Aharoni, O., Hofstein, A., Bogner, F., Sturm, H., Hoeksema, K. (2006, July). Visualizing the invisible: The CONNECT approach for teaching science. In Sixth IEEE international conference on advanced learning technologies (ICALT'06) (pp. 1084–1086). IEEE.

39.

Spradley

J. P.

(1980). Participant observation. Holt, Reinhart & Winston.

40.

Sung

S. H.

Chen

Huang

Xie

Massicotte

Shen

(2021). How does augmented observation facilitate multimodal representational thinking? Applying deep learning to decode complex student construct. Journal of Science Education and Technology, 30(2), 210–226.

41.

Tarng

K. L.

(2012, March). A study of campus butterfly ecology learning system based on augmented reality and mobile learning. In 2012 IEEE seventh international conference on wireless, mobile and ubiquitous technology in education (pp. 62–66). IEEE.

42.

Turan

Atila

(2021). Augmented reality technology in science education for students with specific learning difficulties: Its effect on students’ learning and views. Research in Science & Technological Education. Advance online publication. https://doi.org/10.1080/02635143.2021.1901682.

43.

Washington

Samuel-Gama

K. G.

Goyal

Ramaswami

Riedel-Kruse

I. H.

(2019). Interactive programming paradigm for real-time experimentation with remote living matter. Proceedings of the National Academy of Sciences, 116(12), 5411–5419.

44.

Wilson

(2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9(4), 625–636.

45.

Xie

(2011). Visualizing chemistry with infrared imaging. Journal of Chemical Education, 88(7), 881–885.

46.

Xie

Hazzard

(2011). Infrared imaging for inquiry-based learning. The Physics Teacher, 49(6), 368–372.

47.

Pardos

Z. A.

Scott

(2019). Student behavioral embeddings and their relationship to outcomes in a collaborative online course. In EDM (workshops) (pp. 23–29).

48.

Zimmerman

H. T.

Land

S. M.

Jung

Y. J.

(2016). Using augmented reality to support children’s situational interest and science learning during context-sensitive informal mobile learning. In A. Peña-Ayala (Ed.), Mobile, ubiquitous, and pervasive learning (pp. 101–119). Springer.