Patterns of Using Multimodal External Representations in Digital Game-Based Learning

Abstract

Although prior research has highlighted the significance of representations for mathematical learning, there is still a lack of research on how students use multimodal external representations (MERs) to solve mathematical tasks in digital game-based learning (DGBL) environments. This exploratory study was to examine the salient patterns problem solvers demonstrated using MERs when they engaged in a single-player, three-dimensional architecture game that requires the acquisition and application of math knowledge and thinking in game-based context problem solving. We recorded and systematically coded the behaviors of using MERs demonstrated by 20 university students during 1.5 hours of gameplay. We conducted both cluster and sequential analyses with a total of 2654 encoded behaviors. The study indicated that the maneuverable visual-spatial representation was most frequently used in the selected architecture game. All of the participants performed a high level of representational transformations, including both treatment and conversion transformations. However, compared to the students in the second cluster who were mostly non-game players, students in the first cluster (composed of mainly experienced video game players) displayed a higher frequency of interacting with various MERs and a more cautious and optimized reflective problem-solving process.

Keywords

multimodal external representation math problem solving game-based learning cluster analysis sequential analysis

Research on mathematics education suggests that students gain better performance by practicing problem solving in a real-world context (Kilpatrick, 2014). Unfortunately, prior research indicated that many students are not able to solve mathematical problems embedded in contexts because they lack sufficient learning opportunities to practice mathematical representations of the situation (Ke & Clark, 2020). Students are often taught how to solve well-structured mathematical problems that utilize standardized phrases and keywords in classrooms (Daroczy et al., 2015). Nevertheless, multiple studies reported that teaching students with standardized phrases and keywords often led to incorrect mathematical translation of problem situations (Daroczy et al., 2015; Griffin & Jitendra, 2009; Hegarty et al., 1995; Nesher & Teubal, 1975; Pongsakdi et al., 2020). Students, especially less successful problem solvers, tend to be attracted by verbal cues and then directly interpret texts into a mathematical representation without deep thinking of contextualized situations (Nesher & Teubal, 1975). Therefore, it is crucial for students to identify the meaning and structure of the problem to make sense of problem situations when they are involved in problem solving.

Recently, a growing body of empirical evidence suggests that digital game-based learning (DGBL) is a natural and dynamic learning platform that improves students’ mathematical performance, by creating a meaningful context-based learning environment with multiple representations of mathematical problems (Bullock et al., 2021; Ke, 2016; Ke & Clark, 2020; Moyer-Packenham et al., 2021; Siew, 2018). These external representations refer to the concrete and visual objects that represent abstract math knowledge in a simulated context. It is one of the most frequently used approach for designing the game objects that help students make meaning between abstract math concepts and concrete objects in mathematics education (Ke, 2016). More importantly, the platform of DGBL enables students to interact in a systematic way with large sets of multimodal external representations (MERs, e.g., verbal, visual-spatial, and notation, Ke & Clark, 2020), which are the embodiments of the mathematical concepts purposefully designed as the game objects in the game world. MERs are supposed to foster cognitive processing and encourage in-depth learning (Ainsworth, 1999), which in turn enhance students’ abilities to construct mathematical representations of the situation (Adu-Gyamfi et al., 2019; Goldin, 2003, 2014).

Prior research reported that interacting with purposefully designed MERs in DGBL could improve students’ performance of mathematical problem solving (Bullock et al., 2021; Ke, 2019; Moyer-Packenham et al., 2019), performance of mental rotation (Ke, 2019; Ke & Clark, 2020), learning engagement (Moon & Ke, 2020), and mathematics connections (Moyer-Packenham et al., 2019). Thus, we speculate that there is a positive association between the use of MERs in DGBL and students’ mathematical learning outcomes (Moyer-Packenham et al., 2021). However, most of these studies used conventional learning performance assessment or self-report questionnaires, studies that investigate learning process through exploring learners’ behavioral patterns of interactions with various MERs in DGBL for mathematical learning remain inadequate. In-depth analyses of the students’ behavioral patterns should yield more comprehensive results that provide guidance for educators, researchers, and game designers to better understand students’ learning process of interacting with MERs in the context of DGBL. Therefore, in the current study we investigated of how problem solvers interact with MERs in DGBL by analyzing their in-game actions. We implemented the data mining technique, including both cluster and sequential analyses, to obtain a deeper understanding of the learning process by visualizing learners’ behavioral pattens of interactions with various MERs in DGBL. The overarching research question guides the current study is as follows: What are learners’ behavioral patterns of interacting with MERs in the context of DGBL? Specifically, the study addresses the following two questions:

1. What are the cluster patterns of using various MERs demonstrated by participants in DGBL?

2. What are the sequential behavior patterns of using MERs demonstrated by participants in each cluster?

Literature Review

External Representations for Mathematical Learning

There is no universally accepted definition for external representations for mathematical learning, but most researchers acknowledge that mathematical external representations refer to the physical (and therefore observable) embodiments of the abstract math ideas, concepts, and procedures that conceptualize students’ internal structure (Ainsworth, 1999; Lesh, 1981). According to Lesh’s (1981) conceptual model of the representation system, external representations can be categorized into real-world situations, manipulatives, pictures, spoken symbols, and written symbols with respect to their different aspects of the structure of the concept.

Numerous researchers have reported or claimed the paramount importance of external representations on the development of mathematical thinking (Duval, 2006; 2017; Goldin, 2003; Hurst et al., 2020; Lesh et al., 1987). Mathematics itself is a special and artificial language that heavily relies on signs and numeral expressions. For instance, the number of 1 can represent the quantity of a set, such as one apple; it can also represent a position on a number line. Compared to other scientific disciplines, mathematical phenomena cannot be directly perceived and observed (Duval, 2006). Therefore, the representations are used not only to stand for mathematical objects, but also to communicate the meaning of the problem and provide the possibility of working on mathematical objects. In other words, the employment of external representations builds upon the foundation of performing mathematical processing (Duval, 2006).

In Duval’s Theory of Registers of Semiotic Representation (TRSR) (Duval, 2006, 2017), the external representations are considered as “representation registers” (Duval, 2006). Students will make a variety of register transformations, which include either treatments or conversions, when they are engaged in solving mathematical problems. Treatment refers to a transformation within the same register, but conversion happens when changing one representation register to another (Duval, 2006, 2017). Both transformations play vital roles in developing conceptual understanding of mathematics (Duval, 2006, 2017; Hurst et al., 2020; Lesh et al., 1987; Moyer-Packenham et al., 2019; 2021).

Conversion is generally considered as more complex than treatment (Lesh et al., 1987; Moyer-Packenham et al., 2021; Pino-Fan et al., 2017), because it is built upon the recognition of the representation registers and extends the “recognition of the same represented object between two representations whose contents have very often nothing in common” (Duval, 2006). For example, Figures 1 and 2 are two examples of word problem solving of ratios and proportions. The former question primarily requires students to make “treatment” transformations, while the latter one involves the application of “conversion” transformations. Specifically, to answer the question in Figure 1, students are requested to perform computations within the same representation system (i.e., written symbols/formal mathematical notation). To solve the question in Figure 2, students must first recognize the representation system of the written symbol (i.e., formal mathematical notation), then make a conversion transformation from a written symbol to a representation register of static pictures. According to the theory of TRSR (Duval, 2006, 2017), students who are able to flexibly make more transformations across different registers (i.e., conversions) have a greater understanding of the mathematical concepts than those who make fewer conversion transformations (Calder et al., 2018; Pino-Fan et al., 2017). Empirical research also supports this proposition. For example, Brenner et al. (1997) found that middle school students who participated in a training program for representation performed significantly better in the posttest than those without practice in representational transformations. Kellman et al. (2008) reported that students who were given opportunities to practice their transformation competencies in a computer-supported system obtained significant higher learning gains than those who did not receive any instruction. Sidney et al. (2019) explored the effects of registers of static pictures (number lines and area models) on middle school students’ accuracy and conceptual understandings of fraction division and found that students in the group of number lines outperformed those without any picture registers. However, researchers argued that some “treatment” transformations, particularly those corresponding to symbolic registers (i.e., formal mathematical notations to formal mathematical notations), are as complex as “conversion” transformations (Pino-Fan et al., 2017).

Figure 1.

An example of a word problem with ratios and proportions that requires treatment transformations.

Figure 2.

An example of a word problem with ratios and proportions that requires conversion transformations.

Overall, prior research (Ainsworth, 1999; De Bock et al., 2017; Duval, 2006; Goldin, 2003; Hurst et al., 2020; Lesh, 1981; Lesh et al., 1987) revealed the importance of external representations for mathematical learning, but most of these previous studies focused on a conventional learning setting. Recently, research on the field of external representations for mathematical problem solving in DGBL is increasing (Bullock et al., 2021; Duval, 2017; Hung et al., 2014; Moyer-Packenham et al., 2019; 2021), but still scarce.

Multimodal External Representations in DGBL for Mathematical Problem Solving

DGBL could present a multimodal environment by embedding dynamic and interactive 2D or 3D external representations (e.g., graphics and “physical objects”). According to the theory of situated learning, the processes of cognition “extend to external states and structures (or representations)” (Ke & Clark, 2020, p. 105), students will form new insights when they are involved in interacting with external representations that are tentative to their internal cognitive processes. The platform of DGBL provides students with multimodal external representations that involve not only verbal associations but also visual-spatial and physical accounts (Ke & Clark, 2020). According to the characteristics of DGBL as well as the MER perspectives of Goldin (2003), MERs in DGBL can be categorized as verbal/syntactic task narratives, maneuverable visual-spatial representations, formal mathematical notations, and interactive mathematical tools (Ke & Clark, 2020). The current study adopted this category to explore how students interact with MERs in DGBL.

Prior research (Bullock et al., 2021; Ke & Clark, 2020; Moyer-Packenham et al., 2019; 2021) suggested that interactions with well-structured MERs in DGBL could enhance students’ abilities to construct a mental model for mathematical problem solving. Specifically, DGBL provides opportunities of translations (i.e., transformations) among different mathematical representations. The translations among different modes of representations (i.e., conversion transformation) provide opportunities for students to perceive the same mathematical concept from different perspectives, which then foster an enriched and deeper math understanding prior to constructing mathematical representations of the situation for problem solving (Adu-Gyamfi et al., 2019; Goldin, 2003, 2014). Recently, Ke and Clark (2020) reported that students’ abilities to translate math problem representations could be enhanced through the interactions with MERs in DGBL. Based on the data collected from 46 seventh graders, they found that the students who played E-Rebuild for 5 hours over six in-class sessions performed significantly better than the control group students in both mathematical problem solving and mental rotation test performance. Furthermore, Moyer-Packenham et al. (2021) examined 145 4–6 graders’ gameplay and pretest-posttest to investigate the relationships between students’ mathematical performance and their abilities of representational transformations. Their finding indicated that students who recognized and made more representational transformations among MERs during gameplay demonstrated better performance in the mathematical posttest than those with fewer transformations.

Overall, recent studies have revealed the positive effects of interactions with MERs in DGBL on students’ mathematical problem solving (Bullock et al., 2021; Ke, 2019; Moyer-Packenham et al., 2019; 2021), but few of them investigated how students interact with various MERs. In-depth analyses of how students interact with different MERs in DGBL for mathematical problem solving remain inadequate. Therefore, in the current study we aimed to conduct a naturalistic inquiry to facilitate a deep understanding of problem solvers’ interactions with MERs in DGBL.

Data Mining Techniques on Students’ In-Game Behaviors

Data mining techniques, including cluster and sequential analyses (Hou, 2015; Hsu & Cheng, 2021), have gained increasing attention in the research of technology-assisted, visualized learning processes, such as the investigation of interactions in threaded discussions in online courses (Jeong, 2003), and students’ problem-solving sequences and learning processes in DGBL environments (Hou, 2015; Hsu & Cheng, 2021; Tsai et al., 2016). Cluster analysis is one of the most frequently used educational data mining techniques (Castro et al., 2007), by which the data set that is implicit and previously unknown will be analyzed in a manageable number of variables (Vogt & Nagel, 1992). Cluster analysis in educational contexts frequently categorizes students based on the similarities and differences of their performance (Feldman et al., 2014). Sequential analysis, another data mining technique, is particularly useful for studying students’ interactions with the game world (Hwang et al., 2017; Sun et al., 2021). Students need to perform a series of actions to complete a game task. Such a game-based problem solving or learning process is normally complex and embeds special behavioral structures or patterns. These behavioral patterns will inform on students’ in-depth learning process and disclose hidden yet important issues of the design of DGBL (Bakeman & Gottman, 1997).

Multiple studies have applied cluster analysis and sequential analysis in investigating students’ behavioral patterns in DGBL (Hou, 2012; 2015; Hsu & Cheng, 2021; Hwang et al., 2017; Lin et al., 2015; Moon & Ke, 2020). For example, Hsu and Cheng (2021) employed both cluster and sequential analyses to find that students in the cluster of immersion experiences demonstrated in-game behavioral patterns of more heuristic and analogical thinking strategies than those in the cluster without immersion experiences. Hwang et al. (2017) used a progressive sequential analysis to explore how anxiety affected students’ language learning behaviors in a puzzle game. Hou (2015) used cluster and sequential analyses to examine the relation between the experience of flow and behavioral patterns in game-based science education.

Although different data mining techniques have been employed to explore students’ behavioral patterns of game-based learning, research incorporating both cluster and sequential analyses to investigate participants’ patterns of interactions with MERs in DGBL is still scarce. In this study, we conducted both cluster analysis and sequential analysis to analyze students’ gameplay behaviors to explore the cognitive operations they perform when interacting with various MERs during gameplay.

Methods

Research Design

In this case study, we explored learners’ behavioral patterns of interacting with different MERs in DGBL. Data mining techniques, including cluster and sequential analyses (Bakeman & Gottman, 1997; Vogt & Nagel, 1992), were used to analyze the encoded behavioral data governing learner-MER interactions during game-based mathematical problem solving.

Participants

We recruited 25 college students from a university in the Southeastern United States. Five students did not complete the study session, so their video recordings were excluded from the final data analysis. Across all the 20 participants, 40% were male, 45% were Caucasian, 40% were African American, and 15% were Latino. Participants’ average age was 20.3 (SD = 2.4), and their grade levels ranged from freshman to senior. There were 45% identified as occasional gamers or gamers and 55% considered as non-gamers.

Game-Based Learning Platform: E-Rebuild

The study employed a 3D architectural simulation game E-Rebuild. The game aims to foster students’ performance of mathematical problem solving and conceptual understanding in the context of reconstructing a disaster-damaged space to fulfill diverse design parameters or needs (Ke & Clark, 2020). Prior research reported that students’ math problem-solving performance was enhanced after the interactions with structured MERs embedded in E-Rebuild (Ke, 2019; Ke & Clark, 2020).

In E-Rebuild, students need to solve contextualized math problems by interacting with multiple MERs that are embodiments of math problem parameters. To conceptualize and solve mathematical context problems, students have to collect, organize, and interpret distributed information that is purposefully inscribed onto MERs situated in the game world. The game encompasses four types of MERs (Ke & Clark, 2020), as illustrated in Figure 3: (1) verbal/syntactic task narratives; (2) maneuverable visual-spatial representations; (3) formal mathematical notations; and (4) interactive mathematical tools. The detailed descriptions and examples of each MER are presented in Table 1.

Figure 3.

Screen shots of four types of multimodal external representations in E-Rebuild.

Table 1.

Coding Scheme of Learning Behaviors in Relation to Using MER.

Code	Category	Description	Modifiers (sub-codes)	Examples
V	Use verbal/syntactic task narratives	Interactions with written text words stating the situation or task within the game that students used or recognized during game play.	1. Task narrative	Read task narrative (i.e., the contextualized math problems and game objectives for players).
			2. Object-oriented narrative
			3. Syntactic cue
			4. Information on help panel
M	Use maneuverable visual-spatial representations	Interactions with interactive 2D/3D game objects that encode the semantic relations embedded in a problem.	1. 2D game objects	Explore the spatial configuration of the landscape designating the size of the shelter to be built.
			2. 3D game objects
			3. Environmental landscape
N	Use formal mathematical notations	Interactions with mathematical symbols, numerals, and operations students referenced during gameplay.	1. Object-oriented notation	Retrieve the numerals inscribed onto different size of families to answer the question.
N	Use formal mathematical notations		2. Object-oriented operation
T	Use interactive mathematical tools	Interactions with tools that students used to explore the distributed mathematical information embedded in the objects and game world.	1. Appropriate use of a measuring tool	Use a measuring tool to measure distance or angle.
			2. Inappropriate use of a measuring tool
			3. Appropriate use of a copying tool
			4. Inappropriate use of a copying tool
S	Successful attempt	Successfully accomplishing a mission within a game level.	N/A	Collect a scattered container by correctly answering attached question.
F	Failed attempt	Unsuccessfully accomplishing a mission within a game level.	N/A	Pay incorrect amount of money to buy a door.

Coding Scheme of Behaviors

We conducted a systematic, structural coding for behavior analyses with the recorded gameplay actions of the participants via BORIS, an analytic tool designed for professional video analysis (Friard & Gamba, 2016). First, two trained researchers conducted independent coding by using an initial coding scheme (based on Ke and Clark’s category (2020) of MERs in DGBL) to systematically label and index a set of randomly picked video recordings of participants’ gaming sessions. The coding was aimed to specify specific events or game actions associated with each of the four types of MERs. After the initial coding, peer debriefing was held to resolve the disagreements, the examples and definitions of the coding categories were updated, and corresponding sub-codes (modifiers) were modified. In the second stage, researchers coded 20% of the video recordings independently with the refined coding scheme. Since the reliability of the multi-rater Fleiss Kappa coefficient κ was 0.96, the researchers moved on to the third stage—coding an equal part of the remaining recordings separately. During the process of structural coding, the coding scheme was constantly refined as the synthesis proceeded and meanwhile, a cross-case pattern analysis of the individual cases was conducted (Miles & Huberman, 1994). Minor revisions were made after peer debriefing. All behavior codes with corresponding sub-codes for the analyses are displayed in Table 1.

In Table 1, six coding categories, depicting the four types of MERs (i.e., verbal/syntactic task narratives, maneuverable visual-spatial representations, formal mathematical notations, and interactive mathematical tools) as well as the successful and failed attempts of game tasks, were presented with the supportive examples and sub-codes (modifiers).

Procedure

The participants signed the consent form before participating in the study. To collect thorough information with respect to the participants’ gameplay behaviors and observe their interactions with a variety of MERs during the game-based problem solving process, we scheduled a one-on-one study session for each participant. On the scheduled observation day, each participant played E-Rebuild for 1.5 hours. At the beginning of the gameplay session, the researcher gave a very brief introduction of the gameplay. The first five participants’ individual gameplay sessions were held face to face in a study room on campus. The remaining 15 participants attended one-on-one sessions through Zoom due to the Covid-19 pandemic situation. Participants’ gameplay actions and conversations were screen recorded.

Data Analyses

We conducted a cluster analysis with the frequency data of the four types of MERs as well as the successful and failed attempts (see Table 1). A two-step clustering procedure was used to analyze the encoded data (Rovniak et al., 2010). First, an agglomerative hierarchical cluster analysis was performed to determine the initial cluster groupings and cluster centers. Ward’s minimum variance method and Squared Euclidean distance were used to form the clusters (Ward, 1963). The number of clusters was selected based on the information shown in the dendrogram and agglomeration schedules. In the second step, the cluster means (centroids) from the previous hierarchical cluster analysis were used as initial seed points in a non-hierarchical, k-means cluster analysis to refine the initial cluster solution (Hair & Black, 2000).

Afterward, each participant’s behavior codes were chronologically arranged so that we can explore participants’ patterns of using MERs in DGBL by conducting sequential analysis. For example, after reading the narrative of the game task (V), a participant headed to the store and traded 2D objects (M) that involved the act of processing formal mathematical notations (N), before acquiring and maneuvering these objects (or building materials) for the following construction task. In this case, the visualized sequence was noted as V→ M → N.

We then conducted a series of frequency transition matrices, including transfer matrix of behavioral frequency, conditional probability matrix, and expected value matrix to determine the sequential behavioral patterns of each cluster (Bakeman & Gottman, 1997). The transfer matrix of behavioral frequency between each behavioral code was used to form a matrix of the frequencies of transfer between each code. The conditional probability was calculated based on the conditional probability between each code from the aforementioned frequency transfer matrix. Subsequently, the matrix of the expected value of sequence transfer between each code was calculated with respect to the aforementioned two matrices. We inferred the z-score value based on the three matrices to examine the significance of each sequence, by testing whether the continuity of each sequence reaches the significant level. Finally, we drew sequential diagrams of behavioral transition with the significant sequences for each cluster to reveal the sequential correlations associated with each behavior.

Results

Cluster Patterns of Using Various MERs

The results of the hierarchical cluster analysis indicated that grouping all the participants into two clusters was the most reasonable and accurate solution. The follow-up k-means cluster analysis revealed the members in each cluster. Table 2 shows the results of cluster analysis of the 2654 coded behaviors with respect to the frequency of using or interacting with each MER.

Table 2.

Cluster Analysis of the Participants’ Behaviors of Using Each MER.

	Clusters		F
Indicators of cluster analysis	Cluster 1 (n = 12, 60%)	Cluster 2 (n = 8, 40%)
V	415	120	0.24
M	730	172	0.14
N	569	152	0.35
T	92	20	8.12^*
F	60	15	4.78^*
S	238	71	1.22
Background of each participant
Percentage of students with gameplay experience	83%	13%
Percentage of students who completed tasks	80%	35%

^*p < 0.05.

The two clusters comprised 12 and 8 participants, respectively, accounting for 60% and 40% of the total participants. In general, participants in Cluster 1 displayed higher frequency of using MERs as well as of successful and failed attempts (V, M, N, T, F, S) than those in Cluster 2 (see Figure 4). Maneuverable visual-spatial representations were the most frequent type of MERs used when participants were involved in solving game-based mathematical problems, followed by formal mathematical notations. The results of the F-test of equal variance of two clusters indicated that the variances on the frequencies of the two coded behaviors—using interactive mathematical tools and failed attempt (T and F)—achieved the level of significance, but the rest of the coded behaviors (V, M, N, S) showed little difference.

Figure 4.

Frequency of using different multimodal external representations for each cluster. Note. V represents using verbal/syntactic task narratives; M represents using maneuverable visual-spatial representations; N represents using formal mathematical notations; T represents using interactive mathematical tools; S represents a successful attempt; F represents a failed attempt.

We further examined participants’ prior gaming experience in relation to the game tasks completed. Since the current study targeted participants with sufficient and advanced prior knowledge of mathematics relevant to the problem solving in E-Rebuild, participants’ prior gaming experience could be a moderator that might impact participants’ game task performance. Table 2 shows that the percentage of the participants with prior gaming experience in Cluster 1 (83%) was significantly higher than that in Cluster 2 (13%). Correspondingly, participants in Cluster 1 (80%) completed more game tasks/levels than those in Cluster 2 (35%). These findings suggest that gaming experience may play a salient role in game-based learning; experienced gaming students are likely to collect more task-relevant information, execute better problem solutions or in a more efficient way.

Behavioral Sequential Analysis

We conducted a sequential analysis with the six behavior codes for each cluster of participants and visualized each behavioral sequence in Figure 5. The adjusted residuals of the participants’ behavioral transition for each cluster are shown in Table 3. The rows of the table display participants’ initial behaviors, and the columns represent the subsequent behaviors. A z-score greater than 1.96 indicates that a sequential behavior in the cluster reaches a level of statistical significance (p < 0.05) (Bakeman & Gottman, 1997). As shown in Table 3, there are 11 and 9 significant sequences occurring in the two clusters, respectively. To further clarify the similarities and differences of the salient behavior sequences between the two clusters, two diagrams of behavioral transition (Figures 6 and 7) were presented to portray these behavior sequences.

Figure 5.

Visualization of each behavioral sequence for two clusters based on adjusted residuals. Note. V represents using verbal/syntactic task narratives; M represents using maneuverable visual-spatial representations; N represents using formal mathematical notations; T represents using interactive mathematical tools; S represents a successful attempt; F represents a failed attempt.

Table 3.

The Adjusted Residual Table for the Behaviors of Using MER in Each Cluster.

	V	M	N	S	F	T
Cluster 1
V	−2.86	6.44^*	1.56	−5.19	−3.24	−2.19
M	5.56^*	−0.39	−5.16	−2.29	−2.17	6.71^*
N	−8.88	−5.45	2.41^*	13.40^*	8.77^*	−2.86
S	8.30^*	5.21^*	−7.24	−5.58	−2.78	−1.44
F	2.22^*	−2.93	4.14^*	−2.79	−0.55	−1.67
T	−2.63	−5.39	11.50^*	−2.84	−1.69	−1.58
Cluster 2
V	−3.27	6.21^*	0.54	−4.21	−1.46	−0.78
M	6.26^*	−0.51	−4.36	−0.69	−2.68	1.79
N	−4.07	−7.39	3.34^*	8.30^*	5.15^*	−0.29
S	2.71^*	5.66^*	−4.54	−3.37	−1.46	−1.70
F	−1.17	−1.89	4.60^*	−1.42	−0.62	−0.71
T	−2.33	−3.11	5.78^*	−1.77	0.62	1.53

^*p < 0.05.

Figure 6.

The behavioral transition diagram of cluster 1. ^*p < 0.05.

Figure 7.

The behavioral transition diagram of cluster 2. ^*p < 0.05.

We found that the participants in the two clusters demonstrated both similar and different behavioral patterns of using MERs in the architectural simulation game. Overall, the sequential patterns in Figures 6 and 7 show that both clusters had four sequential links, 2 bi-directional connections, and 1 self-circular sequence. Specifically, the two clusters had the same sequential links between the use of four types of MERs (V, M, T, N) and successful attempts (S) (i.e., T → N, N → S, S → V, S → M). It indicates that all participants performed a variety of conversion-oriented representation transformations to successfully tackle a math task in the architectural simulation game. Compared to Cluster 2, the z-scores associated with three sequential links (i.e., T → N, N → S, S → V) were much higher in Cluster 1, though the z-scores with the remaining one (i.e., S → M) were quite close. Meanwhile, there were bi-directional connections between the use of verbal task narratives (V) and the maneuverable visual-spatial representations (M), as well as the use of formal mathematical notation (N) and failed attempts (F) in both clusters (i.e., V → M, M → V, N → F, F → N). Similarly, the z-scores associated with the two sequential links (i.e., V → M, N → F) in Cluster 1 were higher than those in Cluster 2. In addition, Figures 6 and 7 show that there was a self-circular sequential pattern in the use of formal mathematical notation (i.e., N → N). Interestingly, the z-score associated with this sequential link in Cluster 1 is slightly lower than Cluster 2. It indicates that there were more behavioral sequences of “treatment” transformations when participants used the formal mathematical notation in Cluster 2.

Compared to Cluster 2, Cluster 1 portrayed additional behavioral sequences between the use of maneuverable visual-spatial representations (M) and that of interactive mathematical tools (T) (M → T) (z = 6.71, p < 0.05), and between failed attempts (F) and the use of verbal task narratives (V) (F → V) (z = 2.22, p < 0.05). It suggests that participants in Cluster 1 tended to better retrieve mathematical information inscribed onto the game objects, through the acts of applying interactive mathematical tools and interacting with the maneuverable visual-spatial representations. It may also indicate that participants in Cluster 1 engaged in more reflective problem-solving processes because they not only went back to check the information obtained from the formal mathematical notation, but also re-gauged the information retrieved from the verbal task narratives when they failed to solve the game-based math task. This pattern helped explain why participants in Cluster 1 portrayed more behavioral sequences of using formal mathematical notations in relation to successful attempts than those in Cluster 2.

Discussion

Discussion of Cluster Patterns

We found that participants tended to use maneuverable visual-spatial representations and formal mathematical notations more than verbal/syntactic task narratives and interactive mathematical tools for game-based math problem solving. This finding is similar to the research by Moyer-Packenham et al. (2021) who reported that the image (i.e., visual pictures or objects) was the most frequently used type of representations in the selected 9 digital math games for students in Grades 4, 5, and 6, followed by symbols (i.e., mathematical symbols and numerals). A potential explanation is that a maneuverable visual-spatial representation acts as “an advanced organizer that aids students in integrating and encoding problem information” (Ke & Clark, 2020). Another potential interpretation is that in the current study, manipulating 2D and 3D game objects and interacting with formal mathematical notation have become an essential part of solving contextualized mathematical problems. It is aligned with the proposition that there is a strong relationship between the design of the most prominent MERs in DGBL and students’ recognitions and the employment of such kind of MERs (Moyer-Packenham et al., 2021). However, it is important to note that more research is needed to further explore how each MER contributes to different learning outcomes.

Meanwhile, the differences in the cluster patterns might be moderated by participants’ gaming experience. Our findings indicate that there might be connections between students’ prior gaming experience, the interactions with MER, and their reflective problem-solving processes. Experienced video game players might solve game tasks more efficiently and strategically than non-players (Smith et al., 2020). The current behavior observations tend to support the previous findings (Bavelier et al., 2012; Smith et al., 2020) that prior gaming experience may cultivate the strategies and propensities that students could leverage to explore, process, and coordinate information in complicated and novel DGBL.

Discussion of Sequential Behavior Patterns

One of the salient findings of this study is that participants in both clusters exhibited similar sequential patterns. First, all the participants engaged in representational transformations, including both treatment and conversion transformations. It indicates that students in DGBL need to perform representational transformations with MERs during math problem solving. It is consistent with what Goldin (2014) suggested that the interactions with a variety of MERs in computer-based learning environments help students construct solutions for contextualized mathematical problem.

Second, participants in both clusters exhibited bi-directional conversion transformations. It indicates that verbal narratives and dynamic visual-spatial objects supplement each other to support mathematical meaning-making. In DGBL, a game-based task is a contextualized math problem, and information is distributed and situated within various objects in the game world. Students need to identify and integrate all the sufficient information to solve the problem. In the current study, verbal narratives aid participants in identifying and formulating initial strategies and solutions, whereas dynamic visual-spatial objects help participants integrate and encode problem-relevant information. Manipulating the virtual mathematical objects helps learners develop alternative insights in relation to the game-based task, thus helping them to adjust their problem-solving strategies and solutions. Participants then seek verbal information to support and verify their problem solutions. After multiple rounds of bi-directional, conversion-oriented representation transformations between verbal narratives and dynamic visual-spatial objects, participants solve the problem and potentially acquire better understanding of the mathematical concepts that are embodied in the game objects. Such a pattern is consistent with the previous study finding that the interactions with a variety of MERs (e.g., manipulating dynamic visual-spatial objects) in DGBL help students internalize conceptual understanding (Ke & Clark, 2020; Kirsh, 2010).

In addition, the current study findings showed that participants in both clusters exhibited bi-directional processes of reflection. The analyses showed that after receiving feedback for failed task attempts, participants would re-analyze, re-process, and re-coordinate the formal mathematical notations, and then retest the hypothesis. Such kind of reflective solution refinement iteratively repeated until the correct solutions were formulated. It supports the argument by Kiili (2007) that students engage in reflective learning processes in DGBL when they iteratively refine their problem-solving solutions.

However, it is worth noting that the frequency of the conversion-oriented representation transformations is higher than that of the treatment-oriented representation transformations in the current study. There was only one type of treatment-oriented representation transformations (i.e., formal mathematical notation to formal mathematical notation) shared by both clusters. This finding suggests that interacting with formal mathematical notation is an essential part of solving contextualized mathematical problems in the selected architecture game. The treatment-oriented representation transformation that uses the registers of the formal mathematical notations, as demonstrated by the current study, is “as cognitively challenging as some conversion transformations” (Moyer-Packenham et al., 2021). This finding is aligned with Pino-Fan et al. (2017) who argued that treatment-oriented representation within symbol registers (i.e., notation) is as complex as some conversion-oriented representation transformations.

Our behavioral analyses showed that participants in both clusters exhibited discrepant sequential behavior patterns. Specifically, the participants in Cluster 1 displayed a higher frequency in using various MERs and attempting game tasks, as well as a more cautious and optimized reflective problem-solving process, while the participants in Cluster 2 portrayed lower frequencies in the coded behavioral events and a relatively less reflective problem-solving process. This finding confirms that students who can freely make more transformations across two registers exhibit deeper mathematical conceptual understanding than those who make fewer conversion transformations (Calder et al., 2018; Pino-Fan et al., 2017).

Furthermore, our findings suggested that there exists a special type of transformations in DGBL: failed-attempt-mediated transformation. After receiving instant feedback for a failed attempt, participants would determine the subsequent refinement of their solutions by referring to the same or a different register. In the current study, failed-attempt-mediated conversion transformations seemed to better promote participants’ task performance than treatment transformations: Participants in Cluster 1 who completed more game levels than those in Cluster 2 demonstrated a frequent behavioral sequence from failed attempts (F) to the use of verbal task narratives (V) (F → V). Therefore, it supports the prior finding that providing instant feedback to students in DGBL could positively impact the way students plan, monitor, and execute problem solutions, thus enhancing students’ performance (Law & Chen, 2016; Nadolski & Hummel, 2017; Tsai et al., 2015).

Conclusion

The study provides an initial and critical reference of how problem solvers interact with a variety of MERs in DGBL. Data mining techniques, including cluster and sequential analyses, were used to extract learners’ behavioral patterns of using various MERs. We observed that the maneuverable visual-spatial representation was the most frequently used type of representations in the selected architecture game for the problem solvers when they were involved in solving mathematical problems, followed by the formal mathematical notations. Manipulating dynamic game objects and interacting with formal mathematical notation composed an essential part of solving contextualized mathematical problems in DGBL. The findings of cluster and sequential analyses coherently indicated that participants have engaged in a high level of representational transformations, including both treatment and conversion, which help them internalize math concepts. Consequently, participants who were capable of making more conversion transformations have exhibited a greater performance of gameplay than those who made fewer. We also observed that participants with gaming experience have solved game-based tasks more efficiently than non-players. Learners should be given more opportunities to practice seeking, processing, and coordinating grounded problem information in a learning environment with a variety of MERs.

Implications

The selected architectural simulation game in the study was found to engage learners in treatment and conversion transformations between various MERs. It shows the feasibility of using digital game-based math context problem solving to promote students’ abilities to recognize, encode and translate grounded problem information embedded in a variety of MERs. The study findings help to inform teachers, game developers, and educators about how to design and implement MERs in DGBL to enhance students’ mathematical problem solving performance.

First, participants with different gaming experience have demonstrated different behavioral patterns of using MER in a game-based learning environment. As such, we recommend that teachers should provide appropriate guidance on how to interact with the MERs. For example, teachers may provide students who lack sufficient gaming experience with instructions on how to extract and recognize mathematical information embedded in a variety of MERs. For students who are insufficiently equipped with prior knowledge of mathematics, teachers can provide exercises and examples of representational transformations before the gameplay and encourage students to share the transformations they have observed and performed after gameplay.

Second, we propose that educational game developers should provide more opportunities for students to engage in representational transformations by designing games with multiple MERs rather than a single-mode external representation. In addition, game design should allow students to practice seeking, processing, and coordinating grounded problem information in a learning environment. This can be achieved by designing scaffolds that introduce situated or contextualized representations of math concepts.

Lastly, we suggest that exploring students’ behavioral patterns through data mining techniques (e.g., cluster and sequential analyses) should be encouraged in future DGBL research. It fosters a deeper understanding of the learning process by visualizing learners’ behavioral patterns of interactions in DGBL.

Limitation and Future Research

This naturalistic inquiry involved a small sample and did not directly measure participants’ math problem-solving knowledge or performance before and after gameplay. A future study can be enhanced with an increasing number of participants and by including an external problem-solving test. It is warranted to conduct future research on the same phenomenon in longitudinal and comparative ways to investigate whether the effect of gaming experience will diminish on the students’ interactions with MERs. It is also suggested that the potential moderating effect of the learner variables (e.g., prior knowledge of relevant content, problem solving skills, or perception of game flow) should be controlled or investigated for the future research.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research,, authorship, and/or publication of this article: This work was funded by the National Science Foundation, Grant No. 1720533. Any opinions, findings, and conclusions or recommendations expressed in these materials are those of the author and do not necessarily reflect the views of the National Science Foundation.

Statements on open data and ethics

This study was conducted with the university IRB (human subject protection) approval and adhered to ethical guidelines as the nature of study demanded. Being constrained by the human subject protection policies, the original study data are not open. The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.

ORCID iDs

Yanjun Pan

Chih-Pu Dai

Author Biographies

Dr. Yanjun Pan graduated from Educational Psychology and Learning Systems Department at Florida State University’s College of Education. Her research interests include digital game-based learning, learning analytics, participatory design, and innovative teaching methods and strategies.

Dr. Fengfeng Ke is a full professor of the Educational Psychology and Learning Systems Department at Florida State University’s College of Education. Her main research interests include digital game-based learning, inclusive design of e-learning, and computer-supported collaborative learning for diverse learners.

Chih-Pu Dai is a PhD candidate in the Department of Educational Psychology and Learning Systems at Florida State University. His research interests include game- and simulation-based learning in Science, Technology, Engineering, and Mathematics (STEM) fields for K-12 and adult learners.

References

Adu‐Gyamfi

Bossé

M. J.

Lynch‐Davis

(2019). Three types of mathematical representational translations: Comparing empirical and theoretical results. School Science and Mathematics, 119(7), 396–404. https://doi.org/10.1111/ssm.12360.

Ainsworth

(1999). The functions of multiple representations. Computers and Education, 33(2–3), 131–152. https://doi.org/10.1016/s0360-1315(99)00029-9.

Bakeman

Gottman

J. M.

(1997). Observing interaction: An introduction to sequential analysis (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511527685.

Bavelier

Shawn Green

Pouget

Schrater

(2012). Brain plasticity through the life span: Learning to learn and action video games. Annual Review of Neuroscience, 35, 391–416. https://doi.org/10.1146/annurev-neuro-060909-152832.

Brenner

Mayer

R. E.

Mosely

Brar

Duran

Reed

B. S.

Webb

(1997). Learning by understanding: The role of multiple representations in learning algebra. American Educational Research Journal, 34(4), 663–690. https://doi.org/10.3102/00028312034004663.

Bullock

E. P.

Roxburgh

A. L.

Moyer-Packenham

P. S.

Bektas

Webster

J. S.

Bullock

K. A.

(2021). Connecting the dots: Understanding the interrelated impacts of type, quality and children’s awareness of design features and the mathematics content learning goals in digital math games and related learning outcomes. Journal of Computer Assisted Learning, 37(2), 557–586. https://doi.org/10.1111/jcal.12508.

Calder

Larkin

Sinclair

(2018). Mobile technologies: How might using mobile technologies reshape the learning and teaching of mathematics? In Using mobile technologies in the teaching and learning of mathematics (pp. 1–7). Springer. https://doi.org/10.1007/978-3-319-90179-4_1.

Castro

Vellido

Nebot

À.

Mugica

(2007). Applying data mining techniques to e-learning problems. In Jain

L. C.

Tedman

R. A.

Tedman

D. K.

(Eds.), Evolution of teaching and learning paradigms in intelligent environment. Studies in computational intelligence (pp. 183–221). Springer. https://doi.org/10.1007/978-3-540-71974-8_8.

Daroczy

Wolska

Meurers

W. D.

Nuerk

H. C.

(2015). Word problems: A review of linguistic and numerical factors contributing to their difficulty. Frontiers in Psychology, 6, 1–13. https://doi.org/10.3389/fpsyg.2015.00348.

10.

De Bock

Neyens

Van Dooren

(2017). Students’ ability to connect function properties to different types of elementary functions: An empirical study on the role of external representations. International Journal of Science and Mathematics Education, 15(5), 939–955. https://doi.org/10.1007/s10763-016-9724-z.

11.

Duval

(2006). A cognitive analysis of problems of comprehension in a learning of mathematics. Educational Studies in Mathematics, 61(1), 103–131. https://doi.org/10.1007/s10649-006-0400-z.

12.

Duval

(2017). Understanding the mathematical way of thinking–The registers of semiotic representations. Springer international Publishing.

13.

Feldman

Monteserin

Amandi

(2014). Detecting students’ perception style by using games. Computers and Education, 71, 14–22. https://doi.org/10.1016/j.compedu.2013.09.007.

14.

Friard

Gamba

(2016). BORIS: A free, versatile open-source event-logging software for video/audio coding and live observations. Methods in Ecology and Evolution, 7(11), 1325–1330. https://doi.org/10.1111/2041-210X.12584.

15.

Goldin

G. A.

(2003). Representation in school mathematics: A unifying research perspective. In Kilpatrick

Martin

W. G.

Schifter

(Eds.), A research companion to principles and standards for school mathematics (pp. 275–285). National Council of Teachers of Mathematics.

16.

Goldin

G. A.

(2014). Mathematical representations. In Lerman

(Ed.), Encyclopedia of mathematics education. Springer. https://doi.org/10.1007/978-94-007-4978-8_103.

17.

Griffin

Jitendra

(2009). Word problem-solving instruction in inclusive third-grade mathematics classrooms. Journal of Educational Research, 102(3), 187–202. https://doi.org/10.3200/JOER.102.3.187-202.

18.

Hair

J. F.

Black

W. C.

(2000). Cluster analysis. In Grimm

L. G.

Yarnold

P. R.

(Eds.), Reading and understanding MORE multivariate statistics (pp. 147–205). American Psychological Association.

19.

Hegarty

Mayer

R. E.

Monk

C. A.

(1995). Comprehension of arithmetic word problems: A comparison of successful and unsuccessful problem solvers. Journal of Educational Psychology, 87(1), 18–32. https://doi.org/10.1037/0022-0663.87.1.18.

20.

Hou

H. T.

(2012). Exploring the behavioral patterns of learners in an educational massively multiple online role-playing game (MMORPG). Computers and Education, 58(4), 1225–1233. https://doi.org/10.1016/j.compedu.2011.11.015.

21.

Hou

H. T.

(2015). Integrating cluster and sequential analysis to explore learners’ flow and behavioral patterns in a simulation game with situated-learning context for science courses: A video-based process exploration. Computers in Human Behavior, 48, 424–435. https://doi.org/10.1016/j.chb.2015.02.010.

22.

Hsu

M. E.

Cheng

M. T.

(2021). Immersion experiences and behavioural patterns in game‐based learning. British Journal of Educational Technology. Advanced Online Publication. https://doi.org/10.1111/bjet.13093.

23.

Hung

C.-M.

Huang

Hwang

G.-J.

(2014). Effects of digital game-based learning on students’ self-efficacy, motivation, anxiety, and achievements in learning mathematics. Journal of Computers in Education, 1, 151–166. https://doi.org/10.1007/s40692-014-0008-8.

24.

Hurst

M. A.

Massaro

Cordes

(2020). Fraction magnitude: Mapping between symbolic and spatial representations of proportion. Journal of Numerical Cognition, 6(2), 204–230. https://doi.org/10.5964/jnc.v6i2.285.

25.

Hwang

G. J.

Hsu

T. C.

Lai

C. L.

Hsueh

C. J.

(2017). Interaction of problem-based gaming and learning anxiety in language students’ English listening performance and progressive behavioral patterns. Computers and Education, 106(1), 26–42. https://doi.org/10.1016/j.compedu.2016.11.010.

26.

Jeong

A. C.

(2003). The sequential analysis of group interaction and critical thinking in online. The American Journal of Distance Education, 17(1), 25–43. https://doi.org/10.1207/S15389286AJDE1701_3.

27.

(2016). Designing and integrating purposeful learning in game play: A systematic review. Educational Technology Research and Development, 64(2), 219–244. https://doi.org/10.1007/s11423-015-9418-1.

28.

(2019). Mathematical problem solving and learning in an architecture-themed epistemic game. Educational Technology Research and Development, 67(5), 1085–1104. https://doi.org/10.1007/s11423-018-09643-2.

29.

Clark

K. M.

(2020). Game-based multimodal representations and mathematical problem solving. International Journal of Science and Mathematics Education, 18(1), 103–122. https://doi.org/10.1007/s10763-018-9938-3.

30.

Kellman

Massey

Roth

Burke

Zucker

Saw

A.T.

Aguero

K. E.

Wise

J. A.

(2008). Perceptual learning and the technology of expertise: Studies in fraction learning and algebra. Pragmatics & Cognition, 16(2), 356–405. https://doi.org/10.1075/p&c.16.2.07kel.

31.

Kiili

(2007). Foundation for problem-based gaming. British Journal of Educational Technology, 38(3), 394–404. https://doi.org/10.1111/j.1467-8535.2007.00704.x.

32.

Kilpatrick

(2014). History of research in mathematics education. In Lerman

(Ed.), Encyclopedia of mathematics education (pp. 267–272). Springer.

33.

Kirsh

(2010). Thinking with external representations. AI and Society, 25(4), 441–454. https://doi.org/10.1007/s00146-010-0272-8.

34.

Law

Chen

C. H.

(2016). Promoting science learning in game-based learning with question prompts and feedback. Computers and Education, 103(1), 134–143. https://doi.org/10.1016/j.compedu.2016.10.005.

35.

Lesh

(1981). Applied mathematical problem solving. Educational Studies in Mathematics, 12(2), 235–264. https://doi.org/10.1007/BF00305624.

36.

Lesh

Post

Behr

(1987). Representations and translations among representations in mathematics learning and problem solving. In Janiver

(Ed.), Problems of representation in the teaching and learning of mathematics (pp. 33–40). Lawrence Erlbaum Associates.

37.

Lin

Y. C.

Hsieh

Y. H.

Hou

H. T.

(2015). Developing a self-report-based sequential analysis method for educational technology systems: A process-based usability evaluation. Journal of Educational Computing Research, 53(1), 95–123. https://doi.org/10.1177/0735633115590772.

38.

Miles

M. B.

Huberman

A. M.

(1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Sage Publications.

39.

Moon

(2020). In-game actions to promote game-based math learning engagement. Journal of Educational Computing Research, 58(4), 1–23. https://doi.org/10.1177/0735633119878611.

40.

Moyer-Packenham

P. S.

Lommatsch

C. W.

Litster

Ashby

Bullock

E. K.

Roxburgh

A. L.

Shumway

J. F.

Speed

Covington

Hartmann

Clarke-Midura

Skaria

Westenskow

MacDonald

Symanzik

Jordan

(2019). How design features in digital math games support learning and mathematics connections. Computers in Human Behavior, 91, 316–332. https://doi.org/10.1016/j.chb.2018.09.036.

41.

Moyer-Packenham

P. S.

Roxburgh

A. L.

Litster

Kozlowski

J. S.

(2021). Relationships between semiotic representational transformations and performance outcomes in digital math games (pp. 1–31). Knowledge and Learning. https://doi.org/10.1007/s10758-021-09506-5.

42.

Nadolski

R. J.

Hummel

H. G. K.

(2017). Retrospective cognitive feedback for progress monitoring in serious games. British Journal of Educational Technology, 48(6), 1368–1379. https://doi.org/10.1111/bjet.12503.

43.

Nesher

Teubal

(1975). Verbal cues as an interfering factor in verbal problem solving. Educational Studies in Mathematics, 6(1), 41–51. https://doi.org/10.1007/BF00590023.

44.

Pino-Fan

L. R.

Retamal

I. G.

Font Moll

Duval

(2017). Analysis of the underlying cognitive activity in the resolution of a task on derivability of the absolute-value function: Two theoretical perspectives. PNA, 11(2), 97–124. https://doi.org/10.30827/pna.v11i2.6076.

45.

Pongsakdi

Kajamies

Veermans

Lertola

Vauras

Lehtinen

(2020). What makes mathematical word problem solving challenging? Exploring the roles of word problem characteristics, text comprehension, and arithmetic skills. ZDM Mathematics Education, 52, 33–44. https://doi.org/10.1007/s11858-019-01118-9.

46.

Rovniak

L. S.

Sallis

J. F.

Saelens

B. E.

Frank

L. D.

Marshall

S. J.

Norman

G. J.

Conway

T. L.

Cain

K. L.

Hovell

M. F.

(2010). Adults’ physical activity patterns across life domains: Cluster analysis with replication. Health Psychology, 29(5), 496–505. https://doi.org/10.1037/a0020428.

47.

Sidney

P. G.

Thompson

C. A.

Rivera

F. D.

(2019). Number lines, but not area models, support children’s accuracy and conceptual models of fraction division. Contemporary Educational Psychology, 58, 288–298. https://doi.org/10.1016/j.cedpsych.2019.03.011.

48.

Siew

P. H.

(2018). Pedagogical change in mathematics learning: Harnessing the power of digital game-based learning. Educational Technology & Society, 21(4), 259–276.

49.

Smith

E. T.

Bhaskar

Hinerman

Basak

(2020). Past gaming experience and cognition as selective predictors of novel game learning across different gaming genres. Frontiers in Psychology, 11, 786. https://doi.org/10.3389/fpsyg.2020.00786.

50.

Sun

Lin

C. H.

Song

(2021). Knowledge-construction behaviors in a mobile learning environment: A lag-sequential analysis of group differences. Educational Technology Research and Development, 69(2), 533–551. https://doi.org/10.1007/s11423-021-09938-x.

51.

Tsai

M. J.

Huang

L. J.

Hou

H. T.

Hsu

C. Y.

Chiou

G. L.

(2016). Visual behavior, flow and achievement in game-based learning. Computers and Education, 98, 115–129. https://doi.org/10.1016/j.compedu.2016.03.011.

52.

Tsai

F. H.

Tsai

C. C.

Lin

K. Y.

(2015). The evaluation of different gaming modes and feedback types on game-based formative assessment in an online learning environment. Computers and Education, 81, 259–269. https://doi.org/10.1016/j.compedu.2014.10.013.

53.

Vogt

Nagel

(1992). Cluster analysis in diagnosis. Clinical Chemistry, 38(2), 182–198. https://doi.org/10.1093/clinchem/38.2.182.

54.

Ward

J. H.

(1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58(301), 236–244. https://doi.org/10.1080/01621459.1963.10500845.