Abstract
In the current technology-driven era, digital game-based learning (DGBL) has emerged as a practical approach for enhancing mathematics achievement among young students. It is believed that digital drill-and-practice educational games can replace traditional pen-and-paper drills in mathematics education and have a better influence on incorporating postgame debriefing activities. The current study applied a quasi-experimental study mixed-method design to evaluate the differences in mathematics learning achievement and learning retention between three types of interventions of PaGamO, PaGamO with postgame group discussions, and pen-and-paper drills among sixth-grade elementary students in mathematics courses. Greater improvements were observed among students learning with PaGamO compared with traditional pen-and-paper drills. Positive perceptions of PaGamO among students were confirmed through semi-structured interviews. However, students who participated in postgame group discussions exhibited impaired learning retention, suggesting that the immediate effects of DGBL with postgame discussions are positive but that the delayed effects are adverse. Implications of the study findings and suggestions for future research are also provided for broader and practical application of PaGamO and postgame activities in educational settings.
Keywords
Introduction
Mathematics is universally recognized as a crucial component of scientific, technological, political, and socioeconomic progress (Sherman et al., 2015). In the current technology-driven world, traditional teaching methods in mathematics education are being transformed by diverse techniques and strategies, with digital game-based learning (DGBL) arising as a predominant new approach to teaching (Gros, 2007; Hwa, 2018). DGBL has undergone rapid advances in the past 30 years and has proven to be an effective method for enhancing cognitive and affective skills among students (Byun & Joung, 2018; C. C. Chang et al., 2017; W. F. Chen et al., 2011; Chou et al., 2020; Ku et al., 2014; Pan et al., 2024). Integrating digital educational games into mathematics education can help shift the negative attitudes toward the subject that young students often hold. Creating enjoyable and exciting learning environments through these games leads to the development of positive perspectives on mathematics (Ke, 2008b; Ke & Grabowski, 2007; Klimmt et al., 2007; Ku et al., 2014; Law et al., 2020). Extensive research has provided robust evidence indicating that implementing DGBL in primary mathematics education improves learning effectiveness across various aspects, including achievement levels and gains in accuracy and calculation speed (Kebritchi et al., 2010; Miller & Robertson, 2011; Tokac et al., 2019; Zapalska et al., 2012). Scholars argue that the use of math games to examine how learners grasp mathematical concepts demonstrates that game-based approaches are effective for learning mathematics (Kiili et al., 2018; K. H. Yang & Chen, 2023).
Among diverse educational games, digital drill-and-practice educational games have gained significant attention as a suitable replacement for traditional pen-and-paper drills in mathematics courses (Beserra et al., 2017). Traditional drills are considered to be disadvantageous for performance, motivation, and engagement among students (Castellar et al., 2014; M. Chang et al., 2015, 2016; Ke, 2008a). The mechanisms and themes of digital drill-and-practice educational games can promote autonomy and engagement in math problem-solving among students, leading to better performance (M. Chang et al., 2015; Hwa, 2018). These games typically feature algorithms that dynamically adjust the difficulty of problems, enabling students to gain a higher level of proficiency (Olive & Vomvoridi, 2006) and develop their cognitive skills (Castellar et al., 2014). In addition, the technological systems of digital educational games could provide each student with instant and personalized feedback, which is considered beneficial for students to monitor and reflect on their learning status (Beserra et al., 2017). However, several researchers have argued against using drill-and-practice games, believing other types of games to be more effective for enhancing learning and skill development among students. Empirical evidence is required to determine whether digital drill-and-practice educational games are effective enough to subtitle traditional drill approaches, which are still widely applied in modern elementary educational settings (Ke, 2008a).
A systematic literature review of game-based learning research highlighted the need for more attention to increasing the rate of knowledge retention due to the fact that papers reported mixed experimental results for the long-term influence of game-based learning on students’ knowledge retention (Tavares, 2022). Scholars are eager to observe the positive effect on both short and long-term retention of students’ important concepts after the incorporation of digital educational games in the course. Some of the studies achieve both (Krop et al., 2023; Y. T. C. Yang & Chang, 2013; K. H. Yang & Chen, 2021), but some of them only achieve short-term influence (Rondon et al., 2013). To provide a deeper understanding, the current study expected the integration of digital educational games would be more effective than the traditional method of improving students’ long-term knowledge retention.
DGBL has received considerable attention in the recent field of educational research, which has brought up postgame debriefing activities such as postgame discussion and reflection, which are considered essential for establishing connections between games and curricula (Bado, 2022). Such connections are crucial for achieving optimal learning effectiveness (Franciosi, 2017; Kangas et al., 2017; Kriz, 2010; Whalen et al., 2018; Y. T. C. Yang, 2012). Postgame debriefing activities are tasks led by instructors, assistants, or students themselves that are aimed to connect skills acquired while participating in-game activities with prior knowledge held by the students through confirmation, correction, clarification, and reflection to reinforce learning (Bado, 2022). Postgame discussions, serving as an interactive exchange between each other in DGBL, afforded the teacher an opportunity to explicitly articulate the connection between the game’s content and the broader curriculum (Poli et al., 2012). Absent this exchange, students may not readily perceive this connection, which might fail to make optimal DGBL in improving students’ cognition (Bado, 2022; Garris et al., 2002). Despite several studies that have highlighted the role of postgame debriefing activities in DGBL, group discussions are still the ones that gain wide attention and could be particularly beneficial for learning outcomes because they provide meaningful opportunities for interaction and opinion-sharing among students (Watson et al., 2011). Postgame group discussions are crucial in facilitating effective learning by enabling students to clarify misunderstandings, confirm concepts and principles, and share strategies and experiences with their peers (Bilgin et al., 2015). A recent systematic review found that 72% of the papers did not report any post-game instructional activity, which showed that students’ game-based learning ended at the gameplay stage (Bado, 2022). More attention is required to the importance of DGBL research. Although some existing research has been conducted on postgame debriefing activities in DGBL, further empirical studies are required to investigate how postgame debriefing activities reinforce knowledge acquired during DGBL activities. To respond to the research gap, the current study was designed to include two groups of students that were involved in DGBL in their mathematics course, but respectively with or without postgame discussions conduction, which could provide empirical evidence of the effectiveness of these activities on students’ knowledge gain and learning retention.
PaGamO is a well-known, internationally recognized, award-winning digital drill-and-practice educational game, developed by a team of scholars from Taiwan. It features motivating game mechanics, an engaging storyline, immersive scenarios, and personalized learning functions, making it highly effective and attractive for learners. Widely adopted across various educational levels, from elementary to secondary schools (I. F. Chang et al., 2020; Lau et al., 2021), PaGamO is perceived by users as both easy to use and highly useful. Notably, the platform has been extensively applied in formal and informal educational contexts in Taiwan, including classroom instruction, learning counseling, reinforcement exercises, independent learning tasks, and adaptive learning environments. Its effectiveness and versatility have earned it strong recommendations from the Ministry of Education (MOE) in Taiwan, contributing to its increasing recognition in other Asian countries such as Malaysia and Singapore (Ministry of Education Republic of China (Taiwan), 2017; PaGamO, 2013). This game was designed to replace traditional pen-and-paper drills with an entertaining gaming experience with diverse game mechanisms that could significantly motivate students to engage in problem-solving tasks actively (A. Yang, 2017; S. F. Yang & OuYang, 2019). The game features a competitive environment in which users build, occupy, and conquer territories by solving as many problems as possible. The competition for territories encourages users to participate actively in the game. The game also has side quests and features, such as the ability to earn badges and acquire props, which enhance the enjoyment of the game and motivate users to perform the drill-and-practice tasks by offering them ownership, possession, and a sense of accomplishment (Cheung & Ng, 2021).
PaGamO is considered to be practical and motivational due to its entertaining game mechanisms, and it has been integrated into education because it can effectively enhance students’ cognitive performance in areas such as learning achievement and teamwork skills (Cheung & Ng, 2021; Law et al., 2020; S. F. Yang & OuYang, 2019), and affective outcomes such as learning attitude and enjoyment (S. F. Yang & OuYang, 2019). PaGamO was designed to arouse students’ competitiveness and engagement in addressing learning tasks, which is highly recommended to be an effective digital educational game to implemented. In addition, students of different ages have positive perceptions of the game, which shows its broad acceptance in educational settings (Cheung & Ng, 2021; Law et al., 2020; S. F. Yang & OuYang, 2019). However, whether PaGamO effectively improves learning achievements and knowledge retention has not been demonstrated, which requires more empirical and rigorous evidence in this field. A study demonstrated that students’ performance in PaGamO could be a significant predictor of students’ achievement in mathematic multiple-choice questions but not for the whole performance of the examination with different question formats (Cheung & Ng, 2021). In addition, some of the aforementioned studies lacked control groups to confirm the game’s effectiveness, which leads to a lack of confidence in attributing the improvement of students’ cognitive or affective skills to the incorporation of PaGamO (Cheung & Ng, 2021). Hence, the present study aimed to address this gap by including a control group to evaluate the difference in effectiveness between PaGamO and traditional pen-and-paper drills in an elementary mathematics course, and the incorporation of postgame debriefing activities were included to gain deeper insights into this topic.
This study investigated whether incorporating PaGamO into a mathematics course improved mathematics learning achievement and learning retention among sixth-grade elementary students. The study also evaluated whether postgame group discussions influenced the effects of PaGamO. Three classes were recruited in the current research. The first class received PaGamO alone, the second class received PaGamO with postgame group discussions, and the third class received pen-and-paper drills. Both the immediate and delayed effects of the interventions were examined. Additionally, qualitative data on student perceptions were collected through semi-structured interviews to support drawing statements and gain a deeper understanding of how PaGamO was implemented in the mathematics classes, which were expected to observe both advantages and challenges that could provide valuable implementing suggestions in educational settings. The current research aims to provide both theoretical and pedagogical contributions to the field of DGBL, with a particular focus on the use of PaGamO. Previous studies on PaGamO have primarily explored its potential in enhancing engagement and motivation in various subjects (Cheung & Ng, 2021; Law et al., 2020; S. F. Yang & OuYang, 2019); however, limited research has systematically examined its effectiveness in comparison to traditional learning methods, such as pen-and-paper drills, particularly in the domain of mathematics education. This study builds on these foundations by empirically investigating not only the differences in learning outcomes and retention but also the added value of integrating postgame group discussions.
Specifically, the inclusion of postgame discussions represents a novel approach in the context of PaGamO, offering insights into how collaborative reflection after gameplay can shape students’ learning experiences and perceptions. By addressing these aspects, this study contributes to a deeper understanding of the mechanisms through which digital educational games, and PaGamO in particular, can be effectively implemented in formal educational settings. These findings aim to inform both academic discourse and practical applications, advancing the broader field of educational technology. In exploring these topics, three research questions were addressed:
RQ1: Is there a difference between the effectiveness of interventions of PaGamO, PaGamO with postgame group discussions, or pen-and-paper drills on elementary students’ mathematic learning achievement?
RQ2: Is there a difference between the delay effect of interventions of PaGamO, PaGamO with postgame group discussions, or pen-and-paper drills on elementary students’ mathematic learning achievement?
RQ3: How do students perceive the incorporation of PaGamO into their mathematics course?
Method
Participants
The current research applied a convenience sampling method to recruit sufficient participants to offer solid empirical evidence in the investigation of the effectiveness of digital drill-and-practice educational games and whether there is a better influence with incorporating debriefing activities. A total of 85 students from three classes in an urban elementary school were recruited. The classes were randomly assigned to receive either PaGamO, PaGamO with postgame group discussions, or pen-and-paper drills. The participants were between 10 and 11 years old, and 52% of the participants were boys. The current research has informed the participants about the purpose and procedures of the research experiment and obtained their consent to be involved on a voluntary basis.
Experimental Design
The study evaluated the immediate and delayed effects of incorporating PaGamO into mathematics courses on learning achievement among sixth-grade elementary students. The intervention lasted for 5 weeks and was conducted in nine sessions. The study employed a quasi-experimental mixed-method design that systematically integrated quantitative and qualitative data collection and analysis. Data were collected using a mathematics achievement test and semi-structured interviews.
The experimental design is illustrated in Table 1. Three classes were recruited and randomly assigned to experimental group 1, experimental group 2, or the control group. The interventions were PaGamO in experimental group 1, PaGamO with postgame group discussions in experimental group 2, and pen-and-paper drills in the control group. A non-equivalent group pre-post-delayed test was applied to investigate between-group differences in immediate and delayed effects. Wilcoxon signed-rank tests were applied to investigate whether three groups of students’ mathematic learning achievement improved after the interventions, and One-way ANCOVA was applied to explore whether there were statistical differences between the three groups’ improvements. The independent variable was the learning method applied in each group, and the dependent variable was mathematics learning achievement. In addition, semi-structured interviews were constructed after the interventions to explore students of experimental groups’ perspectives on the incorporation of PaGamO in mathematics courses, which were analyzed with the application of the content analysis method.
The Experimental Design of the Study.
O1: Pretest of mathematic learning achievement.
O2: Posttest of mathematic learning achievement.
O3: Delayed posttest of mathematic learning achievement.
X1: Incorporation of PaGamO into mathematics courses.
X2: Incorporation of PaGamO with postgame group discussions into mathematics courses.
X3: Incorporation of pen-and-paper drills into mathematics courses.
To ensure the validity of the experimental treatment, the control variables, including the instructor, teaching material, teaching progress, and entry behavior, were controlled using the constant method (see Table 2).
The Control Variables of the Study.
Instructional Design
The study evaluated the immediate and delayed effects of different learning approaches on mathematics learning achievement. The instructional design is illustrated in Figure 1. The study began in the 10th week of the semester. First, students completed the mathematics pretest. Next, the intervention was conducted. The intervention lasted 5 weeks. All groups were taught in the traditional manner for the first 15 min of each class before receiving their respective intervention. The three groups were taught the same content, approach, and progress but were only different from the practice and drill methods in enhancing mathematics concepts, which were respectively PaGamO, PaGamO with postgame group discussions and pen-and-paper drills. Experimental groups 1 and 2 were asked to solve mathematics problems on PaGamO. After completing the PaGamO activity, experimental group 2 held group discussions by freely proposing their own experience, perceived knowledge blind spots, and mistakes to share and discuss. This might effectively address group members’ incorrectly answered questions and confusion, build a connection between the game’s experience and domain knowledge, and further enhance knowledge retention. The control group was asked to complete pen-and-paper drills, the traditional assessment tools requiring students to read questions and respond on worksheets. When students in the experimental groups utilized PaGamO for practice, they received immediate feedback with answers and explanations from the system to clarify the concepts. Conversely, students in the control group, who engaged in pen-and-paper drills, were provided with answers and explanations from the instructors post-practice to correct the concepts. In week 16, semi-structured interviews were conducted, and the students completed the mathematics posttest. Last, in week 18, the students again completed the mathematics posttest.
The instructional design.
Learning System
The current study analyzed PaGamO’s main game functions based on the LM-GM (Learning Mechanics–Game Mechanics) classification framework of educational games in line with enhancing students different order thinking skills, which was constructed by Arnab et al. (2015) that based on Bloom’s theory (Bloom et al., 1964) and digital taxonomy of Anderson and Krathwohl (2001; see Figure 2). The integration of PaGamO in educational settings could construct an entertaining and motivating environment through its 5 main game functions: building, occupying, and conquering territories; gaining experience points and coins; acquiring materials, badges, and props; achieving the top rank on the leaderboard; and learning analysis. The following were the detailed analysis and link between the PaGamO functions, corresponding embedded game mechanics, and the expected effectiveness in enhancing thinking skills.
(1) Build, occupy, and conquer territories: This function is the core mission with a story background of “the end of the world,” and students will become brave fighters to create new land and start the journey of finding the fragments of the Stone of Wisdom, which is embedded with the “Cut scenes/story” and “Roleplay” game mechanics.
(2) Acquire material, badges, and props: Students are required to acquire material, badges, and props through fights, trade, or different approaches that could support them to become more powerful fighters. This function is embedded with the “Selecting/collecting” and “Goods/information” mechanics, which will guide them in making choices and gathering resources in the game.
(3) Gain experience points and coins: Students will gain experience points and coins, which are essential tokens in the game, by successfully solving problems. These tokens can be used to increase one’s own character’s power and level, which will gradually lead students to enter more challenging levels. This function is embedded with “Tokens” and “Progression” mechanics.
(4) Master leaderboard: This function is embedded with “Action points” and “Reward” mechanics, which prompt students to use limited resources effectively in every game turn (remaining blood volume) to increase or maintain their ranking in the leaderboard to win the rewards for the top masters in the game.
(5) Learning analysis: This function is aimed to provide organized and graphical information on students’ achievements in problem-solving based on different dimensions, which is embedded with “Feedback” and “Assessment” mechanics, supporting students in reflecting on their blind spots of the domain knowledge and performance.
Analysis of PaGamO’s main game functions using the LM-GM framework.
To investigate whether incorporating PaGamO into a mathematics course improved learning effectiveness, PaGamO was incorporated into a mathematics course in two classes for 5 weeks, replacing the traditional pen-and-paper drills that would have otherwise been completed. The following are the four main activities of PaGamO that students experienced in the present study:
(1) Try to build, occupy, and conquer territories (see Figure 3)
The main quests in PaGamO involve building, occupying, and conquering territories by solving as many mathematics problems as possible. Territories with black flags are owned by students. Ownership can be strengthened by correctly solving problems, which can prevent occupied territories from being taken over. The difficulty of solving the problems is determined by the level of the territories that contain a certain amount of blood volume. Users’ success in attacking could gradually decrease the remaining blood volume.
(2) Solve mathematics problems (see Figure 4)
Students solved mathematics problems related to the concept of the highest common factor and least common multiple. These problems, which varied in type and difficulty, were either provided by instructors or sourced from the built-in question bank of PaGamO.
(3) Check the mission list and plan for the next step (see Figure 5)
Students can access the mission list to view assigned missions, regardless of the missions’ completion statuses. The mission list provides crucial information about each current mission, including its description, deadline, and completion percentage. This function helps students plan their learning strategies and manage their time.
(4) Analyze their learning status (see Figure 6)
According to class, chapter, discipline, or time, multiple types of analysis are provided in PaGamO for students to reflect on their performance. Corrected items, corrections, and completion rates are considered informative data for promoting student reflection and goal-setting.
Build, occupy, and conquer territories.
Mathematics problem-solving.
Mission list.
Learning analysis.
Measurements
Mathematics Learning Achievement Test
A mathematics test was developed by the current research on the basis of the capability index for sixth-grade elementary students formulated by the Ministry of Education in Taiwan. The tests included 22 multiple-choice questions that aimed to evaluate the students’ comprehension of mathematics concepts of prime numbers, composite numbers, and prime factors, understand the meaning of two numbers being relatively prime, and use prime factorization or the short division method to find the greatest common divisor (GCD) and the least common multiple (LCM) of two numbers.
To ensure content validity, five experienced mathematics instructors reviewed the test by using a two-way specification table to score the item’s relevance to the concepts, correctness, and description comprehensibility. The test items all scored ranging from 3 (appropriate) to 4 (highly appropriate) points, which confirmed the expert validity. A pilot test involving 23 sixth-grade students was then conducted to evaluate item discrimination. The results indicated that the items could effectively discriminate between high-score and low-score groups (t = −9.66, p < .05). Items that did not meet the discrimination index standard of 0.25 were deleted. The final version of the achievement test comprised 22 items and had a maximum total score of 22 points. The test demonstrated strong reliability, with a Cronbach’α of 0.88 (Aiken & Groth-Marnat, 2006).
Semi-Structured Interviews
Interviews were conducted to collect information on the student’s perceptions of the incorporation of PoGamO into their mathematics course. A total of 12 students from experimental groups 1 and 2 were randomly selected to participate in individual interviews. Each interview lasted 30 min, which was sufficient time to gain a comprehensive understanding of each student’s thoughts on PaGamO. The outline of the interview is presented in Table 3.
Semi-Structured Interview Outline.
The current study applied the content analysis method to analyze the semi-structured interviews with the respondents to address research question 3. Students’ responses were comprehensively turned into transcripts after the interviews and respectively coded into themes and categorized by two coders. Discussions were carried out to resolve disagreements during the research. A total of 12 student names were encoded using numerical identifiers to safeguard confidentiality, which were coded from S1, S2…. and S12.
Results and Discussion
Effects on Mathematics Learning Achievement (RQ1)
Wilcoxon signed-rank tests were conducted to investigate whether mathematics ability improved in each group (see Table 4). Significant improvements were observed in mathematics ability in the three groups. Mathematics ability improved in experimental group 1 (Z = 4.4, p < .05), experimental group 2 (Z = 6.66, p < .05), and the control group (Z = 3.29, p < .05). Notably, the effect sizes for experimental groups 1 (d = 0.84) and 2 (d = 1.12) were large and larger than the effect size for the control group (d = 0.54), demonstrating that the incorporation of PaGamO into the mathematics course effectively improved mathematics ability.
Results of Wilcoxon Signed-Rank Test of Students’ Mathematics Learning Achievement (N = 85).
Note. Effect sizes: d = 0.2–0.5 (small effect), d = 0.5–0.8 (moderate effect), and d ≥ 0.8 (large effect; Cohen, 1988).
p < .05.
One-way ANCOVA was performed to investigate differences in improvements in mathematics ability between the groups. Pre-existing between-group differences in mathematics ability were controlled for using the mathematics prettest (see Table 5), which revealed significant differences between the three groups (F = 0.14, p = .14). Post hoc analysis revealed a significant difference between experimental group 2 and the control group; experimental group 2 outperformed the control group (p = .04).
Summary of ANCOVA on Students’ Mathematics Learning Achievement.
p < .05.
All groups underwent significant improvements in mathematics learning achievement. Notably, the effect sizes of the improvements in experimental groups 1 and 2 reached a high range. This promising result is consistent with those of other previous studies (Cheung & Ng, 2021; Law et al., 2020; S. F. Yang & OuYang, 2019), which suggested that the game mechanics of PaGamO could potentially replace the traditional pen-and-paper drill for constructing a more motivating environment to engage them in drill-and-practice tasks. These active behaviors in practicing were effective in improving their mathematics ability (S. F. Yang & OuYang, 2019). Notably, improvements of students from experimental group 2 reached the highest range, which provided valuable insight into establishing these connections by conducting postgame group discussions could maximize learning effectiveness (Franciosi, 2017; Kangas et al., 2017; Kriz, 2010; Whalen et al., 2018; Y. T. C. Yang, 2012). These activities are achieved through processes of confirmation, correction, clarification, and reflection, which are confirmed to be effective in reinforcing short-term knowledge gain (Bado, 2022).
Although no significant differences were observed in the improvement of mathematic learning achievement between the three groups, post hoc analysis revealed a significant difference between experimental group 2 and the control group. This result is consistent with several studies that have demonstrated that the incorporation of DGBL is more effective in improving students’ performance than traditional methods in mathematics elementary courses (Hwa, 2018; Ku et al., 2014). Digital drill-and-practice educational games can dynamically adjust difficulty levels, a feature that is essential in scaffolding, which enables students to develop mathematic proficiency and cognitive skills (Castellar et al., 2014; Olive & Vomvoridi, 2006). PaGamO challenges students to solve problems as fast as possible with diverse game mechanisms, which creates a learning environment that is competitive and more exciting than that created using traditional teaching methods (S. F. Yang & OuYang, 2019). Despite limited empirical evidence, this preliminary finding agrees with previous findings and suggests that PaGamO is an effective replacement for traditional pen-and-paper drills in elementary mathematics courses (M. Chang et al., 2015; Hwa, 2018).
Surprisingly, no statistically significant differences were observed in mathematics learning achievement between experimental groups 1 and 2, indicating that the postgame group discussions might not provide significant cognitive support in the current experiment. A possible explanation for this result is that group-level debriefing activities, such as postgame group discussions, might promote team performance rather than individual performance (Bilgin et al., 2015; Tannenbaum & Cerasoli, 2013). Despite postgame group discussions being highly suggested to be applied as an effective postgame activity (Bado, 2022), scholars still argued that when a learning task is conducted individually, debriefing activities should also be undertaken at an individual level (Peters & Vissers, 2004). Self-debriefing activities could provide more private time for thinking, recall, and reflection after the gameplay stage, leading to an improvement in their performance (van der Meij et al., 2013). Given that PaGamO is played individually with a competitive game mechanism, self-debriefing activities might be a more effective approach to conducting self-reflection and making connections between one’s own game experience and knowledge gain. The current result agreed with previous statements on the timing and length of debriefing forms, and participant familiarity with the games is also an essential factor influencing the effects of debriefing activities (Peters & Vissers, 2004). Different forms of postgame debriefing activities, such as reflection questionnaires or guided and unguided reflective essays, might be suitable to be applied after the PaGamO gameplay stages but still require more investigation (Bado, 2022; Whalen et al., 2018). Further research is required to investigate the optimal reflection and debriefing approach for PaGamO, which could provide valuable suggestions for practical educational implementation.
Delayed Effects on Mathematics Learning Achievement (RQ2)
Wilcoxon signed-rank tests were conducted to investigate the delayed effects of the interventions on mathematics learning achievement (see Table 6). A significant decrease (Z = −3.7, p < .05) was observed in grades in experimental group 2. No significant increase or decrease was observed in the other groups, including experimental group 1 (Z = −0.98, p > .05) and the control group (Z = −0.54, p < .05). This result indicates that incorporating PaGamO with postgame group discussions into the mathematics course was disadvantageous for learning retention and that the other approaches had no effect on learning retention.
Results of Wilcoxon Signed-Rank Test of Students’ Delayed Effect on Mathematics Learning Achievement (N = 85).
Note. Effect sizes: d = 0.2–0.5 (small effect), d = 0.5–0.8 (moderate effect), and d ≥ 0.8 (large effect; Cohen, 1988).
p < .05.
One-way ANCOVA was performed to evaluate whether significant between-group differences existed in the delayed effects of the interventions. The mathematics posttest was used to control for the effects of pre-existing between-group differences (see Table 7). No significant differences between the three groups were observed (F = 0.9, p = .42). This finding indicates that the interventions had similar delayed effects on mathematics achievement.
Summary of ANCOVA on Learning Achievement.
p < .05.
No significant differences were observed in the delayed effects between the three different interventions in the current study. Surprisingly, mathematics achievement significantly decreased in experimental group 2 from posttest to delayed posttest, indicating that the intervention of PaGamO with postgame discussion in this group was disadvantageous to learning retention. The finding of the current study was not consistent with several research studies statements that demonstrated that the integration of digital educational games was more effective than the traditional method of improving students’ long-term knowledge retention (Y. T. C. Yang & Chang, 2013; K. H. Yang & Chen, 2021) but was partly similar to one previous research (Rondon et al., 2013). The current research showed that even digital educational games are more beneficial in promoting short-term knowledge gain than traditional methods of paper-and-pencil practice (Krop et al., 2023), but in some circumstances, they do not improve long-term knowledge retention.
The current result agreed with the systematic literature review on game-based learning, which highlighted the need for increased focus on enhancing knowledge retention, as studies have reported inconsistent experimental results regarding the long-term impact of game-based learning on students’ knowledge retention (Tavares, 2022). Rondon et al. (2013) discussed this result that paper-and-pencil practice could provide more opportunities for pausing, resuming, and covering the ideas presented, which might support students’ long-term knowledge retention. Requiring higher self-reflecting levels, students who learned with digital educational games would put more effort into monitoring their own previous performance than just engaging in the games. In the present research, students were immersed in territory building, occupying, and conquering tasks that required solving as many mathematics problems as possible to expand their sphere of influence in the game. In these circumstances, with the limitation of time, students might put the majority of their effort into solving problems instead of reflecting on the mistaken items, which may not be beneficial for their knowledge retention. On top of that, some authors have explained these results based on cognitive load theory (Chandler & Sweller, 1991), which speculated that learners who use digital time-limited games tend to deal with more information at once compared to the traditional method, straining their memory capacity. Hence, it may cause disadvantages in students’ learning retention, especially in short periods (Rondon et al., 2013). PaGamO, which is considered well-designed and has rich game mechanics, may potentially heighten students’ cognitive load and cause a disadvantage in students’ knowledge retention (S. W. Chen et al., 2019). In addition, students in experimental group 2, who learned through PaGamO with post-game group discussion, were required to handle an overwhelming amount of information and knowledge in a short period of 15 min (Peters & Vissers, 2004). Hence, this might explain the negative delayed effect on the experimental groups’ mathematics achievement learning, which required more attention to these unexpected findings.
Perception of PaGamO Among Students (RQ3)
Table 8 illustrates a summary of representative quotes from the semi-structured interviews conducted with the students from the experimental group 1 and 2. The first category of the response of the interview is “learning support,” which emerged with many students who described how they felt about the incorporation of PaGamO into their mathematics course and how they believed it supported their learning both in cognitive and affective dimensions. These comments highlighted the critical and positive aspects of how PaGamO is considered an interesting learning tool for supporting students’ mathematics learning. In addition, some students mentioned that PaGamO is a novel drill practicing tools compared to traditional approaches. These results indicated that PaGamO could provide students with iterative practice-and-drill experience, allow students to identify gaps in their knowledge, deepen students’ cognition of the topics, increase motivation, and facilitate discussion among peers.
Summary of the Semi-Structured Interview.
The second category is defined by several students having positive perceptions of the game mechanics of PaGamO. These mechanics motivated students to learn mathematics and increased their intention to use the system in other disciplines for learning. The mechanics of PaGamO include building, occupying, and conquering territories and purchasing materials and props. According to the students, these mechanics created a competitive and challenging learning environment, motivating the students to learn. Additionally, the game mechanics led to increased peer interaction, which created an interesting and enjoyable class atmosphere that is beneficial for motivating students’ learning.
To discover some challenges that students might experience to gain a deeper understanding of students’ negative perceptions of PaGamO. It was found that some of the students mentioned some difficulties they faced while using PaGamO in mathematics courses. The study found some shortcomings of PaGamO, which were emphasized in students’ comments, including insufficient energy, lack of calculation function and instabilities of the learning system.
Both positive and negative perceptions were observed among the students regarding the incorporation of PaGamO into their mathematics course. Several students had a positive perception of the game and stated that it supported their learning. This finding is consistent with those of other studies (Cheung & Ng, 2021; Law et al., 2020; S. F. Yang & OuYang, 2019). DGBL enables students to learn from their mistakes, receive immediate feedback, and obtain iterative practice-and-drill experience (Hwa, 2018). These characteristics were identified by the students interviewed in the present study, and these findings indicate that PaGamO is an effective and motivational digital educational game that can be used for mathematics education.
The participants viewed the game mechanics of PaGamO positively and had positive intentions to use the game in the future. The significant educational advantages of PaGamO as a learning tool were highlighted through the participants’ interviews of the study, including offering iterative practice-and-drill experience, supporting students to identify gaps in their knowledge, deepening students’ cognition of the mathematics topics, promoting learning motivation, and increasing peer interactions. These positive findings are consistent with that of S. F. Yang and OuYang (2019) but inconsistent with that of Cheung and Ng (2021). In the research by S. F. Yang and OuYang (2019), seventh-grade students had a highly positive intention to use PaGamO. By contrast, in research by Cheung and Ng (2021), students aged approximately 21 found PaGamO to be monotonous, tedious, and unmotivating and had harmful intentions to use the game in the future. These findings provide an initial understanding of how the age of students affects the suitability of PaGamO, suggesting that further research is required to clarify this point (Cheung & Ng, 2021). Last, functional problems were identified, such as insufficient energy within the game, a lack of calculation functions, buggy app implementation, and unstable Internet. These problems highlight the difficulties of incorporating PaGamO into mathematics education, which should require more attention in this field.
The current findings were partially consistent with our hypothesis, which determined the more considerable effectiveness of incorporating PaGamO to subtitle traditional pen-and-paper drills in elementary mathematics courses. This result is also supported by our qualitative data, which showed students’ positive perceptions of engaging in mathematical problem-solving in PaGamO compared to traditional practicing approaches. However, the current study revealed the integration of PaGamO with postgame group discussions into the mathematics course was disadvantageous for students’ knowledge retention. It offered us valuable insight into the importance of paying attention to students’ cognitive load in the DGBL environment. Appropriate time distribution for gameplay and debriefing stages in instructional designs is crucial for maximizing the effectiveness of incorporating DGBL activities in enhancing students’ knowledge retention. These findings suggested digital drill-and-practice educational games with well-design game mechanisms could potentially replace traditional practicing approaches and offer higher intrinsic motivation, but the advisable design of the time distribution for gameplay/debriefing stages and timing/length of debriefing forms should be considered indispensable for effective DGBL. In addition, some technical issues of the game might affect students’ experience and knowledge gain, which should be cautiously considered before implementation.
Conclusion
This study investigated whether incorporating PaGamO into a mathematics course improved learning achievement. The study also examined whether postgame group discussions influenced the effects of PaGamO. PaGamO was compared with traditional pen-and-paper drills. Sixth-grade students from an elementary school were recruited. Given the limited empirical evidence in this domain, the study adopted a quasi-experimental framework, including quantitative and qualitative data collection strategies, to holistically investigate the effectiveness of deploying PaGamO in elementary mathematics education. PaGamO is recognized for its ability to enhance cognitive skills like learning achievement and teamwork, as well as affective outcomes like enjoyment (Cheung & Ng, 2021; Law et al., 2020; S. F. Yang & OuYang, 2019). However, existing studies often lack control groups, limiting confidence in attributing these improvements to PaGamO. While previous research found that PaGamO performance predicted success in multiple-choice math questions, its broader effects on learning outcomes remain unclear (Cheung & Ng, 2021). The current study recruited three groups for implementing different treatments and further explored both knowledge acquisition and retention which contributed to practical pedagogical implementation. The result showed that PaGamO significantly improved mathematics learning achievement but did not improve learning retention based on some possible reasons as follows which could be considered for noticing and modifying in both future research and practical implementations.
Future studies on the current topic are recommended as follows. Firstly, some human factors such as learning preference, learning motivation, flow, and cognitive load, which are considered essential in DGBL, could be investigated in future works of PaGamO (Bilgin et al., 2015; Y. T. C. Yang & Chang, 2013). These approaches may potentially produce valuable results which could provide a more comprehensive understanding of this area. Given the present findings, there is no difference in the effectiveness of students’ learning outcomes that they learned with or without post-game group discussion after implementing PaGamO in mathematics courses. Another possible area of future research would be to investigate other types of debriefing activities after implementing PaGamO, which take into account the debriefing forms, timing, and length of the familiarity between participants (Peters & Vissers, 2004). These might potentially have valuable implications for this topic.
This research broadens the current understanding of the effect and delayed effect of PaGamO and postgame group discussions on learning achievement among students in elementary mathematics courses. In addition, several findings that are inconsistent with our hypothesis are valuable for alerting educators and researchers to pay more attention and effort to develop the appropriate instructional design and postgame debriefing activities in game-based learning activities, which practically contributed to the DGBL research field. We recommend incorporating PaGamO into mathematics courses as a replacement for traditional pen-and-paper drills. Debriefing activities should be carefully designed to ensure the optimal effectiveness of this learning approach.
Implications
This study offered the insight that digital game-based practice should be highly recommended to replace traditional drill methods. The current study implemented PaGamO, a digital educational drill-and-practice game with engaging game mechanisms, which was shown to have a better impact on students’ learning outcomes compared to traditional paper-and-pencil practice, as evidenced by both quantitative and qualitative analyses. Although the combination of game-based learning and group discussion did not significantly improve students’ short-term learning effectiveness or long-term memory retention, this approach has the potential to bring changes to the classroom, despite various external interference factors.
First, generally speaking, regardless of the subject, elementary students clearly enjoy immersing themselves in game-based learning. However, teachers must ensure that each student is familiar with the game’s operations and is provided with enough time during instruction to prevent additional learning stress and frustration. PaGamO is considered to be a well-designed educational game with multiple functions, gameplay elements, and main/sub-missions, but it requires a more adaptive phase and sufficient time for play and debriefing to avoid excessive cognitive load in the game-based learning process.
Second, some research has suggested that while game-based learning is individually oriented, group discussion is a collective activity that can encourage each student to share their learning experiences and engage more in game-based learning activities. This verbal expression can further strengthen students’ learning memory. However, considering that PaGamO is mainly conducted in single-player mode in the current research, providing sufficient time and space for self-debriefing might contribute to students’ knowledge acquisition and retention, an aspect that should be emphasized in the future implementation of different types of educational games. The time allocated for game-based learning, especially for structurally complex educational games like PaGamO, should be extended to provide students with ample opportunities to practice classroom knowledge through games, thereby facilitating a deeper integration of theory and practice.
Lastly, the game tools used in this study could replace traditional paper-and-pencil assessments as alternative evaluation methods. PaGamO is equipped with learning analysis functions that can collect, store, and visualize students’ data logs and knowledge blind spots to provide teachers and students with effective and efficient identification of learning obstacles, making this one of its significant advantages compared to traditional drill methods.
