Designing Effective Hybrid Course Curriculum: A Design Science Approach to Gamification and Student Outcomes Validation

Abstract

In the modern educational landscape, the integration of gamification into hybrid learning environments has emerged as a promising approach to enhance student outcomes. However, there remains a lack of comprehensive frameworks for designing gamified hybrid courses and validating their impact on student outcomes. This paper proposes a design science-based approach to gamified course design in hybrid learning contexts. Drawing on the principles of design science research, we developed a framework for designing a gamified hybrid course curriculum that incorporates course content, activities, and assessments based on four elements of gamification (achievement elements, utilitarian value, hedonic benefits, and competition). To validate the effectiveness of our approach, we conducted a study with 294 students enrolled in a hybrid business course that implemented the proposed gamification framework. Our findings indicate that all gamification elements of our proposed gamified hybrid courses positively enhance student engagement, achievement, and satisfaction. Ultimately, this paper not only contributes to the ‘gamification in education’ literature by providing a comprehensive framework for designing engaging and effective hybrid courses but also proposes a roadmap for the application of design science to embed gamification in business course curriculum design within the context of modern hybrid learning environments.

Keywords

gamification hybrid learning student engagement satisfaction design science Vietnam

Highlight

• This study provides a new way to design the course by applying design science to solve the learning and teaching problems in the business course.

• The relationship between the gamification element and student engagement was studied to show an impact on the behaviour and learning outcomes of students in higher education.

• The study’s findings indicate that the four gamification elements have a positive impact on student engagement in a hybrid learning model and indirectly influence student academic achievements and satisfaction.

• The course’s learning roadmap proposed in this study will address the gap between curriculum design and student engagement by creating a student performance-tracking dashboard.

Introduction

After a prolonged period of being affected by the COVID-19 epidemic, while online learning offers flexibility and accessibility, it also presents challenges such as reduced student engagement and motivation compared to face-to-face instruction (Al-Hafdi & Alhalafawy, 2024). Even before and after the COVID-19 pandemic, research indicated that a significant percentage of students lacked concentration or interest in online learning (Gavriluță et al., 2022). These challenges persist beyond the pandemic, highlighting the ongoing need for innovative teaching methods that can combine remote and in-person learning to enhance student engagement and motivation (Nichols et al., 2022; Servant-Miklos, 2022; Tolppanen et al., 2022), which is called ‘hybrid learning’. Hybrid learning is a teaching and learning approach that combines in-person and virtual learning via the use of hybrid classroom resources such as learning management systems (LMS), video conferencing, and asynchronous (self-paced) learning (DuBois et al., 2019; Merritt et al., 2022; Nichols et al., 2022). Universities, like other service firms worldwide, are rapidly embracing digitalisation, marking a significant shift from traditional to modern computing-based information and communication technologies (Hoang & Le, 2024). This transition is accompanied by the increasing adoption of hybrid learning techniques (Merritt et al., 2022; Nichols et al., 2022).

Simultaneously, the notions of gamification and game-based learning are two persisting tactical teaching trends due to their ability to facilitate a more intuitive and engaging learning experience, along with the growing popularity of esports and the constant inclusion of new classroom equipment (Al-Hafdi & Alhalafawy, 2024; Navarro-Espinosa et al., 2022). Gamification of learning is an educational approach that uses game aspects and design in both undergraduate and postgraduate learning settings to inspire students (van Gaalen et al., 2021). For example, Kahoot! embodies an innovative evolution of undergraduate student response systems that emphasises gamification to elevate student motivation and engagement (Wang et al., 2016). This platform is ideal for raising students’ interest, engagement, and satisfaction as well as evaluating their comprehension of a course (Lin et al., 2018). Moreover, gamification in postgraduate teaching has proved to be an effective tool to promote more student engagement, learner attrition, and knowledge retention (Nevin et al., 2014; Raju et al., 2021); enhance learning progression and social connectedness (Haque et al., 2017); and possibly create more motivation for learners (Navarro-Espinosa et al., 2022). Besides, gamification is becoming a valuable tool for educators because of the rising focus on student participation and the opportunities given by digital learning (Dunn & Kennedy, 2019). Compared to traditional classes, 67% of students found gamified learning more engaging and motivating (Chapman & Rich, 2018). Race and Powell (2000) also demonstrated the interplay between the good design of teaching activities and the cooperation of students in the learning process, reflected in the quality of learning. It is evident that gamification and game-based learning have become ‘trendy’ teaching strategies as they contribute to making undergraduate and postgraduate learning more intuitive and engaging (Chapman & Rich, 2018; Lin et al., 2018; Nevin et al., 2014; Oliveira et al., 2023), yet their effectiveness in various settings is still unclear, often resulting in unsatisfactory learning outcomes. For instance, gamification has mostly been studied in in-person or virtual learning environments (Oliveira et al., 2023; Sanchez et al., 2020), overlooking hybrid learning’s particularities (O’Byrne & Pytash, 2015). Due to its flexibility and global reach, hybrid courses in higher education have gained better ground (Raes et al., 2020; Singh et al., 2021). However, actively incorporating students in such contexts remains difficult. In conventional settings, gamification can boost student motivation and engagement (Raju et al., 2021), but its full potential in hybrid courses has yet to be confirmed empirically.

The objective of this study is to fill this gap by investigating the effects of incorporating gamification elements, such as competition, utilitarian value, achievement elements, and hedonic benefits, on student engagement, academic achievement, and satisfaction that hope to bring similar successes in the context of hybrid learning. Moreover, we propose to develop a suitable ‘roadmap’ model for the implementation of gamification in a higher education setting using design science theory. Design science (DS) has gained popularity as a research paradigm in the pedagogical realm, offering a systematic approach to creating and evaluating innovative educational artifacts for known problems (Antunes et al., 2021; Hevner et al., 2004; Thuan et al., 2019; Vom Brocke et al., 2020). Nevertheless, there is very limited research yet on applying the design science approach to improve the gamification experience of students, especially in hybrid learning settings (Leite et al., 2021). In this study, we aim to develop effective gamified hybrid learning environments by applying design science theory, which can improve the conventional curriculum through problem-solving and innovation (Henriksen et al., 2017). This will fill gaps in the literature and offer a novel approach to improving hybrid student engagement and learning outcomes in higher education. Current literature has supported that the ‘trendy’ pedagogical choice at all types of undergraduate, graduate, and PhD levels is DS education (Goldkuhl et al., 2017; Hevner, 2021; Winter & vom Brocke, 2021). We expect that this specific choice of teaching and learning approach is meant to help institutions and lecturers increase academic accomplishment and satisfaction among learners in hybrid learning environments by incorporating gamification into instructional design. Overall, the current investigation seeks to address three main research questions:

Question 1

What can be proposed as an alternative educational approach to embedding gamification in curriculum design for hybrid classes based on design science?

Question 2

When embedding gamification in curriculum design for hybrid classes, do gamification elements such as achievement elements, utilitarian value, hedonic benefits, and competition affect student engagement, and how do they affect it (negatively or positively)?

Question 3

When embedding gamification in curriculum design for hybrid classes, does student engagement positively affect academic achievement and student satisfaction in the hybrid learning context?

The main purpose of the study is to provide an explicit grasp of students’ interests in gamification elements, such as achievement elements, utilitarian value, hedonic benefits, and competition. The study additionally deals with the research questions of whether putting gamification into practice (i.e. in curriculum design for hybrid classes) will be effective through higher student satisfaction and academic achievement. Subsequently, universities and colleges can have a better understanding of their student’s needs and expectations in experiencing hybrid learning with gamification. Its implication points out to universities/colleges and lecturers whether gamification should be promisingly sustained. On that account, the suggestions supported by the research findings may guide the development and execution of gamified hybrid learning environments – ultimately augmenting the calibre of higher education.

This research adopts a case study approach, focussing on a specific university in Ho Chi Minh City, Vietnam. For confidentiality purposes, the institution will be referred to as “the University” throughout the paper.

Literature Review

Gamification

Gamification is the ‘use of game design features to inspire user behaviour in non-game environments’ (Deterding et al., 2011). Gamification, as per Domínguez et al. (2013), is the process of introducing game components into non-gaming software applications to improve user involvement, experience, and satisfaction. Gamification has been employed in myriad fields in recent years to improve employment outcomes in the development of their daily duties and jobs (Hugos, 2012; Pedreira et al., 2015). According to Gartner (2015), over half of the firms will enhance their user development procedures by 2015 since gamification provides immediate feedback, clear objectives, and tough activities. Gamification, as per Bíró (2014), has several characteristics of the theory of behaviourist learning, such as the supremacy of positive rewards, tiny step-by-step activities, instant feedback, and escalating difficulties. To affect learners’ behaviour, the utilisation of game-like rule structures, player experiences, and cultural roles are recommended in educational gamification (Su & Cheng, 2013).

Gamification is regarded as an essential novel approach to ensure both undergraduate and postgraduate student involvement and engagement not only in traditional face-to-face learning formats but also in new digital strategies such as fully online learning environments (Antonaci et al., 2019; Johnson et al., 2014; Kyewski & Krämer, 2018). Initially thought to be a strategy for increasing interest in education rather than improving the efficiency and effectiveness of the instruction (Reigeluth, 1999), gamification is anticipated to have direct effects on ‘individual learner performance’, which is the most crucial element in way-away learning approaches. This is clarified by its instinctive characteristics as a fun and effective approach to inspiring students in education and instruction (Lee & Hammer, 2011). The general objective of gamification in traditional face-to-face and fully online learning environments is to analyse the game aspects that make computer games interesting and to adapt and apply these features in educational-instructional processes to increase student involvement and loyalty to learning and educators to teaching activities (Antonaci et al., 2019; Simões et al., 2013). This element of entertainment helps users or students to concentrate on real-world issues by utilising the motivational capacity of computer games (Lee & Hammer, 2011). Gamification tactics to motivate and engage students have thrived in traditional learning contexts like classrooms and in-person encounters (Oliveira et al., 2023). Gamification has also been used in entirely online learning settings to encourage distant learners to participate and compensate for the loss of face-to-face interaction (Antonaci et al., 2019; Kyewski & Krämer, 2018). Gamification has the potential to lower dropout rates and poor individual motivation in remote education programs that use digital technology akin to e-learning (Sun & Rueda, 2012). Gamification, one of the most cutting-edge digital strategies (Johnson et al., 2014), can be used to achieve the set goals or integrated into the curriculum over a term (McFarland, 2017) and is a powerful contemporary technique to validate student engagement and commitment. The majority of participants in gamified learning activities are found to be undergraduate students (Al-Hafdi & Alhalafawy, 2024). Notably, although there have not yet been any studies investigating the differences in the effects of gamification on learners’ outcomes between undergraduate and postgraduate students, several mentions are worth consideration. For instance, Zhang et al. (2017) suggested that certain aspects they observed in their study, such as the usage of information and gamified activities, may be similarly appealing to all students, irrespective of their age or level of study; or a review study conducted by van Gaalen et al. (2020) concluded no clear evidence that the impact of gamification activities relied on contextual variables (e.g. undergraduate vs. postgraduate learning groups). On the other hand, Schlögl et al. (2021) noted that the anticipated results of gamified teaching and learning can differ based on the learners’ proficiency level, personal limitations, and preferences. Cheong et al. (2014) suggested that undergraduate students might be more likely to play games than postgraduate students) checked at the beginning of the educational course.

Gamification has shown clear benefits in traditional and fully online contexts, but its potential in hybrid classrooms remains largely untested. Hybrid classrooms, blending in-person and online learning, require tailored instructional design approaches (O’Byrne & Pytash, 2015). In this context, integrating gamification into hybrid course curricula is crucial for maintaining student engagement across both physical and virtual settings. Studies emphasise the importance of embedding gamification elements seamlessly into hybrid courses rather than using them merely as supplementary tools (Loos & Crosby, 2017; Sailer & Sailer, 2021). When thoughtfully integrated, gamification can significantly enhance student engagement, effectiveness, and satisfaction, fostering a state of flow in learning (Csikszentmihalyi, 1990). Simply adapting traditional teaching methods to online platforms may fail to fully engage students in hybrid learning (Ng & Lo, 2022, 2023). For example, a hybrid marketing class could be enriched by incorporating virtual quests, quizzes, and collaborative puzzles, transforming the learning experience into something immersive and dynamic (Bucchiarone, 2022). As hybrid classrooms and gamification gain traction in higher education (Antonaci et al., 2019), investigating their synergies is critical. Specifically, this research seeks to empirically confirm whether embedding gamification in hybrid learning can enhance student engagement, motivation, and outcomes, ultimately informing more effective instructional practices in today’s evolving educational landscape.

Gamification in Curriculum Design

A DS Approach in Gamifying Hybrid Course Curriculum

The seven elements of the DS framework include motivation, problem statement, research questions, research approach, theory-in-use, research activities, and research artifacts, which imply the steps to apply DS to solve a particular problem (Antunes et al., 2021). These elements are the base to create a modified framework used for this study (Figure 1).

Figure 1.

A DS analytical framework to implement gamification in curriculum development.

Recognising the benefits of applying gamification to the curriculum design in the hybrid-teaching model, the research team was motivated to identify an appropriate approach that academic staff can refer to when designing a course’s curriculum. This study’s outcome also addresses the literature gap of lacking a comprehensive model to implement gamification to improve student engagement (Antonaci et al., 2019; Johnson et al., 2014; Kyewski & Krämer, 2018). The problem statement in DSR is useful for identifying the research problem of low student engagement in hybrid classes (Peffers et al., 2007).

The related DS research question in the study is ‘How to embed gamification in curriculum design for hybrid classes?’. The answer to this research question would address the problem of low student engagement in hybrid classes, and it was formulated by a design process of identifying the course’s learning outcomes (CLOs), delivery content, assessments, course activities, and the mapping and integration of these elements to achieve CLOs (Antunes et al., 2021; Henriksen et al., 2017). The exploratory research type is the approach for the study based on the DS implementation framework proposed by Thuan et al. (2019). Related to research activities, Hevner et al. (2004) suggested two main types: building and evaluating. In this study, we focused on building research activities to develop a holistic gamification-based learning roadmap that lists all learning content, assessments, and course activities in a single and simple map for better tracking and engagement. The artifact of the DS process is the holistic roadmap for the course’s curriculum design that incorporates course content, activities, and assessments (Thuan et al., 2019) based on four elements of gamification (achievement elements, utilitarian value, hedonic benefits, and competition) and various gamification-supporting applications such as learning management system (LMS – Canvas), Kahoot, Miro, and Slido.

Course’s learning roadmap

The next step is identifying the answer to the first research question, ‘How to embed gamification in curriculum design for hybrid classes?’. By applying the DS implementation framework (Thuan et al., 2019) and student preferable gamification features such as progress bar, badges, ranking, rewards, and competition (Al-Hafdi & Alhalafawy, 2024), the research team proposed a holistic course-level learning roadmap in curriculum design, as illustrated in Figure 2.

Figure 2.

Explanation of the course-level learning roadmap in curriculum design.

One of the challenges in a hybrid learning model is that most interactions are spent with an LMS system that lays out the learning process in text-based document formats via Word or websites. This type of curriculum design lacks a big-picture view with attractive visualisation, limits tracking and communicating students’ learning progress, and causes inefficient engagement (Merritt et al., 2022). Thus, the course’s learning roadmap is proposed to address this gap. The roadmap is a simple, graphical, and one-page representation of the course’s learning content, class activities, and assessments designed to achieve the course’s learning outcomes. The roadmap will serve as a student performance tracking dashboard. Each item in the roadmap will include a link back to the source of information.

As schematically presented in Figure 3, there are three main elements in a course’s design: delivery content, class activities, and assessments, which often are divided into smaller components and offered weekly throughout the course’s learning period. The course’s delivery content includes reading materials, learning slides, visual videos, links to various learning resources, textbooks, and recommended articles and news. Class activities can be organised in various formats, such as competitive quizzes, team discussions, and presentations. Assessments can be conducted in quizzes, written reports, and presentations. Gamification features with all four elements of achievement elements, utilitarian value, hedonic benefits, and competition can be embedded in the three elements of a course’s design. When students complete their reading and view the required content, they can earn rewards such as badges, which are accumulated for various individual and team competitions. The roadmap can use a progress bar to show student learning progress throughout the course’s offering time. An achievement dashboard can be utilised to display the status of all competitions with up-to-date rankings. Students’ participation and results of class activities can earn badges and will be factored into competitions and displayed in the roadmap. Assessment results will earn official grades for the course and have different performance dashboards in the roadmap. Toward the end of the course, the results of the competitions can be converted into rewards of additional official grades to the course’s final grades. The conceptual proposal for the course’s level roadmap in curriculum design is summarised in Figure 3.

Figure 3.

Gamification-based learning roadmap in curriculum design.

Hypothesis Development

In this study, our goal is to confirm the positive outcomes of a gamified embedded hybrid learning experience including student engagement, motivation, and learning outcomes in hybrid mode. We believe that the Stimulus-Organism-Response (SOR) model (Mehrabian & Russell, 1974) offers an all-encompassing framework for comprehending the impact of gamification features on student behaviours and outcomes in hybrid classrooms. The model suggests that gamification features, which are stimuli, interact with specific internal processes of students (i.e. engagement), referred to as the organism, to produce certain outcomes such as academic achievement and satisfaction with learning (Robert & John, 1982). This approach allows us to explore the complex interplay between gamification elements, student characteristics, and learning outcomes within the unique context of hybrid classes.

Firstly, achievement elements are a version of gamification to encourage motivation and involvement in various systems (Hakulinen et al., 2015). Achievement elements, such as badges, trophies, stars, levels, and progress bars, are integral components of gamified learning environments that have been shown to positively impact student engagement (Ahmad et al., 2020). Research by Hakulinen et al. (2015) has found badges of achievement elements as a prominent method to motivate and foster students to learn. More specifically, by earning badges, trophies, or advancing through levels, students receive immediate feedback on their progress and accomplishments, which can increase their motivation to engage with course materials and activities and empower students to take ownership of their learning experiences (Van Ginkel et al., 2020). In hybrid learning, due to the lack of face-to-face interaction or in situations where students may face challenges in balancing their time between in-person and online activities, these elements serve as tangible rewards for their efforts, reinforcing their commitment to the learning process (O'Byrne and Pytash, 2015), and even allow them to showcase their accomplishments to their peers, fostering a sense of community and belonging (Mendoza & Venables, 2023). Hence, in the application of games in the hybrid educational environment, we believe that it is necessary to appreciate achievements to promote student learning and reinforce their learning ability.

Achievement Elements (AM) of a gamified hybrid course have a positive influence on Student Engagement.

Next, we argue that utilitarian value (UV) is vital in boosting student engagement in gamified hybrid learning settings. UV is defined by its informational emphasis and its attention to the process of consuming, which leads to its association with efficacy and task-specific outcomes (Henry et al., 2004; Overby & Lee, 2006). In gamified hybrid courses, UV is attained by several means, including the provision of explicit learning objectives, constructive feedback on student’s progress, and tangible incentives for their accomplishments (Rios, 2021). For instance, students have the potential to earn assessment points or physical prizes when they finish game tasks, demonstrate proficiency in particular areas, or achieve specified milestones (Wolf, 2019). This feedback or achievement not only gives students a feeling of satisfaction but also improves their perception of the effectiveness of the learning experience (Yang et al., 2021). Research has also shown that UV directly affects student satisfaction and increased engagement (Aguiar-Castillo et al., 2019; Baptista & Oliveira, 2019). Thereby, we propose that:

Utilitarian value (UV) has a positive influence on Student Engagement.

According to the perspective theory of flow, game challenges create hedonic benefits (Cowley et al., 2008; Csikszentmihalyi, 2014; Sweetser & Wyeth, 2005). Hedonic value is derived from visual appeal, entertainment, escapism, and enjoyment and can stem from having fun competing with others (Wolf, 2019). In game studies, social factors are proposed as predictors of players’ attitudes and intentions, and enjoyment is regarded as the most influential factor motivating gameplay (Akbari et al., 2021; Hsu & Lu, 2007). A similar study indicated that hedonic value would drive continued engagement intention (Högberg et al., 2019). Gamified learning environments greatly enhance student engagement through hedonic rewards, which include enjoyment, fun, and satisfaction (Ng & Lo, 2023). When it comes to gamified hybrid learning, such benefits are vital to motivate students involved in both in-person and online components of their coursework. More specifically, hedonic benefits increase students’ intrinsic motivation to engage with course materials and activities (Oluwajana et al., 2019). When hybrid learning is enjoyable and fun, students are more likely to be motivated to invest time and effort in their studies, leading to higher levels of engagement and active participation in both in-person and online learning activities. This intrinsic motivation has been demonstrated in research that enjoyment foresees the likelihood of using interactive (Akbari et al., 2021; Hamari, 2015; Moon & Kim, 2001; Van der Heijden, 2004; Venkatesh, 1999). Participants who perceive delight in games (gamification) are much more plausible to be prompted to play more (Bernecker & Ninaus, 2021; Hoon et al., 2002; Huang & Cappel, 2005; Yu & Huang, 2022). Moreover, in hybrid classes, particularly during designated online periods when students may experience isolation due to the absence of in-person interaction, the use of gamified learning environments (such as a Bamboozle or Gimkit group game to identify key theories or concepts of the topic) offers students a way to connect with their peers, exchange experiences, and collaborate on group projects (Antonaci et al., 2019; Kyewski & Krämer, 2018). This, in turn, enhances their engagement and sense of belonging, allowing them to actively participate in course activities.

We believe that by creating enjoyable and immersive learning experiences, gamified learning environments motivate students to actively participate in both in-person and online components of their course, leading to higher levels of engagement, motivation, and overall learning outcomes. As a result, we suggest the following hypothesis:

Hedonic Benefits (HBs) have a positive influence on Student Engagement.

Competition persists as a predominant factor that incentivises students to partake in gamification tasks. In complex activities, a lack of skill can be compensated by competition. Competition elements, such as leaderboards, rankings, challenges, and rewards, considered one of the ten main qualities of successful game designs, could improve learning practices and outcomes as an aspect that might increase engagement (Reeves & Read, 2009; Vorderer et al., 2004). Competition elements, such as leaderboards, rankings, challenges, and awards, strongly impact student engagement in gamified hybrid learning environments (Yusof et al., 2021). Within the framework of hybrid learning, these factors are essential in fostering student motivation to actively engage in both the face-to-face and virtual aspects of their learning experience. Competition aspects provide a competitive atmosphere that drives kids to strive for excellence and surpass their classmates. Gamified hybrid learning environments enhance student participation and engagement by including leaderboards, rankings, and challenges. These features establish clear goals and objectives for students to strive toward, fostering a sense of success and appreciation for their efforts (Yusof et al., 2021). Through their presence on leaderboards, attainment of high ranks, and success in challenges, students gain social acknowledgement and validation for their achievements, so reinforcing their sense of self-esteem and accomplishment (Mendoza & Venables, 2023). This, in turn, serves as an additional incentive for them to actively participate in course materials and activities.

According to Sepehr and Head (2013), using competition as a game element to engage students in learning can be an effective strategy. Competition is important in gamification in educational settings due to its motivation for student engagement to fulfil learning objectives. According to Reeves and Read (2009), when students feel a sense of control in a gamification environment, they can be more actively engaged.

Competition (CP) has a positive influence on Student Engagement.

In this study, we chose to focus on the examination of Student Academic Achievement (SAA) rather than pure academic performance. While students’ academic performance has always been recognised as a decisive outcome of student engagement and is often used to anticipate an educational system’s success, evaluate school performance, and provide a general overview of a student’s overall academic abilities and accomplishments (Zhao et al., 2005), SAA focuses specifically on the attainment of learning objectives and standards within individual subjects or courses and is often assessed qualitatively based on the demonstration of knowledge, skills, and competencies (Phelps, 2019). Since the context of our study is mainly based on a gamified hybrid course, it would be more reasonable to assess SAA rather than a broad overview of a student’s overall academic abilities and accomplishments of the whole study programme. In this sense, past studies have confirmed that enthusiasm, hard work, and actively engaging in learning activities can lead to success in student achievement and vice versa (Akbari et al., 2021). These claims indicate that perceived student engagement can positively enhance the academic achievement of students in gamified hybrid courses. Accordingly, the study proposes the following hypothesis:

Student Engagement has a positive influence on Student Academic Achievement.

Finally, student engagement is an influential attribute in academic satisfaction in the hybrid learning context and is attained when ‘students are successful in the learning and are pleased with their experience’ (Moore, 2010). According to Wu et al. (2010), satisfaction refers to the aggregate of how a student feels and thinks, which is derived from combining all the beneficial aspects of a hybrid learning environment. Additionally, student satisfaction, as a momentary mindset deriving from an assessment of a student’s academic experience, is a positive antecedent of student loyalty and reflects the quality of an educational system (Zeithaml, 1988). Student engagement, which helps students to do their tasks, have better results, and know the lesson better, is claimed to positively influence student satisfaction (Akbari et al., 2021). Consequently, these factors indirectly improve student satisfaction regarding the process and the result. Hypothesis 6 (H6), therefore, is proposed as follows:

Student Engagement has a positive influence on Student Satisfaction (SS).

With the combination of all the abovementioned factors, the proposed model is schematically presented in Figure 4.

Figure 4.

Research model.

Methodology

Research Design and Data Collection

The methodology we employ for this study consists of two distinct steps:

Initially, we utilised design science (DS) to include the technique of gamification into a hybrid course known as the ‘Global supply chain management’ which is a mandatory business course offered by the University. Throughout the three semesters of 2022, we implemented the new hybrid style of the course for about 2000 students.

Second, our objective is to conduct a quantitative empirical examination of the beneficial outcomes of incorporating gamification into the curriculum design for hybrid classrooms, utilising design science and analysing the complex mechanisms at play. Our study’s research object shall be the students who are participating in the gamified hybrid courses that followed our adopted DS framework and their study outcomes.

Specifically, a design science (DS) analytical framework, based on the seven core elements proposed by Thuan et al. (2019), was employed to develop a ‘roadmap’ for integrating gamification into university curriculum design. This study utilise design science research (DSR) to systematically design, synthesise, test, and evaluate educational artifacts – such as learning management systems (LMSs) and gamification strategies – that address practical educational challenges (Peffers et al., 2007). DSR’s focus on creating innovative, useful artifacts complements the behavioural sciences, which seek to uncover the underlying truths of phenomena (Hevner et al., 2004; Niederman & March, 2012). By combining these approaches, our research aims to advance the pedagogical field through the development and application of gamified educational tools.

A subsequent survey was conducted to collect empirical data for this research. Measure items are provided in Appendix 1. The target population was the abovementioned students at the University from both undergraduate and postgraduate programs. This institution was selected for the survey due to its status as one of the pioneers in implementing hybrid learning in Ho Chi Minh City during the study period. Students at this university are tech-savvy and experienced in learning gamification. The survey consisted of two phases: a preliminary survey and an official survey. At the preliminary survey stage, the research team collected data from 50 people to check the scale’s reliability. The reliability test of the scale includes testing for Omni directionality, reliability, convergent value, and discriminant validity. If necessary, the scale will be adjusted accordingly. After completing the scale, the research team conducted an official survey. As a result, we received a total of 294 responses (91.8% undergraduate and 8.2% postgraduate due to the number of ongoing classes in the studied semester) based on a non-probability (3-semester quota) sampling approach, equivalent to a response rate of 59%. In our sample of research, the student’s gender is relatively balanced between the two sexes (42.2% and 57.8%). This helps to determine the factors that influence intentional behaviour across genders. Gamification features that affect students the most are ranking and rewards (202 and 203 votes), followed by Competition, Badges, and Collaboration, with a high selection of 157, 150, and 148, respectively. The remaining features are Interaction, Progress bars, and Levels, which also received fewer choices (122, 113, and 96, respectively), referring to Figure 5.

Figure 5.

Profiling of respondents.

Data Analysis

The data was then imported into Smart PLS software version 3.3.7 and the partial least squares structural equation modelling (PLS-SEM) method has opted for this empirical study, as the PLS-SEM method was recommended for the empirical research that aims to examine complex and intricate relationships or seeking theory testing developed on established theoretical foundations.

Assessment of the Indicator Reliability

The factor loading coefficient index is used to measure the reliability of the scale. Scales with factor loading coefficients less than 0.4 should be removed from the research model, whilst strong factor loadings ( $\geq$ 0.7) should be kept. For scales with factor loading coefficients from 0.4 to less than 0.7, the removal of the scale is only done if this removal increases the coefficients of the composite reliability coefficient or the coefficients of the mean extracted variance (Bagozzi et al., 1991; Hair et al., 2011).

After many runs of the ‘PLS Algorithm’ function in Smart PLS software, the scales in the model all have factor loading coefficients greater than 0.4 and strong, and only 3 scales, SE2, SS2, and UV2, are below 0.7. However, it does not affect the coefficients of the coefficient composite reliability or mean extracted variance coefficients; it should be kept in the model as all the scales are reliable.

Assessment of Discriminant Validity

Discriminant validity assessment is based on two indicators: Cross-loading and Fornell–Larcker (Hair Jr. et al., 2017).

With the cross-loading index, the observed variable must have the largest outer loading coefficients on the scale that the observed variable reflects compared to any other scale (Hair Jr. et al., 2017). In Table 1, all observed variables have outer loading coefficients with the scale that the observed variable reflects larger than all other scales. All scales reflect discriminant validity.

Table 1.

Cross-Loading Coefficients.

	AM	CP	HB	SAA	SE	SS	UV
AM3	0.937	0.018	0.053	0.054	0.079	0.195	−0.123
AM4	0.815	−0.03	0.065	−0.015	0.047	0.188	−0.117
CP1	0.012	0.715	0.31	0.411	0.302	0.432	0.38
CP2	−0.029	0.593	0.226	0.323	0.301	0.237	0.21
CP3	0.004	0.787	0.294	0.47	0.454	0.387	0.411
CP4	−0.012	0.809	0.282	0.455	0.44	0.381	0.452
CP5	0.015	0.774	0.296	0.533	0.532	0.466	0.468
HB1	0.092	0.17	0.748	0.225	0.206	0.366	0.078
HB2	0.118	0.09	0.714	0.119	0.146	0.267	0.012
HB3	0.04	0.152	0.709	0.192	0.182	0.279	0.062
HB4	0.012	0.422	0.741	0.548	0.507	0.515	0.472
SAA1	0.036	0.554	0.455	0.822	0.596	0.556	0.501
SAA2	0.017	0.402	0.351	0.687	0.512	0.443	0.387
SAA3	0.001	0.397	0.371	0.752	0.539	0.509	0.432
SAA4	0.04	0.49	0.33	0.8	0.533	0.538	0.439
SE1	−0.011	0.425	0.329	0.552	0.742	0.436	0.522
SE2	0.029	0.33	0.24	0.451	0.674	0.411	0.383
SE3	0.067	0.468	0.369	0.562	0.786	0.542	0.457
SE4	0.126	0.445	0.415	0.534	0.752	0.587	0.443
SS1	0.039	0.399	0.425	0.487	0.451	0.729	0.407
SS2	0.247	0.342	0.389	0.438	0.451	0.686	0.351
SS3	0.237	0.474	0.479	0.552	0.571	0.816	0.423
SS4	0.137	0.39	0.43	0.572	0.587	0.84	0.435
UV1	−0.106	0.447	0.405	0.416	0.44	0.496	0.732
UV2	−0.112	0.177	0.08	0.248	0.288	0.216	0.517
UV3	−0.145	0.373	0.167	0.438	0.483	0.364	0.8
UV4	−0.035	0.466	0.305	0.493	0.492	0.384	0.755

When the indicator loadings were compared to other reflecting indicators in the cross-loading, discriminant validity was determined. Fornell and Larcker (1981) advocated using AVE with a score of 0.50 or more as a general guideline for assessing discriminant validity. For adequate discriminant validity, the square root of the AVE should be greater than the correlations among latent constructs (Fornell & Larcker, 1981). The Fornell–Larcker index compares the square root of a factor’s AVE with the correlation index of another factor. In particular, the square root of the AVE of a factor must be greater than the maximum correlation value with any other factor. In Table 2, the outcome from SmartPLS 3.3.7 shows that all the square root values of the AVE of the factors are larger than the correlation values of other factors.

Table 2.

Fornell–Larcker Coefficients.

	AM	CP	HB	SAA	SE	SS	UV
AM	0.878
CP	−0.000	0.74
HB	0.065	0.38	0.728
SAA	0.031	0.604	0.494	0.767
SE	0.075	0.569	0.464	0.712	0.739
SS	0.216	0.522	0.559	0.669	0.675	0.77
UV	−0.135	0.535	0.352	0.576	0.612	0.525	0.709

Assessment of the structural model

Based on our analytical results, the assessed model does not have multicollinearity when the VIFs are less than 5 (Hair Jr. et al., 2017). Next, to examine the relevance of the path coefficients, this study uses the PLS standard bootstrapping process with 5000 bootstrap samples and 294 instances (Hair Jr. et al., 2014). The estimates for the entire structural model, which contains all variables, are shown in Table 3. Based on the p-value, we can confirm that all the hypotheses are accepted. The p-value, mainly used to determine significance, is interpreted as the probability of errors in a hypothesis rejection (Hair Jr. et al., 2017). With all p-values less than 5%, it can be concluded that the hypotheses in Table 3 are statistically significant.

Table 3.

Structural Model Assessment Direct Relationship.

Hyp	Relation	Original sample (β)	Sample mean (M)	Standard deviation (STDEV)	T-value	p values	Findings
H1	AM -> SE	0.117	0.114	0.052	2.261	0.024	Supported
H2	UV -> SE	0.409	0.408	0.059	6.969	0.000	Supported
H3	HB -> SE	0.210	0.211	0.046	4.612	0.000	Supported
H4	CP -> SE	0.270	0.272	0.062	4.331	0.000	Supported
H5	SE -> SAA	0.712	0.715	0.037	19.196	0.000	Supported
H6	SE -> SS	0.675	0.677	0.040	16.985	0.000	Supported

Specifically, AM had a positive effect on SE (β = 0.117, t = 2.261, p < .05). The variable UV has a positive effect on SE, with the index of H2 being β = 0.409, t = 6.969, p < .05. Variables HB and CP also have similar results (β = 0.210, t = 4.612, p < .05 and β = 0.270, t = 4.331, p < .05). With H5, the value of β = 0.712, t = 19,196, p < .05 shows that SE has a positive influence on SAA, so the hypothesis is accepted. Finally, hypothesis H6 with variable SE has a positive effect with values of β = 0.675, t = 16,985, p < .05 supported this effect.

In Table 4, the adjusted R squared of the variable ‘Student Engagement’ is 0.504, so the independent variables explained 50.4% of the variation of the variable ‘Student Engagement’. The remaining 49.6 % is from systematic error and other factors outside the model. In addition, the R square adjusted for ‘Student Academic Achievement’ and ‘Student Satisfaction’ is 0.506 and 0.453, respectively, which means that the variable ‘Student Engagement’ explained 50.6% of the variation of the variable ‘Student Academic Achievement’ and 45.3% variation of the variable ‘Student Satisfaction'.

Table 4.

R² Value of Dependent Variables.

	R square	R square adjusted
Student Academic Achievement	0.507	0.506
Student Engagement	0.510	0.504
Student Satisfaction	0.455	0.453

Discussion

Impact of Gamification Elements on Student Engagement, Achievement, and Satisfaction

After successfully embedding gamification in curriculum design for the focal hybrid course (previously discussed in 4.1.), the study has positively applied the SOR theory to validate the impact of gamification elements (as stimuli), including achievement elements, utilitarian value, hedonic benefits, and competition, as well as the mediating impact of student engagement (organism) and student achievement and satisfaction (responses).

More specifically, our research shows that UV plays a key part in getting students to become more engaged (with the largest effect size of 0.409 among all gamification elements). Students were highly inspired by UV owing to its focus on efficiency and specific outcomes for each gamified activity. Not only did UV assist students, but it also additionally made learning appear worthwhile by offering them clear goals, relevant feedback, and explicit prizes. Consistent with earlier studies (Aguiar-Castillo et al., 2019; Baptista & Oliveira, 2019; Yang et al., 2021), our finding confirms that UV plays a crucial role in gamified learning settings in increasing the satisfaction and engagement of students in hybrid mode.

In addition, our study highlights how both hedonic benefits and competition elements could fairly boost student engagement and satisfaction (with smaller effect sizes of 0.210 and 0.270). Gamification features, such as stories, rankings, challenges, and incentives, make learning more engaging and immersive, which in turn increases student engagement in both online and in-person learning. This finding is consistent with previous research that students are more engaged and motivated in learning environments when there is some level of competition and hedonic benefits in gamified contexts (Reeves & Read, 2009; Sepehr & Head, 2013; Yusof et al., 2021). Achievement elements, such as badges, trophies, stars, levels, and progress bars, also bring significant impacts, despite the smallest effects on promoting student engagement empirically shown. Moreover, gamified achievement elements, when designed properly, may boost motivation and performance, as Groening and Binnewies (2019) confirmed.

Overall, our study presents empirical evidence that gamification components can improve hybrid course engagement, achievement, and satisfaction. HEI educators may develop more dynamic and immersive gamified learning experiences that promote active involvement, clear goals, and improved learning outcomes among students by incorporating gamification into the design of hybrid learning environments as proposed by our study (see Figures 1 and 2).

Implications of the Study

In this study, the factors applied in gamification have an impact on the behaviour, learning outcomes, and satisfaction of students in higher education. Hence, student insights are drawn for educational administrators to implement appropriate training policies and curriculum design to increase student engagement, performance, and satisfaction. First, lecturers and instructors can convey useful information in familiar traditional forms, such as sentences and pictures, in a trending way to provoke interest from students, who are most sensitive to trends. In addition, learning through traditional books and reading materials can cause boredom, dizziness, and distraction. Therefore, applying more game elements will attract students’ interest in the subject. Additionally, lecturers and instructors can take advantage of the novelty of gamification to guide students to simplify knowledge from subject topics into real life because some knowledge is burdensome and hard to grasp. Lecturers and instructors can build a course-level roadmap with effective gamification features, including a progress bar, badges, ranking, rewards, and competition to provide an attractive learning progress overview and improve student engagement.

Another important factor that should be mentioned is competition. Competition with an impact coefficient of β = 0.270 is considered to have an important effect, after utilitarian value, on students’ active participation, which supports hypothesis H4. According to the result of the items’ outer weight in the variable, CP4: ‘Competition increases my participation in the game’ has the highest outer weight of 0.809. According to the World Economic Forum’s Global Competitiveness Report in the 4.0 period in 2019, Vietnam ranked 97th out of 141 countries in the ‘Skills’ pillar in the ‘Capability’ group. It can be said that the issue of competitiveness is one of the hardships for Vietnamese students in the 4.0-era labour market. Therefore, the research team put forward the second proposal: Universities and administrators need to approve the design of gamification to enhance competitiveness combined with organised lectures to create a competitive learning environment between students on campus and outside.

More specifically, in the classroom application, instructors can design games for groups or individuals with debate. Debates prompt opportunities for students to show opinion and bravery towards the opponent, as well as the satisfaction of winning over others that incites an autonomous feeling as the need to feel free and self-directed. Another way to increase the student experience in a competitive environment is to set goals for students' so-called extinction motivation. The competition gives a sense of competence which defines the need to feel effective when students achieve their goals.

Limitations and Future Research

This research has a few shortcomings. The first limitation is that this study solely focused on students at the University instead of including all students in higher education in a hybrid learning environment in Ho Chi Minh City right after the COVID-19 epidemic. In future studies, researchers could expand the number of participants and vary the sampling process. Face-to-face interviews using a paper survey, going to places where the survey’s segmented people are aimed at, or appealing and useful presents for respondents are just a few examples. This would give more comprehensive responses and samples, allowing for more reliable research.

The lack of qualitative expert viewpoints is the study’s second shortcoming. Many HE curriculum development experts in this discipline were unable to participate in the survey due to time limits. Furthermore, because the study was conducted online, some responses were uncertain. As a result, many participants answered in a shallow and unfocused manner, considerably influencing the research findings. Furthermore, many schools use gamification in the classroom, particularly in the context of hybrid learning; hence, further research should be undertaken in different educational contexts, perhaps across various disciplinary areas, types of HEIs, regional areas, or cross-cultures to validate the generalisability of our framework. It can complement our study results and ensure variety and distinction in identifying the influential aspects of student behaviour.

The third limitation is related to the course’s learning roadmap, which is only a conceptual proposal to implement gamification in curriculum design based on DS. A practical framework with clear and up-to-date gamification-supported applications could be developed. Researchers can further investigate, propose, and validate an implementation process based on DS to assist educators in designing a highly engaging curriculum to improve student academic performance and satisfaction.

Conclusion

Most of today’s students are digital natives, having grown up with digital technologies. Important issues referring to the adaptation of the learning process for students with diverse learning styles and new guidelines for teaching and learning must be resolved by lecturers and instructors. Especially post-COVID-19, it is critical to design innovative teaching tactics that increase students’ motivation and commitment while also maximising their knowledge acquisition. Among the many approaches, gamification has gained the attention of educators, who have been investigating its potential to increase student learning in recent years (Dichev & Dicheva, 2017; Koivisto & Hamari, 2019; Majuri et al., 2018). Gamification is an effective method for promoting positive changes in the behaviour and attitude of students toward learning, thereby increasing their motivation and engagement. In addition to influencing students’ performance and comprehension of pedagogical material, the change can also enable an effective learning process. This study has partly clarified why gamification elements (achievement elements, utilitarian value, hedonic benefits, and competition) applied to learning effect student engagement, performance, and satisfaction in the hybrid learning context. The study also examined the feasibility of incorporating gamification and design science into the teaching methodology. In this paper, we consider gamification an umbrella term for incorporating various games and other interactions to increase student engagement.

In summary, this study attempted to determine factors that improve student engagement, performance, and satisfaction via gamification elements in the hybrid learning context. Referring to a broad literature review of the Stimulus-Organism-Response model (SOR model), a comprehensive integration of cognitive and emotional states of individual variables affecting student engagement, performance, and satisfaction via gamification elements in a hybrid learning context has been proposed and confirmed. Based on the design science approach, a course-level learning roadmap was recommended to provide students with a gamification-based holistic view of the course learning progress. As a result, it would improve student engagement, academic achievement, and satisfaction.

Footnotes

Acknowledgments

The work was supported (1) by the Japanese Government via the project “An Empirical Study on Services Value Chain based on the Experiential and Credibility Values” (Grant-in-Aid for Scientific Research (A) No.25240050), (2) by Japan International Cooperation Agency (JICA) through AUN/SEED-Net Project, No. 022674.242.2015/JICA-AC; and (3) by Japan Society for the Promotion of Science (JSPS) KAKENHI, No. JP18J11566. We gratefully acknowledge Dr Nguyen Thi Minh Thi, a former supply chain manager of Samsung Electronics Vietnam Co., Ltd and Nipro Vietnam Co., Ltd, and Ms Le Thi Cam Huong for their in-depth remarks that contribute to the completion of this paper. Special thanks to our research supporters for their tremendous efforts in data collection for the study: Nguyen Ngoc Trinh, Nguyen Quynh Nhu, Nguyen Dang Hien Nhi and Pham Thi Viet Anh.

Author Contributions

We understand that the Corresponding Author is the sole contact for the editorial process (including Editorial Manager and direct communications with the office). He is responsible for communicating with the other authors about progress, submissions of revisions, and final approval of proofs. We confirm that we have provided a current, correct email address which is accessible by the Corresponding Author.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: We wish to confirm that there are no known conflicts of interest associated with this publication. We confirm that the manuscript has been read and approved by all named authors and that there are no other persons who satisfied the criteria for authorship but are not listed. We further confirm that the order of authors listed in the manuscript has been approved by all of us. We confirm that we have given due consideration to the protection of intellectual property associated with this work and that there are no impediments to publication, including the timing of publication, with respect to intellectual property. In so doing we confirm that we have followed the regulations of our institutions concerning intellectual property.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research is funded by the University of Economics Ho Chi Minh City, Vietnam (UEH).

Ethical Statement

ORCID iDs

Thu-Hang Hoang

Huy Truong Quang

Data Availability Statement

The data are available on request from the corresponding author due to the participant privacy.

Appendix

References

Aguiar-Castillo

Clavijo-Rodriguez

De Saa-Perez

Perez-Jimenez

(2019). Gamification as an approach to promote tourist recycling behavior. Sustainability, 11(8), 2201. https://doi.org/10.3390/su11082201

Ahmad

Zeshan

Khan

M. S.

Marriam

Ali

Samreen

(2020). The impact of gamification on learning outcomes of computer science majors. ACM Transactions on Computing Education (TOCE), 20(2), 1–25. https://doi.org/10.1145/3383456

Akbari

Nguyen

H. M.

McClelland

Van Houdt

(2021). Design, implementation and academic perspectives on authentic assessment for applied business higher education in a top performing Asian economy. Education + Training, 64(1), 69–88. https://doi.org/10.1108/ET-04-2021-0121

Al-Hafdi

F. S.

Alhalafawy

W. S.

(2024). Ten years of gamification-based learning: A bibliometric analysis and systematic review. International Journal of Interactive Mobile Technologies, 18(07), 188–212. https://doi.org/10.3991/ijim.v18i07.45335

Antonaci

Klemke

Specht

(2019). The effects of gamification in online learning environments: A systematic literature review. Informatics.

Antunes

Thuan

N. H.

Johnstone

(2021). Nature and purpose of conceptual frameworks in design science. Scandinavian Journal of Information Systems, 33(2), 2. https://aisel.aisnet.org/sjis/vol33/iss2/2.

Bagozzi

R. P.

Singh

(1991). On the use of structural equation models in experimental designs: Two extensions. International Journal of Research in Marketing, 8(2), 125–140. https://doi.org/10.1016/0167-8116(91)90020-8

Baptista

Oliveira

(2019). Gamification and serious games: A literature meta-analysis and integrative model. Computers in Human Behavior, 92, 306–315. https://doi.org/10.1016/j.chb.2018.11.030

Bernecker

Ninaus

(2021). No pain, no gain? Investigating motivational mechanisms of game elements in cognitive tasks. Computers in Human Behavior, 114(2), Article 106542. https://doi.org/10.1016/j.chb.2020.106542

10.

Bilgihan

Bujisic

(2015). The effect of website features in online relationship marketing: A case of online hotel booking. Electronic Commerce Research and Applications, 14(4), 222–232. https://doi.org/10.1016/j.elerap.2014.09.001

11.

Bíró

G. I.

(2014). Didactics 2.0: A pedagogical analysis of gamification theory from a comparative perspective with a special view to the components of learning. Procedia-Social and Behavioral Sciences, 141(1), 148–151. https://doi.org/10.1016/j.sbspro.2014.05.027

12.

Bryan

R. R.

Glynn

S. M.

Kittleson

J. M.

(2011). Motivation, achievement, and advanced placement intent of high school students learning science. Science Education, 95(6), 1049–1065. https://doi.org/10.1002/sce.20462

13.

Bucchiarone

(2022). Gamification and virtual reality for digital twin learning and training: Architecture and challenges. Virtual Reality & Intelligent Hardware, 4(6), 471–486. https://doi.org/10.1016/j.vrih.2022.08.001

14.

Chapman

J. R.

Rich

P. J.

(2018). Does educational gamification improve students’ motivation? If so, which game elements work best? The Journal of Education for Business, 93(7), 315–322. https://doi.org/10.1080/08832323.2018.1490687

15.

Cheong

Filippou

Cheong

(2014). Towards the gamification of learning: Investigating student perceptions of game elements. Journal of Information Systems Education, 25(3), 233.

16.

Cowley

Charles

Black

Hickey

(2008). Toward an understanding of flow in video games. Computer Entertainment, 6(2), 1–27. https://doi.org/10.1145/1371216.1371223

17.

Csikszentmihalyi

(1990). Flow. The psychology of optimal experience. HarperPerennial.

18.

Csikszentmihalyi

(2014). Play and intrinsic rewards. In Flow and the foundations of positive psychology (pp. 135–153). Springer.

19.

Deterding

Dixon

Khaled

Nacke

(2011). From game design elements to gamefulness: Defining “gamification”. In Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments.

20.

Dichev

Dicheva

(2017). Gamifying education: What is known, what is believed and what remains uncertain: A critical review. International journal of educational technology in higher education, 14(1), 1–36. https://doi.org/10.1186/s41239-017-0042-5

21.

Domínguez

Saenz-de-Navarrete

De-Marcos

Fernández-Sanz

Pagés

Martínez-Herráiz

J.-J.

(2013). Gamifying learning experiences: Practical implications and outcomes. Computers & Education, 63(1), 380–392. https://doi.org/10.1016/j.compedu.2012.12.020

22.

DuBois

Krasny

M. E.

Russ

(2019). Online professional development for environmental educators: Strategies to foster critical thinking and social interactions. Environmental Education Research, 25(10), 1479–1494. https://doi.org/10.1080/13504622.2018.1564247

23.

Dunn

T. J.

Kennedy

(2019). Technology Enhanced Learning in higher education; motivations, engagement and academic achievement. Computers & Education, 137(104), 104–113. https://doi.org/10.1016/j.compedu.2019.04.004

24.

Fornell

Larcker

D. F.

(1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50. https://doi.org/10.2307/3151312

25.

Gartner

G. S. B.

(2015). More than 50 percent of organizations that manage innovation processes will gamify those processes.

26.

Gavriluță

Dalban

C. M.

Ioan

B. G.

(2022). Educational, emotional, and social impact of the emergency state of COVID-19 on Romanian university students. International Journal of Environmental Research and Public Health, 19(7), 3990. https://doi.org/10.3390/ijerph19073990

27.

Goldkuhl

Ågerfalk

Sjöström

(2017). A design science approach to information systems education. In International Conference on Design Science Research in Information System and Technology.

28.

Gray

J. A.

DiLoreto

(2016). The effects of student engagement, student satisfaction, and perceived learning in online learning environments. International Journal of Educational Leadership Preparation, 11(1), n1. https://files.eric.ed.gov/fulltext/EJ1103654.pdf.

29.

Groening

Binnewies

(2019). “Achievement unlocked!”-The impact of digital achievements as a gamification element on motivation and performance. Computers in Human Behavior, 97(2), 151–166. https://doi.org/10.1016/j.chb.2019.02.026

30.

Hair

J. F.

Ringle

C. M.

Sarstedt

(2011). PLS-SEM: Indeed a silver bullet. Journal of Marketing Theory and Practice, 19(2), 139–152. https://doi.org/10.2753/mtp1069-6679190202

31.

Hair

J. F.

Jr. Sarstedt

Ringle

C. M.

Gudergan

S. P.

(2017). Advanced issues in partial least squares structural equation modeling. Sage publications.

32.

Hair Jr

J. F.

Sarstedt

Hopkins

Kuppelwieser

V. G.

(2014). Partial least squares structural equation modeling (PLS-SEM): An emerging tool in business research. European Business Review, 26(2), 106–121. https://doi.org/10.1108/EBR-10-2013-0128

33.

Hakulinen

Auvinen

Korhonen

(2015). The effect of achievement badges on students’ behavior: An empirical study in a university-level computer science course. International Journal of Emerging Technologies in Learning, 10(1), 18. https://doi.org/10.3991/ijet.v10i1.4221

34.

Hamari

(2015). Gamification-motivations & effects.

35.

Haque

O’Broin

Kehoe

(2017). Using game elements to guide postgraduate research students to promote progression and social connectedness. In Proceedings of the 11th European conference on games based learning (pp. 881–889). Academic Conferences and Publishing International Limited.

36.

Henriksen

Richardson

Mehta

(2017). Design thinking: A creative approach to educational problems of practice. Thinking Skills and Creativity, 26(7), 140–153. https://doi.org/10.1016/j.tsc.2017.10.001

37.

Henry

Nigel

Linda

Kevin

(2004). Consumer behavior-A strategic approach. Houghton Mifflin Company.

38.

Hevner

A. R.

(2021). Pedagogy for doctoral seminars in design science research. In Engineering the transformation of the Enterprise (pp. 185–198). Springer.

39.

Hevner

A. R.

March

S. T.

Park

Ram

(2004). Design science in information systems research. MIS Quarterly, 28(1), 75–105. https://doi.org/10.2307/25148625

40.

Hoang

T. H.

Q. H.

(2024). A self-regulatory adapting mechanism to changing work setting: Roles of playful work design and ambidexterity. Current Research in Behavioral Sciences, 6(1), Article 100148. https://doi.org/10.1016/j.crbeha.2024.100148

41.

Högberg

Ramberg

M. O.

Gustafsson

Wästlund

(2019). Creating brand engagement through in-store gamified customer experiences. Journal of Retailing and Consumer Services, 50(September 2019), 122–130. https://doi.org/10.1016/j.jretconser.2019.05.006

42.

Hoon

Park

Yul

Moon

Chun

(2002). E-lifestyle and motives to use online games. Irish Marketing Review, 15(2), 71–72. https://core.ac.uk/download/pdf/301311019.pdf#page=74.

43.

Hsu

C.-L.

H.-P.

(2007). Consumer behavior in online game communities: A motivational factor perspective. Computers in Human Behavior, 23(3), 1642–1659. https://doi.org/10.1016/j.chb.2005.09.001

44.

Huang

Cappel

J. J.

(2005). Assessment of a web-based learning game in an information systems course. The Journal of Computer Information Systems, 45(4), 42–49. https://doi.org/10.1080/08874417.2005.11645854

45.

Hugos

(2012). Enterprise games: Using game mechanics to build a better business. O’Reilly Media, Inc.

46.

Johnson

Becker

S. A.

Estrada

Freeman

(2014). NMC horizon report: 2014 K. The New Media Consortium.

47.

Kankanhalli

Tan

B. C.

Wei

K.-K.

(2005). Contributing knowledge to electronic knowledge repositories: An empirical investigation. MIS Quarterly, 29(1), 113–143. https://doi.org/10.2307/25148670

48.

Koivisto

Hamari

(2019). The rise of motivational information systems: A review of gamification research. International Journal of Information Management, 45(191), 191–210. https://doi.org/10.1016/j.ijinfomgt.2018.10.013

49.

Kyewski

Krämer

N. C.

(2018). To gamify or not to gamify? An experimental field study of the influence of badges on motivation, activity, and performance in an online learning course. Computers & Education, 118(March 2018), 25–37. https://doi.org/10.1016/j.compedu.2017.11.006

50.

Lee

J. J.

Hammer

(2011). Gamification in education: What, how, why bother? Academic Exchange Quarterly, 15(2), 146.

51.

Leite

Santos

Rodrigues

Monteiro

Maciel

(2021). A hybrid learning approach for subjects on software development of automation systems, combining PBL, gamification and virtual reality. In Anais do XXXII Simpósio Brasileiro de Informática na Educação (pp. 113–122). SBC.

52.

Lin

D. T. A.

Ganapathy

Kaur

(2018). Kahoot! It: Gamification in higher education. Pertanika Journal of Social Sciences and Humanities, 26(1), 565–582.

53.

Loos

L. A.

Crosby

M. E.

(2017). Gamification methods in higher education. In Learning and collaboration technologies. Novel learning ecosystems: 4th international conference, LCT 2017, held as part of HCI international 2017, Vancouver, BC, Canada, July 9-14, 2017, proceedings, Part I 4 (pp. 474–486). Springer International Publishing.

54.

Majuri

Koivisto

Hamari

(2018). Gamification of education and learning: A review of empirical literature proceedings of the 2nd international GamiFIN conference, GamiFIN 2018.

55.

McFarland

(2017). Teacher perspectives on the implementation of gamification in a high school curriculum. California Lutheran University.

56.

Mehrabian

Russell

J. A.

(1974). An approach to environmental psychology. The MIT Press.

57.

Mendoza

Venables

(2023). Attributes of blended learning environments designed to foster a sense of belonging for higher education students. Journal of Information Technology Education: Research, 22, 129–156. https://doi.org/10.28945/5082

58.

Merritt

E. G.

Stern

M. J.

Powell

R. B.

Frensley

B. T.

(2022). A systematic literature review to identify evidence-based principles to improve online environmental education. Environmental Education Research, 28(5), 674–694. https://doi.org/10.1080/13504622.2022.2032610

59.

Moon

J.-W.

Kim

Y.-G.

(2001). Extending the TAM for a World-Wide-Web context. Information & Management, 38(4), 217–230. https://doi.org/10.1016/s0378-7206(00)00061-6

60.

Moore

J. C.

(2010). A synthesis of Sloan-C effective practices. Journal of Asynchronous Learning Networks, 14(3), 41. https://doi.org/10.24059/olj.v14i3.167

61.

Navarro-Espinosa

J. A.

Vaquero-Abellán

Perea-Moreno

A. J.

Pedrós-Pérez

Martínez-Jiménez

M. D. P.

Aparicio-Martínez

(2022). Gamification as a promoting tool of motivation for creating sustainable higher education institutions. International Journal of Environmental Research and Public Health, 19(5), 2599. https://doi.org/10.3390/ijerph19052599

62.

Nevin

C. R.

Westfall

A. O.

Rodriguez

J. M.

Dempsey

D. M.

Cherrington

Roy

Patel

Willig

J. H.

(2014). Gamification as a tool for enhancing graduate medical education. Postgraduate Medical Journal, 90(1070), 685–693. https://doi.org/10.1136/postgradmedj-2013-132486

63.

L. K.

C. K.

(2022). Online flipped and gamification classroom: Risks and opportunities for the academic achievement of adult sustainable learning during COVID-19 pandemic. Sustainability, 14(19), Article 12396. https://doi.org/10.3390/su141912396

64.

L. K.

C. K.

(2023). Enhancing online instructional approaches for sustainable business education in the current and post-pandemic era: An action research study of student engagement. Education Sciences, 13(1), 42. https://doi.org/10.3390/educsci13010042

65.

Nichols

B. H.

Caplow

Franzen

R. L.

McClain

L. R.

Pennisi

Tarlton

J. L.

(2022). Pandemic shift: Meeting the challenges of moving post-secondary environmental education online. Environmental Education Research, 28(1), 1–17. https://doi.org/10.1080/13504622.2021.2007220

66.

Niederman

March

S. T.

(2012). Design science and the accumulation of knowledge in the information systems discipline. ACM Transactions on Management Information Systems (TMIS), 3(1), 1–15. https://doi.org/10.1145/2151163.2151164

67.

O’Byrne

W. I.

Pytash

K. E.

(2015). Hybrid and blended learning: Modifying pedagogy across path, pace, time, and place. Journal of Adolescent & Adult Literacy, 59(2), 137–140. https://doi.org/10.1002/jaal.463

68.

Oliveira

Hamari

Shi

Toda

A. M.

Rodrigues

Palomino

P. T.

Isotani

(2023). Tailored gamification in education: A literature review and future agenda. Education and Information Technologies, 28(1), 373–406. https://doi.org/10.1007/s10639-022-11122-4

69.

Oluwajana

Nat

Idowu

Vanduhe

Fadiya

(2019). The adoption of students’ hedonic motivation system model to gamified learning environment. Journal of Theoretical and Applied Electronic Commerce Research, 14(3), 156–167. https://doi.org/10.4067/s0718-18762019000300109

70.

Overby

J. W.

Lee

E.-J.

(2006). The effects of utilitarian and hedonic online shopping value on consumer preference and intentions. Journal of Business Research, 59(10-11), 1160–1166. https://doi.org/10.1016/j.jbusres.2006.03.008

71.

Pedreira

García

Brisaboa

Piattini

(2015). Gamification in software engineering–A systematic mapping. Information and Software Technology, 57(1), 157–168. https://doi.org/10.1016/j.infsof.2014.08.007

72.

Peffers

Tuunanen

Rothenberger

M. A.

Chatterjee

(2007). A design science research methodology for information systems research. Journal of Management Information Systems, 24(3), 45–77. https://doi.org/10.2753/mis0742-1222240302

73.

Phelps

R. P.

(2019). Test frequency, stakes, and feedback in student achievement: A meta-analysis. Evaluation Review, 43(3-4), 111–151. https://doi.org/10.1177/0193841X19865628

74.

Race

K. E.

Powell

K. R.

(2000). Assessing student perceptions of classroom methods and activities in the context of an outcomes-based evaluation. Evaluation Review, 24(6), 635–646. https://doi.org/10.1177/0193841x0002400604

75.

Raes

Vanneste

Pieters

Windey

Van Den Noortgate

Depaepe

(2020). Learning and instruction in the hybrid virtual classroom: An investigation of students’ engagement and the effect of quizzes. Computers & Education, 143(January 2020), 103682. https://doi.org/10.1016/j.compedu.2019.103682

76.

Raju

Bhat

D’Souza

Singh

A. B.

(2021). Effective usage of gamification techniques to boost student engagement. Journal of Engineering Education Transformations, 34(Special issue: January 2021), 713–717. https://doi.org/10.16920/jeet/2021/v34i0/157171

77.

Reeves

Read

J. L.

(2009). Total engagement: How games and virtual worlds are changing the way people work and businesses compete. Harvard Business Press.

78.

Reigeluth

C. M.

(1999). What is instructional-design theory and how is it changing. In Instructional-design theories and models: A new paradigm of instructional theory (Vol. 2, pp. 5–29). Psychology Press.

79.

Rios

(2021). Improving test-taking effort in low-stakes group-based educational testing: A meta-analysis of interventions. Applied Measurement in Education, 34(2), 85–106. https://doi.org/10.1080/08957347.2021.1890741

80.

Robert

John

(1982). Store atmosphere: an environmental psychology approach. Journal of Retailing, 58(1), 34–57.

81.

Sailer

(2021). Gamification of in‐class activities in flipped classroom lectures. British Journal of Educational Technology, 52(1), 75–90. https://doi.org/10.1111/bjet.12948

82.

Sanchez

D. R.

Langer

Kaur

(2020). Gamification in the classroom: Examining the impact of gamified quizzes on student learning. Computers & Education, 144(January 2020), 103666. https://doi.org/10.1016/j.compedu.2019.103666

83.

Schlögl

Roller-Wirnsberger

R. E.

Hernes

S. S.

Perkisas

Bakken

M. S.

Miot

Balci

Dani

Pajulammi

Piaggi

Drenth-van Maanen

Singler

(2021). Teaching geriatric medicine through gamification: A tool for enhancing postgraduate education in geriatric medicine. Aging Clinical and Experimental Research, 34(2), 455–463. https://doi.org/10.1007/s40520-021-01933-9

84.

Sepehr

Head

(2013). Competition as an element of gamification for learning: An exploratory longitudinal investigation. In Proceedings of the First International Conference on Gameful Design, Research, and Applicationms. Association for Computing Machinery.

85.

Servant-Miklos

(2022). Environmental education and socio-ecological resilience in the COVID-19 pandemic: Lessons from educational action research. Environmental Education Research, 28(1), 18–39. https://doi.org/10.1080/13504622.2021.2022101

86.

Simões

Redondo

R. D.

Vilas

A. F.

(2013). A social gamification framework for a K-6 learning platform. Computers in Human Behavior, 29(2), 345–353. https://doi.org/10.1016/j.chb.2012.06.007

87.

Singh

Steele

Singh

(2021). Combining the best of online and face-to-face learning: Hybrid and blended learning approach for COVID-19, post vaccine, & post-pandemic world. Journal of Educational Technology Systems, 50(2), 140–171. https://doi.org/10.1177/00472395211047865

88.

C.-H.

Cheng

C.-H.

(2013). A mobile game-based insect learning system for improving the learning achievements. Procedia-Social and Behavioral Sciences, 103(26 November 2013), 42–50. https://doi.org/10.1016/j.sbspro.2013.10.305

89.

Sun

J. C. Y.

Rueda

(2012). Situational interest, computer self‐efficacy and self‐regulation: Their impact on student engagement in distance education. British Journal of Educational Technology, 43(2), 191–204. https://doi.org/10.1111/j.1467-8535.2010.01157.x

90.

Sweetser

Wyeth

(2005). GameFlow: A model for evaluating player enjoyment in games. Computer Entertainment, 3(3), 3. https://doi.org/10.1145/1077246.1077253

91.

Thuan

N. H.

Drechsler

Antunes

(2019). Construction of design science research questions. Communications of the Association for Information Systems, 44(1), 20. https://doi.org/10.17705/1CAIS.04420

92.

Tolppanen

Kang

Mayoral

(2022). Paradigm shift: Changes in willingness to take pro-environmental behavior in the midst of the COVID pandemic among European pre-service teachers. Environmental Education Research, 29(9), 1–17. https://doi.org/10.1080/13504622.2022.2105818

93.

Tomaselli

Sanchez

Brown

(2015). How to engage users through gamification: The prevalent effects of playing and mastering over competing.

94.

Van der Heijden

(2004). User acceptance of hedonic information systems. MIS Quarterly, 28(4), 695–704. https://doi.org/10.2307/25148660

95.

van Gaalen

A. E.

Brouwer

Schönrock-Adema

Bouwkamp-Timmer

Jaarsma

A. D. C.

Georgiadis

J. R.

(2021). Gamification of health professions education: A systematic review. Advances in Health Sciences Education, 26(2), 683–711. https://doi.org/10.1007/s10459-020-10000-3

96.

Van Ginkel

Ruiz

Mononen

Karaman

De Keijzer

Sitthiworachart

(2020). The impact of computer‐mediated immediate feedback on developing oral presentation skills: An exploratory study in virtual reality. Journal of Computer Assisted Learning, 36(3), 412–422. https://doi.org/10.1111/jcal.12424

97.

Venkatesh

(1999). Creation of favorable user perceptions: Exploring the role of intrinsic motivation. MIS Quarterly, 23(2), 239–260. https://doi.org/10.2307/249753

98.

Vom Brocke

Winter

Hevner

Maedche

(2020). Special issue editorial–accumulation and evolution of design knowledge in design science research: A journey through time and space. Journal of the Association for Information Systems, 21(3), 520–544. https://doi.org/10.17705/1jais.00611

99.

Vorderer

Klimmt

Ritterfeld

(2004). Enjoyment: At the heart of media entertainment. Communication Theory, 14(4), 388–408. https://doi.org/10.1093/ct/14.4.388

100.

Wang

A. I.

Zhu

Sætre

(2016). The effect of digitizing and gamifying quizzing in classrooms.

101.

Winter

vom Brocke

(2021). Teaching design science research.

102.

Wolf

(2019). Intensifying user loyalty through service gamification: Motivational experiences and their impact on hedonic and utilitarian value. ICIS.

103.

J.-H.

Tennyson

R. D.

Hsia

T.-L.

(2010). A study of student satisfaction in a blended e-learning system environment. Computers & Education, 55(1), 155–164. https://doi.org/10.1016/j.compedu.2009.12.012

104.

Yang

Chiu

M. M.

Yan

(2021). The power of teacher feedback in affecting student learning and achievement: Insights from students’ perspective. Educational Psychology, 41(7), 821–824. https://doi.org/10.1080/01443410.2021.1964855

105.

Huang

Y. T.

(2022). Why do people play games on mobile commerce platforms? An empirical study on the influence of gamification on purchase intention. Computers in Human Behavior, 126(January 2022), 106991. https://doi.org/10.1016/j.chb.2021.106991

106.

Yusof

Atan

Harun

Rosli

Abd Majid

(2021). Students engagement and development of generic skills in gamified hybrid service-learning course. International Journal of Emerging Technologies in Learning (iJET), 16(24), 220–243. https://doi.org/10.3991/ijet.v16i24.27481

107.

Zeithaml

V. A.

(1988). Consumer perceptions of price, quality, and value: A means-end model and synthesis of evidence. Journal of Marketing, 52(3), 2–22. https://doi.org/10.1177/002224298805200302

108.

Zhang

Robb

Eyerman

Goodman

(2017). Virtual worlds and gamification to increase integration of international students in higher education: An inclusive design approach. International Journal of E-Learning & Distance Education/Revue Internationale du e-learning et la formation à distance, 32(2). https://ijede.ca/index.php/jde/article/view/1057.

109.

Zhao

Lei

Lai

B. Y. C.

Tan

H. S.

(2005). What makes the difference? A practical analysis of research on the effectiveness of distance education. Teachers College Record, 107(8), 1836–1884. https://doi.org/10.1177/016146810510700812