A Mobile Contextualized Educational Game Framework With ChatGPT Interactive Scaffolding for Employee Ethics Training

ChatGPT and Chatbots for Education

ChatGPT is an artificial intelligence chat technology developed by OpenAI which is capable of prompting and answering questions on different subjects of knowledge (OpenAI, 2023). The important impact of ChatGPT on education is that it provides an opportunity for students to actively explore knowledge; students use ChatGPT and learn by chatting with it (Shalva, 2023). The current common application of ChatGPT in higher education includes knowledge exploring, grammar correction, and strategy selection in problem solving (Oğuz et al., 2024). Various studies have examined the validity of the content, such as Scheschenja et al. (2024), for patient education prior to procedures; the chance of ChatGPT giving incorrect information was below 6% for both the old and new versions of ChatGPT. In a study by Takagi et al. (2023), it was pointed out that ChatGPT’s response on medical knowledge was able to meet the standard of passing the Japanese Medical Licensing Examination. Based on the literature review, ChatGPT has a certain degree of accuracy in knowledge provision, and thus this technology has become an increasingly important research topic as a form of chatbot-assisted learning.

The integration of chatbots into learning is a trend, and Deng and Yu’s (2023) review study categorized the use of chatbots in learning into three types: tutor, teaching assistant, and partner. The tutor is used to provide questions, encourage students, and interact with each other; the teaching assistant is used as a form of scaffolding for professional knowledge and formative feedback; and the partner is used to provide text or voice chat interactions. Chatbots are not only providers of knowledge but also supporters of learning. They can be used to assist in role-playing, storytelling, and writing (Zhang et al., 2023). Chatbots have been a popular gamification element embedded in a variety of educational materials, such as González-González et al.’s (2023) study on incorporating chatroom interactions into mathematics learning to promote student engagement. Even the chatting process can be a game, as in the educational game designed by Al Kahf et al. (2023), where students gain knowledge and points by talking to a chatbot. The introduction of ChatGPT has made it easier to integrate chatbot applications with learning situations. In Maurya’s (2023) study, in order to develop students’ sales skills in a learning activity, background information about customers was given to ChatGPT as prompt data, so that the students could have conversations with the customers played by ChatGPT and learn from them.

However, there is a limitation to using ChatGPT as a direct teaching aid. The source of the data answered by ChatGPT is a black box, and it is difficult for the instructor to ensure that the content conforms to a specific textbook or guideline in terms of professional correctness. Therefore, Khadija et al. (2023) proposed a design that provided the PDF file of a textbook and then allowed ChatGPT to answer based on the PDF content, and the results show that ChatGPT can effectively summarize the content of a textbook and present it in a chat format. Chubb (2023) also used PDFs of qualitative data for research and then asked ChatGPT to focus on the data summaries, and the results showed that this approach helped to focus the use of ChatGPT on the textbooks or guides that instructors wanted to present to their students.

However, problems may be encountered when using PDFs in this way. Ayub et al. (2023) prepared the content of a medical textbook and expected to use ChatGPT to organize format-specific questions from the textbook, but found that the accuracy and complexity were not good, which is different from the results of other studies that used pre-prepared documents for ChatGPT. The study of Khadija et al. (2023) mentioned that too much text or too many tables and images in the PDFs decreased the quality of the PDFs summarized by ChatGPT.

Game-Based Learning With Scaffolding

Game-based learning, also known as learning through playing games, can interact with instructional methods or strategies such as situated learning and scaffolding theory, and promote student motivation, engagement, and positively affect learning transfer (Hou, 2023). Vygotsky (1978) proposed the Zone of Proximal Development (ZPD), which is the gap between a student’s ability to perform on his own and his ability to be assisted by an adult or peers, and the aids to help the student cross this gap are called scaffolding. In a past study, Chen et al. (2023a) incorporated different representational (text, 2D figures, and 3D models) cues as scaffolding in a geometry learning game to facilitate student learning.

Scaffolding can be presented in a way that is not only additional hints to the learning, but can also be integrated into situated learning situations. For example, Chen et al. (2023b) in a business decision-making game have students play the role of an office employee in a virtual office and communicate with the office characters or view office documents in a story situation. The textual content of the knowledge presented in these interactions can be used as scaffolding for the students. The scaffolding is presented in such a way that it feels like part of the game situation rather than an additional source of supplemental information.

Chan et al. (2023) used a history learning game in which students play the role of a person from a past period and solve puzzles in a situation to gain historical knowledge. The entire game is multiplayer, remote and synchronized, and the game also includes real-person NPCs in addition to the learners, who wear historical clothing and speak in the tone of that era through webcam and audio to provide scaffolding. In the process of the game, learners not only interact with other learners, but also interact with real-person NPCs. The results showed that such a design has a positive impact on learning, but it also costs more human resources and time.

Both the document and real-person NPC presentations of scaffolding in context have demonstrated the potential for integrating scaffolding with situations, but the design of the real-person NPCs allowed for more opportunities for students to interact socially in the game than the document presentation.

In past ChatGPT-integrated game-based learning activities, usually the ChatGPT just played the role of an answer provider (Chen & Chang, 2024). However, there is also potential for the ChatGPT to play the role of NPCs as a guide for the game, such as in Stampfl et al.’s (2024) study where the ChatGPT acted as a consultant to the management of the company, and the students were guided through a discussion with the ChatGPT. Chien et al. (2024b) used ChatGPT as a provider of clue scaffolding in a story line in a game and showed that such a design had a positive effect on students’ flow and motivation without feeling overly anxious. In Chen & Hou’s (2024) study, it was also noted that students were in a state of high flow and high motivation when interacting with role-playing ChatGPT scaffolding. Chen & Chang (2024) also noted that the integration of games with ChatGPT as scaffolding was less psychologically burdensome for students than games alone.

Therefore, using ChatGPT to role-play a NPC in a game situation is a design that should be explored, especially in single-player games, where the use of ChatGPT-based NPCs can lead to more social interaction. This study also further proposes a design framework for using ChatGPT-based NPCs as scaffolding in educational games with an emphasis on simulation (e.g., employment ethics in a workplace context), and explores its possible potentials and limitations.

Limitations of Employee Ethics Education

Employee ethics are influenced by culture and many different factors. Van Buren and Greenwood (2013) mentioned that employee ethics in some countries are influenced by the biblical culture. Employees are expected to maintain integrity and honesty at work, and employers are expected to be friendly to their employees by giving them better pay and proper working hours. However, in practice, due to the market mechanism and business interests, sometimes employers do not treat employees in a friendly way. In Berg et al. (2021), it was pointed out that etiquette is also included in the Chinese employee ethics culture, which includes different cultural characteristics such as “introducing oneself by the first name only” and “providing delicious food for others,” in addition to the common values of honesty and responsibility. Woods and Lamond (2011) mentioned that in East Asian countries, such as China, Japan, and Korea, which are influenced by Confucianism, there is a greater emphasis on self-reflection and mentoring in employee ethics. Therefore, it is important to pay attention to whether the culture and context are in line with the learner’s life situation in the first step of ethical education in the workplace.

Bairaktarova et al. (2015) found that the ability to follow employee ethics is an important part of a student’s field work, but employee ethics is currently less frequently included as a compulsory subject in universities. Allen & Simpson (2019) suggest that employee ethics is now one of the most important subjects that students, academics and business owners believe should be included in schools. There are fewer studies that discuss the strategies and methods of teaching employee ethics. A common pedagogical approach to teaching ethics in the workplace is to provide case scenarios where the learner analyzes the situation and allows the student to develop his or her own perspective and evaluate the solution (Rudnicka, 2005).

Floyd et al.’s (2013) study mentioned that there is a lack of teaching of employee ethics, and there are still studies to this day that indicate that senior employees believe that new employees’ knowledge of employee ethics is lacking (Gates et al., 2021). Floyd et al. (2013) mentioned that one of the ways to effectively teach ethics in the workplace is to directly allow students to practice in real-world situations. However, in practice, it is costly to arrange for each student to go to a real workplace for internship or training. Digital games are characterized by providing realistic situations for students to practice in the virtual world (Hou, 2023). Digital games can be used as a training tool in the classroom or before students go to the real workplace by incorporating real-life situations that match the local employee ethics and culture.

Katsarov et al. (2017) point out that presenting contexts and interactions in games that are close to the real world can facilitate students’ perception of specific ethical concepts, further influencing their identification and reflection. Schrier (2017) identifies one of the key designs for facilitating student learning in ethics education games as: allowing students to experience ethical choices and allowing students to experience or observe the outcomes of different choices. Cabellos & Pozo’s (2023) research suggests that by experiencing ethical dilemmas and making choices in games, students’ awareness of and empathy for ethical issues can be promoted. Games also provide real-time evaluation and feedback on the solutions proposed by the students. Learning about ethical issues through games is a well-established method of teaching and learning, and has a positive impact on the training of knowledge to a certain extent.

There are fewer studies discussing scaffolding in ethics instruction. Tammeleht et al. (2020) added scaffolding to support learning in ethics instruction activities and found that scaffolding was helpful for some students in understanding and thinking about the issues by having the instructor observe the discussion and give scaffolding support verbally during the discussion. However, the study also pointed out that such a scaffolding design would be less stable as the timing and content of the scaffolding would be affected by different instructors. Ng et al.’s (2024) study suggests that the situation, text, and other multimedia elements of the game elements are appropriate scaffolding for ethical learning activities. ChatGPT has the potential to be used as scaffolding and contextualization in game-based learning and can be scaffolded appropriately and with quality according to students’ questions. This study aims to investigate the learning achievement of ethical educational games combined with ChatGPT-based NPC for knowledge learning and analyze the students’ experience with this design.

Research Questions

Summarizing the literature review, this study suggests that there is potential for incorporating ChatGPT-based NPCs as a scaffolding instructional design in the previous contextualized educational game, “Newbie, swipe, ethics.” We conducted a pilot study on the game mechanism and content of the game, “Newbie, swipe, ethics” (Chen & Hou, 2023), and the results showed that students had a certain level of high motivation, high flow, and low anxiety during the game. This study aimed to investigate the effects of adding a ChatGPT-based NPC as a scaffold for learning.

Therefore, we designed a document scaffolding group, in which the scaffolding was presented only as a document in the game, and we also had a ChatGPT-based NPCs scaffolding group, in which the students talked to realistic NPCs to get the scaffolding. In this study, learning achievement was measured by pretest and posttest. As research has suggested that ChatGPT has a positive effect on learning motivation (Shalva, 2023), we wanted to investigate how ChatGPT-based NPCs scaffolding affects students. We also aimed to explore whether the addition of ChatGPT-based NPCs scaffolding to a learning game would potentially increase or decrease the flow in the game, or increase engagement or anxiety as a result of the increased interaction methods. The following research questions were posed for this study:

1. What are the performance and differences in learning achievement, flow, motivation, and anxiety between the document scaffolding group and the ChatGPT-based NPCs scaffolding group?

Research Design

The study adopted a quasi-experimental research design and was divided into a document scaffolding group and a ChatGPT-based NPCs scaffolding group. Both groups experimented with the scaffolding-based employee ethics learning game, “Newbie, Swipe, Ethics.” The document scaffolding group was provided with a readily accessible documented file of employee ethics knowledge during the game. The ChatGPT-based NPCs scaffolding group learners had three chances to talk to the NPCs played by ChatGPT during the game to gain knowledge about employee ethics. Both groups took a pretest and posttest on employee ethics knowledge and completed the flow scale, motivation scale, anxiety scale, and scaffolding using experience questionnaire.

Participants and Research Procedure

A total of 61 participants were recruited for this study which was conducted using online open recruitment. Participants were recruited from the age of 20 and above and included those who were interested in learning about employee ethics. The participants consisted of 20 males and 41 females. Since the workplace situations covered a wide range of ages and backgrounds, the ages of the participants were 20–25 (n = 22, 36.1%), 26–30 (n = 15, 24.6%), 31–35 (n = 8, 13.1%), 36–40 (n = 3, 4.9%), 41–45 (n = 5, 8.2%), 46–50 (n = 4, 6.6%), and above (n = 4, 6.6%). Other than that, the study did not restrict the participants to other backgrounds.

All participants were randomly assigned to either the document scaffolding group (n = 29) or the ChatGPT-based NPCs scaffolding group (n = 32). The research process and scales of this study were reviewed by the Office of Research Ethics, National Chengchi University, Case No. (NCCU-REC-202007-E077).

The procedure of the study is shown in Figure 1. Before the experiment, the researcher asked the participant to sign an informed consent form informing them of all their rights. The total time for the whole experiment was 70 minutes. At the beginning, both groups were asked to complete a pretest on employee ethics (20 min). Then the participants were asked to play the game (20 min). Since the scope of learning in this study is not large and only focuses on the knowledge of basic ethical concepts in the workplace, a 20-min experiential game-based learning session is more suitable for the classroom situation. The content and process of the game was the same for both groups, but the difference was in the design of the scaffold. The Document Scaffolding Group game includes a re-readable employee ethics knowledge document as a scaffold. The ChatGPT-based NPCs scaffolding group is a situational ChatGPT-based NPC, where participants could ask questions to the ChatGPT-based NPC to obtain the scaffolding, and the source of information behind the ChatGPT-based NPC was the same as the employee ethics knowledge document available to the document scaffolding group. In order to more realistically reflect the way in which participants asked questions and encountered bottlenecks, we did not conduct any pre-training for participants, who only needed to ask NPCs according to the daily dialogue to obtain the scaffolding, and the dialogue of the query itself is a prompting to the AI and will be responded to by the NPCs in a dialogue manner.OPEN IN VIEWER

At the end of the game, a posttest of knowledge (20 min) was administered. The pretest and posttest used the same questionnaire to examine the learning achievement of the participants. All groups were asked to complete the Flow, Motivation, and Anxiety scales and Scaffolding Using Experience Questionnaire (10 min) after the posttest.

Employee Ethics Learning Game – Newbie, Swipe, Ethics

“Newbie, Swipe, Ethics” is an employee ethics learning game featuring both interactive mechanisms and scaffolding design. In the game, learners need to play the role of a new employee, experience ethical dilemmas in the workplace, and make the right choices to earn more Rank Stars in the game. The interface of the game is shown in Figure 2, and the game is played by swiping cards, which is a common method used by modern dating applications such as Tinder (Dai & Robbins, 2021). In the game, students need to read the situation and make decisions by Swipe Left or Right (Figure 2).OPEN IN VIEWER

As an example in Figure 2, “My immediate supervisor has asked me to give her my newly created e-mail account at the company, but the HR Manager responsible for setting up the account has told me it will have to wait. Should I rush the HR Manager or tell the HR Manager that I will wait until he finishes setting it up?” At this time, if the player chooses to urge (swipe left), he will lose one Rank Star, if he chooses to wait (swipe right), he will get one Rank Star. In addition, there is a countdown time limit for each choice, and simulating interactions with others in a real workplace event does not allow them to wait for a long time without responding. All situations in the game were scripted in discussion with senior managers with 30 years of experience in business to ensure the validity of the content.

In the game, this study designed a virtual NPC, Yuna, as the scaffold provider, who is a best friend of the player, currently works in a human resource management company and has good knowledge related to employee ethics. In the document scaffolding group, Yuna provides scaffolding by leaving documents for players to read, and they can click on the Hint button and read learning support at any time during play (Figure 3). In the ChatGPT-based NPCs scaffolding group, every time there is an opportunity to chat with Yuna after a work day (using the ChatGPT chatbot). At the end of each workday, a chat window pops up and reminds the player that they can talk to Yuna about their work before going to bed, where they can ask questions to get more information about employment ethics and get guidance and support for the next day’s work challenges. (Figure 4).OPEN IN VIEWER

OPEN IN VIEWER

Figure 4.

ChatGPT-based NPCs Scaffolding group using NPCs scaffolding.

In the ChatGPT-based NPCs scaffolding group, Yuna, the scaffolding provider, is played by the ChatGPT-based chatbot, which is paired with ChatPDF (https://www.chatpdf.com/), an integrated application based on ChatGPT that analyzes the researcher’s pre-configured documents as a basis for answering questions (Ayub et al., 2023). In this study, following the findings of Khadija et al. (2023), documents with fewer pages and without graphs and tables were produced to reduce the interference of ChatPDF in parsing text and to improve the accuracy of ChatGPT-based NPC responses. This document is the same as the one read by the Document Scaffolding Group. As shown in Figure 5, after a player sends a question, the question will be presented as a spoken conversation by using ChatGPT’s (ChatPDF) text analytics technology to collate the information from the knowledge document and convert it into spoken text. Before the response is presented to the player, the response string is then subjected to this study’s text-substitution mechanism, which replaces certain words that do not fit the NPC’s situation, such as “According to my documentation, I can tell you about the following knowledge...,” where the word “documentation” is replaced with “According to my profession, I can tell you about the following knowledge.” There are other things like “Assistant” being replaced with “Friend” to make their responses more in line with the NPC’s characterization. Due to the ChatGPT-based NPCs scaffolding Framework was pre-loaded with knowledge documents (including contextual settings) and characterization prompts, players could simply ask questions intuitively and get answers from the NPCs as if they were friends in daily conversations.OPEN IN VIEWER

Research Tools

Employee Ethics Test

To evaluate learners’ knowledge of employee ethics concepts, in the pilot study (Chen & Hou, 2023), the researcher compiled a list of employee ethics concepts as shown in Table 1, based on a literature review and discussions with experts (senior managers with 30 years of work experience). A total of 20 situational questions were designed based on these concepts. Each question is worth 5 points and is a multiple-choice question with a full score of 100. The questions state a situation, such as: “Last week you made an appointment with your co-worker Mei to visit a client today, but Mei not only forgot, but also blamed you for not making it clear. If it were you, what specific actions would you respond to a co-worker like Mei?” and provide four action choices for students to select.

OPEN IN VIEWER

Table 1.

Employee Ethics Concepts.

Concept	Content
Return progress	Ask questions immediately and report progress instantly.
Don’t bypass the immediate leadership	Messages should follow the organizational hierarchy and should not be crossed over.
Comply with the company’s dress code	Dress appropriately according to the company’s rules and culture.
Always be polite	Respecting others.
Don’t gossip.	Avoid gossip about coworkers or managers.
Reduce parochialism	Make decisions that prioritize the interests of the work or group.

Scaffolding Using Experience Questionnaire

This study designed the questionnaire for the document scaffolding group on whether text scaffolding is useful for learning. This questionnaire included questions asking students whether scaffolding is useful for learning (on a 5-point scale), as well as how scaffolding is useful for learning, and an open-ended question on when players would read scaffolding. For students in the ChatGPT-based NPCs scaffolding group, this study designed the questionnaire and asked students about the usefulness of the ChatGPT-based NPC scaffold for learning (5-point scale). In addition, there were two open-ended questions asking players about how the scaffolding helped learning and how players interacted with the ChatGPT-based NPC.

Learning Achievement

In this study, paired-sample t tests were used to examine whether there was a significant difference between the pretest and posttest in the two groups, and to calculate effect sizes (Table 2). The results showed that the posttest of the document scaffolding group was significantly higher than the pretest (t = −4.131, p = .000 < .001) and had a small effect size (Cohen’s d = 0.37 > 0.2). The posttest of the ChatGPT-based NPCs scaffolding group was shown to be significantly higher than the pretest (t = −2.911, p = .007 < .01) and had a small effect size (Cohen’s d = 0.21 > 0.2). Levene’s test was performed on the two groups of pretests and showed that the pretests of the two groups were equal variances not assumed (F = 7.590, p = .008).

OPEN IN VIEWER

Table 2.

Paired Samples t Test of Two Groups.

Group	Pretest		Posttest		t	p	Cohen’s d
	M	SD	M	SD
Document scaffolding group (n = 29)	72.07	13.060	78.28	9.188	−4.131	.000	0.378
ChatGPT-based NPCs scaffolding group (n = 32)	69.69	12.632	73.59	14.041	−2.911	.007	0.214

The Levene’s test was performed after adjusting both sets of pretests using Square Root Transformation, and the results still showed that the pretests of both groups were equal variances not assumed (F = 7.349, p = .009). Therefore, we analyzed whether there was a difference between the pretest and posttest of the two groups using an independent sample t test (Table 3), which showed that there was no significant between-group difference between the pretest (t = −.724, p = .472) and posttest (t = −1.554, p = .126).

OPEN IN VIEWER

Table 3.

Independent t Test of the Pretest and Posttest of the Two Groups.

Group	Document scaffolding group (n = 29)		ChatGPT-based NPCs scaffolding group (n = 32)		t	p
	M	SD	M	SD
Pretest	72.07	13.060	69.69	12.632	−.724	.472
Posttest	78.28	9.188	73.59	14.041	−1.554	.126

As indicated in Deng and Yu’s (2023) review study, the chatbot had a medium-to-high overall effect size on learning achievement, which is because students needed to review the current information and ask questions based on it to get feedback before acquiring new information. In this study, we found that both groups of scaffolding-based game-based learning activities showed improvement in the posttest compared to the pretest; however, there was no difference in the learning achievement of the two groups, and there was no difference between the cognitively assistive effects of the ChatGPT-based NPCs scaffolding and that of the scaffolding provided by the document files.

The study also used an independent t-test to determine whether there were significant differences in the pre-test of learning achievement between the gender differences in the two groups, which showed that there were no significant gender differences in the pre-test of learning achievement of the Document scaffolding group (t = 1.277, p = .211) and the ChatGPT-based NPCs scaffolding group (t = .740, p = .466), indicating that students of different genders had the same prior knowledge about employment ethics before conducting the activity.

Flow

In this study, descriptive statistics were analyzed for the two groups of flow, and an independent sample t test was compared with the median of the scale (i.e., 3). Results showed that in the document scaffolding group, flow antecedent (t = 13.525, p = .000 < .001), flow experience (t = 9.896, p = .000 < .001), and overall flow (t = 12.992, p = .000 < .001) were significantly higher than 3.

The independent sample t test was used to analyze whether there was a significant difference in flow between the two groups (Table 4), which showed that there was no significant difference between the two groups in flow antecedent (t = −.589, p = .559), flow experience (t = .088, p = .930), and flow (t = −.244, p = −.808). This result is consistent with the pilot study’s finding that the instructional games in this study were designed to allow for a high level of student engagement (Chen & Hou, 2023). Also, as in past research, using ChatGPT to engage in learning is able to facilitate student engagement (Ruiz-Rojas et al., 2023).

OPEN IN VIEWER

Table 4.

Independent Samples t Test of Flow Between Two Groups.

Group	Document scaffolding group (n = 29)		ChatGPT-based NPCs scaffolding group (n = 32)		t	p
	M	SD	M	SD
Flow Antecedent	4.106	.440	4.012	.780	−.589	.559
Flow Experience	3.879	.478	3.893	.736	.088	.930
Flow	3.982	.407	3.947	.699	−.244	.808

The results of the study showed no significant difference between the two groups in flow, indicating that the inclusion of ChatGPT-based NPCs in the game would not negatively affect engagement. Chatbots could have been used as a game element and promoted student engagement (González-González et al., 2023). In summary, this study designed the ChatGPT-based NPC as a friend role to provide scaffolding assistance, which is feasible and can be analyzed in more depth in the future with respect to the factors affecting engagement.

Motivation

In this study, the motivation data of the two groups were compared with the median of the scale (i.e., 3) using the independent sample t test. The results showed that the document scaffolding group scored significantly higher than 3 in motivation (t = 9.949, p = .000 < .001). The results showed that the ChatGPT-based NPC group scored significantly higher than 3 in motivation (t = 7.228, p = .000 < .001). An independent sample t test was then used to compare the motivation of the two groups for differences (Table 5), which showed that there were no significant differences between the two groups of motivation (t = −.576, p = .567).

OPEN IN VIEWER

Table 5.

Independent Samples t Test of Motivation Between Two Groups.

Group	Document scaffolding group (n = 29)		ChatGPT-based NPCs scaffolding group (n = 32)		t	p
	M	SD	M	SD
Motivation	4.224	.662	4.109	.868	−.576	.567

Both groups in this study had a certain degree of high motivation, and game-based learning had a positive effect on enhancing motivation (Hou, 2023). There was no significant difference between the motivation of the two groups, which means that the inclusion of ChatGPT-based NPCs scaffolding in game-based learning has no significant effect on learning motivation. In a past study, it was noted that the inclusion of chatbots in research is usually associated with higher motivation compared to traditional instruction (Deng & Yu, 2023), but fewer studies have examined the differences in the impact of the inclusion of chatbots as scaffolding on the motivation of game-based learning. The results of the present study can be used as a reference.

In addition, the inclusion of realistic NPC in games has had a positive effect on students’ motivation in the past (Chan et al., 2023), and the use of ChatGPT-based NPC in games in this study also had a positive effect on motivation, meaning that the use of ChatGPT-based NPCs scaffolding has the potential to achieve the same effect of enhancing or maintaining motivation with fewer human resources than the use of realistic NPC in games, and also has more interactive simulation than just providing documents as a scaffold.

Anxiety

In this study, the anxiety data of the two groups were compared with the median of the scale (i.e., 3) using the independent sample t test. The results showed that the document scaffolding group was significantly lower than 3 on anxiety (t = −2.366, p = .025 < .01). The ChatGPT-based NPC group did not differ significantly from 3 on anxiety (t = −1.567, p = .127).

Using an independent sample t test to compare whether there was a difference in anxiety between the two groups (Table 6), the results showed that there was no significant difference (t = −.576, p = .518). Students’ anxiety was significantly lower than 3 in the pilot study (Chen & Hou, 2023) without the addition of the document scaffolding or the ChatGPT-based NPCs scaffolding, but not significantly lower than 3 with the addition of the simulated NPC in the current study. There were other factors after adding the ChatGPT-based NPC that caused some students to be slightly more anxious than the group that did not use the ChatGPT-based NPC, but the overall level of anxiety was also not greater than 3 (M = 2.656). As pointed out by Hong et al. (2020), game-based learning can reduce learning anxiety, but the same game will have different effects on anxiety when a different instructional mechanism is added.

OPEN IN VIEWER

Table 6.

Independent Samples t Test of Anxiety Between Two Groups.

Group	Document scaffolding group (n = 29)		ChatGPT-based NPCs scaffolding group (n = 32)		t	p
	M	SD	M	SD
Anxiety	2.448	1.256	2.656	1.240	.650	.518

Analysis of the Usability of Scaffolding

Regarding research question 2. In this study, scaffold usage was analyzed for the document scaffolding group, and the mean for the document scaffolding group in terms of their usefulness for document scaffolding was 3.41 (SD = 1.119). From the qualitative questionnaire on the usefulness of the scaffolding, 25 out of 29 (86%) students indicated that the document scaffolding hints were helpful in learning knowledge, while the other four (14%) indicated that they were not helpful. Regarding the time of use, 16 students (55%) reported that they only checked it at the beginning of the game, but not during the game. Another eight (27%) students reported checking back when they encountered decision-making difficulties during the game. Five (17%) noted that they did not view it during the game.

This study analyzed scaffold usage for the ChatGPT-based NPC group, and the ChatGPT-based NPC group had a mean of 3.03 (SD = 1.231) in terms of usefulness for the ChatGPT-based NPC scaffold. The mean of the realistic findings against the ChatGPT-based NPC scaffold was 2.94 (SD = 1.045). In the qualitative questionnaire on scaffold usage, out of 32 students, 20 (63%) responded that interacting with the ChatGPT-based NPC scaffold was helpful and interesting, while two (6%) indicated that they did not know how to start asking questions, and 10 (31%) indicated that it was not very helpful to their learning. Regarding the question “How do you decide what questions to ask ChatGPT-based NPCs?”, 14 (44%) of the students indicated that their questions were based on the game situation, and 10 (31%) indicated that their questions were based on their own life thoughts. Seven (22%) of the students found it difficult to ask questions or had no idea, and one (3%) of the students did not ask any questions.

In the ChatGPT-based NPC group, 31 students used the ChatGPT-based NPCs scaffolding, and some of them did not successfully retrieve the chat logs due to a game system error. A total of 25 students’ chat logs were retrieved in this study, which consisted of 120 questioning logs. Among them, 65 (54%) questions were related to employee ethics, while 55 (46%) were unrelated. The realistic ChatGPT-based NPCs scaffolding was able to search from the researcher’s predefined document and answer the 65 questions related to employee ethics in a manner close to human speech. Ayub et al. (2023) used ChatPDF as a ChatGPT application as in this study and showed that it works for simple questions, but it was wrong for complex questions (such as generating test papers).

After qualitative content analysis of the NPC responses, no incorrect response information was found in any of the questions related to employee ethics. However, as Chubb (2023) points out, the process of analyzing the text to give responses, whether using ChatPDF or other ChatGPT applications, is similar to a black box, and it is difficult for the researcher to know the rationale. As one of the questions in this study, the player asked “What should one wear on the first day of work?”, and the realistic NPC replied: “It is recommended to try to wear plain shirts and pants to match with the clothes in the company which is the safest. You can stay in the company for a few months to familiarize yourself with the rules of the dress code before you start to make changes. Therefore, on your first day at work, we recommend you wear formal and decent clothing, such as a suit or a dress.” The phrases “wear a plain shirt and pants” and “stay in the company for a few months to familiarize yourself with the rules of the dress code before you start to make changes” are extensions of the content of the knowledge text we provided. However, “On your first day of work, we recommend that you wear formal and modest clothing, such as a suit or a dress” was not in the provided knowledge document, and although it can be categorized as a correct response, there is no way of knowing why such a knowledge supplement was generated. This may also lead to potentially imprecise responses in the application of more professionally oriented fields of study, an issue that deserves to be explored in future research.

Of the unrelated questions, 20 (36%) used the ChatGPT-based NPC as a personal assistant to ask questions such as “What time do I have to work tomorrow? and “Do I have a meeting tomorrow?”. Five (9%) of the questions were about other characters in the game, such as “Barbara is mean, what should I do?” and “What kind of lunchbox does Rika like?” In addition to this, six (11%) of the unrelated questions expressed emotions related to the game situation, such as “I’m tired of going to work” and “When can I leave my job?” The remaining 23 (42%) were casual conversations such as asking “What’s for lunch?”.

Because the NPC’s background document only included employee ethics information, the NPC will say it doesn’t know the answer to irrelevant questions. Few studies have looked at using ChatGPT for role-play in learning, and some students might ignore the NPC’s role and treat it as an all-knowing assistant. the application of ChatGPT as a specific role-play in the learning process, and perhaps some students may disregard the role-play of this NPC and inquire about the ChatGPT-based NPC as an all-knowing assistant. Some students also asked questions about other characters in the game, which means that the situation of the game may become one of the factors affecting students’ questions.

In future designs, adding plot details as background information could help the GPT generate more appropriate responses. A few students would express emotions related to the game plot to the realistic NPCs, which may be a response to their feeling empathy for the game situation, but the realistic NPCs would not give feedback related to emotional support, as described by Maurya (2023) when playing a character setup in ChatGPT, which would be less emotionally up and down.

Suggestions and Limitations

It is suggested that future research can focus on the impact of different game presentation methods combined with ChatGPT-based NPC on students, such as Visual Novel or Role-playing educational games with ChatGPT. It is also suggested that we can try to control and refine the response of ChatGPT in a variety of ways, and try to make ChatGPT play the role of a simulated situation as well as a facilitator to enhance learning achievement. At the same time, we also suggest that future studies need to take into consideration the students’ ChatGPT prompting ability. From the results of this study, it was found that there were still a portion of players who did not know how to start to ask questions with the NPCs to obtain the scaffolds, therefore, in future studies, we should consider provide the questioning quotes or examples to help the students to use the NPCs for scaffolding.

The participants in this study were mainly in the age group of 20–30 years old, and it is difficult to extrapolate the results to children or middle-aged and older adults because of the need to take into consideration the acceptance of technology as well as the ability to utilize technology.

The participants in this study were 41 males and 20 females, with more males. Although there was no difference in the prior knowledge between the genders in this study, it is important to note that the preference of game mechanics for game-based learning and the view and behavioral patterns of employment ethics may differ by gender (Aisyah & Hani, 2020; Mardawi et al., 2021) and therefore requires attention in inference and future research. In open-ended questionnaires, this study found that some players did not know how to ask ChatGPT questions effectively. In this era, asking questions via Generative AI and using search engines to find answers are the mainstream channels for acquiring knowledge from the Internet (Zhou & Li, 2024). However, the present study did not investigate the subjects’ ability to use ChatGPT, so it is important to include this ability as a background variable in future studies. In addition, the duration of this study was relatively short, which made it difficult to observe the changes in students’ behaviors and attitudes. In the future, we hope to analyze larger samples and even analyze the behavioral patterns and content of learners’ interactions with ChatGPT. It is also expected that a longer study will be conducted to observe changes in learning retention and attitudes, and unstructured interviews will be added to provide a deeper insight into students’ learning experiences and strategies.