A ChatBot for Learning Chinese: Learning Achievement and Technology Acceptance

ChatBot, a version of virtual assistant, is a program that is able to imitate human conversation (Tatai et al., 2003). Several studies have investigated the possibility of using ChatBot in language teaching and learning (Fryer et al., 2019; Jia, 2004; Lu et al., 2006) and found it to be an effective means of stimulating students’ learning interest. Learning with a ChatBot is an accessible choice for learners, because it allows them to learn anywhere at any time. Moreover, learners felt more confident chatting with a bot even though it is obvious that humans are more intelligent than a programmed robot. Prior research conducted by Lu et al. (2006) investigated the usage of ChatBots for learning English. They developed a ChatBot whose system offered total English environment. The robot had built-in dictionaries and access to authorized conversational materials to conduct both speaking and question-answer functions. The main purpose of the ChatBot was to operate both as an assistant for the teacher and as a partner for the students. The ChatBot succeeded in promoting a one-on-one learning environment.

A one-on-one learning environment concerns the effectiveness of learning. The setting in this environment involves one teacher for one student. Based on that, some benefits of one-on-one learning environment can be delivered. For example, teachers can provide immediate feedback (Gettinger, 1983; Wentling, 1973), and students will have more time to interact with the teacher (Fisher et al., 1978). As shown in the study by Moody et al. (1973), students in the individual class performed better than those in the group class. Glass and Smith (1979) also found a negative correlation between group size and students’ learning achievement. They found that the better students’ learning achievement the smaller their group size. Such result then led to a hypothesis that students’ level of achievement in learning may be affected by changes in the environment.

In language learning, vocabulary is one of the most important part. Huckin et al. (1993) stated that two of the most important components in language learning are reading ability and vocabulary knowledge. For all of the language skills (speaking, writing, reading, and listening), vocabulary is an important key for an effective understanding of meaning (Mutalib et al., 2014). Comparing it with grammar, Wilkins (1972) said, “Without grammar very little can be conveyed, without vocabulary nothing can be conveyed.” (p. 111–112) Memorizing vocabulary tends to be difficult for students, especially memorizing Mandarin Chinese vocabulary, because learning Chinese is different from learning English. Chinese is a Sino-Tibetan-based language while English is a Latin-based language which uses alphabets (Chen & Dunsmoir, 2012).

Several technologies have been used for learning Chinese vocabulary. Edge et al. (2011) created a mobile application for learning Chinese vocabulary. They developed two versions of the mobile application, contextual version, and frequency-based version. Contextual version automatically provides Chinese vocabulary related to a nearby location. Frequency-based version features the use of Web corpus, from which the most to the least common frequency is derived for different levels of Chinese vocabulary to be taught. Both versions were effective tools to help students learn Chinese vocabulary. Also, it was proved that other mobile applications like WeChat may help students learn Chinese vocabulary when used to deliver materials (Yang & Yin, 2018). Another technology used for learning Chinese vocabulary was virtual reality (VR). In the study conducted by Legault et al. (2019), native English students experienced VR for learning Chinese vocabulary and agreed that VR was able to improve their learning achievement. All of the mentioned studies were examples of how technology can help students learn Chinese vocabulary. This further indicates how technology is of paramount importance for learning language. However, none of the studies discussed have incorporated any pedagogical use of a ChatBot. By developing a new platform, or ChatBot, this study is able to compare how different environments equipped with the proposed technology influence learning.

Rote learning is one of the memorization techniques whereby students are taught to memorize the targeted language from a vocabulary list (Yang, 2011). Researchers have stated that memorizing words from a vocabulary list cannot be separated from the context of vocabulary learning (Cortazzi & Jin, 1994; Yang, 2011). Rote learning is categorized as a cognitive strategy in learning vocabulary. Cognitive strategies as mentioned in Mutalib et al.’s (2014) study are “repetition and using mechanical means to study vocabulary” (p. 363) such as using notes for memorizing. Rote learning is done by repeating and rehearsing materials believed to promote academic success (Morton, 2011). Nevertheless, it also takes place as a learning process without or with a minimum of insight (Park, 1937), because it involves the use of surface learning approach, which focuses on the learning material itself (Marton et al., 1996) and in the acquisition of a new language, is indeed inevitable in the early stages (Yang, 2011). Yang (2011) also admitted that a number of initial words can be learned effectively and rapidly through rote learning. However, acquisition of vocabulary cannot be accomplished by rote memorization alone. In fact, the development of the ChatBot is not only based on the rote learning approach but also integrates a personal response system (PRS) approach.

PRS is a technology that makes real-time feedback possible and is able to engage the learner (Gharaie, 2016). Immediate feedback is important for the students to create an effective learning environment based on their current understanding levels (Chen et al., 2010). PRS helps students to gain a better understanding of the learning materials by allowing them to learn at their own pace (Chan et al., 2019). The proposed ChatBot is based on the rote learning approach with a vocabulary list and PRS for memory retention through exercises comprised of multiple-choice questions. The ChatBot, which is based on a mobile instant messaging platform, is convenient for learners to use, because it can be accessed everywhere at any time as long as the user is connected to the Internet, and it is expected to be helpful to students in learning Chinese vocabulary.

A ChatBot can only be described as a successful information technology when it is accepted among users (Nikou & Economides, 2017). Technology Acceptance Model (TAM; Davis, 1985), as a theory of user acceptance, is the most accepted one by far in terms of investigating user’ acceptance behavior (Liu et al., 2009). TAM originally consisted of two factors, perceived ease of use (PEU) and perceived usefulness (PU), both of which may influence people’s behavioral intention (BI). How people expect a technology to improve their job performance defines their PU of the technology (Davis et al., 1992). In regard to its PEU among people, it is measured by how strongly they confirm the technology as easy to use in performing their tasks free of effort. In this study, the measurement of the PU of different items and their PEU is drawn from the previous research by Davis (1989). Several studies have hypothesized PEU as a predictor of PU (Farahat, 2012; Rodrigues et al., 2016; Venkatesh, 2000; Yi & Hwang, 2003). Their hypothesis derives from the original TAM, which posits that the easier the technology is to use, the more users will perceive it as useful. The relationship between the two factors is assumed as having one causal direction as hypothesized below:

H1. PEU positively affects PU.

Research Questions

Based on the background and the literature review above, this study investigated the effect of using the proposed ChatBot for learning Chinese vocabulary and explored the technology acceptance of the ChatBot. The specific purposes of this study are listed below:

Does the use of the ChatBot in different learning environments improve the students’ learning achievement in learning Chinese vocabulary?

Method

This research was conducted with a mixed-methods approach (Creswell & Clark, 2007). Triangulation method was adopted, although quantitative method remained predominant (Chiva-Bartoll et al., 2020). The quantitative method was applied both to the pre- and posttest scores and to the results of the questionnaire. Qualitative method was used toward the analysis of semi-structured interview responses.

Research Tools

Pretest and Posttest

A pre- and posttest for each chapter were implemented to measure the students’ learning achievement. Both tests were developed by the author and were validated by two Mandarin Chinese Level 1 teachers. The pretest was arranged to measure students’ prior knowledge before participating the study. The pre- and posttest for each chapter were of a different type. The test on Chapter 1 contained 10 multiple-choice questions. The test on Chapter 2 contained 10 word-matching questions. The test on Chapter 3 contained 15 word-matching questions. Each chapter used the same pre- and posttests with the sequence of the posttest questions rearranged. The highest score for both tests was 100.

Questionnaire

The technology acceptance questionnaire items were obtained from Davis’ (1989) study (Cronbach’s α of PU was .97 and, for PEU, it was .86) and Liao et al.’s (2008) study (Cronbach’s α of PU was .94 and, for PEU, it was .90) and were modified (subject replaced) by the author. The questionnaire had two constructs, PEU and PU (PU). PEU included seven items (the internal consistency reliability, ICR, among which is .861), and PU consisted of four items (the ICR among which is .912). The extended TAM-PEPE questionnaire was derived from Gopalan et al.’s (2016) study (whose Cronbach’s α was .80) and Liao et al.’s (2008) study (whose Cronbach’s α was .90), and the items were modified (subject replaced) by the author. PE consisted of six items (the ICR among which is .835). The BI items were adopted from Venkatesh’s (2000) study which had ICR of .90. BI consisted of two items (the ICR of which is .947). That questionnaire was only given to coincide with the last posttest for the experimental group.

Interview

A semi-structured interview was conducted in response to the last research question about students’ learning experience of a new learning tool, the ChatBot. The author developed protocols (as shown in Table 1

OPEN IN VIEWER

Table 1.

Interview Protocols.

Question
1.	Please tell me three things that attract your attention when using the ChatBot.
2.	In your opinion, what are two things that are helpful when using the ChatBot and two things that are not helpful? Please give examples and reasons.
3.	According to your experience of learning Mandarin Chinese using the ChatBot, which parts do you like and dislike the most?
4.	How would you use the ChatBot in the future?
5.	What are your recommendations about the ChatBot for the future?
6.	Would you recommend ChatBot to others as a learning tool?

) for the interview and checked their validity with a professor who is an expert in qualitative research.

The ChatBot Mechanism

The proposed ChatBot, named Xiaowen, was developed based both on rote learning for memorizing vocabulary and on a PRS to help with memory retention. This system was implemented via Python programming language and LINE Developer platform; hence this ChatBot can only be accessed through LINE. Rote learning is a technique for memorizing vocabulary from a vocabulary list. Thus, this ChatBot provides vocabulary lists at different levels. To maintain the memorization of the vocabulary, ChatBot was equipped with a multiple-choice practice tool with a PRS.

When learners add “Xiaowen” to their friend lists on LINE and greet the ChatBot with the first message they send, the ChatBot mechanism will be activated and reply with a welcoming message. Xiaowen immediately introduces the levels of vocabulary learning available and instructs the learner to text “start” for entering the learning section. Once the bot received the message, it shows the buttons whereby the users may choose from different levels of practice; currently levels 1 to 4 are available with the bot. These buttons help the user to interact with the bot more easily. Next, the user is asked whether to activate either learning mode or practice mode. If the former was chosen, the bot then displays a list of vocabulary containing preprogrammed Chinese characters, the pinyin of each, what they mean in English, and sentence examples corresponding to the level chosen. If the latter was chosen, the bot randomly selects a word from its vocabulary list to produce a practice question, which contains Chinese characters and their meanings in English. The options are randomly generated from an English wordlist at the same level as previously indicated. And each question contains one correct answer only.

The practice mode functions on a PRS where the bot serves as a substitute for the teacher or instructor. Once an answer is chosen as a response to the question, the bot immediately provides feedback based on the result. If the user choses the correct answer, the bot recognizes it and moves on to the next question. On the contrary, if a false answer is chosen, the bot informs the user of it and reveals the correct response at the same time. The questions loop until the user texts “start” again to change the level of practice. An example of a conversation with the bot is shown in Figure 2 OPEN IN VIEWER

Figure 2.

(a–h) Examples of Conversation With the Bot. (a) Greetings message; (b) choose level; (c) learn or practice?; (d) learning section; (e) continue to practice; (f) multiple choice question (g) correction of wrong answer and (h) feedback of correct answer.

.

Procedure and Instruction

The participants in this study were divided into two groups, a control group (Group A) and an experimental group (Group B). The experimental group was assigned to learn Mandarin Chinese with ChatBot in a one-on-one learning environment while the control group used ChatBot to memorize the vocabulary in a one-on-many classroom. The length of the experiment was 4 weeks in total. Group A was in session every Monday. Group B, with its one-on-one tutoring, took place on different days and times for every student. However, both groups kept the same length of experiment and followed the same weekly agenda. At the beginning of the first week of the experiment, the participants were instructed to take a pretest which lasted 3 minutes. Participants in both groups then spent the next 5 minutes learning about the ChatBot in an introduction. In the next 5 minutes, participants in group A were assigned to use ChatBot for learning Mandarin Chinese vocabulary before receiving further instructions, which lasted for 10 minutes per session, from their teacher. Meanwhile group B participants were assigned with a 15-minute learning session. While Group A spent 5 minutes on learning with the ChatBot and 10 minutes later on with the teacher, whose duty was to explain to them the details of the material, Group B, however, spent an entire 15 minutes on learning with the ChatBot, with which the tutor answer the questions raised by the students. It should be specified herein that the difference in the timespans between the sessions implemented among the two groups does not imply any difference in the amount of material covered in either. Once the groups finished their assigned sessions, the participants spent 3 minutes taking a posttest. The second posttest was conducted the next week along with the pretest on the new chapter. The process repeated three times, each with its own chapter (Chapters 1–3) for learning. On the 4th week, the questionnaire of technology acceptance was administered in both groups, and three participants in each were selected randomly to take part in the one-on-one interview.

Data Analysis

The statistical analysis performed in this study varied from one research question to another. The first research question was explored using two statistical tests: a simple paired t test and analysis of covariance (ANCOVA). The former was used to investigate the differences between the learning achievement of the two groups in the pre- and posttests. ANCOVA was then conducted to determine further differences between the two. Both the simple paired t test and ANCOVA were performed using SPSS 22. The second research question, which concerns the TAM, was tested using partial least square (PLS), conducted via Smart-PLS 3. The third research question was explored using Pearson correlation test on SPSS 22. The last research question, which concerns students’ opinions from the interviews, however, required recorded data and its transcripts, which were then coded for further analysis in the conclusions.

Result

Learning Achievement Within Groups

A pretest was implemented in both groups prior to their exposure to the materials drawn from across all of the three chapters in the textbook. The results of their pre- and posttests are displayed in Table 2

OPEN IN VIEWER

Table 2.

Paired Sample t Tests of the Overall Learning Achievement (Pretest and Posttest).

Group	Pretest		Posttest		n	95% CI for mean difference	t	df	d
	M	SD	M	SD
A	71.63	15.19	88.03	12.35	39	[13.11–19.70]	10.09***	38	1.18
B	49.82	16.06	91.33	11.77	19	[35.45–47.57]	14.38***	18	2.95

***p < .001.

, showing a significant difference each group achieved with their pre- and posttest scores. Both groups had taken three pretests as previously mentioned. The average score of the three pretests indicate that both groups had an average command of Chinese vocabulary. However, it was soon surpassed by the average of their posttest scores. This proved that using the ChatBot may improve students’ learning outcomes. In addition to the above comparison between the pre- and posttest scores, results of the retention tests were also investigated. As shown in Table 3

OPEN IN VIEWER

Table 3.

Paired Sample t Tests of the Overall Learning Achievement (Pretest and Retention Test).

Group	Pretest		Retention test		n	95% CI for mean difference	t	df	d
	M	SD	M	SD
A	71.63	15.19	80.95	15.76	39	[7.02–11.62]	8.193***	38	0.60
B	49.82	16.06	93.21	8.28	19	[36.41–50.36]	13.08***	18	3.40

***p < .001.

, there was a significant difference between the scores achieved in the retention test and those obtained in the pretests. This means that after 1 week of using the learning tools, both groups still remembered their vocabulary and maintained a medium–large effect according to Cohen’s classified effect sizes (0.2 = small effect, 0.5 = medium effect, 0.8 = large effect).

Learning Achievement Between Groups

To compare the differences between the learning outcomes of the experimental group and those of the control group, the ANCOVA test was performed with the pretest scores as the covariates. Table 4

OPEN IN VIEWER

Table 4.

ANCOVA Test of the Overall Learning Achievement (Posttest).

Source	Type III sum of squares	df	Mean square	F	Sig.	Partial eta squared
Corrected model	4,293.950a	2	2,146.975	28.552	.000	.509
Intercept	10,592.802	1	10,592.802	140.871	.000	.719
Pretest	4,154.893	1	4,154.893	55.255	.000	.501
Group	2,095.599	1	2,095.599	27.869	.000*	.336
Error	4,135.728	55	75.195
Total	469,035.111	58
Corrected total	8,429.678	57

^a

R²= .509 (adjusted R²= .492).

*The mean difference is significant at the .05 level.

displays the result of the ANCOVA test of the posttest scores and reveals a significant difference between groups, F(1, 55) = 27.869, p = .000. It means that experimental group was better than control group, as group A has a higher score than group B.

Table 5

OPEN IN VIEWER

Table 5.

ANCOVA Test of the Overall Learning Achievement (Retention Test).

Source	Type III sum of squares	df	Mean square	F	Sig.	Partial eta squared
Corrected model	8,229.114a	2	4,114.557	51.873	.000	.654
Intercept	6,999.628	1	6,999.628	88.246	.000	.616
Pretest	6,308.239	1	6,308.239	79.529	.000	.591
Group	6,514.154	1	6,514.154	82.125	.000*	.599
Error	4,362.595	55	79.320
Total	431,301.778	58
Corrected total	12,591.709	57

^a

R²= .654 (adjusted R²= .641).

*The mean difference is significant at the .05 level.

shows the ANCOVA result of the retention test with the pretest scores as the covariates. The results indicate a significant difference between groups, F(1, 55) = 82.125, p = .000), meanings that learning in a one-on-one environment had led to better memory retention than learning in a one-on-many classroom had.

TAM

The first analysis presented with regard to technology acceptance is the descriptive statistics of each construct (as seen in Table 6

OPEN IN VIEWER

Table 6.

Descriptive Statistics of the Technology Acceptance Result.

	n	M	SD	Min	Max
PEU	58	4.32	0.71	1	5
PU	58	4.19	0.67	2	5
PE	58	4.01	0.82	1	5
BI	58	4.08	0.75	1	5

PEU = perceived ease of use; PU = perceived usefulness; PE = perceived enjoyment; BI = behavioral intention.

). The data used in the analysis were drawn from both groups, which account for a total of 58 respondents. Their ratings for each constructs were calculated first based on their responses and were then averaged for further analysis (Malaquias et al., 2018). The results indicated that both Groups A and B participants perceived the usefulness and the ease of use of the ChatBot with their mean scores of PEU and PU both exceeded four. This means that both groups agreed with the statements on the questionnaire. Moreover, both groups perceived the enjoyment of learning Chinese with the ChatBot, although the average score on this construct was the lowest among all. In terms of the students’ BI, responses in both groups indicated a willingness to use the ChatBot in the future.

Results of the descriptive statistics were applied to the model and the hypothesis tests of PLS developed by Wold (1982). Closely resembling other structural equation modeling (SEM) techniques, in which structural path coefficients and measurement model parameters are measured (Yi & Hwang, 2003). PLS is a good alternative of statistical analysis in cases with a rather small sample size (Wong, 2013); the present research, with as few as 58 participants in total, thus justified its application.

A handful of measurements were available to be chosen by the present research in its initial assessment of latent variables, reliability, and validity. Among all, measurement of reliability must include indicator reliability as well as ICR. While the preferred score of indicator reliability should amount to .70 and above (Barclay et al., 1995; Fornell & Larcker, 1981), the composite reliability in the ICR should reach greater than .7 (Barclay et al., 1995; Fornell & Larcker, 1981). Meanwhile validity consists of convergent validity and discriminant validity. The former checks the average variance extracted (AVE) and should be at least 0.5 (Bagozzi & Yi, 1988), while the latter measures the AVE number and latent variables correlation. Fornell and Larcker (1981) advised that the square root of AVE in each latent variable should be greater than the correlations among the latent variables.

Table 7

OPEN IN VIEWER

Table 7.

Factor Structure Matrix.

	BI	PE	PEU	PU
BI1	0.948	0.763	0.582	0.725
BI2	0.948	0.777	0.629	0.712
PE1	0.646	0.825	0.734	0.707
PE2	0.718	0.878	0.627	0.694
PE3	0.666	0.852	0.672	0.682
PE4	0.737	0.795	0.519	0.695
PE5	0.639	0.852	0.534	0.616
PE6	0.708	0.845	0.533	0.674
PEU1	0.632	0.719	0.821	0.667
PEU2	0.247	0.349	0.666	0.323
PEU3	0.293	0.337	0.579	0.255
PEU4	0.308	0.416	0.770	0.400
PEU5	0.557	0.628	0.713	0.591
PEU6	0.531	0.523	0.635	0.634
PEU7	0.200	0.261	0.633	0.356
PU1	0.652	0.651	0.602	0.826
PU2	0.589	0.600	0.568	0.833
PU3	0.529	0.640	0.555	0.815
PU4	0.774	0.829	0.725	0.924

PEU = perceived ease of use; PU = perceived usefulness; PE = perceived enjoyment; BI = behavioral intention. *Loadings > = .7 in bold.

displays the factor structure matrix for measuring the indicator reliability in each item toward its latent variables. As shown in the results of the study, two items of the BI variables have factor loading greater than .7 with a value greater than those of other latent variables. The PE variable lists six items, and the factor loading of each not only exceeds .7 but also surpasses all the other values drawn from other latent variables. However, in the case of PEU variable, as many as four items (PEU2, PEU3, PEU6, and PEU7) fail to meet the expected criterion value of .7. Thus, elimination of these items was proposed. Finally, the PU variable consists of four items, and the factor loading of each exceeds .7, rendering all of them acceptable.

Table 8

OPEN IN VIEWER

Table 8.

Factor Structure Matrix Without Low Value Loadings.

	BI	PE	PEU	PU
BI1	0.948	0.760	0.582	0.725
BI2	0.949	0.775	0.633	0.712
PE1	0.646	0.830	0.766	0.707
PE2	0.718	0.876	0.630	0.694
PE3	0.666	0.861	0.749	0.683
PE4	0.737	0.786	0.488	0.693
PE5	0.639	0.847	0.527	0.615
PE6	0.708	0.840	0.509	0.674
PEU1	0.632	0.723	0.862	0.668
PEU4	0.308	0.421	0.777	0.403
PEU5	0.557	0.635	0.822	0.592
PU1	0.652	0.648	0.505	0.818
PU2	0.589	0.603	0.600	0.839
PU3	0.529	0.640	0.518	0.812
PU4	0.774	0.829	0.727	0.926

PEU = perceived ease of use; PU = perceived usefulness; PE = perceived enjoyment; BI = behavioral intention. *Loadings > = .7 in bold.

shows the factor structure matrix with PEU2, PEU3, PEU6, and PEU7 removed. All of the factor loadings meet the expected criterion value of 0.7 and thus pass the indicator reliability test.

The next reliability test follows in the form of an ICR test. Table 9

OPEN IN VIEWER

Table 9.

ICR, AVE, Correlation Constructs.

	ICR	AVE	BI	PE	PEU	PU
BI	0.947	0.899	0.948
PE	0.935	0.706	0.810	0.840
PEU	0.861	0.674	0.641	0.751	0.821
PU	0.912	0.723	0.758	0.808	0.700	0.850

PEU = perceived ease of use; PU = perceived usefulness; PE = perceived enjoyment; BI = behavioral intention; ICR = internal consistency reliability; AVE = average variance extracted.

shows the ICR of each latent variable, its AVE, and the result of its comparison between the AVE square root and the correlation among the latent variables. According to Bagozzi and Yi (1988), composite reliability can be used as a replacement of Cronbach’s alpha. This study checks the reliability of its internal consistency with composite reliability. Its results, which show an ICR above .7 on each latent variable, pass the reliability test. As for the validity test, the AVE value on each latent variable has an acceptable score above 0.5 and thus suffices the requirements of the convergent validity test. The last validity test investigates the discriminant validity of the study results by comparing the AVE square root of each latent variable and the correlation among the latent variables. Each AVE square root is expected to exceed that of the correlation among the variables both by row and by column (Yi & Hwang, 2003). The diagonal bold values in Table 9 are the AVE root square between the constructs while the other values indicate the correlations between constructs. The results approve the discriminant validity where each square root of the AVE scores is higher than the correlations among the construct.

The second stage of PLS was to analyzes the structural model and to validate the research hypotheses. The result in Figure 3 OPEN IN VIEWER

Figure 3.

PLS Test for the Model.

shows that the coefficient of determination (R²) for the variable of BI was .598. This means that the PEU and PU variables explain 59.8% of the variance in BI among student participants. The variable of PU has an R² of .49, which shows that the PEU explains 49.0% of the variance in PU. The variable of PEU has an R² of .564, which means that the PE variable explains 56.4% of the variance of PEU.

t tests as well as bootstrapping using 500 subsamples, as suggested by Chin (1998), were conducted to investigate the statistical significance of each path. The results, on the one hand, confirmed H1 (PEU has a significant effect on PU; β = 0.700; p < .001). On the other hand, the effect of PEU on the students’ BI was found inconsistent with H2 (β = 0.216). Also, the effect of PU on the students’ BI was in line with H3 (β = 0.606; p < .05), which posits that PU has a significant effect on their BI. Finally, the results also supported H4 (β = 0.751; p < .001), which claims that the PE among the students has a significant effect on their PEU.

Davis (1989) theorized that ease of use is a determinant of both PU and BI. This, however, as suggested in Davis et al.’s (1989) study, may change over time. They found that learners’ PEU may have a significant effect on their BI and have no significant effect on their PU when it was measured immediately after a 1-hour introduction to the system. However, it was found 14 weeks later that their PEU had a significant effect on their PU and had no significant effect on their BI. In the present study, the results regarding the effect of the students’ PEU on their PU is consistent with those of Davis (1989). Another study conducted by Chen et al. (2013) also yielded similar results, where learners’ PEU had a significant effect on their PU after engaging in a learning process with mobile devices. An increase in the ease of use of the ChatBot did not necessarily entail an increased BI of using a ChatBot, yet an increase in the ease of use indeed leads to an increase in user’s PU. Accordingly, the use of ChatBot is positively influenced by its PU rather than by its ease of use. Users tend to use the ChatBot as a result of its usefulness for learning Chinese vocabulary.

PE has a significant effect on PEU. This is consistent with the research findings of Venkatesh (2000), which asserts that ChatBot interface provides enjoyment to the users by displaying effective choice buttons alongside pictures. Venkatesh also emphasizes design features aimed to establish enjoyment for users and the importance of having a final goal in enhancing the ease of use of the system. Thus, it can be conferred that, in some cases, the absence of enjoyment may render the system less easy to use than it otherwise would appear to users. However, in the present study, the ChatBot did provide enjoyment to its users and thus significantly affected its PEU.

To highlight the benefit each TAM variable brought to the students’ learning achievement, a correlation test was conducted as follows. See Table 10

OPEN IN VIEWER

Table 10.

Correlations Between TAM Variables and Learning Achievement.

		PEU	PU	PE	LA
PEU	Pearson correlation	1	.668**	.678**	.228
	Sig. (2-tailed)		.000	.000	.086
	N		58	58	58
PU	Pearson correlation		1	.799**	.416**
	Sig. (2-tailed)			.000	.000
	N			58	58
PE	Pearson correlation			1	.423**
	Sig. (2-tailed)				.001
	N				58
LA	Pearson correlation				1
	Sig. (2-tailed)
	N

PEU = perceived ease of use; PU = perceived usefulness; PE = perceived enjoyment; LA = learning achievement.

**Correlation is significant at the 0.01 level (2-tailed).

for its results.

According to Table 10, no significant correlation was found between PEU and learning achievement (r = .228). However, a significant positive correlation was found both between PU and learning achievement (r = .416, p = < .001) and between PE and learning achievement (r = .423, p = <.001).

Students’ Experience

The last research question was addressed using qualitative research. A semi-structured interview was conducted with six students, with three of whom drawn from each group. These were one-on-one interviews in English. All of the recordings from the interviews were transcribed and coded in order to provide the qualitative data needed for further analysis and to derive conclusions based on the categorization of various expressions found in the verbatim of the interview responses (Jiménez et al., 2017).

The participants’ interview responses with regard to their experience of using the ChatBot eventually gave rise to a total of four major categories, including technical aspects, the learning process, content, and recommendations. First, the technical aspects refer to all of their opinions about the technical issue linked to the ChatBot such as how easy it was for them to use and to access, the flexibility of its use, and its real-time functions or affordances. Second, both the category of learning process and of content contains a positive and a negative subcategory. The former refers to the advantages of using ChatBot as indicated by some of the respondents in their interviews, and the latter deals with the disadvantages related to such use. This category also includes their opinions about the learning and practice sections of the ChatBot. Third, the content category focuses on their comments about the experience with each level as well as the vocabulary taught. Finally, the recommendation category features their recommendations for future development of ChatBot.

Table 11

OPEN IN VIEWER

Table 11.

The Category of Technical Aspects.

Technical aspect	Verbatim expression
Easy to use	Helpful because of no need for a lot of instruction to understand what should I do to connect with the bot. There is no instruction to use it but I can use it. (S1)
	… easy to use, not so many menus appearing in the ChatBot that sometimes can make people confused. The interface of the ChatBot is simple and easy to interact with … (S2)
	I think it is easy to use. (S4)
	Well, it’s very easy to use … and how to operate it is very easy, just type “start” and we get everything. (S5)
	I think it is easy to use … (S6)
Easy to access	… it is integrated to mobile phone, especially in LINE. I think a lot of people use LINE so it is easy to access. (S1)
	… this kind of method is so practical. Because it uses LINE and LINE is an application that people use every day, almost everybody uses LINE, … The third one is accessible, at anywhere and anytime so anytime I want to use it, there it is, the ChatBot is in my cell phone, in my LINE, just use it. … I can use in my leisure time to learn Chinese. (S2)
	The part I like the most is because it is easy to access, because it is using LINE. (S6)
Flexible	The first reason why that is useful because maybe for some of us, we have a lack of time, a lack of spare time. So, by using that ChatBot, it can help us to learn Mandarin. Whether we are in busy time or in our hectic time. (S4)
Real-time	I think because it is real-time, so if I answer the question now, I can get the answer now. (S4)

lists the verbatim expressions associated with the above category of technical aspects. Almost all of the interviewees agreed that the ChatBot was easy to use. One of them said that she could still figure out how to use it even without the instructions (S1). It should be noted that none of interviewees found the interface of the ChatBot confusing. Their other responses also tend to agree with the findings about the TAM as discussed earlier in the descriptive analysis, which proved that the students had found the ChatBot as easy to use. The ChatBot was also described by the interviewees as convenient thanks to its access via LINE, an instant messaging application with which they are familiar. Three participants stated that the LINE access allowed them to learn Chinese with the ChatBot at any time anywhere. They also appreciated the ChatBot for the flexibility available to them in terms of the amount of time they would like to spend on learning and the real-time answers it offered in that process.

Interview expressions associated with the category of learning process category are shown in Table 12

OPEN IN VIEWER

Table 12.

The Category of Learning Process.

Learning process	Verbatim expression
Multiple-choice section (for practicing)	… when I am trying to practice, I can trial and error so when I am wrong, I can remember the right answer. (S1)
	… I can test my knowledge or my memory using the practice section. So it can help me to train my brain, remember the word in random order, which is good for me. (S2)
	The practice section, which is pretty good is repeat a lot of times. So, if I didn’t get it two or three times, after while I just keep going and learn at some point very fast. (S5)
	… because it replies the correct answer in the practice section. (S6)
	… I want the words that I answer incorrectly to appear more often. (S2)
	… and I think something that you need to improve is that the ChatBot is repetitively giving the same question and answer. So I think for some of the users, I think they get bored with that kind of situation. (S4)
	I think sometimes a little bit boring when I used it because the question is the same. (S6)
Vocabulary list (for learning)	I like the learning section of the ChatBot. Because … I will have a kind of sheet, full of the words, and I can try to memorize all of the words … (S2)
	Learn section, because it makes me easy to remember. (S3)
	… I dislike the learning section. … Because only words that I need to read … (S1)

. In this category, the interviewees mostly commented on the practice section of using the ChatBot, namely, the part where they experienced multiple-choice questions about the meanings of Chinese characters and to each of whose questions they had to choose to the correct answer out of four possible ones in English each time. The preprogrammed logic herein was set to generate questions containing random choices in accordance with a chosen level. Students may answer the questions with no time limit and with no cap on maximum score to stop them from getting as much practice as they wanted. Some interviewees shared that the repetition they got from the section really helped them remember the vocabulary. However, some of them found such repetition boring, especially so when it was the same set of vocabulary that kept on being tested by the questions. The learning section, featuring vocabulary lists of different levels whose Chinese characters were shown alongside their pinyin, meanings in English, and sentence examples, was the part most popular among the interviewees due to the fact that the arrangement of vocabulary lists by levels of learning allowed them to view and memorize a desired level of vocabulary before engaging in the practice section at the corresponding level. However, this section was advised to strengthen its visual appeal by adding additional elements to its current content, which was entirely text-based, to avoid boring the users.

The third category centers on the content of the learning materials by the ChatBot, which mainly dealt with vocabulary. Table 13

OPEN IN VIEWER

Table 13.

The Category of Content.

Content	Verbatim expression
Differentiation of the levels	The ChatBot also provides 4 different levels we can select. This is really helpful in the learning process because I can start from the easy one and then do it until I get used to the words and then progress to the next level gradually. (S2)
Limited to vocabulary	I think because you have limited information. I am only understanding the vocabulary but I can't use it for the conversation. (S1)
	… there is no pronunciation practice there. (S1)
	… I cannot hear the pronunciation. (S2)
	… I think they need grammar or maybe another thing of the lesson, not just vocabulary. (S4)
	And what's not so good because at the moment it only has 4 levels, so there is not much. (S5)
	… and the bad thing is because it is not focused on grammar. (S6)

displays the verbatim expressions associated with the issue of content as described by the interviewees. The ChatBot ranked its vocabulary by four levels of difficulty from the easiest to the most difficult. One interviewee acknowledged that having such configuration of levels in place with the ChatBot was indeed helpful to learners. However, several disadvantages identified by the interviewees with regard to this category of the ChatBot still remained to be improved, for instance, its current lack of coverage for grammar and, in particular, pronunciation of Mandarin Chinese, especially so with the latter, where much of their interest in mastering the language is at stake.

The last category was reserved for the recommendation formed by the interviewees and is of particular importance to the future development of the ChatBot. Table 14

OPEN IN VIEWER

Table 14.

Recommendations.

Recommendation	Verbatim expression
Combine levels	… honestly, when I am trying the first level and now I am in the third level, I forget what is in the first level. So when we are in level three, please include the first level also. (S1)
	And also maybe will be good to implement, I don’t know if it is there already, maybe like practice section for all levels, so you have only one thing, and ask words for every level, or maybe can select only level 1 and 2, so can repeat the words but don’t have to go through each levels by self. (S5)
Add pronunciation	… I hope that maybe in the future you can put this kind of feature in the ChatBot. So the ChatBot can check my pronunciation is correct or not. (S2)
	The hardest part is how I say the words, so it could be good if you have the pronunciation. (S5)
Add more levels	… I hope I can have more levels, instead only has 4 levels. (S2)
	I think you are in the right way, just have to implement more levels maybe. (S5)
Gamification	… I think you need to put the score in the practice section like a quiz. (S3)
	Maybe you can make some game, like gamification like you give some challenges for the user. So the user is interested to try more. (S4)
Expand types of practice	… then the second one is maybe there is another practice section. I mean not only question about word by word but put in a sentence. (S3)

summarizes the recommendations they made. First, it is suggested in the practice section that users should be allowed to select multiple levels to customize their own practice. For instance, they may choose to combine both levels of one and three on the ChatBot to try out questions containing both levels of vocabulary. Also, it was suggested that additional levels may be added to the current ones available. As discussed earlier in the disadvantages to be addressed by future ChatBots in terms of content, voice, or pronunciation of words were recommended to this category as well. Finally, a new scoring mechanism was also recommended by the interviewees to increase users’ level of interest in practice. It is suggested that the ChatBot should add points to the user’s score for choosing the correct answer and deduct points when he or she fails to do so.

A number of opinions and suggestions were provided by the users in the interviews. It can be inferred that they found the ChatBot easy to use and user-friendly and its interface as well as the instruction were clear enough for meaningful interaction. The incorporation of mobile instant messaging with LINE as a well-known and commonly used platform also served as an important leverage in the functioning of the ChatBot. In addition, the practice section was considered useful for students to remember Chinese vocabulary. However, the ChatBot required further improvements to allow users to customize their own practice among different levels, to experiment with a different scoring mechanism with users might to want to practice with the ChatBot more, and allow them to revisit questions they did not get right. Not only does the practice section need further improvement, but the currently missing coverage for grammar and pronunciation should be in place to make up for the content provided by the learning section. One other important element for future ChatBot is to add more levels to its current ones. Finally, it is worth mentioning that all of the participants in the present study expressed willingness to recommend the ChatBot to other learners.

Discussion and Conclusion

This study implemented the technology affordance of a ChatBot in the learning of Chinese vocabulary. The ChatBot commonly known as a virtual assistant was adopted as a substitute for the teacher or instructor. The research was conducted with two different groups, one experimental group and one control group each with a different learning environment. The experimental group was provided with a one-on-one tutoring environment while the control learned in a one-on-many classroom. The objective of this study was to explore the impact of the ChatBot on students’ learning achievement. As it was the first time such technology was deployed in the field of Learning Mandarin Chinese as a Second Language, technology acceptance was also a central topic to be explored among many others, including its PEU, PU, PE, and the BI among users. The users’ experience was also elicited in this study via semi-structured interviews, which was rather important for the present research to understand the feedback of the users and its relations to the future development of the ChatBot.

Both the experimental and the control group showed statistically significant improvement in learning achievement with the proposed technology. These findings imply that the ChatBot can be applied in different learning improvements to improve students’ performance in learning Chinese vocabulary. Such result was also supported by Mahdi (2018) in a previous study, where he conducted a meta-analysis which successfully proved the effectiveness of mobile devices in vocabulary learning. In addition, both the average pre- and posttest scores achieved in both groups across the three chapters instructed by the ChatBot in the present study not only indicated a significant difference between the effectiveness of the ChatBot in the classroom and in the tutoring sessions (the tutoring sessions yield a better score than the classroom), but also in the retention test. A previous study (Mostow et al., 2003) also found that one-on-one tutoring led to better learning outcomes among students in their reading skills than those of giving verbal instructions in a regular classroom. It was then assumed that those who received one-on-one tutoring prevailed due to the advantage they had in their setting where they were allowed to ask questions to the instructor directly in the advent of a problem. This then led to a reduction of the kind of teaching and learning process that were nonproductive (Juel, 1996). Bloom (1984) and Juel (1996) also argued that one-on-one teaching is the most effective method of instruction. It is believed that the guidance provided to the students in their tutoring sessions as well as the use of the ChatBot both increased the effectiveness of their learning process.

The ChatBot was designed to apply its PRS to rote learning. Nacera (2010) argued that memorizing words by repeating them again and again is far from effective because doing so only requires the use of a surface strategy. However, in Yang’s (2011) study, although recognizing the benefit of expanding one’s vocabulary by way of memorizing words during the early stage of learning a new language, it also stressed that such use of rote memorization alone is not nearly as important as opposed to the simultaneous incorporation of the other techniques as well. The ChatBot, in the present study, utilized an important feature to help its users review their newly acquired vocabulary. By generating multiple-choice questions with random candidate responses, it managed to test the retention of the user and reinforced his or her memory of the vocabulary with feedback in cases where incorrect responses were made. The PRS involved also provided instant feedback to help the user achieve better learning (Latham & Hill, 2014). Thus, the combined use of rote memorization and PRS by the ChatBot successfully helped the students improve their learning outcomes on Mandarin Chinese vocabulary through one-on-one tutoring.

Evaluating the technology acceptance of the ChatBot is important for further development of such application of language learning. Those who responded to the survey in this study not only have found the ChatBot easy to use, useful, and enjoyable but also said that they would continue using it in the future, which echoed with the interview responses, where the participants agreed that the ChatBot was easy to use. On the basis of the above findings, the present research also claims the PU among users as the predictor of their BI, their PEU the predictor of their PU, and their PE the predictor of their PEU. In addition, significant positive correlations were found between all of the variables, except for PEU, and the students’ learning achievement. Especially, the result associated with PU was consistent with that of Tsai’s (2014) study, which did not lead to the same conclusion with PEU. This may have to do with the fact that the study had centered mostly on writing while the tools in use required further improvements in various components surrounding their easiness to operate. PE was found to have a significant correlation with learning achievement. This finding is in line with previous research (Fokides & Atsikpasi, 2018), which indicated that the design of the ChatBot may cause its users to have positive learning experience and thus achieve better learning outcomes. This also means that the development of a future ChatBot should focus more on both its PU and PE. The BI among users, however, should also be incorporated. Successful attempts at exploring any of the above would help learners achieve better outcomes in terms of building up Chinese vocabulary.

Regardless of the advantages of the ChatBot in learning vocabulary of a second language, a number of disadvantages were identified by the student interviewees, whose feedback, the bulk of which concerned the content for learning, is important for the future development of the ChatBot. Some of their suggestions include the need for the future ChatBot to reach beyond the realm of vocabulary learning and explore other areas of language learning, such as grammar, pronunciation, and conversation. Another main suggestion was to implement gamified quizzes in the practice section of the ChatBot as once argued by Kozlov and Johansen (2010), who contended that game-like features promote both learning enjoyment and motivation. All of these recommendations should be considered for future studies. In addition, conversation among users should be recorded toward further improvement of their learning experience. It is concluded herein that ChatBots are potential technology to help students learn Mandarin Chinese vocabulary. The ChatBot designed in this study was able to aid students’ vocabulary learning. Student participants also perceived the usefulness, enjoyment, and ease of use while learning vocabulary with the ChatBot. However, limitations were identified with regard to the proposed ChatBot and should be considered for further improvement.