A ChatBot for Learning Chinese: Learning Achievement and Technology Acceptance

Abstract

ChatBot has potential as a language learning tool, especially for learning Chinese vocabulary. This study aimed to investigate the impact of using a newly developed ChatBot to learn Chinese vocabulary by comparing how it works in different learning environments and to explore the ChatBot with reference to the Technology Acceptance Model (TAM). This study was conducted with 58 students divided into two independent groups. The control group used ChatBot in a one-on-many classroom. The experimental group applied the ChatBot in one-on-one tutor sessions. A pretest and a posttest were used to measure the effect of the ChatBot, while TAM was explored through questionnaire and interview. Data analysis includes a paired-sample t test, analysis of covariance, and levels of effect. The results indicated that the ChatBot significantly improved the students’ learning achievement and that having a one-on-one environment may lead to better outcome than what could be achieved in a classroom. The TAM model was tested using partial least square. The result showed that perceived usefulness was the predictor of behavioral intention, whereas perceived ease of use was not. The students agreed that the ChatBot benefited their learning of Chinese vocabulary, with several adjustments need to be made for further progress.

Keywords

ChatBot learning Chinese vocabulary learning technology acceptance rote learning

ChatBot, a version of virtual assistant, is a program that is able to imitate human conversation (Tatai et al., 2003). Several studies have investigated the possibility of using ChatBot in language teaching and learning (Fryer et al., 2019; Jia, 2004; Lu et al., 2006) and found it to be an effective means of stimulating students’ learning interest. Learning with a ChatBot is an accessible choice for learners, because it allows them to learn anywhere at any time. Moreover, learners felt more confident chatting with a bot even though it is obvious that humans are more intelligent than a programmed robot. Prior research conducted by Lu et al. (2006) investigated the usage of ChatBots for learning English. They developed a ChatBot whose system offered total English environment. The robot had built-in dictionaries and access to authorized conversational materials to conduct both speaking and question-answer functions. The main purpose of the ChatBot was to operate both as an assistant for the teacher and as a partner for the students. The ChatBot succeeded in promoting a one-on-one learning environment.

A one-on-one learning environment concerns the effectiveness of learning. The setting in this environment involves one teacher for one student. Based on that, some benefits of one-on-one learning environment can be delivered. For example, teachers can provide immediate feedback (Gettinger, 1983; Wentling, 1973), and students will have more time to interact with the teacher (Fisher et al., 1978). As shown in the study by Moody et al. (1973), students in the individual class performed better than those in the group class. Glass and Smith (1979) also found a negative correlation between group size and students’ learning achievement. They found that the better students’ learning achievement the smaller their group size. Such result then led to a hypothesis that students’ level of achievement in learning may be affected by changes in the environment.

In language learning, vocabulary is one of the most important part. Huckin et al. (1993) stated that two of the most important components in language learning are reading ability and vocabulary knowledge. For all of the language skills (speaking, writing, reading, and listening), vocabulary is an important key for an effective understanding of meaning (Mutalib et al., 2014). Comparing it with grammar, Wilkins (1972) said, “Without grammar very little can be conveyed, without vocabulary nothing can be conveyed.” (p. 111–112) Memorizing vocabulary tends to be difficult for students, especially memorizing Mandarin Chinese vocabulary, because learning Chinese is different from learning English. Chinese is a Sino-Tibetan-based language while English is a Latin-based language which uses alphabets (Chen & Dunsmoir, 2012).

Several technologies have been used for learning Chinese vocabulary. Edge et al. (2011) created a mobile application for learning Chinese vocabulary. They developed two versions of the mobile application, contextual version, and frequency-based version. Contextual version automatically provides Chinese vocabulary related to a nearby location. Frequency-based version features the use of Web corpus, from which the most to the least common frequency is derived for different levels of Chinese vocabulary to be taught. Both versions were effective tools to help students learn Chinese vocabulary. Also, it was proved that other mobile applications like WeChat may help students learn Chinese vocabulary when used to deliver materials (Yang & Yin, 2018). Another technology used for learning Chinese vocabulary was virtual reality (VR). In the study conducted by Legault et al. (2019), native English students experienced VR for learning Chinese vocabulary and agreed that VR was able to improve their learning achievement. All of the mentioned studies were examples of how technology can help students learn Chinese vocabulary. This further indicates how technology is of paramount importance for learning language. However, none of the studies discussed have incorporated any pedagogical use of a ChatBot. By developing a new platform, or ChatBot, this study is able to compare how different environments equipped with the proposed technology influence learning.

Rote learning is one of the memorization techniques whereby students are taught to memorize the targeted language from a vocabulary list (Yang, 2011). Researchers have stated that memorizing words from a vocabulary list cannot be separated from the context of vocabulary learning (Cortazzi & Jin, 1994; Yang, 2011). Rote learning is categorized as a cognitive strategy in learning vocabulary. Cognitive strategies as mentioned in Mutalib et al.’s (2014) study are “repetition and using mechanical means to study vocabulary” (p. 363) such as using notes for memorizing. Rote learning is done by repeating and rehearsing materials believed to promote academic success (Morton, 2011). Nevertheless, it also takes place as a learning process without or with a minimum of insight (Park, 1937), because it involves the use of surface learning approach, which focuses on the learning material itself (Marton et al., 1996) and in the acquisition of a new language, is indeed inevitable in the early stages (Yang, 2011). Yang (2011) also admitted that a number of initial words can be learned effectively and rapidly through rote learning. However, acquisition of vocabulary cannot be accomplished by rote memorization alone. In fact, the development of the ChatBot is not only based on the rote learning approach but also integrates a personal response system (PRS) approach.

PRS is a technology that makes real-time feedback possible and is able to engage the learner (Gharaie, 2016). Immediate feedback is important for the students to create an effective learning environment based on their current understanding levels (Chen et al., 2010). PRS helps students to gain a better understanding of the learning materials by allowing them to learn at their own pace (Chan et al., 2019). The proposed ChatBot is based on the rote learning approach with a vocabulary list and PRS for memory retention through exercises comprised of multiple-choice questions. The ChatBot, which is based on a mobile instant messaging platform, is convenient for learners to use, because it can be accessed everywhere at any time as long as the user is connected to the Internet, and it is expected to be helpful to students in learning Chinese vocabulary.

A ChatBot can only be described as a successful information technology when it is accepted among users (Nikou & Economides, 2017). Technology Acceptance Model (TAM; Davis, 1985), as a theory of user acceptance, is the most accepted one by far in terms of investigating user’ acceptance behavior (Liu et al., 2009). TAM originally consisted of two factors, perceived ease of use (PEU) and perceived usefulness (PU), both of which may influence people’s behavioral intention (BI). How people expect a technology to improve their job performance defines their PU of the technology (Davis et al., 1992). In regard to its PEU among people, it is measured by how strongly they confirm the technology as easy to use in performing their tasks free of effort. In this study, the measurement of the PU of different items and their PEU is drawn from the previous research by Davis (1989). Several studies have hypothesized PEU as a predictor of PU (Farahat, 2012; Rodrigues et al., 2016; Venkatesh, 2000; Yi & Hwang, 2003). Their hypothesis derives from the original TAM, which posits that the easier the technology is to use, the more users will perceive it as useful. The relationship between the two factors is assumed as having one causal direction as hypothesized below:
H1. PEU positively affects PU.
The main idea of TAM is that user BI determines the user’s acceptance of technology, which in other words, is determined by the user’s PU and PEU (Farahat, 2012). Warshaw and Davis (1985) define BI as “the degree to which a person has formulated conscious plans to perform or not perform some specified future behavior.” (p. 214) Logically, if users perceive a technology as being useful and easy to use, they are likely to plan to use it in the future. The present research intends to revalidates such inference about the use of ChatBot through the following hypotheses:
H2. PEU positively affects BI.

H3. PU positively affects BI.
Due to the limitation of the TAM, namely, that some important factors are not included, researchers often extend it by adding other important variables (Hussain et al., 2016). Perceived enjoyment (PE) is one of the important factors and is related to user–system interaction (Venkatesh, 2000). The ChatBot in this study was designed not only for learners to easily use it but also to enjoy using it. Davis et al. (1992) defined enjoyment as the extent to which the activity of using the computer is perceived to be enjoyable in its own right. Venkatesh (2000) found that PE was the predictor of PEU, which means that as long as users feel enjoyment when operating a system, they are likely to perceive its ease of use which will impact their BI. Thus, this research hypothesizes the relationship between enjoyment and ease of use as follows:
H4. PE positively affects PEU.
Consequently, acknowledging the success of the ChatBot is dependent upon the user acceptance study. Moreover, in order to obtain better understanding of the factors influencing students’ learning achievement, one more hypothesis was added to examine how the factors of TAM may affect students’ learning achievement. Naya and Ibáñez (2014) suggested that PE and PEU may influence learning outcomes. Fokides and Atsikpasi (2018) also claimed out that PU was a determinant of learning outcome. These previous studies further propel the present research to look into the correlations between TAM-related variables and students’ learning achievement, and thus the following hypothesis is made:
H5. TAM variables (PEU, PU, and PE) have a positive correlation with students’ learning achievement.

Research Questions

Based on the background and the literature review above, this study investigated the effect of using the proposed ChatBot for learning Chinese vocabulary and explored the technology acceptance of the ChatBot. The specific purposes of this study are listed below:
Does the use of the ChatBot in different learning environments improve the students’ learning achievement in learning Chinese vocabulary?

What are the students’ PEU, PU, PE, and BI when learning Chinese vocabulary using the ChatBot?

Is there any correlation between students’ learning achievement and TAM variables?

What are the students’ experiences of learning Chinese vocabulary with the ChatBot?

Method

This research was conducted with a mixed-methods approach (Creswell & Clark, 2007). Triangulation method was adopted, although quantitative method remained predominant (Chiva-Bartoll et al., 2020). The quantitative method was applied both to the pre- and posttest scores and to the results of the questionnaire. Qualitative method was used toward the analysis of semi-structured interview responses.

Participants

This study involved 58 international students at a university in Taiwan. The students were of different nationalities (26 Indonesians, 7 Thais, 7 Vietnamese, 5 Germans, 4 Czechs, 4 Ethiopians, 1 Austrian, 1 Italian, 1 Mexican, 1 Russian, and 1 Swiss). All of them are either full-time or exchange students and were divided into control and experimental group through convenient sampling. The control group consisted of 39 students (19 female and 20 male) and was referred to as group A. The education background of students in this group was as follows: 8 undergraduates, 26 masters, and 5 doctorates. The experimental group, or group B, was formed by the remaining 19 students (12 female and 7 male). The education background of group B students was as follows: 5 undergraduates, 8 masters, and 6 doctorates. Group A students were enrolled in class Mandarin Chinese Level 1 at the university. Group B students were enrolled in a one-on-one tutor class for those who had never learned Chinese before. Both groups were learning basic Chinese with the language of instruction in English. In the control group, or group A, the respondents learned with a traditional one-on-many classroom and using ChatBot. Students in the experimental group used ChatBot as their learning tool in the one-on-one environment. It was confirmed that each student would and knew how to use LINE, where the ChatBot is based and which is an instant messaging application on their own mobile phones, to interact with the ChatBot.

Learning Materials

Materials used in this research were from Practical Audio-Visual Chinese 1 (refer to Figure 1
Figure 1.
Practical Audio-Visual Chinese 1 Book.
), a textbook that the students used in class. Vocabulary on the ChatBot was from this book and focused on the first three chapters. Chapter 1 contains 30 words including 我 (wǒ) = I, 你 (nǐ) = you, 他 (tā) = he or him, and so forth. Chapter 2 contains 24 words including 忙 (máng) = busy, 天氣 (tiānqì) = weather, 熱 (rè) = hot, and so forth. The last chapter, Chapter 3 contains 30 words such as 貴 (gùi) = expensive, 汽車 (qìchē) = car, 電影 (diànyǐng) = movie, and so forth.

Research Tools

Pretest and Posttest

A pre- and posttest for each chapter were implemented to measure the students’ learning achievement. Both tests were developed by the author and were validated by two Mandarin Chinese Level 1 teachers. The pretest was arranged to measure students’ prior knowledge before participating the study. The pre- and posttest for each chapter were of a different type. The test on Chapter 1 contained 10 multiple-choice questions. The test on Chapter 2 contained 10 word-matching questions. The test on Chapter 3 contained 15 word-matching questions. Each chapter used the same pre- and posttests with the sequence of the posttest questions rearranged. The highest score for both tests was 100.

Questionnaire

The technology acceptance questionnaire items were obtained from Davis’ (1989) study (Cronbach’s α of PU was .97 and, for PEU, it was .86) and Liao et al.’s (2008) study (Cronbach’s α of PU was .94 and, for PEU, it was .90) and were modified (subject replaced) by the author. The questionnaire had two constructs, PEU and PU (PU). PEU included seven items (the internal consistency reliability, ICR, among which is .861), and PU consisted of four items (the ICR among which is .912). The extended TAM-PEPE questionnaire was derived from Gopalan et al.’s (2016) study (whose Cronbach’s α was .80) and Liao et al.’s (2008) study (whose Cronbach’s α was .90), and the items were modified (subject replaced) by the author. PE consisted of six items (the ICR among which is .835). The BI items were adopted from Venkatesh’s (2000) study which had ICR of .90. BI consisted of two items (the ICR of which is .947). That questionnaire was only given to coincide with the last posttest for the experimental group.

Interview

A semi-structured interview was conducted in response to the last research question about students’ learning experience of a new learning tool, the ChatBot. The author developed protocols (as shown in Table 1
Table 1.
Interview Protocols.

Question

1. Please tell me three things that attract your attention when using the ChatBot.

2. In your opinion, what are two things that are helpful when using the ChatBot and two things that are not helpful? Please give examples and reasons.

3. According to your experience of learning Mandarin Chinese using the ChatBot, which parts do you like and dislike the most?

4. How would you use the ChatBot in the future?

5. What are your recommendations about the ChatBot for the future?

6. Would you recommend ChatBot to others as a learning tool?

) for the interview and checked their validity with a professor who is an expert in qualitative research.

The ChatBot Mechanism

The proposed ChatBot, named Xiaowen, was developed based both on rote learning for memorizing vocabulary and on a PRS to help with memory retention. This system was implemented via Python programming language and LINE Developer platform; hence this ChatBot can only be accessed through LINE. Rote learning is a technique for memorizing vocabulary from a vocabulary list. Thus, this ChatBot provides vocabulary lists at different levels. To maintain the memorization of the vocabulary, ChatBot was equipped with a multiple-choice practice tool with a PRS.

When learners add “Xiaowen” to their friend lists on LINE and greet the ChatBot with the first message they send, the ChatBot mechanism will be activated and reply with a welcoming message. Xiaowen immediately introduces the levels of vocabulary learning available and instructs the learner to text “start” for entering the learning section. Once the bot received the message, it shows the buttons whereby the users may choose from different levels of practice; currently levels 1 to 4 are available with the bot. These buttons help the user to interact with the bot more easily. Next, the user is asked whether to activate either learning mode or practice mode. If the former was chosen, the bot then displays a list of vocabulary containing preprogrammed Chinese characters, the pinyin of each, what they mean in English, and sentence examples corresponding to the level chosen. If the latter was chosen, the bot randomly selects a word from its vocabulary list to produce a practice question, which contains Chinese characters and their meanings in English. The options are randomly generated from an English wordlist at the same level as previously indicated. And each question contains one correct answer only.

The practice mode functions on a PRS where the bot serves as a substitute for the teacher or instructor. Once an answer is chosen as a response to the question, the bot immediately provides feedback based on the result. If the user choses the correct answer, the bot recognizes it and moves on to the next question. On the contrary, if a false answer is chosen, the bot informs the user of it and reveals the correct response at the same time. The questions loop until the user texts “start” again to change the level of practice. An example of a conversation with the bot is shown in Figure 2
Figure 2.
(a–h) Examples of Conversation With the Bot. (a) Greetings message; (b) choose level; (c) learn or practice?; (d) learning section; (e) continue to practice; (f) multiple choice question (g) correction of wrong answer and (h) feedback of correct answer.
.

Procedure and Instruction

The participants in this study were divided into two groups, a control group (Group A) and an experimental group (Group B). The experimental group was assigned to learn Mandarin Chinese with ChatBot in a one-on-one learning environment while the control group used ChatBot to memorize the vocabulary in a one-on-many classroom. The length of the experiment was 4 weeks in total. Group A was in session every Monday. Group B, with its one-on-one tutoring, took place on different days and times for every student. However, both groups kept the same length of experiment and followed the same weekly agenda. At the beginning of the first week of the experiment, the participants were instructed to take a pretest which lasted 3 minutes. Participants in both groups then spent the next 5 minutes learning about the ChatBot in an introduction. In the next 5 minutes, participants in group A were assigned to use ChatBot for learning Mandarin Chinese vocabulary before receiving further instructions, which lasted for 10 minutes per session, from their teacher. Meanwhile group B participants were assigned with a 15-minute learning session. While Group A spent 5 minutes on learning with the ChatBot and 10 minutes later on with the teacher, whose duty was to explain to them the details of the material, Group B, however, spent an entire 15 minutes on learning with the ChatBot, with which the tutor answer the questions raised by the students. It should be specified herein that the difference in the timespans between the sessions implemented among the two groups does not imply any difference in the amount of material covered in either. Once the groups finished their assigned sessions, the participants spent 3 minutes taking a posttest. The second posttest was conducted the next week along with the pretest on the new chapter. The process repeated three times, each with its own chapter (Chapters 1–3) for learning. On the 4th week, the questionnaire of technology acceptance was administered in both groups, and three participants in each were selected randomly to take part in the one-on-one interview.

Data Analysis

The statistical analysis performed in this study varied from one research question to another. The first research question was explored using two statistical tests: a simple paired t test and analysis of covariance (ANCOVA). The former was used to investigate the differences between the learning achievement of the two groups in the pre- and posttests. ANCOVA was then conducted to determine further differences between the two. Both the simple paired t test and ANCOVA were performed using SPSS 22. The second research question, which concerns the TAM, was tested using partial least square (PLS), conducted via Smart-PLS 3. The third research question was explored using Pearson correlation test on SPSS 22. The last research question, which concerns students’ opinions from the interviews, however, required recorded data and its transcripts, which were then coded for further analysis in the conclusions.

Result

Learning Achievement Within Groups

A pretest was implemented in both groups prior to their exposure to the materials drawn from across all of the three chapters in the textbook. The results of their pre- and posttests are displayed in Table 2
Table 2.
Paired Sample t Tests of the Overall Learning Achievement (Pretest and Posttest).

Group Pretest
Posttest
n 95% CI for mean difference t df d

M SD M SD

A 71.63 15.19 88.03 12.35 39 [13.11–19.70] 10.09* 38 1.18

B 49.82 16.06 91.33 11.77 19 [35.45–47.57] 14.38* 18 2.95

p < .001.

, showing a significant difference each group achieved with their pre- and posttest scores. Both groups had taken three pretests as previously mentioned. The average score of the three pretests indicate that both groups had an average command of Chinese vocabulary. However, it was soon surpassed by the average of their posttest scores. This proved that using the ChatBot may improve students’ learning outcomes. In addition to the above comparison between the pre- and posttest scores, results of the retention tests were also investigated. As shown in Table 3
Table 3.
Paired Sample t Tests of the Overall Learning Achievement (Pretest and Retention Test).

Group Pretest
Retention test
n 95% CI for mean difference t df d

M SD M SD

A 71.63 15.19 80.95 15.76 39 [7.02–11.62] 8.193 38 0.60

B 49.82 16.06 93.21 8.28 19 [36.41–50.36] 13.08* 18 3.40

p < .001.

, there was a significant difference between the scores achieved in the retention test and those obtained in the pretests. This means that after 1 week of using the learning tools, both groups still remembered their vocabulary and maintained a medium–large effect according to Cohen’s classified effect sizes (0.2 = small effect, 0.5 = medium effect, 0.8 = large effect).

Learning Achievement Between Groups

To compare the differences between the learning outcomes of the experimental group and those of the control group, the ANCOVA test was performed with the pretest scores as the covariates. Table 4
Table 4.
ANCOVA Test of the Overall Learning Achievement (Posttest).

Source Type III sum of squares df Mean square F Sig. Partial eta squared

Corrected model 4,293.950^a 2 2,146.975 28.552 .000 .509

Intercept 10,592.802 1 10,592.802 140.871 .000 .719

Pretest 4,154.893 1 4,154.893 55.255 .000 .501

Group 2,095.599 1 2,095.599 27.869 .000 .336

Error 4,135.728 55 75.195

Total 469,035.111 58

Corrected total 8,429.678 57

^a
R²= .509 (adjusted R²= .492).

The mean difference is significant at the .05 level.

displays the result of the ANCOVA test of the posttest scores and reveals a significant difference between groups, F(1, 55) = 27.869, p = .000. It means that experimental group was better than control group, as group A has a higher score than group B.

Table 5
Table 5.
ANCOVA Test of the Overall Learning Achievement (Retention Test).

Source Type III sum of squares df Mean square F Sig. Partial eta squared

Corrected model 8,229.114^a 2 4,114.557 51.873 .000 .654

Intercept 6,999.628 1 6,999.628 88.246 .000 .616

Pretest 6,308.239 1 6,308.239 79.529 .000 .591

Group 6,514.154 1 6,514.154 82.125 .000 .599

Error 4,362.595 55 79.320

Total 431,301.778 58

Corrected total 12,591.709 57

^a
R²= .654 (adjusted R²= .641).

The mean difference is significant at the .05 level.

shows the ANCOVA result of the retention test with the pretest scores as the covariates. The results indicate a significant difference between groups, F(1, 55) = 82.125, p = .000), meanings that learning in a one-on-one environment had led to better memory retention than learning in a one-on-many classroom had.

TAM

The first analysis presented with regard to technology acceptance is the descriptive statistics of each construct (as seen in Table 6
Table 6.
Descriptive Statistics of the Technology Acceptance Result.

n M SD Min Max

PEU 58 4.32 0.71 1 5

PU 58 4.19 0.67 2 5

PE 58 4.01 0.82 1 5

BI 58 4.08 0.75 1 5

PEU = perceived ease of use; PU = perceived usefulness; PE = perceived enjoyment; BI = behavioral intention.

). The data used in the analysis were drawn from both groups, which account for a total of 58 respondents. Their ratings for each constructs were calculated first based on their responses and were then averaged for further analysis (Malaquias et al., 2018). The results indicated that both Groups A and B participants perceived the usefulness and the ease of use of the ChatBot with their mean scores of PEU and PU both exceeded four. This means that both groups agreed with the statements on the questionnaire. Moreover, both groups perceived the enjoyment of learning Chinese with the ChatBot, although the average score on this construct was the lowest among all. In terms of the students’ BI, responses in both groups indicated a willingness to use the ChatBot in the future.

Results of the descriptive statistics were applied to the model and the hypothesis tests of PLS developed by Wold (1982). Closely resembling other structural equation modeling (SEM) techniques, in which structural path coefficients and measurement model parameters are measured (Yi & Hwang, 2003). PLS is a good alternative of statistical analysis in cases with a rather small sample size (Wong, 2013); the present research, with as few as 58 participants in total, thus justified its application.

A handful of measurements were available to be chosen by the present research in its initial assessment of latent variables, reliability, and validity. Among all, measurement of reliability must include indicator reliability as well as ICR. While the preferred score of indicator reliability should amount to .70 and above (Barclay et al., 1995; Fornell & Larcker, 1981), the composite reliability in the ICR should reach greater than .7 (Barclay et al., 1995; Fornell & Larcker, 1981). Meanwhile validity consists of convergent validity and discriminant validity. The former checks the average variance extracted (AVE) and should be at least 0.5 (Bagozzi & Yi, 1988), while the latter measures the AVE number and latent variables correlation. Fornell and Larcker (1981) advised that the square root of AVE in each latent variable should be greater than the correlations among the latent variables.

Table 7
Table 7.
Factor Structure Matrix.

BI PE PEU PU

BI1 0.948 0.763 0.582 0.725

BI2 0.948 0.777 0.629 0.712

PE1 0.646 0.825 0.734 0.707

PE2 0.718 0.878 0.627 0.694

PE3 0.666 0.852 0.672 0.682

PE4 0.737 0.795 0.519 0.695

PE5 0.639 0.852 0.534 0.616

PE6 0.708 0.845 0.533 0.674

PEU1 0.632 0.719 0.821 0.667

PEU2 0.247 0.349 0.666 0.323

PEU3 0.293 0.337 0.579 0.255

PEU4 0.308 0.416 0.770 0.400

PEU5 0.557 0.628 0.713 0.591

PEU6 0.531 0.523 0.635 0.634

PEU7 0.200 0.261 0.633 0.356

PU1 0.652 0.651 0.602 0.826

PU2 0.589 0.600 0.568 0.833

PU3 0.529 0.640 0.555 0.815

PU4 0.774 0.829 0.725 0.924

PEU = perceived ease of use; PU = perceived usefulness; PE = perceived enjoyment; BI = behavioral intention. Loadings > = .7 in bold.

displays the factor structure matrix for measuring the indicator reliability in each item toward its latent variables. As shown in the results of the study, two items of the BI variables have factor loading greater than .7 with a value greater than those of other latent variables. The PE variable lists six items, and the factor loading of each not only exceeds .7 but also surpasses all the other values drawn from other latent variables. However, in the case of PEU variable, as many as four items (PEU2, PEU3, PEU6, and PEU7) fail to meet the expected criterion value of .7. Thus, elimination of these items was proposed. Finally, the PU variable consists of four items, and the factor loading of each exceeds .7, rendering all of them acceptable.

Table 8
Table 8.
Factor Structure Matrix Without Low Value Loadings.

BI PE PEU PU

BI1 0.948 0.760 0.582 0.725

BI2 0.949 0.775 0.633 0.712

PE1 0.646 0.830 0.766 0.707

PE2 0.718 0.876 0.630 0.694

PE3 0.666 0.861 0.749 0.683

PE4 0.737 0.786 0.488 0.693

PE5 0.639 0.847 0.527 0.615

PE6 0.708 0.840 0.509 0.674

PEU1 0.632 0.723 0.862 0.668

PEU4 0.308 0.421 0.777 0.403

PEU5 0.557 0.635 0.822 0.592

PU1 0.652 0.648 0.505 0.818

PU2 0.589 0.603 0.600 0.839

PU3 0.529 0.640 0.518 0.812

PU4 0.774 0.829 0.727 0.926

PEU = perceived ease of use; PU = perceived usefulness; PE = perceived enjoyment; BI = behavioral intention. *Loadings > = .7 in bold.

shows the factor structure matrix with PEU2, PEU3, PEU6, and PEU7 removed. All of the factor loadings meet the expected criterion value of 0.7 and thus pass the indicator reliability test.

The next reliability test follows in the form of an ICR test. Table 9
Table 9.
ICR, AVE, Correlation Constructs.

ICR AVE BI PE PEU PU

BI 0.947 0.899 0.948

PE 0.935 0.706 0.810 0.840

PEU 0.861 0.674 0.641 0.751 0.821

PU 0.912 0.723 0.758 0.808 0.700 0.850

PEU = perceived ease of use; PU = perceived usefulness; PE = perceived enjoyment; BI = behavioral intention; ICR = internal consistency reliability; AVE = average variance extracted.

shows the ICR of each latent variable, its AVE, and the result of its comparison between the AVE square root and the correlation among the latent variables. According to Bagozzi and Yi (1988), composite reliability can be used as a replacement of Cronbach’s alpha. This study checks the reliability of its internal consistency with composite reliability. Its results, which show an ICR above .7 on each latent variable, pass the reliability test. As for the validity test, the AVE value on each latent variable has an acceptable score above 0.5 and thus suffices the requirements of the convergent validity test. The last validity test investigates the discriminant validity of the study results by comparing the AVE square root of each latent variable and the correlation among the latent variables. Each AVE square root is expected to exceed that of the correlation among the variables both by row and by column (Yi & Hwang, 2003). The diagonal bold values in Table 9 are the AVE root square between the constructs while the other values indicate the correlations between constructs. The results approve the discriminant validity where each square root of the AVE scores is higher than the correlations among the construct.

The second stage of PLS was to analyzes the structural model and to validate the research hypotheses. The result in Figure 3
Figure 3.
PLS Test for the Model.
shows that the coefficient of determination (R²) for the variable of BI was .598. This means that the PEU and PU variables explain 59.8% of the variance in BI among student participants. The variable of PU has an R² of .49, which shows that the PEU explains 49.0% of the variance in PU. The variable of PEU has an R² of .564, which means that the PE variable explains 56.4% of the variance of PEU.

t tests as well as bootstrapping using 500 subsamples, as suggested by Chin (1998), were conducted to investigate the statistical significance of each path. The results, on the one hand, confirmed H1 (PEU has a significant effect on PU; β = 0.700; p < .001). On the other hand, the effect of PEU on the students’ BI was found inconsistent with H2 (β = 0.216). Also, the effect of PU on the students’ BI was in line with H3 (β = 0.606; p < .05), which posits that PU has a significant effect on their BI. Finally, the results also supported H4 (β = 0.751; p < .001), which claims that the PE among the students has a significant effect on their PEU.

Davis (1989) theorized that ease of use is a determinant of both PU and BI. This, however, as suggested in Davis et al.’s (1989) study, may change over time. They found that learners’ PEU may have a significant effect on their BI and have no significant effect on their PU when it was measured immediately after a 1-hour introduction to the system. However, it was found 14 weeks later that their PEU had a significant effect on their PU and had no significant effect on their BI. In the present study, the results regarding the effect of the students’ PEU on their PU is consistent with those of Davis (1989). Another study conducted by Chen et al. (2013) also yielded similar results, where learners’ PEU had a significant effect on their PU after engaging in a learning process with mobile devices. An increase in the ease of use of the ChatBot did not necessarily entail an increased BI of using a ChatBot, yet an increase in the ease of use indeed leads to an increase in user’s PU. Accordingly, the use of ChatBot is positively influenced by its PU rather than by its ease of use. Users tend to use the ChatBot as a result of its usefulness for learning Chinese vocabulary.

PE has a significant effect on PEU. This is consistent with the research findings of Venkatesh (2000), which asserts that ChatBot interface provides enjoyment to the users by displaying effective choice buttons alongside pictures. Venkatesh also emphasizes design features aimed to establish enjoyment for users and the importance of having a final goal in enhancing the ease of use of the system. Thus, it can be conferred that, in some cases, the absence of enjoyment may render the system less easy to use than it otherwise would appear to users. However, in the present study, the ChatBot did provide enjoyment to its users and thus significantly affected its PEU.

To highlight the benefit each TAM variable brought to the students’ learning achievement, a correlation test was conducted as follows. See Table 10
Table 10.
Correlations Between TAM Variables and Learning Achievement.

PEU PU PE LA

PEU Pearson correlation 1 .668 .678 .228

Sig. (2-tailed) .000 .000 .086

N 58 58 58

PU Pearson correlation 1 .799 .416

Sig. (2-tailed) .000 .000

N 58 58

PE Pearson correlation 1 .423

Sig. (2-tailed) .001

N 58

LA Pearson correlation 1

Sig. (2-tailed)

N

PEU = perceived ease of use; PU = perceived usefulness; PE = perceived enjoyment; LA = learning achievement.

Correlation is significant at the 0.01 level (2-tailed).

for its results.

According to Table 10, no significant correlation was found between PEU and learning achievement (r = .228). However, a significant positive correlation was found both between PU and learning achievement (r = .416, p = < .001) and between PE and learning achievement (r = .423, p = <.001).

Students’ Experience

The last research question was addressed using qualitative research. A semi-structured interview was conducted with six students, with three of whom drawn from each group. These were one-on-one interviews in English. All of the recordings from the interviews were transcribed and coded in order to provide the qualitative data needed for further analysis and to derive conclusions based on the categorization of various expressions found in the verbatim of the interview responses (Jiménez et al., 2017).

The participants’ interview responses with regard to their experience of using the ChatBot eventually gave rise to a total of four major categories, including technical aspects, the learning process, content, and recommendations. First, the technical aspects refer to all of their opinions about the technical issue linked to the ChatBot such as how easy it was for them to use and to access, the flexibility of its use, and its real-time functions or affordances. Second, both the category of learning process and of content contains a positive and a negative subcategory. The former refers to the advantages of using ChatBot as indicated by some of the respondents in their interviews, and the latter deals with the disadvantages related to such use. This category also includes their opinions about the learning and practice sections of the ChatBot. Third, the content category focuses on their comments about the experience with each level as well as the vocabulary taught. Finally, the recommendation category features their recommendations for future development of ChatBot.

Table 11
Table 11.
The Category of Technical Aspects.

Technical aspect Verbatim expression

Easy to use Helpful because of no need for a lot of instruction to understand what should I do to connect with the bot. There is no instruction to use it but I can use it. (S1)

… easy to use, not so many menus appearing in the ChatBot that sometimes can make people confused. The interface of the ChatBot is simple and easy to interact with … (S2)

I think it is easy to use. (S4)

Well, it’s very easy to use … and how to operate it is very easy, just type “start” and we get everything. (S5)

I think it is easy to use … (S6)

Easy to access … it is integrated to mobile phone, especially in LINE. I think a lot of people use LINE so it is easy to access. (S1)

… this kind of method is so practical. Because it uses LINE and LINE is an application that people use every day, almost everybody uses LINE, … The third one is accessible, at anywhere and anytime so anytime I want to use it, there it is, the ChatBot is in my cell phone, in my LINE, just use it. … I can use in my leisure time to learn Chinese. (S2)

The part I like the most is because it is easy to access, because it is using LINE. (S6)

Flexible The first reason why that is useful because maybe for some of us, we have a lack of time, a lack of spare time. So, by using that ChatBot, it can help us to learn Mandarin. Whether we are in busy time or in our hectic time. (S4)

Real-time I think because it is real-time, so if I answer the question now, I can get the answer now. (S4)

lists the verbatim expressions associated with the above category of technical aspects. Almost all of the interviewees agreed that the ChatBot was easy to use. One of them said that she could still figure out how to use it even without the instructions (S1). It should be noted that none of interviewees found the interface of the ChatBot confusing. Their other responses also tend to agree with the findings about the TAM as discussed earlier in the descriptive analysis, which proved that the students had found the ChatBot as easy to use. The ChatBot was also described by the interviewees as convenient thanks to its access via LINE, an instant messaging application with which they are familiar. Three participants stated that the LINE access allowed them to learn Chinese with the ChatBot at any time anywhere. They also appreciated the ChatBot for the flexibility available to them in terms of the amount of time they would like to spend on learning and the real-time answers it offered in that process.

Interview expressions associated with the category of learning process category are shown in Table 12
Table 12.
The Category of Learning Process.

Learning process Verbatim expression

Multiple-choice section (for practicing) … when I am trying to practice, I can trial and error so when I am wrong, I can remember the right answer. (S1)

… I can test my knowledge or my memory using the practice section. So it can help me to train my brain, remember the word in random order, which is good for me. (S2)

The practice section, which is pretty good is repeat a lot of times. So, if I didn’t get it two or three times, after while I just keep going and learn at some point very fast. (S5)

… because it replies the correct answer in the practice section. (S6)

… I want the words that I answer incorrectly to appear more often. (S2)

… and I think something that you need to improve is that the ChatBot is repetitively giving the same question and answer. So I think for some of the users, I think they get bored with that kind of situation. (S4)

I think sometimes a little bit boring when I used it because the question is the same. (S6)

Vocabulary list (for learning) I like the learning section of the ChatBot. Because … I will have a kind of sheet, full of the words, and I can try to memorize all of the words … (S2)

Learn section, because it makes me easy to remember. (S3)

… I dislike the learning section. … Because only words that I need to read … (S1)

. In this category, the interviewees mostly commented on the practice section of using the ChatBot, namely, the part where they experienced multiple-choice questions about the meanings of Chinese characters and to each of whose questions they had to choose to the correct answer out of four possible ones in English each time. The preprogrammed logic herein was set to generate questions containing random choices in accordance with a chosen level. Students may answer the questions with no time limit and with no cap on maximum score to stop them from getting as much practice as they wanted. Some interviewees shared that the repetition they got from the section really helped them remember the vocabulary. However, some of them found such repetition boring, especially so when it was the same set of vocabulary that kept on being tested by the questions. The learning section, featuring vocabulary lists of different levels whose Chinese characters were shown alongside their pinyin, meanings in English, and sentence examples, was the part most popular among the interviewees due to the fact that the arrangement of vocabulary lists by levels of learning allowed them to view and memorize a desired level of vocabulary before engaging in the practice section at the corresponding level. However, this section was advised to strengthen its visual appeal by adding additional elements to its current content, which was entirely text-based, to avoid boring the users.

The third category centers on the content of the learning materials by the ChatBot, which mainly dealt with vocabulary. Table 13
Table 13.
The Category of Content.

Content Verbatim expression

Differentiation of the levels The ChatBot also provides 4 different levels we can select. This is really helpful in the learning process because I can start from the easy one and then do it until I get used to the words and then progress to the next level gradually. (S2)

Limited to vocabulary I think because you have limited information. I am only understanding the vocabulary but I can't use it for the conversation. (S1)

… there is no pronunciation practice there. (S1)

… I cannot hear the pronunciation. (S2)

… I think they need grammar or maybe another thing of the lesson, not just vocabulary. (S4)

And what's not so good because at the moment it only has 4 levels, so there is not much. (S5)

… and the bad thing is because it is not focused on grammar. (S6)

displays the verbatim expressions associated with the issue of content as described by the interviewees. The ChatBot ranked its vocabulary by four levels of difficulty from the easiest to the most difficult. One interviewee acknowledged that having such configuration of levels in place with the ChatBot was indeed helpful to learners. However, several disadvantages identified by the interviewees with regard to this category of the ChatBot still remained to be improved, for instance, its current lack of coverage for grammar and, in particular, pronunciation of Mandarin Chinese, especially so with the latter, where much of their interest in mastering the language is at stake.

The last category was reserved for the recommendation formed by the interviewees and is of particular importance to the future development of the ChatBot. Table 14
Table 14.
Recommendations.

Recommendation Verbatim expression

Combine levels … honestly, when I am trying the first level and now I am in the third level, I forget what is in the first level. So when we are in level three, please include the first level also. (S1)

And also maybe will be good to implement, I don’t know if it is there already, maybe like practice section for all levels, so you have only one thing, and ask words for every level, or maybe can select only level 1 and 2, so can repeat the words but don’t have to go through each levels by self. (S5)

Add pronunciation … I hope that maybe in the future you can put this kind of feature in the ChatBot. So the ChatBot can check my pronunciation is correct or not. (S2)

The hardest part is how I say the words, so it could be good if you have the pronunciation. (S5)

Add more levels … I hope I can have more levels, instead only has 4 levels. (S2)

I think you are in the right way, just have to implement more levels maybe. (S5)

Gamification … I think you need to put the score in the practice section like a quiz. (S3)

Maybe you can make some game, like gamification like you give some challenges for the user. So the user is interested to try more. (S4)

Expand types of practice … then the second one is maybe there is another practice section. I mean not only question about word by word but put in a sentence. (S3)

summarizes the recommendations they made. First, it is suggested in the practice section that users should be allowed to select multiple levels to customize their own practice. For instance, they may choose to combine both levels of one and three on the ChatBot to try out questions containing both levels of vocabulary. Also, it was suggested that additional levels may be added to the current ones available. As discussed earlier in the disadvantages to be addressed by future ChatBots in terms of content, voice, or pronunciation of words were recommended to this category as well. Finally, a new scoring mechanism was also recommended by the interviewees to increase users’ level of interest in practice. It is suggested that the ChatBot should add points to the user’s score for choosing the correct answer and deduct points when he or she fails to do so.

A number of opinions and suggestions were provided by the users in the interviews. It can be inferred that they found the ChatBot easy to use and user-friendly and its interface as well as the instruction were clear enough for meaningful interaction. The incorporation of mobile instant messaging with LINE as a well-known and commonly used platform also served as an important leverage in the functioning of the ChatBot. In addition, the practice section was considered useful for students to remember Chinese vocabulary. However, the ChatBot required further improvements to allow users to customize their own practice among different levels, to experiment with a different scoring mechanism with users might to want to practice with the ChatBot more, and allow them to revisit questions they did not get right. Not only does the practice section need further improvement, but the currently missing coverage for grammar and pronunciation should be in place to make up for the content provided by the learning section. One other important element for future ChatBot is to add more levels to its current ones. Finally, it is worth mentioning that all of the participants in the present study expressed willingness to recommend the ChatBot to other learners.

Discussion and Conclusion

This study implemented the technology affordance of a ChatBot in the learning of Chinese vocabulary. The ChatBot commonly known as a virtual assistant was adopted as a substitute for the teacher or instructor. The research was conducted with two different groups, one experimental group and one control group each with a different learning environment. The experimental group was provided with a one-on-one tutoring environment while the control learned in a one-on-many classroom. The objective of this study was to explore the impact of the ChatBot on students’ learning achievement. As it was the first time such technology was deployed in the field of Learning Mandarin Chinese as a Second Language, technology acceptance was also a central topic to be explored among many others, including its PEU, PU, PE, and the BI among users. The users’ experience was also elicited in this study via semi-structured interviews, which was rather important for the present research to understand the feedback of the users and its relations to the future development of the ChatBot.

Both the experimental and the control group showed statistically significant improvement in learning achievement with the proposed technology. These findings imply that the ChatBot can be applied in different learning improvements to improve students’ performance in learning Chinese vocabulary. Such result was also supported by Mahdi (2018) in a previous study, where he conducted a meta-analysis which successfully proved the effectiveness of mobile devices in vocabulary learning. In addition, both the average pre- and posttest scores achieved in both groups across the three chapters instructed by the ChatBot in the present study not only indicated a significant difference between the effectiveness of the ChatBot in the classroom and in the tutoring sessions (the tutoring sessions yield a better score than the classroom), but also in the retention test. A previous study (Mostow et al., 2003) also found that one-on-one tutoring led to better learning outcomes among students in their reading skills than those of giving verbal instructions in a regular classroom. It was then assumed that those who received one-on-one tutoring prevailed due to the advantage they had in their setting where they were allowed to ask questions to the instructor directly in the advent of a problem. This then led to a reduction of the kind of teaching and learning process that were nonproductive (Juel, 1996). Bloom (1984) and Juel (1996) also argued that one-on-one teaching is the most effective method of instruction. It is believed that the guidance provided to the students in their tutoring sessions as well as the use of the ChatBot both increased the effectiveness of their learning process.

The ChatBot was designed to apply its PRS to rote learning. Nacera (2010) argued that memorizing words by repeating them again and again is far from effective because doing so only requires the use of a surface strategy. However, in Yang’s (2011) study, although recognizing the benefit of expanding one’s vocabulary by way of memorizing words during the early stage of learning a new language, it also stressed that such use of rote memorization alone is not nearly as important as opposed to the simultaneous incorporation of the other techniques as well. The ChatBot, in the present study, utilized an important feature to help its users review their newly acquired vocabulary. By generating multiple-choice questions with random candidate responses, it managed to test the retention of the user and reinforced his or her memory of the vocabulary with feedback in cases where incorrect responses were made. The PRS involved also provided instant feedback to help the user achieve better learning (Latham & Hill, 2014). Thus, the combined use of rote memorization and PRS by the ChatBot successfully helped the students improve their learning outcomes on Mandarin Chinese vocabulary through one-on-one tutoring.

Evaluating the technology acceptance of the ChatBot is important for further development of such application of language learning. Those who responded to the survey in this study not only have found the ChatBot easy to use, useful, and enjoyable but also said that they would continue using it in the future, which echoed with the interview responses, where the participants agreed that the ChatBot was easy to use. On the basis of the above findings, the present research also claims the PU among users as the predictor of their BI, their PEU the predictor of their PU, and their PE the predictor of their PEU. In addition, significant positive correlations were found between all of the variables, except for PEU, and the students’ learning achievement. Especially, the result associated with PU was consistent with that of Tsai’s (2014) study, which did not lead to the same conclusion with PEU. This may have to do with the fact that the study had centered mostly on writing while the tools in use required further improvements in various components surrounding their easiness to operate. PE was found to have a significant correlation with learning achievement. This finding is in line with previous research (Fokides & Atsikpasi, 2018), which indicated that the design of the ChatBot may cause its users to have positive learning experience and thus achieve better learning outcomes. This also means that the development of a future ChatBot should focus more on both its PU and PE. The BI among users, however, should also be incorporated. Successful attempts at exploring any of the above would help learners achieve better outcomes in terms of building up Chinese vocabulary.

Regardless of the advantages of the ChatBot in learning vocabulary of a second language, a number of disadvantages were identified by the student interviewees, whose feedback, the bulk of which concerned the content for learning, is important for the future development of the ChatBot. Some of their suggestions include the need for the future ChatBot to reach beyond the realm of vocabulary learning and explore other areas of language learning, such as grammar, pronunciation, and conversation. Another main suggestion was to implement gamified quizzes in the practice section of the ChatBot as once argued by Kozlov and Johansen (2010), who contended that game-like features promote both learning enjoyment and motivation. All of these recommendations should be considered for future studies. In addition, conversation among users should be recorded toward further improvement of their learning experience. It is concluded herein that ChatBots are potential technology to help students learn Mandarin Chinese vocabulary. The ChatBot designed in this study was able to aid students’ vocabulary learning. Student participants also perceived the usefulness, enjoyment, and ease of use while learning vocabulary with the ChatBot. However, limitations were identified with regard to the proposed ChatBot and should be considered for further improvement.

Question
1.	Please tell me three things that attract your attention when using the ChatBot.
2.	In your opinion, what are two things that are helpful when using the ChatBot and two things that are not helpful? Please give examples and reasons.
3.	According to your experience of learning Mandarin Chinese using the ChatBot, which parts do you like and dislike the most?
4.	How would you use the ChatBot in the future?
5.	What are your recommendations about the ChatBot for the future?
6.	Would you recommend ChatBot to others as a learning tool?

Group	Pretest	Posttest	n	95% CI for mean difference	t	df	d
A	71.63	15.19	88.03	12.35	39	[13.11–19.70]	10.09***	38	1.18
B	49.82	16.06	91.33	11.77	19	[35.45–47.57]	14.38***	18	2.95

Group	Pretest	Retention test	n	95% CI for mean difference	t	df	d
A	71.63	15.19	80.95	15.76	39	[7.02–11.62]	8.193***	38	0.60
B	49.82	16.06	93.21	8.28	19	[36.41–50.36]	13.08***	18	3.40

Source	Type III sum of squares	df	Mean square	F	Sig.	Partial eta squared
Corrected model	4,293.950^a	2	2,146.975	28.552	.000	.509
Intercept	10,592.802	1	10,592.802	140.871	.000	.719
Pretest	4,154.893	1	4,154.893	55.255	.000	.501
Group	2,095.599	1	2,095.599	27.869	.000*	.336
Error	4,135.728	55	75.195
Total	469,035.111	58
Corrected total	8,429.678	57

Source	Type III sum of squares	df	Mean square	F	Sig.	Partial eta squared
Corrected model	8,229.114^a	2	4,114.557	51.873	.000	.654
Intercept	6,999.628	1	6,999.628	88.246	.000	.616
Pretest	6,308.239	1	6,308.239	79.529	.000	.591
Group	6,514.154	1	6,514.154	82.125	.000*	.599
Error	4,362.595	55	79.320
Total	431,301.778	58
Corrected total	12,591.709	57

	n	M	SD	Min	Max
PEU	58	4.32	0.71	1	5
PU	58	4.19	0.67	2	5
PE	58	4.01	0.82	1	5
BI	58	4.08	0.75	1	5

	BI	PE	PEU	PU
BI1	0.948	0.763	0.582	0.725
BI2	0.948	0.777	0.629	0.712
PE1	0.646	0.825	0.734	0.707
PE2	0.718	0.878	0.627	0.694
PE3	0.666	0.852	0.672	0.682
PE4	0.737	0.795	0.519	0.695
PE5	0.639	0.852	0.534	0.616
PE6	0.708	0.845	0.533	0.674
PEU1	0.632	0.719	0.821	0.667
PEU2	0.247	0.349	0.666	0.323
PEU3	0.293	0.337	0.579	0.255
PEU4	0.308	0.416	0.770	0.400
PEU5	0.557	0.628	0.713	0.591
PEU6	0.531	0.523	0.635	0.634
PEU7	0.200	0.261	0.633	0.356
PU1	0.652	0.651	0.602	0.826
PU2	0.589	0.600	0.568	0.833
PU3	0.529	0.640	0.555	0.815
PU4	0.774	0.829	0.725	0.924

	BI	PE	PEU	PU
BI1	0.948	0.760	0.582	0.725
BI2	0.949	0.775	0.633	0.712
PE1	0.646	0.830	0.766	0.707
PE2	0.718	0.876	0.630	0.694
PE3	0.666	0.861	0.749	0.683
PE4	0.737	0.786	0.488	0.693
PE5	0.639	0.847	0.527	0.615
PE6	0.708	0.840	0.509	0.674
PEU1	0.632	0.723	0.862	0.668
PEU4	0.308	0.421	0.777	0.403
PEU5	0.557	0.635	0.822	0.592
PU1	0.652	0.648	0.505	0.818
PU2	0.589	0.603	0.600	0.839
PU3	0.529	0.640	0.518	0.812
PU4	0.774	0.829	0.727	0.926

	ICR	AVE	BI	PE	PEU	PU
BI	0.947	0.899	0.948
PE	0.935	0.706	0.810	0.840
PEU	0.861	0.674	0.641	0.751	0.821
PU	0.912	0.723	0.758	0.808	0.700	0.850

		PEU	PU	PE	LA
PEU	Pearson correlation	1	.668**	.678**	.228
	Sig. (2-tailed)		.000	.000	.086
	N		58	58	58
PU	Pearson correlation		1	.799**	.416**
	Sig. (2-tailed)			.000	.000
	N			58	58
PE	Pearson correlation			1	.423**
	Sig. (2-tailed)				.001
	N				58
LA	Pearson correlation				1
	Sig. (2-tailed)
	N

Technical aspect	Verbatim expression
Easy to use	Helpful because of no need for a lot of instruction to understand what should I do to connect with the bot. There is no instruction to use it but I can use it. (S1)
… easy to use, not so many menus appearing in the ChatBot that sometimes can make people confused. The interface of the ChatBot is simple and easy to interact with … (S2)
I think it is easy to use. (S4)
Well, it’s very easy to use … and how to operate it is very easy, just type “start” and we get everything. (S5)
I think it is easy to use … (S6)
Easy to access	… it is integrated to mobile phone, especially in LINE. I think a lot of people use LINE so it is easy to access. (S1)
… this kind of method is so practical. Because it uses LINE and LINE is an application that people use every day, almost everybody uses LINE, … The third one is accessible, at anywhere and anytime so anytime I want to use it, there it is, the ChatBot is in my cell phone, in my LINE, just use it. … I can use in my leisure time to learn Chinese. (S2)
The part I like the most is because it is easy to access, because it is using LINE. (S6)
Flexible	The first reason why that is useful because maybe for some of us, we have a lack of time, a lack of spare time. So, by using that ChatBot, it can help us to learn Mandarin. Whether we are in busy time or in our hectic time. (S4)
Real-time	I think because it is real-time, so if I answer the question now, I can get the answer now. (S4)

Learning process	Verbatim expression
Multiple-choice section (for practicing)	… when I am trying to practice, I can trial and error so when I am wrong, I can remember the right answer. (S1)
… I can test my knowledge or my memory using the practice section. So it can help me to train my brain, remember the word in random order, which is good for me. (S2)
The practice section, which is pretty good is repeat a lot of times. So, if I didn’t get it two or three times, after while I just keep going and learn at some point very fast. (S5)
… because it replies the correct answer in the practice section. (S6)
… I want the words that I answer incorrectly to appear more often. (S2)
… and I think something that you need to improve is that the ChatBot is repetitively giving the same question and answer. So I think for some of the users, I think they get bored with that kind of situation. (S4)
I think sometimes a little bit boring when I used it because the question is the same. (S6)
Vocabulary list (for learning)	I like the learning section of the ChatBot. Because … I will have a kind of sheet, full of the words, and I can try to memorize all of the words … (S2)
Learn section, because it makes me easy to remember. (S3)
… I dislike the learning section. … Because only words that I need to read … (S1)

Content	Verbatim expression
Differentiation of the levels	The ChatBot also provides 4 different levels we can select. This is really helpful in the learning process because I can start from the easy one and then do it until I get used to the words and then progress to the next level gradually. (S2)
Limited to vocabulary	I think because you have limited information. I am only understanding the vocabulary but I can't use it for the conversation. (S1)
… there is no pronunciation practice there. (S1)
… I cannot hear the pronunciation. (S2)
… I think they need grammar or maybe another thing of the lesson, not just vocabulary. (S4)
And what's not so good because at the moment it only has 4 levels, so there is not much. (S5)
… and the bad thing is because it is not focused on grammar. (S6)

Recommendation	Verbatim expression
Combine levels	… honestly, when I am trying the first level and now I am in the third level, I forget what is in the first level. So when we are in level three, please include the first level also. (S1)
And also maybe will be good to implement, I don’t know if it is there already, maybe like practice section for all levels, so you have only one thing, and ask words for every level, or maybe can select only level 1 and 2, so can repeat the words but don’t have to go through each levels by self. (S5)
Add pronunciation	… I hope that maybe in the future you can put this kind of feature in the ChatBot. So the ChatBot can check my pronunciation is correct or not. (S2)
The hardest part is how I say the words, so it could be good if you have the pronunciation. (S5)
Add more levels	… I hope I can have more levels, instead only has 4 levels. (S2)
I think you are in the right way, just have to implement more levels maybe. (S5)
Gamification	… I think you need to put the score in the practice section like a quiz. (S3)
Maybe you can make some game, like gamification like you give some challenges for the user. So the user is interested to try more. (S4)
Expand types of practice	… then the second one is maybe there is another practice section. I mean not only question about word by word but put in a sentence. (S3)

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was financially supported by a Grant from the Ministry of Science and Technology, Taiwan (MOST 108-2511-H-011-007-MY2).

ORCID iDs

Hsiu-Ling Chen

Hendri Sutrisno

Author Biographies

Hsiu-Ling Chen is an associate professor of Digital Learning and Education at National Taiwan University of Science and Technology. She has earned her PhD from the University of Texas at Austin. Her current main field is ICT integration in Education. Her other interests include flipped classroom, CSCL (computer supported collaborative learning), digital storytelling, and high-order cognitive thinking (critical thinking, creative thinking, problem-solving, and metacognition).

Hendri Sutrisno received his BS in Industrial Engineering and MS in Industrial Management from Petra Christian University, Indonesia, and National Taiwan University of Science and Technology, Taiwan, in 2016 and 2018, respectively. He is currently a PhD candidate in Industrial Management at National Taiwan University of Science and Technology, Taiwan. His main research interests are operations research, high-dimensional data analysis and optimization, metaheuristic algorithms, multivariate statistics, and applied machine learning.

Gracia Vicki Widarso received her BS in Industrial Engineering and MS in Digital Learning and Education from Petra Christian University, Indonesia, and National Taiwan University of Science and Technology, Taiwan, in 2016 and 2020, respectively. Her main research interests are data analysis in education, applied artificial intelligence for education, and natural language processing.

References

Bagozzi

(1988). On the evaluation of structure equation models. Journal of the Academy of Marketing Science, 16(1), 74–94. https://doi.org/10.1007/BF02723327

Barclay

Higgins

Thompson

(1995). The Partial Last Squares (PLS) approach to causal modelling, personal computer adoption and use as an illustration. Technology Studies, 2, 285–309.

Bloom

B. S.

(1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4–16. https://doi.org/10.3102/0013189X013006004

Chan

S. C. H.

Wan

C. L. J.

(2019). Interactivity, active collaborative learning, and learning performance: The moderating role of perceived fun by using personal response systems. The International Journal of Management Education, 17(1), 94–102. https://doi.org/10.1016/j.ijme.2018.12.004

Chen

Sivo

Seilhamer

Sugar

Mao

(2013). User acceptance of mobile technology: A campus-wide implementation of Blackboard’s Mobile^™ learn application. Journal of Educational Computing Research, 49(3), 327–343. https://doi.org/10.2190/EC.49.3.c

Chen, J. C., Whittinghill, D. C., & Kadlowec, J. A. (2010). Classes that click: fast, rich feedback to enhance student learning and satisfaction. Journal of Engineering Education, 99(2), 159–168.

Chen, Y.-F., & Dunsmoir, J. W. (2012). Chinese/English vocabulary learning tool (U.S. Patent No. US8137105B2). International Business Machines Corporation, Armonk, NY (US).

Chin

W. W.

(1998). The partial least squares approach to structural equation modeling. In Marcoulides

G. A.

(Ed.), Modern methods for business research, 295–336. Lawrence Erlbaum Associates.

Chiva-Bartoll

Moliner

M. L.

Salvador-García

(2020). Can service-learning promote social well-being in primary education students? A mixed method approach. Children and Youth Services Review, 111, 104841. https://doi.org/10.1016/j.childyouth.2020.104841

10.

Cortazzi

Jin

(1994). Changes in learning English vocabulary in China. In Coleman

Cameron

(Eds.), Changes and language (pp. 153–165). British Association for Applied Linguistics in association with Multilingual Matters.

11.

Creswell

J. W.

Clark

V. L. P.

(2007). Designing and conducting mixed method research. SAGE.

12.

Davis, F. (1985). A technology acceptance model for empirically testing new end-user information systems [Thesis]. Massachusetts Institute of Technology.

13.

Davis

F. D.

(1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008

14.

Davis

F. D.

Bagozzi

R. P.

Warshaw

P. R.

(1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35(8), 982–1003.

15.

Davis

F. D.

Bagozzi

R. P.

Warshaw

P. R.

(1992). Extrinsic and intrinsic motivation to use computers in the workplace. Journal of Applied Social Psychology, 22(14), 1111–1132. https://doi.org/10.1111/j.1559-1816.1992.tb00945.x

16.

Edge, D., Searle, E., Chiu, K., Zhao, J., & Landay, J. A. (2011). Micro Mandarin: Mobile learning in context. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 3169–3178. https://doi.org/10.1145/1978942.1979413

17.

Farahat

(2012). Applying the technology acceptance model to online learning in the Egyptian universities. Procedia—Social and Behavioral Sciences, 64, 95–104. https://doi.org/10.1016/j.sbspro.2012.11.012

18.

Fisher

C. W.

Filby

N. N.

Marliave

Cahen

L. S.

Dishaw

M. M.

Moore

Berliner

D. C.

(1978). Teaching behaviors, academic learning time, and student achievement: Final report of phase III-B, beginning teacher evaluation study. San Francisco.

19.

Fokides

Atsikpasi

(2018). Development of a model for explaining the learning outcomes when using 3D virtual environments in informal learning settings. Education and Information Technologies, 23(5), 2265–2287. https://doi.org/10.1007/s10639-018-9719-1

20.

Fornell

Larcker

D. F.

(1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50.

21.

Fryer

L. K.

Nakao

Thompson

(2019). Chatbot learning partners: Connecting learning experiences, interest and competence. Computers in Human Behavior, 93, 279–289. https://doi.org/10.1016/j.chb.2018.12.023

22.

Gettinger

(1983). Effects of learner ability and instructional modifications on time needed for learning and retention. The Journal of Educational Research, 76(6), 362–369. https://doi.org/10.1080/00220671.1983.10885484

23.

Gharaie, E. (2016). Sequential Exercises and Personal Response System in Project Management Courses. Procedia – Social and Behavioral Sciences, 228, 106–111. doi:https://doi.org/10.1016/j.sbspro.2016.07.016

24.

Glass

G. V.

Smith

M. L.

(1979). Meta-analysis of research on class size and achievement. Educational Evaluation and Policy Analysis, 1(1), 2–16. https://doi.org/10.2307/1164099

25.

Gopalan

Zulkifli

A. N.

Abubakar

(2016). A study of students motivation based on ease of use, engaging, enjoyment and fun using the augmented reality science textbook (Vol. 31). Revista de la Facultad de Ingeniería U.C.V

26.

Huckin

Haynes

Coady

(1993). Second language reading and vocabulary learning. Ablex Publishing Coorporation.

27.

Hussain

Mkpojiogu

Yusof

(2016). Perceived usefulness, perceived ease of use, and perceived enjoyment as drivers for the user acceptance of interactive mobile maps (Vol. 1761). AIP Conference Proceedings.

28.

Jia, J. (2004). The study of the application of a web-based Chatbot system on the teaching of foreign languages. [Paper presentation]. Society for Information Technology & Teacher Education International Conference, Atlanta, GA, USA, https://www.learntechlib.org/p/13633

29.

Jiménez

M. Á. F.

Rodríguez

E. M.

Vidal

L. I. E.

(2017). The tutor’s roles and functions in online education. Qualitative study within the context of worker training. Procedia—Social and Behavioral Sciences, 237, 196–202. https://doi.org/10.1016/j.sbspro.2017.02.063

30.

Juel

(1996). What makes literacy tutoring effective? Reading Research Quarterly, 31(3), 268–289.

31.

Latham

Hill

N. S.

(2014). Preference for anonymous classroom participation: Linking student characteristics and reactions to electronic response systems. Journal of Management Education, 38(2), 192–215.

32.

Liao

C.-H.

Tsou

C.-W.

Shu

Y.-C.

(2008). The roles of perceived enjoyment and price perception in determining acceptance of multimedia-on-demand. International Journal of Business and Information, 3(1), 27–52.

33.

Liu

S.-H.

Liao

H.-L.

Pratt

J. A.

(2009). Impact of media richness and flow on e-learning technology acceptance. Computers & Education, 52(3), 599–607. https://doi.org/10.1016/j.compedu.2008.11.002

34.

Legault

Zhao

Chi

Y.-A.

Chen

Klippel

(2019). Immersive virtual reality as an effective tool for second language vocabulary learning. Languages, 4(1), 13. https://doi.org/10.3390/languages4010013

35.

C.-H.

Chiou

G.-F.

Day

M.-Y.

Ong

C.-S.

Hsu

W.-L.

(2006). Using instant messaging to provide an intelligent learning environment. Intelligent Tutoring Systems, 4053, 575–583. https://doi.org/10.1007/11774303_57

36.

Kozlov

Johansen

(2010). Real behavior in virtual environments: Psychology experiments in a simple virtual-reality paradigm using video games. Cyberpsychology, Behavior and Social Networking, 13(6), 711–714. https://doi.org/10.1089/cyber.2009.0310

37.

Mahdi

H. S.

(2018). Effectiveness of mobile devices on vocabulary learning: A meta-analysis. Journal of Educational Computing Research, 56(1), 134–154. https://doi.org/10.1177/0735633117698826

38.

Malaquias

R. F.

Malaquias

F. F. O.

Hwang

(2018). Understanding technology acceptance features in learning through a serious game. Computers in Human Behavior, 87, 395–402. https://doi.org/10.1016/j.chb.2018.06.008

39.

Morton, Z. C. (2011). Constructivism or rote learning: promoting greater academic achievement. (Master of Education), LaGrange, LaGrange, Georgia.

40.

Marton, F., Alba, G. D., & Kun, T. L. (1996). Memorizing and understanding: the keys to the paradox? In D. A. Watkins & J. B. Biggs (Eds.), The Chinese learner: cultural, psychological, and contextual influences (pp. 69–83). Hong Kong & Melbourne: CERC & ACER.

41.

Moody

W. B.

Bausell

R. B.

Jenkins

J. R.

(1973). The effect of class size on the learning of mathematics: A parametric study with fourth-grade students. Journal for Research in Mathematics Education, 4(3), 170–176. https://doi.org/10.2307/748512

42.

Mostow

Aist

Burkhead

Corbett

Cuneo

Eitelman

Huang

Junker

Sklar

M. B.

Tobin

(2003). Evaluation of an automated reading tutor that listens: Comparison to human tutoring and classroom instruction. Journal of Educational Computing Research, 29(1), 61–117. https://doi.org/10.2190/06AX-QW99-EQ5G-RDCF

43.

Mutalib

A. H.

Kadir

Robani

Majid

(2014). Vocabulary learning strategies among Malaysian TEVT students in German-Malaysian Institute (GMI). Procedia—Social and Behavioral Sciences, 123, 361–368. https://doi.org/10.1016/j.sbspro.2014.01.1434

44.

Nacera

(2010). Languages learning strategies and the vocabulary size. Procedia—Social and Behavioral Sciences, 2(2), 4021–4025. https://doi.org/10.1016/j.sbspro.2010.03.634

45.

Naya

V. B.

Ibáñez

L. A. H.

(2014). Evaluating user experience in joint activities between schools and museums in virtual worlds. Universal Access in the Information Society, 14(3), 389–398. https://doi.org/10.1007/s10209-014-0367-y

46.

Nikou

S. A.

Economides

A. A.

(2017). Mobile-based assessment: Integrating acceptance and motivational factors into a combined model of Self-Determination Theory and Technology Acceptance. Computers in Human Behavior, 68, 83–95. https://doi.org/10.1016/j.chb.2016.11.020

47.

Park, R. E. (1937). A Memorandum on Rote Learning. American Journal of Sociology, 43(1), 23–36.

48.

Rodrigues

L. F.

Oliveira

Costa

C. J.

(2016). Does ease-of-use contributes to the perception of enjoyment? A case of gamification in e-banking. Computers in Human Behavior, 61, 114–126. https://doi.org/10.1016/j.chb.2016.03.015

49.

Tatai

Csordas

Kiss

Szalo

Laufer

(2003). Happy chatbot, happy user. Intelligent Virtual Agents, 2792, 5–12. https://doi.org/10.1007/978-3-540-39396-2_2

50.

Tsai

Y.-R.

(2014). Applying the Technology Acceptance Model (TAM) to explore the effects of a Course Management System (CMS)-assisted EFL writing instruction. CALICO Journal, 32(1), 153–171. https://doi.org/10.1558/calico.v32i1.25961

51.

Venkatesh

(2000). Determinants of perceived ease of use: Integrating control, intrinsic motivation, and emotion into the technology acceptance model. Information Systems Research, 11(4), 342–365.

52.

Warshaw

P. R.

Davis

F. D.

(1985). Disentangling behavioral intention and behavioral expectation. Journal of Experimental Social Psychology, 21(3), 213–228. https://doi.org/10.1016/0022-1031(85)90017-4

53.

Wentling

T. L.

(1973). Mastery versus nonmastery instruction with varying test item feedback treatments. Journal of Educational Psychology, 65(1), 50–58. https://doi.org/10.1037/h0034820

54.

Wilkins

D. A.

(1972). Linguistics in language teaching. Edward Arnold.

55.

Wold

(1982). Systems under indirect observation using PLS. In Fornell

(Ed.), A second generation of multivariate analysis, Volume 1: Methods (pp. 325–347). Praeger.

56.

Wong

K. K.-K.

(2013). Partial least squares structural equation modeling (PLS-SEM) techniques using SmartPLS. Marketing Bulletin, 24.

57.

Yang

Yin

(2018). Learning Chinese colloquialisms through mobile technology. Journal of Technology and Chinese Language Teaching, 9(1), 35–47.

58.

Yang

Dai

(2011). Rote memorization of vocabulary and vocabulary development. English Language Teaching, 4(4), 61–64. https://doi.org/10.5539/elt.v4n4p61

59.

M. Y.

Hwang

(2003). Predicting the use of web-based information systems: Self-efficacy, enjoyment, learning goal orientation, and the technology acceptance model. International Journal of Human-Computer Studies, 59(4), 431–449. https://doi.org/10.1016/S1071-5819(03)00114-9