Given the importance of conversation practice in language learning, chatbots, especially ChatGPT, have attracted considerable attention for their ability to converse with learners using natural language. This review contributes to the literature by examining the currently unclear overall effect of using chatbots on language learning performance and comprehensively identifying important study characteristics that affect the overall effectiveness. We meta-analyzed 70 effect sizes from 28 studies, using robust variance estimation. The effects were assessed based on 18 study characteristics about learners, chatbots, learning objectives, context, communication/interaction, and methodological and pedagogical designs. Results indicated that using chatbots produced a positive overall effect on language learning performance (g = 0.484), compared to nonchatbot conditions. Moreover, four characteristics (i.e., educational level, language level, interface design, and interaction capability) affected the overall effectiveness. In an in-depth discussion on how the 18 characteristics are related to the effectiveness, future implications for practice and research are presented.
AdamM.WesselM.BenlianA. (2021). AI-based chatbots in customer service and their effects on user compliance. Electronic Markets, 31(2), 427–445. https://doi.org/10.1007/s12525-020-00414-7
2.
*AhnS. (2022). The effects of chatbot on grammar competence for Korean EFL college students. Journal of Digital Convergence, 20(3), 53–61. https://doi.org/10.14400/JDC.2022.20.3.053
3.
AyedounE.HayashiY.SetaK. (2019). Adding communicative and affective strategies to an embodied conversational agent to enhance second language learners’ willingness to communicate. International Journal of Artificial Intelligence in Education, 29(1), 29–57. https://doi.org/10.1007/s40593-018-0171-6
4.
*BaoM. (2019). Can home use of speech-enabled artificial intelligence mitigate foreign language anxiety – Investigation of a concept. Arab World English Journal, (5), 28–40. https://doi.org/10.24093/awej/call5.3
5.
BaskaraF. R.MukartoF. X. (2023). Exploring the implications of ChatGPT for language learning in higher education. Indonesian Journal of English Language Teaching and Applied Linguistics, 7(2), 343–358.
6.
Belda-MedinaJ.Calvo-FerrerJ. R. (2022). Using chatbots as AI conversational partners in language learning. Applied Sciences, 12(8427), 1–16. https://doi.org/10.3390/app12178427
7.
BenottiL.MartínezM. C.SchapachnikF. (2014). Engaging high school students using chatbots. Proceedings of the 2014 Conference on Innovation & Technology in Computer Science Education - ITiCSE ’14, 63–68. https://doi.org/10.1145/2591708.2591728
8.
BibauwS.FrançoisT.DesmetP. (2015). Dialogue-based CALL: An overview of existing research.Critical CALL – Proceedings of the 2015 EUROCALL Conference, 57–64. https://doi.org/10.14705/rpnet.2015.000310
9.
BibauwS.Van Den NoortgateW.FrançoisT.DesmetP. (2022). Dialogue systems for language learning: A meta-analysis. Language Learning & Technology, 26(1), 1–24. https://hdl.handle.net/10125/73488
10.
BorensteinM.HedgesL. V.HigginsJ. P. T.RothsteinH. R. (2009). Introduction to meta-analysis. John Wiley & Sons.
11.
ChenH. L.Vicki WidarsoG.SutrisnoH. (2020). A chatbot for learning Chinese: Learning achievement and technology acceptance. Journal of Educational Computing Research, 58(6), 1161–1189. https://doi.org/10.1177/0735633120929622
12.
CheungA. C.SlavinR. E. (2012). Effective reading programs for Spanish-dominant English language learners (ELLs) in the elementary grades: A synthesis of research. Review of Educational Research, 82(4), 351–395. https://doi.org/10.3102/0034654312465472
13.
CheungA. C.SlavinR. E. (2016). How methodological features affect effect sizes in education. Educational Researcher, 45(5), 283–292. https://doi.org/10.3102/0013189x16656615
14.
CoburnK. M.VeveaJ. L. (2019). Weightr: Estimating weight-function models for publication bias (R package version 2.0.2) [Computer software]. https://CRAN.R-project.org/package=weightr
de BoerH.DonkerA. S.van der WerfM. P. (2014). Effects of the attributes of educational interventions on students’ academic performance: A meta-analysis. Review of Educational Research, 84(4), 509–545. https://doi.org/10.3102/0034654314540006
18.
*DizonG. (2020). Evaluating intelligent personal assistants for L2 listening and speaking development. Language Learning & Technology, 24(1), 16–26. https://doi.org/10125/44705
19.
DokukinaI.GumanovaJ. (2020). The rise of chatbots – New personal assistants in foreign language learning. Procedia Computer Science, 169, 542–546. https://doi.org/10.1016/j.procs.2020.02.212
20.
EngestromY. (1987). Learning by expanding: An activity-theoretical approach to developmental research. Prienta-Konsultit Oy.
21.
EpleyN.WaytzA.CacioppoJ. T. (2007). On seeing human: A three-factor theory of anthropomorphism. Psychological Review, 114(4), 864–886. https://doi.org/10.1037/0033-295x.114.4.864
FryerL. K.AinleyM.ThompsonA.GibsonA.SherlockZ. (2017). Stimulating and sustaining interest in a language course: An experimental comparison of chatbot and human task partners. Computers in Human Behavior, 75, 461–468. https://doi.org/10.1016/j.chb.2017.05.045
24.
FryerL. K.NakaoK.ThompsonA. (2019). Chatbot learning partners: Connecting learning experiences, interest and competence. Computers in Human Behavior, 93, 279–289. https://doi.org/10.1016/j.chb.2018.12.023
25.
GodaY.YamadaM.MatsukawaH.HataK.YasunamiS. (2014). Conversation with a chatbot before an online EFL group discussion and the effects on critical thinking. Journal of Information and Systems in Education, 13(1), 1–7. https://doi.org/10.12937/ejsise.13.1
26.
*HanD. E. (2020). The effects of voice-based AI chatbots on Korean EFL middle school students’ speaking competence and affective domains. Asia-Pacific Journal of Convergent Research Interchange, 6(7), 71–80. https://doi.org/10.47116/apjcri.2020.07.07
*HsuH. L.ChenH. H. J.ToddA. G. (2023). Investigating the impact of the Amazon Alexa on the development of L2 listening and speaking skills. Interactive Learning Environments, 31(9), 5732–5745. https://doi.org/10.1080/10494820.2021.2016864
29.
*HsuM. H.ChenP. S.YuC. S. (2023). Proposing a task-oriented chatbot system for EFL learners speaking practice. Interactive Learning Environments, 31(7), 4297–4308. https://doi.org/10.1080/10494820.2021.1960864
30.
HuangW.HewK. F.FryerL. K. (2022). Chatbots for language learning—Are they really useful? A systematic review of chatbot-supported language learning. Journal of Computer Assisted Learning, 38(1), 237–257. https://doi.org/10.1111/jcal.12610
31.
HuangX.ZouD.ChengG.ChenX.XieH. (2023). Trends, research issues and applications of artificial intelligence in language education. Educational Technology & Society, 26(1), 112–131. https://www-jstor-org-443.web.bisu.edu.cn/stable/48707971
32.
HultF. M. (2018). Foreign language education policy on the horizon. Foreign Language Annals, 51(1), 35–45. https://doi.org/10.1111/flan.12315
33.
HwangG. J.ChangC. Y. (2023). A review of opportunities and challenges of chatbots in education. Interactive Learning Environments, 31(7), 4099–4112. https://doi.org/10.1080/10494820.2021.1952615
34.
*HwangW. Y.GuoB. C.HoangA.ChangC. C.WuN. T. (2022). Facilitating authentic contextual EFL speaking and conversation with smart mechanisms and investigating its influence on learning achievements. Computer Assisted Language Learning, 1–27. https://doi.org/10.1080/09588221.2022.2095406
35.
JannachD.ChenL. (2022). Conversational recommendation: A grand AI challenge. AI Magazine, 43(2), 151–163. https://doi.org/10.1002/aaai.12059
36.
*JeonJ. (2023). Chatbot-assisted dynamic assessment (CA-DA) for L2 vocabulary learning and diagnosis. Computer Assisted Language Learning, 36(7), 1338–1364. https://doi.org/10.1080/09588221.2021.1987272
37.
JeonJ.LeeS. (2023). Large language models in education: A focus on the complementary relationship between human teachers and ChatGPT. Education and Information Technologies, 28, 15873–15892. https://doi.org/10.1007/s10639-023-11834-1
38.
JeonJ.LeeS.ChoiS. (2023). A systematic review of research on speech-recognition chatbots for language learning: Implications for future directions in the era of large language models. Interactive Learning Environments, 1–19. https://doi.org/10.1080/10494820.2023.2204343
39.
JiH.HanI.KoY. (2023). A systematic review of conversational AI in language education: Focusing on the collaboration with human teachers. Journal of Research on Technology in Education, 55(1), 48–63. https://doi.org/10.1080/15391523.2022.2142873
40.
*JiaJ.ChenY.DingZ.RuanM. (2012). Effects of a vocabulary acquisition and assessment system on students’ performance in a blended learning class for English subject. Computers & Education, 58(1), 63–76. https://doi.org/10.1016/j.compedu.2011.08.002
41.
JohnsonC.UrazovM.ZanoliE. (2022). Escapeling: A gamified, AI-supported chatbot for collaborative language practice [Symposium]. In GuralnickD.AuerM. E.PoceA. (Eds.), Innovations in learning and technology for the workplace and higher education. TLIC 2021. Lecture notes in networks and systems (Vol. 349, pp. 141–148). Springer. https://doi.org/10.1007/978-3-030-90677-1_14
42.
*KimN. Y. (2016). Effects of voice chat on EFL learners’ speaking ability according to proficiency levels. Multimedia-Assisted Language Learning, 19(4), 63–88. https://doi.org/10.15702/mall.2016.19.4.63
43.
*KimN. Y. (2018a). A study on chatbots for developing Korean college students’ English listening and reading skills. Journal of Digital Convergence, 16(8), 19–26. https://doi.org/10.14400/JDC.2018.16.8.019
44.
*KimN. Y. (2018b). Chatbots and Korean EFL students’ English vocabulary learning. Journal of Digital Convergence, 16(2), 1–7. https://doi.org/10.14400/JDC.2018.16.2.001
45.
*KimN. Y. (2019). A study on the use of artificial intelligence chatbots for improving English grammar skills. Journal of Digital Convergence, 17(8), 37–46. https://doi.org/10.14400/JDC.2019.17.8.037
KuhailM. A.AlturkiN.AlramlawiS.AlhejoriK. (2023). Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28, 973–1018. https://doi.org/10.1007/s10639-022-11177-3
48.
LawlessK. A.PellegrinoJ. W. (2007). Professional development in integrating technology into teaching and learning: Knowns, unknowns, and ways to pursue better questions and answers. Review of Educational Research, 77(4), 575–614. https://doi.org/10.3102/0034654307309921
LeeJ. Y.HwangY. (2022). A meta-analysis of the effects of using AI chatbot in Korean EFL education. Studies in English Language & Literature, 48, 213–243. https://doi.org/10.21559/aellk.2022.48.1.011
51.
LesterJ. C.ConverseS. A.KahlerS. E.BarlowS. T.StoneB. A.BhogalR. S. (1997). The persona effect: Affective impact of animated pedagogical agents. Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, 359–366. https://doi.org/10.1145/258549.258797
52.
LiakinD.CardosoW.LiakinaN. (2015). Learning L2 pronunciation with a mobile speech recognizer: French /y/. CALICO Journal, 32(1), 1–25. https://doi.org/10.1558/cj.v32i1.25962
53.
LiangJ. C.HwangG. J.ChenM. R. A.DarmawansahD. (2023). Roles and research foci of artificial intelligence in language education: An integrated bibliographic analysis and systematic review approach. Interactive Learning Environments, 31(7), 4270–4296. https://doi.org/10.1080/10494820.2021.1958348
LipseyM. W.WilsonD. B. (2001). Practical meta-analysis (2nd ed.). Sage.
56.
LiuC. C.LiaoM. G.ChangC. H.LinH. M. (2022). An analysis of children’ interaction with an AI chatbot and its impact on their interest in reading. Computers & Education, 189, 104576. https://doi.org/10.1016/j.compedu.2022.104576
57.
*LiuY.-F.HwangW.-Y.SuC.-H. (2023). Investigating the impact of context-awareness smart learning mechanism on EFL conversation learning. Interactive Learning Environments, 1–16. https://doi.org/10.1080/10494820.2023.2194931
*MageiraK.PittouD.PapasalourosA.KotisK.ZangogianniP.DaradoumisA. (2022). Educational AI chatbots for content and language integrated learning. Applied Sciences, 12(3239), 1–16. https://doi.org/10.3390/app12073239
60.
MokS. Y.RuppD.HolzbergerD. (2023). What kind of individual support activities in interventions foster pre-service and beginning teachers’ self-efficacy? A meta-analysis. Educational Research Review, 40, 100552. https://doi.org/10.1016/j.edurev.2023.100552
61.
*NghiT. T.PhucT. H.ThangN. T. (2019). Applying AI chatbot for teaching a foreign language: An empirical research. International Journal of Scientific and Technology Research, 8(12), 897–902.
62.
*NicolaidouI.PissasP.BoglouD. (2023). Comparing immersive virtual reality to mobile applications in foreign language learning in higher education: A quasi-experiment. Interactive Learning Environments, 31(4), 2001–2015. https://doi.org/10.1080/10494820.2020.1870504
63.
PageM. J.McKenzieJ. E.BossuytP. M.BoutronI.HoffmannT. C.MulrowC. D.ShamseerL.TetzlaffJ. M.MoherD. (2021). Updating guidance for reporting systematic reviews: Development of the PRISMA 2020 statement. Journal of Clinical Epidemiology, 134, 103–112. https://doi.org/10.1016/j.jclinepi.2021.02.003
64.
ParkM. H.KimJ. R. (2021). An analysis of high school students’ task interaction using AI chatbots. Journal of Learner-Centered Curriculum and Instruction, 21(9), 1–13. https://doi.org/10.1080/10447318.2024.2303557
65.
PetrovićJ.JovanovićM. (2021). The role of chatbots in foreign language learning: The present situation and the future outlook. Artificial Intelligence: Theory and Applications, 313–330. https://doi.org/10.1007/978-3-030-72711-6_17
66.
PicaT. (2013). From input, output and comprehension to negotiation, evidence, and attention. Contemporary Approaches to Second Language Acquisition, 9, 49–70. https://doi.org/10.1075/aals.9.06ch3
67.
PigottT. D.PolaninJ. R. (2020). Methodological guidance paper: High-quality meta-analysis in a systematic review. Review of Educational Research, 90(1), 24–46. https://doi.org/10.3102/0034654319877153
68.
PustejovskyJ. (2023). clubSandwich: Cluster-robust (sandwich) variance estimators with small-sample corrections (R package version 0.5.10) [Computer software]. https://CRAN.R-project.org/package=clubSandwich
69.
R Core Team. (2023). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/
70.
RodgersM. A.PustejovskyJ. E. (2021). Evaluating meta-analytic methods to detect selective reporting in the presence of dependent effect sizes. Psychological Methods, 26(2), 141–160. https://doi.org/10.1037/met0000300
71.
*RuanS.JiangL.XuQ.LiuZ.DavisG. M.BrunskillE.LandayJ. A. (2021). Englishbot: An AI-powered conversational system for second language learning.Proceedings of the 26th International Conference on Intelligent User Interfaces, 434–444. https://doi.org/10.1145/3397481.3450648
72.
SansonnetJ.-P.LerayD.MartinJ.-C. (2006). Architecture of a framework for generic assisting conversational agents [Symposium]. In GratchJ.YoungM.AylettR.BallinD.OlivierP. (Eds.), Intelligent virtual agents. Lecture notes in computer science, Springer, Berlin, Heidelberg (Vol. 4133, pp. 145–156). https://doi.org/10.1007/11821830_12
73.
SharplesM.TaylorJ.VavoulaG. (2007). A theory of learning for the mobile age. In AndrewsR.HathornthwaiteC. (Eds.), The SAGE handbook of e-learning research (pp. 211–247). Sage. https://doi.org/10.4135/9781848607859
74.
SinghS.ThakurH. K. (2020). Survey of various AI chatbots based on technology used.Proceedings of the 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), 1074–1079. https://doi.org/10.1109/icrito48877.2020.9197943
75.
SlavinR. E.LakeC.ChambersB.CheungA. C.DavisS. (2009). Effective reading programs for the elementary grades: A best-evidence synthesis. Review of Educational Research, 79(4), 1391–1466. https://doi.org/10.3102/0034654309341374
76.
SmutnyP.SchreiberovaP. (2020). Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Computers & Education, 151, 103862. https://doi.org/10.1016/j.compedu.2020.103862
77.
Soler CostaR.Mauri MedranoM.Lafarga OstárizP.Moreno-GuerreroA. J. (2020). How to teach pre-service teachers to make a didactic program? The collaborative learning associated with mobile devices. Sustainability, 12(9), 3755. https://doi.org/10.3390/su12093755
78.
*SongD.KimD. (2021). Effects of self-regulation scaffolding on online participation and learning outcomes. Journal of Research on Technology in Education, 53(3), 249–263. https://doi.org/10.1080/15391523.2020.1767525
79.
SungY. T.YangJ. M.LeeH. Y. (2017). The effects of mobile-computer-supported collaborative learning: Meta-analysis and critical synthesis. Review of Educational Research, 87(4), 768–805. https://doi.org/10.3102/0034654317704307
80.
*TaiT.-Y. (2022). Effects of intelligent personal assistants on EFL learners’ oral proficiency outside the classroom. Computer Assisted Language Learning, 1–30. https://doi.org/10.1080/09588221.2022.2075013
81.
*TaiT.-Y.ChenH. H.-J. (2022). The impact of intelligent personal assistants on adolescent EFL learners’ speaking proficiency. Computer Assisted Language Learning, 1–28. https://doi.org/10.1080/09588221.2022.2070219
82.
*TaiT.-Y.ChenH. H.-J. (2024). The impact of intelligent personal assistants on adolescent EFL learners’ listening comprehension. Computer Assisted Language Learning, 37(3), 433–460. https://doi.org/10.1080/09588221.2022.2040536
83.
TerrinN.SchmidC. H.LauJ. (2005). In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. Journal of Clinical Epidemiology, 58(9), 894–901. https://doi.org/10.1016/j.jclinepi.2005.01.006
84.
TiptonE. (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375–393. https://doi.org/10.1037/met0000011
85.
TliliA.ShehataB.AdarkwahM. A.BozkurtA.HickeyD. T.HuangR.AgyemangB. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(15), 1–24. https://doi.org/10.1186/s40561-023-00237-x
86.
TsvilodubP.ChevalierE.KlützV.OberbeckT.SigetovaK.WollatzF. (2022). Improving a gamified language learning chatbot through AI and UX boosting [Symposium]. In GuralnickD.AuerM. E.PoceA. (Eds.), Innovative approaches to technology-enhanced learning for the workplace and higher education, TLIC 2022. Lecture notes in networks and systems (Vol. 581, pp. 557–569). Springer. https://doi.org/10.1007/978-3-031-21569-8_52
87.
*Vázquez-CanoE.Mengual-AndrésS.López-MenesesE. (2021). Chatbot to improve learning punctuation in Spanish and to enhance open and flexible learning environments. International Journal of Educational Technology in Higher Education, 18(33), 1–20. https://doi.org/10.1186/s41239-021-00269-8
88.
VeveaJ. L.HedgesL. V. (1995). A general linear model for estimating effect size in the presence of publication bias. Psychometrika, 60(3), 419–435. https://doi.org/10.1007/BF02294384
89.
ViechtbauerW. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03
90.
VygotskyL. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.
91.
*WambsganssT.KuengT.SoellnerM.LeimeisterJ. M. (2021). ArgueTutor: An adaptive dialog-based learning system for argumentation skills. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–13. https://doi.org/10.1145/3411764.3445781
92.
WangF.CheungA. C.NeitzelA. J.ChaiC. S. (2024). Does chatting with chatbots improve language learning performance? A meta-analysis of chatbot-assisted language learning [Data set]. Johns Hopkins Research Data Repository. https://doi.org/10.7281/T1/XOL4BR
93.
WeizenbaumJ. (1966). Eliza—A computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1), 36–45. https://doi.org/10.1145/365153.365168
94.
*XuY.AubeleJ.VigilV.BustamanteA. S.KimY. S.WarschauerM. (2022). Dialogue with a conversational agent promotes children’s story comprehension via enhancing engagement. Child Development, 93(2), e149–e167. https://doi.org/10.1111/cdev.13708
95.
*XuY.WangD.CollinsP.LeeH.WarschauerM. (2021). Same benefits, different communication patterns: Comparing children’s reading with a conversational agent vs. a human partner. Computers & Education, 161, 104059. https://doi.org/10.1016/j.compedu.2020.104059
96.
XuZ.WijekumarK.RamirezG.HuX.IreyR. (2019). The effectiveness of intelligent tutoring systems on K-12 students’ reading comprehension: A meta-analysis. British Journal of Educational Technology, 50(6), 3119–3137. https://doi.org/10.1111/bjet.12758
97.
YangH.KimH.LeeJ. H.ShinD. (2022). Implementation of an AI chatbot as an English conversation partner in EFL speaking classes. ReCALL, 34(3), 327–343. https://doi.org/10.1017/s0958344022000039
98.
*YeY.DengJ.LiuX. (2021). The effect of oral practice via chatbot on students’ oral English accuracy [Symposium]. In JiaW., et al. (Eds.), Emerging technologies for education. SETE 2021. Lecture notes in computer science (Vol. 13089, pp. 344–354). Springer. https://doi.org/10.1007/978-3-030-92836-0_30
99.
ZhangS.ShanC.LeeJ. S.CheS.KimJ. H. (2023). Effect of chatbot-assisted language learning: A meta-analysis. Education and Information Technologies, 34, 15223–15243. https://doi.org/10.1007/s10639-023-11805-6
100.
ZuritaG.NussbaumM. (2007). A conceptual framework based on activity theory for mobile CSCL. British Journal of Educational Technology, 38(2), 211–235. https://doi.org/10.1111/j.1467-8535.2006.00580.x
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
