Unpacking the Rejection of L2 Students Toward ChatGPT-Generated Feedback: An Explanatory Research

Technological limitations

Technology itself is reported as the challenge and barrier to technology integration with classroom instruction and self-directed learning activities. It includes (1) incapability to offer answers and responses in a reliable and expected way (Kuhail et al., 2023; Winkler et al., 2020), (2) unavailability of emotional support to the users (Escalante et al., 2023; Gallacher et al., 2018; Goda et al., 2014), and (3) limitation in engaging and maintaining conversations (Yang & Zapata-Rivera, 2010).

In relation to the first point, chatbots may not be able to accurately respond to users’ inquiries due to the limitations or insufficiency of training datasets in the early stage of the development of the AI systems. For example, Winkler et al. (2020) designed a conversational bot named Sara aimed to solve the challenge of creating meaningful interactions in online video lectures. They evaluated the bot by collecting self-reflection reports from 182 participants who were undergraduate and graduate students. The findings indicated that bots with limited datasets posed challenges to learning, including the inability to provide correct answers and respond to questions. These limitations led to learners’ frustrations and thus possibly influenced their learning process.

Furthermore, chatbots lack human-like emotional support, which can be linked to unnatural and ambiguous voices and the use of non-academic language. According to Goda et al.'s (2014) study, students have reported feeling that computer-generated voices are unnatural compared to human voices, ultimately limiting their engagement with the immersive learning environment. Moreover, Gallacher et al. (2018) noted that the absence of emotional and visible cues during interactions with chatbots undermined students’ positive state of language learning. While Escalante et al. (2023) collected 48 undergraduate students’ perceptions of feedback provided by teachers and feedback given by generative AI, the result revealed that half preferred feedback provided by teachers rather than AI-generated feedback. The primary reason cited for this preference was the positive emotional impact of in-person communication.

Lastly, the limitation of chatbots in engaging in extended conversations is another drawback. Research by Yang and Zapata-Rivera (2010) revealed that students’ ideas diverge in unforeseen directions as communication progresses, and chatbot systems fail to recognize and address these constantly shifting topics.

The challenges and barriers to teachers and learners

Regarding chatbot limitations in education, a subset of research has primarily focused on teachers and learners. Four areas were identified from the perspective of teachers: (1) teachers’ negative beliefs and attitudes toward the limited use of technology in teaching (Howard, 2013; Shin et al., 2014), (2) teachers’ discomfort or lack of confidence in working with technology, which can lead to the rejection of new technology (Leem & Sung, 2019), (3) teachers’ low level of computer literacy can make it difficult for them to operate chatbot technology (Su et al., 2023), and (4) limited access to resources in schools can also pose a challenge that can hinder the implementation of AI tools (Reiser, 2002; Shin et al., 2014).

More importantly, from the learners’ perspective, several factors may be relevant to the limitations of implementing new technologies in teaching and learning: (1) students’ low perceived usefulness can lead to negative behaviors during their interactions with the technology (Chen et al., 2020; Kaur et al., 2021; Marchante, 2022), (2) students’ limited competence and computer literacy may hinder their effective use of chatbot technology (Celik, 2023; Hatlevik et al., 2018; Fryer et al., 2020), and (3) the disconnect between students’ external requirements and their interactions with technology has resulted in minimal use of chatbots in their learning process (Ericsson et al., 2023; Guo et al., 2023; Hew et al., 2023).

To be more specific, students’ attitudes and beliefs toward the use of technology in supporting their learning process refer to their technological acceptance. This encompasses their willingness to experiment with new technology, perceived usefulness, and overall attitudes toward technology as an educational tool. For instance, Chen et al. (2020) found that students’ beliefs about chatbot usefulness in Chinese language learning significantly impacted their acceptance and utilization of the technology. Moreover, an insightful study by Hatlevik et al. (2018) examined the connection between technology and students’ self-efficacy in nearly 60,000 individuals from 15 countries. The results revealed that familiarity with technology significantly influenced students’ information and communication technology self-efficacy. Fryer et al.'s (2020) study revealed that learners are unlikely to develop an interest in oral tasks during language classes without sufficient training in feedback-seeking strategies and computer literacy. As a result, human-supported oral training sessions achieved better outcomes, with higher levels of motivation and engagement. Furthermore, the successful implementation of chatbots in educational environments depends on the alignment between students’ learning interest in AI and assignment requirements established by educators (Hew et al., 2023).

Framework and research questions

Currently, no existing research utilizes a theoretical framework to systematically analyze the challenges associated with chatbot-assisted writing and corrective feedback practices. This study aims to fill that gap by utilizing the Unified Theory of Acceptance and Use of Technology (UTAUT), first introduced by Venkatesh et al. (2003). Upon examining the breadth of research on Generative AI within educational settings, we found that the UTAUT model elucidates multiple barriers to the optimal and efficient deployment of these AI systems. The model identifies four principal areas impacting effectiveness and integration, which have already been widely discussed separately in the field of AI-assisted learning and teaching. Firstly, there is “Performance Expectancy” related to the extent to which AI can visualize and be assisted in achieving learning outcomes. For instance, previous studies have shown that AI is a reliable tool for scoring and peer review (Jiang et al., 2023; Kaebnick et al., 2023). Secondly, “Effort Expectancy” considers the ease with which these tools can be used to facilitate the learning process, and sometimes even in an unethical manner, such as plagiarism and academic integrity issues (Cotton et al., 2023). Thirdly, “Social Influence” is tied to the endorsement and contempt from other stakeholders in the educational community. McDonald et al. (2024) reviewed the policies of 116 US higher education institutions on the use of Generative AI, and most of the policy concerns were issues of ethical practice of AI, with most references to its application remaining ambiguous. Lastly, “Facilitating Conditions” refers to the infrastructural and resource-related aspects that enable AI utilization. As indicated in prior research (Reiser, 2002; Shin et al., 2014), constraints related to accessing necessary resources in academic settings can greatly hinder the successful implementation and effective use of AI technologies. In our study, the four core factors are “Mismatch with learners’ expectations,” “Anticipated high workload to respond,” “Mismatch with external references,” and “Impending conditions.” We have provided definitions for each of these core factors in Table 2.

These potential challenges can be suggested by observing students’ revision strategies to deal with AI-generated feedback. As for the framework for coding students’ revision strategies, two categories are identified from previous corrective feedback studies based on the rate of uptake or acceptance of received feedback (Cheong et al., 2023): (1) accept and revise accordingly and (2) show a certain level of rejection, such as ignorance, disagreement, and confusion by arguing with the feedback providers, especially for non-error corrective feedback (Mohebbi, 2021; Yang et al., 2023; Zhang, 2020). Moreover, we used two established frameworks, called “levels of revisions” and “taxonomy of revision strategies,” to label the dependent variable. These frameworks were informed by prior research on corrective feedback and automatic written feedback. Fan and Xu's (2020) research shows two types of feedback: form-focused and content-focused. Form-focused feedback, typically given by teachers or peers, targets errors in language usage, such as sentence structures, vocabulary, and verb conjugations. Interestingly, Fan and Xu (2020) found that content-focused feedback was primarily delivered through negotiation. This means that feedback providers prefer to engage in oral discussions and debates with receivers on topics related to evidence and examples.

Without any doubt, partnerships with Generative AI technology in language teaching and learning will be one of the trends. The use of AI technologies in language teaching and learning has gained significant attention in recent years (Hinostroza et al., 2015; Lai, 2015; Morales & Roig, 2002). Although the digital affordance of these AI-driven learning systems has been widely investigated (Fitria, 2021; Ghufron & Rosyida, 2018), the challenges of using Generative AI technologies have not been adequately examined. Open AI released a powerful generative AI technology called ChatGPT in late November 2022. To date, most of the research related to ChatGPT has mainly concentrated on academic writing (Rozencwajg & Kantor, 2023; Salvagno et al., 2023; Zimmerman, 2023). However, an in-depth exploration of ChatGPT's distinctive influence on enhancing student engagement and their responses to formative corrective feedback is notably limited.

In that case, this research aims to incorporate ChatGPT technology into the language learning feedback process. Guided by the UTAUT, the study aims to identify the key factors that lead to negative responses to ChatGPT feedback. Consequently, the study is guided by two primary questions:

What are the challenges faced by L2 learners when engaging with ChatGPT-generated feedback?

Participants

Undergraduate students from a Computer Science program at a public university in Macao, aged between 20 and 21 (SD = 1.72), were invited to participate in an academic writing course. The sample consisted of 52 male and seven female students who had all completed the first-semester courses and understood computer operational systems, AI technologies, and English academic writing well. The language of instruction in the classroom was English, so the participants felt comfortable communicating with the AI system in English. Among the sample, 55 were native Cantonese or Mandarin speakers, and the remaining four were Portuguese, English, and Filipino native speakers. Most participants were at the B1 or B2 level at the CEFR band scale (Common European Framework of Reference for Languages), based on their scores in the English language test of the university entrance examination. At the time of data collection, most participants had already used ChatGPT, the chatbot, for academic and entertainment purposes since its official launch at the end of November 2022. These students were first contacted to participate in this study via Canvas, a centralized online learning system. Forty-five students expressed interest in the study, and written consent forms were obtained from each of them. Data protection strategies included removing all personal identifiers and replacing them with ID numbers.

Data collection

After that, these students were briefed about the procedure of data collection. First, they were invited to draft a 250-word essay individually without the assistance of ChatGPT in a classroom condition. The writing task is “To what extent do you agree or disagree that AI technologies may replace human working forces?”. This task was selected for several reasons: (1) this topic has been discussed in one of their compulsory subjects in their first semester, Ethics and Professional Issues, and (2) they were familiar with relevant ideas, scientific reports, and linguistics resources. Ethics approval was obtained by the organization of the corresponding author. The data collection for this study is an integral part of our routine work. No additional academic work is placed on participants specifically for this study.

Secondly, they were trained to use ChatGPT to offer corrective feedback on their draft and revise their essay in 2-week advance. The training includes (1) the academic integrity issues in the use of ChatGPT to complete academic assignments, (2) the university guidelines on the use of ChatGPT for academic purposes, and (3) the guidelines and examples of writing reflective journals to report their feedback experiences with ChatGPT.

The participants in the study were asked to interact with ChatGPT in two ways: (1) by providing corrective feedback on their essay, which they attached, and (2) by making suggestions to improve their essay based on the attached marking rubrics. These rubrics were designed to evaluate both the form-focused aspects of writing (grammatical accuracy and lexical sophistication) as well as the content-focused aspects (quality of argument structures and use of source evidence) (Wei et al., 2022). Participants were encouraged to ask ChatGPT follow-up questions about its feedback, including asking for clarifications, providing more examples, or even disputing the feedback if they disagreed. Finally, they were required to submit their responses to ChatGPT's feedback item by item in either their first language or English. The collected data included the participants’ first draft of their essays, the corrective feedback and suggestions from ChatGPT, and the learners’ responses and revisions, as shown in Table 1.

OPEN IN VIEWER

Table 1.

Example of L2 writers’ response on ChatGPT generated feedback.

Feedback from ChatGPT	Your response
1. Use proper grammar and punctuation—here are several grammatical errors in the essay, including improper use of commas and run-on sentences. Make sure to proofread your work before submitting it.	I agreed that, and I provide made revision accordingly. I change “get in touch with people” to “easily connect with people”, “had convenient our daily lives” to “has made our daily lives more convenient”, “book for position” to “make reservations”.
2. Use appropriate language—Choose your words carefully and use appropriate language for your audience and purpose. Avoid using overly technical terms or jargon that your readers may not understand. Also, make sure to vary your sentence structure to keep your writing interesting.	I don’t agree that, think my words are all very easy. I don’t use any jargon in my essay. The evidence I chosen were closed to our lives.
3. Clarify and elaborate on your points—Some of your arguments could be more specific and detailed. For example, you could provide examples of jobs that have been replaced by machines or describe how addiction to virtual technology affects young people's lives.	I agree with that, but I can’t do so. Because that will over the word count.

Data coding

To answer the first research question, the UTAUT model was adapted and developed to identify and code possible barriers to students’ negative responses. The first author and corresponding author jointly coded the responses of the first 15 participants and developed the framework collaboratively. After resolving all disagreements, they coded the rest of the responses from the remaining 30 participants with an agreement rate of 87%. The specific explanation of coding factors is presented in Table 2.

OPEN IN VIEWER

Table 2.

Coding framework on the students’ negative responses to ChatGPT-generated feedback.

Categories	Explanations
Mismatch with learners’ expectations	ChatGPT provided unhelpful feedback, such as (1) misinterpreting authors’ intentions and ideas, (2) lack of clarity and examples, and (3) extraneous feedback, which made students believe using this technology would not help them attain work/study performance gains.
Anticipated high workload to respond	ChatGPT provided inconvenient feedback, such as (1) information overload in feedback and (2) a general description rather than highlighting specific mistakes, which decreased students’ degree of ease associated with using the particular technology in their work.
Mismatch with external references	ChatGPT gave feedback that did not align with the expectations or preferences of other important people (e.g., peers and teachers), making students refuse to take the AI feedback.
Impeding conditions	ChatGPT did not provide personalized suggestions based on proficiency levels and lacked emotional support, causing discomfort and unexpected behavior in students.

To answer the second research question, an Ordinal Logistic Regression model (OLR) was developed to estimate the prediction power of the identified barriers to learners’ rejection of the AI-generated feedback. To label the dependent variable, the coding process was guided by two established frameworks: levels of revisions and the taxonomy of revision strategies, both informed by prior research on corrective feedback and automatic written feedback. This study considered four critical, independent variables in this model, including “Mismatch with learners’ expectations,” “Anticipated high workload to respond,” “Mismatch with external references,” and “Impeding conditions” to predict users’ intentions to engage with ChatGPT-generated feedback. The OLR model can create an analysis model that treats the dependent variable as an ordered categorical variable based on the cumulative odds principle, similar to a conventional multiple regression approach (Hosmer et al., 2013).

Qualitative findings

To answer the first research question, the research conducted a thorough analysis of the feedback provided by L2 learners to understand their rejection of AI-generated feedback better. We carefully examined the transcript, identified relevant themes that could serve as suitable study questions, and answered the first research question. By analyzing 167 L2 students’ rejections of generative AI feedback, we found that students rejected the feedback more often because of “Mismatch with learners’ expectations” (37.1%) and “Anticipated high workload to respond” (34.1%) than “Mismatch with external references” (15.0%) and “Impeding conditions” (13.8%). Specifically, the findings have been summarized in Table 3, which provides an overview of the barriers encountered by L2 learners when attempting to act upon ChatGPT-generated feedback guided by the UTAUT model.

OPEN IN VIEWER

Table 3.

The reasons behind students’ rejection of AI feedback guided by the UTAUT model.

Factors of the UTAUT model	Sub-category
Mismatch with learners’ expectations	misinterpreting authors’ intentions and ideas (31)
	lack of clarity and examples (21)
	extraneous feedback (10)
Anticipated high workload to respond	a general description rather than highlighting specific mistakes (33)
	information overload (24)
Mismatch with external references	not aligned with other teachers’ and peers’ expectations or preferences (25)
Impeding conditions	lack of personalized suggestions based on proficiency levels (18)
	lack of emotional support (5)

The “Mismatch with learners’ expectations” aspect of feedback refers to the feedback provided by ChatGPT that led students to think that using this technology would not help them improve their work or study performance. First, ChatGPT may not fully understand the nuances and subtleties of the learners’ writing. As a result, it may misinterpret the authors’ intentions and make revisions that deviate from the original meaning. For example:

I think ChatGPT's comment is wrong because I just want to speak from a neutral point of view to show some good and bad things coming out. Therefore, I will not make any changes on this point.

Nevertheless, the learners pointed out that the suggestions lacked clarity and specific examples, which in turn made it challenging for them to identify and address their particular areas of improvement, as one student stated:

I do not understand what “changed the formatting” means; therefore, I cannot make revisions.

In ChatGPT, extraneous feedback refers to the frustrating experience of L2 learners when they receive irrelevant corrections from AI models meant to assist them. Extraneous feedback frustrates L2 learners who use AI language models for assistance, wasting time and confusion. For example:

I think ChatGPT's comment is wrong because my first sentence of the paragraph is, “On the other hand, technology also has some disadvantages.” And ChatGPT gave me the same sentence.

The “Anticipated high workload to respond” challenges in AI feedback decreased students’ degree of ease in using the particular technology in their work. For instance, overload-specific suggestions given by ChatGPT are often disregarded and rejected by the learners. The abundance of tips can overwhelm learners and cause them to overlook valuable insights or fail to allocate their attention appropriately. For example:

Its summary is too long to be suitable for its summary. It will be better if it gives me a short one.

Furthermore, ChatGPT provided a generic explanation of an error, resulting in L2 learners failing to make the necessary revisions and seeking clarification on the specific occurrences of these errors. This challenge may hinder the learners’ ability to recognize and rectify their linguistic shortcomings by not highlighting the exact locations of errors. For example:

ChatGPT did not give me the exact sentences, like which sentence is unclear; it just told me to revise all the paragraphs, so I will not make any changes on this point.

The “Mismatch with external references” challenge in AI feedback refers to ChatGPT providing feedback that does not align with the expectations or preferences of essential people, such as peers and teachers, leading students to reject the AI feedback from ChatGPT despite its accuracy and helpfulness. Additionally, students may prioritize the feedback of individuals they trust over AI feedback, as one student mentioned:

I think ChatGPT is wrong and right. More examples can support a convincing essay. Nevertheless, as I write above, I should follow the requirements of the 3 + 7 + 7 + 3. So, for all the advantages and disadvantages, I can only add one example to each of them.

Our research identified challenges concerning “Impeding conditions” related to students’ reluctance to use AI feedback in their essays. Specifically, ChatGPT could not provide personalized suggestions based on proficiency levels and lacked emotional support, leading to discomfort and negative behavior in some students. This challenge might have made the learning process tedious and demotivating for students, as some students stated:

I agree with that, but I cannot do much to avoid it because of my weak English skills.

Quantitative findings

In order to answer the second research question, descriptive statistical analysis was used to demonstrate the main challenges of the interaction between students and AI. Although the AI chatbot provides more content-focused feedback (58.8%) than form-focused feedback (41.2%), the participants rejected more content than form-focused feedback. It suggests that they are less prepared to engage with content-focused than form-focused feedback. A comparison of rejected AI feedback between form-focused and content-focused revealed a similar pattern, with “Mismatch with learners’ expectations” and “Anticipated high workload to respond” being rated as the top two reasons for rejection. As shown in Table 4, “Mismatch with external references” is mentioned less frequently to reject form-focused AI feedback, while “Impeding conditions” is reported less frequently to reject content-focused feedback.

OPEN IN VIEWER

Table 4.

Descriptive data and proportion of the students’ rejection.

Factors of UTAUT	Form—Rejection		Content—Rejection
Mismatch with learners’ expectations	25	36.2%	37	37.8%
Anticipated high workload to respond	24	34.8%	33	33.7%
Impeding conditions	16	23.2%	7	7.1%
Mismatch with external references	4	5.8%	21	21.4%
Sum of rejection	69		98

Ordinal logistic regression was conducted to determine which factors were statistically significant in predicting learners’ rejection of AI feedback after controlling pre-experiment academic writing proficiency. Regardless of the focus of the feedback, the chi-square test shows that the assumption of parallelism tested was fulfilled, including Form-focused (χ² = .000, p > .05) and Content-focused (χ² = .000, p > .05) feedback. As shown in Table 5, the model fitness (χ² = 129.912, p < .05) showed a relationship between Form-focused feedback rejected by students and the challenges from the interaction. An increase in the “Anticipated high workload to respond” feedback results in a faster decline in the overall number of students rejecting feedback, which is the only factor statistically significant (Wald χ²(1) = 2.432, p < .05). In other words, students are more likely to reject form-focused feedback (such as formatting and grammatical errors) when they are cognitively overloaded by the overwhelming amount of feedback.

OPEN IN VIEWER

Table 5.

Reasons for rejecting AI-generated form-focused feedback.

Effect	Estimate	SE	95% CI		p
			LL	UL
Mismatch with learners’ expectations	−33.339	21.387	−75.239	8.562	.119
Anticipated high workload to respond	−33.332	13.623	−60.032	−6.632	.014
Mismatch with external references	−11.137	7.336	−25.516	3.241	.129
Impeding conditions	−22.414	11.948	−45.832	1.003	.061
Model fit (−2 log-likelihood)					.000

As for the Content-focused AI feedback, the model fitness (χ² = 151.836, p < .05) showed a relationship between Content-focused feedback rejected by students and the challenges from the interaction. Notably, all the main factors in the model had statistical significance, as demonstrated in Table 6. The feedback provided by AI had a greater impact on students’ rejection in terms of “Mismatch with learners’ expectations” (Wald χ²(1) = 13.677, p < .000), “Anticipated high workload to respond” (Wald χ²(1) = 18.253, p < .000) and “Mismatch with external references” (Wald χ²(1) = 14.541, p < .000), when compared to “Impeding conditions” (Wald χ²(1) = 6.877, p < .000). In summary, it may be interpreted from four perspectives. Firstly, AI-generated content-focused feedback may not align with participants’ expectations or be perceived as useful, creating a disconnect between the AI's output and the student's learning objectives. Secondly, the level of effort required to engage with AI feedback can be significantly higher than what students are able to accommodate, leading to cognitive overload and disengagement. Thirdly, AI-generated feedback may not correspond to or support the feedback provided by teachers and peers, causing confusion and inconsistency in the learning process. Lastly, students may lack the necessary skills to comprehend and respond to AI-generated content-focused feedback, further exacerbating the issue.

OPEN IN VIEWER

Table 6.

Reasons for rejecting AI-generated content-focused feedback.

Effect	Estimate	SE	95% CI		p
			LL	UL
Mismatch with learners’ expectations	−32.096	8.679	−49.106	−15.086	.000
Anticipated high workload to respond	−23.059	5.397	−33.638	−12.481	.000
Mismatch with external references	−25.894	6.790	−39.203	−12.585	.000
Impeding conditions	−7.433	2.834	−12.988	−1.878	.009
Model fit (−2 log-likelihood)					.000