Building Critical AI Literacy in the Business Communication Classroom

Abstract

The public release of ChatGPT in 2022 ushered in a new era, affirming the present reality of AI-assisted writing and the critical role business instructors play in preparing students. This study presents the results of a pedagogical experiment. Specifically, it evaluates strategies for integrating and teaching about AI in the business communication classroom, focusing on the impact of generative AI on students’ understanding of business writing principles and how different levels of engagement with AI influence students’ critical AI literacy and attitudes toward AI-assisted writing in the workplace.

Keywords

bad-news messages business communication instruction generative AI AI literacy

The public release of ChatGPT by OpenAI in November 2022 has led many scholars and experts to conclude that AI-assisted writing is a reality for the here and now. ChatGPT, a Large Language Model (LLM) trained on enormous amounts of data, makes the recent progress in natural language processing available to anyone with internet access and has captured the attention of the public with its capability to “generate human-like text, answer questions, and complete other language-related tasks with high accuracy” (Kasneci et al., 2023). The rapid, widespread implementation of AI technologies such as ChatGPT is changing the business communication landscape, offering many benefits but also presenting challenges and ethical dilemmas (Getchell et al., 2022; Hancock et al., 2020).

As these tools become increasingly entrenched in workplaces and more developed in their abilities, business communication instructors need to consider how these new technologies influence the practice and instruction of business communication. Moreover, investigations into the perspectives of university students regarding genAI technologies like ChatGPT reveal that while students generally hold a positive view of GenAI’s potential benefits, they also harbor apprehensions. Concerns regarding accuracy, privacy, ethical implications, as well as potential impacts on personal development, career opportunities, and societal values, have been voiced in recent studies (Chan & Hu, 2023; Kelly et al., 2023).

Specifically in business communication, Getchell et al. (2022) call for instructors to “develop teaching practices to improve students’ ethical proficiency alongside technical proficiency. Students must learn to evaluate implications for use of AI tools in the workplace through methods such as case studies, in-classroom discussion, and other reflective exposure” (22). This call underscores the critical need to respond proactively through pedagogy to cultivate a generation of professionals who are both skilled workplace communicators and critical users of AI applications.

Seeking to address the imperative set forth by Getchell et al. (2022), this article shares the results from a pedagogical experiment to evaluate strategies for integrating and teaching about AI in the business communication classroom. In particular, we set out to better understand (a) how generative AI (genAI) impacts students’ understanding of business writing principles and (b) how different levels of engagement with genAI impacts students’ critical AI literacy and attitudes toward AI-assisted writing in the workplace. In this article, we formulate the type of connections we saw in our data between students’ understanding of and attitudes toward AI and their assessment and application of business writing principles. As we explore these connections, this article strives to contribute to a sustained conversation on AI in business communication pedagogy by proposing that the way we approach teaching students about AI has the potential to influence whether and to what extent they become critical users of genAI for business communication purposes.

Frameworks for AI Literacy

The rapid integration of Artificial Intelligence (AI) into various aspects of society has raised concerns about the need for individuals to develop AI literacy. As Long and Magerko (2020) define it, AI literacy is “a set of competencies that enables individuals to critically evaluate AI technologies; communicate and collaborate effectively with AI; and use AI as a tool online, at home, and in the workplace” (2). AI literacy involves the ability to understand, evaluate, and engage with AI technologies in a thoughtful and informed manner.

Recognizing the increasing role of AI in various aspects of society, different frameworks for AI literacy have been put forward in recent years. Some of these frameworks focus on the technical aspects of AI use (Kandlhofer et al., 2016; Ng et al., 2021). Kandlhofer et al. (2016) include topics such as machine learning, problem solving by search, sorting, and graphs and data structures in their AI education modules aimed at fostering AI literacy from kindergarten to higher education.

Increasingly, scholars and educators are recognizing the importance of including the ethical, social, and cultural dimensions alongside the technical. Exactly how much varies depending on the framework. As part of their AI for K12 project, Touretzky et al. (2019) frame five “big ideas” to guide instruction about AI, the fifth of which emphasizes identifying positive and negative ethical or societal impacts of both existing and future AI technologies. In their frequently cited review of interdisciplinary literature on AI literacy, Long and Magerko (2020) propose a conceptual framework composed of five overarching themes, each with a set of core competencies and design considerations. Three of the five themes (“What can AI do?,” “How should AI be used?,” and “How do people perceive AI?”) incorporate ethical, social concerns as well as the other two themes (“What is AI?” and “How does AI work?”) that more explicitly emphasize technical knowledge. Bali’s (2023) critical AI literacy framework includes how AI works but also centers understanding inequalities, bias, ethics, prompt engineering, and why/when/where AI helps in order to empower individuals to navigate the complex landscape of AI.

Indeed, in their review of work about AI and business communication, Getchell et al. (2022) call for scholars to develop frameworks to discuss the technical, practical, ethical, and social implication of AI-assisted communication. Based on their study of 343 communication instructors, Cardon et al. (2023) propose a framework for AI literacy that aims to cultivate students as proficient managers of AI tools. Through the four key components of application, authenticity, accountability, and agency, the framework (a) stresses the importance of developing a nuanced understanding of AI’s application, (b) encourages familiarity with diverse generative AI tools and the strategic alignment of their capabilities with specific communicative tasks, and (c) emphasizes human agency and the need for individuals to retain control over decision-making processes in the context of AI tools.

As this body of scholarship points out, business communication students’ awareness can and should extend farther than the technical know-how needed for using AI tools. Business communication instructors need to develop students’ awareness of technical, practical, ethical, and social implications for AI tools in business communication. Effectively integrating critical AI literacy into the curriculum demands the development of pedagogical strategies. This work, however, is complicated by a host of challenges, including the lack of teachers’ AI knowledge, the need to do this pedagogical work rapidly, instructor time constraints, limited resources, and institutional constraints (Cardon et al. 2023; Su et al., 2023). In addition, students’ attitudes about AI may complicate their ability to build critical AI literacy (Chan & Hu, 2023; Kelly et al., 2023). Addressing these challenges is paramount for fostering innovation in AI literacy instruction and ensuring that business communication students receive a comprehensive education that prepares them for the nuanced landscape of AI in professional settings.

Description of Study

As Cardon et al. (2023) point out, the widespread use of generative AI necessitates significant changes to learning and teaching business communication. Following Cardon et al.’s (2023) call, this study was born out of our recognition that “business as usual” in the business communication classroom would not suffice. So we set out to revise our approach to teaching business communication principles while addressing AI literacy skills.

We teach in a stand-alone writing department at a midsized public university in the U.S. Midwest. In Fall 2023, Professor A was scheduled to teach three sections of Business Communication (Writing 350) in an online asynchronous modality. WRT 350 is a three-credit class that emphasizes training in communication skills for business and the professions. While open to Writing majors and minors as an elective, this course primarily enrolls nonmajors, and it satisfies a university requirement for supplemental writing skills. Many of the students in this course are computer science or information technology majors, as this is currently a requirement in their curriculum.

One of the primary methods of instruction in this class (and for many business communication classes) is the case study. Previous iterations of Professor A’s WRT 350 course asked students to practice responding to different business scenarios to help them learn genre conventions, apply rhetorical principles, and practice standard business writing principles (e.g., you-attitude, positive emphasis) that are covered in assigned textbook readings (Locker et al., 2019; Kolin, 2022) and lectures. But faced with the reality of genAI and the overall vulnerability of the existing assignment to genAI, we decided to experiment and combine what students learned about business writing principles with instructions on AI in a new learning module.

We developed a four-week module that introduced key business communication principles interwoven with an exploration of generative AI (e.g., how it works, what it can/can’t do, the ethics of using AI, its role in workplace communication). Our planning was informed by Long and Magerko’s (2020) framework and 17 AI competencies. Given the scope of the class and a semester’s time constraints, we focused our attention on specific competencies (1, 2, 5, 9, 10, 16) both in terms of the module content and assessment.

This module took the place of the previous assignment in which students practiced responding to different business scenarios. The updated module required all students to (a) complete a pre- and post-unit self-assessment of their knowledge and ideas about AI; (b) complete short readings and style exercises on business writing principles; (c) watch a selection of short instructional videos on how AI and machine learning works; (d) research and create a slide about the functioning, capabilities, and limitations of a genAI program (e.g., Dall-E, CodePilot, Claude) of their choice; (e) critique or modify genAI output; and (f) compose a written reflection on their engagement with all of these tasks in the module.

Exactly how students worked with genAI output varied by section. All students were presented with the same workplace scenario involving communicating the news to the employees of a fictitious company about a new merger and the resulting layoffs (see Appendix A in the online Supplemental Material). But in two sections, students were given a company persona and ChatGPT’s output in response to the scenario and asked to analyze its use of key business communication principles, rhetorical principles, and genre conventions and reflect on the aspects of the AI’s letter that were well-done and those that could be improved. In the third WRT 350 section, students were given the same scenario and company persona and instructed to collaborate with ChatGPT to write an effective, ethical response that smoothly integrated AI output with their own writing. Effective responses followed key business communication principles, rhetorical principles, and genre conventions. Along with their response, students submitted an appendix with a “revealed” version of their response with the ChatGPT contributions highlighted. Any student who might have had ethical concerns (Chan & Hu, 2023) about directly working with ChatGPT were provided with an alternative way of completing the assignment (in this case, using other sections’ instructions).

In the final week of the module, all students were asked to reflect on the previous weeks and the work they had been doing to become acquainted with both long-standing business communication principles and new tools, like ChatGPT. They were asked to reflect on their experience of applying business writing principles (e.g., you-attitude, clarity, positive emphasis) as well as their experience of using ChatGPT in their case study responses. They were also asked to reflect on what roles they thought AI-assisted writing should play in the workplace, what ideas about writing, AI, business, or humanness were affirmed, tested, or changed, and under what circumstances (if any) they would use ChatGPT in the future.

Methodology

The methods for this research study are based on a mixed methods research paradigm, combining qualitative and quantitative data. More specifically, we have employed the “explanatory sequential design” as described by John Creswell (2015, p. 37), a well-known proponent of mixed methods research. This approach combines quantitative survey questions with qualitative ones, and additional insights are gained from written documents generated throughout the study. The analysis stage consists of statistical methods to interpret quantitative data and simultaneous coding of qualitative data to gain insights into patterns within survey answers as well as documents. The findings from both types of analysis were then triangulated and presented while showing connections between statistically significant results and patterns that arise from qualitative analysis.

This mixed methods design ensured that we did not solely rely on one-dimensional quantitative data in our findings but that we were able to augment our insights gained through the survey results through textual evidence. With the qualitative data from the open-ended survey answers and the case studies and reflection documents, we were able to see potential themes emerging about the effectiveness of each specific instructional method applied in the two groups of students. The open-ended qualitative survey questions such as the question about the definition of AI and additional notes on what makes students concerned about AI allowed participants to express their independent thoughts without restrictions of predesigned answers for quantitative questions. In addition, the case studies and reflections revealed (a) students’ individual perspectives and attitudes about AI, (b) how these perspectives and attitudes impacted their writing choices, and (c) how they applied the knowledge gathered through the module to course assignments. The mixed methods data helped us not only understand students’ thinking patterns and perceptions of AI but also how these patterns impacted their understanding and application of business communication principles when analyzing text created by ChatGPT or when using ChatGPT-generated text as the basis for completing a case study.

Participants

Participants in the study were recruited from three online asynchronous sections of a business communication course. From the total 66 undergraduate students in these sections, 51 students opted to participate in the study when filling out the consent form as part of the pre-unit survey at the beginning of the module. One student dropped the class, and as a result, 50 students filled out the post-unit survey. Documents for qualitative analysis were also gathered from these 50 students. As mentioned above, students were placed in two groups. The group that analyzed ChatGPT output (control group) had 33 assigned students, and the group that collaborated with ChatGPT to generate an output had 17 assigned students.

Based on general information gathered about participants through the survey, we also know that 26 of the 50 students had some programming experience. In addition, 34 students stated that they had some experience using AI-driven technology, primarily ChatGPT (60%) and Snapchat (16%). The majority of the study participants (67%) were not regular users of AI technology; only a quarter of the participants used AI at least once a week or more.

Quantitative Data Set

Quantitative data were gathered from the students using pre- and post-unit surveys that assessed their knowledge and attitudes about AI. Both the pre- and post-module surveys consisted of 38 questions. We drew upon Long and Magerko’s (2020) conceptual framework composed of five overarching themes to structure the survey: What is AI? (6 questions); What can AI do? (12 questions); How does AI work? (6 questions); How should AI be used? (2 questions); and How do people perceive AI? (7 questions). These questions measured students’ ability to (a) distinguish between technological artifacts that use and do not use AI; (b) identify strengths and weaknesses of AI; (c) understand the steps involved in machine learning; (d) recognize that humans play an important role in programming, choosing models, and fine-tuning AI systems; (e) recognize that computers often learn from data (including one’s own); (f) identify and describe different perspectives on key ethical issues surrounding AI (i.e., privacy, misinformation); (g) imagine possible future applications of AI and their effect; and (h) understand that agents are programmable. There were also five questions that collected information about participants’ experience with AI. The surveys were administered through Qualtrics, whereas data analysis was performed using SAS. Data were summarized using cross-tabulations with the purpose of revealing any statistically significant differences between students from different sections. Exact 95% confidence intervals for population proportion were obtained for each classification of interest. Additionally, to help visualize the comparisons, forest plots were generated.

Qualitative Data Set

Qualitative data was gathered through three open-ended questions on the surveys. From the answers to these three questions, we focused our analysis on the question where students were asked to provide their own definition of AI. These answers were then coded using the categories listed in Zhang et al. (2023). We also gathered qualitative data from students’ case studies and reflection assignments. As described above, students completed a case study where they had to write a letter to employees of a fictitious company about a new merger and the resulting layoffs. The case studies generated from this assignment were collected, deidentified, and coded through the qualitative data analysis process. In addition to the case studies, we also collected the reflection assignment (see Appendix B in the online Supplemental Material) that students turned in at the end of the unit. These were also deidentified but were paired with the corresponding case studies.

To gain insight from the qualitative data, we divided the documents into groups corresponding to whether the students who created them belonged to the analysis of AI output (control) group or collaboration with AI (variable) group. Initially, these documents were coded based on content to assess what common topics, attitudes, and opinions emerged about AI in the student texts. After this initial coding process was finished, the codes identified in the documents were compared across the variable and control groups and based on patterns in this comparison, themes emerged from the student texts that were identifiable across all documents. To ascertain interrater reliability, the authors independently read the qualitative data and developed thematic categories. These were compared, and both of the raters had a reasonable degree of agreement on these categories. As we found substantial overlap between the themes and categories identified during the initial coding and the three main aspects motivating the design of the AI learning unit, we decided to settle on a set of codes that were grouped into three main categories: (a) business writing principles listed in the assigned readings (Locker et al., 2019; Kolin, 2022), (b) AI literacy competencies based on the framework by Long and Magerko, 2020, and (c) students’ level of collaboration with ChatGPT in the group (the variable group) that was assigned to collaborate with the AI tool to write a letter.

Once these categories were identified, the authors reviewed the data set again with this shared coding scheme in mind and marked the presence or absence of the codes relating to these 3 main categories. Both authors followed the same qualitative data analysis procedure outlined in Creswell’s (2015) description of mixed methods research where they relied on memos during the initial coding process to reflect on patterns observed and to record and share their decision-making process with each other. The patterns that emerged from the coding process were then triangulated with the quantitative data to establish validity and to follow the producers of the explanatory sequential design.

Results

This study revealed that students’ understanding of AI was impacted by the end of the module.

As Figure 1 highlights, there was a decrease in vague and incorrect definitions and an increase in societal and technical definitions by the end of the unit. A majority of students (51%) started the unit with vague definitions of AI. At the outset, fewer students defined AI in terms of technical components (17%), whereas almost as many students (14%) gave incorrect answers. No students defined AI in terms of societal impact on the pre–self-assessment. However, by the end of the unit, students most commonly defined AI as technical (37%). Ten percent of students integrated societal implications into their definitions (with the majority of those participants coming from the control group). We observed both a decrease in vague (22%) and incorrect (6%) definitions. Of the students who started the unit with an incorrect understanding of AI, 57% developed more nuanced understandings of AI by the end of the unit. For students who started the unit with a vague understanding of AI, 50% developed a more nuanced understanding, but 27% remained holding vague ideas.

Figure 1.

Comparison of students’ definitions of AI at the beginning of the unit to the end.

In general, students’ self-reported attitudes toward AI remained consistent, with a trend toward increased positivity where changes occurred. Figure 2 illustrates student responses with options for “agree” (1), “disagree” (2), or “uncertain” (3), except for “daily life” plot, where options were “more concerned” (1), “equally concerned and excited” (2), and “more excited” (3).

Figure 2.

Comparison of students’ self-reported attitudes toward AI at the beginning and end of the unit.

For instance, examining the data on the “privacy” plot reveals that a majority of students comfortable with AI using their personal data at the beginning of the unit (i.e., 1 on the y axis) remained so at the end (93.8%), with only 6% becoming less comfortable. In addition, 100% of the students who were not comfortable at the outset (i.e., 2 on the y axis) remained unchanged. Notably, if student attitudes toward AI did change, they tended to move in a more positive direction. This trend also included students who initially held more concerned or negative stances, many of whom reported more of a mixed attitude over time. We also saw a reduction in the number of students expressing uncertainty about various AI issues at the beginning of the unit. By the unit’s conclusion, more students affirmed that they would continue learning about AI and that they were more aware of AI in the workplace.

While characterizing the full quantitative data set is beyond the scope of this article, we did isolate specific survey questions that aligned with the focus on business writing principles in the qualitative analysis. The first question we isolated came from the section of questions focusing on what AI can do. Students were asked to identify which tasks AI could handle (strengths) and which it could not (weaknesses).

Table 1 above highlights the difference in positive impact levels between the control and variable groups, where positive impact denotes a transition from an incorrect presurvey response to a correct postsurvey response. “Correct” denotes a correctly identified weakness in both pre- and postsurveys. “Noncorrect” denotes either a transition from correct presurvey to incorrect postsurvey response or a missing response.

Table 1.

Comparison of Student Identification of AI Weaknesses From Pre- to Postunit.

AI Weaknesses	Response	Control, n (%)	Variable, n (%)
Deliver news about a layoff	Positive impact	7 (20.59)	2 (11.76)
	Noncorrect	16 (47.06)	11 (64.71)
	Correct	11 (32.35)	4 (23.53)
Understand cause and effect at the level of an adult human	Positive impact	8 (23.53)	0 (0.00)
	Noncorrect	13 (38.24)	5 (29.41)
	Correct	13 (38.24)	12 (70.59)

For instance, when asked if AI can understand cause and effect (e.g., client is very upset after the company does not follow up on a promise) at the level of an adult human, the control group had a significantly higher level of positive impact (i.e., when a participant’s response changed from incorrect in the presurvey to correct on the postsurvey) compared to the variable group.

Similarly, when asked if “delivering news about a layoff” was a strength or weakness of AI, the control group had a much higher level of positive impact. In other words, compared to their preunit responses, more of the students in the control group correctly identified this task as a weakness of AI in their postunit responses. Comparatively, more participants in the variable group got this question wrong in both the pre- and postunit surveys.

These results from our quantitative analysis directly align with the findings we discovered during our qualitative analysis of students’ case studies and reflection assignments. As we coded these student documents for the presence of certain business writing principles, we observed that overall in their assignments many students focused on how effectively you-attitude was present in the AI-generated text (62%). Many assignments also showed that students in both groups were aware of rhetorical principles in the AI generated output (58%) and addressed formatting elements that are required for formal business letters (52%). Overall, fewer students referenced principles such as building goodwill with the reader (36%) and the clarity of the AI-generated message (36%). While all business writing principles were mentioned by some students in each group, certain business writing principles were noticed by the control group significantly more frequently than by the variable group that collaborated with ChatGPT.

The most notable result from our qualitative analysis that separated students based on whether they were assigned to analyze ChatGPT output (control group) or to collaborate with ChatGPT to write a letter (variable group) was that students in the control group were more likely to identify issues with all principles within the ChatGPT-generated text, and they were significantly more likely to do so in all cases except for positive emphasis (64% vs. 59%). Of note, 79% of the students who analyzed ChatGPT output identified issues with you-attitude, compared to 29% of students who collaborated with the AI. Further, 70% of students who analyzed ChatGPT output identified issues with formatting, compared to 18% of students who collaborated with the AI. In addition, 55% of students who analyzed ChatGPT output identified issues with concision while no students who collaborated with the AI did.

Another important finding in the variable group connects to the extent to which students in this group interacted with ChatGPT and engaged in revising and finalizing the output to address the case study requirements. Students in this group were free to create a prompt, change their prompts to get improved results from ChatGPT, and modify ChatGPT output while combining it with their own writing to arrive at an effective letter that contains bad news for readers. While students had all this freedom to make substantial changes to the ChatGPT-generated text, we found that 71% of students in this group made only minimal changes to the generated output. Further, 24% of students made several, more involved changes to the text and structure of the AI generated text, but only 1 of the 17 students in the variable group made substantial change to the generated output to ensure that the resulting letter was successfully addressing all aspects of the case scenario and represented effective use of the business writing principles covered in the class.

These findings from our qualitative analysis of students’ documents show that students in the variable group who collaborated with ChatGPT were less likely to identify and change the shortcomings of the generated text with regard to effective use of business writing principles. These results also align with the quantitative findings described above where students in the variable group were less likely to identify “delivering news about a layoff” as a weakness of ChatGPT, while the control group had a more accurate assessment of ChatGPT’s strengths and weaknesses when it came to creating effective business messages.

Discussion and Implications for Teaching of Business Communication

The results of this study shed light on the evolving dynamics of students’ understanding, attitudes, and engagement with AI in the context of business communication. The findings suggest that the specific pedagogical approach employed in integrating genAI into the curriculum can have a significant impact on students’ perceptions and cognitive development in relation to AI. The two approaches used in this study have produced significantly different results.

When students were asked to analyze AI-generated text (in the control group) they were more likely to correctly identify the strengths and weaknesses of the ChatGPT output and its use (or lack of use) of important business writing principles in the case of delivering bad news. One student from the control group wrote, “ChatGPT excelled in generating coherent text but had limitations in capturing nuances, empathy, and context. It could generate content that sounded professional, but it lacked the ability to understand the emotional or cultural aspects of communication” (Participant 9, “Reflection”). Text analysis and critique is a common practice in the business communication classroom and based on our results, it appears to also be an effective approach for making students aware of this AI tool’s often ineffective application of some business communication principles. Thus, this teaching approach resulted in more critical awareness of AI use for business communication tasks.

In our variable group, students were allowed to collaborate with AI to generate a bad news letter that also reflects the application of effective business communication principles. Our motivation for incorporating this approach in our module for the variable group is connected to the many calls for open engagement with AI in classrooms and to the practice that some instructors describe as “letting students play with AI.” This approach that we often hear in the hallways of universities and at conferences is becoming quite common, but we were curious what type of results it will create for our students’ understanding of business communication principles and their perception of generative AI tools such as ChatGPT. We found that students in the variable group were “dazzled” by the style of the ChatGPT output, which made them less critical of this text’s use of business writing principles. This is visible in the following quote from a student’s reflection in the variable group: “ChatGPT adds a touch of professionalism to my writing that I sometimes struggle to achieve on my own. It helps me refine my language and structure, making my written communication more polished” (Participant 30, “Reflection”). Similarly, another student in the variable group wrote: “without AI, I would not have been able to sound as professional” (Participant 11, “Reflection”).

This “dazzling” effect of ChatGPT was certainly seen in the variable group as these students’ overly positive attitude about AI resulted in less willingness of critiquing and modifying ChatGPT output. As instructors of business communication, this certainly makes us question the effectiveness of “letting students play with AI” approach without other types of scaffolding activities first that allows the emergence of students’ critical stance toward AI. Based on this study, we would certainly caution instructors against separating engagement with AI tools from the teaching of business writing principles. Making students aware that the seamless style of ChatGPT-generated text can hide a lack of effective use of business writing principles should be an important learning objective of AI-focused business communication modules in our classes.

Limitations

While this study contributes valuable insights into strategies for integrating and teaching about AI in the business communication classroom, it is essential to acknowledge certain limitations that may impact the interpretation and generalizability of the findings. One notable limitation of this research pertains to the relatively small sample size. While the recruitment level was high, the number of participants may limit the representativeness, generalizability, and transferability of the results to a broader population. Another challenge we encountered was the inconsistency in participant engagement, both in the pre- and postunit surveys. Some participants did not respond to all questions, and in certain cases, there was a lack of uniformity in responses between the pre- and postunit surveys. This variability in participation restricted our ability to comprehensively assess the impact of the level of engagement with AI on student learning. We also acknowledge that the written directions in the assignment description communicated a certain level of freedom about students’ extent of engagement with the AI output, and this may have influenced their approach to modifying text generated by ChatGPT especially as this could have been some students’ first attempt at AI-human collaborative writing.

Additionally, some of the data collected in this study relies on self-reported measures, introducing the possibility of subjectivity, unintentional memory biases, or systematic distortions. It is also important to note that the findings of this study are situated within a specific context, specifically that of an online asynchronous classroom. The unique circumstances of the study environment should be considered when extrapolating the results to broader contexts. These limitations should be taken into account when interpreting and applying the findings. Future research endeavors could address these limitations through larger sample sizes, improved survey design, and enhanced participant engagement strategies.

Conclusions and Future Research

This pedagogical experiment shed valuable insight into strategies for integrating and teaching about AI in the business communication classroom. A notable finding is that less direct engagement with genAI had a greater impact on students’ understanding of business writing principles. Despite ChatGPT’s human-like interactions, students in the control group recognized its inability to truly identify with people. Our results underscore the significance of analyzing machine-generated output in enhancing students’ awareness of the human elements of communication, especially in the context of business writing. The question of whether this awareness transfers to application is yet to be answered.

Unsurprisingly, the findings of this study show that the design of our pedagogical approaches does have an impact on student learning. Experimenting with different approaches for teaching our students about AI and its effective use in business communication will lead to a deeper understanding about the approaches that result in more critical use of these tools and in more effective deliverables. While ChatGPT’s wide availability is fairly recent and we may feel compelled to quickly adapt our pedagogy to include the use of this novel technology, we should not abandon the communication principles that were developed long before the advent of AI in the field of business communication based on our shared understanding of rhetoric.

As we help our students navigate the new realities of workplace communication where they will work with people and artificial intelligence to collaborate on documents, we owe it to our students to teach them critical AI literacy skills that intersect with successful application of business writing principles. Thus, we need more research on different pedagogical designs where students not only learn about the strengths and weaknesses of AI when generating business messages but are also able to apply their critical insight gained from analysis as they collaborate with AI on writing messages. In a future iteration of this project, we hope to build on the strategies we used in the control group to develop students’ critical AI literacy and business writing knowledge. Our goal is to scaffold direct engagement with AI so that students can apply the acquired skills and knowledge. We hope that this ongoing experimentation and additional pedagogical research in our field about AI literacy will lead to pedagogically sound teaching practices about this novel technology and the cultivation of a generation of professionals who are not only adept communicators but also critical and ethical users of AI applications in the workplace.

Supplemental Material

sj-docx-1-bcq-10.1177_23294906241253199 – Supplemental material for Building Critical AI Literacy in the Business Communication Classroom

Supplemental material, sj-docx-1-bcq-10.1177_23294906241253199 for Building Critical AI Literacy in the Business Communication Classroom by Danielle DeVasto and Zsuzsanna Palmer in Business and Professional Communication Quarterly

Footnotes

Acknowledgements

The authors would like to thank the Statistical Consulting Center at Grand Valley State University for their assistance with the statistical analysis for this study. Their support and expertise were greatly appreciated.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Danielle DeVasto

Supplemental Material

Supplemental material for this article is available online.

Author Biographies

Danielle DeVasto is an assistant professor in the Department of Writing at Grand Valley State University. Her research interests reside at the intersections of visual rhetoric, science communication, and uncertainty. Her work has been published in Technical Communication Quarterly, Communication Design Quarterly, Community Literacy Journal, and Journal of Business and Technical Communication.

Zsuzsanna Palmer, Ph.D., is an associate professor in the Department of Writing at Grand Valley State University where she teaches professional writing, business communication, and document design. Her research interests include intercultural communication, virtual exchange projects, website accessibility, and translingual writing.

References

Bali

(2023, April 1). What I mean when I say critical AI literacy. Reflecting Allowed. https://blog.mahabali.me/educational-technology-2/what-i-mean-when-i-say-critical-ai-literacy/

Cardon

Fleischmann

Aritz

Logemann

Heidewald

(2023). The challenges and opportunities of AI-assisted writing: Developing AI literacy for the AI age. Business and Professional Communication Quarterly, 86(3), 257–295. https://doi.org/10.1177/23294906231176517

Chan

C.K. Y.

(2023). Student’ voices on generative AI: Perceptions, benefits, and challenges in higher education. International Journal of Education Technology in Higher Education, 20, 43. https://doi.org/10.1186/s41239-023-00411-8

Creswell

J. W.

(2015). A concise introduction to mixed methods research. Sage.

Getchell

K. M.

Carradini

Cardon

P. W.

Fleischmann

Haibing

Aritz

Stapp

(2022). Artificial Intelligence in Business Communication: The Changing Landscape of Research and Teaching. Business and Professional Communication Quarterly, 85 (1), 7–33. https://doi.org/10.1177/23294906221074311

Hancock

J. T.

Naaman

Levy

(2020). AI-mediated communication: Definition, research agenda, and ethical considerations. Journal of Computer-Mediated Communication, 25(1), 89–100. https://doi.org/10.1093/jcmc/zmz022

Kandlhofer

Steinbauer

Hirschmugl-Gaisch

Huber

(2016). Artificial intelligence and computer science in education: From kindergarten to university [Conference session]. 2016 IEEE Frontiers in Education Conference (FIE) (pp. 1–9), Erie, PA, USA. https://doi.org/10.1109/FIE.2016.7757570.

Kasneci

Sessler

Küchemann

Bannert

Dementieva

Fischer

Gasser

Groh

Günnemann

Hüllermeier

Krusche

Kutyniok

Michaeli

Nerdel

Pfeffer

Poquet

Sailer

Schmidt

Seidel

. . . Kasneci

(2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and individual differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274

Kelly

Sullivan

Strampel

(2023). Generative artificial intelligence: University student awareness, experience, and confidence in use across disciplines. Journal of University Teaching and Learning Practice, 20(6), 1–18. https://doi.org/10.53761/1.20.6.12

10.

Kolin

P. C.

(2022). Successful writing at work. Cengage Learning.

11.

Locker

Mackiewicz

Aune

Kinzler

(2019). Business and administrative communication (12th ed.). McGraw Hill.

12.

Long

Magerko

(2020, April). What is AI literacy? Competencies and design considerations [Conference session]. Proceedings of the 2020 CHI conference on human factors in computing systems, Honolulu, HI, USA, (pp. 1–16).

13.

D. T. K.

Leung

J. K. L.

Chu

K. W. S.

Qiao

M. S.

(2021). AI literacy: Definition, teaching, evaluation and ethical issues [Conference session]. 84th Annual Meeting of the Association for Information Science & Technology, Salt Lake City, UT, USA. https://doi.org/10.1002/pra2.487

14.

D. T. K.

Chu

K. W. S.

(2023). Artificial intelligence (AI) literacy in early childhood education: The challenges and opportunities. Computers and Education, 4, 100124. https://doi.org/10.1016/j.caeai.2023.100124

15.

Touretzky

Gardner-McCune

Martin

Seehorn

(2019, July). Envisioning AI for K-12: What should every child know about AI? [Conference session]. Proceedings of the AAAI conference on artificial intelligence, Honolulu, HI, USA, (Vol. 33, pp. 9795–9799).

16.

Zhang

Lee

Ali

DiPaola

Cheng

Breazeal

(2023). Integrating ethics and career futures with technical learning to promote AI literacy for middle school students: An exploratory study. International Journal of Artificial Intelligence in Education, 33(2), 290–324.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.09 MB