Adapting to the Unsanctioned Use of AI-Supported Technologies in Student Assessments

Introduction

Educators have enjoyed a few decades of mostly anxiety-free innovations in education-focused technologies. It would be difficult to imagine returning to a world before Canvas, Moodle, Blackboard and Turnitin existed. Digital technology has permeated the profession at both the practical and theoretical levels during the twenty-first century. Not only have we become dependent on technology in the education sector, but few topics have also generated more scholarly production over the last decade than the use of digital tools in the classroom. A 2-year pandemic only deepened our reliance.

For the first time, there are hints that the honeymoon with technology in education is coming to an end. The release of ChatGPT-3.5 in 2022 sent ripples of angst through the education sector. Educators saw in the AI model a real threat to the integrity of student assessment methods. ChatGPT appeared to do everything that our students could do—often at a higher level. In the months that followed, universities and colleges around the world worked frantically to propose temporary solutions to the ‘AI problem’, while administrators and faculty tried to formulate some basic understanding of the technology and its capacities. It would be difficult to name another technology that has blindsided higher education like ChatGPT.

In this article, I offer a brief description of large language models (LLMs) and discuss ChatGPT’s capabilities and limitations from an educator’s perspective. In the process, I examine the complex problem over which many university educators are losing sleep: Student use of AI systems to cheat on assignments. Finally, I propose a multi-level strategy for addressing the problem systematically.

It is important to acknowledge that many educators will find positive uses for AI-based tools in the classroom and beyond. As the title of this article indicates, I have limited my analysis to addressing unsanctioned uses of AI-supported technologies. I fully support the claim that educators should strive to find ways to work with these technologies when possible because the potential benefits for learners and teachers are obvious. The rapid expansion of AI-based tools into the education sector (and every other sector) is likewise inevitable. As always with paradigm-shifting technologies, however, ‘good’ usage and ‘bad’ usage are eventualities that must be discussed and debated during the initial stages of integration.

ChatGPT and LLMs

ChatGPT is a general designation used to refer to multiple generations of AI-chatbot iterations developed by OpenAI between the years 2018 and present. A stable version of the next iteration, ChatGPT-4, was made public in March 2023 (ChatGPT-4 Turbo announced in November 2023). ChatGPT-3.5, the initial launch version of the technology, was released in 2020 but was not made public until November 2022. In 2024, OpenAI made ChatGPT-4 available to the public for free. This section will focus on the capabilities of ChatGPT-3.5 and ChatGPT-4 since these are the iterations to which students have access most often. It is important to note, however, that indirect access to more advanced models has become common through the use of plug-ins and through the integration of that model into various consumer applications. Universities are beginning to make more advanced models available to students as well.

ChatGPT is an example of an LLM that has been trained to interact with questions or prompts in a conversational way. When asked to define LLMs, ChatGPT-3.5 says they ‘are artificial intelligence models that use deep learning techniques to analyze and understand human language. These models are designed to process vast amounts of text data and generate human-like language’. They ‘consist of multiple layers of neural networks that are trained using large data sets of text’. From this training, an LLM learns to ‘recognize patterns in the data’ and to ‘generate new text that is similar to the training data’ (OpenAI, 2023b). In addition, the next generation, GPT-4, is multi-modal, which means that it can encode and decode non-textual data to produce image, sound, video and text responses.

It is important to understand that LLMs are, in essence, prediction-making algorithms. ChatGPT is able to analyze ‘the context and meaning’ of the input, allowing it to generate ‘the most probable response’ to a question or prompt (OpenAI, 2023b). LLMs are not programmed to provide factual information. They predict the most probable language responses to a question or prompt based on language patterns that they identify in the data set on which they have been trained. A consequence of the way that LLMs work is that they have a tendency to ‘hallucinate’ responses at times, meaning that they generate answers that humans classify as fabrications (Athaluri et al., 2023; Azamfirei et al., 2023; Rudolph et al., 2023a). If patterns of thought exist in the data set, an LLM will identify those patterns and generate probable responses (predictive inferences drawn from those patterns). The factualness of a response is not guaranteed since responses are more of a ‘prediction’ than an attempt to describe reality. ChatGPT-3.5’s responses to requests for citations often include completely fabricated article titles, for example, Choi (2023). In short, the model ‘hallucinates’ the titles of articles that could be written by a certain researcher based purely on inferences drawn from the researcher’s actual publication history (a probability response). These hallucinations can be understood as ‘predictions’ about the kinds of articles that an author could write, with the actual publication record used as a pattern for that prediction. OpenAI has worked to reduce the frequency of ‘hallucinations’ in their newest iteration of ChatGPT. As of November 2023, GPT-4 and 4-Turbo led the pack among competing LLM-based AI models with a 97 per cent accuracy score (Hughes et al., 2023). Factual consistency rates have continued to improve in 2024, with OpenAI’s latest iterations, GPT-4o-mini (98.3 per cent), GPT-4o (98.5 per cent) and OpenAI-o1-mini (98.6 per cent) (Hughes et al., 2023).

Notably, LLMs can be trained to exclude certain responses or classes of response but this requires human intervention. The process of training a model to ignore certain patterns in the data set and to promote other patterns is referred to as ‘alignment’. The idea behind alignment is to try to ensure that the model’s responses remain consistent with the values and ethics of human societies. The method that OpenAI employs to train ChatGPT is called reinforced learning from human feedback (RLHF). At this point, human evaluations of the model’s responses still play a central role in the way that ChatGPT’s algorithm is trained. For now, this means that the model does not generate responses that we would categorize as strictly independent from human intervention. However, as the models develop over time, OpenAI has stated that less direct human intervention will be required since more advanced models will be able to train/align themselves to a pre-determined set of values (Altman, 2023; Leike et al., 2022).

Initial Thoughts about the Effects on Education

Students have had access to ChatGPT-3.5 since November 2022; however, the company’s popularity spiked in March 2023 when OpenAI launched ChatGPT-4. Discussions about the abilities and implications of AI models began to permeate both traditional and social media immediately. Since then, ChatGPT has exceeded all expectations, setting a record for the fastest-growing consumer application in history, with 100 million active monthly users within 2 months after launch (Hu, 2023).

Education was among the first sectors to sound an alarm about the use of LLMs. Concerns about widespread cheating and/or plagiarism led to a sense of panic, as many educators began to understand that it very difficult, perhaps even impossible, to detect when students were using AI systems to complete assignments. OpenAI acknowledged the scale of the problem, publishing a set of ‘considerations’ for educators on their website (OpenAI, n.d.). Regarding the issue of plagiarism, the document asserts that it is the responsibility of individual institutions to address the problems created by student usage of their technology, while urging caution about attempts to hold students accountable in the absence of clear guidelines:

We also understand that many students have used these tools for assignments without disclosing their use of AI. Each institution will address these gaps in a way and on a timeline that makes sense for their educators and students. We do however caution taking punitive measures against students for using these technologies if proper expectations are not set ahead of time for what users are or are not allowed (OpenAI, n.d.).

OpenAI recommends the use of AI detectors, including its own, but admits that the technology is not entirely reliable:

These tools will produce both false negatives, where they don’t identify AI-generated content as such, and false positives, where they flag human-written content as AI-generated. Additionally, students may quickly learn how to evade detection by modifying some words or clauses in generated content (Educator Considerations for ChatGPT, n.d.).

It is clear from these admissions that education is in a moment of transition. Traditional modes of assessment will need to be reimagined, as they are now vulnerable to an unprecedented kind of ‘exploit’. It will be in everyone’s interest (students, teachers and administrators), therefore, to try to close the ‘gaps’ as quickly and conscientiously as possible.

Identifying a Complex Set of Problems

To say that LLMs—more specifically, the many applications of that technology within the domain of generative AI—will create complex challenges for the world in the near future is a gross understatement. A growing number of researchers are beginning to describe the diverse ways in which generative AI will likely impact human understanding and influence human institutions, both positively and negatively. The scale of the perceived threats ranges from dire predictions about the extinction of our species [with the emergence of artificial general intelligence (AGI)], to warnings about the potential dissolution of Western democratic institutions, as we find it increasingly difficult to distinguish fact from fiction.

Many of these concerns centre on the pace of AI research and development, which appears to be trending towards a continuation of exponential scaling. Following from the work of Nick Bostrom and others, it is now widely assumed that the development of AI will continue to advance, finally reaching a point when that development becomes self-automated, unlocking the potential for an explosive acceleration well beyond the limits of human intelligence (Bostrom, 2016; Grace et al., 2018). More recently, and perhaps more problematically in the short term, are concerns expressed in a recent presentation given to the Centre for Humane Technology, which discusses the current frenetic rush of companies to introduce consumer versions of AI-supported tools to the market (Harris & Raskin, 2023). The launch of ChatGPT has sparked a massive response across many different domains, including social media platforms, search engines, image production, music production, video production, office productivity platforms, education platforms and language learning applications, just to name a few. To remain competitive, markets feel immense pressure either to integrate LLMs into their existing products or to develop entirely new products with AI systems running at the core. In a 2023 presentation, Tristan Harris and Asa Raskin raise questions about the ethical safety of this push, observing wisely, ‘when you invent a new technology, you uncover a new class of responsibility, and it’s not always obvious what those responsibilities are’ (Harris & Raskin, 2023). ¹ The process of discovering new responsibilities requires careful consideration, and careful consideration requires time.

As an important aspect of this consideration, the push to incorporate AI systems quickly and broadly has generated a debate around the future of work and the place of creativity. These are not new concerns, of course. A 2017 survey of 352 researchers in artificial intelligence predicted a 50 per cent chance that AI will replace humans in ‘all tasks’ related to work within 45 years (Grace et al., 2018). The market’s response to the launch of ChatGPT appears to lend credibility to this prediction. Of course, many of these concerns are linked to the potential for an emergence of AGI within the next few years. AGI represents the next step in the ‘evolution’ of AI, bringing it within parameters comparable to human intelligence. Sam Altman, CEO of OpenAI at the time, expressed his own apprehensions about the prospects of AGI:

The first AGI will be just a point along the continuum of intelligence. We think it’s likely that progress will continue from there, possibly sustaining the rate of progress we’ve seen over the past decade for a long period of time. If this is true, the world could become extremely different from how it is today, and the risks could be extraordinary (Altman, 2023).

Since the launch of ChatGPT-4 in early 2023, a few major players in the education sphere have introduced versions of their platforms that now make use of LLM technology. In most cases, this is now direct integration with GPT-4o. A few notable examples from well-established companies are DuolingoMax, GrammarlyGo and Khanmigo. According to Grammarly’s website, the goal of their platform, GrammarlyGo, is to ‘generate high-quality drafts, outlines, replies and revisions when you need them’. The appeal of these claims to students is obvious, and with many similar platforms awaiting imminent launch and even more platforms in development for 2025 release, the impact on student work will likely be enormous. To that point, Microsoft announced the integration of a next-gen version of ChatGPT into its Office productivity suite (the Prometheus Model), which means that students who subscribe to the Copilot service will have access to AI-generated content at the most basic levels of their workflow. Educators will need to have a clear understanding of how these tools are going to impact the ways that students (and the rest of the world) produce individual work.

The Arms Race Metaphor and Developmental Acceleration

ChatGPT is only one of many newly introduced LLM-based platforms. Amazon introduced its AI platform, Bedrock, in April 2023; Anthropic launched Claude 2 in July 2023; Meta released Llama 2 in July 2023; and in March 2023, Elon Musk announced plans to develop his own AI model to compete directly with OpenAI, a company with whom he was initially involved as an investor. Musk’s team (xAI) delivered their model, Grok, in November 2023. Lastly, and perhaps most importantly, Google introduced its own rival models, Bard and PaLM 2, between February and May 2023, respectively. One year later, Google relaunched Bard as Gemini 1.0 Ultra. Within a week of this relaunch, Google announced a next-gen AI model, Gemini 1.5, which appears to outperform ChatGPT-4 on many ‘general reasoning’ tasks. By the time this article is published, even more AI models will have been added to an increasingly crowded space, and most of the abovementioned companies will have introduced updated and thus more powerful models.

As an addendum, I find it important to mention Microsoft’s Bing AI as well, since it represents a new direction in AI implementation and accessibility. Microsoft launched its Bing chatbot in February 2023, although it has since been rebranded as Copilot. Their chatbot, which is freely available to anyone who uses the Bing search engine, makes use of a ‘next-generation OpenAI LLM that is more powerful than ChatGPT and customized specifically for search’ (Mehdi, 2023). A much-discussed limitation of ChatGPT-3.5 is the fact that the data it was trained on was not updated after ‘early 2022’ (OpenAI, 2023a). Whereas ChatGPT-4 was updated to include training data before April 2023, ChatGPT-3.5 is unaware of events that have happened since it ‘went live’. Copilot, on the other hand, maintains a connection to the internet, giving it access to an ever-expanding source of data. ² Rudolph et al. explain how a persistent connection to the internet leads to superior accuracy because the chatbot: ‘is thus aware of current events and not ignorant of events after September 2021, such as the war in Ukraine. It provides footnotes with links to sources and can provide proper academic references upon request’ (Rudolph et al., 2023b, p. 371). Going forward, we can expect that LLM-based tools will maintain persistent links to current data, making them both more useful and more difficult to either prevent or detect in education use cases. For example, OpenAI announced ChatGPT search in October 2024, making it possible for the current models to ‘get fast, timely answers with links to relevant web sources, which you would have previously needed to go to a search engine for’ (OpenAI, 2024a).

Undoubtedly, if the current rate of development persists, we will continue to see companies add increased functionality to their AI tools, making the task of keeping pace exceedingly difficult for policymakers and educators. In short, we are in the initial stages of an ‘arms race’ that is developing between the main players in the technology sector. Each entity is vying to secure market advantage over the other by offering the greatest degree of AI integration to the consumer as quickly as possible. The consequence is that an ever-growing number of AI platforms are being made available to the public. Following the established pattern for mass-market technologies over the last decade, these platforms are entering the consumer marketplace so rapidly that we do not have adequate time to consider the potential complications that may accompany a widespread use of AI-supported technologies.

On a secondary level, we are seeing a different sort of ‘arms race’ develop within the education sector. Traditionally, approaches to instruction and assessment are relatively stable concepts. Ideas usually have long life cycles, which means that advancements in education theory and praxis typically move at a slow pace. It is not uncommon, for example, to rely on theoretical frameworks that were introduced and popularized by educators in the previous century (e.g., Vygotsky’s sociocultural theory or Maslow’s ‘Hierarchy of Needs’).

Likewise, the life cycles of most technologies are often measured in decades. We expect to use most technologies for multiple years before being asked to master a new version/paradigm. Even if we replace a mobile phone each year, for example, we expect the operating system and UI to be recognizable (stable) enough to provide a continuous user experience. We do not expect to master a new technological paradigm each time we purchase a new version of a device. With AI models, however, not only are we encountering a completely new technological paradigm, but the underlying technology develops at a rate that is unprecedented in history. Not only will we need to understand and adapt to a novel technological paradigm, if we hope to adapt, we will need to track rapid iterative changes to the capabilities of that technology in real time.

To give an example of the rate of development that we are witnessing, consider the history of ChatGPT itself. OpenAI developed the first version of their language model, GPT-1, in 2018. The company released the first version of their conversation-based chatbot, ChatGPT, almost 4 years later, in 2022. Between 2018 and 2022, OpenAI developed multiple iterations of GPT, culminating in the now famous launch of ChatGPT-3.5 and the more recent release of ChatGPT-4 in March 2023. There are a few things to notice in the version history: First, OpenAI is able to release new iterations of GPT at the astounding pace of a new version roughly each year (GPT-1 in 2018, GPT-2 in 2019, GPT-3 in 2020, GPT-3.5 in 2022 and GPT-4 in 2023); second, in terms of scaling, each of these versions represents exponential growth beyond previous iterations (GPT-1 made use of 117 million parameters; GPT-2, 1.5 billion parameters; GPT-3, 175 billion parameters; GPT-4, unknown at present, but rumoured to be between 1 and 170 trillion parameters) (OpenAI, 2023b; VanBuskirk, 2023). This brief timeline demonstrates just how rapidly the upscaling of GPT’s capabilities has occurred. This is a trend that most expect to continue, with a more advanced version of GPT being launched possibly by 2025. Sam Altman has spoken about the fact that ChatGPT-5 is already in development (Dorrier, 2023). If this rate of advancement continues, it is not difficult to imagine AI systems drawing close to the stated goal of achieving AGI sooner rather than later. In a recent interview with The Wall Street Journal, CEO of Google’s DeepMind project, Demis Hassabis, confirmed that the goal of developing AI models that possess ‘human-level cognitive abilities’ is achievable within the next decade (Kruppa, 2023).

Thus, predictions that generative AI models will continue to develop at an accelerated pace, thereby extending their capabilities well beyond the threshold of being able to detect differences between human- and AI-generated responses, are not unreasonable. Even if we manage to decelerate these advancements, the last year has provided enough examples of the capabilities of generative AI to convince many experts that we have passed an important developmental milestone already. At present, detecting AI-generated content is extremely difficult, if not impossible (Gill et al., 2024, p. 21). For example, the AI detection tool developed by OpenAI was so unreliable at launch that it could not be used to detect content generated by OpenAI’s own AI model (ChatGPT). Their detection tool has since been shut down. In the light of these failures, one can argue that generative AI models (LLMs) already function beyond our capacity to detect instances of unsanctioned usage in traditional assessments. Educators must find ways to adapt their teaching and assessment strategies to the new realities created by AI-based technology or risk operating from a retrograde position. Problematically, the pace of AI development shows no signs of slowing down making the task of adapting analogous to aiming at a moving target.

Lastly, we are experiencing a third level of ‘arms race’: A struggle to stay ahead of our own students’ understanding and use of AI-based technology. As AI systems become integrated more and more into consumer technologies, especially social media platforms, ³ students will gain valuable competencies at a rate that may begin to outpace the competencies of their teachers. We need to wait for reliable data to provide clarity about the level of engagement students currently have with platforms like ChatGPT, but if anecdotal evidence has any value in this discussion, a significant portion of the 100 million active monthly users are students. A recent investigation by Study.com found that 89 per cent of the 1,000 students it surveyed have used ChatGPT to complete assignments (Study.com, 2023). Other studies have shown that the number of students using ChatGPT to cheat may be smaller (Welding, 2023). In July 2023, it was reported that nearly 400 university students in the UK were being investigated for use of ChatGPT to cheat on coursework, with 40 per cent of all universities in the UK currently conducting investigations into the use of AI (Sanders, 2023).

Complex Solutions for a Complex Problem

In the preceding section, I provided a summary of the current state-of-the-art in LLM-based AI systems, while describing some of the challenges that have arisen from its development. For many educators, the emergence of generative AI has led to unprecedented levels of concern over the potential for cheating on assessments. In the following sections, I present a series of observations and recommendations that I believe will move us closer to finding possible solutions. As a way of trying to make the challenges easier to imagine, I believe it will be helpful to think about the problem (the use of AI for cheating) as a set of system ‘vulnerabilities’. Plagiarism is a problem as old as written culture itself. The novelty that we are facing is this: LLMs can be used to exploit vulnerabilities in the way that we detect and prosecute acts of plagiarism. In response to this novelty, we need to begin to map the points where the current system is vulnerable. Next, we need to understand how LLMs can be used to ‘exploit’ these vulnerabilities. Lastly, we need to outline some of the limitations associated with using AI models to cheat on assessments. If we can arrive at a better understanding of these three areas, we can begin to think about how to ‘resecure the system’.

In the next section, I will argue that the system is vulnerable at four points. The intention is to demonstrate that proposed solutions will need to address these numerous vulnerabilities at their respective levels. I categorize the problem as ‘complex’, meaning that a simple, single-factor approach will not provide a solution because the problem does not exist at a single level. Complex problems require systematic approaches that address multiple weaknesses at their multiple points of origin. I argue that potential solutions will need to address exploits at four distinct levels of vulnerability: The ethical level, the pedagogical level, the technological level and the political (policy) level.

Layers of Security: Prevention, not Detection

It should be acknowledged from the beginning that the immediate strategy for educators will need to be one of prevention, not detection. Currently, there are no reliable tools available to detect the use of generative AI in producing text. Legacy companies like Turnitin have made limited progress with the detection of AI-generated content (Dave, 2023). However, the technologies that they employ for detection operate probabilistically, with success rates that depend on the percentage of AI-generated content in a text (Chechitelli, 2023). As a result, these systems are quite easy to trick. One need only to paraphrase the text created by an AI model or rearrange the text in minimal ways to make the detection algorithm perform less reliably. Unfortunately, tools that help students to rearrange AI-generated text have already proliferated on the internet.

From an administrative standpoint, the bigger problem is that AI detection tools do not provide educators with forensic evidence, portions of both the original and plagiarized text, to compare in the investigation of cases of cheating. In traditional cases of plagiarism, teachers need only locate the original published text being copied, or claimed as one’s own work, to prove the case. With LLMs, there are no original published texts to discover. Evidence in the traditional sense does not exist. Rather, one must trust a detection algorithm tool to predict (accurately) the probability that a text has been generated by AI. Unlike traditional cases, with LLMs, the only ‘evidence’ available is the prediction of an algorithm. Understandably, university administrators have been squeamish about pursuing investigations of cases of suspected plagiarism using such non-traditional and potentially unreliable evidence. For this reason, AI-based plagiarism cannot be understood within the same paradigm as traditional plagiarism. The nature of the problem has changed. With that change should come the recognition that ‘proof’ in the traditional sense may not be an achievable goal in the investigation and prosecution of plagiarism for some time to come. Thus, our efforts should be directed at prevention, not detection.

I believe our current focus should be on creating a more secure academic environment that makes the unsanctioned use of AI systems a losing strategy for students who wish to cheat. My approach will be to propose four ‘layers of security’ that work together to prevent ‘exploits’ at four levels of vulnerability (Figure 1).

OPEN IN VIEWER Figure 1.

Four Layers of Security that Address Four Levels of Vulnerability.

Pedagogic/Didactic Layer

In December 2022, less than a month after the launch of ChatGPT-3.5, Stephen Marche sounded the alarm about the presumed death of the college essay in The Atlantic (Marche, 2022). The supposed demise of the essay format was predicated on the assertion that ChatGPT could produce college- and graduate-level essays with ease. Students around the world cleverly demonstrated the capabilities of the technology by submitting AI-generated essays for university courses. Not only is the technology undetectable, passing as human authors, the essays it generates are good enough to earn decent grades from seasoned assessors (Eke, 2023). To the casual viewer, it appeared that the structure of higher education was beginning to collapse.

Over the course of the last two years, educators have had many opportunities to use ChatGPT for themselves. We have witnessed the astounding, perhaps even revolutionary, capabilities of AI-supported technology—first in the GPT model itself, but then in the many GPT-supported tools and applications that have come to the market in 2023–2024. As the announcement of Microsoft’s initial investment of $10 billion in OpenAI demonstrates, companies are betting on the idea that humanity’s future will be written with the help of artificial intelligence. Still, this brief interaction with LLMs has shown us that AI-supported technologies are far from perfect replacements for human thinking. In short, while the educational paradigm has begun to shift, it has not shifted completely. Educators find themselves suddenly thrust into a transitional moment.

Perhaps the most complex challenge that educators face as we adapt to the capabilities introduced by AI models relates to how we design assessments. Given the narrow focus of this article, a detailed examination of the future of assessment is not possible. However, we can make a few observations about what ChatGPT and other chat-based AI models do well and what they do not do so well. From these observations, we can begin to make recommendations: (a) About the kinds of assessments educators should avoid using and (b) about how we might approach the task of AI-proofing assessments—or, at the very least, how we can make using AI models to cheat more difficult. We should keep in mind, however, that one can only make general recommendations at this point. As I explain above, the capabilities of LLM-supported AI systems are developing continuously and rapidly. Aspects of the recommendations that one makes today will be outdated in a few months (or less). ⁴

ChatGPT is designed to function as a chatbot. This means that users can carry out extended conversations with the model, adding incrementally to context and meaning in the conversation over time. Conversations take place in a prompt-and-response modality. Most incredibly, ChatGPT-4o retains a working memory of the conversation, making it quite easy to refine both the user’s prompts and the model’s responses as context and meaning accumulate over time. In addition, even older models like ChatGPT-3.5 are quite good at summarizing concepts and providing comparative analyzes (Figure 2). It is this functionality, of course, that creates a dilemma for educators. Users can ask questions about any topic on which the model has been trained (which, at this point, is assumed to be most of humanity’s published knowledge) and receive a response in seconds. ChatGPT-3.5’s answers tend to be short, gravitating towards summary-level explanations; yet they are indistinguishable from human responses in most ways.

OPEN IN VIEWER Figure 2.

Example of ChatGPT-3.5 Responding to a Query with Information, then Performing an Analysis of the Original Information in Relation to a Second, Unrelated Query.

There are several areas in which ChatGPT-3.5 struggles to provide human-like responses; however, these ‘struggles’ originate more in the way that the model is trained than in the computational power of its predictive algorithm. For example, ChatGPT is programmed to avoid forming subjective opinions. If asked to provide an opinion, it will inform the user that this does not fall within the bounds of its training (Figure 3). Because ChatGPT-3.5’s training includes programming that prevents it from forming opinions, it does not ‘think’ with the aim of forming conclusions as humans tend to do. When asked if it thinks about its own thinking, ChatGPT informs the user that LLMs analyze data, but they do not have a subjective experience or conscious awareness of their own processes (Figure 4). They do not form opinions. ⁵ They do not have beliefs. As such, they are functionally incapable of metacognition—and perhaps incapable of critical thinking, if we characterize it as a metacognitively regulated process.

OPEN IN VIEWER Figure 3.

Is ChatGPT Capable of Metacognition?

OPEN IN VIEWER Figure 4.

Example of ChatGPT-3.5 Explaining that It does not have Opinions or Beliefs. I have not Included the Information that It provided about Fascism, but It defined the Concept and gave Examples.

In practice, this means that when asked to evaluate competing ideas, ChatGPT will not offer a strong opinion in support of one side of the issue. Instead, it will tend to give a summary of both ideas then fall back on its human training (RLHF) to explain why it cannot take a stand on the issue (Figure 5). From an educator’s standpoint, this is an interesting limitation to note, since it exposes an entire class of cognitive tasks that AI models cannot do well: Namely any type of task that requires forming and defending an opinion or belief based on critical reflection. We can categorize this class of tasks broadly as metacognitive. So, while ChatGPT can aggregate and analyze massive amounts of data, make interpretations of the data and provide statistical predictions based on observed patterns, its training does not allow it to form or evaluate subjective opinions. It is difficult to predict how long this will remain a limitation of GPT models. We have already observed many ‘jailbreaking’ techniques designed to bypass this training. More problematically, hackers have been jailbreaking entire versions of LLMs and making them available online to anyone who wishes to use them (Mozes et al., 2023). ChatGPT-4 has shown early signs of a potential for emergent general intelligence, but it still lacks the cognitive autonomy characteristic of human intelligence (Bubeck et al., 2023). However, it should be noted that the latest model, OpenAI-o1 (a preview of which was released in September 2024) was designed to ‘reason through complex tasks’ (OpenAI, 2024b) and problems before responding, bringing the model closer to the goal of ‘system 2’ thinking (Kahneman, 2011).

OPEN IN VIEWER Figure 5.

Example of ChatGPT Finding a Way to Avoid Making an Ethical Evaluation.

As educators begin to think about how to design assessments with ‘AI-proofing’ in mind, the most important factors will be to avoid assigning tasks that AI models do well (like summary and comparison) while emphasizing the kinds of tasks that AI models do poorly (tasks that rely heavily on metacognition and critical reflection). Ideally, educators need to be prepared to ask students to engage more in activities that require them to go through the processes involved in forming/defending opinions based on the conformity of a claim with evidence. This is not a new approach, of course. Modern educators have employed forms of assessment that highlight the use of metacognition, critical reflection and scientific thinking for decades. The novel assertion being made here is that many traditional forms of assessment hide significant vulnerabilities in the age of AI. Going forward, when educators design assessments for their courses, we cannot think solely about how well the task supports learning outcomes. We must consider first how easily certain elements of a given task may be completed by AI-supported technology. We must be familiar enough with the capabilities of the technology to identify vulnerabilities in the design of assessments. In the process, we may find that many traditional tasks—especially those that require students to summarize information and/or compare perspectives, but which do not encourage strong engagement with metacognition and/or critical reflection—are now obsolete.

Perhaps the best way to approach the problem is to concentrate on designing tasks that simply make reliance on AI technology a losing strategy. In most cases, this would mean designing hybrid assessments that encourage students to draw from many different knowledge domains, while simultaneously engaging in multiple modes of cognition. An example design would require students to blend knowledge from multiple sources, including from recent academic publications, course lectures and peer-group discussions, while requiring that they do more than simply summarize key concepts. In addition, it would require students to apply critical and metacognitive research strategies as they form new, individual understandings/opinions about a topic.

Clearly, I am not ‘reinventing the wheel’ with this example. My intention is to suggest that educators need to establish a new baseline. In the ‘arms race’ that we suddenly find ourselves, assessments must begin to account for and eliminate known vulnerabilities—those tasks that generative AI does well—while encouraging the use of uniquely human capacities to expose weaknesses in the capabilities of AI-supported technologies.

By requiring that students incorporate knowledge from unpublished sources like the classroom, we ensure that students cannot rely completely on AI to complete a task. Most LLMs, like ChatGPT, make use of publicly available data. They cannot account for unpublished information. Additionally, requiring that students use recent academic publications (as current as possible) presents a challenge for older AI models that are not connected to the internet, like ChatGPT-3.5. ⁶ Lastly, by requiring that students apply critical and metacognitive research strategies to form new, individual opinions about a topic, we exploit a weakness in the programming of AI-supported technologies: An inability to engage in metacognitive processes because of training restrictions.

Therefore, it is important to reemphasize the claim that the current best strategy involves prioritizing hybrid assessment designs that require students to make use of multiple sources of knowledge and multiple modes of cognition. Students may still attempt to use AI models to complete parts of a task, but well-designed hybrid tasks will place natural constraints on that usage while exploiting weaknesses in the current abilities of AI systems. Much more can be said about assessment design, of course; yet, it would be risky to make more specific recommendations at this point. We should not expect many strategies to be relevant in a year’s time (or less). This will be an important consideration for the next layer of security.

Concluding Remarks

If researchers in AI systems are correct, we are living in the initial phase of a major shift in the way that humanity lives. This shift extends well beyond the concerns of education. We must strive to recognize the significance of this moment so we may respond from a place of understanding and caution. In this article, I have tried to give an overview of some of the issues that we are facing as educators, currently living and working through a transition. I have made recommendations about how to address new vulnerabilities in the way that we assess our students. Lastly, I have suggested a strategy for ensuring that we approach the problem systematically. Unfortunately, it is impossible to project how AI systems will develop in the near future. We can be certain, however, that the approaches we adopt will need to be flexible. This is true at every level of the issue. For example, how we classify the sanctioned use of AI will have ethical reverberations for decades to come. If we choose to embrace the use of AI-supported technologies, as we likely will, assessment design will matter more than it ever has. Finally, if we formulate policies from the perspective of old paradigms, ignoring or failing to understand the monumental shift in human development that we are witnessing, we may inadvertently contribute to the weakening of higher education, including the potential loss of necessary skills (Stadler et al., 2024). We find ourselves in a situation where we must think deeply but act quickly.