AI or Human? Finding and Responding to Artificial Intelligence in Student Work

Abstract

Introduction

Recent innovations in generative artificial intelligence (AI) technologies have led to an educational environment in which human authorship cannot be assumed, thereby posing a significant challenge to upholding academic integrity.

Statement of the problem

Both humans and AI detection technologies have difficulty distinguishing between AI-generated vs. human-authored text. This weakness raises a significant possibility of false positive errors: human-authored writing incorrectly judged as AI-generated.

Literature review

AI detection methodology, whether machine or human-based, is based on writing style characteristics. Empirical evidence demonstrates that AI detection technologies are more sensitive to AI-generated text than human judges, yet a positive finding from these technologies cannot provide absolute certainty of AI plagiarism.

Teaching implications

Given the uncertainty of detecting AI, a forgiving, pro-growth response to AI academic integrity cases is recommended, such as revise and resubmit decisions.

Conclusion

Faculty should cautiously embrace the use of AI detection technologies with the understanding that false positive errors will occasionally occur. This use is ethical provided that the responses to problematic cases are approached with the goal of educational growth rather than punishment.

Keywords

academic integrity plagiarism artificial intelligence generative AI

Get full access to this article

View all access options for this article.

References

Akram

(2023). An empirical study of AI generated text detection tools (2310.01423; Version 1). arXiv. https://doi.org/10.48550/ARXIV.2310.01423

Andrews

(2023). Comparing AI detection tools: One instructor’s experience. Colorado State University - The Institute for Learning and Teaching. https://tilt.colostate.edu/comparing-ai-detection-tools-one-instructors-experience/

Barshay

(2023). It’s easy to fool ChatGPT detectors. Hechinger Report Proof Points. https://hechingerreport.org/proof-points-its-easy-to-fool-chatgpt-detectors/

Chaka

(2023). Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: The case of five AI content detection tools. Journal of Applied Learning & Teaching, 6(2), 94–104. https://doi.org/10.37074/jalt.2023.6.2.12

Chechitelli

(2023). Understanding false positives within our AI writing detection capabilities. Turnitin Blog. https://www.turnitin.com/blog/understanding-false-positives-within-our-ai-writing-detection-capabilities

Clark

August

Serrano

Haduong

Gururangan

Smith

(2021). All that’s ‘human’ Is not gold: Evaluating human evaluation of generated text (2107.00061). arXiv. https://doi.org/10.48550/arXiv.2107.00061

Cotton

D. R. E.

Cotton

P. A.

Shipway

J. R.

(2024). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 61(2), 228–239. https://doi.org/10.1080/14703297.2023.2190148

Desaire

Chua

A. E.

Isom

Jarosova

Hua

(2023). Distinguishing academic science writing from humans or ChatGPT with over 99% accuracy using off-the-shelf machine learning tools. Cell Reports Physical Science, 4(6), 101426. https://doi.org/10.1016/j.xcrp.2023.101426

Dugan

Ippolito

Kirubarajan

Shi

Callison-Burch

(2023). Real or fake text?: Investigating human ability to detect boundaries between human-written and machine-generated text. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11), 12763–12771. https://doi.org/10.1609/aaai.v37i11.26501

10.

Elkhatat

A. M.

Elsaid

Almeer

(2023). Evaluating the efficacy of AI content detection tools in differentiating between human and AI-generated text. International Journal for Educational Integrity, 19(1), 17. https://doi.org/10.1007/s40979-023-00140-5

11.

Fishman

(2009). “We know it when we see it” is not good enough: Toward a standard definition of plagiarism that transcends theft, fraud, and copyright [Conference session]. 4th Asia Pacific Conference on Educational Integrity, University of Wollongong NSW Australia. https://ro.uow.edu.au/cgi/viewcontent.cgi?article=1037&context=apcei

12.

Fowler

(2023). We tested a new ChatGPT-detector for teachers. It flagged an innocent student. Washington Post, F.7.

13.

Gao

C. A.

Howard

F. M.

Markov

N. S.

Dyer

E. C.

Ramesh

Luo

Pearson

A. T.

(2023). Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. Npj Digital Medicine, 6(1), 75. https://doi.org/10.1038/s41746-023-00819-6

14.

Gegg-Harrison

(2023). Against the use of GPTZero and other LLM-detection tools on student writing. Write Rethink. https://writerethink.medium.com/against-the-use-of-gptzero-and-other-llm-detection-tools-on-student-writing-b876b9d1b587

15.

Gilmore

Strickland

Timmerman

Maher

Feldon

(2010). Weeds in the flower garden: An exploration of plagiarism in graduate students’ research proposals and its connection to enculturation, ESL, and contextual factors. International Journal for Educational Integrity, 6(1), 13–28. https://doi.org/10.21913/IJEI.v6i1.673

16.

Hughes

Bae

. 2023. Vectara Hallucination Leaderboard [Data set]. https://github.com/vectara/hallucination-leaderboard.

17.

Khanna

(2023). How AI content detection works (and why it often doesn’t). How-To Geek. https://www.howtogeek.com/how-ai-content-detection-works-and-why-it-doesnt/

18.

Kramm

McKenna

(2023). AI amplifies the tough question: What is higher education really for? Teaching in Higher Education, 28(8), 2173–2178. https://doi.org/10.1080/13562517.2023.2263839

19.

Kreps

S. E.

McCain

Brundage

(2020). All the news that’s fit to fabricate: AI-generated text as a tool of media misinformation. SSRN Electronic Journal, 9(1), 104–117. https://doi.org/10.2139/ssrn.3525002

20.

Kumar

Mindzak

(2024). Who wrote this? Detecting artificial intelligence–generated text from human-written text. Canadian Perspectives on Academic Integrity, 7(1), 1–9. https://doi.org/10.55016/ojs/cpai.v7i1.77675

21.

Lee

(2023). How hard can it be? Testing the reliability of AI detection tools [Paper presentation]. The Future is Now, Higher Education Research Group (HERGA) conference., Flinders University, South Australia. https://www.researchgate.net/publication/376203561_How_hard_can_it_be_Testing_the_dependability_of_AI_detection_tools

22.

Loftus

E. F.

Palmer

J. C.

(1974). Reconstruction of automobile destruction: An example of the interaction between language and memory. Journal of Verbal Learning and Verbal Behavior, 13(5), 585–589. https://doi.org/10.1016/S0022-5371(74)80011-3

23.

Liu

Cheng

Huang

Liu

(2023). AI vs. Human—Differentiation analysis of scientific content generation (2301.10416; Version 2). arXiv. https://doi.org/10.48550/ARXIV.2301.10416

24.

MacDonald

(2023). Dude, where’s my citations? ChatGPT’s hallucination of citations. Mind Pad, 11(1), 21–25. https://cpa.ca/docs/File/Students/MindPad/MindPad_Winter2023.pdf

25.

Macmillan

N. A.

Creelman

C. D.

(2005). Detection theory: A user’s guide (2nd ed.). Lawrence Erlbaum Associates, Inc.

26.

Moriarty

(2024). The LLM misinformation problem I was not expecting. RIPE Labs. https://labs.ripe.net/author/kathleen_moriarty/the-llm-misinformation-problem-i-was-not-expecting/

27.

Neugebauer

(2023). Understanding LLM hallucinations: How LLMs can make stuff up and what to do about it. Towards Data Science. https://towardsdatascience.com/llm-hallucinations-ec831dcd7786

28.

Nolan

(2023). Two professors who say they caught students cheating on essays with ChatGPT explain why plagiarism can be hard to prove. Business Insider. https://www.businessinsider.com/chatgpt-essays-college-cheating-professors-caught-students-ai-plagiarism-2023-1

29.

Rashidi

H. H.

Fennell

B. D.

Albahra

Gorbett

(2023). The ChatGPT conundrum: Human-generated scientific manuscripts misidentified as AI creations by AI text detection tool. Journal of Pathology Informatics, 14, 1–6. https://doi.org/10.1016/j.jpi.2023.100342

30.

Richards

Waugh

Slaymaker

Petre

Woodthorpe

Gooch

(2024). Bob or bot: Exploring ChatGPT’s answers to university computer science assessment. ACM Transactions on Computing Education, 24(1), 1–32. https://doi.org/10.1145/3633287

31.

Rivas

(2023). Is anyone else noticing overly poetic language in papers these days? “Rich tapestry” this, “symphony” that. I’m assuming this come [Facebook post]. Society for the Teaching of Psychology. https://www.facebook.com/groups/33366519348/search/?q=symphony

32.

Rogers

(2023). How to detect AI-generated text, according to researchers. WIRED. https://www.wired.com/story/how-to-spot-generative-ai-text-chatgpt/

33.

Roig

(1997). Can undergraduate students determine whether text has been plagiarized? The Psychological Record, 47(1), 113–122. https://doi.org/10.1007/BF03395215

34.

Roig

(2001). Plagiarism and paraphrasing criteria of college and university professors. Ethics & Behavior, 11(3), 307–323. https://doi.org/10.1207/S15327019EB1103_8

35.

Rothman

(2023). Metamorphosis: Why the godfather of AI fears what he’s built. The New Yorker, 99(38), 28–39.

36.

Shrivastava

Levine

(2024). Did you use ChatGPT on your school applications? These words may tip off admissions [News]. Forbes. https://www.forbes.com/sites/rashishrivastava/2024/02/05/chatgpt-college-school-applications-admissions-red-flags-ai/

37.

Walters

W. H.

(2023). The effectiveness of software designed to detect AI-generated writing: A comparison of 16 AI text detectors. Open Information Science, 7(1), 20220158. https://doi.org/10.1515/opis-2022-0158

38.

Weber-Wulff

Anohina-Naumeca

Bjelobaba

Foltýnek

Guerrero-Dib

Popoola

Šigut

Waddington

(2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19(1), 26. https://doi.org/10.1007/s40979-023-00146-z

39.

Williams

(2024). The racism behind chatGPT we are not talking about…. This year, I learned that students use chatGPT because they believe it … Kolektiva.Social. https://fediscience.org/@FractalEcho@kolektiva.social/111949021080030890