Evaluating the Use of Artificial Intelligence as a Study Tool for Preclinical Medical School Exams

Introduction

Medical education resources have evolved over time from written textbooks to digital articles and now include innovative study tools to enhance students’ ability to retain vast amounts of information within limited timeframes. Some of these study resources include UptoDate, Pubmed, and Google. ¹ Several studies have been conducted that showed medical students preferred these digital learning modalities to traditional textbooks. In 2002, a survey at the University of Iowa Roy J. and Lucille A. Carver College of Medicine found that 85% of the medical students who responded selected electronic sources as their preferred method of study materials. ² Another survey found that 76% of medical students who were surveyed during their clinical pediatric rotation used digital resources and 30% did not use their textbook for studying at all. ³

In recent years, artificial intelligence (AI) has become a valuable resource for medical students and healthcare professionals to access information and develop a deeper understanding of subject matter. AI was first implemented in general education in the 1920s as a form of self-tutoring, but modern advancements in technology have made it into what we know and see today, 100 years later. ⁴ AI models were introduced into medical education training in the 1980s with the development of CIRCSIM, an AI program designed to teach medical students about cardiovascular physiology.^4,5 Since then, AI has become much more readily accessible via programs like ChatGPT and several studies were conducted regarding its role in medical education. One study discussed situations where using ChatGPT was effective for medical school studies including brainstorming of differential diagnoses, interactive medical practice cases, and multiple choice question review. The same study also showed that AI is limited in its ability to simulate human cognition or intuition. AI was ineffective at tasks where it was asked to use humanistic elements of medical practice learned in both the preclinical and clinical setting, such as interpret sensory and nonverbal cues and build rapport and interpersonal relationships. ⁶ Another study evaluated the performance of ChatGPT on questions used in preparation for the United States Medical Licensing Examination (USMLE) Steps 1 and 2. The study found that ChatGPT is capable of correctly answering enough questions correctly to meet the benchmark passing standards for both Step licensing examinations, indicating its ability to synthesize medical information at the level expected from a third-year medical student. ⁷ This study suggests that AI models could potentially be used as a tutor because of its performance and ability to provide explanations that were “coherent in terms of logical reasoning, use of internal information, and use of external information.” ⁷ As AI develops and new resources are created, physicians who utilize these tools can expect to experience improvements in productivity, thereby limiting their time spent placing orders, writing notes, or precharting, and increasing their face-time with patients. As these effects continue to occur with the development of AI tools to be used in medicine, so too might the relationships of physicians and patients.

While this area of interest has great potential, it is not without its limitations and concerns. In a survey of 3018 medical students, over 85% of respondents agreed that AI resources would improve access to healthcare-related knowledge as doctors and just over 70% of respondents felt AI would lead to a reduction in medical errors. ⁸ Other surveys of medical students demonstrate a concern for AI usage given a lack of formal training on the domain. ⁹ Lack of exposure to this field of development could lead to inappropriate usage of these tools or potentially dangerous decisions by clinicians due to inaccurate or incomplete information. These rising concerns have prompted scholars to advocate for formal training and creations of curricula that emphasize ethical considerations and other themes regarding AI usage in healthcare. Specifically, a review of articles from 2000 to 2020 identified five recurring themes that should be addressed when including AI in medical education. ¹⁰ These themes included a formal training on AI proficiency and ethical considerations as well as emphasizing training on basic people and professional skills. ¹¹ Similarly, another literature review emphasized the call for a “humanistic approach to patient safety” and “early exposure to patient-oriented integration.” ¹² It is clear that there is great concern that AI carries the risk of dehumanizing physicians and medical students and, if integrated, should be done so in a manner that considers the aforementioned curricula themes.

AI focused curriculum development appears to be imminent. One article cited that 94% of the 486 respondents from 17 different medical schools in Canada recognized that AI use in medical education was inevitable. ¹⁰ This impending matter has led scholars to question how AI should be introduced in medical training and what are the ethical considerations of doing so. Some researchers propose the “early stages of a preclerkship curriculum represent an ideal opportunity” to use AI programs to teach early medical students basic science topics and apply them to real-world care. ¹² It is proposed that early exposure to AI tools with an emphasis on ethical usage in the medical setting will assist students in developing the skills needed to utilize AI safely and efficiently as physicians. Harvard Medical School has already taken steps to provide a curriculum for its students that includes a 1 month introductory course on AI in healthcare and expects that the most “successful physicians and researchers will be the ones who can harness genAI for innovation and strategic planning.” ¹³

As scholars advocate for early and formal education on AI resources used in the medical field, it would be valuable to know if there is a quantifiable benefit to those receiving this education. As mentioned previously, medical students are already using AI resources in their studies due to a perceived beneficial relationship, ie, several resources are free and easy to use. What is not well known, is if students who use these AI resources perform better on their preclinical exams than those who do not. The purpose of our study is to identify if there is a correlation between AI usage and preclinical exam performance in first fast medical students. We also want to identify the specific AI tools the subjects use to study and how they use those tools.

Methods

Results

We distributed the survey to the KKSOM class of 2027. Of the 68 total students in the cohort, there were 38 responses (56% response rate) leading to a study population of 38 subjects. Of these 38 students, there were 23 subjects (60.5%) who used AI to study and 15 subjects (39.5%) who did not use AI to study.

The average results on examinations between each study group are reported in Table 1. The yAI group averaged 82.4% over the six exams while the nAI group averaged 82.7%. Each individual exam followed a similar trend with average scores ranging from 80.2% to 85.0%.

OPEN IN VIEWER

Table 1.

Average exam results and standard deviations of the yAI and nAI groups for each organ block NBME exam with associated P-values and the average of all seven exams.

EXAM	yAI RAW SCORE (%)(n = 23)	nAI RAW SCORES (%)(n = 15)	P-VALUE	yAI RANGE (%)	nAI RANGE (%)
Pulmonary	80.7 ± 10.3	80.2 ± 8.9	0.88	48–95	67–92
Cardiovascular	83.4 ± 8.2	85.0 ± 5.2	0.48	58–95	77–97
Renal	82.4 ± 6.4	82.1 ± 6.5	0.91	70–95	73–92
Gastrointestinal	81.8 ± 6.1	81.8 ± 7.4	0.99	67–95	62–93
Endocrine	82.0 ± 9.2	81.08 ± 4.4	0.66	55–93	72–90
Reproductive	82.9 ± 5.1	82.38 ± 7.1	0.81	73–92	70–93
Comprehensive final	83.4 ± 5.9	82.69 ± 2.8	0.60	73–94	77–87
7 Exam group average	82.4	82.7

Of those who stated that they use AI to study, the survey asked participants which AI tools they use. At the time of the survey, 22 of the 23 subjects (95.7%) used ChatGPT 3.5 or 4 as a study tool. Of the subjects in the yAI group 6 (26.1%) used more than one AI tool to study. The final results are tabulated in Table 2.

OPEN IN VIEWER

Table 2.

Total number of users for each AI tool including ChatGPT, THI.AI, and other from the yAI group (n = 23).

AI RESOURCE	ChatGPT	THI.AI	OTHER	COMBINATION OF TOOLS
Number of users	22 (95.7%)	6 (26.1%)	1 (4.3%)	6 (26.1%)

Those in the yAI group were also asked how they use AI to study. Responses were grouped into the following general categories: clarification and simplification of new or confusing material, summarization and content breakdown, practice questions and information application, clinical context and differentiation of confusing material, study guide creation. Many participants used AI in more than one of the ways above to study. Table 3 shows the breakdown of the way participants use AI to study. All responses are shown in Appendix B.

OPEN IN VIEWER

Table 3.

Total number of yAI participant responses to how they use AI grouped into the following categories: clarification and simplification of new or confusing material, summarization and content breakdown, practice questions and information application, clinical context and differentiation of confusing material, and study guide creation.

HOW TO USE AI	CLARIFICATION AND SIMPLIFICATION	SUMMARIZING AND CONTENT BREAKDOWN	PRACTICE AND APPLICATION	CLINICAL CONTEXT AND DIFFERENTIATION	STUDY GUIDE CREATION
Number of users(n = 23)	15	8	3	3	2

Discussion

The purpose of our work was to evaluate the extent to which medical students at a single institution use AI programs to study and whether AI usage correlated to higher exam performance in their preclinical years. Our results showed that there was not a statistically significant difference in exam performance between the two groups on all 7 exams and the overall average of all 7 exams. As previously mentioned, there is very little literature regarding AI usage and exam performance in medical school. Therefore, the results of our study are a unique addition to this field of research.

Much of the literature regarding AI and educational application discusses theoretical benefits as well as more tangible benefits such as saving student time or tailoring their education to meet their individual needs.^15–17 Currently, there is little information on whether using AI actually correlates to higher exam performance. A 2023 literature review of 92 articles regarding AI usage in education reported that approximately 27 of those articles commented on how AI improves students’ academic success in one form or another. ¹⁸ One study, for example, found that AI programs improved students’ speaking skills compared to a control group of human interactions. ¹⁹ Interestingly, another study stated that students who used generative AI programs to write essays, on average, scored over 6% lower than those who did not use these programs. ²⁰ These studies’ results highlighted a negative correlation between AI usage and academic performance. Though important in their implications, these articles did not address a direct correlation between students’ AI usage and exam performance. Previous studies have shown that the introduction of new technologies for studying may not change or worsen school performance when it is first introduced, but as students become more familiar, these novel study methods can improve performance.^21,22 The introduction of AI into studying may follow a similar pattern requiring more longitudinal studies following student use of AI and their performance in academic settings.

The limited research and conflicting findings have created a gap in knowledge that our study hopes to address. The results of this study ultimately indicate that using AI tools to study yields similar results as those who do not use AI. These findings are important as medical students spend significant amounts of money on third-party resources and several of the AI programs that students admitted to using, such as those found in Table 2, are free programs.

There were a number of limitations to our research. First, this research was confined to a 66-student cohort of a single institution and relied on responses to a survey to determine the number of participants leading to a limited sample size that may not represent the population decreasing the study's external validity. Additionally, data collection with a survey introduces many limitations on its own. The self-selection of the participants introduces selection bias which may lead to respondents having different characteristics compared to those who chose not to participate. Furthermore, there are a number of confounding variables that may have affected the average scores within the two groups. Baseline academic performance, preferred study habits, and total time spent studying were not accounted for in this study. In addition, medical education resources are abundant and students typically rely on more than one resource to acquire knowledge in the same manner as those who do not use AI tools. Finally, the brief duration of the study restricts the ability to assess the long-term impacts of AI usage on learning, knowledge retention, and academic performance.

Conclusion

Medical education is a rapidly evolving and often expensive field for medical students to navigate. Advancements of AI improved students’ access to information without the higher costs associated with other third-party resources. Our study shows that students who utilized AI tools to study for subject exams during the first phase of medical school, performed just as well as those who did not use AI. Future research should continue to investigate the tangible benefits of AI tools to medical students throughout their educational experience including basic medical sciences and clerkship rotations.

Supplemental Material