Abstract
Introduction
Artificial intelligence continues to play an increasingly important role in modern health care. ChatGPT-3.5 (OpenAI, San Francisco, CA) has gained attention for its potential impact in this domain.
Objective
To explore the role of ChatGPT-3.5 in guiding clinical decision-making specifically in the context of pancreatic adenocarcinoma and to assess its growth over a period of time.
Participants
We reviewed the National Comprehensive Cancer Network® (NCCN) Clinical Practice Guidelines for the Management of Pancreatic Adenocarcinoma and formulated a complex clinical question for each decision-making page. ChatGPT-3.5 was queried in a reproducible fashion. We scored answers on the following Likert scale: 5) Correct; 4) Correct, with missing information requiring clarification; 3) Correct, but unable to complete answer; 2) Partially incorrect; 1) Absolutely incorrect. We repeated this protocol at 3-months. Score frequencies were compared, and subgroup analysis was conducted on Correctness (defined as scores 1-2 vs 3-5) and Accuracy (scores 1-3 vs 4-5).
Results
In total, 50-pages of the NCCN Guidelines® were analyzed, generating 50 complex clinical questions. On subgroup analysis, the percentage of Acceptable answers improved from 60% to 76%. The score improvement was statistically significant (Mann-Whitney U-test; Mean Rank = 44.52 vs 56.48, P = .027).
Conclusion
ChatGPT-3.5 represents an interesting but limited tool for assistance in clinical decision-making. We demonstrate that the platform evolved, and its responses to our standardized questions improved over a relatively short period (3-months). Future research is needed to determine the validity of this tool for this clinical application.
Get full access to this article
View all access options for this article.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
