A Comparison of Prostate Cancer Screening Information Quality on Standard and Advanced Versions of ChatGPT,Google Gemini,and Microsoft Copilot: A Cross-Sectional Study

Abstract

Purpose

Artificially Intelligent (AI) chatbots have the potential to produce information to support shared prostate cancer (PrCA) decision-making. Therefore, our purpose was to evaluate and compare the accuracy, completeness, readability, and credibility of responses from standard and advanced versions of popular chatbots: ChatGPT-3.5, ChatGPT-4.0, Microsoft Copilot, Microsoft Copilot Pro, Google Gemini, and Google Gemini Advanced. We also investigated whether prompting chatbots for low-literacy PrCA information would improve the readability of responses. Lastly, we determined if the responses were appropriate for African-American men, who have the worst PrCA outcomes.

Approach

The study used a cross-sectional approach to examine the quality of responses solicited from chatbots.

Participants

The study did not include human subjects.

Method

Eleven frequently asked PrCA questions, based on resources produced by the Centers for Disease Control and Prevention (CDC) and the American Cancer Society (ACS), were posed to each chatbot twice (once for low literacy populations). A coding/rating form containing questions with key points/answers from the ACS or CDC to facilitate the rating process. Accuracy and completeness were rated dichotomously (i.e., yes/no). Credibility was determined by whether a trustworthy medical or health-related organization was cited. Readability was determined using a Flesch-Kincaid readability score calculator that enabled chatbot responses to be entered individually. Average accuracy, completeness, credibility, and readability percentages or scores were calculated using Excel.

Results

All chatbots were accurate, but the completeness, readability, and credibility of responses varied. Soliciting low-literacy responses significantly improved readability, but sometimes at the detriment of completeness. All chatbots recognized the higher PrCA risk in African-American men and tailored screening recommendations. Microsoft Copilot Pro had the best overall performance on standard screening questions. Microsoft Copilot outperformed other chatbots on responses for low literacy populations.

Conclusions

AI chatbots are useful tools for learning about PrCA screening but should be combined with healthcare provider advice.

Keywords

prostate cancer cancer screening shared decision making artificial intelligence machine learning African American

Get full access to this article

View all access options for this article.

References

Siegel

Giaquinto

Jemal

. Cancer statistics, 2024. CA Cancer J Clin. 2024;74(1):12-49. doi:10.3322/caac.21830

Giaquinto

Miller

Tossas

Winn

Jemal

Siegel

. Cancer statistics for African American/Black people 2022. CA Cancer J Clin. 2022;72(3):202-229. doi:10.3322/caac.21718

Coughlin

. A review of social determinants of prostate cancer risk, stage, and survival. Prostate Int. 2020;8(2):49-54. doi:10.1016/j.prnil.2019.08.001

Johnson

Mavingire

Woods-Burnham

, et al. The complex interplay of modifiable risk factors affecting prostate cancer disparities in African American men. Nat Rev Urol. 2024;40(1):1-11. doi:10.1038/s41585-023-00849-5

Wolf

Wender

Etzioni

, et al. American Cancer Society guideline for the early detection of prostate cancer: update 2010. CA: Cancer J Clin. 2010;60(2):70-98. doi:10.3322/caac.20066

Wei

Barocas

Carlsson

, et al. Early detection of prostate cancer: AUA/SUO guideline part I: prostate cancer screening. J Urol. 2023;210(1):46-53. doi:10.1097/JU.0000000000003491

Wei

Barocas

Carlsson

, et al. Early detection of prostate cancer: AUA/SUO guideline part II: considerations for a prostate biopsy. J Urol. 2023;210(1):54-63. doi:10.1097/JU.0000000000003492

US Preventive Service Task Service . Screening for prostate cancer: US preventive Services task Force recommendation statement. JAMA. 2018;319(18):1901-1913. doi:10.1001/jama.2018.3710

Smith

Andrews

Brooks

, et al. Cancer screening in the United States, 2019: a review of current American Cancer Society guidelines and current issues in cancer screening. CA. Cancer J Clin. 2019;69(3):184-210. doi:10.3322/caac.21557

10.

Ilic

Djulbegovic

Jung

, et al. Prostate cancer screening with prostate-specific antigen (PSA) test: a systematic review and meta-analysis. BMJ. 2018;362:k3519. doi:10.1136/bmj.k3519

11.

Grene

Cleary

Marcus-Quinn

. Use of plain-language guidelines to promote health literacy. IEEE Trans Prof Commun. 2017;60(4):384-400. doi:10.1109/TPC.2017.2761578

12.

Milne-Ives

de Cock

Lim

, et al. The effectiveness of artificial intelligence conversational agents in health care: systematic review. J Med Internet Res. 2020;22(10):e20346. doi:10.2196/20346

13.

Bérubé

Schachner

Keller

, et al. Voice-based conversational agents for the prevention and management of chronic and mental health conditions: systematic literature review. J Med Internet Res. 2021;23(3):e25933. doi:10.2196/25933

14.

Sanders

Chow

. Chatbot for health care and oncology applications using artificial intelligence and machine learning: systematic review. JMIR Cancer. 2021;7(4):e27850. doi:10.2196/27850

15.

Pan

Musheyev

Bockelman

Loeb

Kabarriti

. Assessment of artificial intelligence chatbot responses to top searched queries about cancer. JAMA Oncol. 2023;9(10):1437-1440. doi:10.1001/jamaoncol.2023.2947

16.

Akpan

Kobara

Owolabi

Akpan

Offodile

. Conversational and generative artificial intelligence and human–chatbot interaction in education and research. Int Trans Oper Res. 2024;31(1):1-31. doi:10.1111/itor.13522

17.

OpenAi . How ChatGPT and our foundation models are developed. at. https://help.openai.com/en/articles/7842364-how-chatgpt-and-our-foundation-models-are-developed. Accessed on 16 November 2024.

18.

Giuffrè

Kresevic

You

, et al. Systematic review: the use of large language models as medical chatbots in digestive diseases. Aliment Pharmacol Ther. 2024;60(2):144-166. doi:10.1111/apt.18058

19.

Lahat

Shachar

Avidan

Glicksberg

Klang

. Evaluating the utility of a large language model in answering common patients’ gastrointestinal health-related questions: are we there yet? Diagnostics. 2023;13(11):1950. doi:10.3390/diagnostics13111950

20.

Samaan

Yeo

Rajeev

, et al. Assessing the accuracy of responses by the language model ChatGPT to questions regarding bariatric surgery. Obes Surg. 2023;33(6):1790-1796. doi:10.1007/s11695-023-06603-5

21.

Antaki

Touma

Milad

El-Khoury

Duval

. Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings. Ophthalmol Sci. 2023;3(4):100324. doi:10.1016/j.xops.2023.100324

22.

Dada

Puladi

Kleesiek

Egger

. ChatGPT in healthcare: a taxonomy and systematic review. Comp Methods Programs Biomed. 2024;245:108013. doi:10.1016/j.cmpb.2024.108013

23.

Potapenko

Boberg-Ans

Stormly Hansen

Klefter

van Dijk

Subhi

. Artificial intelligence-based chatbot patient information on common retinal diseases using ChatGPT. Acta Ophthalmol. 2023;101(7):829-831. doi:10.1111/aos.15661

24.

Moshirfar

Altaf

Stoakes

Tuttle

Hoopes

. Artificial intelligence in ophthalmology: a comparative analysis of GPT-3.5, GPT-4, and human expertise in answering StatPearls questions. Cureus. 2023;15(6):e40822. doi:10.7759/cureus.40822

25.

Sarraju

Bruemmer

Van Iterson

Cho

Rodriguez

Laffin

. Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model. JAMA. 2023;329(10):842-844. doi:10.1001/jama.2023.1044

26.

Fetrati

Chan

Orji

. Chatbots for sexual health improvement: a systematic review. Intl J of Human–Comput Interact. 2024;1-23. doi:10.1080/10447318.2024.2376808

27.

Whiles

Bird

Canales

DiBianco

Terry

. Caution! AI bot has entered the patient chat: ChatGPT has limitations in providing accurate urologic healthcare advice. Urol Times. 2023;180:278-284. doi:10.1016/j.urology.2023.07.010

28.

Chervenak

Lieman

Blanco-Breindel

Jindal

. The promise and peril of using a large language model to obtain clinical information: ChatGPT performs strongly as a fertility counseling tool with limitations. Fertil Steril. 2023;120(3):575-583. doi:10.1016/j.fertnstert.2023.05.151

29.

Gianola

Bargeri

Castellini

, et al. Performance of ChatGPT compared to clinical practice guidelines in making informed decisions for lumbosacral radicular pain: a cross-sectional study. J Orthop Sports Phys Ther. 2024;54(3):222-228. doi:10.2519/jospt.2024.12151

30.

Rossettini

Cook

Palese

Pillastrini

Turolla

. Pros and cons of using artificial intelligence chatbots for musculoskeletal rehabilitation management. J Orthop Sports Phys Ther. 2023;53(12):728-734. https://www.jospt.org/doi/10.2519/jospt.2023.12000

31.

Hristidis

Ruggiano

Brown

Ganta

SRR

Stewart

. ChatGPT vs Google for queries related to dementia and other cognitive decline: comparison of results. J Med Internet Res. 2023;25:e48966. doi:10.2196/48966

32.

Johnson

King

Warner

Aneja

Kann

Bylund

. Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information. JNCI Cancer Spectr. 2023;7(2):1-19. doi:10.1093/jncics/pkad015

33.

Sallam

Salim

Ala’a

, et al. ChatGPT output regarding compulsory vaccination and COVID-19 vaccine conspiracy: a descriptive study at the outset of a paradigm shift in online search for information. Cureus. 2023;15(2):e35029. doi:10.7759/cureus.35029

34.

Walker

Ghani

Kuemmerli

, et al. Reliability of medical information provided by ChatGPT: assessment against clinical guidelines and patient information quality instrument. J Med Internet Res. 2023;25:e47479. doi:10.2196/47479

35.

Yeo

Samaan

, et al. Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol. 2023;29(3):721-732. doi:10.3350/cmh.2023.0089

36.

Wang

Wan

, et al. A systematic review of ChatGPT and other conversational large language models in healthcare. medRxiv. 2024. doi:10.1101/2024.04.26.24306390

37.

Coskun

Ocakoglu

Yetemen

Kaygisiz

. Can ChatGPT, an artificial intelligence language model, provide accurate and high-quality patient information on prostate cancer? Urol Times. 2023;180:35-58. doi:10.1016/j.urology.2023.05.040

38.

Zheng

, et al. Appropriateness of prostate cancer prevention and screening recommendations obtained from ChatGPT-4. 2023. doi:10.21203/rs.3.rs-2898778/v1

39.

Zhu

Mou

Chen

. Can the ChatGPT and other large language models with internet-connected database solve the questions and concerns of patient with prostate cancer and help democratize medical knowledge? J Transl Med. 2023;21(1):269. doi:10.1186/s12967-023-04123-5

40.

Alasker

Alsalamah

Alshathri

, et al. Performance of large language models (LLMs) in providing prostate cancer information. BMC Urol. 2024;24(1):177. doi:10.1186/s12894-024-01570-0

41.

Musheyev

Pan

Loeb

Kabarriti

. How well do artificial intelligence chatbots respond to the top search queries about urological malignancies? Euro Urol. 2024;85(1):13-16. doi:10.1016/j.eururo.2023.07.004

42.

Von Elm

Altman

Egger

Pocock

Gøtzsche

Vandenbroucke

. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007;370(9596):1453-1457. doi:10.1136/bmj.39335.541782.AD

43.

Open

. ChatGPT 3.5. 2023 at. https://chatgpt.com/. Accessed on 6 May 2024.

44.

Open

. ChatGPT 4. 2023 at. https://openai.com/index/gpt-4/. Accessed on 6 May 2024.

45.

Microsoft. Microsoft Copilot

2023 at. https://copilot.microsoft.com/. Accessed on May 6, 2024

46.

Microsoft . Microsoft Copilot pro. 2024 at. https://www.microsoft.com/copilot-pro. Accessed on 6 May 2024.

47.

Google. Google Gemini. 2024 at. https://gemini.google.com/app. Accessed on May 6, 2024

48.

Google . Google Gemini advanced. 2024 at. https://gemini.google/advanced/?hl=en. Accessed on 6 May 2024.

49.

American Cancer Society . American cancer society recommendations for prostate cancer early detection. at. https://www.cancer.org/cancer/types/prostate-cancer/detection-diagnosis-staging/acs-recommendations.html. Accessed on 1 May 2023.

50.

Centers for Disease Control and Prevention . Prostate cancer at. https://www.cdc.gov/prostate-cancer/?CDC_AAref_Val=https://www.cdc.gov/cancer/prostate/basic_info/index.htm%2522%2520/l%2520%2522:∼:text=Cancer%2520is%2520a%2520disease%2520in,common%2520cancer%2520in%2520American%2520men%2522. Accessed on 1 May 2024.

51.

Maksimoski

Noble

Smith

. Does Chat GPT answer otolaryngology questions accurately? Laryngoscope. 2024;134(9):4011-4015. doi:10.1002/lary.31410

52.

Albright

de Guzman

Acebo

Paiva

Faulkner

Swanson

. Readability of patient education materials: implications for clinical practice. Appl Nurs Res. 1996;9(3):139-143. doi:10.1016/s0897-1897(96)80254-0

53.

Good Calculators . Flesch Kinkaid calculator at. https://goodcalculators.com/flesch-kincaid-calculator/. Accessed on 1 May 2024.

54.

Kincaid

Fishburne

JRP

Rogers

Chissom

. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Navy Tech Train. 1975;8:1-48.

55.

Flesch

. How to Write Plain English: A Book for Lawyers and Consumers. New York: Harper & Row; 1979.

56.

Weiss

. Health Literacy and Patient Safety: Help Patients Understand. Manual for clinicians: American Medical Association Foundation, 2007.

57.

Kington

Arnesen

Chou

W-YS

, et al. Identifying credible sources of health information in social media: principles and attributes. NAM perspectives. 2021;2021:10. doi:10.31478/202107a

58.

Thia

Saluja

. ChatGPT: is this patient education tool for urological malignancies readable for the general population? Res Rep Urol. 2024;16:31-37. doi:10.2147/RRU.S440633

59.

Dovey

Nair

Chakravarty

Tewari

. Racial disparity in prostate cancer in the African American population with actionable ideas and novel immunotherapies. Cancer Rep (Hoboken). 2021;4(5):e1340. doi:10.1002/cnr2.1340

60.

Hopkins

Logan

Kichenadasse

Sorich

. Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift. JNCI Cancer Spectr. 2023;7(2):pkad010.

61.

van Bussel

Odekerken–Schröder

Swart

Jacobs

. Analyzing the determinants to accept a virtual assistant and use cases among cancer patients: a mixed methods study. BMC Health Serv Res. 2022;22(1):890. doi:10.1186/s12913-022-08263-5

62.

Owens

Leonard

Singh

. Efficacy of Alexa, Google Assistant, and Siri for supporting informed prostate cancer screening decisions for African-American Men. Journal of Cancer Education. 2023;38(5):1752-1759.