Patient Perceptions of Electronic Patient Portal Artificial Intelligence Use at an Urban Medical Center

Abstract

Background:

Artificial intelligence (AI), particularly large language models (LLMs), has shown potential in health care, including diagnostic assistance and patient education. However, concerns about accuracy, biases, and the loss of human interaction, especially in oncology care, warrant investigation into patient perceptions of AI tools.

Methods:

A survey was conducted among 276 oncology outpatients at Thomas Jefferson University to assess comfort, trust, and familiarity with AI chatbots in three clinical scenarios: medication refills, lab result reviews, and preoperative instructions. Participants rated comfort levels using a 5-point Likert scale, and qualitative responses were analyzed. Demographic data were collected to examine subgroup differences. Statistical analyses included Wilcoxon tests with Bonferroni corrections.

Results:

Patients were most comfortable using AI for routine tasks like medication refills compared to lab result reviews (p = 0.0003) and preoperative instructions (p = 0.003). White and Asian patients reported the highest comfort levels, while African American/Black patients expressed significantly less comfort in some contexts, such as preoperative instructions (p = 0.04). Trust in AI was generally higher among male, older, and more educated patients, although familiarity with LLMs did not significantly influence comfort. Less than 10% of participants were highly comfortable using AI alone, citing concerns about losing the human connection in care. Respondents emphasized transparency and the option to interact with a human as critical for building trust.

Conclusions:

This study highlights the importance of patient-centered approaches to integrating AI in oncology care. Tailored strategies addressing trust, transparency, and cultural sensitivity are essential for equitable AI adoption. Future research should explore ways to enhance patient acceptance and mitigate disparities in AI use to improve health care delivery while preserving human interaction.

Introduction

Artificial intelligence (AI) is defined as the ability of computers to perform tasks typically associated with human intelligence.¹ This rapidly evolving technology has the potential to revolutionize medicine and significantly impact the lives of millions of patients internationally. Large language models (LLMs), a subset of AI, have shown promise in various health care applications. These include providing diagnostic assistance, reducing clinician workload with medical note writing, and improving patient education by engaging in human-like conversations through chatbots.^2,3 Despite these advancements, significant concerns remain regarding LLMs’ validity and ethical implications in patient-facing roles, particularly in oncology care.

One of the primary challenges with LLMs is their tendency to “hallucinate” or generate information with a high degree of confidence that may be inaccurate, as they are probabilistic models that generate text based on learned patterns in their training data, rather than on factual knowledge or understanding. They can sometimes fill in gaps or generate incorrect information, even when presented with accurate context, because they are not perfect at distinguishing between true and false information.⁴ This may be due to overreliance on repetition in training data, internal conflicts from large amounts of training information that may be contradictory, and the presence of outdated or false information. Additionally, these models have demonstrated biases related to gender, race, and age, raising questions about their reliability in diverse patient populations.^5,6 As health care systems increasingly consider integrating LLMs into clinical workflows, mainly through electronic health records (EHRs), evaluating how patients perceive these technologies is critical.

Understanding patient perspectives is particularly important in oncology, where communication and trust are paramount. Literature suggests that disadvantaged populations, including non-White, low socioeconomic status, older, and rural-dwelling individuals, have historically been slower to adopt and benefit from new technologies, often due to systemic barriers and historical mistrust of medical institutions that have been notable in black populations who experience structural racism.^7–9 These disparities are exacerbated by implicit biases in accessing and using health care technology, such as EHR systems.¹⁰ Previous work has shown that non-White patients were significantly less likely to receive EHR engagement from health care providers.¹¹ Studies and surveys of nonmedical applications of AI have demonstrated that the public views nonmedical AI variably, with few existing studies on AI in health care focusing on broad applications rather than patient-specific contexts, leaving critical gaps in understanding patient perceptions.^6,12–14

These factors highlight the need for patient-centered research on the acceptability of AI-based tools in health care. This study aims to explore oncology patients’ perceptions of AI chatbots as clinical support tools within the EHR. By assessing comfort, trust, and familiarity with AI in different clinical scenarios and determining patient characteristics associated with those factors, this research provides insights into the factors influencing patient acceptance of these emerging technologies.

Methods

Recruitment and survey design

Between October 31, 2023, and December 19, 2023, oncology patients were recruited through the Regional Liaison Office (RLO) at Thomas Jefferson University to participate in an IRB-approved study (iRISID-2023-2216).

The survey was adapted from a previously reported survey that assessed general perceptions of generative AI¹⁷ and included questions related to the medical field to measure patients’ perceptions of AI use in health care specifically.

The survey assessed patient comfort, trust, and familiarity with AI-based chatbots in clinical scenarios within the EHR. Participants rated their comfort levels with three configurations: a provider alone, a provider assisted by AI, and AI alone (Supplementary Data). Scenarios included refilling medications, reviewing lab results, and reading preoperative instructions. Responses were measured using a 5-point Likert scale (1 = extremely comfortable/trust to 5 = extremely uncomfortable/distrust). To evaluate trust, participants were asked whether they would “completely trust,” “somewhat trust,” “neither trust nor distrust,” or “not trust at all” an AI chatbot alone to communicate about their health care in the EHR. Familiarity with AI was assessed through two questions: (1) “How familiar are you with AI-based LLMs?” with options ranging from “very familiar” to “not familiar at all,” and (2) “What do you think LLM-based chatbots are?” Participants were classified as having correct or incorrect knowledge based on whether they identified LLMs as a type of AI. Qualitative responses were also collected, asking participants to describe ways to improve trust in AI technologies with an open-ended question. Demographic information such as age, gender, race, ethnicity, education level, and comorbidities was also collected for subgroup analysis.

Data and statistical analysis

Mean and median responses to comfort level using an AI chatbot alone in the three scenarios were calculated for each ethnic, racial, gender, age, education, and medical (cancer or cardiovascular disease) group. Wilcoxon pairwise single-rank tests with Bonferroni corrections were used to evaluate group response differences. Pairing Wilcoxon tests evaluated the response difference among the three scenarios. To assess whether most patients would be more or less comfortable using an AI chatbot alone if it improved the accuracy of their treatment plan, patient responses were recoded so one indicated “more comfortable using AI-powered chatbot technology” and −1 indicated “less comfortable using AI-powered chatbot technology.” A Wilcoxon test was then applied to this outcome. p-Values <0.05 were considered significant. All statistical analyses were carried out using R 4.3.1.¹⁵

Results

In total, 276 oncology outpatients participated in the survey. Table 1 summarizes the patients’ characteristics. Most patients were older than 41 years and were White. There were slightly more females than males, and most patients had a bachelor’s degree or higher.

Table 1.

Patient Characteristics

Characteristic	N = 276
Gender
Female	132 (56%)
Male	98 (42%)
Other (please specify)	3 (1.3%)
Prefer not to say	2 (0.9%)
Race
Asian	11 (4.9%)
Black or African American	21 (9.4%)
Other	8 (3.6%)
Prefer not to say	7 (3.1%)
White	177 (79%)
Hispanic/Latino
Yes	25 (11%)
No	199 (72.1%)
Age
18–40	7 (3.1%)
41–60	56 (25%)
61–70	85 (38%)
71+	76 (34%)
Education
High school or equivalent	28 (13%)
Some college or associate degree	62 (28%)
Bachelor’s degree	62 (28%)
Graduate degree	52 (23%)
Professional degree	15 (6.7%)
Other (please specify)	5 (2.2%)

Patient’s comfort with AI use alone in different use cases

Overall, patients were more comfortable with an AI chatbot alone writing the response for a medication refill than when reviewing lab results (p = 0.0003). They were also more comfortable with an AI chatbot alone writing the response for preoperative instructions than when reviewing lab results (p = 0.003). Table 2 shows the mean and median responses for comfort in using an AI chatbot alone in all three scenarios.

Table 2.

Patient Reported Comfort with AI for Assisting with Specific Tasks

How comfortable would you be with an AI chatbot alone responding to your request when…
	Refilling a medication?	Reviewing lab results?	Reading preoperative instructions?
Characteristic	Mean	Mean	Mean
Total (n = 276)	3.56	3.82	3.62
Ethnicity
Hispanic	3.46	3.74	3.61
Non-Hispanic (n = 199)	3.58	3.83	3.62
Race
Asian (n = 11)	3.1	3.56	3.33
Black/African American (n = 21)	4.16	4.22	4.28
Other (n = 8)	3.43	3.43	3.43
Prefer not to say (n = 7)	3.86	4.29	4.43
White (n = 177)	3.52	3.79	3.53
Gender
Female (n = 132)	3.65	3.98	3.65
Male (n = 98)	3.4	3.56	3.52
Other (n = 3)	5	5	5
Prefer not to say (n = 2)	5	5	5
Age
18–40 (n = 7)	4	3.8	4
41–60 (n = 56)	3.58	4.02	3.7
61–70 (n = 85)	3.55	3.77	3.56
71+ (n = 76)	3.53	3.75	3.59
Education
High school or equivalent (n = 28)	3.71	3.93	3.64
Some college or associate degree (n = 62)	3.68	3.9	3.81
Bachelor’s degree (n = 62)	3.57	3.83	3.56
Graduate degree (n = 52)	3.39	3.68	3.49
Professional degree (n = 15)	3.47	4	3.57
Other (n = 5)	3.33	2.67	3
Current medical condition
Cardiovascular disease (n = 35)	3.62	3.82	3.53
Cancer (n = 89)	3.62	3.84	3.67

Key: 1 = extremely comfortable, 2 = somewhat comfortable, 3 = neither comfortable nor uncomfortable, 4 = somewhat uncomfortable, 5 = extremely uncomfortable.

AI, artificial intelligence.

Racial and ethnic group differences in comfort and trust

Table 2 shows differences in comfort levels among races and ethnic groups when using an AI chatbot alone to refill a medication, review lab results, and read preoperative instructions. General trends show that White and Asian patients tend to be the most comfortable with AI chatbots, followed by African American/Black patients. However, it must be noted that there are small sample sizes for Asian patients (n = 11, 4.9%) and African American/Black patients (n = 2.1, 9.4%), leading to limited generalizability. This trend in overall trust in health care communication with AI chatbots alone across racial groups is illustrated in Figure 1. More specifically, in the category of medication refilling with an AI chatbot alone, African American/Black patients were not significantly less comfortable than White patients (p = 0.052). However, for reading preoperative instructions with an AI chatbot alone, African American/Black patients were considerably less comfortable than White patients (p = 0.04). There are no significant differences between patient races in comfort using an AI chatbot alone when reviewing lab results. When asked how comfortable they would feel if a provider used an AI chatbot to communicate empathetically, Asian patients were significantly more comfortable than White patients (p = 0.039) and African American/Black patients (p = 0.04). There was no significant difference in comfort between White patients and African American/Black patients (p = 1). When asked how they felt about a provider using an AI chatbot to help them communicate empathetically with patients or in any of the three case scenarios, there was no significant difference in comfort level between Hispanic and non-Hispanic patients. There was no significant difference between genders, age, and education level, although there were trends toward higher trust in male patients, older patients, and more highly educated patients.

Fig. 1.

“How much do you trust communication about your health care in the electronic patient portal with an AI chatbot alone?” Key: 1 = extremely comfortable, 2 = somewhat comfortable, 3 = neither comfortable nor uncomfortable, 4 = somewhat uncomfortable, 5 = extremely uncomfortable. AI, artificial intelligence.

Patient perceptions of the use of an AI chatbot in improving the accuracy of treatment plans

Participants were asked about their comfort with using AI chatbots and whether they improved the accuracy of their treatment plans. Results show 45% feeling more comfortable, 29% feeling neither more nor less comfortable, and 26% feeling less comfortable. Patients felt significantly more comfortable using an AI chatbot if it improved the accuracy of their treatment plan (p = 0.001). There was no difference in response between White and non-White racial groups (p = 0.307) and no difference in response between males and females (p = 0.434).

Comfort with different involvements of AI in patient–provider communication

Table 3 shows the mean and median responses for comfort level when using a provider alone, provider and AI, and AI only when refilling medication, reviewing lab results, and reading preoperative instructions. When making medication refill requests, reviewing lab results, and reading preoperative instructions, patients were more comfortable communicating with providers only compared with both providers and AI (p < 0.001) and AI only (p < 0.001).

Table 3.

Patient Comfort Levels in Using a Provider Versus a Provider and AI Versus AI Only in Three Different Scenarios

		Provider	Provider and AI	AI only	Provider vs. provider and AI	Provider vs. AI only
	n	Mean	Mean	Mean	Wilcoxon p-value	Wilcoxon p-value
Refill medication	214	1.33	2.49	3.56	<0.001	<0.001
Lab results	210	1.26	2.5	3.82	<0.001	<0.001
Preoperation instructions	208	1.21	2.38	3.62	<0.001	<0.001

Key: 1 = extremely comfortable, 2 = somewhat comfortable, 3 = neither comfortable nor uncomfortable, 4 = somewhat uncomfortable, 5 = extremely uncomfortable.

Differences in patient perceptions of an AI chatbot based on familiarity

Overall, patients’ familiarity with and knowledge of LLMs did not affect their overall perceptions of their use within a medical context. Patients who were familiar or somewhat familiar with LLMs and had correct knowledge of them (Group 1) were not significantly more comfortable with the use of AI alone when responding to a request to refill a medication than those who were unfamiliar with LLMs (Group 3, p = 0.129) or than those who are familiar/somewhat familiar with them but have incorrect knowledge of them (Group 2, p = 0.187). Patients who were familiar/somewhat familiar with LLMs and had a correct understanding of them were not significantly more comfortable with the use of AI alone when reviewing lab tests than those who were familiar/somewhat familiar with them but had incorrect knowledge (p = 0.073) or than those who were unfamiliar with LLM’s (p = 0.321). The groups also did not differ in their comfort level when using AI alone to read preoperation instructions.

Qualitative findings

The most common concern in response to the open-ended question was the loss of the personal aspect of health care, specifically communicating with other humans (15/43, 34.88%). A large portion of the sample stated that they would never trust AI in health care (14/43, 32.56%), followed by concerns about communication/human oversight (5/43, 11.63%), bias (5/43, 11.63%), and current immaturity of the technology (3/43, 6.98%) These findings are described in Table 4.

Table 4.

Participant Attitudes and Concerns Regarding AI Usage in Health Care (n = 43)

Attitude/concerns regarding AI	Frequency n (%)	Exemplar quote
Human factor	15 (34.88)	“The relationship between the patient and the provider should be a human to human relationship. I do not want to see my condition being relegated to a chat bot.”
Never trust	14 (32.56)	“I’m not at all comfortable with the use of AI in my healthcare plan.”
Communication/human oversight	5 (11.63)	“Be forthcoming about the degree of AI use in making comments so patient could specifically request person to person communication.”
Bias	5 (11.63)	“It has been proven that AI is biased against women and minorities. I have zero trust until AI is developed with an equal number of women and minorities.”
Immaturity of technology	3 (6.98)	“At this point in time, AI does not have the ability to have immediate up-to-date knowledge and that makes me uncomfortable.”

Discussion

This study provides insights into oncology patients’ perceptions of AI-powered chatbots in the electronic patient portal (EHR). For the first time, we explored patient comfort, trust, and familiarity with AI across three clinical scenarios: refilling medications, reviewing lab results, and reading preoperative instructions. The findings are significant because they highlight the potential and challenges of integrating AI tools into patient care. Patients expressed the highest comfort levels with AI for routine tasks such as medication refills, and the lowest comfort levels with AI for more complex activities, such as reviewing lab results. Racial and demographic differences were observed, with White patients generally reporting the highest comfort levels and African American/Black patients expressing less comfort in specific contexts, such as preoperative instructions. These findings, combined with the fact that less than 10% of participants felt highly comfortable using AI alone in any scenario, underscore the need for cautious and thoughtful implementation of these technologies.

A compelling finding was the respondents’ concern that AI chatbots might lose health care’s personal and human aspects. The oncology patients emphasized the importance of trust and empathy in patient–provider relationships. They voiced concern that AI may undermine these relationships by automating previously human-to-human interactions, with some stating they would “never trust” AI in health care. This is congruent with prior findings showing the importance of human connection in medicine.¹⁶ However, among actionable concerns, communication and transparency were emphasized by patients, suggesting that medical centers hoping to implement these technologies should be forthright when such technologies are being used and always provide an option to interact with a human. These findings align with prior research showing that trust and clear communication are paramount to patient acceptance of AI technologies.^15,16

These findings reflect broader patterns of historic mistrust in health care among specific populations, which may influence their comfort and trust in AI tools. While it is well established that historical patterns of unethical medical practices have contributed to persistent mistrust in medical systems among minority groups,^7–9 it is unclear how this mistrust extends to new technologies. This underscores the importance of addressing these systemic issues when integrating AI into patient care. A previous study has even shown that LLM-generated clinical recommendations vary depending on demographics, not just clinical information.¹⁸ Efforts to improve trust should include culturally sensitive approaches, transparency in AI use, and active engagement with diverse communities to ensure that AI technologies are perceived as equitable and trustworthy. Culturally sensitive approaches can manifest in representative data, consideration of social determinants of health, and improving the interface to allow for interactions that consider factors such as different beliefs and communication styles. Diverse groups can be recruited to aid in the design of these technologies to ensure that the technology is more inclusive.

This study has several limitations. The use of convenience sampling at a single urban medical center limits the generalizability of the findings, particularly to rural or underserved populations. Additionally, small sample sizes for specific racial and ethnic groups, such as Black and Asian patients, restrict the interpretability of subgroup analyses. While efficient and palatable to respondents because of the anonymity, the reliance on surveys may have oversimplified complex perceptions of AI. Given the complexity of the topic and the technology, the extent to which each participant fully appreciated our desired meaning of the questions may have varied. Future research should include more representative samples and use additional qualitative methods like focus groups and targeted interviews to better understand patient perceptions and concerns. Furthermore, the bias in AI systems must be mitigated to ensure that this tool can be used in patient populations without exacerbating health care inequities. This can be done by further training of the AI models.

This study highlights the need for patient-centered approaches to integrating AI into health care. The diverse patient perspectives emphasize the importance of transparency, accuracy, and patient autonomy so that AI technologies can be developed to address concerns about trust and equity. This study underscores the importance of designing AI tools that complement, rather than replace, human providers, ensuring that they enhance communication, build trust, and ultimately improve patient outcomes. Future work should explore how to effectively tailor AI tools to diverse patient populations to improve acceptance and use. A longitudinal study would be helpful to see the evolution of perceptions as patients are exposed to AI technologies. By addressing these challenges, AI has the potential to transform health care delivery while preserving the human connection that is fundamental to patient care.

Footnotes

Acknowledgments

The team would like to thank the eager and forward-thinking mentors at Thomas Jefferson University. The authors also thank Nicole Hartman and the RLO for their survey distribution efforts.

Authors’ Contributions

Conceptualization: C.H. and A.P.D. Methodology: C.H., J.K., N.L.S., and A.P.D. Validation: C.H. and A.P.D. Formal analysis: C.H., J.A.L., N.L.S., K.L., and A.P.D. Investigation: C.H., N.L.S., and A.P.D. Data curation: C.H., J.K., N.L.S., and A.P.D. Writing—original draft: C.H., N.L.S., and A.P.D. Writing—review and editing: All authors. Supervision: A.P.D. Project administration: C.H., N.L.S., and A.P.D.

Author Disclosure Statement

No competing financial interests exist.

Funding Information

This research was funded in part by the NCI 5P30CA056036-17 (N.L.S. and A.P.D.) and the Prostate Cancer Foundation Challenge Award (A.P.D.).

Supplementary Material

References

McCarthy

, Minsky

, Rochester

, et al. A proposal for the dartmouth summer research project on artificial intelligence. 1955. Available from: http://jmc.stanford.edu/articles/dartmouth/dartmouth.pdf

Yuan

, Rastogi

, Naik

, et al. A continued pretrained LLM approach for automatic medical note generation. 2024. Available from: http://arxiv.org/abs/2403.09057 [Last accessed: June 12, 2024].

Conway

, Gerard Gonzalez

, Shah

, et al. Racial disparities in diabetes technology adoption and their association with HbA1c and diabetic ketoacidosis. Diabetes Metab Syndr Obes, 2023; 16:2295–2310.

Pal

, Umapathi

, Sankarasubbu

. Med-HALT: Medical domain hallucination test for large language models. 2023. Available from: http://arxiv.org/abs/2307.15343 [Last accessed: June 12, 2024].

Wan

, Pu

, Sun

, et al. “Kelly is a Warm Person, Joseph is a Role Model”: Gender biases in LLM-generated reference letters. 2023. Available from: https://arxiv.org/abs/2310.09219 [Last accessed: June 12, 2024].

Duan

, Tang

, Wu

, et al. The Large Language Model (LLM) Bias Evaluation (Age Bias). DIKWP Research Group International Standard Evaluation; 2024.

Byrne

, Anaraky

, Dye

, et al. Examining rural and racial disparities in the relationship between loneliness and social technology use among older adults. Front Public Health. 2021. Available from: https://www.frontiersin.org/journals/public-health/articles/10.3389/fpubh.2021.723925/full [Last accessed: June 12, 2024].

Ojinnaka

, Adepoju

. Racial and ethnic disparities in health information technology use and associated trends among individuals living with chronic diseases. Popul Health Manag, 2021; 24(6):675–680.

Lee

, Rich

. Who is included in human perceptions of AI?: Trust and perceived fairness around healthcare ai and cultural mistrust. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery: New York, NY; 2021, pp. 1–14. Available from: https://dl.acm.org/doi/10.1145/3411764.3445570

10.

Rozier

, Patel

, Cross

. Electronic health records as biased tools or tools against bias: A conceptual model. Milbank Q, 2022; 100(1):134–150.

11.

Yan

, Zhang

, Yang

, et al. Differences in health professionals’ engagement with electronic health records based on inpatient race and ethnicity. JAMA Netw Open, 2023; 6(10):e2336383.

12.

, Xu

, Bai

, et al. Public perceptions on the application of artificial intelligence in healthcare: A qualitative meta-synthesis. BMJ Open, 2023; 13(1):e066322.

13.

Nader

, Toprac

, Scott

, et al. Public understanding of artificial intelligence through entertainment media. AI Soc, 2022:1–14.

14.

Davis

. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. Mis Q, 1989; 13(3):319–340.

15.

R: A Language and Environment for Statistical Computing. Available from: htttps://www.R-project.org

16.

Cartwright

. Human Relations and Hospital Care. London, Routledge, 2023.

17.

Amani

, White

, Balart

, et al. Generative AI perceptions: A survey to measure the perceptions of faculty, staff, and students on generative AI tools in academia. 2023. Available from: http://arxiv.org/abs/2304.14415 [Last accessed: June 12, 2024].

18.

Omar

, Soffer

, Agbareia

, et al. Sociodemographic biases in medical decision making by large language models. Nat Med, 2025; 31(6):1873–1881.