Patients’ perceptions of using artificial intelligence (AI)-based technology to comprehend radiology imaging data

Data collection

To gauge patients’ general perceptions concerning the use of patient-facing, AI-based systems to interpret imaging data and radiology reports, we conducted semi-structured interviews with 13 participants between August and September 2019. The participants took part in our previous study which focused on understanding patients’ challenges and needs in interpreting diagnostic results. All participants have recent experience with using patient portals to review their diagnostic results before the study took place. This study has been approved and overseen by the first author’s University Institutional Review Board (IRB).

The participants reported their demographics in a short questionnaire, which was deployed online and completed by participants before the interview (see Supplemental material). In addition to general demographic questions (e.g. age, gender, education background, and ethnicity), participants were also asked to rate their health literacy and technology literacy on a scale of 1-5 (“1” denotes low literacy whereas “5” denotes high literacy). Definitions of health literacy ³⁶ and technology literacy ³⁷ were provided to help participants understand the meaning of these two concepts. Almost all participants reported medium to high level of technology literacy and health literacy, with only one participant (P11) rating her technology literacy as “low to medium”. Characteristics of the participants are summarized in Table 1.

OPEN IN VIEWER

Table 1.

Characteristics of study participants.

Participant ID	Gender	Age	Ethnicity	Occupation	Highest degree	Technology literacy	Health literacy
P1	F	50–64	White	Unemployed	Associate degree	4	5
P2	M	50–64	African American	Retired	Associate degree	5	4
P3	F	50–64	White	Retired	Associate degree	5	5
P4	M	50–64	White	Technical analyst	High school	5	5
P5	M	26–49	White	IT clerk	Associate degree	5	3
P6	M	26–49	White	Dog trainer	Associate degree	4	3
P7	F	50–64	White	Landscaper	High school	3	4
P8	F	50–64	White	Executive assistant	Bachelor’s degree	5	5
P9	M	18–25	Asian	Medical assistant	Bachelor’s degree	4	4
P10	M	26–49	Asian	Visual artist	Bachelor’s degree	5	5
P11	F	26–49	White	Office assistant	Bachelor’s degree	2	4
P12	F	26–49	White	Sales	High school	5	5
P13	F	26–49	White	Unemployed	Bachelor’s degree	4	4

The interview protocol was developed in an iterative manner by the researchers and pilot tested with a small group of people to ensure the clarity and relevance of the questions, and to estimate the length of time to complete the interview. The interviews were conducted by two researchers who had completed research ethics online trainings. During an interview, we first asked the participant to read and sign the informed consent form which explained participant rights including the ability to withdraw from the study at any time without penalty. The interview began with general questions about participants’ awareness and experience of using AI-based systems to consult health-related information. It was assumed that many participants had no experience with AI-driven healthcare systems in the past; thus we used a technology probe approach ³⁸ to demonstrate a system prototype which could predict whether a radiology report describes a normal or abnormal finding (Figure 1). The purpose of using this approach was not to explore usability issues or collect information around system use. Instead, we introduced the system prototype to elicit reflections from participants pertaining to the application of AI technology in radiology report interpretation and to reveal technology needs, concerns, and desires.^39–41 The prototype has a web-based interface, showing the report content and the algorithmic predictions (Figure 1). To develop the system prototype, we used a linear support vector machine (SVM) model to train a dataset created in a previous study. ⁴² The dataset consisted of 276 sentences retrieved from de-identified radiology reports that were annotated by the subject matter experts (i.e. radiologists), and 727 sentences that were annotated by crowd workers. ⁴² They annotated the radiology reports as describing either “normal” or “abnormal” findings (we refer to these annotations as “labels”). The AI model was trained using all the crowd-annotated data (727 sentences and labels). We then used the 276 expert-annotated labels as the “gold standard” to evaluate the performance of the model. The AI model had a great performance (around 90%) in predicting the normality of the findings described in a radiology report. Once the system demonstration was completed, the researchers continued the interview by asking about their perceived benefits and concerns of using AI-based health systems, the types of medical information they would like to receive from such systems, and the ways to increase their acceptance and trust of AI outputs. All interviews were audio recorded with participants’ permission. All participants received a $20 compensation after the study.

OPEN IN VIEWER

Figure 1.

The prototype interface.

Data analysis

All interviews were transcribed verbatim. The transcripts were managed using Nvivo (QSR International, version 12), a software for organizing and analyzing qualitative data. Two researchers first independently analyzed a small set of interview transcripts (n = 3) using the open coding technique. ⁴³ We chose this coding technique because it could generate rich thematic analyses through line-by-line, iterative-inductive analysis of interview transcripts, revealing detailed observations of participants’ perspectives and opinions while ensuring study validity. ⁴⁴ The initial list of codes and code definitions were generated and then discussed in a group session until a consensus was reached. This step was followed by creating a codebook which defines each code to standardize the coding process for the rest of the interview transcripts. The researchers then used the codebook to complete the analysis. The agreement of coding results between two researchers was substantial (i.e. greater than 85%). Disagreements were discussed and resolved during weekly group meetings. New codes that emerged through this process were also discussed by the research team to determine whether they should be kept, merged, or removed. Using affinity diagrams, ⁴⁵ the final list of codes was grouped under themes describing participant’s perceptions and concerns of health AI systems, as well as the approaches for enhancing the system’s acceptability and trust. The process of grouping codes is illustrated in Figure 2. For example, “concerns about the quality of AI predictions”, “lack empathy”, “privacy concerns”, and a few other codes (bottom left, Figure 2) were deemed relevant to the participants’ concerns about using the AI system. Thus, we grouped them into a larger theme, namely “concerns”. This step was followed by identifying representative quotes to support the claims.

OPEN IN VIEWER

Figure 2.

Using affinity diagram to group codes into larger themes.

General perceptions of using AI tools to interpret diagnostic results

There was a general lack of familiarity and use of patient-facing, AI-based systems amongst participants. All participants reported never used such systems to consult clinical information and were generally unclear about what these systems can do. However, after interacting with our system prototype, a majority of participants considered AI-driven tools would be useful and suitable for interpreting a radiology report: “I like the idea, I like the concept. I’ve got no problem with using it. It’s gonna happen no matter what. We’re all just in for a future of a lot more AI interaction. [. . .] I am actually looking forward to it.” [P2]

Participants also discussed certain scenarios in which AI-driven systems can facilitate the comprehension of radiology reports. First, a majority of participants perceived AI to be particularly useful when they might struggle to comprehend their physician’s diagnosis. They stated that they could use these systems to seek more personalized and actionable information to better make sense of their radiology report and confirm their physician’s diagnosis. In particular, as people are often confused about the meaning of the imaging findings, they noted that providing a normal vs. abnormal interpretation, as demonstrated by our system prototype, is very useful. They also expressed an interest in obtaining information related to the prognosis and treatment options from AI systems, if possible at all: “If an AI system could access your medical record and tell you this is the most likely thing, that would be great.” [P8] Getting such information could help them determine what actions are needed. However, allowing patients to use AI systems to consult the meaning of diagnostic radiology findings outside of clinical settings may raise practical and ethical issues. For example, there exists legitimate concerns about the accountability of AI systems in decision making. ⁴⁶ When it comes to high-stake conditions, such as communicating medical findings to patients, policies and guidelines should be in place to regulate the use of such systems and clearly define the accountability of AI outputs. ⁴⁷

Furthermore, patients may visit and consult with more than one doctors and undergo multiple radiology imaging studies regarding a medical problem. A few participants discussed the possibility of using AI systems to evaluate and compare different opinions received from their physicians with respect to their diagnostic imaging studies: “I could get a second opinion because I found my doctors interpret my results differently. Maybe it could look into that.” [P8]

Lastly, AI systems were perceived as a useful tool in allowing an individual to prepare for their clinical visits. The systems could provide credible information that they can use to conduct their own research along with contemplating and constructing questions prior to the consultation. By doing so, they can become more prepared for the conversation with their physician, as one participant explained: “Maybe give a user questions that they can ask the doctor, because that’s the other thing I noticed, is that a lot of people don’t get the results they want, or the medical outcomes, because they don’t know what questions to ask the doctor. The doctor has like two minutes to talk to you. [. . .] The patient doesn’t know what to ask, so doctors would be like, ‘Hmmm, they are not complaining about this.’ But if AI could be like, ‘Hey, here is your results, do you feel this? Or do you have problems breathing? Or so on and so forth, and if you do, please bring this up with your doctor.’ My stepmother works in the ER and she’s an RN [Registered Nurse]. And she’s like, ‘Half the time when people come in, if they were just able to ask the right questions, they would be in and out, they’d start treatment immediately.’” [P5]

Concerns

Despite the general positive attitude, participants noted that AI-based systems should be a supplementary service (“a helpful adjunct to doctors” [P3]) rather than a replacement of the professional health force: “I think it can be like a useful tool for the doctor to make the diagnosis, but you can’t remove the human element from it. You still need a doctor to make the final decision.” [P9] They also expressed several major concerns that caused their hesitance with using patient-facing AI systems as part of their healthcare, including accuracy, cyber-security, and lack of empathy.

First, a major concern is related to the quality, trustworthiness, and accuracy of the medical information provided by AI systems: “I would need proof that it works and what you’re actually getting is meaningful information. Like it’s not just some crap. If it’s going to make recommendations to me, I want them to be proven that they’re actually legit.” [P13] Indeed, as healthcare is a high-stake context, users need to ensure the AI-generated predictions and medical recommendations are accurate and reliable, otherwise, they may perceive it as a risk. Our participants further explained that they wouldn’t solely rely on AI to make sense of their radiology report; instead, they may use other approaches for assurance purpose, such as looking up information online (“I may have to Google afterwards just to confirm”), and seeking confirmation from a doctor (“It has to be approved by my doctor, you know, something reassuring that this is not just some crap AI”).

Another major concern expressed by many participants is related to cybersecurity, privacy, and in particular, whether AI systems could maintain confidentiality so that sensitive health information was protected from potential hacking or data leakage, as one participant described: “It’s just those concerns that how safe it is out there. Well, you hear things get hacked. What happens if my health stuff gets hacked and it can be used against me?” [P11].

While a few participants thought that well-designed patient-facing AI systems can be more accurate and logical compared with doctors, the lack of human presence was seen as the main limitation. In particular, radiology imaging studies sometimes are used to diagnose complicated diseases (e.g. brain tumors, lung cancers), communicating an interpretation of radiology report (e.g. normal vs. concerning findings) without the presence of a medical professional may cause needless anxiety and negative emotions. Participants therefore expressed concerns about the inability of AI to understand emotional issues. Aligned with previous work, ³⁴ the responses given by AI were seen as depersonalized and inhuman: “For serious topics like cancer or a disease that may be killing you, you don’t want AI telling you ‘you’re going to die’. [. . .] we are not at the point where AI basically understands emotions. So in kind of sensitive things, you kind of just want it to be like, ‘hey, here’s your results, here are the steps you can do while you’re waiting for your doctor.’ [. . .] Give a warning, but do it in a way that doesn’t freak them [people] out. Again, humans want empathy” [P5]. This echoed the argument that health information systems need to consider patients’ emotional needs when they communicate concerning health news. ⁴⁸

Increasing acceptability and trustworthy of AI-based systems in communicating radiology report findings

As described above, participants had concerns about using AI systems to make sense of their radiology report. In particular, they tended to have doubts about the quality and trustworthiness of AI outputs, such as the predictions and recommended medical information. When asked about how to address this concern, the majority of participants expressed the need to increase system transparency by explaining how AI arrived at its conclusion. This echoes previous research findings stating that AI explanations are often demanded by lay users to determine whether to trust or distrust the AI’s outputs.^49,50 During the interview, we probed for more information with regard to what types of AI explanations participants need for trust-building. We grouped the needed AI explanations into four categories, including logic/reasoning, information sources, system reliability, and personalization. Next, we describe each category in greater detail.

System logic and reasoning: Prior work has demonstrated that providing information related to how a system works, including its policies, logic, and limitations, could be a viable way to enhance users’ understanding and trust in system-generated recommendations.^49,51 In a similar vein, in our study, participants expressed the importance of knowing how AI systems generate medical information to help them quickly and accurately discern whether or not to trust those medical-related recommendations: “I’d be more interested in knowing what kind of machine learning was used for the AI and how it arrives at its results [. . .] I’d be interested in seeing the details, the sample size or what algorithms are being used, because it could just be using a sample of 10 people.” [P10]

System reliability: System reliability has a significant impact on user trust and acceptance of technology. ⁵² Our participants would prefer to know such information as under what circumstances has this system been wrong in the past to determine whether they would trust and act on AI-generated outputs. One participant explained: “It can be useful if offering information about how many errors have occurred in the past. Such as how much of the times it was correct, actual results like it was correct 80% of the time or something like that.” [P10] In addition to the performance history, participants also mentioned it would be useful to know the system’s confidence level/score, which reflects the chances that the AI is correct for a particular prediction.

Information sources: Presenting the information sources underpinning a system has been shown to improve a user’s understanding of system functionality and trust of system outputs. ⁵³ Aligned with the previous work, our participants agreed that being able to see information sources used in AI prediction can help them eliminate any doubt about the qualities of the system’s data and establish appropriate and calibrated trust toward the recommendations: “My level of trust would depend on the source naturally. If it’s from Joe down the street, obviously I wouldn’t be too crazy about it. But if it’s from a trusted source, like a well-respected medical organization or something like that, like John Hopkins or Mayo Clinic, that would probably help build a little bit of trust.” [P4]

Personalization: Providing personalized information is a key feature of intelligent and recommender systems.^53,54 Our participants expressed the necessity of understanding whether and how their inputs (e.g. diagnostic results, demographic information, and past medical history) are taken and modeled by AI, and to what extent system outputs are personalized for them. Understanding such information is important to achieve acceptable levels of user trust and engagement: “I think it would be helpful if I could know how much the information given from the AI was applicable to a person that has given the information to AI.” [P9]