Developing,Purchasing,Implementing and Monitoring AI Tools in Radiology: Practical Considerations. A Multi-Society Statement From the ACR,CAR,ESR,RANZCR & RSNA

A. Why Do AI Algorithms Differ From Previous IT/Informatics Developments in Radiology?

Traditional computer-aided detection (CADe) or diagnosis (CADx) systems as used in radiology for about 30 years are rule-based, using classical machine learning techniques with handcrafted features. The features the system was intended to detect, such as shape, size or texture of a lesion, were manually pre-defined, and then used to detect abnormalities in radiological images. ⁶ Although useful, CAD was limited by the need for manual feature engineering and the inability to learn and adapt over time.

Modern AI algorithms, particularly those based on deep learning, fundamentally differ from traditional CAD by automatically learning relevant features from data without explicit definition and programming. Deep learning algorithms can learn to identify patterns in radiological images by being trained on large datasets and, in principle, can continuously learn and improve their performance as they are exposed to more data. ⁷ Training of deep learning models can use either supervised learning (most used today, presenting pairs of inputs and desired outputs), unsupervised learning (the system clusters the data in classes), or reinforcement learning (the system learns by being rewarded or punished). ⁸

Another key difference is the level of automation that AI algorithms can bring to radiology. While traditional CAD systems can assist in the detection of abnormalities, AI algorithms have the potential to automate many routine radiology tasks, such as image segmentation and measurement, image quality and completeness evaluation, and can provide decision support by analyzing a vast amount of data in real time.^9,10

The implementation of AI in radiology presents new challenges, such as the need for large annotated datasets for training AI algorithms, ensuring the transparency and interpretability of AI decisions, and addressing ethical and regulatory considerations.^11,12

B. Why Do We Need to Evaluate AI Models in New Ways Before They Enter Routine Clinical Use?

Most AI models in Radiology are used to support lesion detection or quantification, or to help radiologists’ decision making. ¹³ Some newer approaches also help with analysing patients’ history or with writing reports and/or impressions of examinations. ¹⁴ To ensure safe operation of AI models in Radiology, it is essential to educate radiologists and other potential end-users about the principles of AI and teach them the limits and potential risks when using AI models.^15,16

It is also important to evaluate the accuracy of AI models on the target population before introducing them into clinical practice, and after that introduction, their performance should be monitored to detect drifts in accuracy.

The integration of AI algorithms into the radiology workflow is key to ensure their safe and consistent operation. The lack of widely accepted standards for AI integration is still a challenge. ¹⁷ In this context, attention should be paid to the interface design. Exposing radiologists to an increasing number of complex interfaces is undesirable, and is liable to diminish utility and acceptance of AI tools. ¹⁸

C. How Can We Differentiate Among the Multiplicity of Products on Offer?

The integration of AI in radiology has the potential to revolutionize healthcare practices, offering advanced solutions to diagnose, quantify, and manage multiple medical conditions. However, the evaluation of AI models extends beyond clinical accuracy, encompassing business and technical considerations. These, and other aspects of how potential users and purchasers can evaluate AI tools before implementation, are explored in detail in Section 6.

Key Statement

AI in radiology should promote well-being, minimize harm, respect human rights such as dignity and privacy, and ensure that benefits and harms are distributed among stakeholders in a just manner.

Given the critical dependency of AI upon data, ethical issues relating to acquisition, use, storage and disposal of data are central to patient safety and the appropriate use of AI. Important ethical issues relate to consent, privacy and data protection, data ownership, bias and fairness, transparency and integration of AI into clinical practice.^11,23

Bias and Fairness

AI systems can unintentionally perpetuate or even amplify existing biases in healthcare, leading to unfair outcomes (Table 1). In particular, AI systems rely on training data, lack context and are more likely to exhibit bias if the data used to train the AI system are not representative of the patient population on which the AI system will be used. This bias is due to differences in populations, and may reflect gender, sexual orientation, ethnicity, social, environmental, or economic factors. The data utilised may also contain inherent biases for other reasons, such as bias derived from the humans who label data for training the AI system. Different scanning devices and protocols may also influence the data used during AI development and induce bias.

OPEN IN VIEWER

Table 1.

Typology of Biases.

Type of bias	Explanation
Data bias	Bias can occur with any dataset. Common sources of bias potentially promote or harm group-level subsets based on gender, sexual orientation, ethnic, social, environmental, or economic factors.
Clinical confounding bias	Radiology AI may be biased by clinically confounding attributes such as comorbidities.
Technical bias	Bias can be introduced due to subtle differences in raw and post-processed data that come from different scanning techniques.
Automation bias	This is the tendency for humans to favor AI decisions, ignoring contrary data or conflicting human decisions. This can lead to errors of omission (when humans fail to notice, or disregard, the failure of an AI tool) and commission (when one erroneously accepts or implements a machine’s decision despite other evidence to the contrary). (See also Section 8).

The interaction between AI systems and humans is also germane. Humans have an appreciation of context and are more likely to understand if AI outputs are inappropriate in a given clinical context and act rather than simply accepting incorrect AI advice. In contradistinction, automation bias is the tendency of humans to favor the decisions of AI systems over human decisions, which can lead to errors if the AI system is incorrect. This automation bias may be accentuated when a radiologist is fatigued or there is a limited radiology workforce and therefore limited capacity to supervise AI output. Risks to patient safety will also be higher when autonomous AI systems are implemented or the AI system continues to learn and adapt over time. In these situations, the need for assessment and monitoring of AI system performance becomes commensurately greater.

Key Statement

AI systems rely on training data, lack context and are more likely to exhibit bias if the data used to train the AI system are not representative of the patient population on which the AI system is used.

Key Statement

Addressing ethical issues in AI will require a combination of technical solutions, government activity, regulatory oversight, and ethical guidelines developed in collaboration with a wide range of stakeholders, including clinicians, patients, AI developers, and ethicists.

A. Clinical Utility of New Products

New products should improve the quality or efficiency of existing workflows in terms of lesion detection, segmentation, diagnosis, or prediction of clinical outcomes. A common mistake among radiology AI developers is the development of solutions reflective of available technology and datasets, rather than those with clinical utility supporting existing workflows. Society-developed resources, such as the ACR DSI Define-AI directory, often serve as a good starting place to ensure the technology being developed meets genuine clinical needs. ³⁰ In the absence of an existing Use Case reference, or an application that is not a direct derivative evolution of an existing application, developers should involve clinicians as early as possible in the design and development process to gain insights into the feasibility and practicality of proposals, well before substantial investments in time and developer resources have been made.

B. Superiority to Existing Clinical/Radiology Tools

Demonstrating superiority over existing clinical processes can be a challenging proposition for developers, particularly those with limited clinical experience in the domain. For solutions backed by pre-existing open-science competitions, where clear performance metrics are defined and leaderboards of competition entrants are maintained, it is generally easier to demonstrate competitive equivalence or superiority. This is particularly notable in the long series of AI Challenges at annual RSNA and MICCAI conferences.^31,32 In situations where no such open data exist, developers should determine the baseline clinical performance, and compare the AI performance with existing or approved software, or radiologist multi-reader control data. Well-designed multi-reader diagnostic accuracy studies are a common method of reporting AI solution superiority, though they can be both difficult and expensive to perform effectively. When human readers are assisted by AI, different modes of algorithm use, such as first-reader, concurrent reader, second-reader, or triage modes, may affect how relative performance is analyzed. ³³

C. Radiomics, Explainability & Transparency

There are certain classes of AI applications that pose particular challenges to model interpretability. Radiomics refers to the extraction of a large number of features from medical images using data-characterisation algorithms to describe pixel intensities, relationships between these pixels, shapes, and textures. Many of these features are non-intuitive or do not map easily to subjective or clinical image findings. ³⁴ There has been much comment about the black box nature of AI models, with early efforts focused on heat map and saliency visualizations; some researchers have called for a combination of visualizations and generated text to improve interpretability of diagnoses.^35-37 Lessons learnt from traditional biomedical research are extremely relevant, and in situations where model transparency and explainability are poor, a higher standard of empirical evidence of performance may be required, including external or multi-centre test data to prove model generalizability, and prospective real-world evaluation, used in clinical settings that most resemble the clinical setting in which the product is most likely to be deployed.

Key Statement

Prior to approval, regulators should request information from AI software developers pertaining to the company, clinical use, implementation, product development, demonstration, cost, and publications (Table 2).

OPEN IN VIEWER

Table 2.

Relevant Information for Regulators Prior to AI Software Assessment [Adapted From Tanguay et al ³⁸ ].

Company information ❏ Company information ❏ Contact information

Clinical information ❏ Product description ❏ Imaging modality ❏ Target population ❏ Target organ ❏ Use cases ❏ Role in clinical workflow ❏ User interaction ❏ Type of application

Product performance ❏ Study design (retrospective vs. prospective) ❏ Single vs. multi-center study design ❏ Software role in study (first-reader, concurrent reader, second-reader, or triage modes) ❏ Sample size of training, validation, and test sets

Engineering information ❏ Supported operating systems ❏ Local or cloud-based ❏ Integration (standalone, PACS) ❏ Access (remote or on-site) ❏ Virtual machine requirement ❏ Security requirements ❏ Hardware requirements ❏ Imaging parameter requirements ❏ Technical performance metrics ❏ Input requirements ❏ Version associated with predictions

Product development information ❏ Source ❏ Sample size ❏ Number of sites ❏ Countries of origin ❏ Clinical validation ❏ Regulatory approval

Demonstration ❏ Illustrative content (image files) ❏ Video content ❏ Option to test software virtually

Cost information ❏ Trial availability and duration ❏ Installation cost ❏ Pricing model (one-time purchase or subscription) ❏ Technical support

Publications ❏ Publications (preferred) ❏ Abstracts (if publications not available) ❏ Links to PDF (if open access)

AI solutions generally present estimates of solution performance using a combination of retrospective and prospective validation trial results upon which their statements of function are based.

Regulators should pay close attention to ensure that the reported information complies with the highest standards of practice; studies should ideally adhere to criteria defined by the multiple established scientific reporting standards.^39-43 Lower quality evidence often has significant gaps in the information reported and only partially fulfills these standard criteria. Two common errors in solution performance reporting include a failure to report a range of expected performance—lower quality solutions often report a single summary accuracy figure—and not reporting specific failure conditions and errors, with lower quality solutions selectively highlighting the best diagnoses made by their systems. In the broader AI safety community, there is a strong embrace of Model Cards or System Cards, in which in-depth analyses of limitations, errors, and biases are explicitly reported, often entirely separate from the primary report of system performance.^44,45 This level of public transparency should be strongly encouraged by regulators to foster a greater culture of AI safety, and should be a primary consideration when evaluating the quality of submissions.

Although clinical risk models differ based upon geographic jurisdiction and historical precedents, we strongly believe that any regulatory model should draw clear categories and boundaries between advisory, semi-automated, and automated systems, with a deeper evidence base and real-world track record required for greater degrees of autonomy. Clinical references often cite, as a relatable metaphor, automation scales that have been proposed for autonomous driving vehicles, for example the SAE J3016 Levels of Driving Automation. ⁴⁶ Multiple attempts to design analogous levels of escalating automation for radiology workflow have been proposed.^24,47 Traditional regulatory frameworks governing medical devices have focused predominantly on monitoring or therapeutic devices, which have very rarely, up until recently, exhibited any functionality with the potential of autonomous action. The general AI literature is replete with examples of negative and often unexpected harms of AI making unsupervised decisions. ⁴⁸ The patient implications of decision making required in clinical medical imaging, even declarations of stability or normality, often have dramatic direct implications on patient care, which we suspect will require human co-supervision for some time.

Regulators should be particularly attuned to ensuring that solutions have an explicit post-market quality assurance plan. The importance of this has several aspects, but mainly relates to the issues caused by concept drift, due to changes in the patient population or occasionally even differences due to upgrades of successive new versions of AI software. ⁴⁹ In practice, what this may entail is prospective performance monitoring of the AI model, for example monitoring for major deviations in month-to-month diagnostic event frequencies, with alerts raised when normal bounds are exceeded, or a control sample approach where a constant reserved held-out set of test case examples is routinely evaluated with the algorithm, to ensure no major deviations on known difficult or borderline cases. ⁵⁰ At a very minimum, a clear reporting procedure for unexpected errors to the vendor, with named responsible contact personnel, should be established and made easily accessible to clinical end users.

Usage Benefits, Risks and Cost

For any AI tool to be successfully integrated into clinical practice, stakeholders should first clearly identify areas that need improvement and define relevant key performance indicators.^52,53 The integration of an AI tool may then be part of a larger strategy devised to attain the goal set for the institution. Alternatively, it might also be the case that a particular AI tool proposed by a vendor offers a potential to improve the quality of the institutions’ services in an area not previously considered. In either case, as outlined in Section 4A, it is essential to determine whether or not the tool solves a real, specific problem that the institution has; tools are solutions, and a solution to a non-existent problem has no value. Note also that different institutions have different problems; a tool that is valuable for one group may not have value for another.

For the positive impact of an AI tool to be measurable, objective and quantifiable goals should be set. It may be useful to consider both what proportion of cases or patients an AI tool is expected to impact, and what the magnitude of impact on each case or patient is expected to be. Purchasers should be aware that the beneficiary of the AI tools’ potential for improvement does not always need to be the radiologist or the radiology department alone. Ideally, all stakeholders involved, from the patient requiring a service to the respective institution and even the wider society could benefit from AI being successfully implemented in a clinical workflow. An example of a strong use-case could be AI as a supporting tool in high-volume radiological screening settings (e.g. mammography). In this case the benefits for patients could include earlier and better detection of breast cancer, leading to better overall outcomes, while benefits for radiologists could include increased productivity, the availability of an additional “safety net” or the potential to increase the time available for interaction with the patient. ⁵⁴ Apart from improvements in productivity and service quality positively reflecting on the institution, they could potentially help reduce costs, while for the wider society positive effects on overall healthcare costs and population health could be envisioned. Such effects could also be expected for other commonly suggested use-cases, such as the detection of large vessel occlusions or in other time-sensitive situations. However, for other applications like organizational AI support tools or as-of-yet more research-driven applications (such as AI-powered opportunistic screening) the benefits might not be as easily definable.^55,56 Depending on the local circumstances and healthcare system in place, such potential benefits need to be carefully weighed against their immediate and mid- or long-term economic impact. Return on investment (RoI) and cost–benefit analyses should be planned and carried out to ensure the viability of the planned AI integration. Depending on the healthcare system, establishment of a viable payment mechanism for AI use may be critical. AI models that primarily benefit a fee-for-service hospital or outpatient imaging center prove RoI through decreasing length of stay, ⁵⁷ improving throughput in the emergency department, ⁵⁸ increasing the volume of findings that require follow-up and/or treatment, decreasing the length of time it takes to perform an imaging exam, and improving operations in the radiology department. Other potential benefits to the radiology practice include decreased mental fatigue, improved radiologist recruitment and retention, and decreased medical malpractice liability, although these tend to be additive as they do not generally cover the cost of the AI.

Lastly, potential costs (both capital and recurrent) and risks associated with the implementation and usage of an AI system are essential components of any purchase analysis and decision. In part, risk assessment can be facilitated by consulting the risk matrix and the risk–benefit analysis provided in the regulatory files by vendors. However, some risks may not be addressed in such regulatory filings or only become apparent during use. The most obvious component of cost is the licensing costs paid to the vendor, but these are typically only a small part of the total cost of ownership. Other sources of cost include contracting and legal agreements, IT effort and professional services for integration with existing systems, training for users and administrators, infrastructure for running the AI, and on-going maintenance and monitoring.

Other essential factors in making an informed decision include evaluating the vendor’s compatibility as a reliable partner, the vendor’s staying power in a competitive environment with limited payor reimbursement (even more important in this era of AI vendor consolidation), optimized model pricing, and opportunities for collaboration beyond product purchase, such as co-development and product resale.

A key component of risk is understanding what the performance characteristics of the algorithm are likely to be in the environment in which it will be used. The error rate in use may differ substantially from what was reported in testing, particularly when the characteristics or distributions of the input data (e.g. scanner manufacturers, scan protocols, patient demographics, disease prevalence, comorbidities) differ from the test data. Ideally, each site considering implementation would perform a statistically rigorous evaluation of performance on their own local data (a method for this evaluation is presented in the Clinical Evaluation Section below). In practice, this may not be feasible. At a minimum, the characteristics of local data should be compared with those of the test data (a typical example might be where a model has been tested only on one manufacturer’s MRI scanner, but will be used on a scanner made by a different manufacturer). Where these are similar, the reported performance metrics may be relied upon with some confidence; where they are not (e.g., an algorithm tested only on adults being considered for off-label use in a pediatric hospital) one should proceed with great caution, if at all. Error frequency, conceptually the inverse of performance, is not the final word on risk, because different errors pose different risks. One should consider the detectability of the errors that are anticipated. That is, for each error, what is the probability that people in the workflow will notice that the AI has produced an erroneous output? For each detected error, what is the probability that the error will be corrected? Finally, if an error is not detected or not corrected, what is the expected impact on patients or other stakeholders? The consideration together of error frequency, detectability, correctability and impact provides a framework for assessing the direct risk of algorithmic errors. Ongoing monitoring of these risks is considered in Section 7.

Another key component of risk is the impact of an AI tool on radiologist performance. Relying on an automated tool to perform a task may lead to de-skilling of radiologists for the task the tool has taken on. This risk is particularly problematic if the radiologist is expected to perform the task manually when the tool fails, but may no longer be skilled enough to do so adequately. User over-reliance and under-reliance also decrease the accuracy of the combined output of the radiologist in combination with the AI model and is discussed further in Section 8.

A final aspect of risk that must be considered is the potential for AI to create or exacerbate healthcare disparities. AI is particularly prone to this because it is generally trained on retrospective data drawn from clinical archives, and these data represent the current and historical healthcare disparities and inequities of our society. Training an AI is a mathematical process of minimizing a cost function that proceeds without ethics or morals. Therefore AI may learn from the inequities and disparities embedded in the training data, and can perpetuate these in implementation. There is no easy or straightforward process for comprehensively identifying these biases, but it is incumbent upon us as physicians and data scientists to think about, search for and mitigate these biases; if these questions are unasked, they will most certainly remain unanswered.

Integration, Verification and Monitoring

Once expected benefits and goals have been decided upon, cost–benefit analysis has been carried out and potential risks have been assessed, integration of the selected AI tools can be planned. Depending on the local IT infrastructure and policies, purchasers can consider different technical integrations—either as local installations with dedicated computational resources on site or as a cloud-based software as a service (SaaS) model. In both types of installation, data orchestration of DICOM and HL7 play a vital role ensuring the right slices from the correct series of the relevant study for the right patient in the right setting are sent to the appropriate AI in an optimized time. To achieve a robust orchestration, understanding and structuring the content of your data is essential. Unfortunately, relying on DICOM metadata is often insufficient due to the high variability and labile nature of study and series names, and the fact that DICOM headers may be incomplete. A more robust option is to use imaging AI to determine the data contents at the studies and series level and use that output for orchestration. Using computer vision AI to determine which body parts are on each image and if intravenous contrast has been administrated are two of the most useful additions. Downstream data orchestration from the AI system requires an intelligent system able to facilitate different workflows depending on an understanding of the AI results. Most current implementations only send the AI results to the Picture Archiving and Communication System (PACS). This limited integration not only allows visualization of AI results by referring physicians, which may not be optimal if these physicians haven’t been educated about the details and accuracy of the AI model, but also has been shown to increase automation bias among radiologists. ⁵⁹ Furthermore, PACS currently offers limited modes for AI results integration and in most instances, the radiologist cannot modify the AI results in PACS. To optimize AI results management and integration, a PACS should enable the radiologist to interact with and modify the AI results and, if results are changed, empower the AI to immediately reprocess a new output. In addition, the updated AI result should be provided to the AI vendor so it can be used for future model improvement. This type of interaction is facilitated in a cloud-native environment where both the PACS and AI models can share radiology data and AI results. Additionally, the ability to accept and store AI results along with radiologist feedback, optimize data security, and continuously monitor AI accuracy are crucial technical aspects that are facilitated in cloud-native systems.

Whatever the integration, ideally AI tools should be well integrated into the usual clinical workflow and information systems in order to avoid additional workload by requiring users to switch between applications. A recently published survey revealed concerns about additional workload to be one of the main reasons for respondents not intending to acquire AI tools for their clinical practice. ⁶⁰ The same survey found that another major concern was that the AI system would not perform as well as advertised. This concern is important and should not be overlooked. Of course, vendors will have performed testing and quality assurance of the respective AI tools during regulatory approval, but purchasers should consider validation of the AI’s performance on a local dataset, and adjust parameters if needed prior to implementation in clinical practice. This process should be repeated whenever relevant changes are made to the AI software or the equipment used in combination with the AI. In the example of a commercially available breast screening AI model an update of the AI tool resulted in a substantially different recall rate, requiring recalibration of the decision threshold to ensure continued usage with clinically acceptable diagnostic accuracy. ⁶¹ These findings highlight that it cannot be taken for granted that diagnostic performance claimed in premarket publications translates to a comparable and stable performance during clinical usage, emphasising the need for continuous post-market surveillance of the AI systems used. The exact approaches to how this should be done are currently being discussed by the respective regulatory bodies. For example, the UK’s Medicine and Healthcare products Regulatory Agency (MHRA) Guidance for manufacturers on reporting adverse incidents involving Software as a Medical Device under the vigilance system details various circumstances in which an adverse event should be reported—including “[failure] to identify clinically relevant brain image findings related to acute stroke” and “[degradation of MRI image] appearance of anatomical and pathological structures”. ⁶² Similarly, the FDA’s Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device would expect manufacturers “to commit to the principles of transparency and real world performance monitoring” when making updates to their products. ⁶³ Stakeholders in implementation of AI tools in clinical practice should therefore familiarize themselves with the relevant methods and metrics for clinical evaluation and devise strategies to verify performance claims prior to tool introduction, and should continuously monitor performance during routine usage. ⁶⁴ This is especially important as the previously mentioned survey found that a large majority of respondents did not assess the AI’s diagnostic accuracy on a regular basis. ⁶⁰ Post-market monitoring is discussed in greater detail in Section 7 (below).

Human-AI Interaction

Besides technical performance details and the practical workflow integration of AI tools in radiology, the importance of difficult-to-measure human factors should not be underestimated. AI has undeniably made impressive progress and for many use-cases can reach diagnostic performance comparable to that of human readers. This has especially been shown in the context of breast cancer screening.^65-69 However, as discussed above, many factors can influence the technical diagnostic performance of AI tools in clinical practice. While it has been suggested that the combination of human reader and AI tool could help increase overall diagnostic accuracy by either the human detecting an error made by the AI or vice versa, recent studies question this premise and highlight the need to further study the psychological phenomena that can bias decision making in a setting of human-AI interaction. It is well known that automation bias—the tendency to over-rely on automated systems, such as AI-powered decision support tools—can influence human readers and negatively impact their ability to exercise oversight. ⁷⁰ Recently, a study focused on mammography found that even the most experienced readers exhibited this bias in an experimental setting and had significantly worse performance when a purported AI system suggested a wrong BI-RADS category. ⁷¹ Conversely, the opposite effect described as algorithmic aversion—where information is rejected in a decision making process solely based on it being AI-generated—can also be observed. ⁷² A recent study showed that radiologists and other physicians rated the same information about a chest X-ray as being less reliable when it appeared to come from an AI system than when it appeared to come from a human expert. ⁷³ These issues are further complicated by the fact that human-AI interaction may be influenced by details of the user interface’s (UI) design. For example, while many radiologists preferred image overlays to detect pulmonary nodules, it was found that this configuration of the UI did not improve reader performance, while a minimalistic setup with text-only UI output did. ⁷⁴ Similarly, a study evaluating eye gaze in endoscopy found that the usage of a computer-aided system for polyp detection led to significantly reduced eye movements while evaluating endoscopic videos and an increase of misinterpretation of normal mucosa. ⁷⁵ These findings highlight the need for further education on those topics to increase awareness amongst users and stakeholders and allow for safe and successful implementation of AI into clinical routine. ⁷⁶ Opportunities to help mitigate human-AI bias are discussed in Section 8. More focused research into this area is needed to provide reliable evidence on how to best design human-AI interaction.

Clinical Evaluation

While FDA or other relevant authority approval/clearance data provides some insights, testing the AI model on local data, with the local systems and workflows used in practice, is essential to ensure accuracy and relevance when the model is deployed. While local evaluation will need to be tailored to the specific AI model and local resources, Table 3 outlines tactics which may help practices decide if a given model is relevant to local practice and performs with suitable accuracy on local data (Table 3).

OPEN IN VIEWER

Table 3.

5-Steps for Assessing Clinical Accuracy of AI Model.

Step	Process	Detail
1	Review Model Accuracy	Evaluate AI model performance on local data. Look carefully at user facing metrics (i.e., PPV and NPV) as these affect user engagement. Use this information and case based examples to craft educational content for the radiologists to help mitigate human-AI bias.
2	Calculate Optimized Enhanced Detection Rate (EDR)	EDR = (# of AI positive exams, not included in the rad report) / (# of rad reports with the identified pathology). This value represents an improvement in sensitivity and patient care that could be reached by optimally combining the radiologist and AI results.
3	Identify “WOW” Cases	“WOW” cases are those that could affect patient care or hospital operations as seen through the lens of any of the radiology stakeholders including the radiologist, referring clinician, hospital administrator, patient, or payor.
4	Categorize Model Pitfalls	AI models will have false positives (FP) and false negatives (FN). Try to categorize the FP and, if possible, the FN cases so these can be used to set radiologist expectations and help mitigate the human-AI bias.
5	Summarize & Decide	Based on the above data, determine if the model is clinically worthwhile to roll out in your environment.

A clinical accuracy evaluation process can be performed efficiently and does not require model implementation into your clinical workflow. The first step involves comparing the AI model’s performance on local data against regulatory authority documentation, specifically evaluating accuracy through the lens of radiologist acceptance and engagement with the AI tool. Hence, parameters that are radiologist-facing, including positive and negative predictive values for the disease prevalence are more relevant than overall accuracy, Area Under the Curve (AUC), or sensitivity/ specificity. Secondly, calculate an “Enhanced Detection Rate,” the optimized detection that could be obtained through a combined detection of radiologist plus AI true positive results. Thirdly, impressive, or "WOW cases," should be identified to demonstrate the AI model’s value to users and stakeholders. Fourthly, categorizing AI false positives and, when possible, false negative cases can set radiologist expectations and improve their acceptance of an imperfect AI model (all AI models are imperfect). Finally, all the findings should be reviewed to determine if the AI model is worthy of clinical deployment.

Ultimately, the decision lies in the balance between positive predictive value (which is highly dependent on disease prevalence) and the value and number of “WOW” cases. Radiologists are more willing to accept false positives, if the model also identifies pathology that impresses the radiologist or would add value for the patient or other stakeholder. Disease prevalence also has a strong impact on downstream model acceptance. Low disease prevalence AI models produce results with numerous false positives limiting user acceptance. Disease prevalence in a patient group presented to an AI model can be modified by properly selecting patient imaging locations, such as Emergency Department, inpatient, or outpatient. Hence, some AI models may be deployed on a subset of exams because disease prevalence in that exam subset is increased from baseline. For example, pneumothorax (PTX) on Chest XRay (CXR) has a higher prevalence in the inpatient rather than the average population. Limiting a PTX AI model to only inpatient CXRs will provide fewer false positive results and will more likely be accepted by the radiologists from an accuracy standpoint.

Utilizing information from the above 5-step clinical evaluation for radiologist education, coupled with change management, is vital to set user expectations before AI model implementation. A local AI champion plays a significant role in promoting AI adoption among radiologists. Finally, continuous user education throughout the lifecycle of AI utilization and monitoring radiologist AI usage and the combined accuracy of radiologist plus AI are instrumental in ensuring optimal patient care.

Purchasing considerations are summarised in Table 4:

OPEN IN VIEWER

Table 4.

Purchasing Considerations for AI Models in Radiology.

Strategic	● Which problem is the AI helping to solve? ● What benefit can be expected from the AI’s usage? ● How much improvement can be expected? ● Are there any risks associated with the AI’s usage? How can those be mitigated?
Regulatory	● What is the AI’s intended use? ● At which risk category was the AI certified?
Performance	● How can the AI’s performance be monitored? ● Can AI failures be detected and reported? ● Is performance on local data comparable to claimed performance? Are differences between local data and training data known? ● Does performance vary depending on the imaging device used? Does performance vary depending on patient characteristics (gender, ethnicity, etc.)?
Workflow	● How is the AI integrated into the radiologist’s workflow? ● Are radiologists biased by the AI’s predictions? ● What training is required for proper usage and bias avoidance?
Technical	● How does the AI integrate into local IT infrastructure?
Economic	● What is the direct cost of the AI (e.g., licensing)? Which other costs need to be considered (e.g., legal, training, etc.) ● Can return on investment be estimated and monitored?

Periodic Monitoring of Model Performance

Re-evaluation of AI model performance using updated local data sets at specified intervals, but at least annually, may be an appropriate monitoring mechanism for models where gathering real-time data on model performance is limited (quantification, workflow enhancement, etc.) or in instances where patient safety will not be immediately impacted. ⁸⁰ Such a system requires that a new up-to-date evaluation data set be created using an appropriate number of clinical cases and parameters similar to the initial validation set to re-evaluate the performance of the model under current conditions. While this type of performance monitoring system could be useful for many of the AI models in clinical use, limitations include the time delay between ongoing use of the model and the occurrence of the discrete monitoring activity, which delays the institutions’ ability to take corrective actions should degradation occur. Another specific scenario which may require a re-evaluation of a model’s performance would be the issuance of a new model version by the manufacturer. It cannot be simply assumed by user sites that intended benefits introduced by a manufacturer’s deployment of the latest version of a model automatically generalize to the local practice.

Monitoring for Causes of Data Drift Affecting Model Performance

Since changes in equipment, protocols and naturally occurring changes in population demographics are known causes of input data (source image) drift and potentially reduced AI model performance, institutions could elect to define baseline input data characteristics at the time of model acceptance and then monitor for data drift against that baseline state specific to each AI model. ⁸² Identification of relevant changes in input parameters could trigger the re-evaluation process described above for periodic monitoring. By monitoring for individual components of data drift, institutions could trigger re-evaluation of model performance depending on timing and severity of changes and initiate appropriate steps to safeguard patient care.

Continuous Monitoring of Model Performance

Strategies for continuously monitoring AI model performance cover many of the risks which should be evaluated before AI model introduction (outlined in Section 6 above). While real-time determinations of statistical parameters such as sensitivity, specificity, and positive predictive value are not possible during continuous monitoring, the algorithm’s performance compared to the interpreting radiologist’s final report, where possible, can be used as a surrogate for model accuracy. As explained above, harvesting of metadata about the examination should form part of this monitoring, and should include equipment manufacturer, protocol used, radiation dose and patient demographics. When available, the contemporaneous radiologist interpretation is considered a surrogate for ground truth, but the strength of this opportunistic labelling may be different from labelling provided during initial validation studies. Ideally this data collection could occur in the background, comparing information automatically extracted via suitable natural language processing methods against the radiologists’ reports as an AI accuracy measure, and data contained in the DICOM header to monitor the compliance of examination parameters with AI manufacturer specification of input data whenever feasible. Limited patient demographic information may also be found in the DICOM header and should be incorporated in the data collection. More robust monitoring and bias detection solutions may require expanded patient demographics. Continuous AI monitoring offers several advantages over episodic re-evaluations. Relevant information about AI model performance should be recorded in a dedicated AI data registry that allows generation of reports across multiple sites and geographies. Such benchmark data may be useful to individual sites as well as to the AI vendors. ⁷⁹ At the local level, registry reports would allow institutions to identify performance degradation within their own local environment and could enable a systematic evaluation of the sources of potential data drift on a near real-time basis. For example, an institution with multiple CT scanners in their clinical workflow might identify performance degradation relative to their own historic performance in an AI model designed to detect intracranial hemorrhage. Hypothetically, analysis of the aggregate institutional registry data might show the poor performance to be limited to a single machine. Further analysis might also show that the performance degradation occurred after a software upgrade to that machine or change in examination protocol. Systematic analysis of cases that are not processed represent another important monitoring target. Such cases may point to systematic or anecdotal failure in the data acquisition or data transfer, impeding intended AI inference and preventing downstream clinical action to benefit from the same. Monitoring for non-performance represents an important building block of a local quality assurance system for clinical AI, which will be increasingly important as dependency of the clinical enterprise on AI increases in the future.

Aggregation of data from multiple institutions using the same AI models could provide information to developers to identify performance gaps that can be addressed in future versions of the algorithm, as well as meeting any future post-market surveillance regulatory requirements. While none of the AI models in clinical use employ continuous learning as a means for model improvement or local tuning, a hypothetical advantage of continuous monitoring solutions is the ability to inform future adaptive AI models with additional training data for continuous learning. However, there are significant limitations to the approach of continuous monitoring. Today such solutions may not be applicable for many (if any) AI models, including those performing quantitative tasks, and other AI models where performance cannot be measured real time. Furthermore, continuous monitoring requires integration of production systems within a given institution, including information that may not be accessible to a manufacturer without local assistance and requisite infrastructure. Standards for this, specific regulatory guidance and the IT infrastructure for AI registries do not widely exist, and developing internal continuous monitoring solutions is likely to be cost- and resource-prohibitive for most institutions. Pilot projects for AI registries are underway; better understanding of the importance of aggregation and analysis of AI performance signal over time is likely to increase end-user interest in registry participation and may be a cost effective option to support this cause. However, in the absence of any regulatory requirements or availability of continuous learning AI models, demand may be limited. Currently, there are few AI models in limited markets that have regulatory approval for autonomously functioning AI, ⁸³ and the parameters for and frequency of evaluating model performance have yet to be determined. These parameters will vary with the disease process being evaluated, the risk to the patient in the event of model failure, and the prevalence of the disease in the target population. Therefore, one could imagine that performance monitoring could include intermittent random sampling of a pre-determined number of cases with ground truth comparison to spot-monitor performance over time.

Future Local Tuning and Continuous Learning AI Algorithms

Local tuning of AI models and continuous-learning AI algorithms prior to deployment have theoretical potential to improve the local performance of AI products. However, to date all AI tools which have received regulatory approval are static and cannot be locally tuned or undergo modifications using adaptive learning techniques. Recently, the US Food and Drug Administration (FDA) has released draft guidance for a “Predetermined Change Control Plan” ⁸⁴ that would allow future modification to commercial algorithms for both local tuning and continuous learning. Any change control plan must include robust real-time AI model performance and measures that mitigate patient risk. Currently, this guidance has not been implemented but would be for models that are in the process of obtaining approval rather than those already approved.

Other Considerations: AI Governance, Managing Technology Lifecycle and Local User Environment

Given the complexity of managing all aspects of the AI lifecycle in clinical environments, provider entities engaging in the use of clinical AI are well served by formalizing local AI governance oversight and associated processes. ⁷⁸ This is needed to deal with the many challenges in all phases of the AI product life cycle, which include procuring well-functioning AI, monitoring its performance over time, making adjustments to the local environment (e.g. scanner protocols, AI orchestration, device configuration, workflow integration including opportunistic capture of ground truth labels, etc.) over time as needed, and an orderly process to replace currently deployed products with future updates or alternative products. Often forgotten, but no less important, are the effects of the ever-more prevalent staff turnover amongst clinical end-users, radiologists and technical staff, including informaticists. As new users arrive in a local practice, they need to be properly assimilated, oriented, and trained in the available AI tools and associated work processes, to become effective participants in this technology-assisted care delivery paradigm. Ensuring that all local stakeholders are up to date and competent in the use of AI technology is a shared responsibility between vendors and the leaders of local institutional governance and oversight.

Take-Home Points

Monitoring the performance of AI models in clinical practice is needed to ensure that any performance degradation is identified early so that appropriate measures can be taken to ensure patient safety. At a minimum, yearly re-evaluation of the need to assess model performance should be conducted, with monitoring of parameters known to be associated with drivers of input data drift. The need for more frequent re-evaluations should also be considered based on patient risk in the event of model failure and clinical decision relevance of a specific AI output. While not applicable to all AI models and clinical practices, continuous AI monitoring that captures model performance, examination parameters and patient demographics in data registries offers significant advantages over periodic re-evaluation of AI models, including real-time identification of local causes of diminished performance and providing developers with aggregated data for model improvement. Robust continuous performance monitoring will be needed prior to deployment of any autonomously functioning AI algorithms and is also a requisite for continuously-learning AI models.

Augmentative AI

In this scenario, radiologists collaborate with AI systems to enhance diagnostic accuracy and drive efficiency. This collaboration provides an opportunity to increase the value provided by radiologists, but is not without challenges. As discussed in Section 6 (Human-AI Interaction), one crucial issue is the potential introduction of human–computer biases into the radiologic interpretation.^59,70-73 These biases need to be both clarified and managed to ensure the AI’s output does not negatively influence the radiologist’s judgment. There are two general types of bias that can be introduced in a human– computer system, over-reliance, and under-reliance. Over-reliance, also known as automation bias increases the risk of False Positive (FP) and False Negative (FN) results: if the AI is right most of the time, radiologists may stop verifying the outputs, or come to trust the AI more than their own judgment. In this scenario the radiologist will accept incorrect AI results. Under-reliance has the same effect for the opposite reason. If the radiologist does not trust the AI results, they may disregard accurate AI output, also increasing FP and FN results. Ultimately, the output of the combination of radiologist plus the AI system must be optimized. These challenges can be further compounded by negative workplace attitudes ⁸⁵ and factors that decrease personal perception of accountability ⁸⁶ including radiologist burnout, and high workloads, both currently ubiquitous in radiology practice.

A robust approach to mitigating biases and challenges related to reliance involves continuous radiologist education about AI capabilities and limitations. Providing comprehensive information about AI decision-making, its results, and confidence levels can enhance transparency and help radiologists make informed judgments. In addition, categorizing scenarios where AI assistance may falter, and integrating that information into a robust training program, can empower radiologists to recognize and rectify errors. The accuracy of the AI system also affects rad-AI bias—bias is decreased by more accurate AI results. ⁸⁷ Hence, identifying the most accurate AI model has clinical relevance. Finally, the measurement of rad-AI accuracy, along with directed feedback, can further refine the system’s performance.

Ethical considerations surrounding augmentative AI are multifaceted. In settings where subspecialty radiologist coverage is limited, the introduction of AI assistance can significantly impact patient outcomes. However, the reliance on AI may lead to a dilemma where the presence of AI might influence the allocation of resources for training and retaining subspecialists. Careful consideration is needed to balance the ethical implications of AI augmentation in resource-constrained environments.

Autonomous AI

In contrast to augmentative AI, autonomous AI operates without direct human oversight, making independent diagnostic decisions. This scenario raises heightened safety and ethical considerations. ¹¹ Autonomous AI should be subject to stringent performance standards and comprehensive and continuous testing to ensure its reliability and accuracy. It is essential to critically assess the system’s failure modes, considering that statistics from regulatory approval or vendor-provided accuracy rates might not adequately reflect real-world performance across various environments.

For autonomous AI, a rigorous ongoing monitoring program is imperative to detect and rectify errors promptly. ¹¹ Training healthcare professionals in recognizing failure modes and offering a simple mechanism to disable autonomous AI when necessary is essential to avoid unchecked errors that could jeopardize patient care. Holistic continuous AI accuracy monitoring mechanisms are not yet mature. However, relying on such an a posteriori system to detect errors means that AI models may continue to provide inaccurate results for a period before there is sufficient data to confirm these inaccuracies. To gain earlier insights into AI’s accuracy, additional tools for assessing expected AI outcomes based on input data (e.g., determining whether the input data falls within or outside the training data distribution) or comparing the results of one AI model to those of other AI models simultaneously can be employed. ⁸⁸

Autonomous AI should be designed to initiate actions that are transparent, identifiable, and discoverable. The capacity to disable the AI system swiftly and effectively in the event of failure is crucial for patient safety. A streamlined process to address and mitigate failures should be in place to prevent repeating mistakes.

In communities where radiology services are scarce, the deployment of autonomous AI raises complex ethical questions. While autonomous AI can provide diagnostic insights in the absence of skilled radiologists, decisions made by AI systems could potentially lack nuanced human judgment. Striking a balance between accessible healthcare and maintaining diagnostic quality becomes a critical ethical concern.

Ultimately, the successful implementation of AI in radiology relies on an understanding of its implications, and proactive measures, including radiologist education, AI explainability, and radiologist-AI accuracy monitoring to address safety and ethical concerns.