AI in healthcare: Regulatory guidelines and judge-made negligence principles for AI implementers

Standard of care of AI implementers: whether to implement or rely on medical AI, and the monitoring and oversight of AI usage

The more contentious legal issues are whether the doctor has breached the duty by providing diagnosis, treatments, post-surgery monitoring or advice falling below what is expected of a reasonable competent doctor, and in establishing if the breach had indeed caused the patient’s injuries or death.

If the doctor who is adjudged to be liable in negligence in the use of medical AI was acting as an employee of the hospital, the hospital may have to compensate the injured patient on the basis of vicarious liability ⁴⁴ even if it were not at fault. As mentioned above, hospital liability can also arise under the doctrine of non-delegable duties ⁴⁵ towards patients under their custody and care even if an integral function within the hospital’s scope of duty (such as the diagnosis and treatment of the patient) has been delegated to an independent contractor which failed to take reasonable care in discharging the delegated function.

In addition to attributive liability discussed above, the hospital may be directly liable in negligence for any wrongful decision to implement the medical AI. Consider the following scenario: An AI development company has approached hospital XYZ offering a medical AI for diagnosing a type of cancer based on image analysis. The medical AI was trained on data collated from studies conducted in two Western countries and the developer intends to train the model on data obtained from studies on Asian populations. It is based on a deep learning system involving interconnected layers of data with different weights attached. The AI is capable of generating outputs indicating either the presence of the specific type of cancer (positive) or its absence (negative) and the level of confidence associated with each output. The doctor’s decision to accept or reject the AI output with respect to diagnosis or treatment for particular patients can be assessed from the lens of legal liability.

In our discussion below on potential negligence liabilities of hospitals and medical doctors, we will examine one set of guidelines from Singapore namely the Artificial Intelligence in Healthcare Guidelines (October 2021) (‘AIHGle’) ⁴⁶ (pronounced as ‘agile’) that was co-developed by the Ministry of Health (MOH), the Health Sciences Authority (HSA) and the Integrated Health Information Systems (IHiS). The guidelines seek to ‘improve clinical and public trust’ in AI by advocating the ‘safe development and implementation of primarily AI-Medical Devices’ and ‘other AI implemented in healthcare settings’. They complement the AI-MD registration requirements administered by the HSA. Institutions which develop AI-MDs for their own patients, while not obliged to register with HSA, are ‘encouraged’ to register the AI-MDs with HSA to ‘reassure institutions and patients of the AI-MD’s quality, safety, and efficacy’. ⁴⁷

The AIHGle contain recommendations related to AI-MD for AI developers as well as for AI implementers (e.g. hospitals and doctors). The developers design, build, and test the AI systems while the implementers make use of medical AI, monitor its use and conduct reviews periodically. The AIHGle state that the use of AI-MD does not change ‘the liability of the implementing institution or the individual medical professional in their provision of appropriate and safe care’. ⁴⁸ This suggests that the existing liabilities of hospitals and medical doctors in the delivery of medical services remain, notwithstanding the use of medical AI. This statement itself is uncontroversial. What it does not specifically mention (but which we will explore in this section) is how the principles governing legal liabilities of hospitals and medical doctors can accommodate the use of medical AI in clinical practice, or whether these principles have to be modified in order to take account of the manner in which medical AI can impact on the delivery of medical services.

First and foremost, the hospital has to decide whether to purchase and implement the medical AI. There may be a range of considerations for implementing AI-MD, ⁴⁹ such as the intended use and purpose of the AI-MD, its efficacy, the safety, and quality of care provided, the representativeness of the training datasets, the existing regulatory approvals for the AI-MD, the risks of implementation and mitigating measures. Where the hospital or doctor implemented the medical AI contrary to its intended purpose, disregarded the safety protocols or the datasets were non-representative, and there were foreseeable risks of material harms to patients arising from such omissions, the hospital or doctor would likely be in breach of their duty to the patients for implementing AI without adopting reasonable measures to mitigate the risks that eventually materialised.

Assume the scenario that the hospital has made the decision to implement medical AI for service delivery. With regard to a particular patient, the AI model generates an output recommending stage 4 cancer based on image analysis. Would the doctor be regarded as negligent if he relied on the AI output which turned out to be wrong? On what bases did the doctor accept the AI output? Case precedents that support justified reliance on medical technology based on the features of the technology ⁵⁰ may be applied to medical AI. In one Singapore case, the court held that product distributors should not place unquestioning reliance on the approvals given by authorities when there is some suspicion concerning the product. ⁵¹ Similarly, the fact that the medical AI has been approved by the HSA for commercial use does not necessarily justify reliance by the hospital on the medical AI for clinical practice without further checks and verifications. Hence, beyond mere approvals by the authorities, the doctor should examine evidence indicating the accuracy and reliability of the AI technology being used for diagnosing cancers and be satisfied there are no material doubts or suspicions regarding the proper functioning of the medical AI or the accuracy of the AI outputs.

The converse is also true. The doctor may be found negligent for the omission to rely on or use medical AI which has been found to be accurate, affordable, and superior in performance and efficacy to human doctors. This may amount to an unreasonable omission to adopt or access available technology. ⁵² Applying existing precedents, establishing negligence would depend in part on whether there is sufficient evidence of a general practice for such usage. ⁵³ Such evidence cannot be taken for granted especially in the case of novel technology such as AI.

Human implementers may be subject to automation bias, that is, the tendency to favour machine-generated outputs either by disregarding the errors of AI (omission errors) or to accept the AI decisions despite evidence to the contrary (commission errors). ⁵⁴ The level of automation bias may be influenced by the user’s confidence in his own diagnosis or decision-making, trust in the decision support system, past experiences with the DSS, and the complexity and volume of the tasks at hand. ⁵⁵

The decision whether to implement medical AI and the scope of implementation within a hospital may also be affected by medical insurance coverage. The insurance policy may allow reimbursement for claims from patients regardless of whether medical AI was used or not, or limit to claims arising from cases only when the medical doctor adheres to the recommendations of the AI. Another factor relevant to the implementation decision-making might be the medical doctor’s concern for his or her reputation in diagnostic expertise, and not merely for patient well-being. In situations of uncertainty about the diagnosis of patients, studies have shown that medical practitioners may underutilise AI diagnostic tools due to the influence of the peers’ perception of their diagnostic ability even if the AI tool is more accurate in diagnosis than the human doctor. ⁵⁶

Apart from making a judicious implementation decision, AI implementers would likely be expected to monitor the performance ⁵⁷ of AI-MD, respond to adverse events ⁵⁸ and to conduct reviews in the event of errors or to ensure its ‘clinical relevance’. ⁵⁹ In addition, AI implementers should track the AI-MD at the point of deployment (i.e., ‘ground-truthing’) to determine the ‘deployment baseline’ so as to assess whether there are deviations from the intended performance at deployment. The deployment baseline should not fall below the current clinical practice baseline resulting in patients being ‘worse off’. ⁶⁰ These guidelines on monitoring go towards ensuring that the performance of medical AI continues to keep pace with technological and medical developments.

Drawing upon recommendations from manufacturers in the user manuals, AI implementers should maintain oversight over the AI-MD based on the intended use, workflows of the AI-MD and the clinical context. ⁶¹ For AI-MDs that are intended to be used alongside healthcare professionals, the AIHGle remind the implementers to ensure that its staff are trained to operate and interpret results from AI-MD. ⁶²

Importantly, the guidelines anticipate that medical AI can ‘continuously learn and adapt during its deployment’. In this regard, the AI implementers are ‘encouraged to consider’ additional safeguards including utilising locked AI algorithms when in actual deployment, applying the learning post-deployment upon testing and validation, ⁶³ and ensuring the quality, safety, and efficacy of AI-MD during deployment. ⁶⁴

As can be seen from the examples above, with respect to the standard of care expected of hospitals and doctors in situations involving new technologies, regulatory guidelines such as the AIHGle can shed light on the different contexts of use of medical AI in healthcare and the industry expectations of AI implementers. The AIHGle, described as a ‘living’ document, will be ‘periodically updated to incorporate good practices in the rapidly developing AI landscape’. ⁶⁵ Moreover, the fact that these guidelines are not legally binding allow the regulators greater flexibility to implement or adapt them in tandem with technological advances. ⁶⁶ This does not mean that regulatory guidelines will always capture accurately the contexts of use and industry expectations which are inevitably dynamic in nature. In order for the guidelines to have any impact on the law of negligence, they would have to be first accepted and adapted for use within the schema of the tort of negligence, and in particular, the standard of care.

Any judicial uncertainties relating to ascertaining the standard of care may be tempered by negligence rules that can adapt to new situations through the extension of existing rules, drawing analogies with existing precedents and recourse to professional codes and practices. ⁶⁷ At the same time, we should be alert to situations where the legal principles, even if they are extended or analogised, would not adequately address the fundamental nature of the technology. A case in point might be medical AI that will continuously learn as it is applied to different environments with sometimes unpredictable outcomes. We will encounter this issue again below in connection with negligence principles applicable to medical diagnosis and treatment and the giving of medical advice by doctors and hospitals to patients.

Proof of damage from alleged breaches by AI implementers: determining causal responsibility of AI implementers

On the question of proof of damage, the issue of remoteness of damage should not normally pose a major obstacle to the claim in negligence. For hospitals and medical doctors, personal injuries or death suffered by patients arising from AI errors should, save for rare and unexpected health effects, be reasonably foreseeable in the event of the negligent use of AI-MD.

The more difficult issue concerns causation of damage arising from the use of medical AI. A negligent doctor, however blatant or obvious the negligent conduct, will avoid legal liability if the patient cannot demonstrate that the negligence resulted in the harm suffered by him or her. The possible causes of the injuries to the patient include the negligence of the AI manufacturer/developer and/or the AI implementer and the AI software itself.

Let us first consider the scenario where the problem lies with the AI software alone, and assume that the AI developer and implementer (or any other party) are not at fault. In such a case, there is no possibility of a claim in negligence against any legal person notwithstanding that the patient may have suffered injuries resulting from the AI software errors. There is no need for a causation analysis here.

Let us now modify the scenario to one where the AI implementer is negligent but assume that the AI developer (or any other party) is not at fault. The enquiry would be as follows: but for the implementer’s negligence in the decision to implement and use the medical AI, would the patient have suffered the harm? In the case of inadequate medical advice, but for the medical doctor’s negligence in failing to disclose certain material information about the use of medical AI, would the patient have decided to proceed with the treatment and suffered the resulting injuries? ¹⁰⁰ If the response in each case is ‘no’, the breach of the AI implementer would be regarded as having established a counterfactual causal linkage to the patient’s injuries.

Consider an alternative scenario where both the AI developer and AI implementer are at fault. The AI commits an error in the diagnosis of a cancer that is attributable to the AI developer’s negligence in design or testing of the AI model, and the medical doctor, without proper verification and checks, accepts the AI recommendation. As a result, the medical doctor administers the wrong treatment and the patient suffers physical harm. In terms of causation analysis, if each of the breaches by the AI developer and medical doctor has on a balance of probabilities materially contributed to the patient’s injuries, they would be regarded as causally responsible for the patient’s injuries even if their relative contributions to the injuries cannot be ascertained with precision. ¹⁰¹ Here, we assume that the medical doctor’s conduct in administering the wrong treatment, though careless, was not wholly unreasonable or reckless so as to break the chain of causation arising from the AI developer’s initial negligence.

On the contrary, where the AI produces the correct recommendation on the diagnosis of a cancer, but the medical doctor rejects the AI recommendation and takes an alternative course of action that led to the patient’s injury, the effective cause is the incorrect diagnosis by the doctor. But for the doctor’s decision to reject the AI output and to proceed instead with an alternative action, the patient would not have suffered injury.

Complications may arise due to the range of parties (developers, clinicians, researchers and manufacturers) who are involved in building a particular AI system. In instances involving multiple parties, one approach to unravelling the causation issue might be to examine the source of the problem that eventually led to the patient’s injury. Where the problem originated from the defects in the source code, it is prima facie the alleged negligent acts or omissions of the AI developer that we should target. If the source of the problem lay with the training data quality, we can assess the culpability of the supplier of the training data or the implementer who was reasonably aware of the problem. We should also enquire if the AI developer should have taken reasonable steps to prevent the use of such poor-quality data for training and testing the AI model. In practice, when blackbox medical AI is deployed, ascertaining the precise data which the AI had relied on for the flawed recommendation that led to the patient’s injuries can be challenging.

Instead of AI developers and data suppliers, the problem may lie with the actual implementation of the AI model in the provision of medical services. The acts and omissions of the hospital or doctor with respect to the utilisation of the AI system should therefore be scrutinised. The AI developer may argue that the AI implementer’s wholly unreasonable or reckless usage of medical AI constituted the novus actus interveniens leading to the injuries as mentioned above. Alternatively, the deep learning AI system which gave rise to the unpredictable outcomes and/or the unpredictable use of medical AI may have been the supervening cause of the injuries. ¹⁰² In the above instances, the AI developers contend that they should be absolved of any causal responsibility. The supervening cause doctrine has, thus far, been limited to supervening acts or natural events that are unforeseeable. It has yet to consider events arising purely from an AI system’s technological design which generated unpredictable outcomes that are not attributable to the AI developer. If the law were extended to include such unforeseeable supervening technological events, the AI developer would likely not be legally liable.

In the absence of direct evidence on breach and causation, the rule of res ipsa loquitur relies on circumstantial evidence to establish breach on the part of the defendant that is linked to the accident. To succeed, the plaintiff has to show that (1) the defendant must have been in control of the situation which led to the accident, (2) the accident would not have happened in the ordinary course if proper care had been taken, and (3) the cause of the accident was unknown. Where these criteria are satisfied, a rebuttable inference of negligence arises on the part of the defendant. ¹⁰³ The defendant may rebut the inference by showing either that it had exercised reasonable care or that there was another non-negligent cause of the accident. ¹⁰⁴

Although res ipsa loquitur has been successfully applied in medical negligence, ¹⁰⁵ it may fail in certain cases. ¹⁰⁶ It does not, for example, apply where the patient’s injuries were due to the inherent risks of the surgery outside the control of the medical doctor. ¹⁰⁷ Moreover, the inherent risks could have the effect of negating the causal relationship between the alleged negligence and the injuries. ¹⁰⁸ Extrapolating to medical AI, there would be no inference of medical negligence on the part of the doctor in the actual use of AI if there were inherent risks associated with the use of the medical AI outside of the doctor’s control to begin with.

To further analyse the application of the rule, let us assume the patient suffered injury due to an AI error in prescribing treatment to the patient, and the AI was opaque. For a start, the cause of the accident would be unknown because the opaque AI system could not provide a causal explanation for the incorrect treatment. The AI error constitutes the ‘res’ that calls for an explanation rather than the defendant’s act or omission though the defendant may point to evidence of other plausible causes to rebut any inference of negligence. ¹⁰⁹ On one level, it may be argued that the criterion of control is satisfied on the basis that the hospital (AI implementer) was in control of the AI system which led to the event that resulted in the patient’s injury. However, this factor of control purportedly exercised by the hospital may be doubted in instances where the AI outputs are opaque to the hospital. Moreover, the hospital may counter that the real control would have been exercised by the AI developer(s) who had designed the AI system or that no single entity or person had exercised the relevant control. ¹¹⁰ Finally, it has to be shown that the accident resulting in the patient’s injury would not have occurred in the ‘ordinary course’ if proper care had been taken. If the AI outputs are not expected, traceable, or explainable, it would be difficult to satisfy this second criterion which is dependent upon events occurring in the ‘ordinary course’. ¹¹¹ Nonetheless, despite the obstacles and challenges, we cannot dismiss entirely the possibility of res ipsa loquitur being applied to patient injuries arising from the use of medical AI. This is because a general ‘common sense’ approach to inferring negligence may be adopted by the courts ¹¹² rather than the strict application of the doctrine discussed above.

Defence of volenti: has the patient willingly and knowingly consented to risks of negligence in the implementation and use of medical AI?

Even if the patient were able to prove the damage, the medical doctor or hospital may raise legal defences to defeat the claim. The defence of volenti relies on the defendant showing that the patient had voluntarily and knowingly assumed the risks of negligence arising from the use of medical AI. ¹¹³ Discharging this burden of proof would be challenging in most situations. For the defence to succeed, the patient must not only be aware of the nature of the risks but also their extent. Where the patient is unaware of the basic functions of the medical AI that has been utilised, there would likely not be sufficient knowledge of the nature of the risks. Furthermore, from the preceding discussions, risks from the use of medical AI can arise from various possible sources, for example, non-representative data producing biased recommendations, opacity of deep learning systems, and non-explainability regarding the AI outputs which sources may not be known to the patient. While the type of harm may be reasonably foreseeable, the extent of the risks is normally not easily anticipated.

Although limited in scope, the consent to risks defence may be tenable in cases where the patient in question possesses expertise or knowledge of the risks of medical AI and the hospital or medical doctor has fully informed the patient of the extent of foreseeable risk, or where the perceived dangers from the use of medical AI are obvious to any reasonable patient. Furthermore, we should note that patients’ knowledge about medical AI technology will likely increase over time as its usage becomes more prevalent. If patients, armed with sufficient knowledge of the nature and extent of the risks, nonetheless consent to proceed with the use of medical AI, doctors and hospitals may in an exceptional circumstance be absolved from liability for medical negligence.