Abstract
Some encouraging uses for AI in medicine will lead to potentially novel legal liability issues. Complex algorithms involve an opacity that creates problems for the medical and legal professions alike. As iatrogenic injury is common in medical malpractice, the medical profession is understandably concerned when AI is introduced in diagnostic and therapeutic devices and events and outcome cannot be fully explained due to the “black box” effect.
A concern about machine learning algorithms is the black box issue and understanding how conclusions or outcomes are reached. The deployment of AI devices in healthcare will require an increase in a clinician’s understanding of AI to increase the transparency of their use.
An important aspect of medical treatment is the notion of “therapeutic privilege”. This will only arise in limited circumstances and requires the clinician to make a judgment, based on reasonable grounds, that the patient’s physical or mental health might be seriously harmed by providing the information.
Given the complexity of AI and the black box effect, could too much AI transparency possibly overwhelm a patient, such that it may dissuade them from giving consent in circumstances where treatment is necessary and essential? In other words, too much AI transparency and information may inadvertently hinder treatment and progress.
Introduction
The roots of Western medicine stem from Ancient Greece where Hippocrates introduced numerous medical terms universally used by physicians, including symptom, diagnosis, therapy, trauma and sepsis. 1 Whilst most clinicians are familiar with the Hippocratic Oath, they are less likely to be familiar with the medical texts of that time. Many now view the Greek physician–patient relationship as paternalistic, in which the physician concealed diagnostic or prognostic information from the patient. 2
With the requirement of informed consent so prevalent in recent decades, has that paternalistic concealment been largely obliterated or is there a residual balancing exercise between the right to be informed and the risk that too much information may, inadvertently or otherwise, lead to harm? The Hippocratic Oath amongst other things adopts the Latin maxim “primum non nocere” (“First, do no harm”). It is followed by, “Then try to prevent it”. Is a clinician who prevents harm due to information overload necessarily acting in the patient’s best interests? The question is open.
Human beings are individuals whose capacity to understand varies greatly. What may seem straightforward and intelligible to one person, is beyond comprehension to the next. Within those extremities there is a wide cognitive spectrum. With the need for informed consent, and the requirements for AI transparency, it may be that therapeutic privilege will play an increasing role in assisting clinicians to achieve optimal healthcare outcomes whilst also providing a measure of legal protection.
Current AI developments in healthcare
Artificial Intelligence (AI) technologies, such as machine learning (ML), are of increasing significance in healthcare. It is a priority area of AI development, with governments and the private sector investing large amounts. AI-enabled medical applications have been developed that promise to improve diagnosis; assist in the treatment and prediction of diseases; improve clinical workflow; enable high quality direct-to-consumer services and devices; aid in genome interpretation and biomarker discovery; and in automated robotic surgery. 3 Gradually, AI-enabled medical devices are gaining regulatory approval and being released to the market. The US Food and Drug Administration (FDA) has approved hundreds of AI-enabled medical devices and this number will continue to rise. 4
AI and the black box
The feature of AI that poses specific challenges to the evaluation of liability and regulation is its “black box” nature. 5 Unless specifically programmed to do so by AI designers and engineers, an algorithm does not provide a rationale for its output and remains opaque.
The “black box” phenomenon is something that is troubling the medical profession – one which has the notion of trust as its foundation. As noted by the Australian High Court in Breen v Williams: “[T]he relationship of doctor and patient is one where the doctor acquires an ascendancy over the patient and the patient is in a position of reposing trust in the doctor.”
6
Black box AI in healthcare is problematic from several perspectives. The functioning of contemporary medicine relies fundamentally on trust. For medical AI to be successful, it must be trusted by governments, health professionals and the public. The quality of medical advice depends on the reasoning being open to scrutiny and evaluation. An inability to clearly explain decisions may impact on how clinicians utilise the information obtained from AI systems when treatment plans are put in place.
Some scholars warn that black box algorithms can hamper patient autonomy in clinical decision making. 7 A patient must be able to make their own autonomous decision about treatment options. That can only be achieved when they have both the decision and the reasons for it explained to them. That is a fundamental requirement of medical care – and a legal duty – with or without AI systems. An AI system, by reason of its opacity or unexplainability, precludes a patient from being fully informed of how a particular recommendation or decision was arrived at. That, in turn, compromises their ability to decide whether to accept or reject the AI recommendation. This may provide an ethical reason to oppose the introduction of black box AI systems in that it would violate the right to informed consent. 8
Being mindful of these issues, AI-enabled medical devices are expected to comply with several ethical principles and policy recommendations. AI ethical guidelines, including healthcare-specific AI ethical guidelines, require AI-enabled medical devices to respect such principles as benevolence, privacy and protection of data, safety, fairness, accountability and responsibility, avoidance of bias, governance and others. A sought-after principle in most guidelines is that AI in healthcare should be transparent and explainable.
There is debate as to the level of transparency that should surround medical AI. That is, exactly how much of the AI system do clinicians and patients really need to understand before they can comfortably make an informed decision as to its use. Enabling patients to understand how AI-determined diagnosis and treatment options are arrived at is crucial but also complicated. Clinicians also must be able to provide clear and cogent explanations of diagnoses and treatment options. This is something that bears upon the principle of informed consent, namely, before a patient can fully consent to something, they should at the very least have knowledge of all material matters. 9 Similarly, clinicians should be able to provide the explanations sought. In healthcare, if AI makes a decision that will impact on a patient, then all material risks need to be recognised, explained, and understood. Obviously, that will include knowing how the AI arrived at a given decision, which needs to be explained in layperson’s terms and not replete with technical jargon.
Drugs and other medical black boxes
Strictly speaking, AI is not the only black box in medicine. In an article researching the ability of AI to improve the prediction and treatment of sepsis in hospital patients, Sendak noted that “[T]he human body is in many ways ‘a black box’, in which the causes and mechanisms of illnesses often elude explanation”. 10
Electroconvulsive therapy (ECT) is a safe and effective treatment for certain psychiatric disorders. It is often the fastest and best treatment available for severe depression and other psychiatric disorders, such as mania and psychosis. During ECT, a small amount of electrical current is passed through the brain while the patient is under general anaesthesia causing a seizure that leads to chemical and cellular changes in the brain that relieve severe depression. Since the introduction of ECT in 1938, the mechanism of action of this highly effective treatment has intrigued psychiatrists and neuroscientists who do not yet fully understand exactly how it works. 11
Certain drugs also remain to be fully explained, for example, lithium. Doctors do not know exactly how lithium works to stabilise the mood of a patient, but it is thought to help strengthen nerve cell connections in brain regions that are involved in regulating mood, thinking and behaviour. Another is acetaminophen (paracetamol). Despite competing explanations for how acetaminophen works, we know that it is a safe and effective pain medication because it has been extensively validated in numerous randomised controlled trials (RCTs). 12
The point to be made is that despite being largely unknown or a black box, these drugs and treatments are regularly used in the healthcare system. That is because they have undergone RCTs which have historically been the gold-standard way to evaluate medical interventions. It should be no different for AI systems.
Informed consent
The law of negligence is premised upon the general rule that those whose acts or omissions might injure another should exercise reasonable care to avoid such an occurrence. The elements that are required to be made out in an action for negligence are well known: the existence of a duty of care; a breach of that duty by a negligent act or omission; and damage suffered. Fundamental to these, but often considered separately, is the requirement of a causal connection between breach and damage.
In the context of healthcare, one area that has gained increasing attention over the past decades when looking at the tort of negligence is the principle of informed consent. This includes being warned of material risks associated with treatment. It is apposite to look at two leading judicial decisions.
Rogers v Whitaker
In the Australian High Court case of Rogers v Whitaker, 13 a patient with pre-existing poor sight in her right eye underwent surgery in the hope of rectifying it. The operation was not successful, but it was performed with the requisite care and skill. Unfortunately, the patient suffered sympathetic ophthalmia post-operatively and, due to complications, lost all sight in the left eye. The patient was rendered almost totally blind.
The question in Rogers was whether the standard of care required information regarding the risk associated with the aftermath of surgery to be given to the patient. The eye surgeon gave evidence that it had not occurred to him to mention sympathetic ophthalmia to the patient.
In England, the approach to the resolution of similar problems had been determined according to the Bolam rule.
14
In Sidaway, the court formulated this as follows: “A doctor is not negligent if he acts in accordance with a practice accepted at the time as proper by a responsible body of medical opinion even though other doctors adopt a different practice. In short, the law imposes the duty of care; but the standard of care is a matter of medical judgment.”
15
It followed from this rule that so long as an acceptable number of medical practitioners adopted the practice in question, that would avail the practitioner a complete defence. On the issue of whether a risk is relevant to a patient, the High Court in Rogers held at
16
that this was a question for the courts themselves stating: “The law should recognize that a doctor has a duty to warn a patient of a material risk inherent in the proposed treatment; a risk is material if, in the circumstances of the particular case, a reasonable person in the patient’s position, if warned of the risk, would be likely to attach significance to it or if the medical practitioner is or should reasonably be aware that the particular patient, if warned of the risk, would be likely to attach significance to it. This duty is subject to the therapeutic privilege.”
Thus, based on autonomy and informed decisions, a patient must be informed of material risks. In Rogers, the court did not entirely rule out the exercise of therapeutic judgment on the part of a doctor as to what information should be given to certain patients and how it is to be conveyed. The qualification the court made to the duty owed to give information about risks, was where there was a danger that the provision of all information would harm an unusually nervous, disturbed or volatile patient.
Montgomery v Lanarkshire Health Board
The facts in Montgomery are familiar to most.16,17 The UK Supreme Court considered liability in negligence for failure to disclose material risks to patients as part of the process of informed consent. The Supreme Court declared at [86]–[87] that Sidaway had represented substantially the correct position, subject to the Rogers v Whitaker “refinement”. Setting out a revised duty to warn test, Lords Kerr and Reed stated at [87]: “The doctor is therefore under a duty to take reasonable care to ensure that the patient is aware of any material risks involved in any recommended treatment, and of any reasonable alternative or variant treatments. The test of materiality is whether, in the circumstances of the particular case, a reasonable person in the patient’s position would be likely to attach significance to the risk, or the doctor is or should reasonably be aware that the particular patient would be likely to attach significance to it.”
What can be distilled from these decisions of superior courts is that a patient must be warned of material risks, and this is a subjective test. That is, what is significant to the individual patient in a particular case. As noted, though, this provision of information must be balanced: “On the one hand, physicians must provide all the information a patient needs to make an informed decision. On the other hand, complex medical information, including all aspects that are somehow relevant to the treatment, would rather prevent informed consent than promoting it.” 18
Therapeutic privilege
There is an assumption that the clinician cares for the patient’s psychological and moral wellbeing and not only their physiological health. The duty to have regard to the best interests of the patient and their welfare, taken as a whole, may clash with the patient’s right to choose treatment based on adequate information. To consider provision of information solely in terms of the rights of the patient (and therefore the correlative duty of the health care professional) discounts the “ethical and social dimension of medical treatments” and may potentially harm the relationship. 19 Some factors governing therapeutic privilege include the patient’s personality, temperament or attitude; their level of understanding; the nature of the treatment; and the likelihood of adverse effects resulting from the treatment.
Multiple studies that have attempted to determine and quantify the anxiety-generating effect of informed consent provide mixed results about whether a more detailed consent process is physiologically or psychologically harmful to a patient.
20
Some patients do not have the educational or intellectual ability to understand the choices before them, particularly if the choices are scientifically complex as with AI. Similarly, language and cultural barriers may also impose limits. As stated by the High Court in Rogers, at [14]: “the skill is in communicating the relevant information to the patient in terms which are reasonably adequate for that purpose having regard to the patient's
Clearly, the more complex, scientifically advanced, and intellectually demanding information becomes, the greater the difficulty for patients to provide informed consent. Put simply, there is a point where people who ordinarily have capacity to make their own decisions find it all but impossible to fully understand, and truly informed consent will be impossible. 21
When faced with these complex clinical contexts, clinicians may wonder about the most appropriate ethical and legal conduct. The doctrine of informed consent in specific clinical contexts may need to be looked at through this new prism. Informed consent has become so central and important to the way clinicians practice that there may be situations in which patients’ ability to provide informed consent may be compromised or overlooked, particularly where complex information is involved.
Consider this example: Patient A has been diagnosed with a form of cancer. A sophisticated AI system has recommended a course of drug treatment that is quite particular and unique to Patient A, given all the circumstances and data that is relied upon. It might be aptly described as “personalised” treatment.
Patient A’s prognosis will be much better with the treatment but there are contraindications as with all chemotherapy. Clinician X knows that this a treatment regime that will assist but is unable to understand how the AI system arrived at its decision due to the black box effect, let alone explain it in meaningful layperson’s terms to the patient.
What do they do? Save the patient from themselves at risk of compromising their autonomy based on therapeutic privilege? Or admit that they cannot explain the decision adequately and risk not receiving consent for what will be a necessary life-preserving treatment? Time may be of the essence for the proposed treatment.
Faced with an avalanche of information and unable to make a personal decision, the patient is often reduced to blind trust. To assume that the patient’s decision is based firmly on a complete understanding of the issues, and without outside influence, is neither realistic nor achievable.
Conclusion
There can be no escaping that AI in healthcare has well-arrived and will continue to exponentially increase. It is trite to say that it is a complex integration. Understanding AI poses difficulties to those that design and construct it, let alone the hospitals and clinics that deploy it, the clinicians and others who use it, and the patients who trust and rely upon it.
Where does this leave consent in terms of AI healthcare? The legal and clinical dilemma arises because of the tension between legal duties and constraints, and ethical practice. There must be an analysis on a case-by-case basis. Patients will still need to be given a broad overview but not to an intricate technical level. Clinical validation develops a heightened importance and trust is essential – both in the AI system and in the clinician.
With the development and introduction of AI, arguably there should not be a hardline insistence on obtaining informed consent from patients who are clearly overwhelmed with information due to transparency requirements. Steps should be taken to provide the assistance that patients in each specific situation require whilst affording legal protection to the duty-bound clinician. If this necessitates a tailored multi-factorial approach beyond the scope of this paper, then so be it. Put simply, too much information may be just that – too much for the wrong patient. Consequently, it becomes a finely balanced legal dilemma for the responsible clinician on a duty to warn issue.