Between a logic of disruption and a logic of continuation: Negotiating the legitimacy of algorithms used in automated clinical decision-making

Algorithms as disruption and continuation

As described above, contemporary accounts of the use of algorithms in healthcare, both popular and academic, hold projections of how algorithms will revolutionise and disrupt healthcare. Other scholars have pointed to algorithms often being approached in terms of their opacity and ‘blackboxing’ (Lee and Björklund Larsen, 2019; Schwennesen, 2019). Thus, some critical scholars have an analytical interest in ‘going under the hood’ of algorithms, as Lee and Björklund Larsen (2019) characterise the approach of analytically exposing the assumptions and hidden politics inscribed in the ‘machinery’ of algorithms. This could for example be by drawing attention to the extent to which algorithms ‘blackbox’ the grounds on which decisions are made and results reached, with negative implications for human autonomy, public oversight and justice (Obermeyer et al., 2019; Prainsack, 2017). Along the same lines, the use of advanced big data analytics in healthcare (Lupton, 2018; Prainsack, 2017) and machine learning systems such as for example IBM’s Watson for oncology diagnostics (Tupasela and Di Nucci, 2020), have in particular prompted discussion of algorithms and their ‘authority’ (Lupton and Jutel, 2015; Schwennesen, 2019) vis-à-vis that of the human clinical expert. While this analytical interest takes its point of departure, to some extent, in the popular discourse regarding the disruptive potentials of algorithms, most studies of algorithms in the field of STS have an interest in understanding algorithms as unstable objects (Seaver, 2017) embedded in, and enacted as part of, practices (Lee and Björklund Larsen, 2019; Neyland, 2019: 151; Schwennesen, 2019). In the context of health, Schwennesen (2019), for instance, shows how the authority of an algorithm instructing patients in physical rehabilitation is contingent on active delegation and continuous human ‘repair work’. Lomborg et al. (2020) similarly describe how heart patients engaging with Fitbit’s algorithms for managing their health, mobilise and make sense of the data output in constant interaction with the care infrastructure in which these patients are embedded (Langstrup, 2013). And with regard to PRO-tools, Torenholt and Tjørnhøj-Thomsen (forthcoming) have shown how algorithmic sorting and hence the decontextualised nature of PRO-data entail that nurses mobilise complementary information in order to make PRO-data actionable in clinical decision-making. As pointed out by Lee and Larsen, studies like these offer an alternative to only ‘going under the hood’ and instead consider algorithms as ‘contingent upshot of practices, rather than [as] a bedrock reality with a focus on the intertwining of human and non-human actors’ (Lee and Björklund Larsen, 2019).

Similarly, as an alternative to approaching algorithms as disruptive per se, we suggest a need to consider also the history and the institutionalised practices within healthcare – in which algorithms have, for a considerable period of time, played a significant role – as part of the algorithmic system (Seaver, 2017). Hence, the expectation and hope that automation – an implicit part of applying algorithms in clinical decision-making – will lead to an improvement in healthcare, are not new. On the contrary, Berg in his work from 1997 presents these hopes and discussions, which at that time were already a decade old: how standardised tools, such as protocols and clinical guidelines for decision-support would rectify the problem of large variations and increasing costs in healthcare (Berg, 1997a). A different name for such protocol or guideline – even if on paper – is an algorithm. An algorithm can be defined as ‘an abstract, formalised description of a computational procedure’ (Dourish, 2016: 3). In conversation with early STS work on standardisation (e.g. Berg, 1997b; Bowker and Star, 1998; Star and Ruhleder, 1996) Timmermans and Berg (2003), studied how standards and digitised ‘expert systems’ affect medical practice, and highlighted how the ‘construction and application of standards transform the practices in which they become embedded’ (p. 22). The studies of Timmermans and Berg thereby demonstrate that algorithms, if understood as tools for automated decision-making/support, are not new in healthcare, nor in sociological studies. Rather, a strong tradition of delegating agency to standardised tools in clinical decision-making exists and current developments should be seen in this light. On an analytical level, this challenges the narratives of algorithms as per se disruptive when talking about healthcare and suggests that a sensitivity to the possibility of more continuation-oriented narratives is needed.

To align our analytical sensitivity and allow for both the disruptive and the more continuation-oriented narratives around algorithms in healthcare, we developed an analytical framework. This framework was not solely based on contemporary studies of algorithms within the social sciences and classical STS on standardisation and automation as described above. It also emerged through our empirical observations carried out in clinical practice, in which we observed that algorithms were often enacted as ‘normal procedure’ and not as ‘disruptive’. Thus, in a conversation between the existing theory and these early observations, we developed an analytical frame approaching algorithms in the light of two co-existing logics: a logic of disruption and a logic of continuation.

In applying the term ‘logic’ we follow Mol, as we intend to explore ‘what it is appropriate or logical to do in some site or situation, and what is not’ (Mol, 2008: 8). Hence, with ‘logics’ we do not refer to a stabile coherence, but rather to a coherence that ‘is not necessarily obvious to the people involved (. . .) It may be implicit: embedded in the practices, buildings, habits and machine’ (Mol, 2008). Thus, as opposed to consider a logic as a thought system expressed through for instance a managerial technology (as in the idea of institutional logics) (Allen, 2014), drawing on Mol’s approach we consider how logics come into being, can be identified in, and have an impact on the everyday activities involved in implementing algorithms in clinical work.

Disrupting inefficient work and unsystematic patient involvement

To set the scene, we start by describing how PRO-tools were first introduced in Denmark and how it was especially the framing of the disruptive potential of PRO-tools that led the propagation. Thus, the optimistic coupling of algorithms, automation, and efficiency, which forms part of the socio-technical imaginary we initially described, was also seen in relation to the Danish national investment in PRO-tools. This investment took off after a report in 2015 (PA Consulting Group, 2015), ordered by the Government, Danish Regions, and Local Government Denmark, argued that PRO-data had great potential to be used in automated triage-systems (see also Langstrup, 2019). The report showed that with PRO-data provided by patients through online systems, clinical visits could be scheduled based on individual needs. This differs from how, for instance, follow-up care had been, and still is, typically arranged, where patients are seen, for example, every 3 months based on a generic calendar system. According to the report, PRO-tools could provide a resource-saving and patient-involving alternative: Using algorithms to automate the processing of PRO-data, patients would be classified according to need (green, yellow or red) preselecting which patients should be seen in person for a clinical check-up and which patients should only receive an automated response or a phone call. The main message of the report was a business case, arguing that automation of follow-up visits held immense potential for saving money in a health system with an increasing number of patients in chronic care trajectories or in follow-up care after cancer:

A national use of PRO would imply a reorganisation of the clinical procedures at the outpatient clinics, which means that patients can be referred to PRO-based care; whereupon it is the patients’ responses that constitute the point of departure for the clinical judgement (. . .). In other words, it is a premise that an appointment in the ambulatory assessment unit is not made until a need is identified, on the basis of the PRO [and the algorithmic sorting, ed.]. (Our translation) (PA Consulting Group, 2015)

The consultancy report thus demonstrated how PRO-tools could disrupt previous inefficient ways of organising healthcare, bringing financial benefits. Hence, in line with other reports and strategies on the digitalisation of healthcare produced by health authorities in collaboration with consultancy bodies (Danish Ministry of Health, 2018; Danish Ministry of Health, Danish Regions, Local Government Denmark, 2013), the narratives found here are those of disruption: healthcare needs to change radically: digitalisation, automation, and algorithms provide the key. The consultancy report on PRO-tools influenced and mobilised stakeholders. Thus, from the onset of the Danish PRO-movement, the economic potential and a logic of disruption came to the fore.

However, in the public presentations and strategic documents that followed after the first national introduction of PRO-tools, the value of the data input – patients’ own experience of their health and wellbeing – became the central figure. Economic arguments could not in and of themselves provide legitimacy. As expressed in a report published by the Danish Knowledge Center for User Involvement in Health Care, part of the nongovernmental organisation Danish Patients: ‘With the implementation of PRO-data in clinical practice and in the quality of care development, the aim is to put into reality the vision of involving patients more in their own treatment’ (Trygfonden and ViBIS, 2016: 37).

The potential economic value of automation and the automation itself were downplayed, and little attention given to the algorithms involved. PRO-data and algorithms were consequentially recast, no longer being tools for ‘process automation’, as they appeared to be in the consultancy report, but rather tools for ‘systematic’ and ‘efficient’ patient involvement. These digitised capabilities are referred to in various and less conspicuous ways: ‘data are sent to the database for further processing’; data are used for ‘advanced decision-support’, or in ‘weighting’, or as ‘a smart selection of patients’ – here quoting from a number of internal reports, policy documents and public meetings. The objective around the use of PRO-tools for automated triage therefore seemed to change when PRO-tools were presented closer to clinical practice.

In the so-called clinical coordination groups, consisting of both clinicians and patient representatives who had the responsibility of developing national, cross-sectoral PRO packages including algorithms, the tension between the potential for saving time and money and the potential for patient involvement also appeared. Some clinicians hailed the disruptive potential in economic terms: one clinician stated ‘PRO is smart! We can use it to sort our patients. It’s more rational’. Yet most would only consider an automatic prioritisation of patients if aligned with a goal of increased patient-centredness. The public servant responsible for the workshops was very attentive to this tension:

She [the public servant] says to the group that she has asked in the ministry, and they have ensured her, that this is not a ‘money-saving exercise’. ‘It is to help prioritise – some patients will need more [clinical visits], some will need fewer, she says to the group. (Fieldnote).

Even clinicians who had experienced that the application of algorithms – as the first point of contact with patients – could reduce numbers of controls, were worried that PRO would never gain support from patients and health professionals if perceived as a ‘money-saving tool’. Instead, PRO-tools and the use of algorithm gained its legitimacy based on the assumption that it would ensure and systemise patient involvement.

Despite the gradual shift in rationale for the use of PRO-tools to sort patients, what the examples above demonstrate is that from the onset of the Danish PRO-movement a logic of disruption played a significant role. Both the consultancy report, the political documents and the clinical coordination group expressed a belief that PRO-tools and the delegation to an algorithm would disrupt existing inefficient and unsystematic ways in which healthcare was organised. However, in the following sections, we will demonstrate how this logic of disruption was accompanied by and partly or fully replaced by a logic of continuation as PRO-tools were developed and implemented closer to, and within, clinical practice.

PRO as a continuation of standardised and evidence-based diagnostic procedures

Members of the clinical coordination group at other times discussed PRO-tools not only regarding their potentiality to disrupt unsystematic work procedures, but also in terms of their similarity to established clinical practices. ‘It’s not much different from what we already do’ a physician and member of the clinical coordination group said, referring to standardised procedures and guidelines for interviewing and triaging patients. The algorithmic sorting of patient data is seen as a ‘natural’ prolongation of existing guidelines and best practices, thus expressing a logic of continuation. When PRO-tools are not discussed as something new, but rather as a continuation, it should be seen in the light of standardised and evidence-based procedures which already define clinical work (Allen, 2009; Timmermans and Berg, 2003). In a Danish setting, national guidelines are developed by the Danish Health Authorities with the participation of representatives from relevant scientific societies so that both for heart rehabilitation and breast cancer follow-up, national evidence-based guidelines are developed in order to ensure standardised procedures. Within breast cancer, Danish Breast Cancer Group and their work on preparing evidence-based and national guidelines regarding diagnostic procedures and treatment closely defines the clinical work around breast cancer. Comparable groups exist within other cancer areas.

The same references to ‘business as usual’ were seen in the local PRO-initiative. Here patients’ PRO responses were sorted by an algorithm based on predefined thresholds. To the physician leading the daily management of the project, the automatic sorting performed by the algorithm was comparable to the work she would carry out in face-to-face consultations, when considering how she would habitually ask the same clarifying questions and pay attention to the same symptoms. The physician’s enactment of the PRO-tools as a continuation of regular medical practice, indirectly appeared in terms of what the physician did not tell the patients. In the initial consultation, when patients were briefed about their follow-up care, the physician very rarely mentioned how the task of sorting the PRO responses and the following triage were performed on the basis of an algorithm, with no human involvement. Extra time had been allocated to this consultation as it contained much information and served several purposes including summarising the patient’s treatment, explaining about new preventive medication and radiotherapy, and advising about side effects. Lastly, the physician introduced the follow-up treatment and explained the programme, most often saying:

We’ll follow up on you through questionnaires which you’ll receive by email and where you’ll be asked about the symptoms you’re experiencing. Then a nurse compares your responses with the responses you gave the last time.

The automatic sorting of the responses and automated triage were not mentioned. The physician described how, due to the comprehensiveness of this specific consultation and its many purposes, she was very careful not to give too much information. For her, explaining about the use of automated triage was considered less important information. She instead emphasised the greater importance of informing patients about their treatment, and potential side effects. The physician appeared very dedicated and thorough in her work, and attentive towards the patient’s understanding and any questions raised. When asked why she did not mention or explain the automated screening, the physician said:

The ones who are experiencing symptoms which we are worried about, we get to call those (. . .) But you’re completely right, I consistently didn’t do it [mention the algorithm]. But I don’t believe there’s anything to hide, absolutely not, at any time I would [explain it] if the patient asked. There are a few who have asked about it; ‘How do you do it?’ And then I’ve told them. In that sense, I don’t think there is any jiggery-pokery in it.

The physician prioritises information she considers more important to the patient’s complex and challenging situation. As representing standardised care, the algorithm is established as good clinical practice. The example demonstrates how in clinical practice the algorithm is considered a natural extension of what the physician normally does when applying other standardised procedures.

PRO as a continuation of the physicians’ expertise and authority

In the local PRO-initiative, the PRO-tool – including its algorithm – was developed and introduced into clinical practice as part of an intervention study. The physician leading the daily management of the PRO-initiative had herself led the work of developing the PRO-tool. At first, PRO questionnaires were chosen and formulated based on her professional knowledge and experience. The number of response categories and headings were decided upon based on an already-validated scales. Using her clinical expertise, the physician designed the algorithm by determining what the thresholds should be for each question. The PRO-tool and its predefined thresholds were afterwards sent to six experienced oncologists who commented and made suggestions for improvements in both wording and thresholds. The PRO-tool was then adjusted in discussion between the physician and the six oncologists (Saltbæk et al., 2019). Therefore, to the physician, the algorithmic sorting constituted an extension of her own, and her experienced colleagues’ expertise, which ensured the authority of the algorithm:

I think that it’s a kind of inserted clinical judgement (. . .) I consider it a clinical judgement, which we made when we decided upon the thresholds (. . .) we had six consultants (physicians) contributing by looking at it, and if they didn’t want to call the patient at that time, then I don’t need to know anymore.

Here, the physician emphasises the role that the six experienced physicians played when developing the PRO-tool. To the physician, the algorithm constitutes a legitimate stand-in not only for herself, but also for six other experienced physicians. It is interesting that she uses the word ‘judgement’ as this has traditionally been seen as being in contrast to rule-based decision-making (Dreyfus, 1992; Simon, 1973). Hence, PRO-tools are enacted as an extension and continuation of her own clinical expertise and authority, and not in contrast to expert judgement. At another point in the interview, the physician, using a peculiar expression, described the algorithm as ‘a brain a long time ago’. However, as the quote above illustrates, the algorithm not only comes to represent one brain but the brains of all the healthcare professionals involved in developing the algorithm.

Likewise, the primary investigator behind the project also expressed the algorithm as being more than a representation of one physician. He considered the algorithm as an advance, as it is the result of an agreement between several leading experts. During an interview, Henriette questioned the algorithm’s complexity, referring implicitly to its fixed thresholds and the lack of machine-learning capabilities, something that she as an outsider considered the hallmark of complex algorithms. The primary investigator countered this by saying:

(. . .) we actually made an algorithm for recurrence, which is fairly complicated. Because (regarding) recurrence, we want to see those who we believe have recurrence or dissemination, right? So, it’s rather complicated I believe. It’s been pretty difficult to make. We’ve taken a long time. But I think that we got away with it pretty well.

In this quote, the primary investigator – a physician by training – considers the efforts of the six experienced consultants, and also projects their expertise and authority into the algorithm: its authority again rests on the clinical expertise behind the algorithm. It may take some power of judgement away from individual clinicians, but it rests on and constitutes a continuation of the authority from the expertise of a collective of clinicians (Timmermans and Berg, 2003).

However, when describing the needs for initiating the local PRO-project, the primary investigator simultaneously seemed to draw on algorithms’ more disruptive characters, showing that the two logics should not be seen as mutually exclusive. This appeared at the initiation of the project when he argued strongly for a need to change follow-up practices in oncology as, according to him and supported by research results, these practices rested too heavily on pre-scheduled control-visits and too little on objective indications of recurrences and other needs. He argued that using PRO and algorithms was a necessary disruption of established but costly and inefficient practices and directed a mistrust towards the physicians, who believed that they were effective at finding recurrences, but who – according to scientific studies – did not have a great success rate in spotting recurrences during clinical check-ups. Here, a logic of disruption seems very present. However, as demonstrated in the previous quote from the primary investigator, he simultaneously emphasised the close link between the algorithm and the physicians. This example demonstrates two things: on the one hand, it shows how the physician can tap into more logics, further strengthening the legitimacy for delegation to the algorithm. On the other hand, it may also suggest that the division of the two logics is in some cases too streamlined: that it is too simplistic to believe that a logic of disruption is present only at the political level, or that closer to clinical practice legitimacy is established based only on a logic of continuation.

Moments of tension: Negotiating the transparency of algorithms

Where the examples above demonstrate how the primary investigator seemed to juggle both logics, in this section we will show how during the developmental phase of the local PRO-initiative the logic of continuation was challenged, which momentarily caused some tension and uncertainty in the legitimacy of delegation. Hence, as we took part in the finalising meetings about the design and planning of the PRO-intervention, we observed how questions about the transparency of the algorithm were negotiated. During one of these meetings the IT developers, the physician leading the daily management, and researchers discussed if and how the delegation of decision-making power to the algorithm should appear to the patients, and hence how recognisable it should be that PRO-responses were automatically screened by a computer. The negotiations around transparency emerged as the researchers tried to decide whether patients should receive an auto-generated email shortly after submitting their responses, or if there should be a delay. The following discussion took place during a meeting with the technical company designing the software system running the PRO-tool:

In the example, the physician (the developer of the algorithm) expresses how they should be honest with patients, not letting them assume that she or the nurses had already examined the responses. Her trust in the algorithm, and her associated desire for full transparency, rest on a logic of continuation, as the algorithm represents the authority of a physician and a continuation of how to perform standardised care. Researcher 1 does not represent the same logic of continuation, being concerned with patients’ emotional response to having a ‘machine’ rather than a healthcare professional doing the screening – yet it is not the automatic screening itself that she considers problematic: she feels uncomfortable with how automation will be experienced by patients – such as whether patients will consider this a break with existing practices, and with the trusted relationship between patient and healthcare professional. Nevertheless, Researcher 1 is then convinced by the argument of the physician and tries to make it into a matter of continuation. In this situation, a logic of continuation becomes more persuasive, and it is decided that the email will be sent to the patients immediately after their responses, with no attempt to conceal the algorithm’s involvement.

However, contrary to what had been assumed in the meeting mentioned above, patients did not perceive the immediate-response email as a sign that an algorithm had screened their data rather than a human being. Instead, in interviews with patients it became clear, that the email was often perceived as an autoreply indicating that their data had been received, not indicating anything about who or what had been involved in evaluating their data. And as the physician only very rarely informed them about it, most patients had an inaccurate understanding of how their data were evaluated. A few patients had figured out that some sort of automatic screening probably took place, but none of the patients Rikke interviewed were fully aware of the two stages of the screening process: the first screening by an algorithm and the second by a nurse, and that the nurses only saw data from the patients not screened out in the first stage. This was also the case with Janne, who Rikke asked as to what she thought happened to her responses after submitting them.

I have the impression that a nurse looks at them [the responses], I don’t know if the physician maybe also looks at it, just to see if there is anything [the responses] that stands out, is there anything we [the healthcare professionals] should pay attention to? And then, if it’s something that we’ve already talked about, well, then that’s just it. And if there’s anything new, well, then they get hold of me. That’s sort of my impression of what happens.

This perception, that a nurse always received their response, was present among most of the patients interviewed; the delegation to algorithm remained unknown to them. To the physician and the nurses, the use of algorithms is enacted as a continuation based on their professional experiences and beliefs; while to the patients, treatment is perceived as ‘business as usual’, since they are unaware of the algorithmic sorting. It could be that patients too would consider the use of the algorithm as a matter of continuation, if they were aware of its function and the reason for delegation to an algorithm. However, it may also be that patients would actually perceive it in more disruptive terms, as predicted by Researcher 1.