Mors ex Machina : Technology,Embodiment and the Organization of Destruction

Introduction

‘In the past,’ wrote Frederick Winslow Taylor (1911) ‘man has been first. In the future the system must be first.’ Henceforth, ‘man’ – and ‘his’ current successor ‘the human’ (resource) – would constitute merely a component, often unreliable, of the system. More than a century later, the shadow of ‘the system’ still falls heavily on the field of organizational studies, particularly on its uneasy engagement with ethics. In mainstream accounts (e.g. Cohen, 2010; Dunlop & Lee, 2003) organizations’ ethical credentials are presented as constantly imperilled by the moral shortcomings of individual members (‘bad apples’, ‘rogue agents’, ‘unreliable managers’, etc.) who must therefore be constantly monitored and re-trained to ensure compliance with corporate ethical codes (e.g. Cohen, 2010; Tang, Chen, & Sutarso, 2008; Treviño & Youngblood, 1990; Zahra, Priem, & Rashid, 2005). In critical accounts (e.g. Jones, Parker, & ten Bos, 2005; Wray-Bliss, 2009, 2012), on the other hand, organizations are seen as intrinsically geared towards the displacement of moral responsibility; towards ‘neutralizing the disruptive and deregulating impact of moral behaviour’ (Bauman, 1989, p. 215). Here, it is precisely the systemic nature of organization which, aided by modern technology, facilitates this process of ‘adiaphorisation’, enabling moral responsibility for one’s actions to be diffused so that organizations end up becoming ‘instrument[s] to obliterate responsibility’ (Bauman, 1989, p. 163; Arendt, 1969).

In this article we set out to re-examine some of the issues raised by the still recent ethical turn (Parker, 1995) in organization studies and the way they intersect with other longer-standing concerns, such as embodiment and the role of technology (Hassard, Holliday, & Willmott, 2000). However, we do so from the viewpoint of a form of organized activity that has hitherto received relatively little consideration within the discipline – namely, the organization of destruction. Pickering (2001, p. 165) has argued that social scientific accounts of how agency unfolds need to pay special attention to the ‘great, enduring and conspicuously visible sites of encounter of human and nonhuman agency, such as the factory (standing for the whole field of organized production) and the battlefield (standing for organized destruction)’. Historically of course, the discipline of organization studies has had a lot to say about factories and organized production but considerably less to say about battlefields and organized destruction. Indeed, aside from miscellaneous studies of military organization (e.g. Catino & Patriotta, 2013; Roberts, 1990; Weick & Roberts, 1993), most of the important contributions in this area have tended to come from beyond the boundaries of the discipline (e.g. Bauman, 1989; De Landa, 1991; Virilio, 1989, 2000). Perhaps this is not altogether surprising. It could be argued, for instance, that in organization studies, and organizational ethics in particular, the very notion of an organized destruction constitutes an ambiguous, even illegible, object. Thus, in studies of the military for example, it is ‘the organization’ that tends to be the object of inquiry, while that which is being organized (i.e. the destruction of human bodies and lifeworlds) tends to – as a rule – remain in the shadows (see Banerjee, 2008; Burrell, 1998; Godfrey, Lilley, & Brewis, 2012; Godfrey, Brewis, Grady, & Grocott, 2014 and Grey, 2009 for some notable exceptions). As a way of beginning to address this absence, this article undertakes a critical exploration of an ongoing programme by the US Army Research Office led by Ronald C. Arkin (2009), which seeks to ensure adherence to ethical codes against the backdrop of the rapidly expanding military use of robotic systems.

Armed robotic vehicles and unmanned aerial vehicles are now being deployed or being developed by many militaries from the ‘First World’ and beyond. Indeed, over the last decade the forms of ‘action at a distance’ that such systems facilitate have emerged as key dimensions of whatever we might mean by ‘globalization’ (Bauman, 2001; Blount, 2011). Although today’s robotic systems are, on the whole, remotely operated, there is a widespread expectation that, like financial market systems before them, such technologies may in the near future acquire considerable autonomy – here in making life-and-death decisions. For instance, USAF’s Unmanned Aircraft Systems Flight Plan 2009–2047 predicts deployment within that timescale of fully autonomous aerial vehicles, where humans will play the role of ‘monitoring the execution of decisions’ rather than actually making those decisions themselves: ‘advances in AI [artificial intelligence] will enable systems to make combat decisions and act within legal and policy constraints without necessarily requiring human input’ (USAF, 2009, p. 41).

Similarly, a 2011 UK Ministry of Defence report (MOD Joint Doctrine Note 2/11) claims that it is

only a small technical step to enable an unmanned aircraft to fire a weapon based solely on its own sensors, or shared information, and without recourse to higher, human authority. Provided it could be shown that the controlling system appropriately assessed the [Law of Armed Conflict] principles (military necessity; humanity; distinction and proportionality) and that [Rules of Engagement] were satisfied, this would be entirely legal.

The push towards autonomous systems resonates with the drive to reduce the significant economic costs of training military personnel (particularly pilots) in times of austerity as well as the increasing Western sensitivity towards the tragedies of war and the political need to avoid their (‘our’) own military casualties (The Economist, 2007). At the same time, the dawn of the age of autonomous killing machines – as represented by artefacts such as the US Navy’s new $813 million X47B ‘fully autonomous’ armed drone – has understandably stoked fears and given rise to moral and political concerns. Jakob Kellenberger, President of the International Committee of the Red Cross, for instance, is quoted in the Los Angeles Times arguing that

The deployment of [armed autonomous systems reflects] … a major qualitative change in the conduct of hostilities … The capacity to discriminate, as required by [international humanitarian law] will depend entirely on the quality and variety of sensors and programming employed within the system … policymakers must deal with these ethical questions long before these lethal autonomous drones go into active service. (Hennigan, 2012)

Set against these developments, Ronald C. Arkin, an expert with a distinguished career in robotics, has proposed a controversial design for an ‘ethical governor’ as part of an architecture which will enable future autonomous military systems to use lethal force while adhering to the Law of Armed Conflict or Laws of War (LOAC or LOW) and the Rules of Engagement (ROE) more closely than human combatants (Arkin, 2009, 2011; for some critical responses see: Asaro, 2009, 2012; Sharkey & Suchman, 2013; Sparrow, 2009; Suchman, in press; Sullins, 2010). Arkin and his co-workers argue that unlike soldiers – whose embodiment and associated behavioural drives/imperatives (which they construe as flaws) means that they cannot maintain adherence to their own ethical codes – suitably programmed ‘ethical robots’ will be exempt from fear, anger or stress-induced scenario fulfilment. For this reason, such robots could not only be entrusted with life-or-death decisions but they could also be relied upon to identify and report human violators of the Laws of War and the Rules of Engagement. The Ethical Warrior Robot (our term) thus exemplifies, perhaps paradoxically, a double commitment: to what we might call a humanitarian ethics and, simultaneously, to a transcendence or displacement of the human subject.

In one of his fantastical stories Italo Calvino (1959) narrates the tale of the non-existent Knight, an empty medieval suit of armour that conducts itself in strict accordance with the code of chivalry. Modern technological societies, we might say, no less than medieval ones, have their own fables and the ethical warrior robot is a particularly resonant one. In the 21st century, it appears, computer code has come to be seen as the most reliable means for upholding an ethical code (Mackenzie & Vurdubakis, 2011). Therefore, one way of understanding Arkin’s (2011) system is by asking the question: how did the ethical warrior robot’s suit of armour come to be empty?

The structure of our argument is as follows. The next section seeks to make sense of this disembodiment in the light of recent work on organizational ‘agencements’ and on the various human-technological convertibilities and miscegenations that such ‘assemblages’ encourage; whether in the organization of production or that of destruction (the so-called ‘revolution in military affairs’). The subsequent two sections provide an account and an analysis of how this disembodiment is described and justified in the technical and promotional writings of Arkin and his co-workers. As we shall show, however, the disposal of the body turns out to be a fraught enterprise, constantly plagued by questions of lack and excess prompting in turn constant recourse to (Derridian) deferrals and supplementations. Against what certain strands of post-humanist theorizing might lead us to expect, such assemblage-work does not render human-machine dualisms and distinctions irrelevant but, on the contrary, remains haunted by them. Prompted by this, we propose that the attempted disposal of the body (and of the ethical centrality of the body) needs to be understood not (merely) in terms of its instrumental effectiveness in rendering the organization of destruction more manageable, but also as an occasion for the rehearsal of the moral and philosophical conflicts characteristic of Atlantic Modernity. Finally, our concluding discussion revisits the main threads of the argument and the need to open up the organization of destruction as a topic in organization studies.

The Machine at War

‘Nations make war the same way they make wealth’ (Cebrowski & Garstka, 1998, p. 29)

In recent years, a number of commentators have stressed the need for ‘ontological inquiry into what [moral] agency means’ in organizational settings (Painter-Morland, 2011, p. 84; our emphasis). Indeed, what often appears to be at stake in debates over organizational (Phillips & Margolis, 1999) ethics is the status and attributes of moral agency. For example, the commonplace reliance on ‘management’ as the (self-evident) agent of organizational ethics (e.g. Cohen, 2010; Deckop, 2006; Treviño & Nelson, 2007; Treviño, Weaver, Gibson, & Toffler, 1999) is questioned in critical accounts, which stress that it has the effect of ‘further removing ethical responsibility from individual organizational members’ (Wray-Bliss, 2012, p. 590; 2009). Relatedly, the ethical formalism of mainstream accounts, with its emphasis on ‘formal rules for the determination of behaviour’ (Atkinson & Butler, 2012, p. 17), is seen to go hand-in-hand with a neglect of ‘the bodily and embodied character of moral practice’ (Hancock, 2008; Kuepers, forthcoming). Thus, critical accounts increasingly seek to challenge this denial of ‘the ethical centrality of the body’ which such ‘rational and self-interested uses of the ethical to pursue corporate interests’ entail (Pullen & Rhodes, 2014, p. 6; our emphasis). As Rosalyn Diprose (1994, p. 16) puts it, the ‘ethical’ should not be understood (merely) in terms of the ‘logical status of moral judgements’ or as the setting down (and adherence to) ‘a set of universal principles for regulating behaviour’ but as ‘the problematic of the constitution of one’s embodied place in the world’ (see Hancock, 2008; Pullen & Rhodes, 2014).

Such issues acquire additional urgency in the context of what many view as the increasingly ‘posthuman’ (e.g. Braidotti, 2013; Hayles, 1999) forms which contemporary ‘cyborganizations’ (Parker, 1998) take. Recent studies of the organization of financial markets for instance (sites which in many ways exemplify these developments) have sought to place in question any hard-and-fast ontological distinctions between human and nonhuman agencies. Instead, emphasis is placed upon their various enfoldings, enmeshments and imbrications in Deleuzian assemblages or agencements (e.g. Beunza, Hardie, & MacKenzie, 2006; Raviola & Norbäck, 2013). As will be recalled, for Deleuze (2007, p. 177), ‘in assemblages you find states of things, bodies, various combinations of bodies, hodgepodges; but you also find utterances, modes of expression, and whole regimes of signs.’

An assemblage therefore is

first and foremost what keeps [such] very heterogeneous elements together: e.g. a sound, a gesture, a position, etc., both natural and artificial elements. The problem is one of ‘consistency’ or ‘coherence’, and it is prior to the problem of behavior. How do things take on consistency? How do they cohere? Even among very different things, an intensive continuity can be found. (Deleuze, 2007, p. 179)

Thus, the ‘assembling [of] an economic actor’, for instance, is seen to involve ‘human beings, technical devices, algorithms and so on’ (Hardie & MacKenzie, 2007, p. 57). These heterogeneous elements are brought together in specific agencements ‘from which action springs’ (p. 57). In line with this work then, we might view Arkin’s ethical warrior robot as the ‘assembling of a moral actor’, an ideal(ized) agent of organized destruction.

In many ways, the ethical warrior robot’s attempt at an ethics completely divorced from the human body seems like a parody of the managerialist ‘formalism’ critiqued by Hancock (2008) or Pullen and Rhodes (2014). How then can we begin to make sense of this kind of ontological move? Over 30 years ago Paul Virilio noted the early signs of a withdrawal of the (Western) body from the workings of war-fighting apparatuses and highlighted the evident parallels with capital’s gradual abandonment of labour in the organization of production:

Robots and computers will take care of production. War is automatized, and along with it the power of decision. They no longer need men, soldiers or workers, only means of absolute extermination, on the commercial level as elsewhere. (Virilio, 1983, p. 102)

As (US Vice-Admiral) Arthur Cebrowski and John Garstka sum it up in their 1998 Naval Institute report, the same forces which have ‘changed the nature of American business today…have changed and will continue to change the way we conduct the sometimes violent business of the military’ (p. 29). By the close of the 20th century this process of reorganization of destruction had acquired its name: a Revolution in Military Affairs (RMA) (e.g. Lefebvre, Fortmann, & Congora, 1996). We can sum up this ‘revolution’ in terms of three closely interconnected processes: first, changes in the technological apparatuses of warfare; second, changes in the conduct of warfare; and third, claimed changes in the objectives of warfare. Changes in the technological apparatuses of warfare are most visibly exemplified by the drones (UAVs – unmanned aerial vehicles) and other ‘smart’ weapons and positioning systems with which ‘First World’ militaries have sought to ‘shock and awe’ their enemies abroad and television audiences back home. UAVs, ‘smart weapons’, stealth technologies and the rest of the apparatus of the RMA reflect attempts to engineer a redistribution of the risks of warfare by instituting ‘the technical capability and ethical imperative to threaten and if necessary, actualise violence from (a “safe”) distance – with no or minimal casualties’ on the US side (Der Derian, 2009, p. xv). Western ‘risk societies’, we might say, have increasingly sought to ‘let robots do the dying’ (Ramo, 2011) on their behalf. In the course of this revolution the human body comes to be increasingly problematized – as Taylor (1911) had prophesized – as an intrinsically unreliable component of the war machine.

Given the increased pace of warfare, humans have in some respects become the weakest link in the military arsenal and are thus being taken out of the decision-making loop. The reaction time of autonomous systems far exceeds [is superior to] that of human beings. (Heyns, 2013, p. 10)

At the same time, Euro-American technical superiority is said to underwrite a moral superiority. Thus the technological apparatus of the RMA is frequently described (contra Bauman, 2001) as the vehicle of an ongoing ‘humanitarian revolution’ (Pinker, 2011; Singer, 2011) in both the conduct and the objectives of organized destruction. We are told for instance that ‘The whole point of drones is to minimize loss of life compared to indiscriminate forms of destruction’ (Pinker, 2013; our emphasis).

In the wake of atrocities like the Mai Lai massacre, demands have intensified in the West that war should no longer be granted exemption from the moral standards and norms of conduct which apply to the rest of social life:

we may now be applying to war the standards from which it has previously been exempt. ‘Thou shall not kill’ has been tightened as a social norm, with fewer and fewer exceptions allowed; many Western states even decline to impose the death penalty. (Shaw, 2005, p. 317)

The ‘ethical warrior’, as the The Marine Corps Gazette would have it, is now to be ‘a protector of life’ (Hoban, 2010, p. 20; our emphasis). Thus, the end of the Cold War (and, it was claimed, of history) saw the objectives of warfare increasingly rearticulated in the language of protecting human rights (e.g. Kosovo, Libya) and of humanitarian intervention (e.g. Somalia). US/Western military commanders now could, it was said, ‘not only keep their own casualties levels low, but also respect the expectation – bordering on moral presumption – that fire will be directed with precision and only against targets of evident military value’ (Freedman, 1998, pp. 76–7). ‘By narrowly targeting our action against those who want to kill us and not the people they hide among’, argued President Obama (2013) in a recent speech, ‘we are choosing the course of action least likely to result in the loss of innocent life’.

The current round of North–South conflicts that are said to comprise the still ongoing ‘War on Terror’ has seen an escalation of ‘targeted killings’ carried out by UAVs. While the objective of this strategy is to eliminate clearly defined ‘high value targets’ (HVTs, such as high-ranking Al Qaeda terrorists) it has, for all the rhetoric of ‘targeting’ and ‘precision’ led (inevitably) to numerous (and occasionally well-publicized, e.g. Benjamin, 2013; Cloud, 2011) civilian casualties. For instance, Hudson, Owens and Flannes (2011), in their alternative analysis of ‘collateral’ death statistics relating to UAV operations in Northern Pakistan, point to an alarming increase in the numbers of ‘innocent lives’ lost. As Hudson et al put it, when ‘five or six people were killed for each defined high-value target’ (as was the case in 2002–04) these could be said to be ‘part of that high-value target’s immediate entourage’ and thus ‘more likely to be militants than civilians’. With the increase in such ‘collateral’ deaths to 66 per HTV in 2008–09, and to 147 per HVT in 2009–10, this is highly unlikely to still be the case. This then undermines the distinction between terrorists and anti-terrorists, ‘Them’ and ‘Us’ – i.e. the notion that ‘terrorists’ target ‘innocent civilians’ while ‘we’ (Euro-Americans) only kill non-combatants accidentally. ¹ Framed in this way, the loss of ‘innocent lives’ comes to appear not as an inevitable part of what Aeschylus long ago described as ‘war’s harvest of sorrows’, but as problems of good organization amenable to technical-managerial fixes – for instance, in the shape of more accurate or discriminatory systems of destruction. ²

Body Trouble

‘The ethical warrior is a protector of life. Whose life? Self and others. Which others? All others.’ (The Marine Corps Gazette, Hoban, 2010, p. 20).

The imbroglios of Iraq and Afghanistan have seen the ideal of the ‘ethical warrior’ ³ engaged in humanitarian warfare falter. The call for an ethical warrior robot, then, should therefore be understood against the backdrop of a mounting civilian death toll together with the reports of war crimes carried out by Western military personnel generating revulsion and political fallout throughout the world.

Is it not our responsibility as scientists to look for effective ways to reduce human inhumanity to other people through technology? And if such inhumanity occurs during warfare, what can be done? (Arkin, 2009, p. 33; emphasis in original)

What then is the code of ethics of the flesh-and-blood soldier and in what ways is it seen to be ‘failing’? Official discourses hold that the Laws of War (LOW) are incorporated into the Rules of Engagement (ROE) that soldiers must conform to. Thus US soldiers marched into battle during operation ‘Iraqi Freedom’ (the Second Gulf War) carrying with them cards encoding their Rules of Engagement: for example, a soldier has the right to defend themself; a soldier is not allowed to intentionally target or otherwise mistreat non-combatants; civilian property should be respected, and so on. Arkin (2011) draws on a report by the Surgeon General’s Office Mental Health Advisory Team (2006) reviewing the ethics of US soldiers and marines during that operation which, among a number of related findings, documents disconcerting ‘attitudes’ regarding the treatment of non-combatants (or their property) as well as a lack of compliance with ethical standards or the inability to apply such standards to specific situations they had been confronted with. Here is a sample:

And so on. Such attitudes are highly problematic, particularly when military personnel are expected to win ‘hearts and minds’, to be ‘nation builders as well as warriors’ (Petraeus & Amos, 2009, p. xiii). In line with many mainstream views of corporate ethics, the ethical behaviour of the organization is seen here as in danger of being compromised by the failure of particular members (bad apples?) to comply with the behavioural requirements of their organizational ethics codes. Unethical behaviours are thus perceived as symptoms of disorganization – manifestations perhaps of what Godfrey et al. (2012, p. 559) call (after Burke, [1935] 1992) an ‘unfit fitness’ – where the soldiers’ military training renders them unfit for their present duties.

In warfare, Arkin (2011) argues, human emotions are potentially sources of both strength and weakness for the soldier. Strength because it allows for empathy (arguably an important characteristic in situations where the exercise of lethal force is involved); but also a weakness insofar as it might prevent a human from carrying out an authorized and legitimate action, or on the other hand drive a person to the excessive use of force (in revenge, for instance, for the loss of a comrade). ‘People,’ Arkin (2011) concludes, ‘have not evolved to function’ in the demanding and stressful conditions imposed by the battlefield – a problem that only becomes further exacerbated as warfare becomes more fast-paced and information intensive (e.g. Adams, 2001). ‘Robots’, however, ‘can be engineered to function well in them’ and eventually the (autonomous) ethical warrior robot can be capable of behaving in ‘a more humane manner than even human beings’ (Arkin, 2009, p. 31; our emphasis). Incapable of emotions, and deliberately so, the robot would neither be impelled by the blood lust of battle nor driven to questionable or disproportionate actions through fear or the fight for self-preservation. Ironically then, it is humans’ very humanity that prevents them acting humanely. Indeed, Arkin (2011) highlights a number of real-world conflict scenarios where it can be argued that military personnel behaved in questionable ways. For instance, he describes an incident in the Iraq war where a US Apache helicopter engaged three men on two trucks; these were designated as insurgents involved in suspicious activity (e.g. about to place an improvised explosive device by the roadside). The Apache crew destroyed one of the trucks, killing two of the men and wounding the third. There was then some deliberation among the crew on whether to target the second truck, after which the third man (now arguably unarmed and posing no threat) was deliberately targeted (along with the second truck) and executed. Arkin uses this and other scenarios to demonstrate how an autonomous robot built to his design would have acted differently (that is to say ‘ethically’) and, in this case, would have sought to further determine the level of threat posed by the wounded insurgent and (following the Laws of War), had he been deemed no longer a threat, would have spared him. By extension, Arkin envisages a robot with the ‘right to refuse an order which is determined (by the robot) to be unethical’ (Arkin, 2009, p. 33). As a programmed machine, the robot would be capable of giving an account of its actions or inactions (cf. Sacks, 1963) – its reasoning would have to be legible – and furthermore the capacity to monitor and assess (in)action would extend to the (mis)behaviour of other (i.e. human) combatants. Hence, not only would the ethical robot decline to commit illegal acts but it could also serve as a deterrent to any humans operating alongside it. Thus the unreliable, trigger-happy soldier (see Godfrey et al., 2012) who fails to live up to the ‘ethical warrior’ ideal is to be replaced by the ethical warrior robot. This will allow, it is conjectured, the system to better uphold the distinction between combatants and innocent civilians, and through it the distinction between ‘Us’ and ‘Them’ (who fail to observe such distinctions). Organized destruction, it appears, cannot be carried out ethically unless the suit of armour is emptied, the human body transcended.

‘Grown Up and Encased in a Suit of Armor’

As Mary Douglas (1966) has famously argued, different forms of social organization can be best understood by attending to how they dispose of whatever is deemed to be ‘out of place’. Thus, there is a by now significant body of research in organization studies which has sought to understand contemporary organizations as ‘architectures of disposal’ (Munro, 2001, p. 110). Cooper (1986, p. 328) for instance describes ‘organization’ in terms of the processes through which ‘order is extracted as form’, and whatever is taken to represent disorder is ‘refused as non-form’ (our emphasis; see also Bloomfield & Vurdubakis, 1999; Bloor, 1978; Knox, O’Doherty, Vurdubakis, & Westrup, in press; Thompson, 2008). We have already referred or alluded to how, in what we might call (after Parker, 1998) the ‘cyborganization’ of destruction, these dramas of displacement come to constitute occasions for the rehearsal of the moral and philosophical dichotomies – human versus machine, free will versus determinism, body versus mind, self versus other – characteristic of Atlantic Modernity (Pfaffenberger, 1992).

Consider, for instance, the question of empathy as a key dimension of ethical judgement (Prinz, 2007). Echoing Bauman’s (2001) argument regarding adiaphorization, Joseph Weizenbaum’s Computer Power and Human Reason (1984) argued that it is precisely the lack of genuine (rather than simulated, as in Picard, 1995) empathy that renders artificially intelligent machines unsuitable for the performance of particular roles such as judge or therapist. Reflecting upon a conversation with Weizenbaum, Pamela McCorduck (2004, p. 324) recalls, ‘I probably spoke for most women, minorities, and others excluded from power in my own culture when I said that I’d rather take my chances with an impartial computer.’ Indeed, we might note that empathy (from the Greek empatheia: physical affection, passion, partiality) often gets in the way of fairness and the administration of impersonal justice. The implication of McCorduck’s position is that a machine might in this sense realize the ideal of impersonality that has always eluded Weberian bureaucracy.

Nonetheless, reason without empathy – a quintessentially human quality – is strongly reminiscent of the ‘cold’ implacable reason of the killer robots that populate science fiction, such as the cinematic Terminator(s) (e.g. dir. Cameron, 1984) or Philip K Dick’s (1952) Second Variety. Such tales can be read as caricatures of the RMA strategies of disembodiment and action-at-a-distance; what Human Rights Watch (2012) calls the ‘ongoing dehumanization of warfare’. Indeed, as cultural studies of organization (e.g. Hassard & Holliday, 1998; Parker, 2002; Parker, Higgins, Lightfoot, & Smith, 1999) have argued, the importance of representations of technological and organizational developments in popular culture must not be overlooked. Not surprisingly then, the idea of out-of-control ‘killer robots’ has increasingly served to focus opposition on autonomous military systems (such as the Campaign to Stop Killer Robots; see www.stopkillerrobots.org). Government and military commentators too acknowledge the prevalence of such cultural imagery:

In theory…autonomy should enable more ethical and legal warfare. However, we must be sure that clear accountability for robotic thought exists and this in itself raises a number of difficult debates… There is a danger that time is running out – is debate and development of policy even still possible, or is the technological genie already out of the ethical bottle, embarking us all on an incremental and involuntary journey towards a Terminator-like reality? (UK Ministry of Defence, 2011, pp. 11–12)

Similarly, a report prepared for the US Office of Naval Research stresses that in science fiction:

the use of robots in society is [usually] in tension with ethics and even the survival of humankind. The public, then, is already sensitive to the risks posed by [autonomous military] robots – whether or not those concerns are actually justified or plausible. (Lin, Bekey, & Abney, 2008, p. 9).

Likewise, the RAF Air Power Review argues that

media reporting, has engendered much speculation and discussion, often not well-informed…Beyond the, unlikely, nightmare of a ‘Terminator’-style killer robot running amok, there are some fundamental questions that must be addressed as we integrate increasingly autonomous weapon systems into the battlespace. (Blount, 2011, p. 32)

Arkin’s ethical warrior robot (the humane non-human) cannot, it appears, rid itself of the image of science fiction’s unfeeling killers.

In a discussion of disposal that has a number of commonalities with our own, Munro (2001, p. 129) notes the ‘impossibility of actually disposing of meaning – as if by volition’. Munro’s comments occur in the course of an ironic critique of postmodernism. Noting the ‘supposed omnipresence of meanings’ in postmodernist accounts of social life, he poses the question of how it is that meanings ‘do not just accumulate and jumble up the world?’ (Munro, 2001, p.128). While for Munro this is an ironic move – he draws an analogy with the well-known convention in crime novels where the effective disposal of the body is seen as the best means of making the crime itself disappear – our argument is that in this case there is indeed a crime to be concealed. ‘Humanitarian War’, we suggest, routinely seems ‘jumbled up’ – to opponents and proponents alike – and therefore the disposal of conflicting ‘meanings’ appears not as a postmodernist conceit but as a pressing practical concern. Accordingly, Arkin’s ethical warrior robot can itself be viewed in this light as an architecture of disposal.

Arkin’s project, it will be recalled, is to ensure compliance with the Laws of War and Rules of Engagement via the introduction of constraints to the behaviour of the autonomous robots currently under development by the military. This is in the form of an ‘ethical governor’ – imagined as analogous to the mechanical governor of Watt’s steam engine (a long-standing cybernetic icon). Arkin’s design is complicated and multifaceted, incorporating many considerations that are beyond the scope of the present article. There is however a foundational aspect of the design as a whole that merits close attention here. To get to that point we first need to consider his framing of the basic problem that the robot is designed to solve. Within the context of a military engagement let’s imagine an autonomous robot in receipt of various stimuli (i.e. data derived from its sensors) and that above a certain threshold value (that which would be sufficient to enable the identification/discrimination of hostile enemy combatants, for instance) these evoke a set of possible responses (including lethal actions such as firing a weapon against a target identified as the hostile source of the stimuli). This part of the architecture is the ‘reactive’ subsystem. What underpins the claims to ‘ethical behaviour’ on the part of the robot is the ability to assess the set of possible courses of action while ‘governed’ by a set of ethical constraints; for example, ruling out those behaviours that are expressly forbidden (such as the ‘killing of civilians’, ‘unarmed’, ‘surrendering’ or ‘wounded’ combatants, or exercising force against targets in close proximity to an object of cultural significance, etc.) and thence to select an appropriate/permissible response (including the possibility of inaction) consistent with the requirements of military necessity and proportionality. ⁴ This ability represents a deliberative part of the design, hence Arkin’s description of the robot as possessing a ‘hybrid reactive/deliberative’ architecture (Arkin, 2011, p. 21). Before it can exercise lethal force the robot must conform to the following procedural steps: ‘ASSIGN RESPONSIBILITY (A priori). ESTABLISH MILITARY NECESSITY. MAXIMIZE DISCRIMINATION. MINIMIZE FORCE REQUIRED (PROPORTIONALITY+DOUBLE INTENTION)’ Arkin (2011, p. 58).

First, it must be authorized to use lethal force by the humans in the military situation; second, it must determine that the use of force is necessary from a military point of view; third, it must ensure the maximum discrimination between combatants and non-combatants; and fourth, it must use the minimum of force, this constrained both by the principle of proportionality, the intention to neutralize the threat determined in step 2 as well as that of avoiding collateral damage (e.g. to non-combatants, cultural and other objects/property). These procedures can be seen to reflect the current organization of the US military’s ethics compliance bureaucracy. With its procedures enshrined in technological form, it is conjectured, a higher ethical standard will be met than that reached by humans who are expressly meant to follow such procedures but whose corporeality, all too often, ‘gets in the way’.

Arkin’s design is not meant to mimic the ethical reasoning of a human being in that

we do not want the agent (i.e. the robot) to be able to derive its own beliefs regarding the moral implications of the use of lethal force, but rather to be able to apply those that have been previously derived by humanity as prescribed in the LOW and ROE. (Arkin, 2011, p. 55; our emphasis)

And again: ‘Nor is it particularly relevant that the same models of ethical reasoning that are postulated for humans be applied to battlefield robots, especially given the typical failings of humanity under these extremely adverse conditions’ (Arkin, 2011, p. 55).

Thus the autonomous robot must be guided by rules implemented according to the Laws of War and Rules of Engagement; it is these that determine the set of constraints governing the robot’s responses. The ethical governor would therefore be given the task of conducting ethical evaluations of the set of responses derived from the reactive subsystem so as to ensure that only permissible responses can be followed through by the robot. The operation of the ethical governor would thus involve the application of a set of constraints [C] derived from the Laws of War so as to differentiate between lethal responses that are permissible (i.e. ethical) and those that are not (and are therefore unethical).

The set of ethical constraints C defines the space where lethality constitutes a valid and permissible response by the system. Thus the application of lethality as a response must be constrained by LOW (laws of war) and ROE (rules of engagement) before it can be used by the autonomous system. (Arkin, 2011, p. 18)

Represented by Arkin in set-theoretic terms, the task essentially involves drawing a clear boundary to demarcate permissible lethal responses from non-permissible lethal responses – as illustrated in Figure 1.

OPEN IN VIEWER

Figure 1.

(a) P – Set of Possible Robot Responses. (b) Permissible and Non-Permissible Responses.

The set of possible responses by the robot is denoted by P [see Figure 1(a)]. This contains a subset of responses that are lethal [P_lethal] which in turn includes a subset of responses that are both lethal and ethical [P_l-ethical].

Applying the constraints C to the set of responses P [see Figure 1(b)] is meant to allow the separation of the permissible [P_permissible] from the non-permissible and in particular the lethal responses that are ethical [P_l-ethical] from those that are not [P_l-unethical], such that only permissible responses are allowed. In short, the design assumes the possibility of, as it were, abolishing the ‘grey areas’ and thus bifurcating the set of lethal responses, to evaluate them unambiguously so as to assign them to either one subset (permissible) or the other (non-permissible). Getting rid of the ‘killer robot’ image then has been translated into the seemingly more tractable problem of the disposal of the lethal unethical responses [P_l-unethical].

The Fog of War

The problem however may not be as easily solvable as it first appears. A number of practical questions are raised by Arkin’s overall design. These include what defines the set of constraints; how are they operationalized within the robot’s design; and how are they enacted in practice? Arkin acknowledges that the various representational formalisms and ethical frameworks for deriving a set of constraints from laws or moral codes are subject to several flaws – including imprecision, variable meaning across contexts, conflicting rules, etc. – but nonetheless is optimistic about the prospects of lessening them. ‘It is my belief that battlefield ethics are more clear-cut and precise than everyday or professional ethics’ (Arkin, 2011, p. 39). Against that there is of course the view that the Laws of War are not as transparent as his design presumes. As Asaro (2009, p. 21) notes, what we have in practice is:

a menagerie of international laws and agreements (such as the Geneva Conventions), treaties (such as…the anti-personnel land-mine ban) and domestic laws regulating the procurement, design, and use of various weapons and tactics. By the very nature of law, these rules are open to challenges and interpretations in various courts, and may not be effectively enforceable.

Though acknowledging the cogency of that line of critique, let us for speed of argument sidestep it and proceed as if such a coherent, clearly expressed and internationally accepted corpus of rules governing armed conflict did indeed exist. This would, theoretically, make the task of building an ethical robot more straightforward; but even so it would still face another crucial problem – that of knowing how such rules (etc.) actually pertain to any given situation that confronts it (e.g. Garfinkel & Sacks, 1970). In other words, such laws and rules, while necessary, are not sufficient by themselves. A key issue concerns their interpretation in the context of any given battlefield situation. The principles (e.g. ‘Anticipate attack. Use force if, but only if, you see clear indicators of hostile intent’ [Arkin, 2011, p. 69]) and categories (‘combatant’, ‘non-combatant’, ‘armed’, ‘unarmed’, ‘surrendering’, ‘wounded’, etc.) of the ethical framework built into the robot might indeed be clearly represented. However, reliably recognizing a specific situation in terms of these categories is likely to prove intractable, especially in the current forms of North–South conflict in which the identity of the enemy (‘insurgents’ in the war-on-terror terminology) is intentionally difficult to differentiate from the civilian population (‘non-combatants’). After all, perhaps the key dimension of guerrilla warfare is precisely that guerrillas seek to make themselves indistinguishable from the surrounding population out of which they emerge, and back into which they melt, as the exigencies of conflict demand (Taber, 1972). Any information therefore as to whether a potential target is, or is not, an ‘enemy combatant’ is irredeemably ambiguous and likely to remain so. This problem is replicated when it comes to situations for which the set of constraints [C]) imposed on the behaviour of the robot do not apply.

If the agent encounters any situation outside of those governed by C, it cannot be permitted to issue a lethal response – a form of Closed World Assumption preventing the usage of lethal force in situations which are not governed by (or outside of) ethical constraints. (Arkin, 2011, p. 18)

A fundamental issue with this formulation centres on the problem of recognizing any particular situation as being within (or outside) the set of situations for which the constraints imposed by the Rules of Engagement and Laws of War apply (e.g. Garfinkel & Sacks, 1970). This kind of problem has of course a complex genealogy within the field of artificial intelligence (AI) research. From the early days of AI in the late 1950s and 1960s, esoteric problem solving posed by the playing of intellectual games such as chess (a game bounded by clearly specified rules) was seen as the obvious challenge for the field, so that the status of artificial intelligence might be ascribed to a machine capable of, for instance, performing such a task as well as, or better, than a human player. The US military’s interest in computers and AI had from the very beginning been a driving force behind many of the developments in the field (Edwards, 1988). As computing applications broadened out from the early problems of deciphering codes and the calculation of missile trajectories to decision making and game playing, they seemed to open the prospect of an (even) more automated form of warfare, with military tactics reduced to a computable decisional calculus. In his reflections on the role and possible place of the then still emergent computer sciences and technologies within the ethical norms and values of human society, Norbert Wiener (1950) voiced his unease: ‘von Neumann and Morgenstern[’s]…Theory of Games has made a profound impression…in Washington. When Mr Shannon speaks of the development of military tactics, he is not talking moonshine, but is discussing a most imminent and dangerous contingency.’

Wiener proceeds to bemoan ‘the dire implications of the chess-playing machine grown up and encased in a suit of armor’ (Wiener, 1954, p. 178, our emphasis). It is, in this view, one thing for a machine to win at chess – a game of fixed rules with a large but finite set of moves – and quite another to deal with the complexity and contingencies inherent in human warfare. Elsewhere, Wiener (1964, p. 59) writes of technology’s literal-mindedness: ‘if it grants you anything at all it grants what you asked for, not what you should have asked for…’

Despite many well-publicized achievements, AI researchers had to struggle with the question of common sense that humans seemed to possess (but not necessarily articulate) and that computers lacked (see for instance Hobbs et al., 1985). Thus, a recent UN report (Heyns, 2013, p. 10) voices concerns that autonomous military robots lack the (human) qualities that compliance with international humanitarian law presupposes, such as ‘human judgment, common sense, appreciation of the larger picture, understanding of the intentions behind people’s actions, and understanding of values and anticipation of the direction in which events are unfolding’. Common sense cannot be reduced to a set of rules. Furthermore, except for well-defined domains it is not possible to anticipate all the possible exceptions to a set of rules. We might address this by stipulating new rules with all manner of conditional clauses but we cannot presume to know all the possible exceptions that such clauses are meant to anticipate. Speaking about the addition of further rules when existing rules do not apply, AI pioneer Marvin Minsky (1991, p. 43) notes:

This approach usually works well at first, but whenever we try to move beyond the realm of toy problems and start to accumulate more and more rules, we usually get into trouble because each added rule is increasingly likely to interact in unexpected ways with the others.

Another way of explaining this problem is to note (as Charles Taylor, 1992, and others have pointed out) that rules do not contain the rules of their own application: ⁵ ‘Nothing in the rule itself fixes its application in a given case…there is no “fact of the matter” concerning the proper application of a rule’ (Barnes, 1995, p. 202; see also Tsoukas, 2005, p. 125). In law, rule-following presupposes regular practices, ‘forms of life’ (Wittgenstein, 1978).

Arkin deals with this problem, i.e. how to recognize a particular situation as falling within or without ‘those governed by C’ (Arkin, 2011, p. 18), by means of a strategy of deferral; that is, by proposing that technoscientific progress will, at some unspecified near-future date, deliver the necessary capabilities for ‘effective situation assessment’ (Arkin, 2011, p. 22). For example:

Although we are nowhere near providing robust methods to accomplish this in the near-term … in my estimation, considerable effort can and should be made into this research area by the [Department of Defence], and in many ways it already has, e.g., by using gait recognition and other patterns of activity to identify suspicious persons. These very early steps, coupled with weapon recognition capabilities, could potentially provide even greater target discrimination than simply recognizing the weapons alone. Unique tactics (yet to be developed) by an unmanned system to actively ferret out the traits of a combatant by using direct approach by the robot or other risk-taking (exposure) methods can further illuminate what constitutes a legitimate target or not in the battlefield. (Arkin, 2011, p. 11)

The acknowledged problems of rule-based systems led some AI researchers to seek ever more sophisticated forms of knowledge representation; others have sought to develop programs that can ‘learn’ from their performance – an approach that Arkin explicitly rules out in his robot design. His robot cannot, so to speak, ‘learn on the job’. What he envisages instead is some form of post-hoc analysis of the system’s behaviour and the appropriate modification of the programming by human operators so as to minimize any emergent gap between ethically prescribed and actual behaviours.

Thus, although Arkin’s ‘ethical governor’ serves as a final, real-time check on the robot’s ethical operation, his architecture also provides for other components that would come into play both prior to and following its deployment. Specifically, the ‘responsibility advisor’ is a part of the system for the management of the deployment of the robot. The responsibility advisor would aid in making clear to human operators the potential consequences of the deployment of the robot – including situations in which they might deliberately choose to override the system’s inaction due to its ethical governor – and thus allocate ultimate responsibility for any consequences (cf. Sharkey & Suchman, 2013; Sullins, 2010). On the other hand, the ‘ethical adaptor’ would allow for the reflective modification of the robot’s potential responses following an evaluation of the consequences of its actions or inactions in specific engagements. For example, should the robot be determined to have violated the Laws of War then the set of ethical constraints governing its behaviour would require modification. Again it is worth noting that Arkin does not envisage an infallible machine: ‘Remember that the system will never be perfect…but it is designed and intended to perform better than human soldiers operating under similar circumstances’ Arkin (2011, p. 72; our emphasis).

Ironically, given Arkin’s ambition to escape the unreliability caused by human emotions insofar as these are deemed to interfere with ethical rule-following, one aspect of human emotional responses – ‘guilt’ – is reintroduced into the design, in effect serving as an anthropomorphic Derridean (1976) supplement. Arkin thus recognizes that (emotions of) guilt play a key role in human ethical behaviour and seeks ways of incorporating guilt into the calculations performed by the ethical adaptor. Simply put, the robot would be programmed in order that any negative evaluation of its actions (whether by human overseers or the robot’s own monitoring capabilities) would influence the value of a parameter (‘V_guilt’) in such a way that once a threshold value was exceeded the robot would (via the ethical adaptor) abstain from any further exercises of lethal force pending a review (Arkin, 2009, pp. 140–3; see Sharkey & Suchman, 2013 for a discussion). We might say, however, that what is presented as a mere technical refinement also constitutes a symbolically loaded move that disrupts the ontological claims made by Arkin on behalf of his apparatus. Arkin insists that his project is not aimed at the creation of an artificial form of moral agency, but merely of a governor able to regulate an autonomous system’s behaviour in compliance with Laws of War and Rules of Engagement. Ironically however, in the Judeo-Christian tradition, Adam and Eve’s experience of guilt – a result of tasting the fruit of the Tree of Knowledge of Good and Evil – signifies the dawning of full awareness of their own moral agency. ⁶

Derridean deferrals and supplementarities (Derrida, 1976; Cooper, 1989) plague attempts to unambiguously locate moral agency in some part of the (post-human) system. As a recent Ministry of Defence (2011) report puts it:

The role of the human in the loop has, before now, been a legal requirement which we now see being eroded, what is the role of the human from a moral and ethical standpoint in automatic systems? … Is a programmer guilty of a war crime if a system error leads to an illegal act? Where is the intent required for an accident to become a crime?

Arkin’s design, therefore, seems to replicate the problem of the ‘obliteration’ of moral responsibility (Bauman, 1989, p. 215) that has been highlighted in many critical accounts of organizational ethics (e.g. Jones et al., 2005; Wray-Bliss, 2012). Arguably, it creates a situation where, in Arendt’s (1969, pp. 38–9) words (written about bureaucracy but in the shadow of the Holocaust and the Vietnam war) ‘there is no one left who could even be asked to answer for what is being done’.

In sum, we might say that the effort to convert the Laws of Armed Conflict and Rules of Engagement into a set of rules which can then be safeguarded by an ‘ethical governor’ presupposes and requires the solution of technical and theoretical problems that have long bedevilled AI research, and which may indeed not be solvable in the manner that Arkin’s project demands. But perhaps this is an indication that the ethical warrior robot also needs to be assessed in terms other than its instrumental effectiveness as a war-fighting apparatus. What seems to be at stake here is the maintenance of what we might call a culture of denial of the essentially tragic nature of warfare. The wish for an efficient and just war, that is war carried out in an ethical (even ‘humane’) manner, relies, as Judith Butler (2010, p. xviii) notes, on the ideal of the controllability of the instruments of destruction. ‘But because uncontrollability is part of that very destructiveness, there is no war that fails to commit a crime against humanity, a destruction of civilian life.’ Contemporary institutions however tend to reproduce the fantasy ‘that there can be a war without such crimes…a ‘clean’ war whose destruction has perfect aim’ (Butler, 2010, p. xviii, our emphasis).

Concluding Comments

Organization studies have in recent years sought to expand their remit beyond ‘the organization’, narrowly understood as a ‘discrete, bounded, economic-administrative entit[y]’ (Chia, 1998, p. 5) and have begun to engage with much broader, and arguably more fundamental, questions concerning the organized character of modern social life (and death: e.g. Banerjee, 2008; Burrell, 1998; Godfrey et al., 2014). Within this expanded perspective, the ways and means through which the destruction of human beings and their worlds is currently ‘managed’ cannot be but a key concern (Bauman, 2001; Suchman, in press). Organized destruction should not be considered as a regrettable aberration but rather as a logical extension of the desire to maintain the ‘organized character’ of the contemporary world. In this connection, we have argued that the conception of an ethical autonomous killing machine constitutes a culturally and historically situated event, the conditions of intelligibility of which are in need of elucidation.

While Arkin’s work remains highly controversial even among roboticists, it has, as we have seen, been taken very seriously indeed by the US military. The opposition of concerned scientists (including roboticists) – voiced most notably via the International Committee for Robot Arms Control (ICRAC; e.g. Sharkey & Suchman, 2013) – is, if anything, a recognition of the social and political urgency of the questions automated warfare poses. We make the claim that the discipline of organization studies also needs to have a voice, that it too must seek to understand such developments and the kinds of realities they attempt to bring into being. This article, then, is part of such an endeavour. In line with recent calls for ontological inquiry into how moral agency is constituted in contemporary organizational processes and settings (e.g. Painter-Morland, 2011), we have sought to understand the ethical warrior robot as the ‘assembling’ (e.g. Hardie & MacKenzie, 2007) of a moral actor, the device by means of which organized destruction might be ethically carried out. As we have shown, Arkin’s system enacts what we might call a particular managerialist (Costea & Amiridis, 2013) understanding, where the death of civilians and other tragedies of war tend to be reframed as lapses in good organizational practices and procedures, calling for ever more sophisticated technical-managerial fixes. For instance, as we have seen, the notion of the programming of ethical rules appears as a viable solution for the perceived failure of ‘ethical socialization’ (also a common complaint in business ethics, e.g. Dunlop & Lee, 2003; Ponemon, 1992) of military personnel. Unreliable bodies, are thus to be replaced by (more) reliable machines. As in the organization of production, F. W. Taylor’s (1911) now triumphant ‘system’ is, it seems, always rearranging its parts.

The Deleuzian concept of agencements has been used to account for how such sociotechnical rearrangements work when ‘considered from the point [of] view of their capacity to act and give meaning to [such] action’(MacKenzie, 2008, p. 20). Although the extant (often ANT influenced) literature on organizational agencements (e.g. Beunza et al., 2006; Callon, Muniesa, & Millo, 2007; Raviola & Norbäck, 2013) provides us with a useful vantage point from which to begin to make sense of the ever-mutating assemblages of the ‘Revolution in Military Affairs’, our analysis of the ethical warrior robot differs in a crucial respect. This literature tends to argue that such assemblages ultimately render human-machine ontological differences inconsequential. We on the contrary want to argue that an ontology of the specific form of ethical agency that the ethical warrior robot attempts requires attention to how, in the course of such assemblage-work, the (embodied) human, the (programmed) machine and the difference(s) between them come to be symbolically and instrumentally enacted in, and as, a series of disposals, deferrals and supplementations (Cooper, 1989). Such dis-placements, we have argued, are constituted as ‘technological dramas’ (Pfaffenberger, 1992), that is to say, as occasions for the rehearsal of various moral and philosophical conflicts characteristic of Atlantic (post?)modernity: free will versus determinism, body versus mind, human versus machine, self (‘Us’) versus other (‘Them’). We have seen for instance how the integrity of the categories ‘Us’ (‘US’?) and ‘Them’ (the ‘Other’ side in the ‘War On Terror’ and related conflicts) are seen to depend upon the provision of a ‘technical fix’ for the problems posed by the physical and ethical unreliability of the human components of the RMA’s war machine. ‘Lethal behaviour’, it appears, cannot be effectively ‘governed’ (Arkin, 2011) unless the soldier’s body, so prone to fear, injury and the ‘savage joys’ (Elias, 2000, p. 371) of war, is displaced from the system. At the same time, this attempt to purge the system from the characteristics associated with embodied subjectivity brings about new problems of disposal by conjuring the spectre of a kinship between the now autonomous system (and thus – ultimately – ‘Us’) and the out-of-control killer machines which have since its inception haunted science fiction.

Set against this background, the importance of Arkin’s managerial-technological ‘fix’ for the tragic nature of human conflict lies primarily not in its instrumental value as a functioning artefact (as we have made clear, we have strong reservations regarding its functionality) but in its role as an ideal model, a means through which the longing for a rational, even humane, organization of destruction has come to be articulated. As such, as we tried to show, it speaks to a number of key concerns in organization studies, namely, ethics, embodiment and the increasing deployment of technological ‘solutions’ for perceived human failings in the course of ongoing managerial(ist) re-makings of the modern world.