Our Bodies in the Trolley’s Path,or Why Self-driving Cars Must *Not* Be Programmed to Kill

What Is the Fat Man Thinking? or, from Quandary Ethics to the Question of Action

The short answer to the first question—is it reasonable to assume that principles upheld by experimental ethics are sufficient to resolve those same ethical situations if faced in real life?—is no. To illustrate why, we can highlight the improbable and binary framing of these scenarios, noting how distant they are from actual life situations. As a result, the values they uphold may or may not sufficiently serve lived ethical situations.

Consider, for example, the fact that we do not learn any details about the five workers in the original framing of the trolley problem: Who are they? Are they young or old? How are they positioned in relationship to the trolley? Are they capable of seeing, hearing, and reacting to the trolley as it approaches them? Would I, the trolley driver, be unfairly targeting the one worker, given that he is just a bystander in the situation until I decide to target him, thus giving him a lot less time to reflect and react? These and other similar questions point to the uncertain, complex, and living nature of the situation that I, the driver of the trolley, could plausibly be facing. As a result, while it may seem that the utilitarian principle of “saving most lives” is applicable when presented with the simplified version of the scenario, this principle may or may not serve in an actual lived situation.

The trolley experiment and its variations are cases of quandary ethics. ⁸ In quandary ethics, the parameters are predefined and fixed and the choices are clear. These two characteristics make quandary ethics particularly attractive for algorithmic forms. Both quandary ethics and algorithmic morality hold the premise that if we identify a set of principles, we can “solve” ethical problems, thus proffering a false sense of being in control.

Here, extending my past research on ethical situations and their characteristics (broadly based on pragmatic and care ethics), I outline three key tenets of ethical situations that set them apart from quandary ethics. First, ethical situations are marked by a deep sense of uncertainty and an organic character. Second, our place within ethical situations matters greatly. Third, the impact of our actions in response to ethical situations is not limited to immediate outcomes, consequences are broad and long ranging. Therefore, and as the examples above show, principles that appear to solve the scenarios of experimental ethics may or may not serve similar ethical situations encountered in real life. Their usefulness and appropriateness depend on the concrete facts of the situation inclusive of the condition and position of the person who is encountering the situation.

What If Grandma Is Pushing a Stroller? or, Why Principles Cannot Simply Be Applied

The underlying premise for going to experimental ethics for a source of principles is that all we need for creating moral algorithms is a set of universal and/or agreed-upon principles. Yet, if principles upheld by experimental ethics are inadequate to address real-life ethical situations, the second question is would identifying or agreeing on “a” set of principles be sufficient for creating moral algorithms that adequately control the behavior of self-driving cars? The answer to this question is also no for two reasons: first, resolving or addressing ethical situations is not solely a matter of identifying a set of values to apply to a clearly defined situation ¹⁰ ; rather, it involves deliberating on what the situation is and what those values entail in action. Second, algorithms depend on sets of parameters that categorize various situations as the bases for triggering a series of action scripts. ¹¹ Not only are such representations reductive, the fixed categories that such algorithms rely on introduce biases into the system in ways that would disproportionately and unjustly impact different segments of the population.

Imagine, for instance, that we agree on the principle of maximizing life. Moreover, to make this principle easy to translate into algorithmic form (i.e., make it calculable), we might define it purely in terms of life expectancy. But what does this mean in action? Consider, for instance, the seemingly most straightforward of cases that of an inevitable accident in which the car is faced with different types of people on its various paths. Imagine an older adult on one side and a young child on the other. Here, interpreting the principle in terms of life expectancy would result in deciding to save the child’s life. In another scenario, we might imagine four older adults on one side and two young children on the other. Making a decision based on life expectancy is clearly not as simple anymore. What if there is a teenager on one side and an older woman pushing a stroller on the other? What does maximizing life entail in this case? In any such calculation, we need to consider the changing nature of the situation (or, in mathematical terms, the probabilities of how the situation might evolve). For example, let’s suppose that there are four young adults on the one side and an elderly woman walking slowly with a walker on the other. In this case, the algorithm needs to account for how these pedestrians might react to the car approaching them. The four young adults, as opposed to the elderly woman, are better-equipped and more likely to react quickly to clear the path in a way that might save their lives. Maximizing life in this circumstance might entail veering the car off toward the four young adults.

Settling on the value of maximizing life is by no means sufficient to address the situation adequately, as maximizing life is not one and the same in all circumstances. The same principle of maximizing life may entail actions that are contrary in different situations. ¹² Of course, resolving such ethical situations becomes even more complicated if we note that real-life ethical situations don’t always fall neatly into distinct and clear choices as the scenarios above suggest. ¹³

More broadly, we recognize in these very small variations of a simplified scenario, even if we identify a value such as maximizing life and define it in the most simplified and mathematical terms, that it cannot be simply applied to yield a course of action in accordance with it. Figuring out what maximizing life is in each case requires deliberation, and that’s what algorithms cannot do. Settling on a value or a set of values, or what Suchman (2015) has characterized as a “super morality,” is neither possible nor sufficient for algorithms (or people for that matter) to decide what the situation is and the action that it demands.

One might argue that this is fine, since, as humans, we do not have time to deliberate when faced with an inevitable accident. Our understanding of situations is partial and our reflexes are limited. But I would hold that even in emergency situations, our human ways of understanding and reacting are much more nuanced and inclusive than inputting a set of variables into an ethical calculus. It is important to note here that the idea of algorithmic morality resides within the paradigm that understands humans as information processors. The reductive nature of this perspective has been challenged by many, including researchers within the fields of cognitive science and artificial intelligence (see, e.g., Epstein 2016; Zarkadakis 2016). Algorithms can create sophisticated mathematical models of the world. Those models, however, are only as good as the specific kinds of data that can be collected and stored in digital form (see also Gitelman 2013) as well as the categories and organization of data within relational databases. This is one of the reasons why there has been very little improvement in creating algorithms that can effectively deal with the uncertainty in such seemingly mundane tasks such as housekeeping, as noted recently by Suchman (2016).

There is another reason to be extremely cautious about algorithmic morality. Algorithms are a series of action scripts triggered when certain conditions, represented by a set of variables, are met. The relationship between the variables and action scripts is either hard-coded or derived from a set of examples as in the case of machine learning.

Moral algorithms working based on a set of agreed upon principles (e.g., maximizing life) necessarily rely on a set of variables/categories representing ethical dilemmas to weigh and select among a set of action scripts. In such an algorithm, what is included as a category and how it is included can be a source of deeply problematic biases. ¹⁴ At the most basic level described above, the algorithm relies on categories such as human/nonhuman, young/old. Any such categorization poses ethical and political challenges. Not only is the definition of each category problematic in mathematical terms (e.g., how do we define young or old?), the inclusion or exclusion of a category is also a source of bias. We might ask if young/old is the only criteria of relevance. What if we add to the above scenarios a case where there is a younger person wearing a helmet who might thus stay alive but will be gravely injured on one side, and an older adult with no headgear who will likely die if hit on the other? Is the category of wearing/not wearing helmet relevant? Knowing that some studies have linked obesity to life expectancy, should the weight of the subjects be included as a category?

This very simplified case demonstrates the ethical valence of the inclusion or exclusion of categories, recalling Suchman’s (1993) Politics of Categories, as well as how each category is defined and operationalized as code. By programming moral algorithms based on such categories, and with a deciding principle such as maximizing life, one can expect that in the majority of cases, the car’s algorithm will target older people (or those who are overweight or wear helmets!). In other words, the combination of the categories with all-encompassing principles to decide the cases introduces deeply problematic biases, surfacing issues related to the politics of algorithmic ordering that have been documented in other contexts, such as Web version 2.0 (e.g., Beer 2009; Bucher 2012) and search engines (e.g., Introna and Nissenbaum 2000).

The addition of each new category only makes the system more complex, producing what Bowker and Star (1999, 21) described in their ethnographic work of classification as “a panoply of tangled and crisscrossing classification schemes held together by an increasingly harassed and sprawling…bureaucracy.” The fluidity of the categories in the case of machine learning and genetic algorithms by no means removes the biases or reduce complexity. Rather, in such cases, the algorithms turn into “black boxes,” making such biases inscrutable: more hidden and difficult to discern and scrutinize (Introna 2016, 25-26). ^{15 , 16}

Moreover, such algorithms would have to rely on sensing technologies such as image recognition, the inherent biases and reductiveness of which have been well-documented. It has been demonstrated, for example, that African Americans, Asians, dark skinned persons, and older people could be subjected to disproportionate scrutiny because they are easier to identify by surveillance technologies (Introna and Wood 2002, 192). ¹⁷

Settling on one or a set of values for designing self-driving cars’ algorithms is not only insufficient for resolving the ethical situations that cars would face, it also introduces systemic biases that have disproportionate negative impacts on specific segments of the population. Principles such as maximizing life cannot be applied to ethical situations to produce a course of action like recipes in a cookbook; rather, they need to be employed as hypotheses to assess what the situation is and the action that it demands. Given the nuances and complexity of ethical issues that come to matter when we are in such situations, and even in the unlikely scenario when the designers, manufacturers, and public can agree upon a set of principles, those principles are inadequate for driving the behavior of self-driving cars.

Remembering the Fat Man, or Beyond Net Gains

The above leads us to consider an argument often used by proponents of algorithmic morality, which is captured in the third question identified earlier: that in spite of the fundamental limitations and potential for error articulated above, we should accept algorithmic morality in the context of self-driving cars, given that by adopting self-driving cars, in theory, we save more lives than we lose. But is this a valid argument?

The case for self-driving cars is often made by highlighting foreseen advantages in eliminating fatal accidents: computers have much faster reflexes, they do not drink and drive, and they follow rules such as speed limits and safe distances from other vehicles. As such, self-driving cars could potentially prevent many deaths that are caused by human error and distractions. However, as the supporters of self-driving cars also admit, these cars do not eliminate all (fatal) accidents. Accidents will still happen due to software errors or other similar machine failures.

The argument above appears simple and straightforward at first: self-driving cars potentially save lives by eliminating accidents due to human error; this number is larger than the number of deaths they cause due to machine errors such as bugs or unexpected scenarios, so we need to accept the risks of algorithmic morality because the net gain of lives saved is positive. ¹⁸ Looking closely at the above argument, it is easy to recognize that a utilitarian principle is at work: by adopting self-driving cars, we are able to save more lives than we lose. Thus, we need to focus on the net gain of self-driving cars and accept their potential errors and accept fatalities by machine errors as a price we pay for the benefits.

But the situation is far more complex than the binary suggests, as an alternate reading of the trolley thought experiment can help demonstrate—if we reflect on and attempt to take the position of the fat man in the scenario. ¹⁹ Doing so, we might first note how the fat man is treated as a living object but an object nonetheless. All we know about him, after all, is his physical composition: he is fat and heavy, so much so that what is visible is the material weight of his body and the tragedy of his humanity disappears under this weight. ²⁰ The inhuman nature of this framing is clearer when we try simple replacements: A fat girl? A heavy pregnant woman? My fat teenage brother? My neighbor Ahmed—who retired last month after forty-five years, who had had a rough time after his partner passed away two years ago and had gained some weight; but who was feeling much better lately and even seemed excited to be planning a long backpacking trip in New Zealand? These simple replacements highlight the biased perception of the situation that the referent “fat man” creates, working to remove his humanity from the situation in ways that are advantageous to the logic of a utilitarian calculus. ^{21 , 22}

In a similar fashion, utilitarian arguments that highlight the net gain of lives saved by introducing self-driving cars fail to consider the complexity and nuance of the ethical problems at hand, veiling the devastation and pain involved in fatal accidents in abstract numbers. In these arguments, bodies are rendered interchangeable, merely counted, and recounted by their physical makeup and age. Yet, if we consider the above scenarios with attentive care, a serious consideration of the lives lost in fatal accidents involving self-driving cars is brought to the fore. The lives that are lost by the introduction of autonomous cars are not the same as the ones that would have been lost otherwise (Lin 2014). ²³ Much like the fat man’s case above, we need to ask about these people and their stories and the systemic biases and power structures that make them most vulnerable. In addition to considering algorithms’ biases, we should, for instance, consider the fact that the people who may be most affected by this technology are those who have the least power in deciding its makeup. Here, an idea suggested at end of the MIT Review article is worth noting: the idea that manufacturers can offer multiple versions of moral algorithms in the future so people can choose based on their preferences. I could be presented with the choice of buying a selfless car with algorithms that will sacrifice my life to save others or a protective one that will save my life at any cost. This line of reasoning finds it acceptable for the people who will be driving these cars to decide to save their own lives at any cost. Not unlike the case of the fat man in the experimental ethics scenario above, we (the consumers) are given a choice with the understanding that it is acceptable to “use” some people’s lives to save others’ lives. Car manufacturers, and not pedestrians or bikers, are the ones with the power and privileges to offer this choice and to profit from it in return.

Rethinking the Trolley’s Trajectory, or Ethical Inquiry and Design Imagination

What if we decide that moral algorithms are entirely unacceptable? What if we choose to reject them no matter what?

In the previous sections, I illustrated how utilitarian principles seemingly upheld by a narrow and literal framing of the trolley problems fail to do justice to what’s at stake in real-life situations when and where fatal accidents may occur. They do so by masking the uncertain and organic characteristics of ethical situations, by being dismissive of the significance of our place within those situations, and by being inattentive to the broader consequences of actions.

In a similar manner, it is my concern that a dominantly utilitarian framing that emphasizes the inevitability of sacrificing some lives to save others is obscuring our ability to see what’s at stake with this new technology. By this, I do not mean to downplay the urgent need of action for reducing death toll as a result of car accidents nor to undermine the many advantages that some of the features of this new technology might bring about. I would argue, however, that a genuine concern with the many lives lost in car accidents now and in the future—a concern that transcends false binary framings of technology enthusiasts’ opportunity-centered approach (Hargraves and JafariNaimi 2012)—and the liability considerations of the automotive industry, could serve as a starting point to rethink mobility as it connects to the design of our cities and the future of our communities. Indeed, a serious consideration of the ethical issues raised by self-driving cars opens a space for the kind of design that is innovative, inclusive, and mindful of both immediate and broad consequences.

It is neither the scope nor ambition of this article to outline design alternatives that would eliminate the need for algorithmic morality. However, the three tenets of ethical engagements that I have identified in this article gesture toward possibilities for the kinds of radical rethinking of this design space that we might pursue.

The first step for breaking free from the narrow framing that implies the adoption of self-driving cars as inevitable is to restore the deep sense of uncertainty accompanied with this new technology. Indeed, the systemic and complex nature of designing self-driving cars and integrating them into the urban landscape, and their interplay with the infrastructural and social makeup of cities, makes it particularly challenging to predict self-driving cars’ full technical, social, and cultural impact. There are many gaps in what we know about self-driving cars. Think, for example, of the recent fatal accident that was caused because the self-driving car didn’t recognize an incoming white tractor trailer against a brightly lit sky (Vlasic and Boudette 2016). Or, potential social challenges such as hacked software and other forms of abuses. Taking these issues seriously is reason to question whether self-driving cars will actually save lives. With so much at stake and so many questions, isn’t it wise to step back and carefully consider the adoption of this technology?

The second step for breaking free from the dominant utilitarian framing is to observe the places from which the arguments for self-driving cars arise and to bring to fore the historical trajectory that this technology extends. We might note and question, for example, the roles and powers of strong industry interests in defining the terms of the discourse. We might observe the groundings of research agendas that champion algorithmic morality to trace their intellectual roots, largely sympathetic to Artificial Intelligence (AI) and the potentials of machine learning. We might bring under scrutiny the historical trajectories that this new technology rests upon. How did we get to be so reliant on cars for our daily transportation? How did the many deaths as a result of car accidents become normalized? Consider, for example, the legal cases that led to a dramatic simplification of a complex object, the automobile (Jain 2004). As Jain observes, an early categorization of the automobile in the same group as bikes and carriages instead of steamships and trains configured it as a common sense object. In doing so, it extended notions of danger and chains of responsibility to the consumer (Jain 2004, 82) instead of car designers and manufacturers and those responsible for the urban infrastructures that accommodate cars. A similar logic is being drawn upon in the recent discourses on self-driving cars in which people (drivers, pedestrians, bikers, etc.) are positioned as reckless and unpredictable, making it seem logical to replace them with this rational-albeit-imperfect technology. But what might ensue if we were to challenge the very notion that this new technology is a car? How would the terms of discourse change if we refer to this new technology as dangerous instrumentality (Jain 2004) or, even more aptly, “deadly robotics?” While the forces of the historic trajectories that led to car-centric cities and lifestyles are strong, we are not bound to follow them. To be sure, there are other models and trajectories that we can draw upon. Examples include the Netherlands’ bike-friendly laws, public policy that resulted in a reliable and safe biking infrastructure, or the more recent Swedish Vision Zero with the ultimate target of redesigning urban infrastructures as to entirely eliminate deaths or serious injuries by cars. We might indeed think of the introduction of self-driving cars as an occasion for a radical rethinking of mobility that challenges and reorients dominant car-centric visions. For this, however, we need to engage multiple disciplinary and social and historical perspectives and to embrace more nuanced framings of the problems of mobility.

The third step in this process is to draw broad and wide-ranging consequences of the adoption of self-driving cars into consideration, highlighting that the decision to accept algorithmic morality cannot be simply made based on the number of lives potentially saved. We must consider both tangible and intangible effects, such as how this new technology might change the character of our cities and the quality of movement day to day. How much of public funds and public spaces would have to be devoted to it and to whose benefit? How would it impact the legal system, one that is already biased against bystanders, bikers, and small children (Jain 2004)? What would it be like to live in a city where at every moment you might be the target of an accident decided solely by the preprogrammed logic of an algorithm? Would it be safer to travel with a stroller at all times to assure safety?

The problems of mobility are perplexing. A pragmatic rethinking of them involves a systemic consideration of all that is at stake, including the safety of pedestrians and bikers, the livability of cities, and environmental sustainability—all center stage. In addressing these problems, we might draw on established strategies, such as small-scale experiments and interdisciplinary collaborations, while inventing new strategies that are commensurate with the scale and complexity of the task.

The design of self-driving cars opens up a space for ethical inquiry rethinking theories such as literal readings of trolley experiments and acceptance of the principles they appear to uphold. Sophisticated and nuanced ethical analyses of what is at stake in the design and implementation of self-driving cars give us an opportunity to rethink mobility and the instrumental and cultural values we assign to cars. No doubt there is much that is unknown. What is sure, however, is that succumbing to algorithmic morality in the name of increased safety would be a grand failure of both our ethical and technical imagination.