The problem of the missing dead

Abstract

This article examines what scholars can learn about civilian killings from newswire data in situations of non-random missingness. It contributes to this understanding by offering a unique view of the data-generation process in the South Sudanese civil war. Drawing on 40 hours of interviews with 32 human rights advocates, humanitarian workers, and journalists who produce ACLED and UCDP-GED’s source data, the article illustrates how non-random missingness leads to biases of inconsistent magnitude and direction. The article finds that newswire data for contexts like South Sudan suffer from a self-fulfilling narrative bias, where journalists select stories and human rights investigators target incidents that conform to international views of what a conflict is about. This is compounded by the way agencies allocate resources to monitor specific locations and types of violence to fit strategic priorities. These biases have two implications: first, in the most volatile conflicts, point estimates about violence using newswire data may be impossible, and most claims of precision may be false; secondly, body counts reveal little if divorced from circumstance. The article presents a challenge to political methodologists by asking whether social scientists can build better cross-national fatality measures given the biases inherent in the data-generation process.

Keywords

civilian civil war conflict fatality mortality newswire data South Sudan

Introduction

Civil war datasets offer scholars an ‘international funeral parlour’ to view the dead.¹ This presentation of death obscures a murky reality, fraught with uncertainty about who has died and who is worth counting. Behind each observation in a newswire dataset is the story of a person who falls victim to an atrocity; a killer who disfigures, buries or conceals a corpse; a human rights officer or a journalist who arrives once the killing is done, and decides whom to talk to and what to record; a report that bureaucrats or newspaper editors make public; and a researcher who locates the report, and codes it into a dataset. This article is about what scholars can learn about the causes and consequences of civilian killings from newswire data in contexts wrought with non-random missingness. It asks: How do technical procedures, political agendas and normative judgements alter inferences about violence drawn from the newswire? Can social scientists build better cross-national fatality measures given those biases?

Scholars have long recognized that fatality data are nasty, noisy, and muddled (Williams, 2016; Spagat et al., 2009; Kalyvas, 2006). How researchers code determines what they see (Merry, 2016; Sambanis, 2004). Governments fabricate data or obscure it to produce ‘official ignorance’ (Aronson, 2013: 30; Werth, 1997). Epidemiologists and statisticians must adjudicate among assumptions to model observations missing not-at-random (Manrique-Vallier, Price & Gohdes, 2013; Checchi, 2010; Dulic, 2004). These problems motivate some to reject civil war research because the data preclude design-based inference, while others proceed believing that patterns in messy data reflect stronger trends in reality (Gerber, Green & Kaplan, 2014).

This article charts a course between nihilism and blind faith. It starts from the premise that newswire data remain invaluable to conflict scholarship in spite of non-random missingness. Not all fatality count biases are equal: some distort our inferences about civilian killings in ways that preclude precision, while others prove trivial. I argue that conflict scholars should train their methodological eye on the first type. But to do so, they need to understand what they are.

I contribute to this understanding by offering a unique view ‘under the hood’ of newswire data for South Sudan. Most quantitative datasets in the social sciences stand on qualitative foundations, which demand careful source interpretation (Kreuzer, 2010). I begin by demonstrating how this feature of newswire data creates confusions in defining civilians, biases in counting methods, and estimation problems. I then unveil the data-generation process by recording the experiences of people who handle the dead and file newswire reports. Drawing on 40 hours of interviews with 32 human rights advocates, humanitarian workers, and journalists, I investigate the biases we should expect to distort data for South Sudan. I focus subnationally on killings in Jonglei State to assess patterns of non-random missingness. I conclude with lessons about which biases matter most and assess the merits of using categories rather than counts to analyze violent deaths in low-information contexts.

Counting the dead

In war, ‘something is always wrong with the facts one is given’ (Nordstrom, 1997: 43). Scholars, journalists, and human rights advocates disagree about who dies in crises for many reasons. They diverge on who counts as a civilian, and face political and technical choices about how they handle the dead. This leads to variations in how researchers record killings and estimate what they do not see. Newswire datasets offer starting points for navigating these problems by offering systematic records of the dead over time. The problem for civil war researchers is that they can rely neither on a missing-at-random assumption nor on a general pattern of missingness that applies across contexts.

Which civilians?

How researchers define dead civilians determines how they code, and how they code determines what they see (Sambanis, 2004). This confronts conflict scholars with two challenges.

The first is to distinguish civilians from combatants – a difficult exercise in irregular wars without subjective judgements (Gade, 2010). Belligerents may remove uniforms or put them on the dead, and people shape shift between civilian and combatant categories at unpredictable times and in unpredictable ways. For example, South Sudan’s ‘White Army’ is an ad-hoc mobilization of cattle camp youth without clear central command, which sometimes aligns with insurgents (Thomas, 2015). Whether observers count these youth as civilians or combatants depends on what they use death counts for (Ratnayake, Degomme & Guha-Sapir, 2009). For example, human rights advocates may choose an expansive definition to obviate excluding ‘victims’.

The second challenge is that conflict researchers study ‘direct’ civilian deaths, requiring an ‘event grammar’ that identifies a perpetrator, a victim, and an act (Landman & Gohdes, 2013: 78; Ball, 1996). Even where the act is observable, scholars must disaggregate whether a homicide is conflict-related, criminal or both (Williams, 2016). This is difficult where criminal networks run deep into the battlefield and conflict-related killings shape criminal homicides (Kleinfeld, 2017).

Political and technical bias

Once researchers settle on a civilian measure, they face technical biases, which arise from counting methods, and political biases, which arise when people’s agendas shape what they report (Aronson, 2013). Two issues make this distinction a useful but blurry framing device. The first is that technical choices can motivate political biases that would not otherwise pose a challenge. For example, wire journalists may interview survivors at an atrocity site. Due to political pressure, civilians might misreport what they observed. Thus, the recording technique motivates political bias.

The second issue is that everybody, from survivors to researchers, has agendas that shape what they say, choose methodologically, and judge credible. These agendas may be technical. For example, epidemiologists and forensic statisticians arrive at different counts because they have different objectives that motivate different survey questions (epidemiologists seek data to improve the health of the living; forensic statisticians investigate homicides) (Aronson, 2013). These agendas may deprioritize accuracy to advance a normative cause (De Waal, 2016). Political authorities may also distort official records to propagate a particular social outlook (Merry, 2016; Nelson, 2015; Tishkov, 1999).

Biases arise at four levels of data production and use, when: (1) investigators count the dead; (2) researchers code data; (3) scholars analyse the coded data; and (4) people and institutions make secondary use of this

Table I.

Biases in newswire data

Level	Challenge	Bias type
(1) Investigation	Urban bias: When investigators perceive rural conflict from towns and cities.	Both
	Communication networks: Journalists receive news of killings disproportionately from areas with cellphone coverage.	Technical
	Concealment: When killers preclude identification of the dead.	Both
	Questioning: What investigators count depends on whom they interview and what they ask (UNMISS, 2015).	Technical
	Truth modification: Question format, social desirability, recall problems, and vested interests affect how interviewees describe the truth (Eck, 2011; Foster Lynch, 2013).	Both
	Record keeping: Investigators format records to target particular audiences, producing description bias (Weidmann, 2015). Different actors also keep different records based on political objectives and administrative capacities (Wood, 2003: 32).	Technical
	Record publication: What gets published depends on the agendas of political authorities and media houses.	Political
	Event size: Larger events generate more reports, producing undercounts in conflicts where killings happen across many small-fatality events (Friedman, 2015).	Technical
	Capacities and priorities: Fatality counts can correlate with the size and priorities of agency and media teams (UN, 2017).	Both
(2) Coding	Reliability vs accuracy: Codebooks define sources to make coding replicable, often excluding evidence like declassified intelligence. This can produce systematic omissions of deaths belligerents wish to conceal (Twagiramungu et al., 2019).	Technical
	Rashomon effect: Events produce multiple, sometimes contradictory reports. Coders must subjectively choose what to trust, and how to set parameters in a data integration algorithm (Davenport & Ball, 2002; Weidmann, 2015; Donnay et al., 2018).	Both
	Boiling ‘hard’ facts from ‘soft’ reporting: Coders draw ‘hard’ facts about numbers, dates and locations from narratives not necessarily written for numerical accuracy (Weidmann, 2015; De Waal, 2016).	Technical
(3) Analysis	Over-aggregation: Scholars may attribute a particular trend to an entire country, when one event or region may drive an aggregate count (Kalyvas, 2006).	Technical
(4) Use	Manipulation: Political entities may suppress, invent and discredit counts to fit agendas (Aronson, 2013).	Political

analysis. As Table I shows, conditions on the ground determine what evidence coders access. The size of a media presence or investigations team may determine how many fatalities get reported. For example, a team of five human rights investigators might count fewer than a team of ten. Then, what observers count depends on where they go. Political authorities may suppress investigations by confining journalists to towns (Kalyvas, 2004, 2006). Access restrictions also exacerbate event size bias (a situation where observers miss ‘small’-scale atrocities) and communication bias (where observers rely on reports from areas with cellphone coverage) (Weidmann, 2016). Thus, ‘violence that is too dangerous to investigate and report is violence that remains unreported’ (Landman & Gohdes, 2013: 81).

These issues magnify up the chain through coding and analysis to shape political discourse. Controversies over death counts in Rwanda and Libya illustrate this problem. In 1994 Rwanda, African Rights provided the first systematically gathered evidence of killings (Omaar, 1994), which later provoked accusations that the investigator was a Rwandan Patriotic Front sympathizer (Van Oijen, 2018). In Libya, Human Rights Watch estimated fewer civilian fatalities than the US government claimed (Kuperman, 2013). In such cases, coders must choose which sources to trust (Weidmann & Rød, 2015).

Political organizing and fluid advocacy agendas add contingency to Table I, altering the direction of biases non-randomly. For example, human rights entities’ growing attention to rape in Bosnia and Herzegovina may explain the apparent increase in reported incidents over time (Cohen, 2013: 466). Interest levels also affect the intensity of human rights investigations into past events. In Guatemala, journalists missed the surge in rural killings of Mayan communities in the 1980s, in a typical case of urban bias (Ball, Kobrak & Spirer, 1999: Chapter 9). Yet, in the 1990s, Mayan communities powerfully mobilized to demand that human rights organizations intensify their investigations into abuses in the countryside (Nelson, 2015).

The bottom line is that newswire data comprise interpretations of observers (Davenport & Ball, 2002). This leads to conflicting accounts open to arbitrary selection.

Estimation options

The scholars who make newswire data reliable, free, and public have long acknowledged these problems. As the Uppsala Conflict Data Program Georeferenced Event Dataset (GED) codebook asserts: ‘The goal […] is not to present the most complete and accurate image of a certain conflict at a certain point in time, but rather be a tool for the global understanding of subnational conflict patterns and trends’ (Högbladh, 2019: 3). Nonetheless, social scientists frequently assume, implicitly or explicitly, that media reports of killings correlate with actual killings (Krauser, 2020; Berman et al., 2017; Fjelde & Hultman, 2014; Wood, 2014; Toft, 2010). This assumption is more credible in some conflicts than others (Sloboda et al., 2013). Fighting forces target individuals and groups for specific reasons, and make differential choices about which atrocities they conceal or put on display (Fujii, 2013). Political authorities also make non-random decisions about where investigators can go after conflict events. Thus, newswire datasets often omit accurate information about atrocities in the most insecure environments, obstructing micro-level analysis (Landman & Gohdes, 2013; Eck, 2012).

Statisticians have advanced promising remedies for missingness. However, these require high-quality data and explicit probability models to yield precise point estimates. For example, the Matching Event Data by Location, Time and Type (MELTT) method assuages missingness in individual datasets by integrating them using an algorithm to identify matching entries. As the developers of MELTT acknowledge, this still requires high quality data and random missingness to support meaningful estimation (Donnay et al., 2018). Methods like multiple imputation also demand sufficient information about the cause of missingness to allow a missing-at-random assumption (Bauer, Ruby & Pape, 2017).

Multiple systems estimation (MSE) offers another promising alternative by using overlaps between kill lists to estimate deaths recorded nowhere (Landman & Gohdes, 2013; Hoover Green, 2013). The challenge is that MSE is unviable in many contexts, and cannot be performed with newswire data alone. The method, like other epidemiological techniques, demands information at the level of individuals rather than events, and ideally requires at least three different kill lists. Thus, MSE offers a powerful method in contexts with some bureaucratic capacity but does not correct newswire deficiencies.

Given these problems of non-random missingness, what can civil war scholars learn from newswire data about the most volatile conflicts? To answer this question, I turn to South Sudan.

Counting the dead in South Sudan

‘We’ve lost count’ (Hervé Ladsous, cited in Martell, 2016). As the UN peacekeeping chief confessed, humanitarians have struggled to document killings in South Sudan (Maxwell et al., 2018). From 2014 to 2018, officials cited 50,000 dead (Martell, 2014; Reuters, 2016; Council on Foreign Relations, 2018) – a figure that jumped four-fold overnight when the London School of Hygiene and Tropical Medicine (LSHTM) issued a mortality estimate with transparent confidence intervals (Checchi et al., 2018). This section offers vignettes from South Sudan to illuminate how biases in data generation distort what observers think they know about the civilian dead.

Empirical strategy

This article studies South Sudan for methodological and normative reasons. South Sudan’s newswire data exhibit useful variations in types and degrees of bias, with higher quality data for some areas than others. These challenges lead some scholars not to study contexts like South Sudan. Yet, sole focus on data-rich environments eases causal inference at the expense of understanding whole categories of conflict; empirical bias feeds theoretical bias (Arjona & Castilla, 2020). This produces an imperative to optimize rather than reject newswire data in turbulent, low-information conflicts.

I begin by plotting South Sudan’s Armed Conflict Location and Event Data Project (ACLED) and GED data, from December 2013 to December 2015, disaggregating it to investigate missingness (ACLED, 2019; Raleigh et al., 2010; Croicu & Sundberg, 2017; Sundberg & Melander, 2013).² I then conduct more than 40 hours of interviews with 32 people responsible for monitoring civilian killings in South Sudan and other relevant contexts.³ These span 17 UN officials, including

Figure 1.

South Sudan, October 2011 (based on UN map)

peacekeepers; 11 NGO workers, including members of major advocacy organizations; and four wire journalists. I selected interviewees by seeking the original sources in ACLED and GED. For example, ACLED frequently cites Radio Tamazuj, a South Sudanese news outlet based in Nairobi, and thus I interviewed its former editor-in-chief. Where news articles refer to UN statements about killings, I sought the officials involved in investigations. I spoke to Casie Copeland, who produced the 50,000 count. I also pursued sources shared by respondents.⁴

I asked people to describe what they did when they reported on an event, at the scene and afterwards. I prompted interviewees to reflect on whom they systematically missed and could count well. I also asked people’s views on a range of issues, including the feasibility of distinguishing civilians from combatants, and civil war-related violence from ‘intercommunal’ violence. I then extracted references to incidents, time periods, and locations from the transcripts, examining how different accounts clashed or aligned with newswire data.

I received perspectives that covered South Sudan unevenly. Therefore, I focused subnationally on a location and time period for which I could more feasibly ‘know the unknowns’. The resulting case study of Jonglei State, from December 2013 to December 2014, guides the analysis.

Turbulence

South Sudan’s civil war has roots in 50 years of conflicts. When the country gained independence from Sudan in July 2011 (Figure 1), the ruling Sudan People’s Liberation Army/Movement (SPLA/M) inherited myriad political divisions. A shootout between soldiers in the presidential guard on 15 December 2013 triggered violence that spread from

Table II.

Dataset characteristics

Dataset	Civilian definition	Sources	Violent deaths
GED	Unarmed bystanders (Croicu & Sundberg, 2017)	Global newswire reporting; global monitoring, translation of local news by BBC; secondary sources such as local media, NGO and IGO reports, field reports, books	01.12.13–31.12.16: 2,394 (civilians); 5,512 (civilians + unknown)
ACLED	Unarmed people not engaged in political violence (ACLED, 2019)	Local, regional, national and continental media; NGO reports; Africa-focused news reports and analyses	01.12.13–31.12.16: 6,088 (civilians)
LSHTM	NA (estimates general mortality)	Mortality survey data and other sources previously collected by humanitarian agencies	12.13–04.18: 190,000

the army headquarters across the country. Civilians fled their homes to escape ethnically targeted killings, taking refuge on UN bases or fleeing into the bush. Army battalions mutinied and insurgent groups multiplied, with former Vice President Riek Machar launching the Sudan People’s Liberation Movement-in-Opposition (SPLM-IO).

This situation created military and humanitarian turbulence (De Waal, 2015). The UN estimates that the conflict had displaced four million people by 2018, while causing food insecurity and ‘the longest-running cholera outbreak in its history’ (UNOCHA, 2017: 5). By 2016, UNICEF claimed armed groups had recruited more than 17,000 children (UNICEF, 2016). Human rights organizations have accused armed groups of using forced starvation as a tactic, and of perpetrating ‘epic proportions’ of sexual violence (Mednick, 2017; Turse, 2016).

These numbers represent advocacy agendas. But they describe a conflict of chaotic complexity where governing institutions kept few records, and belligerents destroyed what they found.

Who has died: According to the newswire

At first glance, GED and ACLED show large discrepancies (Table II). GED captures less than half of ACLED’s civilian killings, in part because it denotes killings as ‘unknown’ where civilian or combatant identities are unclear. If we assume that most of these ‘unknowns’ are civilians, and GED would code some of ACLED’s killings as ‘unknown’, the gap narrows. However, these counts do not converge across months. Figure 2 presents South Sudan’s aggregate monthly killings; ACLED and GED differ in trend and magnitude. For example, from June to July 2015, ACLED out-counts GED, whereas the opposite is true for the same period in 2014.

Figure 2.

Monthly fatalities, South Sudan, Dec 2013–Dec 2016

The divergences also appear subnationally (Figure 3).⁵ ACLED captures more observations, but GED’s counts exceed ACLED in some periods, notably during an April 2014 government offensive in Unity State.⁶

How counting happens

To understand what these data represent, I interviewed the authors of three types of source dominant in GED and ACLED: South Sudan-focused outlets such as Radio Tamazuj; human rights reports (principally by Human Rights Watch, Amnesty International, and the UN); and civil society and think tank statements, which the newswire reports cross-reference.

Figure 3.

State-level fatalities aggregated by month

Newswire journalism

Journalists expressed surprise when I described how scholars use newswire data. Interview respondents did not believe they had a mandate to produce accurate fatality counts. Instead, they sought to publish first-hand observations in a format that allowed readers to interpret reliability (A26, Martell; A31, Van Oudenaren; A29, Gridneff; A28, journalist). Ilya Gridneff, who had reported for the Associated Press and Bloomberg, explained: ‘The nature of wire reporting is that you hedge your bets by quoting someone else, such as “the SPLA spokesman said there were ten dead.” You never say how many yourself, because you are just reporting what someone said.’

Journalists adopt this approach to fit their professional mandate. Peter Martell, who had reported for AFP, BBC World Service, and IRIN, described newswire reporting as a haphazard, hand-to-mouth existence: ‘Stringing meant living from story to story trying to make a living’, where agencies paid USD 50 for a report. In this context, ‘the objective was to get something onto the wire’. That information was intended for others, like Al Jazeera, to verify further. The problem, Martell explained, is that upstream media houses would revert to his original source, making verification circular. This produced a dilemma:

Do you report nothing [if you can’t triple source], or report what you’ve got noting who has said it […] saying here is the information as we know it at the time? In South Sudan, the second happens more often than not.

Daniel van Oudenaren, former Editor-in-Chief of Radio Tamazuj, explained how this approach fits the ethics of reporting:

Journalists’ responsibility is to tell stories and not collect data for social scientists. The responsibility is to readers […] It requires case-by-case editorial judgements as to whether we trust a person: Even if he is lying, is the number worth printing? Even if the number is questionable, is it worth printing? We may interview a county commissioner who says there was a cattle raid and 20 people were killed. We don’t verify that but we print that […] We can’t wait around nine months for the UN to produce a human rights report. We are not doing our job as journalists if nobody knows there is a massacre in a city until a year afterwards […] So, let’s transcribe what we know, and who knows what and put it out there.

The objective of ‘telling a story’ risks producing self-fulfilling narratives. People noted how the media presence in South Sudan had dwindled with declining international interest as the civil war persisted (A8, human rights advocate; A14, human rights advocate; A16, Copeland; A26, Martell). This meant that reporters prioritized stories they judged relevant to an international narrative about the conflict. As Gridneff commented, ‘you can only write so many famine stories and genocide stories’. Similarly, Van Oudenaren reported a colleague at a wire agency as saying, ‘it’s not enough to run an editorial that has information about fighting […] This is a non-story to Western news outlets’. This means that journalists struggle to publish stories that appear internationally insignificant. One journalist revealed that they rarely reported on intercommunal killings because these were difficult to tie to a global context. Other interviewees judged intercommunal violence a central feature of the civil war, responsible for a large proportion of civilian killings. Instead, on the occasions one journalist filed stories on killings, they were about targeted killings with an apparent ethnic angle, because media houses perceived these stories to explain the dynamics of the war (A28, journalist).

In spite of these problems, interviewees felt pressured to cite numbers. Copeland spoke of journalists’ desperation for death counts. When she voiced an estimate of 50,000 dead, she recounts how the information spread like wildfire through media houses. As Martell explained: ‘If you can quote numbers, it changes the situation […] Numbers make reporting more effective’. However, as Copeland commented, it can take a committed journalist several months to arrive at an accurate account of a single incident, if they can arrive at one at all.

Human rights investigations

Human rights investigators face similar dilemmas. In the case of human rights NGOs, small teams deploy from a regional base for advocacy purposes (A8, human rights advocate). This disposes human rights investigators to report the most egregious cases and not others. Interviewees explained how they sought to register the quality of human rights violations, for which a death count can be counterproductive (A8, human rights advocate; A14 human rights advocate). Indeed, one interviewee noted how death counts derailed debates about the killing of healthcare workers in Syria. UN officials distrusted his organization’s reports because of numerical imprecision, although they had access to robust evidence of targeted killings at a health facility. This was sufficient to demonstrate a violation of international humanitarian law, for which it did not matter whether an armed group had targeted one or 100 healthcare workers. In this context, ‘the accuracy of the count became a distraction’ (A4, human rights advocate).

Accordingly, interviewees emphasized that they sought to establish scale and types of violations, rather than comprehensive accounts of events (A14, human rights advocate). For example, on viewing the aftermath of massacres in Bor town, South Sudan, the Human Rights Watch investigator reported that ‘scores died’, rather than estimate what they could not see (Wheeler, 2014). However, numbers do appear in human rights reports, which coders transfer as data points. Like their journalist colleagues, human rights advocates expressed surprise when I described how scholars use their data. One interviewee commented that, like wire reporters, they include numbers through anecdotal quotes, giving sufficient context for the reader to interpret accuracy. For example, they might write, ‘According to John in X village, his two brothers were killed in the attack’, without any intention of conveying that two was a credible count for the event (A14, human rights advocate).

Civil society and think tank statements

Civil society organizations and think tanks also produce numbers that travel through the newswire. These are attractive sources to journalists, because they derive from ‘expert’ opinion and intentional estimation. For example, Copeland produced the 50,000 count through a multimethod strategy, where she interviewed displaced people and traveled to hospitals across South Sudan to check records. However, Copeland conducted the count alone, since international agencies were over-stretched. She urged readers to interpret her estimate as ‘at least 50,000’ at one moment in time (A16, Copeland).

Remembering the Ones we Lost provides another data source journalists have turned to. South Sudanese volunteers launched the initiative to memorialize the deceased or missing. People submit testimonials online or by text message, while the volunteers conduct their own investigations (A15, South Sudanese human rights advocate). The initiative draws substantially on community lists – local authorities’ informal records of the dead (A25, UN official). However, lists often include missing people who arrive alive in internally displaced people (IDP) camps (A13, UN official; A25 UN official; A30 UN official; A32 UN official). Some saw the lists as useful starting points for investigations due to the quality of local knowledge (A13, UN official; A32 UN official), while others judged them limited in reliability and coverage (A14, human rights advocate; A15, South Sudanese human rights advocate).

This discussion of who produces source data illustrates how technical and political constraints create non-random omissions and distortions in the evidence available to coders. I now turn to how this affects the patterns evident in newswire data.

Who is missing?

Interviewees produced common stories about which populations journalists and human rights reporters count, and which they miss. Investigators have access to a subset of IDPs in UN protection of civilian camps (PoCs), which housed 209,000 of 1.9 million displaced across South Sudan in 2017 (UNOCHA, 2017). The PoCs provide concentrated spaces to interview witnesses of attacks. Consequently, investigators have ‘mapped the PoCs left, right and centre’, using the camps to record testimonies (A15, South Sudanese human rights advocate).

By contrast, a human rights investigator estimated that international agencies were unable to account for a quarter of the population due to poor access to 85% of South Sudan’s terrain (A14, human rights advocate); populations beyond airstrips remained a mystery (A30, UN official). The UN Mission in South Sudan (UNMISS) developed a fatality tracking system in 2012 that corroborated this uncertainty, delivering zero counts for places like Upper Nile state in months where insecurity limited surveillance. Officials kept the counts confidential, to obviate external misinterpretation (A18, UN official).

Interviewees also reported limited data on the elderly and people with disabilities, whom fleeing families often left behind during attacks (A14, human rights advocate; A25, UN official). One investigator falsified allegations that an armed group had executed the elderly in Malakal when he discovered combatants had brought them to safety: ‘The soldiers weren’t angels, but treated them decently’ (A14, human rights advocate).

Interviewees also feared that information was more prevalent about some ethnicities than others. For example, an investigator alleged that local authorities suppressed information about the Shilluk dead in Malakal (A8, human rights advocate). Others found it difficult to count the Dinka dead due to cultural taboos around talking about the deceased (A14, human rights advocate). Some predicted undercounts, while others feared Dinka civilians were overcounted due to the (Dinka-dominated) government’s interests in revealing rebel-perpetrated abuses (A16, Copeland; A3, UN official).

I also heard concerns that newswire data undercounts ‘intercommunal’ civilian killings, although ACLED captures more of these incidents than other sources (A5, South Sudanese academic). To compound journalists’ problems with pitching these stories, interviewees disagreed about whether intercommunal killings counted as direct conflict deaths. Some saw intercommunal violence as criminal, unrelated to civil war politics (A12, NGO worker), while others judged it inseparable from the broader conflict narrative. One UN official saw the line between crime and armed conflict as so fluid in South Sudan as to be meaningless (A32, UN official). Others offered examples of politicians manipulating cattle raids as proxies for political retaliation. Such acts bind local disputes to the national politics driving civil war (A15, South Sudanese human rights advocate; A14, human rights advocate; A5, South Sudanese academic; A3, UN official).

Fatalities in Jonglei state

Even within subnational units, sources of missingness vary in ways that preclude capturing them in a probability model. Vignettes from Jonglei state illustrate this. Jonglei sits on the Sudd, the largest marsh on earth (Figure 4): ‘Multilingual; muddy and remote; mutinous and divided against itself: Jonglei exemplifies South Sudan’ (Thomas, 2015: 278). It is an illuminating case due to its political significance, which should draw journalists’ attention.

In December 2013, Jonglei became the frontline between government forces and the SPLM-IO. The violence initially hit the state capital, Bor town, which changed hands four times in four weeks when armed groups killed and evicted civilians. Many took refuge on the UNMISS base, to be terrorized when armed youth attacked the PoC camp on 17 April 2014.

Other areas of Jonglei, previously wrought with violence, experienced calm. Pibor – a year earlier, the center of an insurgency against the government, led by David Yau Yau’s SSDM/A-Cobra Faction (Cobra Faction) – became one of the most stable places in South Sudan in 2014. When the civil war erupted, both the government and SPLM-IO sought to co-opt the Cobra Faction. Yau Yau used this to bargain for special autonomy in Pibor.

These dynamics appear inconsistently in the newswire data. Figure 5 illustrates fatalities aggregated by month; months without data points signal either no killings or no information. Observable data points at ‘zero’ code recorded conflict events without reported civilian deaths. Figure 6 presents these data padded with zero counts in periods without observations, and Figure 7 decomposes the data for Jonglei’s 11 counties.

Figure 4.

Jonglei state, December 2016 (based on UN map)

Figure 5.

Monthly fatalities, Jonglei, Dec 2013–Dec 2014

If no observations reflect no violence, Fangak, Pibor, Uror, and Nyriol appear peaceful. While this assumption fits Pibor in 2014, it seems suspect for the other counties. I explore these data gaps in vignettes of a correct count, an undercount, an overcount, and missing data.

Pibor

A former UN official (A25) described visiting Pibor in March 2015: ‘It was remarkable – an oasis of peace with sheep grazing by the river. There were Cobra

Figure 6.

Monthly fatalities, Jonglei, Dec 2013–Dec 2014 (padded)

Faction soldiers drinking tea next to SPLA – a real contrast to the 2012–13 situation.’ Few observations in ACLED and GED may accurately reflect a situation where there were few conflict events in Pibor in 2014.

Bor town, December 2013–January 2014

In November 2013, Bor was much like Pibor in March 2015. That month, I (the author of this article) had sauntered through a lively town, drinking tea and eating fresh perch from the Nile. On 16 December 2013, I started evacuating colleagues from the spreading violence. My colleague in Bor told me things were calm and to prioritize the team in Juba. He called me the next morning as he fled towards the UN along a road littered with bodies.

‘After this, things got blurry for days’ (A25, UN official). International agencies evacuated. On 26 December the UN redeployed civilians under movement restrictions. In the first month of the conflict, Bor became a ‘black hole’ to international observers while armed groups killed and evicted civilians (A13, UN official).

The Bor town mayor forbade the few remaining residents from moving bodies (A13, UN official). Once government control returned, the mayor brought journalists and human rights advocates to observe the scene:

Figure 7.

Fatalities by county in Jonglei state aggregated by month, Dec 2013–Dec 2014 (Nyirol and Pibor excluded due to no recorded events)

Table III.

Bor town killings

Source	FatalitiesDec 13– Feb 14
GED, Bor town (civilians)	163
GED, Bor town (civilians + unknown)	196
ACLED, Bor town (civilians)	246
Collective graves (para 114, UNMISS, May 2014)	525
Local authorities’ estimates for Bor county (Bor town and surrounding rural area) (para 114, UNMISS, May 2014)	2,007
UN official’s gut estimate for Bor county (A13)	2,000–6,000
Human Rights Watch	‘Scores’

There were bodies everywhere. We would go into houses and find civilians shot […] I will never forget this woman hiding under her bed and seeing her hand coming out. It stank appallingly […] The whole of Bor was a massive crime scene. It was pathetic. (A8, human rights advocate)

Following this visit, local authorities requested that the UN clean the town:

There were all these dead bodies, all over town, weeks into the crisis. The authorities asked the UN to help with body removal. The ICRC was not there to do it, and only had a written guide available, which is what we relied on […] We tried to do some mapping of dead bodies by sector, but it was a very rough estimation. A drop in the bucket. (A13, UN official)

UNMISS personnel removed the bodies to collective graves, documenting what they could:

The numbers that made it into the UN’s public report were the numbers buried in the collective graves. We are confident of these numbers because the UN military picked up the bodies and buried them. But they were not identified, and we didn’t check if they were civilians. (A13, UN official)

Many more died crossing the Nile and remain uncounted (A13, UN official; A25, UN official). Table III illustrates the divergence between counts.

Attack on UNMISS Bor base, 17 April 2014

An attack on the UNMISS Bor base illustrates an overcount in a situation where events unfolded before investigators’ eyes in a confined space (A25, UN official). Officials faced confusion as youths started shooting at civilians on UN premises: ‘There were good intentions that meant bodies were removed and the injured treated. But it was done by [untrained] humanitarians and military personnel’ (A13, UN official). For several days, a security lockdown restricted UNMISS staff from investigating evidence outside the base (A13, UN official). This meant that the UN required months to confirm the attack’s death count: ‘The situation involved a fairly finite physical space […] and yet we still had to contend with controversy over the numbers’ (A13, UN official).

Community leaders immediately called media outlets, claiming hundreds of deaths. They later notified the UN of 146 dead, of which they could identify 55 by name, some of whom appeared alive among the displaced. The UN found forensic evidence of 53 dead (UNMISS, 2015: paras 107–108). GED counts 47 dead, which accurately reflects the UN’s count of the IDP dead (UNMISS, 2015: para 105). However, ACLED counts 60 civilian dead.

Northern Jonglei

The most difficult missing observations to interpret relate to places unmonitored by any agency, such as northern Jonglei: ‘We didn’t know what we weren’t seeing’ (A21, UN official); ‘An intelligent person would know the single digits we were getting [from the UN fatality tracking system] were wrong’ (A18, UN official). But the scale of the undercount was unclear: ‘In the early months, we had no idea of what was going on in the Nuer areas of northern Jonglei […] we presumed they [Dinka and other non-Nuer] had to have been killed’ (A25, UN official). This assumption proved wrong when UN officials discovered rebels had imprisoned rather than killed many Dinka (A25, UN official). An aid worker summed up the uncertainty in Canal/Pigi county specifically:

It is a place that’s caught in the middle. We have no sense of how bad things really are. It is clearly not the worst place in Jonglei. At the same time, the security is not good enough for a long-term presence […] The only thing we see is food security data from the World Food Program, which looks bad for the country but is not the worst in Jonglei […] There is not enough evidence of things going really wrong for people to look at it. (A12, NGO worker)

Counting the dead better

The Jonglei vignettes illustrate empirical dilemmas with elusive solutions. Nevertheless, researchers may consider two steps when handling newswire data: (1) elucidate biases; and (2) measure prevalence.

Biases

This inquiry suggests scholars should train their eye on at least seven types of bias.

Accessibility bias appears the most serious cause of data distortions. Access challenges produce non-random blackholes, obscuring entire regions. Political meddling is not necessarily the cause. While a challenge, interviewees judged: ‘There is no single place in South Sudan to which people want to go, are not able to go and haven’t been able to go’ (A16, Copeland); ‘Access issues are an excuse’ (A28, journalist). Instead, interviewees explained that agencies’ priorities determine where they invest resources and thus where they access.

This feeds self-fulfilling narrative bias. What international media houses perceive the conflict to be about determines where reporters go and what they wire. If media houses perceive intercommunal violence to be irrelevant to the civil war, then newswire datasets will not capture these incidents. Similarly, if agencies perceive areas like northern Jonglei to be non-stories, they will register few events in datasets.

Journalists’ urban bias exacerbates self-fulfilling narratives: ‘Journalism is primarily an urban phenomenon. Most of the international and South Sudanese journalists I worked with were urban people. So, their contacts with South Sudanese in rural areas were tenuous’ – although that is where most of the violence was happening (A31, Van Oudenaran). Gridneff described the difficulties of wiring stories outside Juba: ‘If you went to Bentiu, you would wait around and have nothing to file. Meanwhile, you would miss the offensive in Malakal, but if you were there, you would have no transport and no means to file’ (A29, Gridneff). Thus, the fatality patterns we see in newswire data often reflect what reporters can observe from capitals.

Respondents feared that event-size bias produced undercounts. Several noted that most events involved less than ten people, which they would rarely investigate (A5, South Sudanese academic; A14, human right advocate; A16, Martell; A30, UN official; A31, Van Oudenaren): ‘I realize that my own threshold for remembering or counting an incident […] was about 40 deaths upwards’ (A30, UN official).

The South Sudan case also highlights data distortions from inconsistencies in the categorization of civilians and combatants. While, on paper, uniforms connote combatants, ‘uniforms [in South Sudan] are not uniformized’; anybody can go to the market and buy one (A3, UN official). Conversely, combatants have allegedly changed uniforms for civilian clothes at the gates of PoC sites (A13, UN official). Fuzzy distinctions preclude UN agencies from disaggregating their fatality data and drive human rights advocates to qualify their statements with caveats.

I also heard accounts of political bias where belligerents had circulated disinformation and meddled with evidence (A16, Copeland; A29, Gridneff; A31, Van Oudenaren). The overall effect is difficult to estimate because political bias can vary over time in a single context. As Copeland describes:

The chaos of the first month or two of the war created a space for well-intentioned people to communicate what was taking place with relatively minimal interference. Once the government was competent again in Juba structures formalized again […] All of a sudden, a national security person was with the people you were interviewing. Either people were brave and would tell the truth, or they would pick selectively […] Space has closed since the beginning of the conflict.

Lastly, my qualitative data illustrate the rarity of eyewitness testimonies (A8, human rights advocate; A13, UN official; A14, human rights advocate; A26, Martell; A28, journalist; A29, Gridneff): ‘Few people have seen fighting, because it happens in small spurts all over the country’ (A28, journalist). People flee pre-emptively. This can produce overcounts as people extrapolate from an incident several kilometers behind them. Self-reinforcing conversations among traumatized civilians in displaced camps then solidify these perceptions (A14, human rights advocates).

Prevalence

These biases generate non-random missingness, threatening statistical inferences based on point estimates in places like South Sudan. Absent a fix, researchers might consider analyzing the prevalence of violence using categorical measures. This method is neither novel nor perfect. However, in some situations it illuminates magnitudes, patterns, and uncertainties more reliably than counts. To illustrate, I compare a measure of fatality prevalence against numbers for Jonglei.

The following approach adapts a decades-old method. The human rights field developed categorical variables for violence in the 1980s, when the Political Terror Scale project translated a coding scheme from Freedom House to measure violations (Stohl, Carleton & Johnson, 1984).

Figure 8.

Monthly fatalities by county in Jonglei (padded), Dec 2013–Dec 2014

Cohen and Nordås refined this idea to measure reported prevalence of sexual and gender-based violence (SGBV) on a four-point scale, ranging from ‘none’ to ‘massive’ (Cohen, 2013; Cohen & Nordås, 2014: 420). Their coding rules supplement thresholds for victim counts with principles for interpreting qualitative descriptions of SGBV. For example, countries that fit the ‘massive’ reported SGBV category in a given year must yield reports that use adjectives like ‘systematic’ and ‘innumerable’ to describe violations.

This approach optimizes qualitative information when numbers are uncertain. Consider, for example, what descriptive data show that reported numbers obscure about an attack on the UN base in Akobo, Jonglei state on 19 December 2013. ACLED registers a zero count, and GED two ‘unknown’ killings.⁷ Yet, public reports describe numerous fatalities:

Many civilians were killed in the attack. Due to the chaotic circumstances and the inability of survivors to observe the full attack as their movements were restricted, the HRD [Human Rights Division] has not been able to determine an exact figure of fatalities. However, any Dinka civilians not evacuated by UNMISS are presumed dead, and at least 20 civilians, as well as two peacekeepers, were killed. (UNMISS, 2014: para 143)

This report portrays a stark contrast to ACLED and GED’s counts.

While attractive in these situations, it is important to note that categorical variables blunt rather than resolve the problems this article explores. They also generate their own dilemmas. As Cohen recognizes, report-based prevalence measures face similar biases to newswire data (violence that is too dangerous to report will go unreported, descriptively and numerically). Therefore, researchers may only be able to discern patterns at a high level of aggregation – in her case, the country-year (Cohen, 2013: 466–467). Prevalence measures are also normative projects. Categories make populations legible for a political or scholarly purpose (Scott, 1998). They require decisions about who counts as a victim, and what thresholds indicate moderate or extreme abuses (Nelson, 2015).

Nonetheless, prevalence measures reveal fatality patterns that numbers can obscure. I illustrate for county-months in Jonglei, December 2013 to December 2014 (Figures 8 and 9). Table IV describes my coding rules for

Figure 9.

Monthly fatality levels by county in Jonglei, Dec 2013–Dec 2014

designating county-months as involving ‘unknown’, ‘low’, ‘medium’ or ‘high’ civilian fatality levels.⁸ At first glance, this protocol adopts a low bar for classifying county-months as ‘medium’ or ‘high’ fatality. However, the rules hinge on a contextual interpretation of what it means for an event to register in any report. Incidents that happen beyond airstrips rarely get investigated (A30, UN official). Therefore, recorded fatalities tend to be the tip of an iceberg. I assume that accounts of any killings, forced displacement or armed group activity signal at least some fatalities.

Figures 8 and 9 show how civilians in two counties that look alike in the newswire may have divergent experiences of killings. For example, Fangak and Pibor both register zero counts in ACLED and GED. Yet, qualitative descriptions indicate that Fangak experienced violence that likely killed civilians, while Pibor remained calm in 2014.

Table IV.

Coding rules

Civilian fatality level	Rule
High	Published source registers some civilian or unknown fatalities, AND interview data or published source suggests civilian killing was ‘massive’, ‘widespread’, ‘large’, ‘collectively targeted’ or ‘indiscriminate’, or that ‘large numbers’, ‘many people’ or ‘scores’ were killed.
Medium	Published source or interview registers some fatalities or some fighting between combatants, or displacement of civilians due to fighting, or evidence of major troop movements, or civilians being ‘shot at’ or ‘attacked’, BUT does not use the descriptors in the ‘high’ category.
Low	Published source or interview describes ‘calm’ or cites political events that should preclude civilian killings.
Unknown	Interview and published data absent.

Conclusions

This study has normative and empirical implications. It raises questions about whether counting the dead is analytically useful and politically ethical in contexts like South Sudan. Death counts are prerequisite to memorialization, criminal justice, humanitarian targeting, and meaningful scholarship on political violence. Yet, ‘there is a risk when things are so messy and complicated that throwing something like a death toll out there is a shiny distraction from the issue at hand’ (A16, Copeland). Why, as scholars, human rights advocates, and policymakers, do we obsess about numbers in these situations? Can data that miss so many of the dead advance knowledge?

Maintaining the hope that it can, this study has two implications for social science. First, precise point estimates about the causes and consequences of violence may be impossible using newswire data. Scholars face neither statistical nor qualitative silver bullets to resolve bias problems. Categorical variables can reveal patterns where newswire data are difficult to interpret, but do not solve non-random missingness. Thus, this article poses a challenge to political methodologists: Can social scientists build better cross-national fatality measures, given the biases inherent in the newswire-based data-generation process? While this article offers few answers, it is meant to stimulate this debate.

Secondly, body counts reveal little if divorced from circumstance. It is impossible to construct theories about violence with numbers decoupled from the human experiences that lie beneath. As a former aid worker (A23) shared: ‘It is the nature of killing that matters, not the number.’ He recounted how a man with disabilities was castrated, killed, and left on display in the Malakal PoC camp:

There were riots a month later and the Dinka left […] [because they feared] they would be ethnically cleansed […] If you look at fatality data this man appears as only one death but that body was used to convey something important.

Social scientists owe it to both knowledge advancement and the dignity of the dead to ground statistics in context.

Footnotes

Replication data

The datasets and R scripts for the empirical analysis in this article, along with the Online appendix, can be found at .

Acknowledgements

I thank Kate Baldwin, Ali Zeynel Gökpınar, Stathis Kalyvas, Jana Krause, Keith Krause, Anne-Kathrin Kreft, Jule Krüger, Roxani Krystalli, Jason Lyall, Ian Shapiro, Jessica Stanton, Jason Stearns, Alex de Waal, Martin Waehlisch, Skye Wheeler, Elisabeth Wood, and participants at the International Studies Association Annual Convention 2019 and Yale African Politics Working Group.

Funding

This article was supported by a US Institute of Peace (USIP) Peace Scholar Award. The views expressed in this article are those of the author and do not necessarily reflect the views of USIP.

ORCID iD

Sophia Dawkins

Notes

References

ACLED (2019) Armed conflict location and event data project (ACLED) codebook (https://acleddata.com/acleddatanew/wp-content/uploads/dlm_uploads/2019/04/ACLED_Codebook_2019FINAL_pbl.pdf).

Arjona

Ana

Castilla

Juan Pablo

(2020) The violent bias in the study of civil war. Working paper, presented at Yale University, 27 February.

Aronson

Jay

(2013) The politics of civilian casualty counts. In: Seybolt, Aronson & Fischoff, 29–50.

Ball

Patrick

(1996) Who did what to whom? Planning and implementing a large-scale human rights data project. American Association for the Advancement of Science (https://hrdag.org/publications/who-did-what-to-whom-planning-and-implementing-a-large-scale-human-rights-data-project/).

Ball

Patrick

Kobrak

Paul

Spirer

Herbert F

(1999) State violence in Guatemala, 1960–1996: A quantitative reflection. American Association for the Advancement of Science (https://hrdag.org/wp-content/uploads/2013/01/state-violence-guate-1999.pdf).

Bauer

Vincent

Ruby

Keven

Pape

Robert

(2017) Solving the problem of unattributed political violence. Journal of Conflict Resolution 61(7): 1537–1564.

Berman

Nicolas

Couttenier

Mathieu

Rohner

Dominic

Thoenig

Matthias

(2017) This mine is mine! How conflict minerals fuel conflicts in Africa. American Economic Review 107(6): 1564–1610.

Checchi

Francesco

(2010) Estimating the number of civilian deaths from armed conflicts. Lancet 375(9711): 255–257.

Checchi

Francesco

Testa

Adrienne

Warsame

Abdihamid

Quach

Burns

Rachel

(2018) Estimates of crisis-attributable mortality in South Sudan, December 2013 – April 2018. London School of Hygiene and Tropical Medicine (https://www.lshtm.ac.uk/south-sudan-full-report).

10.

Cohen

Dara Kay

(2013) Explaining rape during civil war: Cross-national evidence (1980–2009). American Political Science Review 107(3): 461–477.

11.

Cohen

Dara Kay

Nordås

Ragnhild

(2014) Sexual violence in armed conflict: Introducing the SVAC dataset, 1989–2009. Journal of Peace Research 51(3): 418–428.

12.

Council on Foreign Relations (2018) Global conflict tracker: Civil war in South Sudan (https://www.cfr.org/global/global-conflict-tracker/p32137#!/).

13.

Croicu

Mihai

Sundberg

Ralph

(2017) UCDP georeferenced event dataset codebook version 17.1. Department of Peace and Conflict Research, Uppsala University (https://ucdp.uu.se/downloads/ged/ged172.pdf).

14.

Davenport

Christian

Ball

Patrick

(2002) Views to a kill: Exploring the implications of source selection in the case of Guatemalan state terror, 1977–1995. Journal of Conflict Resolution 46(3): 427–450.

15.

De Waal

Alex

(2015) The Real Politics of the Horn of Africa: Money, War and the Business of Power. Cambridge: Polity.

16.

De Waal

Alex

(2016) Writing human rights and getting it wrong. Boston Review 6 June (https://bostonreview.net/world/alex-de-waal-writing-human-rights).

17.

Donnay

Karsten

Dunford

Eric T

McGrath

Erin C

Backer

David

Cunningham

David E

(2018) Integrating conflict event data. Journal of Conflict Resolution 63(5): 1–28.

18.

Dulic

Tomislav

(2004) Tito’s slaughterhouse: A critical analysis of Rummel’s work on democide. Journal of Peace Research 41(1): 85–102.

19.

Eck

Kristine

(2011) Survey research in conflict and post-conflict societies. In: Högland

Kristine

Öberg

Magnus

(eds) Understanding Peace Research: Methods and Challenges. London: Routledge, 165–182.

20.

Eck

Kristine

(2012) In data we trust? A comparison of UCDP GED and ACLED conflict events datasets. Cooperation and Conflict 47(1): 124–141.

21.

Fjelde

Hanne

Hultman

Lisa

(2014) Weakening the enemy: A disaggregated study of violence against civilians in Africa. Journal of Conflict Resolution 58(7): 1230–1257.

22.

Foster Lynch

Meghan

(2013) Collecting data on civilian casualties: Scientific challenges and ethnographic solutions. In: Seybolt, Aronson & Fischoff, 123–143.

23.

Friedman

Jeffrey A

(2015) Using power laws to estimate conflict size. Journal of Conflict Resolution 59(7): 1216–1241.

24.

Fujii

Lee Ann

(2013) The puzzle of extra-lethal violence. Perspectives on Politics 11(2): 410–426.

25.

Gade

Emily Kalah

(2010) Defining the non-combatant: How do we determine who is worthy of protection in violent conflict? Journal of Military Ethics 9(3): 219–242.

26.

Gerber

Alan S

Green

Donald P

Kaplan

Edward H

(2014) The illusion of learning from observational research. In: Teele

Dawn Langan

(ed.) Field Experiments and Their Critics: Essays on the Uses and Abuses of Experimentation in the Social Sciences. New Haven, CT: Yale University Press, 9–32.

27.

Högbladh

Stina

(2019) UCDP GED codebook version 19.1. Department of Peace and Conflict Research, Uppsala University (https://ucdp.uu.se/downloads/ged/ged191.pdf).

28.

Hoover Green

Amelia

(2013) Multiple systems estimation: The basics. Human Rights Data Analysis Group (http://hrdag.org/2013/03/11/mse-the-basics/).

29.

Kalyvas

Stathis N

(2004) The urban bias in research on civil wars. Security Studies 13(3): 160–190.

30.

Kalyvas

Stathis N

(2006) The Logic of Violence in Civil War. Cambridge: Cambridge University Press.

31.

Kleinfeld

Rachel

(2017) Reducing all violent deaths, everywhere: Why the data must improve. Carnegie Endowment for International Peace (https://carnegieendowment.org/files/CP_297_Kleinfeld_Crime_Final_Web.pdf).

32.

Krauser

Mario

(2020) In the eye of the storm: Rebel taxation of artisanal mines and strategies of violence. Journal of Conflict Resolution 64(10): 1968–1993.

33.

Kreuzer

Marcus

(2010) Historical knowledge and quantitative analysis: The case of the origins of proportional representation. American Political Science Review 104(2): 369–392.

34.

Kuperman

Alan J

(2013) A model humanitarian intervention? Reassessing NATO’s Libya campaign. International Security 38(1): 105–136.

35.

Landman

Todd

Gohdes

Anita

(2013) A matter of convenience: Challenges of non-random data in analyzing human rights violations during conflicts in Peru and Sierra Leone. In: Seybolt, Aronson & Fischoff, 77–93.

36.

Manrique-Vallier

Daniel

Price

Megan E

Gohdes

Anita

(2013) Multiple systems estimation techniques for estimating casualties in armed conflicts. In: Seybolt, Aronson & Fischoff, 165–182.

37.

Martell

Peter

(2014) 50,000 and not counting: South Sudan’s war dead. Agence France-Presse 15 November (http://www.dailymail.co.uk/wires/afp/article-2836137/50-000-not-counting-South-Sudans-war-dead.html).

38.

Martell

Peter

(2016) South Sudan is dying, and nobody is counting. Agence France-Press 10 March (http://news.yahoo.com/south-sudan-dying-nobody-counting-045835407.html).

39.

Maxwell

Daniel

Hailey

Peter

Kim

Jeeyon Janet

McCloskey

Erin

Wrabel

Maria

(2018) Constraints and complexities of information and analysis in humanitarian emergencies: Evidence from South Sudan. Feinstein International Center, Tufts University (https://fic.tufts.edu/wp-content/uploads/SouthSudan-Case-Study-Report.pdf).

40.

Mednick

Sam

(2017) Sexual violence reaches ‘epic proportions’ in South Sudan’s civil war. Boston Globe 25 March (https://www.bostonglobe.com/news/world/2017/03/25/sexual-violence-reaches-epic-proportions-south-sudan-civil-war/ODXhTsiM8I7LpWiP061neO/story.html).

41.

Merry

Sally Engle

(2016) The Seductions of Quantification: Measuring Human Rights, Gender Violence, and Sex Trafficking. Chicago, IL: University of Chicago Press.

42.

Nelson

Diane M

(2015) Who Counts? The Mathematics of Death and Life after Genocide. Durham, NC: Duke University Press.

43.

Nordstrom

Carolyn

(1997) A Different Kind of War Story. Philadelphia, PA: University of Pennsylvania Press.

44.

Omaar

Rakiya

(1994) Rwanda: Death, Despair and Defiance. London: African Rights.

45.

Raleigh

Clionadh

Linke

Andrew

Hegre

Håvard

Karlsen

Joakim

(2010) Introducing ACLED: An armed conflict location and event dataset. Journal of Peace Research 47(5): 651–660.

46.

Ratnayake

Ruwan

Degomme

Olivier

Guha-Sapir

Debarati

(2009) Coming together to document mortality in conflict situations: Proceedings of a symposium. Conflict and Health 3(2). https://doi.org/10.1186/1752-1505-3-2.

47.

Remembering the Ones we Lost (2018) Website (http://rememberingoneswelost.com/).

48.

Reuters (2016) UN official says at least 50,000 dead in South Sudan war. 3 March (https://www.reuters.com/article/us-southsudan-unrest-un/u-n-official-says-at-least-50000-dead-in-south-sudan-war-idUSKCN0W503Q).

49.

Sambanis

Nicholas

(2004) What is civil war? Conceptual and empirical complexities of an operational definition. Journal of Conflict Resolution 48(6): 814–858.

50.

Scott

James C

(1998) Seeing Like a State. New Haven, CT: Yale University Press.

51.

Seybolt

Taylor

Aronson

Jay

Fischoff

Baruch

(2013) Counting Civilian Casualties: An Introduction to Recording and Estimating Nonmilitary Deaths in Conflict. Oxford: Oxford University Press.

52.

Sloboda

John

Dardagan

Hamit

Spagat

Michael

Hicks

Madelyn Hsiao-Rei

(2013) Iraq body count: A case study in the uses of incident-based conflict casualty data. In: Seybolt, Aronson & Fischoff, 53–76.

53.

Spagat

Michael

Mack

Andrew

Cooper

Tara

Kreutz

Joakim

(2009) Estimating war deaths: An arena of contestation. Journal of Conflict Resolution 53(6): 934–950.

54.

Stohl

Michael

Carleton

David

Johnson

Steven E

(1984) Human rights and US foreign assistance from Nixon to Carter. Journal of Peace Research 21(3): 215–226.

55.

Sundberg

Ralph

Melander

Erik

(2013) Introducing the UCDP georeferenced event dataset. Journal of Peace Research 50(4): 523–532.

56.

Thomas

Edward

(2015) South Sudan: A Slow Liberation. London: Zed.

57.

Tishkov

Valery

(1999) Ethnic conflicts in the former USSR: The use and misuse of typologies and data. Journal of Peace Research 36(5): 571–591.

58.

Toft

Monica Duffy

(2010) Ending civil wars: A case for rebel victory? International Security 34(4): 7–36.

59.

Turse

Nick

(2016) Next Time They’ll Come to Count the Dead: War and Survival in South Sudan. Chicago, IL: Haymarket.

60.

Twagiramungu

Noel

Duursma

Allard

Berhe

Mulugeta Gebrehiwot

Waal

Alex de

(2019) Re-describing transnational conflict in Africa. Journal of Modern African Studies 57(3): 377–391.

61.

UN (2017) Public reporting on human rights by United Nations peace operations: Good practices, lessons learned and challenges (http://dag.un.org/handle/11176/400896).

62.

UNICEF (2016) Rise in child recruitment as conflict in South Sudan enters fourth year. 15 December (https://www.unicef.org/media/media_94185.html).

63.

UNMISS (2014) Conflict in South Sudan: A human rights report (https://unmiss.unmissions.org/sites/default/files/unmiss_conflict_in_south_sudan_-_a_human_rights_report.pdf).

64.

UNMISS (2015) Attacks on civilians in Bentiu & Bor, April 2014 (http://www.ohchr.org/Documents/Countries/SS/UNMISS_HRDJanuary2015.pdf).

65.

UNOCHA (2017) Humanitarian response plan, January–December 2018: South Sudan (https://reliefweb.int/sites/reliefweb.int/files/resources/SS_2018_HumanitarianResponsePlan.pdf).

66.

Van Oijen

Jos

(2018) When facts cease to matter: The polarised world of Rwanda research. African Arguments 29 January (http://africanarguments.org/2018/01/29/when-facts-cease-to-matter-the-polarised-world-of-rwanda-research/).

67.

Weidmann

Nils B

(2015) On the accuracy of media-based conflict event data. Journal of Conflict Resolution 59(6): 1129–1149.

68.

Weidmann

Nils B

(2016) A closer look at reporting bias in conflict event data. American Journal of Political Science 60(1): 206–218.

69.

Weidmann

Nils B

Rød

Espen Geelmuyden

(2015) Making uncertainty explicit: Separating reports and events in the coding of violence and contention. Journal of Peace Research 52(1): 125–128.

70.

Werth

Nicolas

(1997) Un État contre son people: Violences, répressions, terreurs en union soviétique [A state against its people: Violence, repression and terror in the Soviet Union]. In: Courtois

Stéphanie

Werth

Nicolas

Panné

Jean-Louis

Paczkowski

Andrzej

Bartosek

Karel

Margolin

Jean-Louis

(eds) Le livre noir du communisme: Crimes, terreur, répression [The black book of Communism : Crime, terror, repression]. Paris: Robert Laffont, 39–312.

71.

Wheeler

Skye

(2014) Dispatches: Counting the dead in South Sudan. Human Rights Watch 30 January (https://www.hrw.org/news/2014/01/30/dispatches-counting-dead-south-sudan).

72.

Williams

Paul D

(2016) War and Conflict in Africa. Cambridge: Polity.

73.

Wood

Elisabeth Jean

(2003) Insurgent Collective Action and Civil War in El Salvador. Cambridge: Cambridge University Press.

74.

Wood

Reed M

(2014) Opportunities to kill or incentives for restraint? Rebel capabilities, the origins of support, and civilian victimization in civil war. Conflict Management and Peace Science 31(5): 461–480.