Where are human subjects in Big Data research? The emerging ethics divide

Introduction

Critical data studies is in its infancy, but it faces a substantial challenge: as the practice of data science surges ahead, we lack a strong and rigorous sense of ethical parameters for scientific research. There are several problems emerging. First, there is a growing divide between established systems of research ethics in more traditional disciplines and the dynamic norms and research methods of Big Data. Big Data research methods exacerbate a long-standing tension between the social sciences and research regulations that are geared to the methods and harms of biomedical research. Second, US research regulations (both the current rules and proposed revisions) exempt projects that make use of already existing, publicly available datasets on the assumption that they pose only minimal risks to the human subjects they document. ¹ But this assumption is founded on a misconception. Publicly available data can be put to a wide range of secondary uses, including being combined with other data sets, that can pose serious risks to individuals and communities. This is one of several risks that are being overlooked in the current debates about the ethics of Big Data studies.

For example, in 2016, a group of researchers published a study that sought to reveal the identity of British artist Banksy, who has sought to keep his real name out of the public domain (Hauge et al., 2016). They used geographical profiling, a technique of statistical inference traditionally used in serial crimes like rape and murder, to hone in to a suspected person. They analyzed the spatial patterns of Banksy’s artworks around London and Bristol, and then tracked a particular individual who had been named by the Daily Mail as likely to be Banksy. ² They searched the electoral rolls for this person’s former addresses as well as those of his wife, and places where he likely went to school and played football. Then Banksy’s public artworks were mapped against these streets and neighborhoods. They investigated no other “suspects” but argue that their findings support those of the Daily Mail. The researchers claim that their approach could be useful for early identification of terrorists, as “terrorists often also engage in low level activities such as vandalism, graffiti, anti-government leaflet distribution, and banner posting” (Hauge et al., 2016: 5).

There are many questions that could be asked of this study, not least about the correlation between graffiti and terrorism. But for our purposes, we will only focus on the “ethical note” that appeared at the end of the article: “the authors are aware of, and respectful of, the privacy of [subject name removed] and his relatives and have thus only used data in the public domain” (Hauge et al., 2016: 5). This claim is particularly striking, as it is difficult to see how tracking a specific individual (and their family) to such an invasive degree could be considered respectful of their privacy. But there are now so many data sets about individuals in the public domain, that, while relatively innocuous in themselves, become highly identifying when brought together. The Banksy study is not a large-scale data study, but it echoes the argument made by many Big Data researchers that they are absolved of ethical concerns by pointing to the “publicness” of the data they use. By applying specialized tools for tracking terrorists, Hauge et al. revealed sensitive patterns of movement over several decades. Though they only delved into public data stores, they exploited everything they could find about an artist’s personal (and creative) life, and cross-referenced it with the details of a private citizen, in order to expose an identity that the artist sought to keep secret.

The researchers who published the Banksy study say they went through review from an independent ethics board, and while we cannot see their determination, it is likely that they were allowed to track their suspected individual because the data was public as that is a common standard across research ethics regulations. ³ We argue it is a useful case study of why public data can be incredibly invasive, and potentially harmful. Critical data studies has an important role to play in analyzing and clarifying these issues by situating questions of data ethics regulations and norms within a historical and discursive analysis of the core concepts and norms of research ethics in general. By historicizing extant research ethics norms and regulations, we are able to see the disjunctions with the epistemic conditions of data sciences as one more site of negotiation and improvement rather than an implacable conflict.

Big Data stretches our concepts of ethical research in significant ways (boyd and Crawford, 2012). It moves ethical inquiry away from traditional harms such as physical pain or a shortened lifespan to less tangible concepts such as information privacy impact and data discrimination. It may involve the traditional concept of a human subject as an individual, or it may affect a much wider distributed grouping or classification of people. It fundamentally changes our understanding of research data to be (at least in theory) infinitely connectable, indefinitely repurposable, continuously updatable and easily removed from the context of collection. By doing so, it forces us to grapple with the ways in which familiar and practical ethical constraints depended upon research data being temporally and contextually constrained and restricted by technical infrastructures and financial cost. Further, data science methods create an abstract relationship between researchers and subjects, where work is being done at a distant remove from the communities most concerned, and where consent often amounts to an unread terms of service or a vague privacy policy. Together, these shifts are hard to quantify and ameliorate (Zwitter, 2014), frustrating the familiar ethical practices outside of biomedical research. So while extant research ethics and regulations are far from a perfect fit for the methods of Big Data, there is real urgency to define what a “human subject” is in Big Data research and critically interrogate what is owed to “data subjects.” What lessons might we learn from the history and implementation of human-subjects research protections in order to better address these growing conceptual and structural discontinuities? How have other non-biomedical fields of science confronted the question of ethics through a critical lens?

Part of the difficulty here is that the precursor disciplines of data science—computer science, applied mathematics and statistics—have not historically considered themselves as conducting human-subjects research. Even though statistics do ultimately represent people, research into math, computational capacity and other numeric modes of analysis rarely exhibited the types of human subjects concerns that are baked into research ethics regulations designed to handle the types of harms found in biomedical research. Such regulatory definitions rest on a set of ethical and epistemic assumptions which are now under contestation due to Big Data methods.

For example, data analytics techniques rarely appear as a direct “intervention” in the life or body of an individual human being, which is one of the key requirements for research to be regulated in the USA (Department of Health and Human Services, 2009). The action of Big Data analytics happens mostly at a remove from the point of data collection, which is the most plausible analog for an “intervention.” Instead, it is focused on data sets that likely have a long lifespan and may be continuously updated and re-analyzed. Similarly, the Common Rule assumes that data which is already publicly available cannot cause any further harm to an individual. ⁴ Yet this fails to account for data analytics techniques that can create a composite picture of a person from disparate datasets that may be innocuous on their own but produce deeply personal insights when combined (Crawford and Schultz, 2014). The assumption (codified in law) that individual harm is the only type of risk researchers are required to track and mitigate undercuts the ability to see and account for harms that affect communities or produce “networked harms” (boyd et al., 2014).

Implicitly, the existing ethics regulations promote a historically situated understanding of “research subjectivity” that is clearly eroded by data science. The assumptions about what constitutes an intervention, when and how consent should occur and what types of harms are relevant, all add up to a picture of the human-research subject that is out of step with large-scale data practices. If the familiar human subject is largely invisible or irrelevant to data science, how are we to devise new ethical parameters? Who is the “data subject” in a large-scale data experiment, and what are they owed?

In this paper, we offer a preliminary examination of how critical data studies might generate a theory of data subjectivity that would enable responsible scientific practice with Big Data methods. We map the discontinuities between research regulations and data science, focusing in particular on human-subjects protections and the 30 year debate in the USA about the regulation of human-sciences research. We show that while the proposed revisions to the Common Rule are helpful in terms of making research ethics regulations more flexible and scalable to different research methods and types of risk, they problematically exclude data science wholesale in situations that still present serious risks. These exclusions are based on questionable assumptions about publicly available data, researcher–subject relationships and the very nature of “intervention” into the daily lives of those whose data is held within research databases.

Data science, social science and the complicated human subject

There are a variety of reasons why the predecessors of data science—applied mathematics, statistics and computer science—have had little contact with the infrastructures of ethics review. For the most part, the basic science conducted in these fields has had only distant contact with human data. Researchers represent themselves as dealing with systems and math, not people—human data is treated as a substrate for testing systems, not the object of interest in itself. The infrastructures of human-subjects protections have largely accepted this position, but where Institutional Review Boards (IRBs) have engaged with data science there appears to be mutual confusion. University-based IRBs are overwhelmingly oriented toward the methods common to biomedical and psychological experimentation in which interventions carry clear risks to individual subjects. Now that data science techniques profoundly affect human lives, the computational and mathematical disciplines are in urgent need of strong, adaptable ethical frameworks.

A robust approach to data ethics should interrogate how subjectivity is constructed in research datasets. Critical data studies have routinely demonstrated that it is deeply mistaken to treat research data as neutral and raw (see, for example, Bowker, 2005; Gitelman, 2013). Datasets and algorithms have historical, material specificity that is laden with political and ethical values. As data science moves toward interpreting and manipulating social structures and behaviors, often drawing on the interpretative tools of social science, these values become both more evident and more consequential. Hence, there is a need for more nuanced ethical research processes. We suggest that as computer science is being drawn into a closer orbit with social science we need to re-examine the rocky relationship between the social sciences (and to a lesser extent, the humanities) and research ethics infrastructures. In this closer conversation between the norms of social science research and the emerging practices of data science, there have been no clear conclusions about what counts as a human subject, and little research into what protections they might deserve.

Yet, it may be unnecessary to create an entirely new definition of what counts as a research subject in data science. Instead, we advocate for an approach to research subjectivity that is co-emergent with the conditions of research. From the earliest biomedical research ethics documents and policies, the question of how human-subjects get defined has been contested by scientists, physicians and ethicists (Annas, 1992). These debates revolve around norms of trust between researchers and subjects that run deeper than regulatory definitions. Situating critical data studies within the ongoing, dynamic debates about human subjects—rather than treating it as an entirely new field with unique problems—can remind data scientists and ethicists that we are engaging with a rapidly changing set of research dynamics that should be addressed in context, rather than solely through regulatory decisions.

Cases of research harms to data subjects

There have been several recent cases where de-identified data that was released publicly was able to be re-identified, or where data that was assumed to have no identifying features could be correlated with specific populations. For example, in 2013, the New York City Taxi & Limousine Commission released a dataset of 173 million individual cab rides, and it included the pickup and drop-off times, locations, fare and tip amounts. The taxi drivers’ medallion numbers were anonymized (hashed), but this was quickly de-anonymized—revealing sensitive information such as any driver’s annual income and enabling researchers to infer their home address (Franceschi-Bicchierai, 2015). A data scientist at Neustar Research showed that by combining this data set with other forms of public information like celebrity blogs you could track well-known actors, and predict likely home addresses of people who frequented strip clubs (Tockar, 2014). Another researcher demonstrated how the taxi dataset could be used to speculate which taxi drivers were devout Muslims by observing which drivers stopped at Muslim prayer times (Franceschi-Bicchierai, 2015). From one seemingly innocuous and anonymized data set came many unexpected and highly personal forms of information.

The taxi dataset is arguably a case of open data gone wrong—had the dataset been hashed properly it may have been much harder to de-anonymize. However, other research that makes use of publicly available private data from multiple databases has been used to make potentially risk-laden correlations. Danyllo et al. (2013) correlated a dataset from a financial institution with Twitter profiles from a geographic region of Brazil. They were able to produce a social network graph demonstrating that social and geographical relationships cluster around similar levels of credit access. The authors note that this research can be used by financial institutions to gauge credit worthiness based on one’s social relationships.

Of course, a case can be made that academic researchers should have access to public datasets in order to fully understand their potential and risk. Furthermore, we would caution against presuming that the worst case uses are inevitable with new forms of knowledge—the same research that could be used to discriminate against credit seekers could be used to track and ameliorate that discrimination. However, we find it concerning that we know so little about the data subjects in these studies and their expectations about how their private data is used in research. Should Twitter users now expect that their social media activities could affect their ability to get a loan? Is it reasonable to assume that social behavior on Twitter is the same as social relationship outside of Twitter, or is this a spurious correlations that might cause economic harm to particular individuals and communities? If human-subjects research regulations assume that public datasets are inherently harmless, it will be nearly impossible to review the material consequences to the affected data subjects.

These cases and the Banksy data tracking study (Hauge et al., 2016) remind us that datasets will often contain surprises, even when they are ostensibly public and anonymous. Beyond the issue of joining data sets, there is the question of the ethics of experimentation (Crawford, 2014; Grimmelmann, 2015b). The most public example to date was the public furor over the Facebook “emotional contagion” study in 2014. After using large-scale A/B testing to manipulate the emotional valence of the news feeds of nearly 700,000 users, Facebook shared the results with then-Cornell social scientist Jeff Hancock, who co-published the study in the Proceedings of the National Academy of Science (Kramer et al., 2013). Susan Fiske, who edited the article for PNAS, relayed in public statements that the Cornell IRB had approved Hancock's role in the study because the dataset was ‘pre-existing' as Facebook's data when he was first invited to participate in the analysis. His role in the study therefore did not technically rise to the standard of an ‘intervention' in a human life that qualifies a study as human-subjects research requiring further review, and the Cornell IRB therefore granted approval (Meyer, R., 2014). It does not appear that Facebook used any independent review process to approve the research that created the ‘pre-existing' dataset under question, nor would they be required to do so under the Common Rule as a private entity. Instead of quelling concern, this response ignited a broad debate about the ethics of such experiments (Auerbach, 2015; Crawford, 2014; Grimmelmann, 2015a; Meyer and Chabris, 2015; Waldman, 2014; Watts, 2014).

In an analysis of the Facebook emotional contagion controversy, Michelle Meyer argued that critics are mistaken if they examine only the antecedent (the “B”) of A/B testing and not the precedent (the “A”) (Meyer, 2015; see also Meyer and Chabris, 2015). In what she identifies as the “A/B illusion,” we have a tendency to focus on the ethics of changes resulting from an experiment and not the prior state. The “A/B illusion” illustrates what is essentially a variant of the naturalistic fallacy for the Big Data era: the way things are is the way that things should be and any change must be ethically interrogated. Meyer argues that it is equally important to ethically interrogate the precedent state as the antecedent state in order to avoid falling victim to this illusion. Yet she notes that the historically-situated codifications of research ethics are are calibrated to a different model of experimentation such that this logical parity remains largely invisible, and Big Data ethics debates tend to focus on the antecedent state alone. Thus, she links the future epistemic success of A/B testing, and its role in the Internet economy, to a regulatory environment that is not burdened by ill-fitting research ethics regulations that were historically designed to manage different scientific regimes.

While we largely agree that A/B testing is a loose fit for existing research ethics regulations, we argue that there are significantly different lessons to be drawn from those gaps. Meyer’s argument hinges on the limitations of the practice/research distinction—and on this point we agree. As we have discussed above, much of research ethics regulation can be viewed as managing the line between physician-as-caregiver and physician-as-researcher. Yet, there is no clear analogue of the practice/research distinction in data science because the practice of data science is iterative research. Indeed, it is problematic to import wholesale the ethical standards regulating that distinction. But we do not see this as reason to reject the application of research ethics regulations (or other responsive methods and enforceable standards) in data science. Rather, it is possible to read research ethics through a different framing that emphasizes ethics regulations as a form of community assent that enables self-regulation. Although the particularities of current research regulations cannot be directly ported to data science, the history of biomedical research ethics regulation indicates that the success of the field will depend on collectively assenting to transparent, enforceable norms of trust, responsibility and accountability. We see formal research ethics regulation as a route to that goal rather than a goal in itself. It is crucial that research ethics norms are established with a nuanced and empirically-informed assessment of the potential harms of data—public, semi-public and private—and a critical understanding of emerging forms of human-subjects research.

Conclusion

Large-scale data experimentation in academia and industry is playing a significant role in shaping both scientific endeavor and much of everyday life. From social media platforms to city streets, data is being gathered and used to conduct experiments on the public. And yet there is very little research on how to identify, track and mitigate the risk imposed on people who are (often unwittingly) participants in these experiments. The current debate, and the HHS revisions as they are currently framed, might lock in potentially risky forms of research as exempt from review, and maintain a problematic sense that Big Data research does not directly impact people’s lives.

Social scientists have a long and sometimes fraught relationship with the framing and reach of research ethics. As we have shown, this is due to the history of shaping ethics regulations around the epistemic conditions and particular scandals of biomedical research. Critical data studies should help articulate how new methods of knowledge production are co-constitutive with emergent ethical norms and modes of subjectivity (Jasanoff, 2004; Reardon, 2004; Thompson, 2013). Because the boundaries of human-subjects research are continually contested, it is crucial for new fields like data science to be attuned to the potential human impact of their work if they are to earn and maintain community trust. We argue that any move to exclude data science research from review, and more broadly, to consider it outside of human-subjects research, is thus premature and potentially dangerous. Rather, we propose that critical data studies contribute to a deeper understanding of data subjectivity, including an account of the fundamental responsibility that researchers have to care for the well-being of their subjects.

The changes proposed in the NPRM are claimed to be scaled toward empirical measurements of harm. But what is to be done with a field such as data science where practices for measuring and mitigating harms are still taking shape? What is “public” and “private” is not easily answerable by looking at the conditions of a database, but the proposed changes to the Common Rule appear to eliminate any formal point at which these questions could be asked. If adopted in a manner that does not allow for tracking the evolving risk profiles of data-intensive research, these new regulations could prematurely close off significant questions about data ethics. Both the NPRM and the National Academies report do recognize that risk profiles are rapidly changing with data-intensive research techniques, and suggest establishing an independent body capable of providing continuing advice to IRBs about how to measure and mitigate such risk (Committee on Revisions, 2014: 112–115). More accurate assessments of harms and risks are critical to ensure accurately and consistently assigning projects to the correct regulatory categories.

Finally, we should reject the belief that the risk borne by research subjects depends on what kind of data is obtained and how, rather than what is done with the data. In the context of data science, it simply does not hold. Instead, large-scale data practices begin with the assumption that new insights—some extremely sensitive—can be generated through connecting previously disparate data sets. Thus, the Common Rule needs to reflect that even anonymous, public data sets can produce harms depending on how they are used. The best way to do this in academic settings remains the IRB. As for industry, there needs to be a more serious commitment to review and assessment of human data projects. Facebook, for example, responded to the public outcry about the emotional contagion experiment by setting up an internal review process for future experiments. Legal scholar Ryan Calo has argued that a body like the Federal Trade Commission could commission an interdisciplinary report on data ethics, and that those public principles could guide companies as they form small internal committees that review company practices (Calo, 2013). Polonensky et al. (2015) have similarly argued for a two-track ethics review model for use outside of the purview of the Common Rule that would blend internal and external perspectives. Dove et al. (2016) recently surveyed how research ethics committees have grappled with data-intensive research with “bottom-up” approaches when more traditional “top-down” approaches have fallen short. Others have also offered promising insights for integrating ethical reasoning into data science research and practice prior to the typical timing of formal ethical review (Shilton and Sayles, 2016; Steinmann et al., 2015; Tractenberg et al., 2015). We think these are valuable approaches going forward, with an emphasis on bringing data science practices into frameworks of trust and accountability. Rather than seeking to exempt entire classes of new and emerging research, we should be establishing more flexible and informed structures of review, both within the academy and in industry.

This article is a part of Special theme on Critical Data Studies. To see a full list of all articles in this special theme, please click here: http://bds.sagepub.com/content/critical-data-studies.