Evaluating the prospects for university-based ethical governance in artificial intelligence and data-driven innovation

Human participants, data subjects, and ethical governance

The key mechanism for governance of research involving human participants within universities in the UK is the ethics committee (the equivalent body in the US would be the Institutional Review Board or IRB). A full review of the emergence of university-based ethical review and the responses to those developments is outside the scope of this paper (see Hedgecoe (2009)). Suffice to say that the current system has arisen in a somewhat piecemeal fashion and there is not full consensus that the forms of ethical review that we have arrived at meet the needs of all research communities: some question the applicability of practices derived from biomedical ethics to the social sciences (Schrag, 2011) while others are concerned about the bureaucratization of ethical review into a set of procedures enacted by committee (Haggerty, 2004; van den Hoonaard and Hamilton, 2016). Such concerns remain for the most part in the background for this paper, which begins from the current context within which ethical review committees have become embedded in UK universities as a means to meet expectations around ethical conduct and that as such they form an important feature in the landscape of research governance for any university-based researcher whether or not the researcher is warmly disposed to their activities.

As Edwards (2009, 2010a, 2010b, 2010c, 2011) describes in a series of papers on the role and remit of the ethics committee, ² the primary focus of review is on the human participants in the research process itself. The committee takes a protective view of these human participants, balanced against the future potential benefits of the research. The focus of review homes in most closely on the wellbeing of the human participants in the research and tends to narrow in on issues of informed consent and the limits of consent (Edwards, 2010c). Whilst the committee is required to balance the risks of the research against the potential gain to society (Edwards, 2010b), in practice understanding of potential gain is a somewhat speculative and uncertain business. Hence, the focus of ethical review can often come to concentrate on the research process and in particular the impact on the immediate participants in that process.

In its focus on the research process, much of the practice of university ethics committees is founded on the basis forged for biomedical ethics. In a biomedical context, while the same principles of beneficence, non-maleficence, autonomy, and justice are held to apply across research and everyday practice (Beauchamp and Childress, 2019), the monitoring of these principles happens in quite different ways between research and practice. In everyday medical practice the application of ethical principles is largely reliant on the judgment of healthcare professionals who are offering established treatments in the best interests of patients (Gillon, 1994). By contrast, medical research is subject to an additional level of ethical scrutiny through formal review considering whether it is appropriate to expose participants to the potential for additional levels of risk that research entails. Modeled on this scenario, university ethical review might be expected to examine data-driven research from the perspective of participants but not necessarily to have the capacity to look downstream at the products of such research when they find their way into implementation. In medical contexts, ethical review beyond research focuses on the suitability of new treatments, and there are calls for regulators, policymakers, and developers to be mindful of the need for thorough ethical assessment of AI (Morley and Floridi, 2020; Morley et al., 2020). Within healthcare there is a complex patchwork of responsibilities regarding ethical compliance for AI across clinical settings and direct-to-consumer products (Nebeker et al., 2019). There is as yet even less clarity on the processes of ethical review for the products of university research on data-driven technologies outside the healthcare domain as they might move into an implementation phase, whether through spin-off company or other forms of technology transfer.

A focus on direct participants may itself become challenging in data-driven research: Metcalf and Crawford (2016) note that it is difficult for a university ethics committee comprehensively to consider potential harms to humans from data-driven research, since harms may be diffuse, occur well downstream of the original research context and affect those who were not involved or indeed identifiable at all within the narrowly defined scope of the research project that would receive ethical review. A comprehensive review of data-driven research shifts away from the protection of immediately identifiable humans as participants to include humans who are rendered as data subjects and onto the review of algorithms, training datasets, arrangements for regulations and governance, and implementation of human oversight. There may also be a need to expand the definition of human participants, since the research may involve what appears to be simply secondary data or textual analysis but have downstream consequences for identifiable people.

The focus of ethical review on the immediate research process contrasts with the more recent movement to promote responsible research and innovation or RRI (Von Schomberg, 2013) that builds in a stronger focus on looking downstream to implementation and aims to enhance deliberation at the stage of determining research directions and priorities. RRI expands the perspective beyond the individual research protocol that tends to be the focus of ethical review and builds in an explicit orientation toward the future:

Approaches to responsible innovation extend the governance discussion to encompass questions of uncertainty (in its multiple forms), purposes, motivations, social and political constitutions, trajectories and directions of innovation. (Stilgoe et al., 2013: 1570)

Stilgoe et al. (2013) describe four key aspects of responsible research and innovation: anticipation; inclusion; reflexivity (particularly in relation to the limits of one’s own knowledge); and responsiveness. It would be difficult for a university-based system of ethical review meaningfully to develop such an evaluation on the researchers’ behalf without this becoming purely speculative. It seems more reasonable that ethics committees should expect researchers to come to review having already conducted a broad ranging evaluation of their proposed direction of travel, including the views of funders, user constituencies. and other stakeholders as proponents of RRI suggest and that the outcomes of this process should be presented to the committee. In the field of data-driven technologies it is important that such an evaluation should draw on the emerging principles of ethical AI and keep in touch with current debate on potential harms and benefits.

The university ethics committee currently provides a peer-based scrutiny of proposed research largely from the perspective of its impacts on the immediately identifiable participants in the research. This may indeed be important for research on data-driven technologies, where humans are providers of the data that feeds machine learning or where humans are the recipients of interventions occasioned by as yet untried predictive models derived from machine learning. People impacted in this way by data-driven research would quite reasonably expect that their involvement be underpinned by a degree of ethical scrutiny beyond the individual researchers and in proportion to the degree of risk involved and that they might reasonably be treated as active participants in the research rather than passive data subjects. However, taken alone this falls far short of the kind of process-based governance that Leslie (2019) recommends for ethical AI in the public sector that encompasses an array of requirements for stakeholder consultation, diversity impact, safety assessment, and transparency statement. This process-based approach also allows for issues to arise throughout the course of a project rather than confining attention to those issues that can be anticipated and addressed in advance as is often the case for university ethical review. A similar concept is expressed by Dove (2020) in proposals for ethics committees to see themselves as engaged in a regulatory stewardship that takes a deliberative approach to achieving ethical research with applicants and accepts a need for ongoing engagement throughout the research process rather than one-off review. ³ Such an ongoing approach to ethical assurance may be particularly important given the contingent and emergent nature of outcomes in machine learning. Unless we extend ethical review to include a wider array of issues downstream into the potential impacts of the products of the research, in line with Leslie’s (2019) recommendations for the public sector approach to AI and with the broader agenda of RRI, it is unlikely that university-based ethical review would be in a position to encompass some of the key principles outlined as key for ethical AI. Ultimate implications for beneficence and maleficence, for example, might well only be reasonably definable in relation to ultimate contexts of use beyond the confines of immediate research scenarios.

The remit of ethical review in universities

It might seem obvious that university ethics committees should confine themselves to matters strictly related to ethics, but behind this straightforward statement considerable controversy plays out concerning the appropriate object of scrutiny. University ethical review does not in practice cover all of the activities carried out by members of university staff. As outlined above, the focus on research ethics often narrows down the remit of scrutiny in practice to the time-bounded process of active research involving human participants. There are also limits that are mandated by the requirement for other institutions to conduct reviews, such as the requirement for review by NHS research ethics committees where NHS patients are involved. Here NHS ethical review takes precedence and obviates the need for university review. In addition to this narrowing of the remit of the university ethics committee, a number of other issues act either to limit or to extend the remit of the ethics committee within a university such that not all research comes under their purview and not all that they consider falls under a strict definition of research ethics. The nature of these limits and extensions poses some challenges for the capacity of university ethics committees to accommodate an effective ethical review of research on data-driven technologies. These challenges focus on three distinct issues: the focus of review on research rather than consultancy; the tendency of review to include issues beyond ethics; and the counter-productive effect on ethical thinking of an intensification of ethical review.

Possibly the most challenging exclusion from university ethical review in the case of data-driven technologies is the focus of review on academic research rather than consultancy. Given the extent of commercial involvement in university research in this field and the tendency for consultancy work to be explicitly excluded from formal processes of ethical review there is potential for significant ethical issues with technologies developed by university researchers to miss out on peer review. A high-profile example concerns the case of Cambridge Analytica, the company accused of making inappropriate use of Facebook users’ personal data in political campaigning. In written evidence to a Parliamentary committee inquiry on disinformation and fake news, Aleksander Kogan (a researcher at the University of Cambridge who played a key role in providing the dataset used by Cambridge Analytica) stated that “as with all of my research, the research that used the app was first ethics approved by the University” (Digital Culture Media and Sport Committee, 2019). A statement by University of Cambridge (2018) about the case developed a more nuanced position pointing out a distinction between academic research that was subject to review and various commercial associations that a researcher might develop in a personal capacity. The statement also highlighted that an application to the university for ethical approval to use data collected within the commercial context for academic research purposes had been rejected by them. The case makes apparent the complexity attendant on distinctions between academic research and commercial activity and the potential for this to operate against joined-up ethical review. Effective ethical review of research into data-driven technologies will require equivalent standards to be in operation for both academic and commercial research.

Ethics committees do not always confine themselves to ethics. Hedgecoe (2016) aligns the growth in university ethics committees with the extension of a particular form of managerial practice into universities and notes that over time ethical review becomes increasingly a tool for managing risks to institutional reputation rather than focusing on strictly ethical issues. In practice it is difficult for committee members with a commitment to their institutions to exclude from their ethical deliberations considerations on how a particular research topic might impact on the public standing of the university. Committees are thus often accused of being excessively risk averse for reasons beyond the strictly ethical. Such concerns might well pervade research into data-driven technologies where there is a potential to impinge on sensitive or politically volatile areas concerning automation and replacement of human labor or decision-making. Researchers in data-driven technologies might reasonably ask whether there are risks that their research might also come to be constrained by reputational rather than ethical concerns. Submitting for ethical review also has the tendency to expose researchers to further forms of scrutiny beyond ethics and reputational connotations. In recent years ethical review within many British universities has become embedded within a wider framing of research integrity and governance, following on from the publication of the Universities UK (2012) “concordat to support research integrity.” In many cases arrangements for ethical review will now be overseen by research integrity and governance offices that also take responsibility for oversight of compliance with other aspects of research regulations such as data protection, health and safety, and security-sensitivity (Universities UK, 2016). It is indeed important for research into data-driven technologies to submit to the requirements of research integrity and governance but the coupling of these forms of additional scrutiny with ethical review may present both practical and principled difficulties. In practice, the risk assessments and review of data protection raised by data-driven technologies are likely to be highly technical documents requiring specialist scrutiny. In principle, it can be important for ethics to be decoupled from other aspects of research integrity and governance in order for the ethical issues to be fully appreciated.

Against the tendency for important areas of work to be excluded from review by university ethics committees thus sits an opposing tendency for their remit to extend and for ethical concerns to become dominated by bureaucracy. As a bureaucratic system, university ethical review is prone both to extension into new areas and intensification: Haggerty (2004) identifies a problem of “ethics creep” as the mandate of ethics committees extends to encompass more activities and those activities deemed within scope are regulated more intensely. The attendant problem, Haggerty (2004) argues, is that ever more formalized ethical review may be counter-productive if it shifts researchers’ attention away from actually acting ethically and rule-based compliance takes over from more fundamental attention to doing the right thing. This argument was made in the specific context of ethical review extending into social science, but we might consider whether the arguments have any relevance for the situation of data-driven technologies. Arguably, the social sciences did have a more established culture of ethical reflection before the formalization of ethical review than is currently the case for data-driven technologies, although Jennings (2012) suggests that even in the social sciences there are significant shortfalls in the ability of researchers to self-reflect sufficiently on ethical concerns. In the case of research into data-driven technologies we may have an opportunity to build a culture of self-reflection on ethics and a system of ethical governance simultaneously and hence to take care that the latter does not over-ride the former. Concerns about bureaucratization are not, in themselves, reasons to resist the extension of ethical review by committee into data-driven technologies, but there is a case for thinking very carefully about the nature of the work to be reviewed and the boundaries between research and consultancy, ethics, and reputation that are enacted. There is on the one hand every possibility that the scope of university ethical review could be extended again to include a wider array of data-driven research, whilst on the other hand there is potential for such extension to work either for or against the reinforcement of a culture of ethical practice in the field, depending on how it is implemented.

Lay and expert input in ethical governance

The final set of issues to consider is what forms of expertise the ethics committee can offer to the ethical review of data-driven technologies. The concept of appropriate expertise is a complex one for ethics committees, since the review process is explicitly designed to include voices beyond the researcher who might be deemed the one with most expertise on the project itself. While ethics committees need training in the fundamentals of ethical review, they are not necessarily either subject specialists in the material that they review, nor are they professional ethicists. Their expertise is, rather, in the process of evaluation of the researcher’s self-assessment and the collective identification of the potential risks attendant on research directly involving humans (Edwards, 2011). Working within that model of expertise it is common for university ethics committees to include lay members from beyond the research community. The Economic and Social Research Council, for example, recommends inclusion on ethics committees of members who are users of health, education, or social services where these are the focus of the research (Economic and Social Research Council, n.d.). Lay input to ethical review offers the prospect of expanding the committee’s appreciation of risks, benefits, and harms (Edwards, 2009). Drawing on this array of voices aims at informing a balanced, inclusive assessment that does not favor the interests or perspective of the research community above those of wider society. Within this framing, lay members are being invoked as experts in their own condition and their ethical expertise is an everyday competence, albeit one that has to be balanced out by the committee through processes of deliberation (Edwards, 2011). Similarly, one of the key dimensions of responsible research and innovation is the broadening out of decision-making on new technologies, enabling a wider array of sectors of society to have a legitimate say on the direction of research and innovation (Owen et al., 2012).

Against this idealized model, there are, however, frequent concerns expressed about the extent to which ethics committees in practice actually display an appropriate degree of expertise such that they are able to reach sensible or consistent decisions (Schrag, 2011). Such complaints pose a direct challenge to ethics committees’ claims to hold sufficient expertise for adequate decision-making. There is therefore a live question on how far a committee should need to display domain-specific expertise, and how far the committee should handle areas where the research is highly technical and difficult for a non-specialist to understand. This may well be an issue for many areas of data-driven research where explicability is often in question. Whilst holding to the model of lay expertise as relevant and valuable, it may be important for ethics committees to draw on specialists to advise them in such areas. As Rawbone (2010) describes, where committees find themselves lacking in the expertise to identify risks and benefits they can draw on specialist advice and indeed specialist committees may convene to review research, for example at the level of an individual university faculty rather than the university as a whole, in order to balance the requirement for an inclusive community with the specialist expertise to interrogate the details of the research. Any system of ethical review for data-driven technologies in universities will need to consider the question of where appropriate expertise will be drawn from and the extent to which specialist advice or sub-committees may be required. It is important, however, that the recourse to specialists does not neglect the need for independent decision-making or preclude the inclusion of diverse lay voices as experts in their own condition.