Abstract
We examine compensation as a tool to encourage the thoughtful use of audit experimentation. If researchers had to compensate subjects or communities, they would be more likely to avoid excessively large studies and to only conduct high-value studies. We build on Desposato and propose that when conducting an audit study, (1) researchers donate time or money to the communities potentially affected by the study, (2) researchers preregister this donation, and (3) researchers provide evidence of the donation they made in publication and presentation.
Keywords
Introduction
Audit studies have been used for decades to study discrimination in housing and labor markets (see review in Gaddis (2017)) and have become popular in political science since the study conducted by Butler and Broockman (2011). At their most basic, audit studies initiate some interaction with the subjects being studied (a request for help, a job application, an application for rental housing, etc.) and then measure how the subjects respond. 1 Typically, subjects do not know that they are part of a research study or that they are interacting with a researcher or confederate.
Audit studies have become popular for at least two reasons. First, audit studies allow researchers to measure social phenomenon, especially unacceptable behaviors, that would be difficult or impossible to measure in other types of studies where subjects are fully informed and consenting. Subjects are unlikely to respond truthfully when asked directly about racism, sexism, discrimination, corruption, or any number of other attitudes or behaviors. In addition, if subjects know they are being observed by scholars, they are unlikely to behave in socially unacceptable ways. 2 But when subjects are unaware of the research and believe they are interacting naturally with others, such behaviors can be observed and identified. Because such behaviors may be quite harmful, it is important that researchers identify and find ways to address them. Indeed, some have argued that we are “Obligated to Deceive,” as fully informed and consenting research might severely bias results. 3
Second, audit studies are more frequent today because in the Internet age, they are very affordable. Pre-Internet audit studies often required hiring and training actors who would apply for jobs, seek housing, or request public assistance under assumed identities. These studies were labor-and capital-intensive. But with email correspondence and online applications for housing, jobs, and government benefits, it is almost costless for scholars to conduct an audit study with thousands of subjects. A large-scale study can be conducted by a scholar with no research funding, who just has the patience to enter email addresses into a computer.
In other disciplines, audit studies have used diverse subject pools that include members of the public, businesses, politicians, or other elite actors as subjects. 4 In political science, many audit studies focus on the behavior of elite political actors, including elected (and unelected) political leaders, powerful bureaucrats, and others in related institutions or businesses. Our discussion herein focuses on these cases of audit studies of elites, as studies involving private citizens involve a different set of issues.
The Objections
The features that make audit studies attractive to researchers also contribute to reasons that some observers object to them. First, subjects often do not want to participate in research, especially without their consent and knowledge (Desposato, 2018). In other types of experiments, subjects choose to participate after an informed consent process. Subjects often object to such studies because they resent the deception, because they feel their time has been wasted, and because they might be embarrassed by the results. This certainly seems to be the case for elites in audit studies—when they detect or learn of audit experiments, their reactions are rarely favorable. 5
However, concern about these harms to public officials may be offset by their unique positions of power and public responsibility. Many would argue that given the broad impact and power wielded by these officials as well as their choice to enter public life, some deceptive audit studies are often appropriate to monitor the performance and underperformance of public officials.
Second, audit studies may waste public resources. The public officials or bureaucrats in such studies are public employees, paid with public money, and with public responsibilities. Audit studies may be very valuable but they effectively leverage public time for research purposes. This may be a misappropriation of public resources. 6
Third, the combination of the first two objections might be harmful to the political science discipline. If too many of these studies are done, the public and politicians may take a hostile view of the discipline that affects their willingness to participate in interview, surveys, or even fund research. The low cost of conducting audit studies exacerbates these concerns by increasing the number of studies done; more studies make it more likely that politicians object to being studied in this way.
How should scholars balance the potential harms of audit studies with their benefits as tools to monitor and research the performance of powerful public actors? Scholars are highly skilled in self-deception regarding the benefits and value of their own research. In addition, trends suggest a premium to larger projects. Given the low cost of extremely large-scale audit studies and the career pressures of academia, self-monitoring and restraint might not be sufficient to avoid excessive use of public resources for low-value or marginal studies. In the next section, we consider compensation as a tool to help the discipline self-monitor and balance the harms and benefits of these types of studies.
Solutions
We suggest adapting the proposal in Desposato (2016) and applying it to audit studies. Desposato (2016) proposed that researchers should pay some amount for every subject included in a field experiment conducted without informed consent. Such compensation shows respect for the time of the individual subjects that has been taken up by researchers for their own purposes. In the case of field studies using members of the public, this compensation would presumably go directly to the subjects or their community. In the case of audit studies of public institutions or actors, it seems appropriate for the compensation to go to the community served by the public official.
Furthermore, the imposition of a small cost on an otherwise costless study will encourage scholars to conduct a power analysis and choose the study size appropriate for their question. Right now, the marginal cost of increasing an email audit study from 1000 subjects to 10,000 subjects is very small. Under our proposal, a 10,000-person study will cost 10 times as much as a 1000-person study. A modest cost will discourage unnecessary n-maximization and will likely lead scholars to focus research efforts on the highest impact and most important questions.
We propose the adoption of a compensation plan for audit studies, specifically (1) that researchers make a donation (of money or of their time) to charity for each audit study they conduct, (2) that researchers preregister who will receive the donation prior to the study being conducted, and (3) that researchers provide evidence of the donation after the study is completed.
Other Potential Solutions
Before discussing this plan, we first review a few other potential solutions. First, one option would be to simply ban audit studies, with journals refusing to publish them. Because audit studies are a major tool for studying discrimination in action (Butler and Crabtree, 2017), a complete ban would be an overreaction that would make it harder to understand and reduce discrimination. We think that this approach would be a bad idea.
Second, Butler and Crabtree (2021) call on researchers to use existing studies whenever possible to test hypotheses. Essentially, scholars should recycle data from previous experiments to test new hypotheses, whenever possible. While this is a reasonable proposal, it is not always possible because existing studies cannot be used to answer some of the relevant question; indeed, sometimes researchers want to measure the change over time which is only possible by conducting a new study. And even when it is possible, researchers may not wish to use this approach if journals are more likely to reward them for conducting their own independent experiments. As a result, this proposal is unlikely to change the number or character of audit studies being conducted.
Third, one could create a board, through a group such as the Political Studies Association (PSA) or the American Political Science Association (APSA), that must approve any audit study and would only approve some set number each year. One potential downside to this approach is that it gives a group of scholars the ability to block other studies. If there is a bias in that group, it will prevent scholars from potentially conducting important studies not valued because of those biases.
Fourth, one could ask the potential subjects or institutions of the audit study to agree to the concept of audit studies as part of a research partnership in the public interest. One could imagine that a public agency might agree to such a study in order to improve performance or service. This could be combined with the previous option where there is a limited number of such studies. However, although some individual politicians might agree to participate, it is likely that the worst performers (individuals and institutions) will refuse to participate, consistent with the findings of Findley and Nielson (2016). This selection will bias results and will prevent researchers from performing audits among the population(s) that may need to be studied most.
Fifth, one might use existing institutions to address excessive or harmful audit studies. The most obvious solutions include ethical review committees and journals. Regarding the first, most scholars conducting research on human subjects must have the approval of an ethical review committee in order to proceed. In the US, these committees are commonly known as Institutional Review Boards or IRBs. In Brazil, they are Committees of Research Ethics. 7 In the UK, they may be referred to as Research Ethics Committees. Ethical review committees like these may play a role in addressing the potential harms of audit studies, but we do not see them as a primary solution for several reasons. One reason is that many of these committees are highly bureaucratic, ensuring compliance with law or regulations. In most cases, these regulations (like the Common Rule in the US) focus on individual harms and do not address the broader costs to the public that audit studies may cause. Another reason is that these committees may lack the expertise and awareness of the ethical issues involved in political science research, especially since they frequently comprised volunteer faculty from other fields or by community members. Finally, in some countries, there simply are no social science ethical review committees. In addition, we believe it critical that scholars not outsource ethical judgments to an external body, instead preferring that individual scholars be thoughtful and transparent in their ethical judgments and choices.
An alternative institution that might play such a role is the review process, shaped by journal policies, editors, and reviewers. Here, we see more potential for addressing ethical issues. Editors and journals can require that authors address ethical issues in their work, they can reject unethical research, and they can publish letters of concern when a publication is judged to have crossed the line. But we do not see the ethics of audit studies in simple black and white and we are hesitant to outsource control exclusively to editors and reviewers. Norms in this area of research are evolving and will require time to develop, and for the time being, we prefer our approach which emphasizes individual scholars’ responsibility, transparency of practices, and respect for communities affected by research.
The Outlines of a Proposal
Desposato (2016) proposed that researchers conducting field experiments on people without their consent should pay those participants for their time. We outline how that suggestion can be adapted for audit studies.
First, researchers should make a donation to a charity that benefits the population being studied. Many audit studies in political science are done on elected officials. Because paying elected officials directly is often not allowed, researchers have used donations to charity instead to thank elected officials for their participation in research (Butler and Pereira, 2018). Because the audit study is taking up the officials’ time that they could have spent on working for their constituents, the donations should benefit the people served by the officials studied whenever possible. For example, if an audit study in being conducted on high school principals (Pfaff et al., 2011), the donation might be directed toward an organization serving high school students.
Second, the amount donated should reflect how much time the officials are putting into responding to the request. The actual amount will depend on the context but should reflect the time that the officials put into responding. A simple place to start might be to tie the donation amount to an hourly wage of $20, with the minimum amount paid set at $1 per individual in the sample. So, if the reading and responding to an email is likely to take 15 minutes, then the researcher should pay $5 for each official included in the sample. Other compensation schemes are also possible. One might pay the actual compensation cost of the subjects’ time, including wages and benefits. One might also pay an amount equal to the researcher’s compensation, or the greater of subject and researcher compensation. While this is an imperfect solution, we think that a form of compensation would reduce the likelihood that researchers in the discipline abuse the public trust.
Third, researchers should include their donation plan in the pre-analysis plan that they register. The scholars planning on conducting an audit study will need to identify the charity they will support and justify the calculation they used for the donation rate per subject. And after completing the study, researchers should provide proof of payment at the place they have deposited their pre-analysis plan. Donations should be made for all studies conducted, not just those that are published.
We propose these steps in the hope that they become a norm. If there is little uptake among the researchers, further steps could be taken to incentivize compliance. Specifically, journals could incentivize compliance by requiring that an audit study follow the steps we have proposed in order to be considered for publication. And registries could further incentivize compliance by only registering a new audit study if the researcher has provided proof of donation from any prior audit studies they have done.
Potential Problems
Would This Proposal Favor the Haves Too Much?
Any form of regulation will create winners and losers and potentially exacerbate inequality in research opportunities. One of the benefits of audit studies is that they are relatively inexpensive. Increasing the costs of audit studies might prevent scholars with less access to research resources from conducting these studies. Indeed, it could exacerbate inequality in research and could prevent scholars from poorer institutions from conducting important and high-quality research.
One solution would be for scholars to donate time instead of money. A scholar might volunteer at a community organization in order to compensate the public for time lost. The choice of organization would be highly contingent on context and the nature of the intervention. This approach does not remedy all inequality in research, but creates a mechanism for scholars at any university or college to conduct audit studies without a need for funding.
We should also consider the challenges of scholars and subjects located in countries with very different levels of compensation. We do not wish this proposal to encourage or reward research imperialism, with scholars in the developed world conducting inexpensive audit studies on elites in the developing world, nor to punish or deter research by scholars in the developing world on elites in the developed world. We can imagine a number of approaches to address these types of inequalities. For scholars in the developing world, one approach might be for scholars to donate the lesser of their compensation and that of the subjects. Another mechanism might be for scholars in developing countries to donate time to a local charity in their country, if their subjects are located somewhere in the developed world. 8
Indulgences
One risk of our proposal is that it might, if taken in the extreme, allow research-fund wealthy scholars to buy their way out of ethical considerations. In this scenario, scholars from well-endowed universities or with generous financial support might justify otherwise unethical studies via compliance with our proposal. To be clear, payments to subjects or subject communities do not provide absolution of ethical concerns—it is not the modern researcher’s equivalent of an indulgence. Instead, this is a starting point for ethical audit studies and a mechanism to deter excessive use of public time. Payment of compensation in no way frees scholars from the work of broadly considering harms and benefits of any particular study.
Discussion
We see additional issues that should be addressed in future work. First is whether compensation is required when studying business or non-governmental elites. In these cases, time or other resources are taken from the shareholders or other owners, or from the nonprofit or political party. A second issue is that some research that is clearly in the public interest might justify a smaller compensation schema. In cases where the public strongly supports the research. Suppose half of the public supports the research. If so, could compensation be reduced by half? In a survey conducted by Desposato (2018), more respondents looked favorably on an audit study investigating discrimination by legislators than on an audit study on communication. Perhaps the latter study should pay slightly more and the former slightly less. Any effort to evaluate a given audit study in this way should learn about the public’s view of the value of that study. 9 Researchers should turn to the public both because scholars may not objectively evaluate the public value of their own work and because researchers should consider the public’s will on how public goods (including the time of public servants) are spent.
Audit studies of elites can make valuable contributions to political science, but their low cost and large scale can lead to excessive experimentation with potential public and disciplinary harms. This proposal provides an actionable outline for getting researchers to have a greater investment in any audit studies they pursue. We encourage scholars to consider the costs of their research on subjects, the public, and the broader discipline.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
