Fairness perceptions of algorithmic decision-making: A systematic review of the empirical literature

Establishing the research question

A precise research question is vital for a systematic literature review (Booth et al., 2016). We applied the “PICOC” method (Petticrew and Roberts, 2006) to break down the research question into five components: Population, Intervention, Comparison, Outcome, and Context. This approach helps identify possible search terms for subsequent searches in databases (Booth et al., 2016). First, the population includes all individuals, irrespective of sociodemographic characteristics. This includes the general public, but also AI experts and decision-makers such as public officials or managers. Second, the intervention is defined as fairness in and through algorithms. Third, we did not specify a comparison, as it was not considered beneficial to include additional interventions. Fourth, the outcome involves individual fairness perceptions about ADM. Fifth, the context—conceived here as a country-specific and domain-specific setting—was not narrowed down, thereby making the literature review global in scope. Using these criteria, we derived the following research question: How do individuals perceive the fairness of algorithmic decision-making?

Inclusion criteria

The keywords in the research question (“fairness,” “algorithmic”) provide the basis for the search terms. We adopted the “pearl-growing” method for additional terms for the subsequent search (Booth et al., 2016). It draws on relevant articles (“pearls”) to identify further relevant search terms or keywords. For this study, we initially identified three articles as “pearls” because they are widely cited in the literature: Binns et al. (2018), Grgić-Hlača et al. (2018a), and Lee (2018). As all these articles correspond with our research interest, we added relevant keywords to the search terms we derived from our research question: “justice,” “discrimination,” “machine learning.” Subsequently, we clustered the search terms into two components: terms referring to ADM, and terms referring to the theoretical concept of fairness.

We inputted these search terms into the following Boolean operators (Lefebvre et al., 2008): (“big data” OR “artificial intelligence” OR “machine learning” OR “algorithm*”) AND (“fair*” OR “unfair*” OR “just*” OR “discrimina*” OR “bias” OR “disparate”).

If the predefined combination of search strings appeared in the title of a publication, we included that publication in the preliminary sample, which we then used for the first screening process. Due to the recent surge of literature on algorithmic fairness, we adopted the reasoning suggested by Favaretto et al. (2019) and only included studies written in English and published since January 2010.

Comprehensive literature search

We selected the final sample of empirical studies through a stepwise process (see PRISMA chart, Figure 1). As fairness of ADM has been investigated in different research disciplines, the selection of the databases was based on thematic classification. We applied the Boolean logic to the following electronic databases: Web of Science, PsycINFO, IEEE Xplore, and Scopus. Our study also includes “gray literature”—working papers, pre-prints, or reports—as they account for recent research efforts. We used Google Scholar to manually search for relevant publications by searching for publications by key institutions (e.g. AI NOW, AlgorithmWatch).

OPEN IN VIEWER

Figure 1.

Flowchart documenting the selection process adopted by Moher et al. (2009).

Applying the search terms to the selected databases, we identified a total of 5117 contributions (24 February 2022), of which 3097 remained after filtering for duplicates. In a subsequent research step, we identified 101 potentially relevant articles by manually searching gray literature. We extracted the title, authors, keywords, journal information, and abstract for all publications.

Screening & critical evaluation

Two expert raters examined the 3198 publications based on their titles and abstracts for the initial screening. The screening was twofold. In the first step, for a publication to be considered for further analysis, both raters had to agree that it generally addressed the research interest. In this step, 389 publications were selected. In the second step, all publications were rated in terms of their applicability to the research question of the review and the use of an empirical method. All publications that were rated 1 (applies) on both criteria were selected for the literature review. This filtering process resulted in a sample consisting of 46 publications. Finally, we used Google Scholar to scan all publications that cited at least one of the “pearls” (Binns et al., 2018; Grgić-Hlača et al., 2018a; Lee, 2018), identifying twelve other relevant publications that we included in the final sample.

Two raters assessed the reliability of the selection process with a sample of 72 abstracts in the first test (Cohen's κ = .55, “fair agreement”) and 39 abstracts in the second test (Cohen's κ = .74, “good agreement”) (Higgins and Deeks, 2008).

Descriptive results

The descriptive results (shown in Table 1 in the Supplemental Material) indicate high homogeneity in terms of the national context. Data on citizens' perceptions of algorithms' fairness was almost exclusively collected in Western democracies: 32 studies were conducted in the United States (US), three in China, three in Germany, two in the United Kingdom (UK), two in the Netherlands, one in South Korea, and one in Cyprus, with four studies collecting data in multiple countries. ¹

Looking more closely at the empirical methods used to investigate citizens' perceptions of algorithmic fairness, we find great diversity. Eight studies used a qualitative design, 37 studies used quantitative methods, and 13 studies combined qualitative and quantitative approaches in mixed-method designs.

With regard to the different domains within which the studies were located, the descriptive results reveal a focus on work-related decisions (16 studies), especially in hiring. Moreover, emphasis is also given to the criminal justice system as an area of application (13 studies)—most prominently pretrial risk assessment. Other domains include news recommendations, allocation of donations, university admissions, loan decisions, and targeted advertisements.

Concepts of (algorithmic) fairness

Despite the growing interest in algorithmic fairness “there is no consensus on a precise definition of (un)fairness” (Srivastava et al., 2019: 1). We find much heterogeneity in relation to algorithmic fairness notions (see Table 1 in the Supplemental Material). Twenty-three studies focus on the perceived fairness of decision outcomes—that is, distributive fairness, although many computer science studies do not use this term explicitly. Thirteen of them distinguished between different formal definitions of algorithmic fairness (e.g. demographic parity, equalized odds) (Saxena et al., 2020; Srivastava et al., 2019) or among equality, equity, and need (Schlicker et al., 2021). Thus, they conceived of algorithmic fairness in distribution norms. In this reasoning, fairness can be achieved by decreasing “discriminatory consequences for certain groups of individuals” (Dodge et al., 2019: 275).

Twenty-two studies went beyond distributive fairness and investigated the fairness of algorithmic processes. However, conceptual ambiguity exists. While some studies defined procedural fairness more narrowly as the inclusion of selected (sensitive) input features, others conceptualized it in broader terms, also addressing such criteria as the consistency or the revocability of decisions (Hsu et al., 2021; Schlicker et al., 2021). Thus, procedural fairness can be increased even if the decision outcome is biased by using appropriate input features and ensuring people can appeal an ADM decision. We further find that five studies also included interactional fairness, and six studies investigated informational fairness (Acikgoz et al., 2020; Binns et al., 2018; Schoeffer et al., 2021). These studies zoom into the decision process and emphasize its social aspects, such as treating individuals with respect and receiving an explanation for an ADM decision (Schlicker et al., 2021). In this reading, fairness can be fostered by endowing ADM systems with fundamental aspects of social interactions indicating that algorithms should adhere to the same social norms as humans.

Measurements of algorithmic fairness

Most quantitative studies used a two-step process to measure fairness perceptions of ADM: respondents were confronted with an ADM process or outcome, and then they were asked about their perceived fairness. However, the results show much diversity in the measurement of algorithmic fairness. Fourteen studies simply used single items measured on 5-point or 7-point Likert scales, along the lines of “how fair did you perceive the decision?.” A total of 17 studies drew from existing fairness scales (e.g. Colquitt and Rodell, 2015) and adapted them for ADM. Five studies indirectly gauged fairness via stated preferences for one ADM over another. Ahnert et al. (2021) summarized the different measurement steps in their FairCeptron framework.

However, other measurements also exist. Pierson (2017) predefined a fair distribution of resources and asked participants’ approval on a 7-point Likert scale. Other studies used a conjoint design to investigate how algorithmic fairness hinges on what, who, and how resources are allocated (Hannan et al., 2021) and the relative importance of fairness in relation to other attributes such as transparency or accountability (Kieslich et al., 2022).

Algorithmic predictors of perceived algorithmic fairness

A significant strand of literature investigates how an ADM system's technical design affects people's perceptions of fairness. Overall, fairness is a crucial factor when evaluating algorithms (Bankins et al., 2022). Studies that tapped into people's general notions of algorithmic fairness (Dodge et al., 2019; Shin and Park, 2019) yielded mixed results: some respondents perceived “the very idea of an algorithmic system making an important decision on the basis of past data […] unfair” (Binns et al., 2018: 9), while other participants argued that algorithms are by definition impartial (Lee and Rich, 2021).

On a general note, Zhou et al. (2021) showed that a decrease in the de facto fairness of ADM led to a decline in respondents' perceived fairness. However, fairness perceptions are highly context-dependent. Some studies revealed that algorithmic fairness is perceived as more problematic in some domains than in others (Hannan et al., 2021). For instance, discrimination by ADM in housing or job recommendations is viewed as more harmful than in music or movie recommendations (e.g. Spotify, Netflix) (Smith et al., 2020). Also, less complex algorithmic tasks elicited higher fairness perceptions than more complex ones (Hsu et al., 2021).

A large group of studies goes beyond people's basic understanding of algorithmic fairness and investigates how people's perceptions of fairness are related to different outcome distributions. This is particularly intriguing as some formal fairness definitions cannot coexist (Kleinberg et al., 2017). Lee and colleagues (Lee et al., 2017; Lee et al., 2019b) tested people's fairness perceptions regarding the allocation of resources based on equality, equity, or efficiency. They found much variation in the preferences for the three fairness concepts, both within and across different social groups differently impacted by the decision. Most respondents considered an outcome fair when it mirrored their input (equity). Yet, some respondents also believed that an equal allocation of tasks or resources was fair, emphasizing moral norms such as self-sacrifice (equality) (Lee and Baykal, 2017).

While these qualitative studies focused on basic fairness concepts, other quantitative studies tested more nuanced notions of fairness. Srivastava et al. (2019) used criminal risk and skin cancer as examples and matched people's fairness choices with six different notions of group fairness. Their results showed that demographic parity best matched the fairness choices most respondents made in both scenarios. Thus, people favored algorithms aiming to equalize the positive rate across different groups. For instance, if ten percent of all applicants to a university get admitted, this rate should be equal for all gender groups. While the authors further found that in high-stakes situations, respondents weighed accuracy higher and inequality lower (Srivastava et al., 2019), a qualitative study by Koene et al. (2017) revealed that participants deemed ethical considerations more important than higher accuracy. Three other studies shed light on the relationship between fairness and accuracy: Kieslich et al. (2022) showed that fairness and accuracy were similarly important to respondents, Cheng et al. (2021) found that participants rather accepted disparities in accuracy across groups than give up overall accuracy, and Hsu et al. (2021) highlighted that accurate ADM is also perceived as fairer than inaccurate ADM.

Adding to this, two studies (Kasinidou et al., 2021a; Saxena et al., 2020) investigated a different set of formal fairness definitions in the context of loan decisions: equal distribution (“money is split equally among candidates”), meritocratic distribution (“all the money is distributed to the candidate with the highest payback rate”), and calibrated/proportional distribution (“money is split proportionally to candidates payback rates”). The results indicated that people perceived the calibrated model to be the fairest. Then again, Cheng et al. (2021) compared three group fairness approaches in a child maltreatment predictive system and found that respondents most supported equalized odds, followed by statistical parity and unawareness.

Another study shed light on the trade-offs between different incompatible fairness definitions in the criminal justice context. The results indicated that respondents favored an algorithm that equalizes the false positive rate between groups over one that equalizes accuracy (Harrison et al., 2020).

However, the technical design of an algorithm refers not only to the decision outcome but also to the decision process. Six studies investigated the perceived fairness of input features. Grgić-Hlača et al. (2018b) used predictive policing as a case study. They showed that respondents perceived feature-accuracy fairness (“a feature is perceived as fair if it increases the accuracy of an algorithm”) to be the fairest process, followed by feature-a priori fairness (“a feature is perceived as fair, independent of its effect on the outcome”) and then feature-disparity fairness (“a feature is perceived as fair even if it increases disparity in the outcomes of an algorithm”).

Two other studies tested eight feature properties that determine if people perceive the use of said feature in an ADM system to be fair: while Grgić-Hlača et al. (2018a) indicated that relevance, causes outcome, and reliability are most important for respondents, Albach and Wright (2021) found that relevance and increases accuracy are the essential features when deciding whether it is fair to use a feature in an ADM system.

Other studies shed further light on procedural fairness showing that respondents perceived features that directly relate to the issue at hand to be fairest and perceived unrelated features to be the most unfair (Grgić-Hlača et al., 2018b; Plane et al., 2017; van Berkel et al., 2019). Nyarko et al. (2021) qualified this finding by discovering that respondents were generally opposed to including sensitive features, like race and gender, in an ADM system. However, after respondents were told that including these features in the model can lead to better outcomes for minority groups, support for such “non-blind algorithms” increased substantially. In one of the few studies conducted in a non-Western context, Sambasivan et al. (2021) further found that missing data and misrepresentation of subgroups in the existing data were critical reasons for algorithmic unfairness perceived. Moreover, respondents' fairness perceptions hinged upon different design factors of the algorithmic process, such as the possibility to appeal a decision made by an algorithmic system (Hsu et al., 2021) and people's control to avoid algorithmic discrimination (Sun and Tang, 2021).

Another set of studies looked at explanations for a decision as a critical aspect of perceived fairness. Explanations for an algorithmic decision significantly increased respondents' perceptions of fairness in several studies (Binns et al., 2018; Dodge et al., 2019; Shin, 2021; Shulner-Tal et al., 2022). However, the results are very nuanced. Schlicker et al. (2021) found that while explanations for ADM had no impact on interpersonal and distributive fairness, they influenced perceptions of informational fairness. Lee et al. (2019a) added that the direction of the effects largely depended on the context. When explanations helped respondents understand biased distributions, perceived fairness decreased, yet, when explanations helped respondents understand equality in utility distribution, perceived fairness increased. Moreover, several studies showed that different explanation styles (e.g. case-based, sensitivity-based, demographic-based, input influence-based) affected respondents' perceived fairness differently (Binns et al., 2018; Dodge et al., 2019; Schoeffer et al., 2021; Shulner-Tal et al., 2022).

Looking at transparency more generally, Perez Vallejos et al. (2017) found that young people demanded more information about an algorithm to perceive it as fair. However, empirical, experimental evidence offers inconclusive results. While Wang (2018) found that algorithmic transparency increased perceptions of fairness, Wang et al. (2020) observed that different degrees of transparency had no significant effect on algorithmic fairness. Van Berkel et al. (2021) focused on visualizations as a specific form of transparency and showed that scatterplots led to lower perceived fairness levels than text.

OPEN IN VIEWER

Figure 2.

Summary of the results in a process model.

Human predictors of perceived algorithmic fairness

Another strand of empirical studies investigates human predictors of perceived algorithmic fairness. Only a few studies found an impact of sociodemographic variables. For instance, three studies suggested a significant influence of gender: female respondents opposed gender as an input feature more strongly than male respondents (Grgić-Hlača et al., 2020; Pierson, 2017) and overall showed a lower level of perceived algorithmic fairness (van Berkel et al., 2021). Helberger et al. (2020) found that education and age affected both perceptions of algorithmic fairness. In terms of age, young people perceived the inclusion of a sensitive feature in ADM systems as unfair (Grgić-Hlača et al., 2020) and demanded global approaches to regulating fairness for algorithms (Perez Vallejos et al., 2017). Moreover, higher educated respondents showed higher levels of perceived algorithmic fairness (van Berkel et al., 2021) and deemed fairness an important feature when evaluating algorithms (Kieslich et al., 2022). Other studies found that algorithmic fairness perceptions can vary between ethnic groups (Albach and Wright, 2021; Lee and Rich, 2021).

Five studies found that perceived algorithmic fairness hinged on self-interest, indicating that people tend to perceive algorithms as fairer when the ADM yields a positive outcome for them (Bankins et al., 2022; Grgić-Hlača et al., 2020; Noble et al., 2021; Shulner-Tal et al., 2022; Wang et al., 2020). Along similar lines, respondents perceived decision outcomes as fairer when their social group benefitted from them, suggesting in-group favoritism (Hannan et al., 2021). In addition, Lee et al. (2019a) presented evidence that the more significant the gap between prediction and the actual outcome, the more unfair an algorithm was perceived.

Another aspect that received considerable attention is people's familiarity with data and algorithms. While participating in a workshop about ethical AI raised awareness about algorithmic fairness (Kasinidou et al., 2021b), understanding the mathematical definition of the fairness concept led respondents to reject the fairness concept (Saha et al., 2020). While some other studies indicated that higher levels of AI literacy were associated with lower levels of perceived algorithmic fairness (Lee and Baykal, 2017; Schoeffer et al., 2021), Araujo et al. (2020) found that the effects ran in the opposite direction.

Two other studies examined political ideology as a predictor of algorithmic fairness perceptions. Grgić-Hlača and colleagues (Grgić-Hlača et al., 2020; Grgić-Hlača et al., 2018a) found that conservative users perceived the inclusion of sensitive features such as gender and race in an ADM system as fairer than liberal users did.

Comparative effects (HDM vs. ADM)

Our literature review found 24 studies that examined whether decisions made by humans or algorithms are perceived to be fairer (see Figure 2), with ambiguous results. After being asked, “Who would, according to you, make a fairer decision: a human or artificial intelligence/computer?,” 54% of respondents answered that they believed AI makes fairer decisions (compared to 33% for humans) (Helberger et al., 2020). Schoeffer et al. (2021) investigated open-ended questions and also found that respondents perceived ADM as less biased than HDM. Other studies support this general finding in university admission decisions (Marcinkowski et al., 2020) and algorithmic work assignments (Bai et al., 2020).

However, several studies provided different evidence that suggests that humans are viewed as fairer compared to algorithms, in criminal justice (Harrison et al., 2020; Wang, 2018), healthcare (Lee and Rich, 2021), work-related decisions (Acikgoz et al., 2020; Bankins et al., 2022; Newman et al., 2020; Noble et al., 2021), sport referee decisions (Wonseok et al., 2021), and social division tasks (Lee and Baykal, 2017). The main reason for this finding is that respondents believed that algorithms do not consider qualitative information or context. However, Wang (2018) found that respondents were more willing to accept discriminatory outcomes when the outcomes were attributed to algorithms rather than humans. Other studies found evidence for both positions, as HDM and ADM had different effects on different proxies of procedural fairness: while algorithms were rated higher in terms of consistency, human agents were rated higher in terms of personableness (Kaibel et al., 2019; Schlicker et al., 2021). Then again, other studies found no significant differences in perceived fairness between HDM and ADM (Plane et al., 2017; Suen et al., 2019).

The inconsistency of the empirical evidence suggests that fairness perceptions of HDM versus ADM are highly context-dependent. Consequently, several studies looked at conditional effects or distinguished between different kinds of decisions. An experiment on ADM versus HDM in policing suggested that procedural fairness perceptions hinge on social group representation (Miller and Keiser, 2021). Black respondents rated ADM as fairer—but only when they felt their social group lacked representation in the HDM condition.

Lee (2018) compared different decisions and found that ADM and HDM were perceived as equally fair for tasks requiring mechanical skills. However, respondents perceived HDM as fairer than ADM for tasks requiring human skills. Araujo et al. (2020) investigated high- and low-impact decisions. Overall, the findings suggested no significant differences between ADM and HDM in perceived fairness; however, ADM was perceived as fairer than HDM in high-impact decisions in the health and justice sectors. In contrast to this finding, Nagtegaal (2021) distinguished between high- and low-complexity decisions and found that HDM was perceived as fairer than ADM in high-complexity tasks. However, for low-complexity tasks, respondents viewed algorithms as fairer.

Another strand of literature went beyond the binary distinction between HDM versus ADM and included hybrid decision-making forms. For instance, Nagtegaal (2021) found that decision-making systems involving algorithms and humans are perceived as the fairest in high-complexity situations. However, Newman et al. (2020) suggested that while algorithmic decisions with human oversight increased fairness perceptions, it was still outranked by pure HDM. Lee et al. (2019a) showed that allowing respondents to adjust the algorithmic allocation of resources and thereby overrule decisions made by the ADM system increased people's perceptions of algorithmic fairness. Other studies varied the degree of human involvement in the decision-making process: While Wang et al. (2020) found no significant differences across the experimental conditions, de Cremer and McGuire (2022) suggested that respondents even incurred financial costs to avoid the algorithm led to the decision because they perceived it to be unfair.

Consequences of perceived fairness

Only eleven studies investigated the implications of the perceived fairness of algorithms. Shin and Park (2019) found that perceived fairness has a significant positive impact on satisfaction with algorithms. Several studies focused on the relationship between perceived algorithmic fairness and trust. While Kasinidou et al. (2021a) found no correlation between the two variables, other studies (Shin, 2020, 2021; Shin et al., 2020; Sun and Tang, 2021) suggested that fairness perception had a positive effect on trust in an algorithm. Woodruff et al. (2018) asked respondents belonging to traditionally marginalized groups about possible consequences of perceived algorithmic unfairness. The results indicated that “algorithmic fairness (or lack thereof) could substantially affect their trust in a company or product” (p. 1). The authors concluded that perceptions of fairness play an essential role in adopting algorithms.

Concerning ADM in human resources decisions, empirical evidence suggests that perceptions of procedural and interactional algorithmic unfairness were associated with lower organizational attraction or commitment and lower job pursuit intention (Acikgoz et al., 2020; Newman et al., 2020). Furthermore, low levels of perceived interactional fairness increased the likelihood of pursuing litigation against a company using ADM systems (Acikgoz et al., 2020). In a field experiment, Bai et al. (2020) found higher perceived fairness was associated with higher productivity among workers in a warehouse.

Marcinkowski et al. (2020) tested how fairness perceptions influence students' intentions to protest, students' willingness to exit, and the institution's reputation if ADM was used for university admissions. The results yield three main insights. First, distributive and procedural fairness perceptions negatively influenced students' intention to protest against an ADM system. Second, perceptions of procedural fairness negatively affected students' likelihood of exiting the university. Third, perceptions of distributive fairness had a positive effect on the university's reputation.

We summarize four main insights: First, preferences for different distribution norms are highly context-dependent and can vary substantially across domains, tasks, and algorithmic designs. However, respondents favored more straightforward fairness definitions over more complex ones. The literature further yielded tentative evidence that explanations can increase perceived fairness. Second, while studies on human predictors delivered inconclusive results regarding sociodemographic variables, they also indicated that political ideology and self-interest influence citizens' fairness perceptions of ADM. Third, studies comparing fairness perceptions of HDM versus ADM revealed ambiguous results: fairness perceptions are highly context-sensitive, making generalizations about the perceived fairness of HDM versus ADM infeasible. Fourth, while little empirical research examined the consequences of perceived algorithmic (un)fairness, initial empirical insights suggested adverse effects for institutions using an ADM system if that system is perceived to be unfair, especially to its reputation.

Theoretical groundwork

A key takeaway of this review is that the perceived fairness of ADM systems is highly context-dependent. Fairness perceptions are determined not only by the technical design of the algorithm but also by the area of application (e.g. pretrial risk assessment, hiring) and the specific task at hand (e.g. high-stakes vs. low-stakes). However, we attribute some of the inconclusivenesses of the empirical results to the lack of coherent theoretical frameworks for perceived algorithmic fairness.

Three aspects stand out. First, to avoid conceptual confusion, researchers should clearly state the fairness dimension (e.g. distributive, procedural) they investigate and discuss their results through the respective lens. Second, existing theoretical fairness approaches are not used consistently in the literature. For instance, while some authors define procedural fairness as input features (e.g. Grgić-Hlača et al., 2018b), others conceive of it in broader terms that also include consistency and revocability (Marcinkowski et al., 2020). Third, most studies rely on fairness concepts that have been developed for decisions made by humans (Greenberg, 1990). However, empirical evidence suggests that people base their evaluations of ADM on different factors than those they use to evaluate HDM (Dietvorst et al., 2015). It is likely that people also include other criteria for assessing the fairness of ADM and HDM. Following these arguments, we echo the call voiced by other authors (Lepri et al., 2018; Wong, 2020) for more theoretical groundwork on human perceptions of algorithmic fairness. We argue that a fruitful avenue for future research lies in developing a coherent, multi-dimensional theoretical concept of perceived algorithmic fairness. For instance, such a framework could encompass the four main fairness dimensions distributive, procedural, interactional, and informational fairness, and scrutinize the specific differences between human–human and human–machine interactions.

Diversification versus harmonization of research

We argue that a need exists for diversification and harmonization of empirical research on algorithmic fairness perceptions. First, while the descriptive results show a variety of research methods, they also reveal that the studies included in this review were almost exclusively conducted in Western democracies, predominantly the US. The problematic aspects of generalizing so-called WEIRD (White, Educated, Industrialized, Rich, Democratic) samples are widely acknowledged (Henrich et al., 2010). The seminal work of Henrich et al. (2010) showed that fairness in decision-making is considerably dependent on the sociocultural context. The same likely applies to fairness perceptions of algorithms as cross-national variance exists globally in citizens' perceptions of AI (Kelley et al., 2021). Diversification of countries under investigation—especially those in which ADM systems are already widely implemented, such as China and South Korea—would greatly enrich the existing literature.

In addition, we also call for more diversification in terms of the investigated domains and tasks. Thus far, the literature has been dominated by fairness perceptions around ADM in the criminal justice system and human resources. However, ADM systems have recently surged in many other areas of society, such as distributing social benefits (Noriega-Campero et al., 2020). Thus, more empirical research is needed to compare fairness perceptions across various domains and tasks systematically.

Second, we argue in favor of more harmonization in measurements. Ideally, reliable measures of perceived algorithmic fairness should be developed and validated following the theoretical groundwork outlined above. These would make new findings more comparable and allow for more nuanced interpretations of the results. A multi-dimensional measurement would provide information about the specific deficiencies of an algorithmic process instead of merely indicating that a process is perceived as (un)fair. For example, knowing whether consistency or accuracy concerns drive a perception of unfairness could enable developers to fine-tune ADM processes according to citizens' desires. Recent studies have tackled this challenge for ADM-adjacent concepts, such as “threats of AI” (Kieslich et al., 2021), introducing validated scales easily adaptable for use in different domains.

Understanding consequences of ADM systems

This review shows that the literature dedicates more attention to investigating the drivers of algorithmic (un)fairness than to its social implications. This emphasis is understandable since insights into the predictors of fairness perception are vital to designing fair ADM systems. However, only eleven studies investigated the consequences of algorithmic fairness perceptions. This is surprising because attitude changes or actions resulting from perceived (un)fairness may have profound societal ramifications. Not only may reputation losses offset potential gains in reducing costs or increasing impartiality. More importantly, understanding the social implications of algorithmic (un)fairness is vital to shedding light on what is at stake. Suppose public and private institutions fail to design and implement ADM systems perceived as fair and deny citizens a voice in this process. In that case, citizens could become alienated or lose trust in public institutions and thereby become more vulnerable to populist rhetoric.

Conceptualizing perceptions of algorithmic fairness as a mediator variable in structural equation models can be a valuable way to study both the drivers and the consequences of algorithmic fairness (Shin, 2021; Shin and Park, 2019). We encourage more research that builds on such approaches and examines the potential implications of perceived algorithmic unfairness for other attitudinal and behavioral variables.

From human-in-the-loop to society-in-the-loop

The results of the perceived fairness of HDM versus ADM are inconclusive; they provide evidence for and against the assumption that algorithms are seen as fairer than HDM. This variation of results can be found across domains (e.g. recidivism risk prediction vs. hiring) and different tasks within the same domain (e.g. two different hiring algorithms). This shows that fairness perceptions are highly context-specific and that every algorithm requires thorough investigation before being widely used. However, simply differentiating HDM and ADM falls short of capturing the natural world's complexity (Binns, 2022). In most real-life tasks, ADM systems do not decide entirely independently; instead, humans are also involved in the decision-making process. For instance, COMPAS makes a prediction, which serves as a recommendation for the final decision by a human judge. Such forms of hybrid decision-making (Starke and Lünich, 2020) can be fine-tuned and deserve more nuanced attention from the empirical literature. For instance, the European Commission (2019) outlines three approaches to human oversight. First, humans are involved in every step of the decision cycle of an ADM system and can intervene at any point (human-in-the-loop). Second, humans can intervene during the design cycle of an ADM system and monitor the system's operation (human-on-the-loop). Third, humans oversee an ADM system's overall economic, societal, legal, and ethical impacts and have authority over its use in any situation (human-in-command).

This leads to a broader point of institutional implementation. Introducing an ADM system into an institutional context raises critical questions that extend far beyond the algorithmic design (e.g. input data, code) and the specifics of human involvement in the decision-making process. This review revealed a blind spot in the existing literature, as few studies considered the broader institutional context and the emerging questions in implementing an ADM system. For instance, how are humans who might use ADM systems within an institution trained? How are decisions made by an algorithm communicated to those most affected by the decision (e.g. job applicants, defendants)—are they even made aware that an algorithm made the final decision? Do affected citizens have an opportunity to appeal, and who should ultimately be liable for false and/or discriminating classifications?

These are just a few examples of the many emerging questions that are likely to affect citizens' fairness perceptions. Addressing these blind spots requires more interdisciplinary research involving computer scientists, social scientists, legal scholars, and ethicists (Lepri et al., 2018). For instance, computer scientists add technical expertise in how ADM systems work and allow using real algorithms instead of hypothetical scenarios. Social scientists add the methodological expertise necessary to research human perceptions, attitudes, and behaviors. Such research collaborations can help put “society in the loop,” which Rahwan (2018) defined as human-in-the-loop plus social contract. This approach argues that the more ADM takes over decisions that profoundly affect citizens' livelihoods, the more essential it becomes to embed social values and norms in designing and implementing such systems.