Algorithmic reparation

Why does ML reproduce inequality?

The fundamental reason that ML algorithms continue to reproduce inequality is because these technical systems are intrinsically and fundamentally social (Ames, 2018; Bucher, 2018; Kitchin, 2017; Seaver, 2017). Put simply, algorithms are animated by data, data come from people, people make up society, and society is unequal. Algorithms thus arc towards existing patterns of power and privilege, marginalization and disadvantage (Benjamin, 2016, 2019; Broussard, 2018; Browne, 2015; Costanza-Chock, 2020; Davis, 2020; D’Ignazio and Klein, 2020).

Barocas et al. (2017b) summarize the ML process as a pipeline that proceeds in four steps: capture and quantify what is (measure)→model generalizations from the training data (learn)→apply the model to novel inputs (action)→collect feedback and refine. Through the course of this pipeline, there are several specific, overlapping ways algorithmic inequalities materialize. They can be a product of unjust goals rooted in racist, sexist, heteronormative, ableist, nationalist, and/or colonialist priorities; they can derive from biased, non-representational data; they can use biased proxies (e.g. arrest rates as an indicator of actual crime rates); and they can take real population differences that have been created through structural oppression and treat these differences as unproblematic and essential (e.g. health insurance pricing that penalizes Black men and rewards White women based on differential rates of chronic illness) (Caplan et al., 2018; Hoffmann, 2019).

FML and algorithmic idealism

The problem of algorithmic inequality is not lost on computer scientists and engineers. Indeed, a vibrant field of FML has emerged with the shared goal of rectifying biases in ML systems (e.g. Barocas et al., 2017b; Chouldechova and Roth, 2020; Corbett-Davies and Goel, 2018; Kearns and Roth, 2019; Suresh and Guttag, 2019). A recent review categorizes technical FML solutions into three categories, which map onto distinct definitions of fairness: anti-classification, classification parity, and calibration (Corbett-Davies and Goel, 2018). We define and discuss each of these in a subsequent section. For now, the relevant point is that each of these solutions proposes a computational path towards fair algorithmic outcomes. However, despite laudable aims, the proposed solutions consistently fall short.

FML approaches fall short because they stem from what we refer to as algorithmic idealism, enacting computation that assumes a meritocratic society and seeks to neutralize demographic disparities. Such an approach will always be inadequate in a context that is fundamentally unjust (Fazelpour and Lipton, 2020, Green and Viljoen, 2020). Algorithmic idealism begins with a base belief in equal opportunity, defining the problem of stratification as one caused by fallible human biases on the one hand, and imperfect statistical procedures, on the other. This perspective derives from illusory cultural narratives that misalign with the world that is – a world in which discrimination is entrenched, elemental and compounding at the intersections of multiple marginalizations. From their current theoretical packaging, FML proposals emerge disinterested and objective; they seek optimal precision to apportion risks and rewards evenly across neatly bounded identity-based groups. Such proposals are consistently eluded by the fairness they mean to achieve.

We take FML's idealism as our point of departure, proposing instead algorithmic reparation, which re-conceives society through a critical Intersectional lens. This approach strives not for social equality, which treats everyone the same, but for social equity, which provides resources based on differential need, thus accounting for axes of historical (dis)advantage (Cook and Hegtvedt, 1983; Deutsch, 1975; Rawls, 1971). This means doing away with fairness and instead, coursing resources to those who have been systematically denied. This approach pairs the logic of Intersectionality with the praxis of reparation.

Intersectionality as a lens on ML

Intersectionality is not a singular theory, but an approach and a prism with a set of orienting assertions, goals and tools. It undergirds critical theories across subfields – critical race theory, critical feminist studies, queer theory – all of which share a fundamental focus on systemic power relations that privilege and penalize centralize and silence (Cho et al., 2013; Collins, 2019; Crenshaw, 1990; Hooks, 2000; Rahman, 2010). An Intersectional orientation is premised on the notion that identities are multiple and interrelated, shaped by and filtered through, societal structures and institutions. These structures and institutions concentrate and compound opportunities and constraints in ways that reflect and reinforce essentialized hierarchical arrangements. However, these hierarchical arrangements are not predetermined, and practitioners of Intersectionality task themselves with revealing and undoing, systems of injustice (Chepp and Collins, 2013; Collins, 2002; Collins and Bilge, 2020).

Intersectionality has taken on various meanings and been deployed towards varied ends while sustaining a core set of tenets (Cho et al., 2013; Collins, 2019; Ferree, 2018; McCall, 2005). The main tenets of Intersectionality are that inequalities are systemic and entangled, meaning that identities cannot be understood apart from their interrelation with each other and from their imbrication with socio-structural systems; ‘objectivity’ is never neutral, meaning positionality matters and marginal subjects provide a necessary but undervalued lens; that inequalities manifest through legal, personal, and professional (dis)advantage; and that hierarchies of power and privilege hide behind essentialisms, rendering their mechanisms imperceptible by default. These tenets combine with imperatives to expose and negate essentialisms; empower the marginalized; and to name, highlight, and challenge agents and structures of domination (Carastathis, 2016; Collins and Bilge, 2020; Ferree, 2018).

Although Intersectionality has become embedded in academic texts and activist movements, it originates in the legal sector. Intersectionality arose in response to legal codes that erased and ignored co-occurring identity axes (e.g. Black women), working to account for discriminatory policies and practices that affect doubly marginalized legal subjects. With these legal foundations, proponents of Intersectionality emphasize the approach as an active political project (Cho et al., 2013; Collins and Bilge, 2020). Intersectionality is not just something to think with, but something to do. It is an intellectual method, but also, and in the first instance, a tool for empowering people and fostering social justice (Collins and Bilge, 2020). Thus, beyond identifying cases of systemic disadvantage, an Intersectional project also works to surge resources to those who are marginalized and deprived. This imperative to treat Intersectionality as a grounded, practical, material endeavor, can be served through the application of Intersectionality to ML evaluation and design.

As an approach to ML, our deployment of Intersectionality joins with and builds on a growing body of work attending to socio-historical power relations within computational systems. These include proposals for critical race methodologies for algorithmic fairness (Hanna et al., 2020), critical race theories applied to human–computer interaction (Ogbonnaya-Ogburu et al., 2020), decolonial AI (Mohamed et al., 2020), decolonial computer science (Birhane and Guest), computing for social change (Abebe et al., 2020), and affirmative action in algorithmic policing and criminal sentencing (Humerick, 2019). Inspired by, and combining elements from each of these projects, algorithmic reparation has a fundamental foundation in praxis, an emphasis on the multiplex of intersecting identities, and an explicit position of compensatory resource redistributions accomplished proactively through a reparative approach.

A reparative approach

Bringing Intersectionality to bear on ML, and bringing ML to bear on Intersectionality, grounds Intersectional politics in material conditions that interplay with contemporary lived experience through computational forms of governance and mundane technical engagements. That is, the doing (and undoing) that drives Intersectionality converges directly with issues of algorithmic inequality (Benjamin, 2019; Costanza-Chock, 2020; Mann and Matzner, 2019). We suggest animating Intersectional politics through practices of reparation.

‘Reparation’ is a historically grounded mechanism by which offending parties symbolically and materially mend wrongdoings enacted against individuals and groups (Torpey, 2006). Reparations have been assigned in the context of war (Lu, 2017; Young, 2010), in acknowledgement of and apology for acts of colonialism (Gunstone, 2016; Lenzerini, 2008), and they remain a point of mobilization for Black civil rights activists in the United States, demanding material recompense for the multigenerational damages of slavery and segregation (Bittker, 2018 [1972]; Coates, 2014; Henry, 2009). Reparative acts are not just backward-looking, but also proactive, aiming to address the way historical wrongdoings affect current and future opportunity structures by channeling resources to make up for and overcome existing deficits.

Although traditionally applied in a legislative, often geopolitical context, we use ‘reparation’ in a broader sense, arguing for structural redress through algorithmic reform. This is more than the conceptual loosening of a legal term. Legal and political systems hinge reparation on identifiable culprits and victims along with demonstrable links between the wrongdoing of one party and the consequences of wrongful actions upon the aggrieved. However, this is rarely how structural, Intersectional oppressions operate. What makes Intersectional oppressions so pervasive and pernicious is their diffusion through institutional infrastructures, policies of governance, language, culture, individual attitudes and interpersonal dynamics. The systematic, multifaceted, often subtle nature of Intersectional inequality is at odds with linear relations of harm and blame. Algorithmic reparation thus incorporates redress into the assemblage of technologies that interweave macro institutions and micro-interactions, embedding an equitable agenda into the material systems that govern daily life ³ .

Our call for reparative algorithms is motivated by a broader mandate for equity and social justice, but it is also motivated by the specific conditions of automation that leave no neutral option (Broussard, 2018; Bucher, 2018; Mann and Matzner, 2019; Noble, 2018). In general, the distribution of resources can either reinforce inequalities, make them worse, or make them better. However, ML systems are intrinsically self-perpetuating in ways that ossify and intensify the outcomes they engender. This is because algorithms render decisions seemingly objective and divorced from human discretion; because they are opaque and inscrutable; and because their outcomes often have no technical means of undoing, even if circumstances call for correction (Bucher, 2018; Eubanks, 2018; Gillespie, 2014, 2018; Pasquale, 2015; Vaidhyanathan, 2018). Our proposal for algorithmic reparation assumes a moral duty to ameliorate, rather than aggravate, structural and historical stratifications as they manifest in computational code. This proposal sits in direct opposition to the prevailing logic of FML, which seeks to de-bias algorithms and make them fairer. In contrast, a reparative approach assumes and leverages bias to make algorithms more equitable and just.

Anti-classification

Anti-classification stipulates that algorithmic estimates do not consider protected class attributes such as race, class, gender, or (dis)ability. This includes direct consideration of these characteristics as well as proxies for them. Corbett-Davis and Goel (2018) equate this to principles of equal protection under the law (Karst, 1977) and ‘taste-based’ discrimination in economics (Becker, 2010 [1957]), by which advantages and disadvantages cannot be assigned based on demographic preference. Algorithmically, anti-classification systems strive to encode indifference to the identities of individuals who will be subject to automated outcomes.

Anti-classification principles underlie automated employment programs that aim to circumvent managerial biases in candidate selection, avoiding the historical race–class–gender–age–nationality (dis)advantages that have historically shaped which candidates make it past initial screenings (Lahey and Oxley, 2018; Oreopoulos, 2011; Quillian et al., 2017). In practice, these algorithmic systems reproduce social hierarchies pervasive to the populations from which they select. Technology conglomerate Amazon's use of anti-classification algorithms exemplifies this point.

In 2014, Amazon developed a recruitment tool to aid in its own hiring processes. The tool used ML to sort applicants based on optimal fit for each position, removing social identity characteristics from consideration (Dastin, 2018). The trifold purpose was to increase efficiency, select the best candidates, and avoid implicit biases, especially against women, as this group has been (and remains) underrepresented in the technology sector (Beede et al., 2011; Harrison, 2019). However, by 2015, it became evident that the automated system was not operating as planned. Consistently, the recruitment algorithms assigned higher scores to men and lower scores to women. The reason for this is that the system was trained on the company's previous 10 years of employment data, which reflected a male-dominated sector. That is, Amazon's workforce, like the broader technology workforce, was populated disproportionately by men. Consequently, using existing data, the hiring system learned that men were the preferred candidates. This self-perpetuating cycle was so pronounced that any indicator of feminine gender identity in an application lowered the applicant's score. A degree from a women's college, participation in women-focused organizations, and feminized language patterns all reduced the evaluative outcome. Although Amazon attempted to adjust for these issues, the system continued to find proxies for gender and reward men at the expense of women. Amazon eventually retired the program (Dastin, 2018).

From an Intersectional perspective, anti-classification systems are intrinsically faulty. These systems are premised on the erasure of difference, a flattening of demographic traits. Such an approach ideologically sidesteps the empirical reality of systemic inequality, but it cannot statistically or mathematically address it. The data that feeds these systems and the people who are subject to them, operate from hierarchically differentiated positions. These distinctions are, and will continue to be, captured and reproduced through computation.

In contrast, a reparative approach would highlight, name and encode hierarchical distinctions as they manifest across social identity categories. From this foundation, Amazon's algorithms would not invisibilize gender but would instead define gender as a primary variable on which to optimize. This could mean weighting women, trans, and non-binary applicants in ways that mathematically bolster their candidacy, and potentially deflating scores that map onto stereotypical indicators of White cisgender masculinity, thus elevating women, trans, and non-binary folks in accordance with, and in rectification of, the social conditions that have gendered (and raced) the high-tech workforce. Moreover, it would not treat ‘woman’ as a homogenous (binary) category, but would label and correct for intersections of age, race, ability and other relevant variables that shape gendered experiences and opportunity structures.

This reparative system would literally value the contributions underrepresented applicants bring to the company while normalizing Intersectional gender diversity in tech, such that high-level positions and the pathways to them, are recast as plausible and expected across gender groups. The technical solution (women, trans, and non-binary individuals get a statistical boost) would thus have direct effects on the company's work environment (more women, trans, and non-binary employees are hired at Amazon) and broader social effects on Intersectional gendered social relations (women, trans, and non-binary folks are normalized in the technology sector and the pathways to technology careers more seamless for these individuals to pursue). If these ends remain untenable, a reparative approach would indicate that ML ought not to be used in hiring decisions ⁵ .

Classification parity

Classification parity is defined in terms of equal errors in classification across social identity groups. This aims to achieve parity in the error rates of predictive performance measures. Corbett-Davies and Goel (2018) identify several measures of classification error: false-positive rates, false-negative rates, precision, recall, the proportion of decisions that are positive, and the area under the ROC curve (AUC) (see Berk et al., 2018; Skeem and Lowenkamp, 2016). They focus in particular on false positives and the proportion of decisions that are positive, as these are the error metrics that FML researchers have given the most attention (Corbett-Davies and Goel, 2018). We also focus on those metrics here, along with false negatives, as these are relevant to high-profile cases of algorithmic inequality.

False positives and false negatives are errors in predicting how likely it is that something will (or will not) happen. Proportion of positive decisions, also known as ‘demographic parity’ (Feldman et al., 2015), means that a given outcome distributes equally across social identity groups. These measures – false positives, false negatives, and demographic parity – have been central to debates about (and critiques of) ML in criminal sentencing, the most notable case of which is the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) recidivism risk assessment tool.

COMPAS is a widely used and commercially available tool designed to predict the likelihood that a criminal defendant will reoffend. In 2016, a ProPublica report analyzed pre-sentencing data from Broward County, Florida, a large jurisdiction using the COMPAS system. The report found that Black defendants were systematically assigned higher risk scores than White defendants, and that risk was overpredicted for Black defendants and underpredicted for White defendants (i.e. Black defendants recidivated at a lower rate than what the algorithm predicted, and White defendants recidivated at a higher rate than what the algorithm predicted) (Angwin et al., 2016a, 2016b). The overall disparity in Black–White risk assessments represents an error of demographic parity, while the over- and under prediction of Black–White recidivism represents errors of false positives (Black defendants) and false negatives (White defendants).

Classification parity is rooted in the assumption that if errors distribute evenly, then decisions will be fair, and that if data are accurate and representative enough, fair distributions can be achieved. Intersectionality indicates that these assumptions are misguided. They are misguided because the effects of risk are not equivalent across groups and because the data that feed into the ML system are infused with social policies and practices that shape statistical inputs and outputs. In terms of sentencing decisions, a criminal record and time in prison are undesirable for anyone. However, the negative effects of conviction and incarceration compound for individual Black defendants, flow on to their families and communities, and reinforce racial disparities in wealth, health, family stability, and mental well-being (Pettit and Western, 2004; Travis et al., 2014; Western and Pettit, 2010; Western and Sirois, 2019). Moreover, the likelihood of contact with police and of conviction is significantly higher for poor Black men than any other group (Alexander, 2010). Indeed, criminal justice data are self-perpetuating, such that those groups defined as criminally ‘risky’ are in fact, at disproportionate risk of ensnarement by the criminal justice system (Brayne, 2017; Brayne et al., 2015; Christin et al., 2015; Ferguson, 2017; Richardson et al., 2019).

A reparative approach would supplant the goal of ‘parity’ with, instead, systemic redress, beginning with the social facts of disproportionate risk between racial groups and the history of race–class dynamics that inform training data. From this, reparative decision aids would work to actively protect poor communities of color, especially poor Black men, over and above other subpopulations. This means the production and deployment of algorithms that keep Black men out of prison and keep police out of Black communities, defending against the criminalization of Blackness and rectifying racialized prison pipelines.

Calibration

Calibration specifies that ‘outcomes should be independent of protected attributes conditional on risk scores’ (Corbett-Davies and Goel, 2018: 6). Calibration can be thought of as a more nuanced take on anti-classification. The calibration approach is such that identity characteristics should only be considered by an algorithmic equation if those characteristics have demonstrable, empirical effects on the outcome under consideration. That is, the system calibrates to differential risk levels between groups and assigns scores according to those base-level differences.

To illustrate calibration, we remain with the COMPAS example. We do so because Northpointe, the company behind COMPAS, has responded to critics by claiming that in fact, their algorithms are fair because they satisfy calibration. What they mean is that a Black defendant classified as high risk by COMPAS is equally likely to recidivate as a White defendant classified as high risk. In an open letter to ProPublica, the company states:

ProPublica focused on classification statistics that did not take into account the different base rates of recidivism for blacks and whites. Their use of these statistics resulted in false assertions in their article that were repeated subsequently in interviews and in articles in the national media (Dietrich et al., 2016: 1) (emphasis added).

Defending itself, Northpointe justifies its product based on calibration standards. Their defense is inadequate on both technical and social grounds.

On a technical level, although errors calibrate for base differences between groups, the kinds of errors are inconsistent. As detailed by ProPublica, Black defendants remain subject to disproportionate false positives and White defendants are rewarded with disproportionate false negatives (Angwin et al., 2016a). Black people are mis-assessed with overly strong risk scores and White people are mis-assessed with overly lenient risk scores. Concretely, this means that more Black people end up in jail and more White people remain free.

There are also non-technical reasons to be dissatisfied with Northpointe's response and in turn, dissatisfied with calibration as an algorithmic standard. In particular, the data used by Northpointe to train their algorithms reflect racist policing tendencies in the United States that over-indict Black men, creating (not just reflecting) different base rates between raced, classed, and gendered groups (Brayne, 2017; Brayne et al., 2015; Ferguson, 2017; Richardson et al., 2019). Moreover, the carceral system not only responds to criminality, but through a constellation of mechanisms, also begets further violations (Alexander, 2010). Thus, Northpointe's reliance on calibration as a technical justification affirms and entrenches a system in which existing injustices act as the basis for their own amplified reproduction.

Like anti-classification, calibration seeks to remove identity from the decision equation (though in qualified form). Like classification parity, calibration works to achieve equivalence between groups (by adjusting for differential base risk). As detailed in the subsections above, both of these objectives are ineffective for reducing inequality and may intensify inequitable social arrangements. In contrast, algorithmic reparation rejects the notion of identity erasure, even on the grounds of empirically distinct risk rates. It instead takes stock of social disparities as they map along axes of identity, exposing the underlying causes of differential risk and undoing the stratification that those differences both represent and produce. In practice, this could mean higher recidivism risk thresholds for Black defendants, lower thresholds for bail and parole, and weighted statistical adjustments that account for over-policing in poor communities of color. Evaluated through a reparative Intersectional lens, any algorithm that did not address these base inequities would be deemed inadequate.

Methods of implementation

Archivist curation is one promising approach to implementing algorithmic reparation. This draws on the professional expertise of archival practice, honed by librarians and museum curators, applying these skills to ML data (Donovan, 2020; Jo and Gebru, 2020). Unjust algorithmic outputs are inextricable from problems with source data. These problems can be a function of representation in datasets and/or social factors that crystalize in data form. Managing these data issues can be prohibitively complex. However, professionals trained in collection and curation have skill sets that are transferrable to the ML sector, with Jo and Gebru (2020) noting consent, inclusivity, power, transparency, ethics, and privacy as data-relevant issues that have been well addressed in library sciences.

Drawing on their extant skill sets, curation professionals are capable of managing, collecting, arranging, and auditing data in ways that not only avoid re-entrenched inequalities, but optimize for marginal elevation, enacting targeted precision unachievable by those who are not professionally trained in curatorial methods. This includes the capacity to account for complex identity configurations in which advantages and disadvantages are in simultaneous operation, and the insight to determine which pieces of data are relevant to collect and, more importantly, what data ought not be collected. Such skills and practices are well suited to the problems discussed above, such as hiring and criminal sentencing, in which the complexity of the data and its entanglement with a multitude of confounding and compounding variables have proven intractable for data practitioners alone.

Distributed AI power is a second potential method. This method is premised on undoing standard power asymmetries between those who make, and those who are affected by, ML systems. The approach argues for tools that are legible to, and co-created with, impacted communities, especially those communities with histories of vulnerability prior to, and re-entrenched with, automation (Kalluri, 2020). Distributed AI power tactics rely on reciprocal engagement between developers and community stakeholders, with reverse pedagogies by which community stakeholders serve as experts in their lived experiences (Mohamed et al., 2020).

This method is exemplified by academic–activist collaborative projects, undertaken by groups such as the Algorithmic Justice League, Data for Black Lives, and the Carceral Tech Resistance Network, among others. Each of these organizations leverages community knowledge to challenge and partner with commercial, governance, and regulatory bodies to enact technical, social, and policy changes. The Algorithmic Justice League, for example, has performed audits of race and gender in facial recognition technologies, leading several companies to revamp their programming in ways that improve the classification accuracy for dark-skinned women in image search tasks (Raji and Buolamwini, 2019). Data for Black Lives, which coordinates thousands of engineers, mathematicians and activists, is training former inmates in data science so that this directly affected population can actively participate in the reform of the criminal justice system (Heaven, 2020). In turn, the Carceral Tech Resistance Network (2020) trains in and with communities, mobilizing towards the abolition of carceral tech and reparations for these systems’ racialized damages. All three organizations have joined with others to activate against the use of facial recognition in policing, demonstrating the fundamental incongruity between these tools and racial justice, eventuating a cascade of corporate and legislative moratoria (Flynn, 2020; Heilweil, 2020; Lazar et al., 2020). These projects begin with, are led by, and develop through, affected communities, with a record that demonstrates the capacity to enact reparative approaches to ML evaluation and design.

Barriers

Grounding algorithmic reparation means identifying both opportunities and challenges. The methods just discussed represent encouraging prospects, but there are empirical reasons that ML keeps reproducing inequality, and these realities are robust and obdurate. Enacting algorithmic reform requires unvarnished realism about the conditions under which any sociotechnical intervention will go into effect. For algorithmic reparation, implementation will face interrelated social, legal, and institutional barriers. Although addressing each barrier is beyond the scope of the present work, we lay them out to set clear terms for the path ahead.

Socially, reparation relies on a base logic that diverges from normative conceptions of fairness, opting instead for uneven resource allocations targeted at the margins. As evidenced by the backlash against affirmative action policies and resistance to critical race curricula (Ray and Gibbons, 2021; Vought, 2020), an intentional reallocation of resources will, undoubtedly, come up against significant friction. Rectificatory tactics will be difficult to accept for those who ascribe to an image of society that is functionally meritocratic, and this baseline assumption is indeed, deep-seated.

There will also be legal and institutional challenges. Reparation calls for centralized knowledge about and action based upon protected class attributes. This is difficult under legal conditions that prohibit the collection of such data and/or its consideration in consequential decisions like employment, lending, school admissions, and criminal sentencing (Lieberwitz, 2008; Long and Batemen, 2020; Skeem and Lowenkamp, 2020). Similar prohibitions written into institutional policies will create blockades against algorithmic reparation within organizational settings.

There are also real challenges to the kinds of interdisciplinary and socially engaged collaborations necessary for reparative algorithmic projects. Power and compensation disparities persist between computer scientists and social scientists, and between academic and non-academic organizations (Carrigan and Bardini, 2021; Hackett and Rhoten, 2011; Stavrianakis, 2015; Viseu, 2015), along with epistemological schisms that are difficult to reconcile (Bauer, 1990; Richter and Paretti, 2009). These impediments to meaningful inter/trans/non-disciplinary collaboration are exacerbated by academic incentive structures that reward traditional intra-disciplinary outputs over and above hybrid and expansively defined research products (Woelert and Millar, 2013), despite widespread statements about the value of disciplinary blending and community-engaged science (Hackett and Rhoten, 2011; Viseu, 2015). Contending with these institutional challenges means considering not only who will do the work of algorithmic reparation, but also how it can be done across sectors, with the support of leadership, mechanisms of accountability, democratic oversight, and equitable returns for practitioners’ labor.

Summary

Technologies reflect and create the societies from which they stem and in which they proliferate. By default, technologies will embody the values of the powerful and reconstitute the stratified hierarchies those values represent (Benjamin, 2016, 2019; Broussard, 2018; Browne, 2015; Costanza-Chock, 2020; Davis, 2020). These patterns of reflection, reconstitution, and in turn, amplification of structural inequality have borne out in spectacular fashion with the integration of ML systems into personal and institutional life.

The field of FML has emerged in response, with computer scientists and engineers proposing myriad technical fixes to the injustices of automation. Yet, algorithmic inequalities persist. In their efforts to hide, distribute evenly between, and calibrate social identity traits, FML practitioners operate with a goal of fairness and equality when instead, equity and reparation are required. We make this case in the body of the text above, suggesting a move away from fairness, replaced by an anti-oppressive, Intersectional approach. We intend for this approach to guide algorithmic design and to act as an evaluative standard by which existing algorithmic systems are judged, adjusted, and where necessary, omitted or dismantled. Our proposal is thus geared towards building better systems and holding existing ones to account.

We highlight two possible methods of implementation – professionalized archival data curation and distributed AI power. Both methods are consonant with the base assumptions and objectives of algorithmic reparation and they both show promise as practical means for algorithmic reform. We also take stock of social, legal, and institutional barriers to implementation, providing a realistic perspective on the work ahead.

Next steps

Continuing this focus on the work ahead, we conclude by considering next steps in the ongoing project towards social and technical restructuring. Here, we emphasize the need for context-specific attention, more and multiple tools, and multipronged approaches that converge technical, social, and institutional efforts.

Instruments of social change – technical or otherwise – never operate in a vacuum. In the final substantive section of this paper, we selected two newly introduced mechanisms by which algorithmic reparation might be implemented. Testing these in diverse empirical settings will reveal how they function, where they fall short, and what kinds of infrastructural conditions will be required for these methods to take meaningful effect.

It will also be vital to explore and create a cache of methods and tools, addressing specific needs, specific conditions, and creating interoperability between social and technical systems. The acute need for a constellation of methods and tools becomes clear when we consider the varied and engrained structural reasons why inequalities continue to manifest in algorithmic form. Algorithmic reparation will, necessarily, run against the grain of multiple status quos, requiring numerous iterations, agile applications, and persistent adjustments for this uphill endeavor.

In service of creating a robust toolbox, this paper's third author (Yang) is currently leading a project to devise technical instruments that audit and optimize for inequality reduction in decision systems. This is a computational mechanism that centers impact estimations that most reduce inequality in automated decision outputs. These auditing tools are intended specifically for institutional decision aids, such as those used in hiring processes, loan allocations, and admission decisions, calibrated to the particular inequalities of the communities affected. Projects such as this, which are currently in development, portend a new and critically informed landscape of sociotechnical relations.

We also note that ‘next steps’ cannot be technical alone. Any algorithmic solution to social problems is necessarily partial and incomplete, requiring complementary social, legal, and institutional evolutions. Concretely, this means rethinking discrimination policies that erase and thus ignore identity attributes; reworking institutional incentive structures and power arrangements that silo academic disciplines from each other and from the public sector; introducing regulatory implements that capture and censure discriminatory algorithmic outputs; and forming organizational bodies dedicated to auditing technical systems and assuring their allocative and representational ends.

In practice, the problems of algorithmic systems are the problems of social systems, and meaningful solutions will be technical and social in nature. These solutions will not come easy, nor fast, nor with absolute finitude. Just as undoing racism, sexism, classism, and colonialism are continuous, evolving, non-linear projects, so too is the journey to unmake the inequalities of algorithms and code. Algorithmic reparation is not a final or encompassing answer, but a critical, equitable, Intersectional foundation.