Addressing discrimination in algorithmic profiling: Examining risk governance in Dutch public social security agencies

Regulating discrimination risks in algorithmic profiling

The legal basis upon which the agencies studied operate, the Wet SUWI, does not specify safeguards against discrimination. Thus, agencies are tasked with interpreting and implementing legal obligations aimed at safeguarding against discrimination, stemming from the European Convention on Human Rights and the GDPR, while also anticipating requirements that will be applicable under the AI Act. ⁶ Furthermore, when these agencies act within the scope of EU law, such as when applying the GDPR or later the AI Act, they are also bound by the Charter of Fundamental Rights of the European Union. ⁷

Under fundamental rights legislation, the right to non-discrimination entails the substantive prohibition of discrimination based on protected grounds such as race or ethnicity, while equal treatment requires any differential treatment to be reasonable and fair (Borgesius, 2020; Eder, 2021; Gerards and Borgesius, 2022; Tilburg University, 2021). ⁸ The right to privacy and data protection regulate the collection and use of data to prevent discriminatory outcomes in algorithmic profiling (Ligue des droits humains, 2022; ⁹ Roberts, 2015: 2127; The Hague District Court, 2020 ¹⁰ ). ¹¹ Additionally, the right to an effective remedy aims to ensure that individuals can challenge and seek correction for discriminatory decisions made by organizations employing algorithmic profiling systems (O'Sullivan, 2024). ¹² These rights impose both ex-post protections and ex-ante obligations, requiring organizations to address risks of discrimination when designing and deploying algorithmic systems (Borgesius, 2020; Tilburg University, 2021).

More specific to the context of algorithmic profiling, the GDPR and AI Act introduce complementary safeguards requiring agencies using these technologies to identify and mitigate risks of discrimination. The GDPR applies more broadly to the automated processing of personal data, including profiling, which refers to processing personal data to evaluate personal aspects such as one's behavior. ¹³ This can include, for example, the use of both simple rule-based and complex trained risk profiling systems to select groups where there is a risk of misallocation of benefits (Autoriteit Persoonsgegevens, 2024).

In relation to such systems, the GDPR requires, for example, that the processing is lawful, fair, data such as race and ethnicity are not processed, and that there is meaningful human intervention (Article 29 Working Party, 2017a). ¹⁴ Importantly, where data processing poses a high risk to the fundamental rights of citizens, as in the case of algorithmic profiling, Data Protection Impact Assessments (DPIAs) are prescribed to identify and mitigate risks to the privacy and data protection of data subjects, including discrimination (Article 29 Working Party, 2017b). ¹⁵

Similarly, the recently adopted AI Act, once applicable, will require organizations to manage risks across the life cycle of high-risk AI systems, including governance practices, assessments of risks to fundamental rights, and the management of such risks. ¹⁶ While the AI Act provides a broad definition of AI systems, it remains unclear exactly when risk profiling systems fall within the scope of the regulation (European Commission, 2025; Hacker, 2023; Sousa e Silva, 2024). ¹⁷ Thus, while these legal frameworks seek to impose obligations to address discrimination risks associated with technology such as algorithmic profiling, they offer limited guidance on how agencies should operationalize these requirements in practice.

Challenges in addressing the systemic risk of discrimination

Addressing the systemic risk of discrimination in algorithmic profiling is, however, shrouded in complexity, uncertainty, and ambiguity (Van Asselt and Renn, 2011). Risks are complex when they are multicausal, which makes it difficult to attribute discriminatory outcomes to specific factors (Van Asselt and Renn, 2011: 436–437). Discriminatory outcomes are multicausal as they can be traced back to choices and unmitigated social and technical risks in the life cycle of such systems, including their design, implementation, and use (Haitsma, 2023; Marabelli et al., 2021). These include the use of biased training data, directly or indirectly discriminatory predictive criteria, biases in the manual review of selected cases, insufficient monitoring and evaluation mechanisms, and opacity toward external parties including citizens (Amnesty International, 2021; Constantaras et al. 2023; DDPA, 2024; Haitsma, 2023; Mehrabi et al., 2021; SyRI, 2020).

Uncertainty refers to when there is limited scientific knowledge on how to assess and manage risks and potential negative outcomes (Klinke and Renn, 2012: 276). In the context of algorithmic profiling, it is well-established that discrimination is a recurring and structural phenomenon, as demonstrated by countless incidents that clash with the right to non-discrimination, both within and outside of the Netherlands (Amnesty International, 2021; Bouwmeester, 2023; Chowdhury, 2024; Constantaras et al., 2023; NOS, 2024; Pelgrim, 2025; SyRI, 2020). However, interdisciplinary research into the causality of these discriminatory outcomes is still in its infancy (Algorithm Audit, 2024a; Dankloff et al., 2024; Haitsma 2023; Marabelli et al., 2021). While methodologies for addressing discrimination in algorithmic profiling are being developed, further research is still needed to examine their utility and effectiveness (Boonstra et al., 2024; Franzke et al., 2021; Meuwese et al., 2024; Tilburg University, 2021; Vale et al., 2022; Vetter et al., 2023).

Finally, risks are ambiguous when views differ on how to identify, appraise, manage, and justify risks and potential adverse effects (Renn et al., 2011; Van Asselt and Renn, 2011: 437). The legal framework for algorithmic discrimination is broad, key concepts such as fairness, bias and discrimination have multiple and often ambiguous definitions, case law is limited, and the academic field is still developing—together, these factors create socio-legal ambiguity (Gerards and Xenidis, 2021: 28, 107–117; Al Khatib et al., 2024; Mehrabi et al., 2022; Meuwese et al., 2024; Zuiderveen Borgesius, 2023). In particular, views in society and within organizations may differ regarding when the use of algorithmic profiling technologies is appropriate, if and when it is justified to use certain predictive indicators, to what extent bias and overrepresentation of specific groups is acceptable, or how to facilitate meaningful transparency or explainability (Amnesty International, 2024; Loi et al., 2020; Meuwese et al., 2024; Tweede Kamer den Staten Generaal, 2024). This sociotechnical complexity, scientific uncertainty, and sociolegal ambiguity leave organizations on an uneasy footing regarding how to effectively meet their legal obligations to address discrimination.

From risk regulation to risk governance

Synthesizing decades of interdisciplinary research and global efforts directed towards risk governance, authors van Asselt and Renn developed the principles of systemic risk governance to inform responses to systemic risk. The principles of communication and inclusion, integration, and reflection stress the co-production of knowledge to govern risks, with the robustness of this approach being demonstrated by the wide variety of contexts in which it is applied (Van Asselt and Renn, 2011). These include environmental and climate risk, urban planning, supply chain risk, disaster management, public health, and, more recently, data governance (Ahlqvist et al., 2020; Taebi et al., 2020; Di Guilio, 2023; Hanssen et al., 2018; McHugh et al., 2021; Micheli et al., 2020; Renn, 2015; Renn and Klinke, 2016).

Regarding communication and inclusion, communication between a variety of actors included in risk governance is at the core of effective risk governance processes. This involves ensuring the exchange of knowledge, experience, concerns, and perspectives between relevant parties, which include governmental actors, scientific experts, and affected parties or citizen interest groups (Van Asselt and Renn, 2011: 440). Inclusion stipulates that organizations must consider which actors to include, the mechanism for their inclusion, and the scope or mandate of their role in addressing systemic risks (Renn, Klinke & Van Asselt: 246). Mechanisms can, for example, include expert panels, citizen advisory committees, and impact assessments (Renn, 2015: 14; Renn, Klinke & Van Asselt: 246).

Secondly, the integration principle refers to establishing how the resulting knowledge is combined and used to identify risks and address those risks, having due regard for potential consequences, controllability, tolerability, and acceptance (Van Asselt and Renn, 2011: 441–443).

Lastly, according to the reflection principle, organizations must, together with a variety of actors, consistently and intentionally reflect on their approach to addressing the systemic risk in question, to avoid mistreating it as a simple risk (Van Asselt and Renn, 2011: 442–443). These principles aim to promote social learning and co-production of knowledge among parties, thereby fostering legitimacy and trust, and socially robust risk governance solutions based on collective input (Renn, 2015: 14; Renn and Klinke, 2016: 254; Renn et al., 2011: 246).

Algorithmic profiling in Agency 1 and Agency 2

While their processes and types of systems differ, both agencies conduct risk-based investigations using risk profiles to combat potential misallocation of social security benefits. Regarding the prompting of a risk-based investigation, in Agency 1 risk profiles are created according to the risk-based investigation process, which is laid down in a process description (Risk-based investigations process description, 2024). This process begins with an internal or external signal—such as insights from prevention and enforcement agents or a fraud scandal in the media—indicating a phenomenon of misallocation. The organ responsible for risk-based investigations in the prevention and enforcement department receives this signal, investigates it, and, if warranted, can take mitigating measures to prevent the risk.

In Agency 2, during the Integral Fraud Risk Analysis process (hereafter referred to as IFRA), legal provisions enforced by the agency are examined to determine if there is a risk of misallocation. These risks are further investigated according to the “Research and Analysis Process” (hereafter R&A Process) laid down in the “Research and Analysis Protocol” (hereafter R&A Protocol) (R&A Protocol, 2024). The R&A process is carried out by multidisciplinary “Research and Analysis teams” (hereafter R&A teams) in the prevention and enforcement department, consisting of risk researchers, data analysts, risk mitigation advisors, enforcement agents, and sometimes a behavioral scientist. The goal of the R & A Process is to clarify the nature and scope of a risk of misallocation and investigate the population in order to inform the potential mitigation of the risk.

Query. In Agency 1, if warranted and legally and ethically justified, a simple rule-based risk profile, consisting of selection criteria, is developed and applied to select a population corresponding to the identified phenomenon. This profile enables a one-time selection—called a “query”—of a population in a data base which fit the risk profile for investigation. Likewise, in Agency 2, if during the R&A process, behavioral research, process research, and data analysis provide insufficient insight into a risk, client investigations may be used. In this case, a risk profile in the form of a simple rule-based “query” is created and applied, to select a sample of clients matching the potential risk phenomenon. If the selection itself provides insufficient insights into the risk, then the cases are investigated, clients are spoken to, and where warranted are subject to an enforcement decision.

Risk selection. In Agency 1, if the query-based investigation proves effective, and other mitigating measures inadequate, the same risk profile may be applied more than once to a larger population, to further tackle the misallocation. This requires all required steps, assessments, and approvals in the “risk-based investigation process description” to be followed again. Similarly, in Agency 2, following the R&A process and if necessary and proportionate, algorithmic systems with a more repressive character may be used. In particular, if a high rate of misallocation is found and preventive measures prove inadequate, the query can be further developed into a recurring risk selection used to select and mitigate the risk within a larger population. Employees involved in building these systems explained that risk selections focus on “singular relationships because someone is selected based on if they fulfill all criteria (within a profile)”. While these selections typically consist of fewer variables, a data analyst clarified that they can be made up “of quite a few lines of code to select the correct population”.

Trained models. Finally, the agencies studied have either experimented with developing, or are actually using trained models. Previously, Agency 1 was developing a machine learning system for risk assessments to detect and predict non-compliance with benefit conditions. However, respondents stated that due to increased societal attention regarding the risks associated with AI and the evolving regulatory landscape stemming from the AI Act, this development is currently on hold, as the agency would like to exercise caution.

Agnency 2, on the other hand, uses risk scans. Risk scans are models trained using machine learning techniques, which use more variables corresponding to a phenomenon to make selections. These models use combinations of weighted variables to process data provided by clients, resulting in a score relevant to making selections. A data analyst explained that these systems are also more systematic, in that every month thousands of clients go through the system, which also brings the risk that mistakes and biases can have more far-reaching consequences (Table 1).

OPEN IN VIEWER

Table 1.

Algorithmic profiling in Agency 1 & 2

Agency	Profiling process	Types of systems employed	Characteristics of system
Agency 1	Risk-based investigation process	- Query	- Simple/rule-based - One-off selection
		- Risk selection	- Simple/rule-based - Multiple selections
Agency 2	Research & analysis process	- Query	- Simple/rule-based - One-off selection
		- Risk selection	- Simple/rule-based - Multiple selections
		- Risk scan	- Complex/trained using AI - Systematic selections

Systemic risk governance in Agency 1 and Agency 2

Overarching risk governance landscape. The risk governance approaches in both agencies are embedded in the context of algorithm policies, process descriptions and protocols, and assessment methodologies are used to facilitate the identification and mitigation of risks of discrimination. Regarding algorithm policies, Agency 1 has an overarching algorithm policy that is currently in the later stages of development (Algorithm Policy, 2025). This document lays down policy principles aimed at providing further norms for the responsible development, application, and use of algorithmic systems within the organization. To ensure the application of these principles, overarching roles and responsibilities relevant to the governance of algorithmic systems are established. These include figures and bodies such as the privacy officer and data protection officer, the agencies ethics center, an external ethics committee, and a working group on algorithms. In Agency 2 an algorithm policy is also in place, called the Model Risk Management Policy (hereafter MRM policy). This policy seeks to ensure the legally and ethically responsible development of models that are explainable and auditable. It delineates the life cycle of models, assigns roles and responsibilities along “three lines of defense” including internal and external actors, such as the Client Council, and prescribes assessments (MRM Policy, 2024).

In both agencies, there is currently uncertainty about how rule-based systems—queries and risk selections—fit within their respective algorithm policies. Agency 1’s algorithm policy was drafted with a focus primarily on trained algorithms. According to the section manager, it was only this year, with increased attention to simpler risk profiles, that it became “clear to us that risk profiles should also fall under the policy”. Thus, “the idea is that simpler risk profiles should fall under the algorithm policy, but we still need to figure out how”.

Likewise, in Agency 2 the MRM policy was designed with their risk scan systems in mind, with these systems constituting a “model” and thus falling within the scope of the policy. Accordingly, the MRM policy applies to risk scan systems and prescribes a Data Protection Impact Assessment (DPIA) and Fundamental Rights and Algorithms Impact Assessment (hereafter FRAIA) to assess risks of discrimination associated with these systems. However, risk selection systems technically fall outside the scope of the definition of a model, and the policy. As a result, the DPIA currently serves as the main assessment for risks of discrimination associated with risk selection systems. Several respondents indicated that the agency tries to apply elements of the MRM policy to new and revised risk selections, but is in the process of creating a policy tailored to these systems.

Given the lack of clarity on how simpler risk profiles fit within the algorithm policy, process descriptions and risk-based investigation protocols act as the most relevant guiding documents, formalizing processes and prescribing assessment points. In Agency 1, while not a formal policy, the “Risk-based investigation process description” reflects key trade-offs and policy choices and is strictly followed. The document is particularly relevant to discrimination, as it formalizes points for ethical and legal assessments during risk-based investigations. In Agency 2, for client research using simpler queries, the R&A protocol describes the methodology for the desired execution of the R&A process and prescribes points for assessments. This protocol prescribes the “ethical deliberation” and the DPIA as mechanisms for identifying risks of discrimination.

Overarching governance landscape

The risk governance approaches of both agencies are embedded in algorithm policies, process descriptions and protocols, and assessment methodologies. This landscape varies depending on the type of system used. For more complex trained models, both organizations have algorithm policies—either established or in development—that outline principles for responsible development and assign roles and responsibilities to internal and external actors, such as external ethics committees and client councils. However, the relationship between simpler rule-based systems—including queries and risk selections—and these algorithm policies remains unclear, as these systems currently technically fall outside the scope of these policies. The applicability of the policies affects the applicable principles, the actors involved and their responsibilities, and the prescribed assessment methodologies. Currently, both organizations are exploring ways either to integrate simpler rule-based systems into existing algorithm policies or to develop additional policies tailored to these systems.

Communication and inclusion

Regarding communication and inclusion, the findings indicate that assessments play a crucial role in responding to the regulatory call to action, by bringing actors together to identify and mitigate risks of discrimination in profiling systems. In particular, DPIAs and quick scans, ethical assessments, and the FRAIA were identified as offering differing but complementary lenses to consider discrimination. Both agencies make choices regarding the assessment methodologies employed, the structure and frequency of assessments, and the actors involved. Interestingly, currently only assessments of trained systems included both internal and external actors, whereas rule-based system assessments engage only internal actors.

The combination of assessments and actors highlights the symbioses between fundamental rights, data protection legislation, ethics, and interdisciplinary dialogue in identifying and mitigating risks of discrimination and justifying risk-related decisions. However, for simpler systems, this dialogue within assessments is currently limited to internal actors, with the underlying rationale for this not being explicit. For these simpler systems, the question then arises whether external perspectives should be included in assessments of these profiles and what their role could be, or if it is sufficient to include them in reflective discussions regarding responsible risk profiling. Here, assessment methodologies such as the FRAIA and the example of Agency 2 potentially involving clients in the ethical deliberation could prove insightful regarding the value of their inclusion and their role.

Integration

Regarding integration, actors collectively discuss and answer questions in assessments relevant to discrimination. Typically, for simpler systems, prevention and enforcement actors initially play a significant role in both organizations in identifying risks and mitigation measures, with other actors playing consulted/supporting, reviewing, or approving roles, although the exact roles of the actors and the relevant process descriptions, protocols, and embedded assessment methodologies are currently still somewhat fluid. For trained systems, however, the roles and responsibilities of various actors in assessments are established in algorithm policies and in the various sections of the FRAIA that dictate the expertise and roles needed to answer specific questions. This difference in the maturity of roles and assessments between types of system is noteworthy, and seems to reflect the recent public and political focus on trained systems.

During assessments, ambiguity was noted regarding when certain selection criteria in risk profiles can lead to indirect discrimination. The fact that bias tests are not always feasible due to legal limitations on data for bias testing further complicated this issue, as reported. Even when such tests can be conducted, uncertainty remains about when bias should be considered unacceptable. This struggle highlights the socio-legal ambiguity of discrimination in algorithmic profiling and the need for complementary lenses in assessments and interdisciplinary dialogue to support well-reasoned decisions on risk profiles.

Reflection

Finally, reflection on risk governance was firstly seen in periodic evaluations of assessment methodologies, landscapes, and compliance with mitigation measures. Both agencies emphasized the need to experiment with assessment methodologies and to later reflect on their content and integration into organizational processes and policies, based on practical experiences. They also stressed the need to regularly review whether the measures documented in individual assessments are being followed in practice, in order to maintain oversight of compliance. In addition, the agencies noted the value of evaluating the overall assessment landscape and whether it serves its intended purpose, as, for example, a high volume of assessments for each profile—though well intended—can increase the workload and complicate risk oversight, ultimately hindering compliance.

Furthermore, the agencies cited engagement with incidents, other organizations, and clients as stimulating broader reflection. Engaging with incidents proved useful, to build a better fundament for risk-based investigations and risk governance. Likewise, engaging with other organizations was noted to be helpful in this regard. However, currently both agencies only engage with other organizations on an ad hoc basis, thereby limiting the structural incorporation of risk governance knowledge stemming from the unique experiences of other organizations addressing similar challenges. Finally, the agencies’ involvement of clients was cited as stimulating reflection; however, this involvement currently focuses primarily on trained systems rather than simpler, rule-based systems. Since risks of discrimination can also arise through the use of these simpler systems, the concerns, experiences and values of potentially affected clients could prove useful.