Algorithms,artificial intelligence and automated decisions concerning workers and the risks of discrimination: the necessary collective governance of data protection

From automatic data handling to automated processing

In the information age, much of the work done by human resources’ (HR) experts consists of gathering as much information as possible about workers in order to improve decision-making (recruitment, promotion, dismissals, part-time/full-time contract, geographical mobility, payment of bonuses, etc.) (Grensing-Pophal, 2009: 42; Sameen and Cornelius, 2013). Indeed, it is crucial for a company to gather and compile as much information as possible in order to gain a thorough understanding of a worker’s skills, knowledge, capabilities, attitudes, etc. so as to be able to make decisions best suiting its interests (Jackson Lewis, 2016).

In the same way that a company wants to know as much as possible about consumers in order to know what product to offer them or what exact advertisement will convince them to buy its products, employers want to gather as much information as they can about their (potential) employees in order to know whether they will be productive, how well they will fit into the company’s environment or what in particular will motivate them to stay or to work harder (Ajunwa et al., 2017; De Stefano, 2018; Moore et al., 2018b).

The common denominator of these situations is the collection of information (by means of online reputations through customer evaluation, wearables, video cameras, etc.). Gathering information to make better decisions is nothing new. ¹ Indeed, for years, companies have been using interviews in selection processes, group dynamics, performance evaluations, etc. to make such work-related decisions. In recent years, however, HR experts have concentrated on gathering information via new technologies (Facebook, Linkedin and now online reputation) (Ouridi et al., 2016: 240–249; Wolf et al., 2014). While it is of course true that technology potentially allows companies to access larger amounts of data in a very economical way (Daws, 2016), an HR manager was always the person who, once that information had been gathered, had to process it and arrive at a decision. This meant that, despite the increasing amount of information available, there was a natural limit to using that information, namely, the human capacity to process such data.

However, the latest technologies are changing even this. Thanks to algorithms, big data and artificial intelligence, not only is there a reduction in the cost of access to information (that which until now was available thanks to Facebook and Linkedin and other public data), but there is also an unprecedented reduction in the cost of processing this information to make it useful, in turn facilitating decision-making based on this information (automated decisions).

The use of new technologies to assess and monitor workers has thus fundamentally changed i) how information is collected and from which sources; ii) how that information is processed; and iii) how decisions are made.

Likewise, digital reputation systems (customer ratings) make it possible to obtain information about employees’ behaviour in a much cheaper way (Thierer et al., 2015: 7). Employers are even beginning to measure workers' emotions (Moore, 2018a: 18).

In the case of wearables, having an HR employee monitoring the heart rate of all workers (or their location if done by means of GPS) would be excessively expensive and, therefore, impracticable. However, by means of automated systems (and algorithms) it is possible, and very inexpensive, to set up alarms informing an HR manager when a worker is inactive for a long period. This means that there is no longer any need for an HR manager to monitor the information or perform surveillance tasks. Instead the manager will simply be automatically ‘alerted’ when a situation warrants observation.

Some companies in the USA are developing devices fitted with microphones, not with the intention of recording workers’ conversations, but to know the worker’s mood according to his or her tone of voice. This device can also be used to measure a worker’s interactions with colleagues in order to know which of them they interact with and for how long (the Week Staff, 2015). ²

By the same token, in the case of online reputation, analysing and systematising information and evaluations collected from customers can be excessively costly, while a computerised rating system allows information to be categorised and averages and alerts to be obtained when a worker’s behaviour deviates from the acceptable standards. Reducing the need for the interaction of an HR manager obviously gives rise to more economical monitoring methods.

The simplified level basically consists of automating the process in question (promotion, bonus payments or dismissals) by establishing a command in a computer programme (if X happens, perform Y). Hence, it would be possible to develop an automated process such that, if a worker’s activity (measured by heart rate) decreases for more than three hours, an email is automatically sent with a letter of dismissal. ³ According to the inspection report of the Labour Inspectorate of Valencia, if a Deliveroo rider is not in motion (detected by GPS), he or she automatically receives a warning message, being told to get moving again ⁴ (as a ‘mental whip’ (Moore, 2018a: 23)). Or, for example, if a worker’s average online reputation – customer ratings – drops below 4.6 out of 5, he or she is ‘automatically’ disconnected from the platform ⁵ (or prevented from entering the workplace by automatically deactivating his or her credentials).

A more complex system would use artificial intelligence (AI). Determined by a firm’s programming decisions, true artificial intelligence could, of course, take many more factors into account when making a decision to promote, dismiss, etc. one of the company’s employees.

In short, the reduced cost of these three levels allows companies easily to step up the monitoring of workers: the cheaper monitoring is, the more measures an employer will take to protect its legitimate business interests. At present, European labour legislation grants employers the power to choose which forms of worker surveillance and monitoring they deem appropriate. However, these regulations were enacted at a time when surveillance and monitoring were limited by their very nature – in short, because they were expensive.

In contrast to what is commonly believed, technology rarely enables monitoring that was not previously possible in any shape or form. What technology does allow is for monitoring to be performed at a lower cost. Similar to worker monitoring, installing several video cameras in a shop has the same effect as having a greater number of security guards. Video cameras allow the same monitoring to be performed in a more efficient and cheaper way. Hence, with the appearance of these new technologies and the lower cost of monitoring, there is now a need to reconsider this unilateral employer power.

While we have no intention whatsoever of questioning the legitimacy of an employer to be able to monitor the work carried out by its employees, the starting hypothesis of this article is that cheaper monitoring and new methods may give rise to unjustified or abusive interference in workers’ fundamental rights and freedoms, in principle justifying legal censorship. In the face of these risks, we advocate the collective governance of workers’ data as a way of minimising the discrimination risks and potential violations of fundamental rights at work.

Scope of the specific protections: decisions based solely on automated processes

Technological improvements to workforce-related decision-making are not limited to increasing the capacity to accumulate and process that information, but also allow some HR tasks previously done by an HR officer to be completely automated. Indeed, artificial intelligence, or machine learning, now allows fully automated decision-making without any human intervention (or only to a minimal extent). The European legislator, in view of this advance in technology, considers that special and specific safeguards are needed to protect citizens from automated decision-making. In this sense, it is understood that, as the degree of automation rises, not only is there a greater risk of discrimination or bias, but also that the more parts of the process are automated, the more control there will be (as it becomes cheaper).

Thus, when a company’s decisions are automated, the protections set out in GDPR Article 22 will be applied. These protections will be set out in the following. However, first of all an analysis is necessary to determine in which cases a decision is made based solely on an automated process.

As technology improves, there will be more possibilities for artificial intelligence to take autonomous action. One can already envisage selection processes in which filters are automatically activated by grades found in academic records or by the university where a degree was obtained, with those applicants not meeting the requirements being screened out and never reaching the head of HR’s ‘desk’ for further analysis. This kind of automatic filtering is in itself an automated decision (denial of employment). Analysing other hypothetical situations, however, may be more complicated. Think, for example, about the dismissal of an employee because his digital reputation is less than 4.6 out of 5 (for further debate about the digital reputation of workers, see Todolí-Signes, 2019). If in this case the dismissal is automated, it will come under the scope of GDPR Article 22. However, the dismissal may not have been enacted directly by the computer system, but by an HR officer receiving an ‘alert’ when a worker’s rating drops below 4.6. Would this latter case be an automated decision?

A literal interpretation of Article 22 would lead us to think that it is not, as it establishes special protections for decisions based solely on automated processes. As some authors have posited (Wachter et al., 2017a: 92), any level of human intervention, however trivial, would render these protections inapplicable. However, the bulk of the literature seems to be inclined to understand that any type of human intervention would not be sufficient to impede the application of these safeguards (Selbst and Powles, 2017; Veale and Edwards, 2017). In fact, when human intervention is limited to applying the decision taken by the algorithm, we are still dealing with a decision taken solely by an automated system. Hence, the right to control decisions based on an automated process is upheld, provided that human intervention is limited to applying the decision taken by the computer system without any influence on the outcome.

The following – more philosophical – approach can be taken to support this statement. An algorithm does not have any real will; thus, an algorithm does not take decisions, but produces outcomes. In this regard, it is always a human taking the decisions: either the person programming the algorithm or the one applying the algorithm´s outcome. According to this interpretation, mere human intervention cannot rule out the application of these protections since, given the necessary human intervention, these would be without content.

In my opinion, only when the head of HR has the authority to change the outcome produced by a computer system through assessing different aspects can it be understood that there is significant human intervention. ¹¹ In this regard, in order to know whether the level of human intervention is ‘significant’, it will be necessary to assess how often an HR officer takes final decisions in a direction other than that produced by the algorithm or artificial intelligence.

Furthermore, it should be taken into consideration that partial decisions are also subject to these protections if they are automated; i.e., despite the fact that the final decision as to who is hired or who receives a bonus or promotion in a company is made by the head of HR with ‘significant’ intervention, if use has been made of automated systems that have discarded subjects in order to reduce the number of candidates, a denying automated decision will have been taken on them (WP29, 2017: 23). An automated decision is also understood as one sorting workers into categories or profiles, even though an HR officer subsequently takes the final decision (Edwards and Veale, 2017: 46; WP29, 2018a: 23). Indeed, if the computer system classifies workers into categories (e.g. A, B, C, D or assigns them scores from 1 to 5) and the head of HR subsequently decides to give promotion to those in a particular category, the decision, even if made by a human being, would be based on a previous decision made by a machine, thereby falling within the scope of the GDPR Article 22 on automated decisions.

Prohibiting automated business decision-making

The first form of protection in the case of decisions based solely on automated data processing is to prohibit them (right to object). Article 22 of the GDPR establishes the right for a person ‘not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her’. The WP29 (2018a: 23) interprets this right as prohibiting (without any need actively to claim the right) data controllers from making decisions with this automated methodology. Nevertheless, this prohibition is only relative as there are exceptions in GDPR Article 22, namely: ‘the decision i) is necessary for entering into, or performance of, a contract between the data subject and a data controller; ii) is authorised by Union or Member State law to which the controller is subject and which also lays down suitable measures to safeguard the data subject’s rights and freedoms and legitimate interests; or iii) is based on the data subject’s explicit consent.’ Given that, in an employment contract, a worker’s consent will generally not be valid ¹² that means that, in the absence of an internal regulation, the automated decision can only be considered valid if it is necessary for the conclusion or performance of the employment contract.

According to the WP29 (2017), in order to understand whether automated decision-making is ‘necessary’, the data controller must prove that it is the most appropriate way to fulfil the objective of the contract. In fact, the interpretation that can be drawn is that human intervention must be shown to be impractical or impossible because of the amount of data processed, i.e., the company must demonstrate that there are no other less intrusive ways to achieve the same aim. Specifically, the WP29 (2017: 23) establishes that it would be valid to use automated systems in the event that a company, in the case of a job offer, receives tens of thousands of applications. In that case, it would be impracticable for the company to carry out the selection process without first discarding some of the candidates by means of automated processes. However, once the list of candidates has been reduced to manageable numbers, automated decision-making would have to cease.

Accordingly, it would not seem possible to make automated decisions (or decisions based on automatically created profiles) regarding promotion or bonuses and, even less so, concerning dismissals without significant human intervention. ¹³

‘Right to an explanation’

If one of the exceptions to the prohibition applies (e.g. the need to fulfil the contract), Article 22(3) of the GDPR requires a company, as the ‘data controller’, to ‘implement suitable measures to safeguard the data subject’s rights and freedoms and legitimate interests, at least the right to obtain human intervention on the part of the controller, to express his or her point of view and to contest the decision’.

It appears, under this Article, that the data controller has the obligation to inform the data subject of the reasons that led to that decision, i.e., when a company automates decision-making, it must indicate that it has done so and provide details as to what parameters it has used to reach the decision in question (and what weighting it has assigned to each of them). This interpretation is supported by Article 5 of the GDPR, which requires the processing of personal data to be lawful, fair and transparent, and also by Articles 13(2(f) and 14(2)(g), requiring that when the subject is involved in automated decisions, including profiling, the data controller must provide the subject with ‘meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing for the data subject’.

Indeed, it seems that the doctrine unanimously interprets this as meaning that the data subject has the right to receive information on the automated processing of his or her data in three aspects: i) to be informed that he or she is involved in an automated decision-making process, i.e., inform the worker that the HR process will be fully or partially automated; ii) to be provided with meaningful information on the logic of the algorithm, i.e., among other things, indicating the parameters evaluated by the algorithm making the decision and their weighting; and iii) to be informed about the consequences of the process, i.e., what consequences the automatically taken decision will have for the worker, in one sense or another. ¹⁴

Yet, the doctrine is not unanimous in interpreting the necessary extent of the explanation. While a few authors establish that the obligation is limited to requiring an ex ante (i.e., prior to the decision being made) and general description of the data supplied to the algorithm in order for it to make the decision ¹⁵ , the majority considers that GDPR Article 22(3), together with Articles 13(2)(f) and 14(2)(g) and Recital 71, requires an ex post (i.e., after the decision has been taken) and specific explanation of how and why that decision has been taken with regard to that particular worker. ¹⁶

The response given to the controversy is not trivial, as the minority interpretation would fulfil the mandate simply by providing generic information on the three aspects mentioned above, while in the second case it would be necessary to explain how the algorithm processes the data in order to reach its conclusions and how that decision has been reached for the specific data subject.

In my opinion, there are two arguments leading us to opt for the second stance: a literal interpretation and a finalistic interpretation of the precept.

Thus, Articles 13(2)(f) and 14(2)(g) literally require the data controller to provide a data subject with specific and easily accessible information on the automated decision-making concerning him or her, including profiling. Furthermore, Recital 71 expressly requires that automated decisions be ‘subject to appropriate safeguards, which should include specific information to the data subject and the right to obtain human intervention, to express his or her point of view, to obtain an explanation of the decision reached after such assessment and to challenge the decision’.

This interpretation is also in line with the objective of this prescription ¹⁷ , which is to avoid biases in an algorithm over which a data subject has no control and thus no knowledge about the reasons behind a decision. For this reason, it appears that the degree of ‘explanation’ given by a company regarding the reasons for taking such a decision should be sufficient to enable the data subject to exercise the corresponding rights to ‘express his or her point of view and to challenge the decision’. This implies that a generic explanation would be insufficient to avoid the defencelessness of a worker who is denied employment or dismissed on the basis of an automated decision. As the WP29 (2017: 25) says, the explanation must be sufficiently comprehensible for the data subject to understand the reasons for the decision.

In short, in my view, in those few cases where it is deemed necessary to use automated decision-making procedures, a worker should have the right to receive a specific explanation after the decision had been taken. In addition, this explanation should be sufficient to understand the reasons leading to the decision.

Towards collective governance of data protection rights by workers’ representatives?

While the progress made in this area is welcomed, there is still a lack of real collective ‘governance’ of workers’ data protection rights. While the GDPR lays down a number of very strict obligations on transparency and data protection safeguards – also for workers’ data –, such safeguards may always be decided unilaterally by the employer without the GDPR giving trade unions the power collectively to monitor the use of such data by the company within employment relationships. Indeed, it only provides for the exercise of individual rights by a third party. Nonetheless, national implementations of the GDPR stipulations could require that such safeguarding instruments for the protection of workers’ data be negotiated with workers’ representatives, or that the latter are at least consulted, a provision which could be called the collective governance of data protection. In fact, Article 88 calls for national regulations to establish more rights-based safeguards for the protection of workers’ data or for such protection to be provided by collective agreements.

With regard to the protection of consumer data, some authors have stressed the importance of empowering agencies, non-governmental organisations and civil society to ensure that there is no discrimination or bias in decisions made on the basis of big data (Edwards and Veale, 2017: 23). In other words, the GDPR should not simply allow such decisions to be monitored externally on behalf of individuals, but it should also be possible to analyse the legality and legitimacy of the actions undertaken by companies on citizens’ data from the outset. Translated into the framework of labour relations, this, in my opinion, entails the need for workers’ representatives and trade unions to have the power not only to exercise certain rights on behalf of workers, but also to be able to verify from within the use made of workers’ information and to ensure that, in selection, evaluation and dismissal processes, the information used has been obtained and processed lawfully (what we have called ‘data protection governance’).

One of the cross-cutting issues in the new data protection legislation is the obligation to establish ex officio safeguards to protect data subjects against automated data processing. It imposes an obligation of result (the protection of the right to data protection), leaving it up to the data controller to choose the formula (the methods and the safeguards) to fulfil that objective. This formulation – the unilateral power on the part of the data controller to choose the safeguards – makes sense in the consumer field because of the lack of ‘representativeness’ of consumer associations. Precisely the lack of interlocutors with consumers seems to prevent the collective governance of consumer data protection. ¹⁸ In the world of labour relations, however, it would perhaps make more sense if the methods and safeguards to protect the fundamental right to data protection were agreed with the trade unions. Indeed, as set out in Article 88, the establishment of safeguards for the protection of workers’ data may be the subject of collective bargaining.

On the other hand, if nothing is said in collective bargaining about the methods and mechanisms of workers’ protection with regard to data protection, the rights to information and access in favour of workers’ representatives will continue to exist (Article 27 of the Charter of Fundamental Rights of the European Union). ¹⁹ The works council will therefore have the right to be informed and consulted on any business decision affecting the processing of data or any kind of monitoring of workers. ²⁰

The rights to ‘information and access’ granted by The Charter of Fundamental Rights of the European Union (CFREU) Article 27 have the precise aim to restrain corporate powers. Thus, in the face of a regulation (GDPR) obliging a company to establish limits to its corporate powers (safeguards for the benefit of workers), it would be pointless to prevent workers’ representatives from participating in the establishment of such limits – even in their weakest form of participation, as is the case of ‘information and access’. This right to receive information should include access to all information on technical procedures and company use of a worker’s data, including algorithm parameters and its consequences, as described above in the ‘right to an explanation’ section.

However, we consider this form of workers’ participation to be insufficient. The possibilities for the surveillance and monitoring of workers have multiplied as a result of the technological capabilities of information collection. For this reason, faced with the implementation or revision of systems for the organisation and monitoring of work in the future, these legal powers of information and access seem far from sufficient to achieve the desired collective governance of data protection.

A proposal for a joint data protection committee

Hence, in view of this technological change, a regulation is proposed to govern all that information about the worker’s behaviour, performance, attitudes, personality traits, etc. and the decisions taken by a company based on these vast amounts of information collected and subsequently processed.

In regard to our proposal, we find that, in order to allow the collective governance of data protection, it is necessary to introduce, among a union’s statutory powers, certain rights that already exist in the field of occupational risk prevention, such as the right of proposal (Article 11(1) of the ‘Framework’ Council Directive 89/391/EEC) ²¹ , or the existence of a genuine joint data protection committee (analogous to the existing health and safety committee on occupational risk prevention). Prerogatives that exist to govern workplace health and safety should be implemented in the field of data protection at work too.

Finally, in the absence of a specific regulation in law (Ajunwa, 2017: 102), it will be up to collective bargaining to establish instruments that ensure the participation of workers in the choice of measures and safeguards to be imposed to protect their data. ²² Regulation in this field must remain flexible and highly adaptable to technological change. That is why, in my opinion, it would be appropriate, de lege ferenda, to have a basic legislation making it compulsory to negotiate these extremes (the collective governance of these instruments for processing workers’ information), thereby allowing for collective negotiation to specify the necessary safeguards and limits to protect workers’ fundamental rights (De Stefano, 2018).