Legal challenges in bringing AI evidence to the criminal courtroom

AI in law enforcement

In recent years, different fields have been experiencing the transformative power of Artificial Intelligence (AI) ¹ exponentially, and the criminal justice system is not an exception. Notably, AI can support criminal justice practitioners in examining vast amounts of data, thus creating new opportunities to investigate and prosecute crimes. In fact, that is already the case for many Law Enforcement Agencies (LEAs) that rely on AI-driven tools to gather data (which eventually turn into evidence) or to scrutinise previously collected evidence – illustrated by many examples across Europe.

An evidence-recognising tool based on AI, for instance, is used by Spanish and German LEAs to search for clues of child sexual abuse within images, assisting them in the detection of faces, objects, sexual organs and other information which could be indicative of child sexual abuse material in digital devices. ² Such a tool could also help process information of a crime scene by helping detect hints in pictures that may otherwise have been missed by investigating officers. Similarly, police forces in the United Kingdom use an AI forensic tool to examine mobile phones seized during criminal investigations to search for potential evidence. ³ That software can interpret images, analyse communication patterns, match faces and cross-reference data from different devices – functionalities that can help connect the dots when investigating and prosecuting organised crime, for example. A more concrete case in point is the use of an AI-driven system by Europol ⁴ to process tons of data, facilitating a joint LEA operation. The evidence gathered and processed through AI helped dismantle encrypted criminal networks, leading to hundreds of arrests across Europe. ⁵ Only a year later, a similar operation was performed in Belgium to dismantle an encrypted network. ⁶

The possibility of expeditiously processing an astonishing amount of data is an AI feature which brings many opportunities for criminal justice activities given the significant growth of evidentiary material in the ‘big data’ era. ⁷ AI also offers the possibility to uncover hidden insights that the human eye could hardly detect, correlating multiple data points and matching different pieces of evidence. Yet, the vast promise of AI also comes with significant risks and concerns for fair trial rights in criminal proceedings. This paper seeks to elucidate the issues at stake; mainly how AI evidence could entail potential threats to due process guarantees in Europe.

Scope and structure

This contribution concerns evidence generated by or processed through AI-driven systems, usually resulting from the deployment of technologies specifically designed for use in a law enforcement context. ⁸ Although at its core AI evidence is evidence in digital form, features of AI (particularly its inherent opaque and automated nature ⁹ ) add a layer of complexity to traditional digital forensics discussions. This complexity is precisely what this article attempts to address. ¹⁰

Currently, there is no concrete regulatory framework in the European acquis or ruling by the European Court of Human Rights (ECtHR) specifically on AI in law enforcement. ¹¹ As Europe initiates AI regulation efforts, ¹² it is important to reflect upon the specific challenges that AI poses in the context of European criminal law. The message conveyed by a growing body of legal literature is that AI brings forth novel questions regarding fair trial guarantees. ¹³ This article strives to reach beyond these supported claims by focussing on the introduction of AI evidence in court and by envisaging how existing case law on due process principles may apply to such evidence. The overall aim is to examine if and how AI evidence challenges (in a novel way – in comparison to digital evidence from less intrusive and complex systems, for example) the core tenets of the criminal procedure, in an attempt to demonstrate potential gaps in this framework.

In terms of applicable legal framework, the paper focuses on the European legal order. ¹⁴ While criminal procedure remains one of the least harmonised fields of law within Europe, rules and principles on the handling and production of evidence have been formulated, primarily on the basis of European human rights law. In this way, this paper primarily examines the applicability of the ECtHR case law on evidentiary considerations and the overall fairness of criminal proceedings upon AI systems for law enforcement. ¹⁵ On a secondary and complementary level, this paper also touches upon EU legal instruments relevant for the handling of evidence in the algorithmic realm. ¹⁶ The most crucial EU legal instrument in this context is the so-called Law Enforcement Directive (LED) on the processing of personal data within the context of criminal justice. ¹⁷ The latter lays down a framework of data protection principles, rights and obligations applicable to processing activities for law enforcement purposes, excluding national security-related tasks. ¹⁸

The analysis is structured in four sections that tackle the main categories of criminal procedure rules in the European criminal law landscape: admissibility, reliability, challenging by the defendant and evaluation by the judge. Then, the final section offers possible solutions to address these challenges, in an attempt to assist in the effective guarantee of fair trial rights in the criminal justice sector.

Principles

In most jurisdictions, evidence should in principle be lawfully obtained in order to be admissible before national courts. In this regard, it is important to highlight the role of AI when ensuring the lawful collection of evidence. This section first presents the general principles on evidence admissibility within the European legal order, to then focus on ECtHR case law on the admissibility of evidence collected through new technologies.

Evidence is considered unlawfully obtained when it is illegally or improperly collected, in violation of a wide range of legal frameworks applicable during the collection of evidence. Unlawfully obtained evidence may be excluded from criminal proceedings pursuant to the so-called ‘exclusionary’ or ‘fruit of the poisonous tree’ principle, which is similarly shared amongst continental and common law jurisdictions. ¹⁹ Exceptions to the exclusionary principle are often foreseen. ²⁰ For example, several national legal orders accept the admissibility of unlawfully obtained evidence when there is no other evidence to support the conviction of the accused. ²¹ Depending on the legal regime of each state, judges enjoy a narrower or broader margin of discretion in deciding the admissibility of such ‘spoiled’ evidence. ²²

When it comes to the collection of digital evidence, several criminal procedural and other legal rules must be abided by, while protection of human rights on the basis of domestic law and the ECHR must be ensured, as also established by the Budapest Cybercrime Convention. ²³ Most notable examples include violations to fundamental rights – for instance, to life, prohibition of torture and inhuman or degrading treatments, to a fair trail and to privacy and data protection. This contribution focuses on the latter two.

Regarding fair trial, the ECtHR considers itself lacking the competence to rule on the admissibility of specific types of evidence based on Article 6 ECHR given the absence of rules focused on evidence therein. ²⁴ Nevertheless, the ECtHR assessment of the proceedings’ fairness encompasses questions such as whether the fair trial conditions established in Article 6 ECHR are complied with, or whether any other ECHR right (for instance, the right to privacy or to effective remedy) has been violated during the process of evidence collection. ²⁵ It is important to note, however, that a violation of other ECHR rights does not necessarily lead to a violation of the right to a fair trial. ²⁶ The right to a fair trial can be violated if, for example, the defendant was not able to challenge the authenticity and use of the unlawfully obtained evidence, especially when that evidence comprised the sole or decisive element for conviction (see also the Evaluation by the Judge section). ²⁷

Although the ECtHR scrutiny of evidence obtainment in relation to Article 6 ECHR is limited, violations of the right to privacy under Article 8 ECHR during the collection of evidence have been found in numerous cases due to a lack of lawfulness or proportionality, as detailed below. In this regard, compliance with data protection rules and principles for evidence to be considered as lawfully obtained is crucial. Pursuant to Convention 108+ and the LED, for instance, personal data processed by LEAs must be collected for ‘specified, explicit and legitimate purposes’ ²⁸ and ‘to the extent that processing is necessary for the performance of a task’. ²⁹

When investigating privacy violations in cases on the use of surveillance systems and deriving evidence, the ECtHR often starts by examining the purpose for which the technology is built. ³⁰ In other words, the Court distinguishes between systems that are designed and deployed specifically for the purpose of security and surveillance, and other systems that may also be used for surveillance without that being their primary purpose. When a technological system, such as a CCTV camera, is built with a surveillance purpose and its deployment is regulated by law, focus should be put on the alignment with the security purpose and conditions of use established therein. According to the ECtHR, the use of cameras, listening or geolocation devices or audio/video files in a manner that was non-compliant with the procedures set out in the legal instrument regulating the installation and use of the surveillance system, or going beyond the regulated expected use and purpose which the technology ought to serve, violates the right to privacy. ³¹ In the absence of a legal instrument regulating the use of a specific technology or bundle of technologies for the purpose of obtaining evidence or clarifying the scope of police discretion in employing such technology, then such use alone may violate the right to privacy due to the lack of foreseeability. ³²

When information is to be extracted from devices or environments that do not fall under the category of purpose-built surveillance, such as citizens’ private smartphones or electronic communications, the right to privacy under Article 8 ECHR and applicable data protection frameworks ³³ should be respected to ensure that evidence is lawfully obtained. ³⁴ In this regard, the ECtHR commonly examines whether appropriate and sufficient safeguards against possible abuses are in place. ³⁵ Conditions may differ depending on whether the LEA extracts data directly from the citizens themselves or indirectly from a company providing citizens with services, in a targeted or untargeted manner, based on a tailored law, such as a data retention law or an individual warrant. If access relies on a legal framework providing for electronic surveillance, then, according to the ECtHR jurisprudence, it should be sufficiently clear to give citizens an adequate indication as to the circumstances in which and the conditions under which security agents are empowered to access their data, as well as provide for prior (often judicial) authorisation. ³⁶ Evidence obtained in breach of, for example, EU law, should be disregarded by the courts where the defendants ‘are not in a position to comment effectively on that information and that evidence and they pertain to a field of which the judges have no knowledge and are likely to have a preponderant influence on the findings of fact’. ³⁷

Unlawfully obtained evidence and AI

AI designed for law enforcement use should be considered as a purpose-built technology or bundle of technologies, since it is deployed specifically for crime prevention, detection, investigation and prosecution. Nevertheless, it may not always be the case that AI functionalities are specifically regulated. Currently, the development of AI regulation is in talks within Europe, ³⁸ while calls for specific AI regulation have also been made – for instance, in relation to facial recognition and biometric surveillance. ³⁹ In the absence of concrete legislation, determining whether the deployment of law enforcement AI systems remains respectful to their purpose and regulatory framework may not be as straightforward.

Identifying unlawfully obtained evidence and applying the exclusionary principle in the algorithmic context may prove equally challenging, as it requires an unambiguous application of relevant legal frameworks, and a clear definition of what it means to obtain evidence when AI functionalities are employed. Problems may occur already when determining whether AI evidence complies with or violates applicable laws. For example, assessing violations to non-discrimination or data protection rules by AI tools at large is a well-documented equivocal task. ⁴⁰ Lawfulness may be affected during the design phase; for instance, the use of discriminatory or biased datasets to train the algorithms could guide officers towards the biased collection of data or the biased profiling and identification of new suspects. In that case, it is unclear how discrimination law applies in the first place, ⁴¹ and whether such an outcome could be considered as unlawfully obtained evidence. Lawfulness may further depend on the conditions of initial datasets’ collection, the process of datasets’ ingestion to the AI system and further use, and their adherence to the right to privacy and personal data protection principles like transparency and purpose limitation. ⁴² The lawfulness of data collection may be dubious when, for instance, police hacking is involved ⁴³ or due to the opacity of the AI system generating evidence. Similarly, discerning the exact purpose of each AI system and whether any reusing of datasets for different criminal cases complies with the purpose limitation principle and its permissible restrictions may necessitate a case-by-case analysis, creating a degree of uncertainty. ⁴⁴ Finally, given that unlawfully obtained evidence is not necessarily inadmissible, assessing the weight such violations hold on the overall fairness of the trial may be particularly difficult for judges, as further discussed under the Evaluation by the Judge section.

Principles

Whereas it is primarily a task for the national judge to assess the reliability of the evidence, the ECtHR has set out some requirements in terms of reliability and accuracy of evidence. ⁴⁵ For evidence to be considered reliable, there must be no doubt over its authenticity and integrity. Authenticity refers to the irrefutable provenance of the evidence. In other words, LEAs must be able to establish the source of the evidence. ⁴⁶ The notion of integrity refers to the fact that evidence remains intact and that it is not tampered with during its collection or subsequent handling by LEAs. ⁴⁷ The ECtHR also pays attention to whether the defendant had the opportunity to challenge both the authenticity and the use of the contested evidence (infra). ⁴⁸ Finally, reliability strongly correlates with the weight of evidence in a criminal trial. ⁴⁹

The ECtHR regularly applies the above-mentioned principles. In Khodorskovskiy and Lebedev v Russia, for instance, the defendants alleged that LEAs had planted evidence on the crime scene, as some of the seized materials had only been added to the criminal file after the closure of the investigation. Moreover, the Russian LEAs had copied information from a hard disk and presented it to the trial judge on paper. The defendants claimed that the authorities had presented more evidence to the court than had originally been seized, as the servers had not been properly locked up, and that the evidence was not reliable. In this case, the ECtHR decided that the use of the evidence did not breach Article 6 ECHR, as the defendants had had the opportunity to interrogate witnesses that were present at the time of the search and seizure operation. ⁵⁰ In addition, the ECtHR could not detect any manifest flaw in the process of seizing and examining the hard drives, which would make the information obtained unreliable. ⁵¹ This case shows that the ECtHR only considers obtained evidence as unreliable if there are manifest flaws to be detected in its processing. National courts thus have a considerable margin of appreciation when assessing the reliability of evidence, which boils down to a case-by-case approach.

In addition to the general principles stemming from the case law of the ECtHR, both Article 5(4) Convention 108+ and Article 4(1)(d) LED contain a data quality requirement relating to the reliability of evidence as well. Those provisions state that the processing of personal data shall take place in an accurate manner, ensuring also that the personal data being processed are up to date. Consequently, LEAs that are considered as data controllers when using advanced data-driven analytics have to verify and maintain the quality of the personal data they process in the fulfilment of their duties.

Problems regarding and means towards reliable AI evidence

Prior to presenting it before a criminal court, it should be ensured that the obtained evidence is reliable. ⁵² The reliability of AI-generated evidence can be affected at three levels. First, the authenticity and integrity of the ‘raw’ digital data can be at stake. During a criminal investigation, information is obtained from different data sources, including persons such as witnesses or informants. When testifying, individuals might have various degrees of reliability or trustworthiness based on their motives. For example, ideological reasons may drive an informant to provide inaccurate information or even fabricate facts. Furthermore, information provided by a human being is more prone to be speculative and may be subject to memory faults. ⁵³ The processing of such information can yield unreliable algorithmic predictions.

The authenticity and integrity of digital data becomes all the more important when it is introduced in data-driven tools (such as forensic tools underpinned by AI technology). Reliability could arguably be improved by automating the process of examining large datasets. Automation could enhance the accuracy of results that are not subject to the cognitive biases that humans tend to be prone to in decision-making processes. ⁵⁴ However, the results of the data-driven analysis inevitably reproduce the quality of the input data. If the input data are not accurate, complete and up to date, the outcomes of the system will be flawed and inaccurate. ⁵⁵ Poor quality of data can lead to false positive or false negative results and thus wrong, biased or erroneous evidence, potentially resulting in wrongful accusations or even convictions. For that reason, both Article 5(4) Convention 108+ and Article 4(1)(d) LED proclaim the principle of accuracy (supra). In other words, the higher the reliability of the input data, the more reliable the investigative outcomes of analytic tools.

Second, AI tools can contain miscodes that could remain hidden because of the opacity issue (also known as the ‘black box’ problem), which is one of the most significant hurdles discussed in AI literature. ⁵⁶ It refers to the fact that the inner operations of most AI systems are not visible to humans, raising questions as to whether their processes can be understood at all. This problem might affect the reliability of the AI evidence in the processing of the ‘raw data’ by AI tools. ⁵⁷ Third, human experts who are operating AI tools can jeopardise the reliability of AI evidence. A study showed low reliability between digital forensics experts examining the same evidence file with the same contextual information, in their observations, interpretations and conclusions. ⁵⁸ There is a similar risk when it comes to human experts operating AI tools.

How can those risks affecting the reliability of AI evidence be prevented and/or addressed? An important notion in this regard is the chain of custody, which refers to the process of maintaining and documenting how the (digital) evidence has been handled throughout a criminal investigation. ⁵⁹ The chain of custody can be defined as ‘the chronological documentation of evidence as it is processed during the investigation (i.e., seizure, custody, transfer, and analysis)’. ⁶⁰ From the moment the chain of custody has been breached, evidence can no longer be considered reliable. Therefore, great attention should be paid to the chain of custody when collecting and processing AI evidence, despite the lack of enforceable European rules on this topic. Many guidelines, however, exist at the European level, some of them being only accessible for LEAs. ⁶¹ In the following analysis, we make a distinction between the collection of ‘raw’ digital data and the subsequent processing of this data by AI tools.

First, when it comes to ‘raw data’, respecting the chain of custody implies that the source of the material obtained is recorded, as well as all elements (both human and digital) that came into contact with the evidence, including all relevant dates and times. ⁶² This can be summarised in a few questions: who, when, how and why? ⁶³ The chain of custody documentation starts at the crime scene, where forensics investigators conduct a careful study of the event or place and collect the pieces of evidence found on site. ⁶⁴ Accordingly, the first stage of the chain of custody starts with the records kept by the end-users of law enforcement AI tools when collecting evidence. Those records should include a detailed account of issues such as location, time and date of evidence recovery, as well as a description of each evidence item. One of the methods that is frequently being used in order to establish the authenticity and integrity of digital files is ‘hashing’. A unique code (or calculation) is created at the moment of collection of the evidence. When the outcome of the calculation is not the same at the moment the evidence is presented to a court, this proves that it has been tampered with in the meantime. ⁶⁵

Finally, as explained above, LEAs are not always responsible for the quality of data they collect. Therefore, the technical design of analytic tools should incorporate a distinction between facts and purely factual statements on the one hand and statements containing assumptions or personal opinions on the other. ⁶⁶

Second, the chain of custody remains important when the collected evidence is subsequently subject to analysing tools. It should be possible for LEAs using those tools to obtain records that prove the irrefutable provenance and integrity of the large quantities of processed data, in case the results of the analytics are questioned in court. The party that presents the AI evidence before a court carries the burden of proof over its authenticity and integrity after all. For this reason, AI evidence must be based on precise and scientifically proven methods. The reliability has to be proven for the scientific methods as well as their correct use. As a result, evidence can be brought forward by describing the results of the analyses, the logic of the process, or through expert testimony. ⁶⁷ Also, transparency on the contextual information that experts operating AI tools have received beforehand could improve the reliability of the AI evidence. ⁶⁸ However, a combination of these approaches would generally increase the reliability of AI evidence in court.

Principles

The right to a fair trial under the ECHR comprises the right to have the time and facilities to prepare one’s defence, ⁶⁹ meaning that defendants must have the opportunity to put forward relevant arguments (and evidence, where applicable) such that their views are considered in the trial court’s decision. The fair trial standards in the ECHR also contemplate the right to be presumed innocent until proven guilty ⁷⁰ and the right to participate effectively in a criminal trial. ⁷¹

For the criminal proceedings as a whole to be considered fair, the defendant must be given the opportunity to challenge the admissibility and reliability of the evidence brought against him or her and to oppose its use. ⁷² Doing otherwise could entail a breach of the adversarial principle and render the entire proceedings unfair. ⁷³ In this context, the defendant may petition the court to exclude the evidence that the prosecution obtained unlawfully, as discussed in the Principles subsection in the Admissibility section, or in the case where the chain of custody is broken for any reason, as discussed in the Principles subsection in the Reliability section. Whenever the lawfulness of the evidence adduced in court is challenged, it is for the party bringing that evidence to court to dispel any doubts about it. ⁷⁴

The defendant must be given the opportunity to assess the relevance and weight of the (incriminating) material that may have been considered in the determination of the facts, as well as to formulate relevant comments in that respect. ⁷⁵ In cases where complex questions are posed to the court, the defendant should be given written notice of the evidence to be presented in trial to be able to review the material, assess its authenticity and evaluate its probative value. ⁷⁶

Fair trial guarantees also require that equality of arms between the parties is upheld. ⁷⁷ In cases where expert evidence is adduced, this has at least three consequences. First, favourable expert evidence must be disclosed to the defence ⁷⁸ and must be considered in the proceedings. ⁷⁹ Second, the defendant must be allowed to bring counter-expertise to examine the material evidence ⁸⁰ as otherwise it might be hard to challenge an expert opinion commissioned by the prosecution. Third, if the competent court appoints an expert witness, the designated person must be effectively neutral; otherwise, that expert opinion may lead to a breach of the equality of arms. ⁸¹

Another right underpinning fair trial guarantees is the right to an adversarial process. It implies that, in principle, both parties must have the opportunity to have knowledge of and comment on all evidence adduced or observations filed with a view to influencing the decision of the court. ⁸² This right closely relates to the principle of equality of arms, and thus the ECtHR often finds a violation of Article 6(1) ECHR looking at both concepts jointly. The importance of the adversarial principle in this context has also been highlighted by the CJEU, which stated that ‘[i]f a court takes the view that a party is not in a position to comment effectively on evidence pertaining to a field of which the judges have no knowledge and that is likely to have a preponderant influence on the findings of fact, it must find an infringement of the right to a fair trial and exclude that evidence in order to avoid such an infringement’. ⁸³

Furthermore, the defendant must be granted access to its case file or other documents (such as computer files) ⁸⁴ relevant for the accusations against him or her. Failure to afford such access has led to the finding that the principle of equality of arms had been breached. ⁸⁵ In this regard, the ECtHR has found that, as a general rule, defendants must be given unrestricted access to their case file and unrestricted use of any notes, including copies of relevant documents (where necessary), as important guarantees of a fair trial. However, limitations may apply to the disclosure of evidence due to legitimate competing interests, insofar as the restrictive measures are strictly necessary and are counterbalanced by relevant procedures. ⁸⁶ For instance, when assessing the right to an adversarial trial in cases involving bulk data collected by the prosecution, the ECtHR specified that the defendant does not have a right to access the entire (primary) dataset, but must have the possibility to access the data selected by the public prosecutor (the so-called secondary dataset). ⁸⁷

Challenging AI evidence in court

When applying the above principles to criminal cases involving AI evidence, it is possible to identify many ways in which the introduction of AI may threaten the defendant’s ability to challenge, contest and appeal against decisions made based on automated outputs. Two major areas of concern in this context are (i) the contestability of decisions based on AI outputs; and (ii) the practical capacity of the defendant to challenge AI evidence.

First, opacity is a significant barrier for the exercise of the right to confrontation, given the inability to contest AI evidence put forward by the accusation. Contestability may be considered a crucial element of a rule of law framework, ⁸⁸ which requires the conditions to challenge the actions by government actors to ensure that incorrect, insufficiently supported or arbitrary decisions can be corrected (e.g. the prosecution’s decision to bring someone to the stand based on potentially unreliable evidence). However, the inherent nature of AI compromises the possibilities for defendants and their lawyers to scrutinise and critically assess (incriminating) AI evidence, and to challenge the accuracy and lawfulness of decisions based on such evidence. Put simply, there is an ‘inverse relationship between algorithmic opacity and contestability’. ⁸⁹

The second issue to discuss in this section is the barriers to challenge AI evidence given the defendant’s (lack of) practical capacity to do so. The technical complexity of AI inevitably requires expert assistance to challenge AI evidence. In the absence of specific AI expertise, it is difficult to conceive how the defendant could scrutinise AI-powered tools and challenge AI evidence. At the same time, it is important to consider whether the defendant can afford the required expertise in AI. Currently, not all suspects and accused persons can access the expert assistance (legal, technical or otherwise) necessary to challenge scientific knowledge relied upon in criminal proceedings, which raises questions about equality of arms. ⁹⁰ This issue might be aggravated in the prosecution’s reliance on AI evidence since the expertise necessary to challenge it can be expected to be rather expensive considering all the training required in this field. Even in cases where the defendant can access the relevant expertise, concerns still remain given the inscrutability of AI systems by defendants and their lawyers. In this context, it is difficult to envisage how the defence can question the accuracy and legality of AI evidence without access to the technical information and knowledge about the processes that generated it. This has serious implications for the principle of equality of arms, the right to participate effectively in the trial and the right to an adversarial process.

In the light of these findings, some of the critical concerns and legal implications of AI evidence become apparent when attempting to apply due process principles in this context. These challenges combined with others, such as the risk of automation bias (infra), could aggravate the concerns. In that way, AI evidence may be given undue legitimacy due to the uncritical over-reliance on machine results, while the defendant is not even able to review (let alone challenge) the incriminatory evidence relied upon in court. Therefore, discovering and representing evidence through AI systems might turn such evidence into absolute truth, risking lowering or even reversing the burden of proof, ⁹¹ and thereby entailing breaches to fair trial rights.

Principles

An ultimate outcome of the criminal proceedings is the determination, beyond any reasonable doubt, of three core aspects: (i) whether and how an action was carried out; (ii) whether that action is of a criminal nature; and (iii) whether the defendant executed the said action and, in that case, how s/he should be punished. The criminal court has to reach a decision on those aspects, delivered through a judgment where the reasoning underpinning the ruling must be explained beyond reasonable doubt. This relates to the fundamental safeguards of transparency and accountability of public bodies in the administration of justice more broadly – conditions that constrain the arbitrary exercise of government power in a democratic society. In particular, according to established case law closely related to the proper administration of justice, such judgments should adequately state the reasons on which they are based. ⁹²

Reasoned decisions help to demonstrate that both parties have been heard. They also require that courts indicate with sufficient clarity the grounds on which they based their decision ⁹³ and uphold the defendants’ rights. As a result, the defendant would be able to exercise any right of appeal that is available. ⁹⁴ Evidence, as evaluated during the fact-finding process, is one of the elements influencing the legal reasoning of the judicial decision. In particular, the evidence gathered during a criminal investigation should provide the judge (as the ultimate guardian of the proceedings) with the ability to digest the findings presented in trial and determine the defendant’s guilt. ⁹⁵

In this context, when assessing the fairness of the proceedings as a whole, the ECtHR must review the domestic courts’ evaluation of evidence to determine whether the domestic courts’ assessment of the weight of the evidence could be considered unacceptable or arbitrary. Subsequently, the Court must make its own assessment of the weight of the evidence if the domestic courts did not indicate their position on that issue or if their position is unclear. ⁹⁶ This is particularly the case in situations where the ‘sole or decisive rule’ (supra) is at stake.

Another determining factor of the fairness of the proceedings is the principle of the presumption of innocence, ⁹⁷ which requires, inter alia, that (i) when carrying out their duties, triers of fact should not start with the preconceived idea that the accused has committed the alleged facts; (ii) the burden of proof is on the prosecution; and (iii) the principle of in dubio pro reo (i.e. doubts should benefit the accused). ⁹⁸ Since the presumption of innocence is a procedural guarantee in the context of a criminal trial itself, this principle must not only be considered during the examination of the merits of a charge, but it has an impact on the criminal proceedings more broadly. ⁹⁹ Therefore, the presumption of innocence applies to the reasons given in the operative provisions of a judgment, and may be violated if the reasoning in an acquittal reflects an opinion that the accused is actually guilty. ¹⁰⁰

The principle of equality of arms is also relevant in the ECtHR’s evaluation of the weight of evidence by domestic courts, particularly regarding the appointment of experts in the proceedings. ¹⁰¹ While the ECHR does not provide specific rules on how expert evidence should be reported in court, ¹⁰² there are three factors in this respect that play a decisive role in the fairness of the proceedings: (i) the position occupied by the experts throughout the proceedings; (ii) how they performed their task; and (iii) the way the judges assessed the expert opinion. When assessing those factors, the ECtHR takes into account the fact that the opinion given by any court-appointed expert is likely to carry significant weight in the assessment of the issues within that expert’s competence. ¹⁰³

Evaluation of AI evidence by the judge

Considering the principles outlined in the foregoing, it is critical to assess whether (and how) AI evidence could influence the evaluation by the judge and eventually put in jeopardy the fair trial guarantees. Of the issues discussed throughout this paper, transparency is a particularly relevant one when talking about AI-generated evidence. This public law principle can be challenged in the evaluation by the judge considering the inherent opaque and automated nature of AI systems, ¹⁰⁴ giving rise to novel risks, such as the inability to explain decisions due to the black box effect and automation bias. In the context of this study, we focus on these two prominent algorithmic challenges given their potential to influence, and probably undermine, the fact-finding process.

Opacity in machine learning (ML), known as the black box problem (supra), ¹⁰⁵ is one of the main impediments to transparency in AI. In a criminal justice setting, the black box effect may generate challenges in terms of providing a reasoned justification on how the defendant’s guilt was established when the employment of AI techniques is used as evidence. ¹⁰⁶ The adequate administration of justice requires that the factors influencing the outcomes of a case are visible or transparent to the parties involved, including those who use, regulate and are impacted by automated systems. That transparency should involve access to and provision of relevant information (e.g. regarding the inner workings of the automated system that produced the evidence) and also relates to the interpretability of such information. When it comes to AI evidence, since the system could affect the correlations between certain facts and their specific consequences in a criminal case, judges must be able to question all steps of the process in their assessment of such evidence. However, interpretability may prove challenging in this context due to the complex interactions that AI technology, particularly ML systems, involve. ¹⁰⁷ Typically, ML solutions operate in ways that the outcomes produced might not have been foreseen, or even explained, by experts, let alone people lacking extensive technical knowledge (likely to be generally the case for judges). ¹⁰⁸

Automation bias is another algorithmic challenge that may influence the judge’s ability to evaluate AI evidence. Although generally introduced to guard against machine error, ¹⁰⁹ human oversight might not always have the expected effects due to the automation bias issue, consisting of the human tendency to over-rely on machine outputs primarily due to the perception that they are generally trustworthy and reliable. ¹¹⁰ Machine processes are often considered objective and free from human biases based on the assumption that, as maths are involved, those processes must be naturally neutral. Automated devices can fundamentally change how people (e.g. judges) approach their role in a decision-making process, rendering human oversight ineffective as a result. Accordingly, criminal investigators and triers of facts may tend to place greater weight on automated assessments over other sources of advice or evidence, even though AI-powered tools replicate (and may even exacerbate) pre-existing human biases, as shown in growing literature. For instance, a study conducted by ProPublica showed that algorithmic systems used in the criminal justice sector generated ‘remarkably unreliable’ evidence when forecasting violent crime. ¹¹¹ Another example is the research performed by researchers at the Massachusetts Institute of Technology, which provided evidence of discriminatory outcomes in ML systems based on protected attributes such as race and gender. ¹¹²

Therefore, while AI evidence can be used as supporting material during the criminal fact-finding process, the algorithmic challenges may play a particular role in this context, raising novel concerns. Specifically, the characteristics of AI seem to be at odds with the transparency and accountability principles in the adequate administration of justice. On the one hand, the black box problem results in an inability to explain certain AI-generated outcomes in human terms. This could lead to situations where it might be hard for judges to fully understand and adequately assess AI evidence, not even relying on expert testimony. ¹¹³ On the other hand, in the presence of automation bias, judges might not be able to remain critical when faced with AI outputs or may often not contradict those, irrespective of the accuracy of the technology. Thus, based on the technology’s appearance of objectivity and certainty, the fact-finder may give undue legitimacy to a flawed AI result, improperly affording the presumption of trustworthiness and reliability to certain (incriminating) evidence, which could in turn lead to miscarriages of justice. This is where the specific dangers of AI evidence inevitably arise. In particular, the black box effect and the risk of automation bias cast doubt as to whether the court would be able to make an informed decision on the defendant’s guilt when admitting AI outputs as evidence and when justifying court decisions through AI evidence. These two algorithmic challenges threaten to hamper the judges’ ability to provide sufficient reasoning for their decisions, posing challenges to the fulfilment of fair trial standards and undermining the quality of judicial decision-making based on AI evidence.

Policy and regulation

The findings above call for the development of a specific legislative framework defining minimum standards on AI in law enforcement. With an adequate legal framework, the possibilities that the outcomes of AI technologies are compliant with fundamental rights and fair trial standards could be much higher. While Council of Europe and EU-wide initiatives to regulate AI more broadly, ¹¹⁴ and those on AI in the criminal law field more specifically, ¹¹⁵ are welcome, they might not address all of the concerns outlined in this contribution.

Regulating specific AI tools, especially on a national basis, ahead of any deployment, can more efficiently substantiate their lawfulness and clarify their treatment under criminal procedural law. ¹¹⁶ In particular, pursuant to the ECtHR guidelines on surveillance instruments, the use of law enforcement AI should be preceded by a concrete regulatory framework giving citizens sufficient information on the circumstances under which their personal information may be processed by such systems for the latter to meet the threshold of foreseeability and lawfulness and respect the right to privacy.

Guidance should be provided with respect to the application of traditional criminal procedural law on algorithmically processed or produced evidence. Assessing the lawful obtainment of evidence through data collection and processing may become more complex. Therefore, guidelines should be developed for judges and their discretion in accepting AI outcomes as sole incriminating evidence, especially in cases where the AI system is known to present high risks of errors or where establishing that the lawful obtainment of evidence may be equivocal. In that regard, establishing thresholds of acceptable error rates across law enforcement AI systems would also be helpful in levelling the playing field.

Further, provided the (technical) accessibility of AI systems, ¹¹⁷ public authorities in the criminal justice field should effectively make their AI systems open to public scrutiny before and after deployment, in line with their accountability obligations. In other words, law enforcement AI systems should meet the level of transparency required to protect fundamental rights and due process guarantees in their use by criminal justice practitioners. That would allow interested parties (such as research communities from relevant fields – including law, ethics, social science, criminology, public policy, but also statistics, computer science or data science) to assess AI systems. This could help determine whether public policy goals are adequately reflected in the tools and contribute to the progress made in multidisciplinary research literature on fairness, accountability and transparency of AI systems in the criminal justice field. ¹¹⁸ It is also encouraged that the investigation how independent experts for interpretation and explicability of AI-processed evidence be equally accessible to the prosecution and the defendant.

Architecture of law enforcement AI

Several of the challenges explained in this contribution may be mitigated by measures implemented during the design of the AI tools. For instance, transparency, widely identified as one of the core requirements for AI technology, ¹¹⁹ may be achieved through by-design measures that enable end-users to assess the validity of the evidence generated when deciding on a course of action, and eventually facilitate the challenging by defendants. Considering the important public policy decisions involved, and the fact that transparency is a necessary precondition to public accountability, ¹²⁰ law enforcement AI tools need to be sufficiently transparent to enable effective independent scrutiny and oversight. Achieving transparency by design entails taking measures towards the traceability and explicability of the AI system, as well as open communication about the limitations of the technology. ¹²¹

Adequate architectural choices should also be made to ensure that the technology can result in evidence that is legally acceptable and reliable, and can contribute to the respect of the presumption of innocence. In order for the chain of custody to be respected, data provenance should be in place for all data. That means that a detailed account of the obtainment and handling processes as to the location and the reliability of each source of data being processed should be maintained from beginning to end (see supra 3.2). It needs to be clear what the original (form of the) data was and which adaptations were made and by whom. For instance, when a text document is translated through the automatic language translation service within law enforcement AI, both the original and translated text must be recorded. The law enforcement AI should enable the verification that the data has not been unlawfully altered during the analysis. In this regard, the use of timestamps in datasets as well as the logging within an AI tool of any collection, alteration, consultation, disclosure, transfers, combination, and erasure of data, recording the time such action took place, along with the identities of users involved is highly recommended. The logs kept by the law enforcement AI should be related to a specific case file, so that they will allow human operators to maintain an evidence log, thereby capturing the chain of custody of all data pertinent to the case at hand. In that regard compliance with ISO standard ISO/IEC 27037:2012, which provides guidelines for specific activities in the identification, collection, acquisition and preservation of potential digital evidence, might contribute to a meticulous respect for the chain of custody. ¹²²

It is further necessary to refrain from designing systems that can pre-designate individuals as criminals before trial or that can assist end-users in carrying out unjustified or disproportionate actions against persons without observing the fair trial guarantees. Moreover, when law enforcement AI detects a degree of suspicion of individuals, it should alert only the authorised users of the tool. The human operator has to later on decide to whether the item resulting from the AI tools will be added as part of an existing investigation or instigate a new investigation. AI developers should continuously strive towards the highest degree of explicability and impartiality of the AI outcomes that inform decisions taken by criminal justice practitioners. Finally, a key feature of technological due process ¹²³ is the requirement of audit trails that document correlations between rules and decisions of an algorithm. The records kept of the audit trail should include a map of the facts and rules that were applied to each decision made in an algorithm. Accordingly, the audit trail should not only allow the oversight of the system, but should also facilitate testing to detect problematic aspects in the decisions made – for instance, classifications based on protected characteristics such as race, nationality, gender and sexual orientation.

Training of professionals

Finally, legislative action and the development of law enforcement AI must be complemented by training and support for judicial staff (prosecutors, judges, clerical workers, etc.) and defence lawyers. Those training activities should be aimed at raising awareness of the available AI tools, their success or error rates, their adequate use and the correct interpretation of their outputs. The special training and support tools should ensure that all the persons concerned have at least some awareness of how AI works and how AI evidence is generated. That way, it can be ensured that all the parties in trial are able to formulate adequate questions and strategies when AI evidence is used in criminal proceedings.

When it comes to LEAs using AI systems, the training programmes should include detailed information on the technical functionalities of the technology, complemented by specific information on the legal and ethical aspects involved. Clarity should be provided on aspects such as why AI works the way it does and delivers a certain output. Emphasis should be placed on the role of algorithms in a decision-making process and the critical importance of the meaningful human intervention in AI-assisted processes. This can be expected to positively impact the ethical and lawful use of law enforcement technology and eventually reduce the potential detrimental effects that AI tools (and their outcomes) involve.