Artificial intelligence-assisted criminal justice reporting: An exploratory study of benefits,concerns,and future directions

Introduction

Artificial intelligence (AI) technology has been increasingly adopted in various domains. One area that has attracted interest is the use of AI in report writing, which is a critical aspect of the justice system. Reports serve as a record of incidents and provide a basis for investigations, prosecutions, and court proceedings (Reynolds, 2012). However, quality report writing is a complex and time-consuming task that requires a high level of skill and attention to detail (Yu and Monas, 2018). The use of AI in report writing holds the promise of improving the accuracy, operational efficiency, and consistency of reports while reducing the burden on justice personnel (Redden and Dix, 2020). Indeed, Redden and Dix (2020) in their National Institute of Justice (NIJ) report asked this, “The question for the criminal justice community is not, Is AI ready for deployment today? Rather, the question is, How can we be proactive co-creators of an intelligent criminal justice future that balances the risks and opportunities of this new technology?” (p. 2).

Despite the potential benefits of AI-assisted reporting, concerns have been raised regarding the implications of relying on automated systems for decision-making in the criminal justice (CJ) system. One of the primary concerns is that AI systems could reinforce existing biases. For example, if the data used to train the AI system is biased, the resulting decisions could perpetuate this bias (Bauchard et al., 2023). This could lead to unfair outcomes for individuals, particularly those from marginalized communities who are already disproportionately impacted by biases in the CJ system (Dukes and Kahn, 2017).

Critics also argue that relying on AI could erode the human element of justice work (Dupont et al., 2019; Redden and Dix, 2020). The CJ system is built on the principles of fairness, equity, and justice for all, and human judgment is critical to upholding these values. AI systems may lack the capacity to consider the nuances of complex situations or to exercise empathy and discretion (Bauchard et al., 2023). Therefore, it is crucial to understand the perspectives of active professionals in the CJ system regarding the use of AI in report writing. This will help to identify and address any concerns that may arise and ensure the use of AI is ethical, fair, and transparent.

To date, AI and the CJ system have been mainly studied in relation to predictive policing and risk assessment (Redden and Dix, 2020; Rigano, 2018). However, report writing, which is a vital part of justice work in police, court, and carceral systems, has received little attention. This study aims to fill the gap by examining the advantages and challenges of AI-assisted narrative generation and offering insights into the future direction of the field. By exploring the perspectives of justice professionals on the use of AI in report writing, the authors will provide important insights into the potential of AI to transform the way justice reports are produced and used. Findings will inform the development of AI-assisted reporting systems responsive to the needs and concerns of justice professionals.

This article presents an exploratory study of the benefits, concerns, and future directions of AI-assisted CJ reporting. The study’s research questions are focused on (RQ1) identifying what is happening in the field around narrative generation and (RQ2) the perceptions and concerns CJ practitioners have about the use of AI in report writing. The study involved conducting in-depth interviews with 23 active professionals in the police, court, and carceral systems in Oklahoma. The corpus of data collected from the interviews informs six themes related to justice narrative generation and the use of AI.

Literature review

Report writing is an essential part of the CJ system, used by police officers, attorneys, and correctional staff to document incidents, investigations, and decisions (Miller and Whitehead, 2018). However, traditional report writing practices can be time-consuming, prone to errors, and difficult to interpret (Yu and Monas, 2018). Advancements in AI have shown promise in improving the efficiency and accuracy of writing in general, but concerns exist regarding the use of AI in this context (Bauchard et al., 2023; Redden and Dix, 2020). This literature review explores current practices in report writing in the CJ system and potential benefits/concerns of AI-assisted CJ reporting.

Methods

Findings

Our research aimed to explore the intersection of current field writing practices with the views of CJ professionals on the use of LLMs in their work. Through the analysis of 23 semi-structured interviews, six themes emerged, shedding light on the potential benefits, concerns, and challenges of AI-assisted report writing in the CJ system.

One of the most prevalent themes to emerge was the high frequency of poor-quality reports seen by participants in this study, which can have far-reaching consequences for the justice system. Supervisors and stakeholders who rely on these reports consistently encounter documents riddled with spelling and grammatical errors, missing details, a lack of organization, and inappropriate style and tone. This ubiquitous problem undermines the credibility and effectiveness of the information conveyed in justice reports. Poor reports also impose an undue burden on those who must approve them. Not surprisingly, interviewed supervisors across the system lament the large percentage of their day focused on reviewing reports and either making corrections themselves or returning them to the writer for revision. Indeed, one mid-level supervisor in a large police department said less than 20% of reports he reviews receive approval on the first attempt. He reviews between 50 and 75 reports a day. Not only does this delay the involved cases, but the original authors are also forced to spend additional time editing their work instead of moving on to other tasks.

Surprisingly, despite the digital age, handwritten reports continue to persist in certain agencies. The use of handwritten reports exacerbates the existing challenges related to accuracy, legibility, editing, and sharing. Handwritten documents are prone to illegible handwriting, making it difficult to decipher critical information accurately. Furthermore, editing handwritten reports becomes a laborious and time-consuming task, impeding the efficiency of the overall report-writing process. The limited capability for electronic file sharing among agencies adds another layer of complexity, as agencies reported resorting to manually scanning printed documents to transform them into a shareable digital format. This unnecessary and resource-intensive step not only introduces the potential for human error but also delays the timely dissemination of information, hindering effective collaboration among justice professionals.

Training on how to write and recognize standard elements emerged as a significant variable in generating a quality report. Interviewed prosecutors indicated they frequently provide writing workshops for the departments and agencies they support to improve the usability of the documents they receive. Police, carceral supervisors, and prosecutors alike said onboarding writing instruction for new hires is limited and that on-the-job training was expected to fill the gap in writing skills.

While discussing narrative generation and training with interviewees, it became evident that a considerable portion of reports across the various roles and domains within the justice system consist of boilerplate sections. Across all sampled LE agencies, participants confirmed the existence of commonly used phrases and sections in their reports. For instance, one supervisor disclosed that when officers completed driving under the influence (DUI) reports, they were specifically instructed to include boilerplate phrases such as “staggered gait” or “belligerent” in order to align with the expectations of judges in their jurisdiction. Two of the defense attorneys interviewed underscored this observation and lamented this lack of detail. This observation has noteworthy implications for the potential utilization of LLMs in training, offering guidance, and streamlining the drafting process using templates. However, variations in report writing requirements across different district attorney’s offices pose challenges for statewide LE agencies, underscoring the need for standardization and comprehensive training programs.

Attorneys, too, acknowledged the presence of boilerplate elements in their work. Pleadings, motions, and forms often encompass sections that are common across cases. Interviewed attorneys reported engaging in the practice of copying and pasting specific sections from their own documents or even requesting completed documents drafted by colleagues, which could then be duplicated and customized for a particular case or filing. This reliance on boilerplate language signifies the potential for AI-powered assistance in assembling the necessary components of legal documents. Given quality templates or guides, LLMs could provide attorneys with suggestions for appropriate language, or facilitate the extraction of relevant sections from previous filings, thereby streamlining the drafting process and ensuring consistency in the presentation of legal arguments.

Another finding was a general lack of knowledge about and a prevailing negative perception of AI among the interviewees. Many participants had limited or no understanding of AI and held negative perceptions influenced by media portrayals. Concerns raised included fears of plagiarism, cheating, job displacement, security risks, impediments to legal defense, and the stigma associated with the term “AI” and its use. Frequently cited as examples shaping these perceptions were pop culture references such as The Terminator, Minority Report, and 1984.

Despite this prevailing negative perception, we found a growing acceptance of AI in certain areas of the legal field. One interviewed civil attorney—discovered through a referral by a prosecutor who was a law school classmate—has been using AI for document analysis in large corporate and malpractice suits since 2016. This adoption of AI technology demonstrates a progressive attitude toward AI and its potential benefits. While use in civil law practices has primarily been driven by cost reduction initiatives, this successful use of AI may serve as a foundation for wider acceptance and adoption of AI tools within the realm of criminal law. Indeed, in 2012, the American Bar Association’s amended Model Rule of Professional Conduct 11 to include Comment 8 (Lingos, 2023), which states: “To maintain the requisite knowledge and skill, a lawyer should keep abreast of changes in the law and its practice, including the benefits and risks associated with relevant technology, engage in continuing study and education and comply with all continuing legal education requirements to which the lawyer is subject” implying the need for AI literacy. Among interviewed attorneys, we also found criminal law adapts slower than civil law to technological changes, such as AI tools.

Our interviews with forensic examiners revealed they also employ various forms of AI in their investigative processes, utilizing AI-powered tools to search digital devices and the Internet for criminal evidence. It is interesting to note that when these examiners testify in court regarding the evidence uncovered through AI tools, they refer to it by specific software names rather than using the term “AI.” This observation suggests the focus is placed more on the practical applications and outcomes of the technology rather than on the abstract notion of AI itself.

Interviewees expressed significant concerns about privacy and confidentiality when utilizing web-based LLMs. One of the main apprehensions was regarding access to sensitive information that may be loaded into LLMs, particularly in cases involving juveniles. Data retention and review policies of AI providers such as OpenAI, Microsoft, and Google were mentioned as areas of concern. Although each of the big three claim to protect the LLM chat sessions of users from public access, data breaches and changes in administrators could potentially release sensitive content. Indeed, Bard warns users against including any identifying information and invites a review of Google’s privacy policy.

It was observed that justice professionals handling non-juvenile cases exhibited much less concern regarding privacy and confidentiality. When asked about this, practitioners reported that much of their work eventually becomes public record anyway, which can be accessed by anyone through a variety of means. For those professionals with legitimate or perceived concerns, one detective suggested using a generic term like “victim” when referring to sensitive information during prompt engineering. By doing so, the LLM can still generate a useful narrative draft, which the practitioner could later edit in a more secure form, such as Microsoft Word, to incorporate the relevant sensitive details.

The critical need for human oversight and responsibility throughout the entire narrative generation process clearly emerged during the interviews. Regardless of the method used, and especially in situations using an LLM, supervisors, prosecutors, and defense attorneys all emphasized the importance of street-level practitioners possessing a comprehensive understanding of their reports and taking personal responsibility for the information presented. According to one supervisor, “once you put your name on it,” the practitioner is responsible for the narrative regardless of how it came to be.

Once educated on the capabilities and limits of LLMs, interviewees universally agreed practitioners should never view AI-generated narratives as the final product; instead, they treat them as a draft. This is consistent with the general caution extolled by Teubner et al. (2023) that users are ultimately responsible for generated narratives and must be sensitive to the potential for nuanced errors in the LLM. CJ professionals, especially those at the street level, must meticulously review and verify the accuracy of all information, ensuring the narrative aligns with the facts and details of the case. For example, the inclusion of precise quotes, detailed descriptions of a victim’s appearance or behavior, and other pertinent information are vital for the proper functioning of the justice system. These details provide crucial data to those relying on the report to bring charges, build a defense, or otherwise move a case forward. It is worth noting the interviewed defense attorneys reported they frequently challenge the specifics contained within an officer’s, or other CJ professional, report further emphasizing the importance of accuracy and attention to detail.

Finally, practitioners, LE supervisors in particular, saw the potential of LLMs as a productivity tool. The ability of AI to shift the focus of report writing from “how to do it” to “what to do” was viewed as a valuable improvement. For example, the time required by street-level officers to generate reports could be vastly reduced. This perspective was particularly emphasized considering the ongoing officer shortages and the urgent need to deploy LE personnel back onto the streets. Moreover, the supervisor’s own time could be more liberated. Indeed, many supervisors interviewed said much of their day is consumed with reviewing and editing reports. If the quality of reports submitted to them improves, then the time needed to review them is reduced.

Discussion

The results of this study have significant implications for practice and policy in the CJ system. The findings underscore the pressing need for technological advancements to address the challenges and issues permeating CJ report writing. The introduction of LLMs holds immense promise for streamlining the report writing process, improving overall quality, and enhancing the efficiency of information dissemination. The emergent themes suggest that while there is interest in the potential of LLMs to improve writing quality and efficiency, there are also concerns about their use.

First, there are significant gains to be realized through the integration of LLMs into the training of justice professionals. By utilizing LLMs, novices, who are notoriously poor writers, can gain exposure to common narrative structures and boilerplate elements. Moreover, the utilization of LLMs in narrative generation or editing allows practitioners to expedite the drafting process while ensuring good spelling and grammar, consistency, and adherence to established standards. With access to pre-existing templates or training guides used by the LLM, justice professionals can benefit from standardized language and phrases, facilitating the inclusion of necessary information in a cohesive and comprehensive manner.

Second, the utilization of domain-specific knowledge and proper training in prompt engineering boosts the potential for fine-tuning LLMs to cater to the unique requirements of different justice system domains. An important caveat, however, is that “low-effort prompts will yield low-quality results” (Teubner et al., 2023, p. 4). Prompts containing all the necessary details for narrative generation are essential for quality output. This is an iterative process of refining, adding to, or omitting information as needed to reach a final version that satisfies the requirements of the report. Indeed, this is where domain-specific knowledge is essential for training AI models to recognize the specific requirements, terminology, and legal standards associated with different roles, such as street officers, corrections personnel, and attorneys. Each domain has its own distinct terminology, legal nuances, and contextual circumstances that need to be considered.

Incorporating domain-specific knowledge into the LLM training process produces more tailored and contextually appropriate results, thereby improving the overall quality and usefulness of the generated reports. By tailoring the AI models to the unique needs of different domains and equipping practitioners with the skills to effectively guide AI’s output, the potential for generating accurate, relevant, and contextually appropriate narratives significantly increases. This not only enhances the efficiency of report writing but also ensures narratives align with legal requirements, maintain the necessary level of detail, and effectively convey the intended message. This aligns with and supports Teubner et al. (2023), who noted that “knowledge workers [CJ professionals in this case] interacting with ChatGPT [an LLM] will hence prove their worth based on a combination of skilled prompting and rapid control and adaption of responses” (p. 4).

Third, streamlining the report drafting process frees up valuable time and resources, enabling practitioners to allocate their attention and expertise to more critical aspects of their work. For example, by utilizing LLMs, agencies can boost writing efficiency and return officers to the street to engage with the community more quickly. In the face of nationwide staffing and recruiting problems, the integration of an LLM as a productivity tool enables officers to accomplish more with limited resources, alleviating some of the challenges associated with personnel shortages. This can be applied to the critical functions of carceral or legal professionals as well.

Fourth, the integration of LLMs in report writing holds the potential to enhance collaboration and knowledge sharing among justice professionals. Interoperability between agencies even within the same jurisdiction is problematic. Reporting requirements and software tools vary almost county by county. Even software provided by the state is not universally adopted. By utilizing AI-powered tools, practitioners can exchange best practices, templates, and suggestions, fostering a culture of continuous improvement and knowledge transfer. This collaborative approach can help ensure practitioners benefit from collective expertise, enabling them to generate high-quality reports that meet the expectations and requirements of the CJ system. For example, OpenAI has made it possible to share chat sessions from its LLM, ChatGPT, with others, eliminating the need for screenshots to distribute prompts or output.

Fifth, considering the significance of privacy and confidentiality in the CJ system, it is imperative to address these concerns when implementing LLMs or any AI technologies. For street-level report writers, using generic terms or variables as proxies for sensitive data during prompt engineering can help alleviate this concern. This still allows the LLM to do its work of accelerated narrative generation into which the practitioner can insert the sensitive details later using a word processor. At a higher level, developing robust protocols and guidelines to govern the access, storage, and sharing of sensitive information within LLMs is crucial. Collaborative efforts between AI providers, legal professionals, and IT experts can help establish industry standards and best practices to mitigate privacy risks. Furthermore, maintaining transparency regarding the data retention and review policies of AI providers is essential for building trust within the CJ community. Open dialogue and clear communication between stakeholders can help address concerns and ensure the implementation of LLMs aligns with legal and ethical standards.

Practitioners should adopt a balanced approach considering the potential benefits and the privacy implications of utilizing LLMs. Our findings suggest this will vary by agency based on their needs and the needs of those who use their work. It is important to highlight the utilization of LLMs hosted on local servers may provide an enhanced level of information security. This approach helps ensure sensitive data remains within the organization’s infrastructure, reducing the risk of unauthorized access or data breaches associated with external AI providers. An internal LLM also allows easier customization to the specific needs of the agency. However, hosting an LLM internally could be expensive as they require enormous computing power and storage space. For now, the most likely candidates for housing their own LLM are large police departments and state-level agencies.

Finally, it is important to strike a balance between leveraging LLMs for efficiency and maintaining the integrity of the narrative. Practitioners must exercise judgment and critical thinking to ensure the narrative accurately reflects the facts and conveys the intended message. While LLMs can provide guidance and assist in the assembly of reports, CJ professionals with domain-specific expertise remain essential to verifying the appropriateness of language, context, and address any nuanced aspects the LLM may not fully comprehend. AI tools are indeed powerful and efficient, but they are ultimately just tools. Humans still bear the responsibility of verifying factual accuracy and ensuring the narrative effectively conveys the intended message. By equipping CJ practitioners with the necessary knowledge and skills to effectively utilize AI tools, they can make informed decisions, critically evaluate AI-generated narratives, and ensure the highest quality standards in their reports.

Conclusion

This study sheds light on several significant aspects related to report writing within the CJ system. Our findings emphasize the widespread use of boilerplate elements in generating narratives, highlighting the potential for integrating LLMs to streamline and enhance the report writing process. The incorporation of LLMs presents an opportunity to expedite drafting, provide training support, and ensure the inclusion of standardized sections, thereby improving efficiency and promoting consistency across reports.

These interviews also revealed the growing acceptance and adoption of AI technologies in civil law practices, such as document analysis for corporate and malpractice lawsuits. This acceptance sets a promising precedent for the integration of LLM tools within the CJ system. However, it is crucial to address the prevailing lack of knowledge and negative perceptions surrounding AI among justice professionals. Educating practitioners about the capabilities and limitations of AI can help alleviate concerns and foster a more informed and receptive mind-set.

Of special significance in our findings is the essential role of human oversight throughout the report writing process. While LLMs can significantly enhance efficiency and standardization, it is imperative for justice professionals to exercise critical thinking and judgment. Human oversight ensures the accuracy, relevance, and compliance of AI-generated content with legal requirements, professional standards, and the unique circumstances of each case. The synergy between AI and humans, where AI serves as a valuable productivity tool, empowers justice professionals to focus on higher-level tasks that demand analysis, synthesis, and nuanced decision-making.

In addition, the importance of domain-specific knowledge and prompt engineering training is underscored by our work. Fine-tuning LLMs for specific domains, such as street officers, corrections personnel, and attorneys, can optimize the AI output, ensuring it is relevant and applicable to the diverse needs of the CJ system. Investing in prompt engineering training equips justice professionals with the skills to effectively utilize LLMs, refine prompts, and achieve the desired form and content in the generated narratives.

Privacy and confidentiality concerns also emerged as significant considerations when implementing AI tools. Safeguarding sensitive information, particularly with respect to juveniles, is paramount. Street-level data safeguards and collaboration with AI providers are essential to address these concerns and maintain trust in the use of LLMs.

Ultimately, the successful integration of LLMs in the CJ system has the potential to revolutionize report writing practices. By leveraging the advantages of AI, justice professionals can enhance productivity, improve report quality, and contribute to a more standardized and coherent body of documentation. However, it is crucial to strike a balance between embracing AI as a valuable tool and preserving the critical role of human judgment and expertise in ensuring the integrity, fairness, and accuracy of the CJ process.