Data Science Approaches in Criminal Justice and Public Health Research: Lessons Learned From Opioid Projects

DOMIP

The DOMIP provides community surveillance capabilities in Delaware to help reduce its opioid and other substance use–related problems (Anderson et al., 2018a). DOMIP triangulates data on overdose deaths, toxicology reports, criminal incidents and arrests, population characteristics, and community resources for two main efforts. First is a user-friendly dashboard called the DOMIP Mapping app (Anderson et al., 2018b). The dashboard contains 7 years (2013–2019) of data for community surveillance and mapping of a wide range of opioid and crime-related metrics, at the U.S. census tract, Zip code, Delaware House District and County levels. Users can select geographic layers to view where overdose deaths are concentrated and plot treatment resources therein. Pop-ups provide additional relevant metrics for chosen areas. Users can also download graphics for tailored purposes. Second, DOMIP permits scientific analysis on hot spots of opioid use consequences and related crime problems as well as statistical analysis to assess patterns, trends, and relationships among the metrics.

The DOMIP project began in 2015 with funding from the National Association of State Controlled Substances Authorities (NASCSA) and the Harold Rogers program of the Bureau of Justice Assistance (BJA). With funding from the National Institute of Justice (NIJ), DOMIP was expanded to 7 years (2013–2019) of data for the dashboard as well as for advanced surveillance, mapping, and scientific analysis (Anderson et al., 2019; Wagner et al., 2019).

FROST

FROST is an integrated data warehouse to support surveillance and promote collaborations between public health and criminal justice agencies (Wang et al., 2020). FROST is updated based on feedback from stakeholder meetings and expert panels. FROST consolidates data from multiple sources and transforms it into a standard format to facilitate the reporting and analytical requirements for surveillance. For instance, FROST analyzes toxicology reports to present the demographic and toxicological characteristics of drug overdose decedents (DuPont, 2018).

FROST’s interactive visualizations are customized based on the data granularity and availability permitted to provide deep insight while protecting data confidentiality. FROST was initially built to track drug-related outcomes in Florida (e.g., drug overdose deaths, controlled substances dispensing records, drug-involved traffic crash reports) but has expanded with national data (e.g., drug submissions in the National Forensic Laboratory Information System) to support federal, state, and local surveillance efforts. Future directions include building a coordination center to facilitate data sharing across workgroups and ad hoc queries capability in a secure environment based on data sharing agreements, user affiliations, and roles.

The FROST project began in 2016 as a research tool for organizing different data sources and disseminating findings. It was supported by two grants from BJA, Office of Justice Programs, and funding from the ONDCP through the South Florida High Intensity Drug Trafficking Area. With new funding from the National Institute on Drug Abuse (NIDA), the FROST will be upgraded to be a national platform to support objectives of the National Drug Early Warning System.

Locating Data and Formulating Agreements

Our projects began by identifying state agencies that tracked relevant information. Data on prescription drug dispensing, calls for police service for overdoses, and fatal drug poisonings are usually collected by distinct state agencies in different branches of government. Sometimes, universities have long-standing research relationships with agencies that provide a pathway to begin the data requesting process (e.g., a research center regularly coordinates with an agency to receive arrest data). Other times, collaborations begin by university researcher–initiated efforts to analyze agency data. Researchers can make a partnership more alluring by completing analyses at a minimal cost, even when they are secondary to primary research questions. In either scenario, finding a statewide data source, rather than making a specific request to a local agency, may improve data coverage. For instance, a city police department might collect police service information, but a centralized state criminal justice information system would amplify the collaboration.

Once data were located, we entered into agreements known as Memoranda of Understanding (MOUs) and/or Data Use Agreements (DUAs) to obtain the desired data. Public health and criminal justice data are usually owned and held by state agencies governed by state and federal laws. Thus, agreements to obtain and use the data must be a priority for researchers. MOUs outline a basic set of expectations between parties and usually include names of the parties involved, a description and scope of the project, and details of each party’s roles and responsibilities. DUAs are more formal as they are typically used for data requests from covered entities under the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule and must describe how the data will be used and protected. Use of MOUs and DUAs are common in the fields of criminal justice and public health (Lynch, 2018). One salient example of the complexity of these arrangements can be found in the Research Guidance and Procedures document from the Washington/Baltimore HIDTA’s Overdose Detection and Mapping Application (a link can be found here.)

Negotiating MOUs or DUAs begins with bringing state officials, university legal counsel, and research teams to the table. For both DOMIP and FROST, data agreement negotiations were a lengthy process that required coordination between three university parties (i.e., the researchers, the university institutional review board (IRB), and the university legal council) and each state agency that housed a particular type of data sought. Using templates provided by state agencies (see a blank example of a DUA for emergency medical services or EMS data from the Florida Department of Health here and corresponding data request procedures here), our universities’ legal counsel worked with state officials on our behalf.

Often, contracts with state agencies require consultation with the state attorney general assigned to that agency. For example, DOMIP’s integrated data set requires DUAs and MOUs with different agencies that are uniquely supervised and regulated by the Delaware legislature and the Delaware Attorney General (DAG). Senior research staff worked for approximately 1 year to negotiate the initial agreements, and they had to be renegotiated with the NIJ award. IRB approval for the project also had to be renewed (i.e., established as a separate approved protocol) with each grant funded.

The negotiation of these agreements must be a top priority, and researchers are best advised to work with university legal services to secure them. Researchers should also be prepared to work closely, diligently, and likely for an extended period with state officials. Some agencies require persons having access to data to complete training (e.g., research security and risk management) and to house the data in secure environments and/or submit applications for agency-IRB review. Many researchers, especially early career scholars, are unlikely to have experience negotiating legal contracts or working with agency leadership. Early and responsible consultation with the correct professionals will help ease this process and overcome obstacles. Without proper sensitivity of the legal framework protecting the data, academics’ requests and “cold calls” to agency personnel for data will likely hinder the effort.

Researchers should be mindful that DUAs should clearly specify the scope and terms of the data request (see above links for example documents). Because criminal justice and public health data are sensitive and protected, it is advisable to limit the request to only those data actually needed and to expect expiration dates. For example, in the project that preceded DOMIP, a Business Associate Agreement (BAA) permitted our research team to perform work for a state agency whose data were governed by a specific law. Under the law and a BAA, DOMIP staff could have access to sensitive data when performing work for the state. However, with the NIJ-funded expansion of DOMIP for university research and a public dashboard, the Delaware law prohibited us from the same level of data. The state offered a work-around, but limited the geographic attributes needed to meet DOMIP goals. Thus, the data source was removed. This is an example of how laws and officials governing sensitive data must be considered ahead of time by researchers when developing their projects.

MOUs and other agreements often stipulate how long researchers can have access to the requested data and abide by state requests to destroy or return the information. Agencies can require data destruction when the research is complete (Lynch, 2018). Therefore, researchers may need to re-request data for continued use and replication by others, which are fundamental best practices in science. Researchers should also be aware of university policies to have new IRB protocols approved for each new research concept. IRBs may require new MOUs or DUAs to be in hand before giving approval. This was the case for both DOMIP and FROST.

A related data agreement challenge is maintaining good relationships with state personnel and agencies as staffing and policy landscapes change. This is especially true for projects spanning multiple years with repeated data requests over time. Researchers must adapt to forms of communication preferred by state officials (e.g., phone calls instead of emails or vice-versa) and anticipate slow response times. Research data requests are not often a top priority for state officials or policymakers as the demands of their office prioritize other matters that directly affect constituents. Elections or staff turnover in agencies may stall or change data requests as well as alter the terms of standing agreements. Even so, researchers should not be expected to compromise their independence in pursuit of maintaining good relationships with state agencies.

Data Integration

Single-source data hubs are ideal for epidemiological criminology data science projects (Culhane et al., 2018; Lynch, 2018), but such entities are difficult to find. Thus, research on opioid-related outcomes requires the third-party integration of data sets housed by various government agencies. To comprehensively examine various dimensions of opioid use, researchers must request crime data from law enforcement and criminal justice agencies, death information from vital statistics agencies, controlled substance dispensing data from prescription drug monitoring programs (PDMPs), health condition and expense data from Medicaid programs, sociodemographic and community data from the U.S. Census Bureau, and EMS data, to name a few. Records are rarely collected for cross-agency use or research (Lynch, 2018). Data fields may contain nonintuitive abbreviations, specialized language, or discipline-specific formats. For instance, one agency may have separate fields for a person’s first and last name, whereas another uses a person’s full name and includes a nickname. Operational definitions may also vary across agencies. Overdoses, for example, may refer to a call for service in police data where naloxone is deployed and a hospital admission (regardless of naloxone use) in EMS data. More importantly, officials do not often collect data to meet academic data quality standards. Researchers often need to take records at face value, knowing measurement and validity issues affecting data quality may be present despite stewardship by a respected agency. Understanding these nuances may seem mundane, but they are critical for understanding analytical bias.

Given that many public databases are not designed for data linkages, data integration involves choice and discretion of the researchers. Under ideal circumstances, data linkage at the person level would be error-free with the use of a unique identifier. Even when person names are available, exact matches can be difficult (McCormack & Smyth, 2017). “Fuzzy” links use probability-based predictions of the same person across databases given information like name, sex, race/ethnicity, and age (Ridge, 2014). Each form of matching introduces new types of errors, as researchers cannot always confirm that records from each agency correspond to the same individuals. The consequences of erroneous data linkage can be high in epidemiological criminology research. For example, many PDMPs attempt to identify criminal “doctor shopping” whereby an individual visits multiple health care providers presumably for the purpose of obtaining controlled substances for abuse or diversion. Individuals can obfuscate their names to evade detection, and “doctor shopping” activity could erroneously identify a patient attempting to obtain legitimate prescriptions.

Data integration problems can arise with area-based estimates when agencies aggregate records to geographic units (e.g., census tracts, ZIP codes). Due to concerns for confidentiality, though, agencies and researchers may be unable to report aggregated measures when sample sizes for geographic units are small (Karp et al., 2008). Stronger restrictions on aggregate reporting apply for publicly sharing data. To comply with standards of de-identification, the HIPAA Privacy Rule requires only the first three numbers of a ZIP code should be reported for areas with populations of <20,000 if the underlying data set contains individual-level data and certain information that could potentially reidentify the subject (HIPAA Journal, 2017). Researchers may then lack a complete picture of the geographic variation of opioid problems within a state because estimates for smaller geographic units cannot be reported or merged with other data. A simple search of PubMed, the leading database for biomedical research, on the keywords “3-digit zip” or “three-digit zip” identifies only 19 studies. This simple search suggests that this geographic level may not be amenable to peer-reviewed public health research.

Another issue to consider is how often the data requested will be refreshed. Getting “real-time” data for research projects is often not possible. For example, overdose death data are seldom available in real time because of the lag in determining causes of death from autopsies and toxicology reports. There are also lags with drug-related arrests and incidents. When making data request plans, researchers should consider the typical delays in reporting or finalizing data, especially when the research project uses data from multiple sources. More importantly, refreshing data daily, weekly, or monthly can raise additional security and confidentiality risks and burden agencies and university researchers with additional work from repetitive data feeds. Our experience shows it is best to refresh data as often as security risks and workload can be kept at a manageable level. Quarterly, semi-annual, or even annual data feeds are usually reasonable and permit meaningful research.

Data and Case Security

Although data science approaches to criminal justice and public health issues promise to both advance science and best practices, they confront similar data security challenges faced by other automated systems in banking, retail, and health industries. Electronic transactions with personal data are ubiquitous with modern computer systems. Even as reliance on e-transactions has grown, so too has public distrust of some systems and entities housing its information. For example, a recent study found a growing mistrust of agencies in safeguarding personal data as data breaches mount (Smith, 2017). Thus, researchers must do more to convince data sharing agencies that they can protect individual confidentiality and anonymity.

Well-developed data security plans, which accompany MOUs and DUAs, can help minimize barriers to sharing sensitive data with researchers. All project protocols (e.g., IRB protocols, MOUs, DUAs, and data security plans) should be based in the ethical principles of beneficence, autonomy, and justice or demonstrate that the research has benefits that outweigh risks, involves parties in decision-making, and features reasonable uses of information (Culhane et al., 2018). Such grounding should be included in all aspects of the research, including staffing, MOUs and DUAs, database management, and other research infrastructure.

Our projects contain sensitive data covered by various laws, including 42CFR and 45CFR that regulate protected health information. Our research protocols also required full IRB review and approval. This not only complicated and prolonged the full execution of MOUs and DUAs (Culhane et al., 2018) but also required high-level security systems and protocols to protect the confidentiality and anonymity of the data. For example, DOMIP’s identifiable data are housed on a secure Microsoft OneDrive, accessible only on two computers in a single locked campus office. Access to these computers and the secure drive is granted through biometric identification of “cleared or vetted staff” (i.e., those with current human subjects training, criminal background checks, and signed nondisclosure agreements). A second Microsoft OneDrive contains de-identified data files and is available on personal workstations via secure login by cleared staff.

FROST, on the contrary, is largely a collection of data sets made publicly available by state contributors. The hosting server for FROST is located within the University of Florida’s Academic Health Center (AHC). The AHC is responsible for operating, monitoring, and controlling both mainframe and mid-range systems. Confidentiality is strengthened by physical security limiting server access to authorized information technology (IT) personnel and physical barriers like hardened doors and walls in a hurricane-proof server room. Servers are on a floor of a building with limited access controlled by electronic keypad locks. Access from individual workstations is limited by specifying private internet protocol numbers. AHC also provides backup, archive, and space management functions to ensure data are both secure and retrievable. We note that public health researchers are often affiliated with academic medical centers at major universities, which provide enhanced resources not always available to criminal justice programs.

We use high-level data transport (secure file transfer or encrypted email servers) to receive data files and also utilize robust encryption methods to secure files and protect confidentiality. Although researchers are likely accustomed to developing their own data hubs or systems for research projects they can access for an extended period of time (Lynch, 2018), it may be advisable to adapt to existing data systems if they are available. Culhane et al. (2018) recommend trending toward single data hubs and clearinghouses to access data. Working with existing data hubs may also circumvent problems with competing efforts in progress among siloed state agencies or external entities such as universities, private think tanks, or commercial providers. Thus, criminal justice researchers would be advised to investigate the existence of data hubs and craft their projects within them. These data hubs are populated with data that are standardized and de-identified into an agreed-upon common data model. Common data models used to analyze opioid-related outcomes include the Prescription Behavior Surveillance System (Paulozzi et al., 2015) and the Medicaid Outcomes Distribution Research Network (MODRN; Kennedy et al., 2019).

Finally, researchers must be cognizant of security risks in reporting data. High data granularity is often desirable for meeting goals and advancing science and practice. However, the problem of reidentification of individuals is a security risk that must be considered and prevented. Having knowledge of data thresholds to prevent reidentification is necessary but not sufficient. Data presentation must often follow data suppression protocols to avoid unintended disclosures.

Closely related are precautions to avoid data sharing beyond that stipulated in DUAs. Identifiable or partially identifiable personal health and criminal justice data are difficult to obtain, yet demand is high. Researchers may field requests from others, including agency leadership, to share data in one form or another, but should adhere to rules in DUAs. For example, an external party requested data from DOMIP rather than doing so from the agency steward (and with which we had a DUA). We declined the request.

Following these suggestions and all existing data, guidelines may not, however, prevail in satisfying concerns about personal anonymity and confidentiality. Researchers should be prepared to scale back their data request and/or accept the fields and granularity of data agencies are willing to provide. For example, DOMIP was able to obtain all personal identifiers from data sources for several years, but recently had to accept fewer ones due to agency policy changes. The best advice is to be flexible in requesting data, willingness to meet officials’ requests, and developing a research plan that can accommodate any changes. Researchers should try adhering to robust research designs despite changes imposed by agencies while simultaneously avoiding ad hoc data requests that overextend researchers’ time.