The Clinical Research Office of the Endourological Society Audit Committee

Introduction

T he Clinical Research Office of the Endourological Society (CROES) ¹ is the premier Society-sponsored clinical research organization in the field of endourology. The primary objective of CROES is to implement large, multinational research studies that are designed to answer important clinical questions. The CROES Research Council is currently overseeing four active studies focused on ureteroscopic management of renal and/or ureteral calculi, management of renal masses, GreenLight™ laser prostatectomy, and narrow band imaging diagnosis of transitional-cell carcinoma of the bladder.

A CROES Audit Committee ² has been established to conduct clinical data audits within the individual studies. This editorial briefly describes the functioning of the Audit Committee and the reason this process has been implemented: To ensure that data collected by CROES for any study are of high clinical standards, ^3,4 reliable, and validated within the possibilities of an audit process.

The CROES is currently collecting data from the four aforementioned studies through its own online Data Management System (DMS), with future studies currently under preparation. Data are remotely added at individual centers by different users (principal investigators or other study personnel such as fellows, residents, medical students, or nurses). For every patient entered into each study, a set of clinical information is collected, varying in type, dimension, and volume. Below are the numbers of patients and centers involved in each study (as of July 15, 2011).

Each study has its own CROES data manager who oversees and coordinates the data collection process.

Data Quality

All studies contain data fields that are included for answering the scientific questions (study objectives). The Audit Committee recognizes that much of this information is “key data.” The collected data potentially present problems at two different levels: At the level of the single case, and at the level of the entire center/institution.

At the patient level the following issues might arise:

• No data entered

At the center level:

• Cases not entered at regular intervals (long periods between one surgery and the next—are key data being entered consecutively or selectively?)

Each of these issues raises questions on the validity of the data collected and increases the need for an immediate and resolute action aimed at:

At the patient level:

• Identifying data or set of data that are potentially invalid or do not conform to industry standards

At the center level:

• Identifying centers that have a poor record for quality of data

Quality Score

The goal of the Audit Committee is that the assessments will result in a Quality Score (QS) for each center. The QS is a combination of two scores: One score assigned to each patient (Qp) and a score assigned to each center (Qc).

The Qp is calculated to identify cases with many missing data points or information that is not available (NA). The score is based on the calculation of the proportion of missing key data or NA key data over the total number of fields required.

The Qc is based on the calculation of a series of quality indicators: Total number of patients included by the center, rate of inclusion (how frequently patients are added to the database), missing or not available data over the total of patients included, and number of key data deemed as outliers. Each of these parameters can be weighted according to the Audit Committee's recommendations. This QS will be dynamic in nature: As the center increases the number of patients, the score will go up, signaling an improvement in the quality of data. The score can also go down; for instance, after a long period of inactivity, or if many key data are missing or contain outliers. This scoring procedure is, therefore, not just a final tool used by the Audit Committee to select centers for an audit, but also and most importantly, it is a monitoring tool that helps each center involved in the CROES activities to receive regular feedback on the quality of the data they provide, with the aim of keeping the highest possible standards.

Audit Process

The results are reviewed by the Audit Committee that determines which centers will be audited (for example, all centers with a Qp or Qc below a certain threshold).

Data auditing can be a complex and laborious process that requires collaboration from all parties involved and the necessary resources to conduct a large scale review of the information available. The CROES Office has developed in-house capabilities of validating key data and identifying centers with questionable quality.

The Audit Committee has begun to audit the data collected on the four currently active studies based on the QS, as well as selecting centers at random to undergo the audit process.

The formal CROES Audit consists of two phases:

• Phase 1. Generation of queries to be answered by the Center.

The Audit Committee realizes that all data cannot be audited at all centers. It will be possible for each center selected for an audit, however, to supply original data (eg, blood work, operative reports, pathology reports, etc) electronically to the data managers. Yet, some challenges will need to be addressed, such as documents that might need to be translated from their original language into English.

Based on the results of the random audit and the audit of the centers with low QS, the Audit Committee will make recommendations to the CROES Steering Committee about excluding data from certain centers that are considered unreliable. Such a process will ensure that data contained in the final database for each study provide the best possible information available for such a complex undertaking. This process will be dynamic, allowing the committee to lower or raise the criteria for auditing at any given time, based on online information.

Conclusions

The Audit Committee will be providing the CROES Council and the Endourological Society membership with routine updates of the audit process. Our goal is to validate, to the best of our ability, the quality of the studies performed by the CROES.

Respectfully submitted,

CROES Audit Committee

Glenn M. Preminger, M.D. – Chair