Abstract
Background/Aims:
Clinical trials evaluating facility-to-facility telemedicine may include sites that have limited research experience. For the trial to be successful, these sites must correctly perform research-related tasks. This study aimed to determine whether health care professionals at community hospitals could accurately identify simulated study eligible patients and submit data to a research coordinating center.
Methods:
Twenty-seven community hospitals in the United States and Canada participated in this study. An electronic survey was sent to one designated health care professional at each site. The survey included a description of trial eligibility criteria and five written neonatal resuscitation scenarios. For each scenario, the participant determined whether the neonate was study eligible. One scenario required participants to submit 14 data elements to the coordinating center. Accuracy of study eligibility and data submission was summarized using standard descriptive statistics.
Results:
The survey response rate was 100% (27/27). Overall accuracy in determining study eligibility was 89% (120/135), and accuracy varied across the five scenarios (range 82–93%). Overall accuracy of data submission was 92% (310/336). Data were >95% accurate for 9 of the 14 data elements, with 100% accuracy achieved for 6 data elements. These results were used to clarify eligibility criteria, inform database design, and improve training materials for the subsequent clinical trial.
Conclusions:
Health care professionals at community hospitals accurately determined trial eligibility and submitted study data based on written clinical scenarios. Research teams conducting telemedicine trials with community hospitals should consider completing pre-trial simulation activities to identify opportunities for improving trial processes and materials.
Introduction
Most hospitals in the United States use video or other technologies to connect with patients and consulting physicians. 1 Hospital-based telemedicine brings expert specialty medicine from the distant site to patients and providers in small or rural originating sites. 2 Large, rigorously designed, controlled studies are needed to evaluate the efficacy of telemedicine. These studies should use methods that address interactions with diverse stakeholders in a natural setting. 3 As such, clinical trials evaluating hospital-based telemedicine should include the smaller, community hospital originating sites. However, these sites often have limited research experience.
Multicenter clinical trials require significant investments of human, financial, and physical resources. Nearly 25% of trials conducted with full financial support from agencies such as the National Institutes of Health are terminated before generating results, 4 most commonly due to problems with trial conduct, coordination, administration, or communication. 5 For hospital-based telemedicine clinical trials to be successful, community-based originating sites must efficiently and correctly perform important research-related tasks, such as screening sufficient numbers of patients and enrolling subjects into the trial. 6 –8 Strong performance by community sites is needed to ensure the trial can be effectively executed and completed—thus, avoiding scientific, ethical, and financial concerns. 5
It is widely recognized that the way multicenter clinical trials are developed, initiated, coordinated, and finalized should be improved. 9 However, there are few studies designed to examine whether knowledge gained from pre-trial activities positively impacts site performance and trial conduct. 10 We have designed a prospective multicenter clinical trial to evaluate the impact of hospital-based telemedicine on the health outcomes of neonates born in 27 community hospital originating sites. Originating site staff have not previously participated in clinical research, and they will need to identify study eligible neonates and submit a limited amount of study data for the trial.
As a pre-trial preparatory activity, this study aimed to use written neonatal resuscitation scenarios to determine whether health care professionals at community sites could accurately identify study eligible neonates and abstract and submit data to a research coordination center.
Methods
Twenty-seven community hospital originating sites in the United States and Canada participated in this study. These hospitals have been identified as study sites for a proposed multicenter clinical trial that will evaluate the impact of teleneonatology on the health outcomes of at-risk neonates born in community hospitals. Teleneonatology is a real-time, audio-video telemedicine service that allows remote neonatologists to connect with community hospital care teams during neonatal emergencies, such as the resuscitation of at-risk neonates. 11 –16
The community hospitals have Level I well newborn nurseries (median [interquartile range, IQR] annual delivery volume 353 [221]) or Level II special care nurseries (median [IQR] annual delivery volume 2,569 [1,940]), as defined by the American Academy of Pediatrics Policy Statement on Levels of Neonatal Care. 17 Community hospital nurseries are staffed by care teams that do not have deep experience with at-risk neonates and may need medical advice on an urgent or emergent basis. The Mayo Clinic Institutional Review Board determined this study to be exempt from its review.
After receiving letters of permission from community hospital leadership, study participants were contacted by e-mail using a recruitment script. The e-mail included a link to the study's electronic survey (Supplementary Appendix SI). The recruitment e-mail and survey were sent to one designated health care professional, for example, medical director/chief, physician, or nurse manager, who was serving as the primary contact for each community hospital site (total n = 27). The survey introduction described the eligibility criteria for the proposed multicenter clinical trial, including the specific inclusion and exclusion criteria. In addition, we provided a table that summarized the criteria (Table 1). The survey included five written neonatal resuscitation scenarios. After each scenario, the participant was asked whether the neonate was eligible for the trial.
Summarizing Study Eligibility Criteria That Was Included in the Electronic Survey
For one scenario (Scenario 4), the study eligible neonate remained in the community hospital upon conclusion of the scenario, and participants were required to abstract and submit study data to the principal investigator. Participants could choose one of two methods for data submission. Study data could be submitted through additional electronic survey questions or by using a templated worksheet that could be downloaded, completed, and uploaded (Supplementary Appendix SII). We offered two data submission methods to assess participant preference and evaluate data accuracy, which would inform the process used for the subsequent multicenter clinical trial.
STATISTICAL ANALYSIS
Accuracy of study eligibility and data submission for the five written neonatal resuscitation scenarios was summarized using standard descriptive statistics. Study eligibility and data submission accuracy were summarized by the level of neonatal care of the responding community hospital. Data accuracy was also summarized according to the submission method used, that is, electronic survey versus templated worksheet. All responses were reported as frequencies and percentages. Given the small sample size, no hypothesis tests were performed.
Results
The survey response rate was 100% (27/27). Forty-four percent of respondents (12/27) were health care professionals from Level I well newborn nurseries with the remaining 56% (15/27) of respondents from Level II special care nurseries. Respondents held various roles within their newborn nurseries. Forty-four percent (12/27) were medical director or chief, 33% (9/27) were pediatricians or family physicians, and 22% (6/27) served as nurse manager or director. Ninety-three percent (25/27) of respondents completed the survey on their own, while 7% (2/27) completed it as a team.
For the five neonatal resuscitation scenarios, respondents achieved an overall study eligibility accuracy of 89% (120/135). Eligibility accuracy differed across the five scenarios, ranging from 82% (Scenario 2) to 93% (Scenarios 1 and 3, Table 2). Respondents from hospitals with Level I nurseries were 87% (52/60) accurate in determining study eligibility, and those from hospitals with Level II nurseries were 91% (68/75) accurate.
Accuracy of Determining Study Eligibility for Each Neonatal Resuscitation Scenario
The survey was not designed to collect details when the respondent incorrectly deemed a patient eligible (when in fact the patient was ineligible). The reasons were inferred based on the exclusion criterion that made the patient ineligible.
CPAP, continuous positive airway pressure; NICU, neonatal intensive care unit; PPV, positive pressure ventilation.
By reviewing the respondents' reasons for ineligibility and the pattern of incorrect responses, we identified that inaccurate eligibility determination was due to 3 main reasons. First, some community hospital health care professionals misinterpreted the inclusion criteria. They thought the neonate had to meet both inclusion criteria to be eligible, when in fact, the neonate only needed to meet one of the two inclusion criteria to be study eligible. Hence, in three of the five scenarios (Scenarios 1, 3, and 4), respondents incorrectly deemed the neonate ineligible because the neonate did not meet both inclusion criteria.
A second reason for incorrect eligibility determination was due to misunderstanding the exclusion criterion “presence of the neonatal transport team at time of birth.” In Scenario 1, the transport team arrived at 10 min of life (indicating arrival after time of birth), but one respondent mistakenly thought this made the neonate ineligible. For Scenario 5, respondents incorrectly thought the patient was eligible, even though the transport team was present for delivery and resuscitated the neonate.
A third cause of eligibility error was failure to identify the exclusion criterion “severe congenital anomaly necessitating intensive care or surgery in the first two weeks of life.” This error occurred in Scenario 2 where respondents incorrectly classified a neonate as eligible, despite presence of “a large lumbosacral myelomeningocele and ventriculomegaly” requiring transport to the neonatal intensive care unit (NICU) for “further evaluation, management, and surgical treatment.” Incorrectly classifying patient characteristics and availability of other support services caused respondents to make eligibility errors in the simulated scenarios.
Eighty-nine percent of community hospital health care professionals (24/27) correctly identified the neonate in Scenario 4 as eligible for the study and not requiring transfer to the NICU. This scenario required respondents to submit study data to the principal investigator. Seventy-nine percent (19/24) of respondents chose to submit data through the electronic survey, and 21% (5/24) downloaded, completed, and uploaded the data worksheet. Overall accuracy of data submission was 92% (310/336, Table 3).
Accuracy of Data Elements Submitted for Scenario 4, Which Described a Study Eligible Neonate That Remained in the Community Hospital Originating Site
Data submission was >95% accurate for 9 of the 14 data elements, with 100% accuracy achieved for 6 data elements. Data accuracy was 92% (141/154) for respondents from hospitals with Level I well newborn nurseries and 93% (169/182) for respondents from hospitals with Level II special care nurseries. When considering the method of data submission, data accuracy was 94% (249/266) when respondents used the electronic survey and 87% (61/70) for the data worksheet. There were three missing values for data submitted through the worksheet compared to no missing values for data submitted using the electronic survey.
Respondents were least accurate when submitting data on maternal antenatal steroid exposure. Some respondents incorrectly selected “no” in response to the question about whether the mother received antenatal steroids, when the correct response was “not indicated due to gestational age ≥37 weeks.” To accurately answer the question, respondents had to recognize that the mother did not receive antenatal steroids and that it was not indicated.
Community hospital health care professionals also made more frequent errors (<90% accuracy) when entering the date and time of birth and selecting the mode of delivery. Difficulties with the time of birth were commonly due to incorrect selection of AM versus PM (on the data worksheet) or confusion about the use of 24-h time (on the electronic survey). For mode of delivery, respondents incorrectly selected “vaginal delivery” when the correct answer was the more specific response, “assisted vaginal delivery (forceps or vacuum).”
Discussion
This study demonstrated that health care professionals at community hospital sites were 89% accurate in determining trial eligibility when presented with written clinical scenarios. We were able to identify specific aspects of the inclusion and exclusion criteria that contributed to incorrect classification of study eligibility. In addition, study data submitted by community hospital staff was 92% accurate. While most data elements achieved >95% accuracy, we found questions that had similar responses, for example, “no” versus “not indicated” or “vaginal” versus “assisted vaginal,” had higher error rates. Study results provided the investigative team with actionable knowledge that will inform the materials, database design, education, and training used during the subsequent clinical trial.
The 11% trial eligibility error rate observed in our study using simulated scenarios compares favorably to results from previous studies. In a report that evaluated a mock enrollment run-in phase before a multicenter clinical trial, the enrollment error rate was 16%. 18 Similarly, the evaluation of a prospective, multinational trial revealed that 14% of patients did not meet inclusion and exclusion criteria. 19 Violations of study entry criteria are one of the most common protocol violations in large, multicenter clinical trials. 20
These errors can be due to vague definitions of inclusion and exclusion criteria, which were observed in our study. Typically, the highest risk of enrollment error occurs when a site recruits their first patient into a new trial. First patient enrollment error rates can be twice that of subsequent patients, and as enrollment increases, the frequency of protocol violations decreases. 21 Adherence to study eligibility criteria is fundamental to conducting successful telemedicine clinical trials as protocol violations have been shown to influence the measurable efficacy of study interventions. 19,21
We found that data accuracy by community hospital health care professionals was 92%. Our observed error rate of 8% is similar to that reported in a review of 42 articles, which found the average source-to-database error rate was 976 errors per 10,000 fields, or about 9.8%. 22 Other telemedicine trials focused on stroke that included community hospital staff have reported proportions of incomplete or erroneous data ranging from 3.1% to 12.0%. 23 –25
Assessments of data quality suggest that most errors occur when data are abstracted from the medical record and transferred to the trial case report form. 26 These transcription errors include incorrect data, missing data, and data on the form that is not in the medical record. 27 All of these error types were observed in our study. The design and clarity of the case report form and training and supervision of data managers are important to minimize error and improve data quality, including both completeness and accuracy.
Evaluating the benefits of adding incremental workflow, communication, and coordination improvements to optimize multicenter clinical trials is crucial. However, there are few published examples of such evaluations, 28 and no published literature on the value of these preparatory activities for telemedicine clinical trials. When Lim et al. simulated activation of a clinical trial to test trial processes, they found the simulation was well received by site investigators and increased investigator confidence in conducting the trial when activated. 29 In a multicenter clinical trial conducted in intensive care units, a mock enrollment run-in phase significantly lowered the actual trial enrollment error rate. 18
In our study, we found pre-trial simulation activities allowed the investigative team to engage and seek input from staff at the community hospital originating sites. We identified how study eligibility errors might occur and discovered a need to clarify inclusion and exclusion criteria. Eligibility errors and qualitative feedback informed revisions to our study eligibility table and development of a separate study eligibility flow chart. These improvements may help ensure that the trial materials and processes work well for staff at the community hospital originating site and facilitate a smooth, efficient trial initiation.
Our study also detected which data elements were at highest risk of error. This knowledge has influenced design of the study database and the content of education and training materials that will be used for the subsequent telemedicine trial. We identified opportunities to optimize database design to avoid errors, which were more frequent for questions requiring a two-step decision to select the correct response. For example, incorrect responses about the mother's receipt of antenatal steroids could be reduced if the question is only visible when users enter an applicable infant gestational age of <37 weeks into the study database.
In addition, to improve accuracy for the question about mode of delivery, branching logic could be used. That is, if the respondent selects mode of delivery as “vaginal,” then a follow-up yes-or-no question about use of vacuum or forceps assistance would appear. Reviewing data elements and their definitions with community hospital study staff during interactive training sessions could improve data accuracy. Results from this study have informed the content for our training materials, including additional time for explaining data fields with higher error rates, for example, attention to the 24 h versus AM/PM time format.
Our study had some limitations. The accuracy of study eligibility and data submission was determined using simulated, written clinical scenarios that were reviewed by a single health care professional at each community hospital originating site. While this study design aligned with our timeline and resources, it did not replicate the actual, clinical situations in which team members must determine study eligibility or the process of abstracting data from the medical record. Participant education on eligibility criteria was limited to the written description and summary table included in the survey.
There was no opportunity for participants to ask questions or clarify the criteria with investigators to improve their understanding, which will be a part of trial initiation. Similarly, data submission was completed by community hospital health care professionals independently, without assistance from a study coordinator or verification of data with source documents. Hence, enrollment and data accuracy may be higher during the actual multicenter clinical trial.
This pre-trial work allowed our investigative team to clarify eligibility criteria, reformat trial resources, and identify data elements at higher risk of error before the planned multicenter telemedicine trial. In addition, these findings will inform staff education and training materials and methods used during trial initiation. Future studies should explore whether pre-trial activities lead to fewer enrollment errors, greater data accuracy, and higher rates of telemedicine trial completion. 29 Distant sites serving as the sponsor and/or coordinating site may need community-based originating sites to achieve a minimum level of protocol-specific experience to successfully conduct a telemedicine research trial. Site investigators, study coordinators, and research coordinating centers must ensure protocols are executed as designed so telemedicine benefits, if present, will be evident. 21
Conclusions
Health care professionals at community hospital originating sites were 89% accurate when determining trial eligibility using written clinical scenarios. We observed an 8% error rate for data submission in this simulated activity, which was similar to previously published averages. Research teams conducting multicenter telemedicine clinical trials with community hospitals should consider completing pre-trial activities, including case-based scenarios and simulations, to identify opportunities for improving trial processes and materials. Completing this preparatory work may improve accuracy of subject enrollment and data collection, thus increasing the likelihood of trial success and confidence in study findings.
Footnotes
Authors' Contributions
Dr. Fang conceptualized and designed the study, designed the data collection instruments, coordinated and supervised data collection, drafted the initial article, and reviewed and revised the article. Drs. Whyte, Umoren, Limjoco, Makkar, and Lo, Ms. Yankanah, and Mr. McCoy provided input on study design, assisted in participant recruitment, and critically reviewed and revised the article. Dr. Herrin carried out the analyses and reviewed and revised the article. Dr. Demaerschalk conceptualized and designed the study, supervised data collection, and reviewed and revised the article. All authors approved the final article as submitted and agree to be accountable for all aspects of the work.
Disclosure Statement
The authors have no conflicts of interest relevant to this article to disclose.
Funding Information
No funding was secured for this study.
Supplementary Material
Supplementary Appendix SI
Supplementary Appendix SII
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
