Automated data collection from an electronic medical record for a prospective real-world study in patients with retinal disease (VOYAGER)

Introduction

Real-world data shed light on treatment use and outcomes in clinical practice, providing critical information that helps to explain the gaps between clinical trial and real-world outcomes and may thus inform ways to optimize treatment outcomes. Real-world data are especially pertinent for retinal diseases, given that vision outcomes following treatment with anti-vascular endothelial growth factor (VEGF) therapies in real-world clinical practice often fall short of those achieved in clinical trials.^1–6 Although there are numerous existing real-world data sources on patients with retinal diseases treated with anti-VEGF therapies across the globe, data covered by these sources are often minimal, incomplete, and do not address holistic aspects of patient management, such as drivers of treatment decisions. ⁷

VOYAGER (NCT05476926) is a 5-year, non-interventional, multi-national, multi-center, prospective study assessing patients with neovascular age-related macular degeneration, diabetic macular edema, or retinal vein occlusion treated with faricimab, the first bispecific antibody for intraocular use that inhibits both VEGF-A and angiopoietin-2, and the Port Delivery Platform with ranibizumab, an anti-VEGF. ⁸ By collecting a diverse range of variables, VOYAGER aims to elucidate real-world treatment patterns and clinician- and disease-related drivers of outcomes across countries and practice settings. These variables encompass clinical and ophthalmic outcomes, anatomical features, clinicians’ experience, and treatment regimens. ⁸

Given the large quantity of study variables, there is a considerable time and resource burden associated with collecting and entering study data on VOYAGER electronic Case Report Forms (eCRFs). The adoption of electronic medical records (EMRs) in clinical practice has helped improve efficiency and accuracy of data collection, but EMRs are rarely leveraged in clinical studies to automatically populate study-specific eCRFs. ⁹ As such, study data routinely captured in EMRs still need to be manually entered into VOYAGER eCRFs, resulting in duplication of effort with inherent inefficiencies and risk of transcription errors. ¹⁰

To address the challenges of duplicative data entry, an EMR-to-electronic data capture (EDC) solution was developed for use in VOYAGER. EMR-to-EDC enables study data from the EMRs of consenting patients to automatically pre-populate eCRFs within a study EDC system. By automating the retrieval and entry of study data that would otherwise need to be manually located within a patient’s EMR and then transcribed into eCRFs, EMR-to-EDC should decrease time and resources spent on data collection and reduce transcription errors and data incompleteness. Ultimately, EMR-to-EDC may enhance the efficiency of clinical studies, while supporting data integrity, accuracy, and completeness. Different EMR-to-EDC methodologies have previously been developed and demonstrated these benefits when applied to clinical studies across various therapeutic areas, including diabetes, cancer, and kidney disease, in different countries,^10–18 highlighting the broad utility of EMR-to-EDC across medical fields and geographies.

Here, we describe the results from the 6-month, small-scale pilot study assessing the implementation of EMR-to-EDC within the ongoing VOYAGER study. We aimed to establish whether EMR-to-EDC could reduce the burden of data entry and improve data accuracy, to inform wider implementation of the methodology in the VOYAGER study. We also gathered end-user feedback to determine whether this methodology is viewed as beneficial by site staff.

Methods

EMR-to-EDC workflow

To enable EMR-to-EDC pre-population for a consented patient, site staff first annotated the patient’s mediSIGHT record with the patient’s unique VOYAGER STUDY_ID at the baseline study visit, to create a link between mediSIGHT and the VOYAGER EDC. Minimal data were then entered in the baseline visit eCRF (Figure 1).

OPEN IN VIEWER

Figure 1.

VOYAGER EMR-to-EDC workflow. (a) Patient enrollment and baseline visit. (b) Follow-up visit.

After pre-population of the eCRF baseline visit with EMR data, EMR-to-EDC identified subsequent visits recorded in the patient’s EMR that were likely to be follow-up visits, then automatically created subsequent follow-up visits in the eCRF before pre-populating them. If site staff have already created a follow-up visit in the EDC, EMR-to-EDC recognized the existing visit and pre-populated it with any missing study-relevant EMR data.

Once pre-population of a study visit was complete, site staff reviewed and edited the pre-populated information, if required, before signing off the form, allowing them to retain full control over data recorded for each patient. Site staff manually filled in fields within study eCRFs that were not pre-populated, such as fields out of scope for pre-population or where data were not available in the EMR.

Further details of VOYAGER EMR-to-EDC development, data pre-population workflow, and architecture are available in Supplemental Appendix 1.

VOYAGER EMR-to-EDC pilot study

Staff from both sites were first trained on the VOYAGER study and EMR-to-EDC pilot (Supplemental Figure S1). Patients being routinely treated for neovascular age-related macular degeneration or diabetic macular edema were invited to participate in VOYAGER; no patients declined. All patients who were enrolled between 25 July 2023 and 31 January 2024 (Central Middlesex Hospital) or 9 November 2023 and 30 April 2024 (Royal Liverpool Hospital) provided informed consent to have their VOYAGER data used in this EMR-to-EDC pilot study. The first five patients enrolled at each site were enrolled using traditional manual data entry; all subsequent patients were enrolled using EMR-to-EDC. This provided staff with the ability to compare manual data entry against EMR-to-EDC pre-population.

eCRF pre-population. The following eCRF pre-population outcomes were defined a priori: (1) the proportion of pre-populated and manually entered eCRF fields for patients with pre-populated data, to assess data completeness and reduction of data entry burden with EMR-to-EDC; (2) the proportion of pre-populated eCRF fields overwritten by site staff for patients with pre-populated data, to assess accuracy of EMR-to-EDC data transfer; (3) the mean number of EDC queries generated per visit for manually entered and pre-populated patients; and (4) the average time spent by site staff on data entry for manually entered and pre-populated patients, to assess reduction of data entry burden. Results were aggregated across sites and reported descriptively.

Thresholds of success were prospectively defined for eCRF pre-population outcomes 1, 2, and 4 (Table 1). The thresholds for outcomes 1 and 2 were based on the proportion of eCRF fields configured for pre-population and determined by an expert consensus between IQVIA, the study sponsor, Medisoft, and key opinion leaders in ophthalmology. For outcome 3, system queries were programmed to be raised automatically if certain parameters are met (e.g. data are missing or data added are out of range). To establish whether the number of queries raised was significantly different for pre-populated versus manually entered patients, p values were calculated with Welch two sample t-tests using Excel Data Analysis Toolpak. All eCRF pre-population outcomes were analyzed using data exports extracted from the Medidata RAVE audit trail at the data cut-off (15 May 2024).

OPEN IN VIEWER

Table 1.

Pre-specified thresholds of success for pilot study outcomes.

Outcome	Thresholds for success
eCRF pre-population outcomes
Outcome 1: Proportion of pre-populated and manually entered eCRF fields for study patients with pre-populated data	≥20% (baseline) or ≥15% (follow-up) pre-populated fields across sites and per site
Outcome 2: Proportion of pre-populated eCRF fields overwritten by site staff for study patients with pre-populated data	≤30% (baseline and follow-up) pre-populated fields overwritten by site staff
Outcome 3: Average number of EDC queries for manually entered and pre-populated patients	No formal threshold of success defined
Outcome 4: Average time spent by site staff for manually entered and pre-populated patients	≥15% (baseline) or ≥10% (follow-up) decrease in average time spent by staff to complete eCRF in pilot site versus other sites using manual data entry only a
	≥20% (baseline and follow-up) decrease in average time spent by staff to complete eCRF in patients pre-populated using EMR-to-EDC versus ≥2 manually entered patients a
Site survey outcomes
Proportion of survey with overall neutral or positive responses for Likert scale questions on site staff satisfaction b	≥60% of survey with neutral or positive responses, b for each completed survey

An outcome was deemed successful if it surpassed its associated threshold(s) of success. EDC: electronic data capture; EMR: electronic medical record; eCRF: electronic Case Report Form.

a

Due to visit completion patterns (piece-meal or multi-user eCRF visit completion) and functional limitations of the EDC audit trail, time spent by site staff on completing visits could not be assessed accurately. It was thus not possible to assess time saved by using EMR-to-EDC pre-population versus manual data entry. In the absence of empirical time-saving data from the EDC audit trail, estimated time saved by using EMR-to-EDC for baseline and follow-up visits, as reported by site staff in the site surveys, was used as a proxy.

b

“Neither agree nor disagree” was considered a neutral response; “Strongly agree” or “Agree” was considered a positive response.

Site staff surveys. Evaluative scales, such as the system usability scale, have been used previously by Wang et al.^14,15 to rate perceived usability and other pre-defined performance characteristics of EMR-to-EDC methodology. In this study, site staff completed an online survey with quantitative and qualitative questions on their experience using EMR-to-EDC, to assess if they considered the solution beneficial. The survey asked site staff to indicate whether they agreed with statements on satisfaction with using EMR-to-EDC on a 5-point Likert scale. The survey also asked staff to estimate time saved using EMR-to-EDC and included free-form fields to report perceived benefits and challenges of using EMR-to-EDC. The full survey is available in Supplemental Appendix 2.

The principal investigator of each site, as well as staff who attended the site training and took part in data entry, was sent links to access the online survey at two different study time points (Supplemental Table S3); survey questions were identical across both sites and time points to track changes in attitudes or perceptions. The first survey was planned for when staff had manually entered five patients and pre-populated five patients using EMR-to-EDC (approximately 1 month from study initiation). Completion of the second survey by the same staff members was planned for 1 month before the end of the study (approximately 5 months from study initiation).

Survey results were aggregated across sites and reported descriptively. A threshold of success was prospectively defined for the Likert scale questions on site staff satisfaction (Table 1).

Results

Included patients and study visits

Overall, 49 patients were enrolled in VOYAGER at Central Middlesex Hospital (N = 29) and Royal Liverpool University Hospital (N = 20) during the pilot study periods and were included in this study. All study visits were manually entered into the VOYAGER EDC for 10 patients (five from each site), and pre-populated using EMR-to-EDC for 39 patients. Of pre-populated patients, all had baseline and ≥1 follow-up visits pre-populated, except for one patient from Central Middlesex Hospital who only had their baseline visit pre-populated.

In total, 49 baseline visits (manually entered: 10; pre-populated: 39) and 151 follow-up visits (manually entered: 36; pre-populated: 115) were completed; all 115 pre-populated follow-up visits were created by EMR-to-EDC rather than site staff. EDC audit trail analysis flagged eight pre-populated follow-up visits as inconsistent. Upon review, six visits were deemed out of scope for VOYAGER (five were non-macular clinic visits recorded in EMRs; one was related to primary care) and two visits were not reviewed/completed by site staff within the study period; these visits were excluded. Ultimately, 192 total visits, comprising the 49 baseline and 143 follow-up visits (manually entered: 36; pre-populated: 107) were included in this study (Figure 2(a)).

OPEN IN VIEWER

Figure 2.

Included visits and eCRF pre-population outcomes. (a) Visits pre-populated by EMR-to-EDC, (b) eCRF fieldspre-populated by EMR-to-EDC,^a and (c) pre-populated eCRF fields overwritten by site staff.^a

eCRF pre-population outcomes

Overall, 146 pre-populated visits, comprising 12,378 unique eCRF fields (baseline: 5,017 (40.5%); follow-up: 7,371 (59.5%)), were reviewed by site staff by the data cut-off. These fields were included in the analysis of eCRF pre-population outcomes 1 and 2. Of these, 27.9% baseline visit and 20.5% unique follow-up visit fields were pre-populated using patient EMR data (Figure 2(b)), surpassing the pre-defined threshold of success (pre-population of ≥20% baseline and ≥15% follow-up visit fields). Only 8.1% and 1.6% of pre-populated baseline and follow-up visit fields, respectively, were overwritten (Figure 2(c)), exceeding the threshold of success (≤30% pre-populated fields overwritten for both visit types).

For outcome 3, the mean number of queries raised was 22.3% lower per baseline visit for pre-populated versus manually entered patients (17.1 versus 22.0; p = 0.22) and 42.3% lower per follow-up visit (4.1 versus 7.1; p < 0.05; Figure 3, Supplemental Table S2).

OPEN IN VIEWER

Figure 3.

Number of queries generated per visit for pre-populated and manually entered patients. (a) Queries raised per baseline visit, (b) Queries raised per follow-up visit.

Outcome 4 could not be assessed accurately due to visit completion patterns and functional limitations of the EDC audit trail (Table 1). eCRF pre-population outcomes for each site are summarized in Supplemental Figures S2 and S3.

Site staff survey outcomes

Both surveys were completed by the same 7 site staff (Central Middlesex Hospital: n = 4; Royal Liverpool University Hospital: n = 3). Overall, 44 patients and 78 visits were completed in the VOYAGER EDC at the time of the first survey, and 48 patients and 132 visits were completed by the second survey (Supplemental Table S3). Survey outcomes met the pre-defined threshold of success of ≥60% neutral (neither agree nor disagree) or positive responses (strongly agree or agree) to questions regarding staff satisfaction with EMR-to-EDC, with 69% positive and 31% neutral responses across both surveys. Results from the first survey were mostly positive (positive: 80%; neutral: 20%), while those from the second survey were more balanced (positive: 57%; neutral: 43%).

Generally, staff strongly agreed or agreed that EMR-to-EDC helped reduce study data entry time and collect high quality data in both surveys (Figure 4). Most staff estimated that EMR-to-EDC helped save 11–20 min of data entry time per patient for baseline visits across surveys (Figure 5). Estimated time saved for follow-up visits increased slightly over time, with the proportion of staff estimating 0–10 min saved decreasing from 5/7 (71.4%) to 4/7 (57.1%), and 11–20 min saved increasing from 2/7 (28.6%) to 3/7 (42.9%).

OPEN IN VIEWER

Figure 4.

Site survey results: site staff satisfaction with using EMR-to-EDC.

OPEN IN VIEWER

Figure 5.

Site survey results: estimated data entry time saved using EMR-to-EDC.

Initially, most staff strongly agreed or agreed that EMR-to-EDC was easy to use, they would recommend EMR-to-EDC to their peers, and EMR-to-EDC helped increase efficiency while retaining control over collected data. These sentiments diminished slightly in the second survey, with approximately half of the staff agreeing and the other half responding neutrally to these statements (Figure 4).

Across both surveys, staff reported that time-saving and quality data collection were the main benefits of EMR-to-EDC. In contrast, the main challenges experienced by staff when using EMR-to-EDC were a high number of system queries generated by the EDC, lack of clarity on resolving queries, the pull-through of study-irrelevant data, and difficulties in tracing EDC data that had been populated using EMR-to-EDC versus manual entry (Supplemental Table S4).

Site-level survey results are presented in Supplemental Figures S4 (Central Middlesex Hospital) and S5 (Royal Liverpool University Hospital).

Discussion

Real-world data are important for understanding real-world treatment responses in patients with retinal diseases, often yielding different results to that of clinical trials. However, manual data entry into study-specific eCRFs is standard practice in prospective real-world data collection, which can impose a significant time and resource burden in busy clinical practices. Furthermore, manual inputs can be subject to transcription errors, which may reduce overall data quality and validity of results. ¹⁰ This pilot study assessed the utility of the EMR-to-EDC solution in automating the entry of routinely collected clinical data in EMRs into the EDC system of VOYAGER, a large-scale real-world data collection study.

Results from this study exceeded the thresholds of success, defined as the pre-population of ≥20% baseline and ≥15% follow-up visit fields and overwriting of ≤30% pre-populated data fields, suggesting that EMR-to-EDC was able to transfer a large quantity of study EMR variables from all visits with few data errors. The high proportion of pre-populated fields indicates that most study-required data elements were pre-populated by EMR-to-EDC in the VOYAGER eCRF. Meanwhile, the low proportion of overwritten eCRF fields is suggestive of the accuracy of EMR-to-EDC transfer, given that any alterations to pre-populated eCRF fields were made by site staff in response to perceived errors in the quality and accuracy of the data pre-populated by EMR-to-EDC. However, overwriting may have also occurred due to site staff perceiving data originally captured in the EMR to be inaccurate after transfer into the EDC, and may thus partly reflect the quality of EMR data; therefore, this outcome is not a direct measurement of the accuracy of EMR-to-EDC data transfer.

The surveys revealed that site staff generally agreed that EMR-to-EDC helped to reduce time spent on data entry and helped to collect high quality data, and estimated that 11–20 min for baseline and 0–10 min for follow-up visits were saved per patient. Taken together, these findings suggest that EMR-to-EDC can substantially alleviate the time burden associated with real-world data collection in studies such as VOYAGER while allowing site staff to maintain control over the collected data and enhancing data completeness and accuracy in long-term studies.

The mean number of queries generated per baseline and follow-up visit was 22.3% and 42.3% lower, respectively, for pre-populated versus manually entered patients, suggesting that EMR-to-EDC may further decrease the time burden associated with data entry by reducing the need to manually resolve system queries. Nevertheless, site staff considered the high number of queries a challenge of using EMR-to-EDC. The disparity between numerical results and site perception may be due to a mismatch between site expectations and reality: site staff may have anticipated much fewer queries to be generated by EMR-to-EDC data transfer than was observed, emphasizing the need for objective measures to evaluate the introduction of new technologies into clinical workflows. In addition, some of the queries appeared to arise from the transfer of study-unrelated clinic visit data into the EDC, leading to staff perceiving the associated EMR-to-EDC-generated queries as avoidably burdensome. Further insights into the causes of these queries may help to refine query programming and improve user experience, but were not examined as this would require an in-depth analysis of the audit trail and query programming, which was considered beyond the scope of this pilot study.

Several methodologies have been developed previously with the goal of automating direct data transfer from different EMRs to eCRFs of clinical studies across medical fields, although the sets of data elements to be pre-populated were narrower and required less complex mapping and transformation compared to the variables pre-populated here by the VOYAGER EMR-to-EDC.^10,12–16 The lack of standardized evaluative measures and varying technological set-up limits direct comparison of these methodologies; ⁹ nevertheless, these previous studies demonstrated substantial time-saving and data quality advantages of EMR-to-EDC methodologies compared to manual data entry, in alignment with the present study.^10,12–16 For example, the EMR-to-EDC developed by Ammour et al. ¹² pre-populated 16.9% of EDC variables across five studies, while the VOYAGER EMR-to-EDC pre-populated 27.9% and 20.9% of baseline and follow-up visit fields, respectively. In addition, we found a low proportion of pre-populated eCRF fields overwritten by site staff, consistent with previous work by Wang et al, ¹⁵ where 1.8% of pre-populated fields were corrected by site staff, and Ammour et al, ¹² where 2.2% of pre-populated fields in a mirror EDC diverged from the manually completed study EDC. Similar to this study, Senerchia et al. ¹³ found that EMR-to-EDC pre-population may result in fewer EDC-generated queries versus manual collection of the same data elements (10 versus 300 queries), although this was estimated via a literature comparison and not a direct within-study comparison of queries, as was done in this study.

Although EMR-to-EDC has the potential to facilitate workflows for site staff, few published studies report end-user satisfaction to date. The survey in this study used a mixture of Likert scale and free-form questions to contribute insights into the qualitative benefits of EMR-to-EDC from the end-user’s perspective, which corroborated with objective measures of success. Challenges of using EMR-to-EDC were also identified in this survey, highlighting specific aspects of the solution to be further refined and optimized to fit individual site needs.

This pilot is unique in that it assessed eCRF pre-population outcomes for baseline and follow-up visits separately, revealing that a greater proportion of baseline visit eCRF fields was pre-populated by EMR-to-EDC than follow-up visits. This was expected, given that baseline eCRFs contain more eligible fields for pre-population compared with follow-up eCRFs (231 versus 170). In addition, most data fields in baseline eCRFs are mandatory, whereas follow-up eCRFs include optional data fields that are pre-populated depending on the procedures performed during the visit. These factors may have led some site staff to perceive there to be lower time savings for follow-up visits than baseline visits. Nevertheless, with follow-up visits representing the majority of visits in clinical practice and real-world studies, EMR-to-EDC may offer substantial time savings when used for large numbers of patients visiting sites frequently and over a long period of time. There is also scope within the EMR-to-EDC solution to increase the number of EMR variables amenable to pre-population beyond those included in this study, to further decrease the burden of manual data entry.

To our knowledge, this is one of the few applications of EMR-to-EDC in ophthalmology. A similar solution has previously been employed in LUMINOUS, an observational study of ranibizumab treatment in patients with neovascular age-related macular degeneration, ¹⁹ to enable the semi-automated transfer of data from the Medisoft Ophthalmology EMR (Medisoft), the precursor to mediSIGHT, into the study EDC. This solution was introduced into the EMR as a separate module, enabling visit eCRFs to be completed within the EMR, pre-populated with routinely captured EMR data where possible. Completed eCRFs were then extracted and uploaded to the LUMINOUS EDC manually by Medisoft staff on an approximately fortnightly basis. In contrast, the VOYAGER EMR-to-EDC solution fully automates the data flow from EMR to eCRFs within the study EDC without the need for manual data transfer. Therefore, while both solutions remove the burden of manual data entry for site staff, the VOYAGER EMR-to-EDC solution further relieves this for Medisoft staff.

Notably, the VOYAGER EMR-to-EDC solution’s generic design allows new sites to efficiently adopt the solution, without requiring sites to upgrade their EMRs or install additional software or hardware, provided that sites are already using an EMR-to-EDC-capable version of mediSIGHT. At the time of writing, 118 sites are using a capable version of the mediSIGHT EMR; this was predicted to increase to 140 sites by the end of 2024. With a growing number of sites using a capable version of the mediSIGHT EMR system, there is potential for EMR-to-EDC to be rolled out on a wide scale for the benefit of other real-world ophthalmology studies beyond VOYAGER. In addition, EMR-to-EDC may make participation in real-world data collection studies more feasible, especially for smaller clinical sites with high time demands and limited research resource. It should be noted that as multiple EMR providers exist within and across countries, parallel EMR-to-EDC solutions compatible with other EMRs would need to be developed if this methodology is to become a global industry standard.

As mediSIGHT is still under active development, new fields may be added to the EMR, which could be used as variables in EMR-to-EDC. There is thus potential to increase the number of variables pre-populated into the VOYAGER eCRFs. In addition, there is potential for real-world data collection studies in other ophthalmology subspecialties already supported within the EMR (e.g. cataract, glaucoma) or requiring EMR development to benefit from mediSIGHT EMR-to-EDC and see a decrease in site burden and increase in data quality, ultimately supporting advancements in ophthalmology patient care. On a wider scale, the results of this study add to the growing body of literature supporting the use of EMR-to-EDC methodology in facilitating data collection in clinical trials and real-world studies across medical fields, representing a means to accelerate new developments across healthcare.

This study is subject to some limitations. The sample of 10 manually entered patients is small; however, this represented a considerable proportion of the 49 total patients enrolled over the limited 6-month study period. In ideal circumstances, patients would be randomized 1:1 to manual data entry and EMR-to-EDC pre-population to facilitate comparison of the two methodologies; however, this would reduce the number of pre-populated patients. The enrollment of five manually entered patients per site therefore ensures sufficient data on EMR-to-EDC use are collected, while enabling staff to compare the methodology against manual data entry. Nevertheless, the small number of manually entered patients means that potential variation in the time required for data entry between different patients may not have been captured or perceived fully.

Another limitation is that the staff survey is not a validated instrument; due to the novelty of EMR-to-EDC methodology, there are currently no validated instruments available to assess end-user perception. To overcome this, the survey questions were reviewed and refined via a collaborative effort between the study sponsor, EMR-to-EDC experts, and key opinion leaders in ophthalmology, to ensure questions were capable of evaluating time savings, efficiency, data quality, ease of use, and user satisfaction.

Conclusion

This pilot study provides initial evidence for the use of EMR-to-EDC to reduce the time burden and risk of transcription errors associated with real-world data collection in VOYAGER, in line with previous studies. Notably, while this study showed that the proportion of fields pre-populated by EMR-to-EDC was highest for baseline visits, the time-saving benefits seen for follow-up visits, which make up the majority of data collection activities, suggest that EMR-to-EDC could potentially provide even greater benefits when used in larger and longer studies. In an environment where clinicians have greater time demands than ever, EMR-to-EDC represents an innovative approach to efficiently transfer a wide range of EMR data to research databases while maintaining data accuracy, integrity, and completeness, ultimately advancing research in ophthalmology and beyond.

Supplemental Material

Supplemental Material