Abstract
Objective
To evaluate patient and provider experiences using telemedicine for gynecologic visits among a diverse, low-income population.
Methods
Patients attending telemedicine visits at a resident-run gynecology clinic completed a modified Telemedicine Usability Questionnaire and providers completed a survey addressing satisfaction and barriers for each visit. The Telemedicine Usability Questionnaire included six subscales to assess telemedicine usability with 1–5 Likert-scale responses. Statistical analyses included Chi-square, Fisher's exact, Wilcoxon rank sum, Wilcoxon signed-rank, and two-sample t-test.
Results
Of 192 patients enrolled, 157 (82%) completed the surveys (87% video visits, 13% telephone visits). Most patients were ethnic minorities (non-Hispanic White—16%, Hispanic—32%, Black—28%, Asian—10%), median age was 40 years (range 18–69), and 63% reported income under $40,000. The total mean Telemedicine Usability Questionnaire score was 4.3/5. The reliability subscale score (3.72/5) was lower compared to all other subscales (p < 0.001). Older respondents were more likely to find telemedicine unreliable (mean age >44 vs <39, p = 0.02). Without telemedicine, 54% would have traveled ≥1 h to appointments, with 46% spending over $35 on travel, and 27% missing ≥ 1 workday. Patients preferred telemedicine for follow-up rather than initial visits (81% vs 33%, p < 0.01). Among providers, residents felt less adequately trained in telemedicine compared to nurse practitioners and fellows (54% vs 46%, p = 0.039).
Conclusion
Low-income women utilizing telemedicine for outpatient gynecologic care report positive experiences with improved access to healthcare, cost, and time savings compared to in-person visits. Provider experiences were also positive; however, teaching hospitals must evaluate whether trainee providers feel adequately trained to deliver telemedicine visits.
Introduction
Telemedicine allows providers to deliver care using virtual visits, remote patient monitoring, and mobile health care. 1 While this form of healthcare delivery has existed for decades, the ongoing coronavirus disease 2019 (COVID-19) pandemic promptly ushered in its wide and rapid implementation in an effort to help minimize the spread of infection and reduce the burden on a strained healthcare system. Telemedicine encompasses a variety of technologies including synchronous live videoconferencing, audio-only visits via patient portals, and other patient messaging technologies.2,3 Benefits of telemedicine include improved patient access to healthcare providers, reduced costs and increased convenience for patients, providers, and the health system. 3 Limitations include the need for specific electronic equipment and training for both provider and patient, limited physical examination, and the potential for breach of confidentiality. 3
Prior research has shown that telemedicine can offer similar health outcomes to traditional modes of healthcare delivery without compromising the patient-physician relationship. 1 However, within obstetrics and gynecology, much of the research focus prior to the COVID-19 pandemic was limited to obstetric care.4–7 In early 2020, the American College of Obstetricians and Gynecologists (ACOG) published recommendations to serve as a guide for the gradual integration of telemedicine services into clinical practice for both obstetrics and gynecology, but could not anticipate the rapid integration that subsequently occurred during the pandemic. 1 Obstetrical care was considered essential during the pandemic, and a large amount of literature was quickly generated regarding the feasibility, uptake, and best practices for telemedicine use in outpatient prenatal care.8–13 Alternatively, outpatient gynecologic care, much of which revolves around “non-urgent” visits, was considered non-essential resulting in a significant decline in access to gynecologic care. 14
Rapid implementation of telemedicine raises concerns about increasing healthcare inequities in outpatient prenatal and gynecologic care among historically underserved patients, many of whom were disproportionately impacted by the pandemic.15–17 Prior research from other areas of medicine highlights decreased telemedicine use among patients from minority populations and those with lower education and income levels citing disparities related to internet access, necessary technology, and digital literacy.18–20 There is limited data on the use of telemedicine for gynecologic care in teaching hospitals caring for these same socially vulnerable, low-income populations from both the patient and provider perspectives.21,22 The objective of this study was to evaluate and assess concordance between the experience and perspectives of patients and providers utilizing telemedicine for outpatient gynecologic care in order to determine how to maximize both the patient and provider experience in an academic teaching hospital serving a diverse, low-income patient population.
Methods
This prospective study of patient and provider acceptability and feasibility of using telemedicine for outpatient visits was performed at an urban academic medical center providing gynecologic care from January 2021 to September 2021. This study was approved by our institution's Institutional Review Board (IRB# 20-11022921). Gynecologic services included benign gynecology, gynecologic oncology, family planning, and infertility care. All patients seen at the Primary Women's Health Clinic have access to health insurance through an approved Medicaid or government-subsidized plan. Eligible patients included women 18 years and older, scheduled for video telemedicine visits for gynecologic indications, English-speaking, and willing to consent to participation in the study. Patients eligible for a telemedicine visit but who declined were asked to complete a voluntary survey explaining why they declined a telemedicine visit. These patients were ultimately excluded from the study's final analysis due to low completion of the decline survey (n = 3).
Providers who participated in the study included resident physicians from an Accreditation Council for Graduate Medical Education (ACGME) approved 4-year obstetrics and gynecology residency program (postgraduate years PGY 2–4), gynecologic oncology fellows (PGY 6,7), and nurse practitioners (NPs) who have been in practice for at least 7 years. All providers participating in the study evaluated patients in the same clinical setting.
All telemedicine visits were conducted as live, two-way synchronous video visits unless connectivity issues arose that necessitated a phone visit. Video visits were performed using the video conferencing software Zoom, which was initiated through the hospital's electronic medical record system (Epic), on a mobile device or computer with video capabilities. All patients were required to have an email address on file to use the electronic platform for the telemedicine visit. Resident physicians were supervised by a faculty attending during their telehealth visits while fellows and NPs completed telehealth visits independently.
At the end of each visit, eligible patients were contacted by a research assistant uninvolved in their care and offered participation in the study. If they agreed, the informed consent process was completed. The patients completed a demographic survey and a telemedicine usability survey specific to that visit while the provider completed a separate provider-focused telemedicine usability survey about the visit. Patient surveys were completed by telephone with the assistance of a research team member at a mutually agreed upon time.
Study tools
The patient-focused telemedicine usability survey was a modified version of the validated Telemedicine Usability Questionnaire (TUQ) developed by Parmanto et al. 23 For our study, we preserved all six original subscales of the TUQ assessing telemedicine usability, including usefulness, ease of use, interface quality, interaction quality, reliability, and satisfaction, but shortened the overall survey from 21 to 16 questions so that it was most relevant to our population (Supplement 1). The provider-focused telemedicine usability survey was a newly developed survey that incorporated relevant questions from the modified patient TUQ in order to assess provider perceptions of the encounter and concordance with individual patient responses (Supplement 2). All responses were scored on a 1–5 Likert scale. Patient demographics and relevant clinical information were collected from the demographic survey and review of the electronic medical record.
Statistical analysis
Descriptive statistics were reported as frequencies (percentage) for categorical variables and as means (standard deviation) or medians (interquartile ranges) for continuous variables. Chi-squared test or Fisher's exact test compared the difference in proportions among patients in different subgroups and was used to examine the concordance between patient and provider responses to corresponding questions. Two-sample t-test and Wilcoxon rank sum test compared the difference in the mean value of the continuous measures among patients in different subgroups. To compare the difference in the TUQ survey subscales for each patient, Wilcoxon signed-rank tests were performed. All p-values were two-sided with statistical significance evaluated at the 0.05 alpha level.
Results
Of 192 eligible patients who consented and enrolled in the study, 157 (82%) completed surveys. The median age was 40 (range 18–69). The cohort was racially and ethnically diverse, with 54 (32%) self-identified as Hispanic, 44 (28%) as non-Hispanic Black, 25 (16%) as non-Hispanic White, and 16 (10%) as Asian (Table 1). Of the patients who responded, 74 (63%), reported an annual household income of less than $40,000 and 65 (41%) were employed either full-time or part-time. Ninety-five patients (61%) had children (Table 1).
Patient demographics (N = 157).
When asked about travel and cost related to attending in-person visits at our clinic, 74 (47%) patients reported less than one hour of travel time and 74 (47%) reported 1–2 h (Figure 1(A)). Approximately half, 80 (51%), used public transportation including rail or bus, 33 (21%) took a car, taxi, or ride-share service, and 44 (28%) used some combination of transportation modalities including walking, biking, public transportation, and/or car (Figure 1(B)). Sixty-seven (43%) reported missing at least a half day or more of work to attend in-person gynecologic appointments at our clinic with 72 (46%) reporting $35 or more in travel or missed work costs (Figure 1(C), (D)).
(A) Travel time for patients to attend in-person appointments. (B) Modes of transportation used by patients to attend in-person appointments. (C) Workdays missed by patients to attend in-person appointments. (D) Cumulative cost for patients to attend in-person appointments.
Most patients 133 (87%) had video visits while 20 (13%) had telephone visits. All 20 telephone visits were due to real-time technical difficulties related to the videoconferencing platform. Most patients who participated in telemedicine visits presented for follow-up appointments 143 (93%), while only seven (5%) were new patient visits. The most common indications for a telemedicine visit included results of a review of recent imaging (36 (24%)) and abnormal uterine bleeding (AUB) (21 (14%)). Of the patients who had post-operative appointments via telemedicine, most had undergone procedures related to AUB or pregnancy termination (Table 2).
Visit types and indications for visit (N = 153).
AUB: abnormal uterine bleeding; PGY: postgraduate year.
*There were no statistically significant demographic differences between patients who had video visits versus phone visits.
Of the 138 patients who completed the modified TUQ, the mean score was 4.25 (s.d 0.61) out of a maximum score of 5.0 (Table 3). Participants reported lower subscale scores on the reliability section of the modified TUQ (3.72/5) as compared to all the other subscales: Usefulness (3.72/5 vs 4.32/5, p < 0.001), ease of use and learnability (3.72/5 vs 4.49/5, p < 0.001), interface quality (3.72/5 vs 4.31/5, p = 0.001), interaction quality (3.72/5 vs 4.64/5, p = 0.001), satisfaction and future use (3.72/5 vs 4.46/5, p < 0.001). Further breakdown of scores for individual items comprising each of the subscales is available in Table 3.
Patient-modified Telemedicine Usability Questionnaire (TUQ; N = 157).
We also examined participant factors that may impact their acceptability of telemedicine. Participants who found telemedicine visits to be as reliable as in-person visits were more likely to be younger (mean age 39 vs. 44, p = 0.02). Race/ethnicity, income, education level, and prior experience with telemedicine had no effect on responses for this subscale. Patients preferred telemedicine for follow-up visits rather than for initial visits (81% vs 33%, p = 0.01). Without telemedicine visits, 83 (54%) would have traveled ≥1 h to appointments, with 72 (46%) spending over $35 on travel and 42 (27%) missing at least one workday for an in-person visit.
There were 28 unique providers that staffed visits for the 157 patients who completed the study. The majority of providers, 19 (68%) were PGY 2–4 resident physicians, under attending faculty supervision, while 9 (32%) were either fellow-level physicians or NPs. At least one telemedicine barrier was reported for 43 (28%) visits; most frequently, platform connectivity issues or background noise on the patient's end.
For the clinician telemedicine survey, the mean total score was 4.36 (0.50) out of a total score of 5, with all individual sub-scale scores above 4 except “I believe I was adequately trained to provide telemedicine visits,” which was 3.74 (0.91). Compared to fellows and NPs, residents felt less adequately trained in telemedicine (57% vs 43%, p = 0.039).
By pairing patient and provider surveys for 134 unique visits, we found concordance between providers’ self-perceived ability to counsel patients using the telemedicine platform and patients’ satisfaction with their telemedicine visit (p = 0.096).
Discussion
In our prospective study of patients and providers at a gynecology clinic caring for a diverse, underserved population within an urban teaching hospital, both patients and providers reported largely positive experiences utilizing telemedicine for outpatient gynecologic care. Our study adds to the limited existing literature around utilization of telehealth for outpatient gynecologic care with the unique aspect of prospectively evaluating the patient/provider interaction simultaneously for each telemedicine visit.24–26 This allowed us to evaluate the interaction more critically, helping to identify any discrepancies in patient and provider perceptions of the same visit.
Prior research has revealed that telemedicine is most commonly utilized for follow-up visits in pediatric and adolescent gynecology and gynecologic oncology outpatient practices.21,24,26 The majority of our patients’ telemedicine visits were also for established follow-up and they rated their experiences highly, with over 80% of patients preferring telemedicine for follow-up visits compared to initial visits.
Although most patients reported high satisfaction with telemedicine, a concern remained regarding reliability. Utilizing the TUQ score, the subscale on reliability scored significantly lower compared to the rest of the survey. Reliability was assessed by asking patients if a telemedicine visit was “as good” as an in-person visit. In our study, we looked to identify factors that may influence the reliability score and found that older patients were less likely to find telemedicine visits to be as good as in-person visits when compared to younger patients. While our study overall had a younger population with 84% of our cohort under the age of 50 years old, other studies have shown that older patients have been able to successfully utilize telemedicine. In fact, a 2019 systematic review that examined the effectiveness of ambulatory telemedicine care among older adults found that not only was telemedicine feasible and acceptable among this vulnerable population, but could actually improve health outcomes and relieve access issues in some cases. 27
Other factors outside of age that have been shown to affect the use of telemedicine include race, ethnicity, and income. McAlarnen et al. 21 who assessed the impact of telemedicine on socially vulnerable groups receiving gynecologic oncology care did find differences in patient perspectives by age as well as race, with a greater number of older patients as well as African American and Hispanic patients opting for phone visits over video visits. Despite these differences, the authors concluded that overall virtual visits helped provide more equitable care across the social vulnerability spectrum in their patient population. Our study's findings support the existing literature that telemedicine can improve access to care and may be cost-effective in reducing costs associated with travel and childcare to attend appointments, across age ranges and race/ethnicity, as many patients reported virtual visits helped them save significant time and money that would have been spent on traveling to and attending in-person appointments. Furthermore, overall patient perspectives on telemedicine visits did not differ significantly by age, race, ethnicity, or annual household income. These findings remained unchanged when we compared patients who had a video visit versus a phone visit.
Providers also rated their experience with telemedicine in the outpatient gynecologic setting highly. Overall, providers’ self-perceived ability to adequately counsel patients using telemedicine was consistent with patients’ reported satisfaction with the corresponding visit. When asked to describe potential barriers to providing care via telemedicine, the most frequently encountered barriers were platform connectivity issues or background noise from the patient's environment. Provider perspectives in our study were similar to those described in prior studies within and outside of obstetrics and gynecology with providers suggesting more resources would have been helpful in easing the integration of telehealth including information technology support to learn how to use the telemedicine platform, as well as departmental support for training modules and videos.8,22,28
Although not a primary aim of our study, we found a difference in providers’ self-perceived ability to adequately counsel patients via telemedicine based on the level of training, with resident-level trainees more likely to feel inadequately prepared to utilize telemedicine for patient counseling compared to NPs and fellows. A similar trend was seen among medicine providers at an academic hospital providing outpatient adult and pediatric telehealth visits. 22 This underscores the need to study trainees’ comfort and ability to deliver telemedicine visits more rigorously in order to identify barriers and modify OBGYN training curricula to specifically address this gap. At our institution, there was no formalized telemedicine training for residents, although a number of telemedicine curricula and electives were developed at medical schools across the country during the pandemic, including for OB/GYN clerkships. 29
Moving forward, as we actively navigate the return of in-person visits, further research is needed to continue to understand the optimal role of telemedicine in the outpatient gynecology setting. Specifically, we must consider which patients and visit types are best suited for the virtual platform. This includes identifying which patients may be appropriate for initial visits to optimize access to care. In our study, we found that the telehealth format was most used to review recent imaging results and for postoperative visits. Consideration should also be given to best practices for the timing of virtual visits and possibly alternating with in-person visits. Additionally, medical institutions must provide training, resources, and standard practice guidelines to providers of all levels so that they are comfortable not only using the telehealth technology but more importantly are adequately trained to provide a standard of care at the same level virtually that they would in-person.
One major strength of our prospective study was evaluating the perspectives of both the patient and provider, capturing in real time any technical issues experienced by either party. Our study participants were racially and ethnically diverse within a predominantly underserved population with the majority having at least a high school education. As the study was conducted within a teaching clinic in an academic medical center, we were able to obtain diverse provider perspectives across a variety of levels of training and experience, representing a real-world outpatient clinical environment.
Since the outpatient gynecology clinic where our study was conducted is NP and trainee run, under supervision of attending OBGYN physicians, we did not survey attending physicians since they did not directly carry out any of the telemedicine visits. Although this is a limitation in our findings regarding provider perspectives, we don’t expect this to have a significant bearing on our findings in that attending physicians are likely to have similarly positive perspectives as the experienced NPs surveyed in our study. Other limitations to our study include selection bias that may have impacted which patients chose to participate and ultimately completed the survey. Additionally, non-English speaking patients were not eligible to participate thus limiting our ability to understand non-English speakers’ perceptions of telemedicine. Lastly, eligible patients were identified if they had an already scheduled telehealth appointment; therefore, patients who may have declined in scheduling a telemedicine appointment for a variety of reasons, such as lack of necessary technology or discomfort with the platform, are not captured. Future studies should specifically include patients who opt out of telehealth visits to better understand their perspectives and determine if there are specific factors limiting utilization.
Despite telemedicine's existence long before the pandemic, COVID-19 led to the rapid, expansive uptake of outpatient telemedicine utilization across specialties throughout the United States. As we move into the post-pandemic era, it is almost certain that telemedicine is here to stay as a standard healthcare delivery tool. Our study's findings share valuable perspectives from providers as well as a racially and ethnically diverse, predominantly low-income patient population receiving outpatient gynecologic care. Overall telemedicine was viewed favorably by both patients and providers. Telemedicine should be considered as an appropriate option for patients presenting for a range of gynecologic follow-up visits, to improve access to gynecologic care and help reduce costs without compromising quality.
Supplemental Material
sj-docx-1-jtt-10.1177_1357633X231197965 - Supplemental material for Patient and provider perspectives on telemedicine use in an outpatient gynecologic clinic serving a diverse, low-income population
Supplemental material, sj-docx-1-jtt-10.1177_1357633X231197965 for Patient and provider perspectives on telemedicine use in an outpatient gynecologic clinic serving a diverse, low-income population by Amita Kulkarni, Ngozi Monu, Muhammad D Ahsan, Chimsom Orakuwue, Xiaoyue Ma, Auja McDougale, Melissa K Frey, Kevin Holcomb, Evelyn Cantillo, and Eloise Chapman-Davis in Journal of Telemedicine and Telecare
Supplemental Material
sj-docx-2-jtt-10.1177_1357633X231197965 - Supplemental material for Patient and provider perspectives on telemedicine use in an outpatient gynecologic clinic serving a diverse, low-income population
Supplemental material, sj-docx-2-jtt-10.1177_1357633X231197965 for Patient and provider perspectives on telemedicine use in an outpatient gynecologic clinic serving a diverse, low-income population by Amita Kulkarni, Ngozi Monu, Muhammad D Ahsan, Chimsom Orakuwue, Xiaoyue Ma, Auja McDougale, Melissa K Frey, Kevin Holcomb, Evelyn Cantillo, and Eloise Chapman-Davis in Journal of Telemedicine and Telecare
Footnotes
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Weill Cornell COVID-19 Diversity Center of Excellence Grant. The author Melissa K. Frey was supported by the following grants NIH/NCATS Grant # KL2-TR-002385.
Ethical approval
This study was approved by the Weill Cornell Medical College Institutional Review Board (IRB# 20-11022921).
Data availability statement
Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
