Validation of Newly Developed Surveys to Evaluate Patients' and Providers' Satisfaction with Telehealth Obstetric Services

Abstract

Background:

Patient and/or provider satisfaction and experience are among the most important indicators for quality assurance of health care services, including telehealth. Validated surveys should be used for this purpose to provide reliable information for a program evaluation.

Objective:

To validate the newly developed satisfaction surveys, report patient, and provider satisfaction with Antenatal and Neonatal Guidelines, Education and Learning Systems (ANGELS) telehealth services.

Methods:

Two self-administered paper surveys were developed by a multidisciplinary team. The surveys were validated among obstetric patients who received telehealth services in 2016 and providers in Arkansas from July to August 2017. Psychometric testing was performed to establish reliability and validity of both the surveys. Descriptive statistics was performed to describe patient and provider satisfaction.

Results:

A total of 89 patient- and 66 provider surveys were analyzed. Construct validity and internal consistency reliability (Cronbach's α >0.7) were confirmed on both the surveys. The majority of patients were highly satisfied with telehealth services and reported positive perceptions toward future use of services. In the past 12 months, telehealth (78.6%) was the most used ANGELS service by providers. Finally, >90% of the providers reported high satisfaction and rated telehealth services as “excellent.”

Conclusions:

The newly developed patient and provider telehealth surveys were reliable and valid. The surveys also demonstrated that the telehealth services were well-accepted and highly valued in a sample of obstetric patients and providers. As both surveys are short, they can be used in routine evaluation of patient and provider satisfaction with similar telehealth programs in the country.

Introduction

Access to health care can be challenging and/or limited for individuals residing in rural/remote areas of the country; however, the advent of telemedicine/telehealth appears to have the potential to address this issue.¹ In obstetrics/gynecology (OB/GYN) area, this mode of health care delivery has been used in providing several services, including perinatal care, psychological health care, monitoring chronic and high-risk conditions during pregnancy, and so on.² In addition to improving access to care, telehealth could lower health care expenditures and increase productivity.^2,3 Even health care providers could benefit from such programs as they reduce their travel for outreach activities, provide care directly from their offices, etc.² One of such programs with a broad range of services for obstetric patients and OB/GYN providers is the Antenatal and Neonatal Guidelines, Education and Learning Systems (ANGELS) program in Arkansas, a state with 84% nonmetropolitan counties.⁴ Being the first of its kind in the nation, the ANGELS program was introduced to improve the delivery of obstetric care to the rural and remote areas of the state and to monitor high-risk pregnancies for timely referral to the board-certified maternal-fetal medicine (MFM) experts.^5,6

An important feature of ANGELS services is telehealth/telemedicine consultations for obstetric patients, especially for those located in rural areas, using advanced information technology. ANGELS telehealth expands health care access to pregnant women in Arkansas and makes health care services more convenient for patients residing in rural areas, those with low income, and those with limited transportation. Literature suggests that implementation of ANGELS program has resulted in improved patient access to obstetric care and better health outcomes.^7

–11 Likewise, utilization of ANGELS services has also increased. In 2016, a total of 3,309 telehealth visits were provided, of which ∼56% were related to “ultrasounds” among pregnant women.¹² There was also a rise in number of calls managed by ANGELS call center in 2016, managing >175,000 calls related to clinic appointment and referrals for MFM evaluations.¹² The overall impact of ANGELS has been positive for Arkansas and there has been a downward trend in number of neonatal deaths after the implementation of ANGELS.¹²

As per the Healthcare Effectiveness Data and Information Set measures, consumer (patients or providers) satisfaction and consumer experience are among the most important indicators for quality assurance of health care services.^13,14 In addition, satisfaction is the voice of consumers to report whether their needs and expectations are met, indicating a positive appraisal of services utilized. Measuring patient/provider satisfaction toward health care services is essential because unsatisfied consumers may refrain from reusing those services. However, patients' satisfaction with ANGELS telemedicine has only been reported once after the introduction of ANGELS.⁶ In addition, evaluation of provider satisfaction with ANGELS telehealth has never been previously reported. Available literature largely describes positive satisfaction with other such programs by patients and providers; however, most studies are unrelated to OB/GYN and are also methodologically deficient limiting the generalizability of findings.^15
–17 One of the major criticisms has been regarding the use of various survey instruments to capture patient/provider satisfaction without reporting or testing validity and reliability of such instruments.^16

–20 However, the use of validated surveys has been recommended for evaluation of consumer satisfaction overtime for any given population of interest.^15,16,20

Herein, we systematically tested reliability and validity of two newly developed surveys to evaluate patients' and OB/GYN providers' satisfaction with telehealth services provided through ANGELS program in Arkansas. The use of validated consumer satisfaction telehealth instruments is essential in obtaining comparable evaluations in the long-term quality assurance of telehealth services.

Methods

Survey Instruments

The research team developed two self-administered surveys, one for patients and another one for OB/GYN providers, which were revised as per the feedback received from OB/GYN clinicians and other staff. Both surveys were paper-and-pencil based and were designed to be short such that they could be easily completed within 5 min. The patient survey (Supplementary Appendix A) was composed of an 11-item telehealth satisfaction questionnaire (ANGELS PSTQ) and a demographic section. Similarly, the OB/GYN provider survey (Supplementary Appendix B) contained a 9-item telehealth satisfaction questionnaire (ANGELS PrSTQ), type and frequency of ANGELS services used within past 12 months, and a demographic section. Statements in the satisfaction section of patient (ANGELS PSTQ) and provider (ANGELS PrSTQ) surveys were scored on a 5-point Likert scale ranging from “disagree entirely (1)” to “agree entirely (5).” Face and content validities for both instruments were established among the research team before mailing them out to consumers.

Design and Study Participants

This cross-sectional, self-administered mailed survey study was conducted among patients and OB/GYN providers in Arkansas during July–August 2017. Patient surveys were sent to those who received telehealth services in 2016, and provider surveys were sent to OB/GYN providers who practiced in Arkansas. The study was determined as a quality improvement project by the authors' institutional review board.

Data Collection

The process of data collection was guided by the modified Dillman's method for mail survey.^21,22 The survey package included a personalized cover letter, the survey, research team's contact details, a request for contact information for follow-up questions, and a prepaid return envelope for a completed survey. Two reminder cards were sent to participants at 1- and 2-week periods after the first contact.

Statistical Analyses

All the information from returned paper surveys were transferred into MS Excel^® spreadsheets using Remark^© software (Gravic, Inc., Malvern, PA) and checked for data validity by two research assistants. For all the descriptive statistics, the survey response categories for satisfaction section in both patient and provider surveys were collapsed into following categories: “disagree” (“disagree for the most part” and “disagree entirely”), “undecided or do not know,” and “agree” (“agree for the most part” and “agree entirely”). Descriptive statistics was generated for demographics and satisfaction toward telehealth services using SAS v9.4 (SAS Institute, Inc., Cary, NC).

Reliability of the surveys was measured by internal consistency (Cronbach's α), which in turn measures how well each item correlates with other items in the scale (or a construct) and with the total scale score. Cronbach's α ≥0.70 is considered as acceptable internal consistency reliability.²³ Construct (structural) validity tests the number of construct(s) captured by the items in a given instrument. Both exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were performed to test the instrument structure. The goodness-of-fit (GoF) criteria were considered for judging the model fit. The GoF indices included root mean square error of approximation (RMSEA), RMSEA 90% confidence interval (CI), comparative fit index (CFI), and standardized root mean square (SRMR). RMSEA (<0.05), RMSEA 90% CI (0.00–0.08), CFI (>0.9), and SRMR (<0.08) are considered acceptable model fit measures.²⁴ All items from satisfaction sections of patient (ANGELS PSTQ) and provider (ANGELS PrSTQ) surveys were theorized to measure one concept, respectively. These analyses were conducted in Stata v15 (StatCorp LLC., College Station, TX).

Results

Patients' Demographics

Ninety-one patients returned the survey (response rate = 7.3%). A plausible reason for such low response is that 360 surveys were undelivered because patients either had moved or had invalid addresses in our database. Among the returned surveys, two were incomplete and were excluded from the final analysis (N = 89). Table 1 presents the demographics of patients. The age of patients was similarly distributed across four age groups with 60.5% patients being 30 years or older. Most patients were white (84.9%), and 31.3% reported annual household income of ≤$25,000. Only 24.4% reported having high school education or lower. Approximately half (51.2%) reported having commercial insurance, and 27.9% were Medicaid beneficiaries.

Table 1.

Patient Demographics

	n (%)
Age, years
<24	10 (11.6)
24–29	24 (27.9)
30–35	21 (24.4)
>35	31 (36.1)
Missing	3
Race
White	73 (84.9)
Non-White^a	13 (15.1)
Missing	3
Household annual income
≤$25,000	26 (31.3)
$25,001–$50,000	14 (16.9)
$50,001–$75,000	20 (24.1)
>$75,000	23 (27.7)
Missing	6
Education
High school or less	21 (24.4)
Some college/vocational school	25 (29.1)
Bachelor's or graduate degree	40 (46.5)
Missing	3
Insurance
Medicaid	24 (27.9)
Commercial	44 (51.2)
Other	18 (20.9)
Missing	3

Includes Black, Hispanic, Asian, and other.

Use of Angels Telehealth by Patients

Table 2 presents information about use of telehealth by surveyed patients. Most patients (95.3%) reported using telehealth one to two times within past 12 months. About three-fourth patients reported that they traveled <30 miles to attend a telehealth session and that they would have had to travel >70 miles for a clinic visit had they not used telehealth. About half of the patients reported that they would have had to spend >$150 out of pocket if they had not used telehealth. The majority (79.5%) also reported that they would have missed ≥1 day of work if they had not used telehealth.

Table 2.

Use of Antenatal and Neonatal Guidelines, Education and Learning Systems Telehealth by Patients

	n (%)
Frequency of using telehealth in past 12 months
1–2 Times	82 (95.3)
3–4 Times	4 (4.7)
Missing	3
Miles traveled to attend telehealth session
<30 Miles	65 (76.5)
31–50 Miles	18 (21.2)
51–70 Miles	2 (2.3)
Missing	4
Miles traveled if not used telehealth
<30 Miles	6 (7.1)
31–50 Miles	7 (8.4)
51–70 Miles	6 (7.1)
>70 Miles	65 (77.4)
Missing	5
Cost incurred when used other service instead of a telehealth session
<$35	10 (11.8)
$35–$75	11 (12.9)
$76–$150	21 (24.7)
$151–$300	28 (32.9)
>$300	15 (17.7)
Missing	4
Number of work days missed if not used telehealth session
0 Day	14 (16.9)
0.5 Day	3 (3.6)
1 Day	57 (68.7)
2 Days	8 (9.6)
>2 Days	1 (1.2)
Missing	6

OB/GYN Providers' Demographics

Ninety-eight OB/GYN providers returned the survey (response rate = 17.9%). Among those, 14 (14.3%) providers reported not using any of the ANGELS services within past 12 months (Table 3). Providers who had not used ANGELS services during past 12 months were more likely to be older (≥55 years) versus those who used ANGELS services (71.4% vs. 39.3%). More than half of the providers who reported using ANGELS services were physicians (56.8%), while most providers (57.1%) who had not used ANGELS services were nurses. Because the number of providers who had not used ANGELS services was small (N = 14), there was insufficient power to test for significant differences between the groups. In addition, as the purpose of this study was to assess the psychometric properties of newly developed survey and to evaluate satisfaction with telehealth, only OB/GYN providers who used telehealth services within past 12 months were analyzed further (N = 66).

Table 3.

Demographic Characteristics of Arkansas' Obstetrics/Gynecology Providers

	USED ANGELS IN PAST 12 MONTHS, N = 84	DID NOT USE ANGELS IN PAST 12 MONTHS, N = 14
	n (%)	n (%)
Age, years
25–34	5 (6.3)	2 (14.3)
35–44	23 (29.1)	2 (14.3)
45–54	20 (25.3)	—
55–64	24 (30.4)	6 (42.9)
>64	7 (8.9)	4 (28.5)
Missing	5	—
Gender
Male	36 (44.4)	3 (21.4)
Female	45 (55.6)	11 (78.6)
Missing	3	—
Race
White	72 (88.9)	14 (100)
Non-White^a	9 (11.1)	—
Missing	3	—
Profession
MD	46 (56.8)	4 (28.6)
APRN	5 (6.2)	2 (14.3)
RN	27 (33.3)	8 (57.1)
Sonographer	2 (2.5)	—
Other	1 (1.2)	—
Missing	3	—
Years of practice
<1	1 (1.2)	—
1–2	1 (1.2)	1 (7.1)
3–4	2 (2.4)	1 (7.1)
5–10	15 (18.3)	—
11–20	24 (29.3)	2 (14.3)
>20	39 (47.6)	10 (71.5)
Missing	2	—
Nature of practice
Private obstetrician/gynecologist	27 (32.1)	1 (7.1)
State supported institution (including ADH)	28 (33.3)	10 (71.5)
Private general/family medicine	6 (7.2)	—
Hospital or hospital system	17 (20.2)	—
Other	6 (7.2)	3 (21.4)
Practice region
Central AR	20 (24.1)	2 (14.3)
NW AR	14 (16.9)	4 (28.6)
SW AR	15 (18.1)	1 (7.1)
NE AR	24 (28.9)	2 (14.3)
SE AR	10 (12)	5 (35.7)
Missing	1	—

Includes Black, Hispanic, Asian, and other.

ADH, Arkansas Department of Health; ANGELS, Antenatal and Neonatal Guidelines, Education and Learning Systems; APRN, advanced practice registered nurse; Central AR, Central Arkansas; MD, doctor of medicine; NE AR, Northeast Arkansas; NW AR, Northwest Arkansas; RN, registered nurse; SE AR, Southeast Arkansas; SW AR, Southwest Arkansas.

Use of Angels Services by OB/GYN Providers

Most providers who reported using ANGELS services during past 12 months were 35–64 years old (84.8%). Half were females (55.6%), and 88.9% were White. The providers were mostly physicians (56.8%) or registered nurses (33.3%). Almost half of the providers reported having >20 years of professional experience (47.6%). Providers also reported broad range of practice types, including private OB/GYN clinics (32.1%), state-supported (33.3%), hospital or hospital systems (20.2%), and private general or family medicine (7.2%). There were similar responses from providers in every region in Arkansas, with most being from central (24.1%) and northeast (28.9%) Arkansas.

Frequently used ANGELS services during past 12 months (Table 4) included telehealth (78.6%), call center (56%), evidence-based guidelines (51.2%), and education (35.7%). In addition, >65% of the providers were frequent users (>6 times) of ANGELS services in the past year.

Table 4.

Use of Antenatal and Neonatal Guidelines, Education and Learning Systems Services in the Past 12 Months

	n (%)
ANGELS services used in past 12 months
None	—
Telemedicine	66 (78.6)
Physician call center	47 (56)
ANGELS evidence-based guidelines	43 (51.2)
ANGELS education	30 (35.7)
Other	15 (17.9)
Colposcopy clinic	9 (10.7)
Frequency of using ANGELS services in past 12 months
1–2	3 (3.6)
3–4	5 (6)
5–6	20 (24.1)
>6	55 (66.3)
Missing	1

Validity and Reliability of Satisfaction Surveys

Patient survey

EFA and CFA were conducted to test the structure of ANGELS telehealth evaluation section from the ANGELS PSTQ. The EFA suggested a two-factor model after excluding three items (q5, q10, and q11). The CFA confirmed the two-dimensional structure of the questionnaire, with acceptable model fit indices (RMSEA: 0.055, 90% CI: 0.000–0.118, SRMR: 0.059, and CFI: 0.971). The two factors were as follows: (1) “Perceived Convenience with Use of ANGELS Telehealth” (q1, q2, and q6) and (2) “Perceived Benefit of Using ANGELS Telehealth” (q3, q4, q7, q8, and q9). The Cronbach's α scores for the two factors were 0.64 and 0.75, respectively, indicating moderate-to-high internal consistency reliability. Figure 1 presents the two-factor model for ANGELS PSTQ.

Fig. 1.

Two-factor model for ANGELS telehealth evaluation by patients (ANGELS PSTQ). ANGELS, Antenatal and Neonatal Guidelines, Education and Learning Systems.

OB/GYN provider survey

The EFA and CFA were conducted to test the structure of ANGELS telehealth evaluation section from the ANGELS PrSTQ. The EFA suggested a two-factor model after excluding one item (q6). The CFA confirmed the bidimensionality of the questionnaire with acceptable model fit indices (RMSEA: 0.040, 90% CI: 0.000–0.124, SRMR): 0.035, and CFI: 0.996). Among the two factors, one factor captured the OB/GYN provider's “Perceived Usefulness of ANGELS Telehealth Specific to the Provider's Patients” (q2, q3, and q4), and the other factor captured the OB/GYN provider's “Perceived Benefit of ANGELS Telehealth in General” (q1, q5, q7, q8, and q9). Both factors indicated high internal consistency reliability with Cronbach's α values of 0.88 and 0.85, respectively. Figure 2 presents the bidimensional model for ANGELS PrSTQ.

Fig. 2.

Two-factor model for ANGELS telehealth evaluation by obstetrics/gynecology providers (ANGELS PrSTQ).

Satisfaction with Angels Telehealth

Patients

Figure 3 presents patients' satisfaction with ANGELS telehealth. Overall, most patients reported being satisfied with the telehealth system (98.8%) and reported positive perceptions toward using it in future (95.3%). Almost all patients (98.8%) believed that “…telehealth is an acceptable way to receive health care services” and they agreed (98.8%) that “…it was easier to talk to the doctor using telehealth system.” Most patients (96.5%) agreed that “…telehealth saved them [me] time traveling to a hospital or specialist clinic,” and 91.8% reported, “Telehealth made it easier for them [me] to see doctor or other specialists.” Most (96.5%) also felt that visits provided over telehealth were as good as in-person visits by agreeing to the statement—”…using the telehealth system, they [I] could see the doctor as well as if they [we] met in person.” Approximately half of the patients (47.7%) agreed that “….without telehealth, they [I] would have not been able to get the pregnancy care they [I] needed.” However, all patients agreed that “…it is important that Arkansas has a telehealth program like ANGELS to serve pregnant women.”

Fig. 3.

ANGELS telehealth evaluation by patients.

Supplementary Appendix Figure S1 presents ANGELS telehealth evaluation of patients, categorized by types of insurance. The level of agreement to many statements was similar across patients with different types of insurance. However, a greater proportion of Medicaid beneficiaries agreed that “…without telehealth, they [I] would have not been able to get the pregnancy care they [I] needed” compared to those with commercial or other insurance (70.8% vs. 34.1% vs. 50%, respectively).

OB/GYN providers

Figure 4 presents the perceptions of providers toward ANGELS telehealth system. Almost all providers (98.5%) agreed that telehealth system is a positive program for Arkansas. They overwhelmingly agreed that telehealth is an acceptable way to provide health care service (98.5%), it improves access to health care needs (97.1%), and it improves the lives of their patients (92.5%). They also agreed that telehealth system is an adequate replacement when in-person visits are difficult or impossible (93.9%). Almost all providers rated the telehealth system as excellent (98.5%).

Fig. 4.

ANGELS telehealth evaluation by providers.

The providers' level of agreement with telehealth evaluation statements was also similar across providers from different regions (Supplementary Appendix Fig. S2). Although all providers practicing in Southeast Arkansas (n = 5) reported that they liked using the telehealth system, one disagreed with the following statement: “I think visits provided over the telehealth system are adequate replacements when in-person visits are difficult or impossible.” Furthermore, 1/19 providers from Central Arkansas disagreed to rate ANGELS telehealth services as excellent.

Discussion

This study established psychometric properties of newly developed surveys, which addresses a crucial limitation of the current literature around this topic. Both the surveys in this study were designed to be short such that they could be easily completed in <5 min. The section on ANGELS telehealth evaluation of patient survey measured two related concepts with moderate-to-high internal consistency reliability. Likewise, the section on ANGELS telehealth evaluation of provider survey measured two related concepts with high internal consistency reliability. Overall, both surveys described in this study are valid and reliable measures to evaluate satisfaction with telehealth services. These instruments can also serve to be useful quality assurance tools for future evaluation of other similar telehealth programs in the country. Importantly, these surveys can be adopted to evaluate health care programs or services, not restricted to only telehealth, and can be easily modified to be delivered in different administration modes such as web-based administration.

Introduction of ANGELS program assured access to high quality perinatal care to eligible obstetric women in Arkansas and empowered local health care providers. Together with the core components and by utilizing the expertise of the board-certified MFM specialists, the ANGELS program has the potential to improve and provide convenient access to health care, reduce neonatal complications, and provide proven obstetric care by promoting the use of evidence-based guidelines. The ANGELS model serves to be replicable for other rural states to improve patient health, particularly for patients with limited access to obstetric care in the country.²⁵

This study presented perceived value of its services from the perspectives of patients and OB/GYN providers. We found that the telehealth services provided by ANGELS were well-accepted and highly valued by patients and providers. Consumers were overwhelmingly satisfied with telehealth by indicating high-perceived usefulness, benefit, and convenience with telehealth program. This high satisfaction from patients is consistent with a previous report on patient satisfaction with ANGELS telehealth,⁶ as well as the large literature about satisfaction with telemedicine/telehealth in other areas of medicine.^15
–17 Notably, in this current study all patients agreed that telehealth was important for pregnant women especially in a rural state like Arkansas. The fact that patients would have had to travel ≥70 miles for their clinic visits if there was no telehealth option strengthens the existence and importance of such program in a rural state. A longer travel for patients was also associated with increased travel costs and absence from work affecting productivity. Similar reasons for use of telehealth and patient satisfaction with telehealth in other areas of medicine have been documented previously.³ The OB/GYN providers also reported that they would use telehealth services again in future and rated ANGELS telehealth as excellent. With high patient and provider satisfaction toward telehealth services in this study, implementation of ANGELS-like telehealth services in other rural parts of the country could benefit vulnerable patients with limited access and greatly improve OB/GYN-related health outcomes in the country.^2,4

Given that the survey instruments are valid and reliable, we intend to collect patient evaluation of various telehealth services provided in our program on a prospective basis. This will address some of the limitations of the current study, including lower response rate and potential recall bias because of retrospective study design. There are additional limitations of this study. We were limited by power to compare characteristics of providers who reported using ANGELS services within past 12 months versus those who did not. The low sample size also limited our ability to conduct regression analysis to determine characteristics associated with high satisfaction toward telehealth services. Furthermore, the convenience sample of study participants also limits the generalizability of findings beyond the study sample. Despite the limitations, taken together with improved outcomes, high consumer satisfaction, and validated survey instruments, other rural states with underserved population could implement ANGELS-like services for better patient health and adopt these surveys for evaluation of their services.

Overall, both patients and OB/GYN providers in the rural state reported greater satisfaction with telehealth services. The results from this descriptive study provide important information about the quality of the health care services provided by an OB/GYN-based telehealth, which could serve as a model for replication by other rural states in the country. Finally, the validated surveys from this study could be readily adopted in evaluating satisfaction with other similar telehealth programs.

Footnotes

Disclosure Statement

No competing financial interests exist.

Funding Information

This study did not receive any funding; however, we acknowledge Arkansas Medicaid for funding the ANGELS program.

Supplementary Material

Supplementary Appendix S1

Supplementary Appendix S2

Supplementary Appendix Figure S1

Supplementary Appendix Figure S2

References

Doarn

. Telehealth: A very useful tool that enables and improves patient access. J Am Board Fam Med, 2016; 29:430–431.

Greiner

. Telemedicine applications in obstetrics and gynecology. Clin Obstet Gynecol, 2017; 60:853–866.

Kruse

, Krowski

, Rodriguez

, et al. Telehealth and patient satisfaction: A systematic review and narrative analysis. BMJ Open, 2017; 7:e016242.

Rhoads

, Eswaran

, Lynch

, et al. High-risk obstetrical call center: A model for regions with limited access to care. J Matern Neonatal Med, 2018; 31:857–865.

UAMS, Arkansas Department of Human Services

S launch nation's first ‘ANGELS’program to reduce premature births. J Ark Med Soc 2003;100:138–139.

Lowery

, Bronstein

, McGhee

, et al. ANGELS and University of Arkansas for Medical Sciences paradigm for distant obstetrical care delivery. Am J Obstet Gynecol, 2007; 196:534.e1–534.e9.

Bronstein

, Ounpraseuth

, Jonkman

, et al. Improving perinatal regionalization for preterm deliveries in a Medicaid covered population: Initial impact of the Arkansas ANGELS intervention. Health Serv Res, 2011; 46:1082–103.

Bronstein

, Ounpraseuth

, Jonkman

, et al. Use of specialty OB consults during high-risk pregnancies in a Medicaid-covered population. Med Care Res Rev, 2012; 69:699–720.

Long

, Angtuaco

, Lowery

. Ultrasound in telemedicine. Ultrasound Q, 2014; 30:167–172.

10.

Rabie

, Canon

, Patel

, et al. Prenatal diagnosis and telemedicine consultation of fetal urologic disorders. J Telemed Telecare, 2016; 22:234–237.

11.

Britt

, Norton

, Hubanks

, et al. A two-period assessment of changes in specialist contact in a high-risk pregnancy telemedical program. Telemed J E Health, 2006; 12:35–41.

12.

ANGELS 2016 Annual Report. Little Rock, AR, 2017. Available at https://angels.uams.edu/wp-content/uploads/sites/81/2010/12/2016-Angels-annual-report-2-revised-1.pdf (last accessed June 18, 2019 ).

13.

Gupta

, Rokade

. Importance of quality in health care sector. J Health Manag, 2016; 18:84–94.

14.

National Committee for Quality Assurance. HEDIS^® and Performance Measurement. Available at www.ncqa.org/hedis-quality-measurement (last accessed June 18, 2019 ).

15.

Mair

, Whitten

. Systematic review of studies of patient satisfaction with telemedicine. BMJ, 2000; 320:1517–1520.

16.

Williams

, May

, Esmail

. Limitations of patient satisfaction studies in telehealthcare: A systematic review of the literature. Telemed J E Health, 2001; 7:293–316.

17.

van den Heuvel

, Groenhof

, Veerbeek

, et al. eHealth as the next-generation perinatal care: An overview of the literature. J Med Internet Res, 2018; 20:e202.

18.

Wakefield

, Turvey

, Nazi

, et al. Psychometric properties of patient-facing eHealth evaluation measures: Systematic review and analysis. J Med Internet Res, 2017; 19:e346.

19.

Whitten

, Love

. Patient and provider satisfaction with the use of telemedicine: Overview and rationale for cautious enthusiasm. J Postgrad Med, 2005; 51:294–300.

20.

Allemann Iseli

, Kunz

, Blozik

. Instruments to assess patient satisfaction after teleconsultation and triage: A systematic review. Patient Prefer Adherence, 2014; 8:893.

21.

Dillman

. The design and administration of mail surveys. Annu Rev Sociol, 1991; 17:225–249.

22.

Hoddinott

, Bass

. The dillman total design survey method. Can Fam Physician, 1986; 32:2366–2268.

23.

Nunnally

, Bernstein

, Berge

. Psychometric theory. New York, NY: McGraw-Hill, 1967.

24.

Schreiber

, Nora

, Stage

, et al. Reporting structural equation modeling and confirmatory factor analysis results: A review. J Educ Res, 2006; 99:323–338.

25.

ANGELS 2013 Annual Report. Little Rock, AR, 2014. Available at https://www.brookings.edu/wp-content/uploads/2014/11/rural_healthcare2.pdf (last accessed June 18, 2019 ).

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.04 MB

0.07 MB

0.76 MB

0.59 MB

0.00 MB