Rapid Development of a Telehealth Patient Satisfaction Survey Using a Multi-Stakeholder Approach

Abstract

Introduction:

The COVID-19 pandemic has hastened the adoption of telehealth and the drastic shift to an unfamiliar process may impose significant impact to the quality-of-care delivery. Many providers are interested in understanding the quality of their telehealth services from the patients' experience.

Materials and Methods:

A telehealth patient satisfaction survey (TPSS) was developed by using an iterative stakeholder-centered design approach, incorporating elements from validated telemedicine and customer service survey instruments, and meeting the operational needs and constraints. A cross-sectional study design was employed to collect survey responses from patients and families of a large pediatric hospital. Finally, we performed exploratory factor analysis (EFA) to extract latent constructs and factor loadings of the survey items to further explain relationships.

Results:

A 22-item TPSS closely matched the existing in-person patient satisfaction survey and mapped to a revised SERVPERF conceptual model that was proposed by the interdisciplinary committee. Survey was implemented in the HIPAA-compliant online platform REDCap^® with survey link embedded in an automated Epic MyChart (Verona, WI) visit follow-up message. In total, 2,394 survey responses were collected between July 7, 2020, and September 2, 2020. EFA revealed three constructs (with factor loadings >0.30): admission process, perceived quality of services, and telehealth satisfaction.

Conclusions:

We reported the development of TPSS that met the operational needs of compatibility with existing data and possible comparison to in-person survey. The survey is short and yet covers both the clinical experience and telehealth usability, with acceptable survey validity.

Introduction

As the pandemic of COVID-19 swept across the globe, most businesses and services shifted to online platforms to reduce risk of exposure. This also led to a wave of rapid adoption of telehealth services by the health care organizations, a sector that has been slow to adopt new technologies.¹ To rapidly expand the use of telehealth across the United States, the Center for Medicare and Medicaid Services issued an emergency declaration of waivers, which removed provider licensing mandates, patient-provider established relationship requirements, and requirements related to location of telehealth offerings.² The Office of Civil Rights waived software privacy requirements, allowing hospital systems to provide telehealth with nonpublic facing video chat softwares, thereby reduced the adoption barriers.^3,4

With the drastic change in health care delivery from in-person to virtual, health care organizations and their providers want to know their patients' telehealth experience and use the feedback to improve telehealth service delivery. There are some existing telehealth surveys, such as the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ),⁵ Telemedicine Satisfaction Survey (TSQ),⁶ and Telehealth Usability Questionnaire (TUQ).⁷ Unlike the traditional patient satisfaction survey, these instruments revolve around the telemedicine modality with strong emphasis placed on the technology and have limited utility in comparison between telehealth and in-person visits. Moreover, these surveys may not be compatible with previously collected in-person visit satisfaction data.

In this article, we share a survey content design process that consolidated the existing telehealth surveys into core data elements, allowing for backward compatibility with in-person visit satisfaction data collection, and prioritized rapid implementation by balancing the survey validity with operational need. To explain the item relationships to further investigate patient attitude and conceptualize the satisfaction, we performed exploratory factor analysis (EFA) to extract latent factors that might reflect the underlying dimensions of telehealth patient satisfaction.

Approach

SURVEY DEVELOPMENT APPROACH

A working group consisting of key stakeholders from administrative, operation, and clinical departments was formed to inform the development of the survey (n = 21). Survey was developed through the following steps: (1) understand operational needs, constraints, and existing efforts on patient satisfaction surveys across the organization; (2) literature review on theoretical framework on patient satisfaction; (3) survey content development; and (4) review and revision. The general process and timeline are depicted in Figure 1.

Fig. 1.

Timeline for survey development.

This study has been reviewed and approved by the Nationwide Children's Institutional Review Board (IRB No. 00001425).

LITERATURE REVIEW ON THEORETICAL FRAMEWORK

We conducted a rapid literature review on the conceptual models for patient satisfaction to develop a survey that matches well to a validated conceptual model. Due to time constraint, we did not conduct an exhaustive literature search, but instead handpicked publications based on conceptual model diagrams, instrument, and validation details provided.^{7

–23} In addition to validation of the conceptual model, the working group considered other factors that may impact operational implementation, such as the frequency and length of the survey. These considerations are provided in Supplementary Table S1.

Following stakeholder deliberation, we decided to adapt the SERVPERF model as the theoretical framework on which to further develop our survey. We replaced the “tangible” construct in SERVPERF with telehealth usability to account for the change from physical to digital environment.

SURVEY ITEM MAPPING AND CONTENT DEVELOPMENT

We mapped the survey questions from the patient satisfaction surveys already developed by some of our clinical departments (four in total) (Supplementary Table S2) to the constructs in the adapted SERVPERF model through group consensus. This allowed us to consolidate similar items and identify gaps in survey coverage. For the constructs that are not well covered, we added items from other developed survey instruments to complete our initial survey design.

SURVEY REVIEW AND REVISION

To finalize the survey design, we presented the initial survey design to the working group, hospital leadership, as well as the NCH family advisory council. Feedback received was discussed with the working group in an iterative process. The finalized survey was reviewed by our literacy and education specialist for readability.

The survey was presented to the working group (n = 21) and the NCH Family Advisory Council (n = 37, including 24 parents) for review and revision. Key feedback from the working group included potential challenges encountered during telehealth visits as well as addressing ambiguity in the questions. Review by the Family Advisory Council ensures that the language and length of the survey are appropriate for our patients to complete with reasonable ease. The readability of the survey met the recommended sixth grade level.²⁴ In addition, the survey content was translated into four other predominant languages of our patient population, including Spanish, Somali, Nepali, and Arabic (Supplementary Table S3).

SURVEY DISTRIBUTION

Telehealth patient satisfaction survey (TPSS) was delivered automatically through HIPAA-compliant REDCap^® Platform with unique survey link sent through MyChart Patient Portal messages after a video encounter was closed. To prevent survey fatigue, each patient was limited to receive one survey invitation in any 2-week period, regardless of number of visits. The survey can be completed by the patient or their caregivers. Each survey is linked to one specific encounter, with date and clinical department noted on the survey. In addition to 16 survey items to be validated as measures of patient satisfaction constructs (Table 1), some demographic questions were collected in the survey as well as directly extracted from the electronic medical record.

Table 1.

Telehealth Patient Satisfaction Survey

ITEM	RESPONSE OPTIONS	ADAPTED SERVPERF CONSTRUCT
I understand the plan of care, which includes why tests and procedures are needed^a	5-Point Likert Scale, NA	Assurance
I was given home care instructions and able to ask questions^a	5-Point Likert Scale, NA	Assurance
The staff was polite and kind^a	5-Point Likert Scale	Assurance
It was easy to schedule my appointment^a	5-Point Likert Scale	Empathy
I feel that I was included in the health care decisions^a	5-Point Likert Scale	Empathy
I know my care and/or my child's care is important to the staff^a	5-Point Likert Scale	Empathy
The provider was able to diagnose problems and treat my or my child's condition^b	5-Point Likert Scale, NA	Reliability
Compared to the level of care received during an in-person visit, the level of care received during the telemedicine appointment was^b:	Better/equal/worse/not sure/no in-person visit before	Reliability
If I had a long wait time, I was kept informed of those delays during my visit^a	5-Point Likert Scale, NA	Responsiveness
My or my child's needs were met promptly^c	5-Point Likert Scale	Responsiveness
I was happy with the check-in process for my appointment^a	5-Point Likert Scale	Usability
I was satisfied with the quality of the video connection during the visit^d	5-Point Likert Scale	Usability
Telehealth improves my access to health care services^d	5-Point Likert Scale	Usability
My overall experience with NCH telehealth was good^d	5-Point Likert Scale	Patient satisfaction
If given the option, I would use video visits for future appointments^d	5-Point Likert Scale	Patient satisfaction
Overall, my needs were met. I would recommend others to Nationwide Children's Hospital^a	5-Point Likert Scale	Patient satisfaction
Please check all the concerns or problems you had regarding telehealth visits	Finding internet access Finding suitable digital device Finding a quiet, private place to carry out the visit Not sure how to carry out a telehealth visit Trouble logging into MyChart Trouble starting the video call (Zoom) Worried about the quality of telehealth visits Concerns about privacy Need for translation services Concerns about whether the visit would be covered by insurance COVID-19-related impact on scheduling a visit Not able to directly get written records Not comfortable with physical exposure on camera over the video visit Other	Barrier
What is your relationship to the patient?^a	Self/parent/guardian (other than biological parent)	Demographics
What is the main language you speak at home?	English/Spanish/Somali/Nepali/Arabic/Other	Demographics
What device did you use for the video visit?	Smartphone/tablet/laptop/desktop computers/NA, phone visit only	Device
Was it your personal device?	Yes/no	Device
How did you complete your telehealth visit?	By video/by phone	Device

Patient Satisfaction Survey for Ambulatory Services (AMB).

Telehealth Patient Satisfaction Survey developed by Center for Surgical Outcome Research (CSOR).

SERVPERF survey.^22,38

Telehealth Usability Questionnaire (TUQ).⁷

NA, not applicable.

Note: Superscripts denote the source survey instrument that the questions were derived from. The survey instruments referenced in this study are detailed in Supplementary Table S2.

SURVEY DATA SET

Our inclusion criteria were as follows: (1) patients of NCH and (2) patients who had completed a video visit at Nationwide Children's Hospital between July 7, 2020, and September 2, 2020. We collected 2,394 survey responses from 22,269 survey sent. Since some patients filled out the survey for different visits, we only selected their first response to ensure independent sample and reduce replicate responses for the purpose of analysis. Of the 2,394 responses, 1,938 responses fit our inclusion criteria.

DATA HANDLING

Survey responses are generally captured on a 5-point Likert scale, except the item comparing level of care between telemedicine appointment and in-person care. The responses for the item are “better/equal/worse/not sure/no in-person visit before,” which does not follow the symmetry and balance of Likert scale. To maintain uniformity of responses among all the items of the survey, we excluded this item for the EFA.²⁵ A few survey items had a “Not Applicable” (NA) response option (Table 1). We noted that one of these items, “If I had a long wait time, I was kept informed of those delays during my visit,” had 52% (∼1,244) of NA responses, which are mapped to “neither agree or disagree,” given the context of the question (Table 1).

This preserved data and prevented introducing bias by removing all those who did not have a long wait time. For the remainder of the survey questions, 482 (25%) had NA responses. For the factor analysis, we employed list wise deletion method,²⁶ that is, deleting all cases with a missing response, to handle these cases.

DATA ANALYSIS

We employed EFA to investigate the relationships between the survey items and explore item loading and grouping to inform the conceptual model.

Results

ADAPTED SERVPERF MODEL

Based on the comparison of existing conceptual models on service quality and patient satisfaction from our literature search (Supplementary Table S1), the SERVPERF model was chosen as the basis for our survey design, with modifications to address telehealth-specific components (Fig. 2).²⁷

Fig. 2.

Adapted SERVPERF conceptual model.

The adapted model consists of four “service dimension” constructs from the SERVPERF model.²⁷ “Assurance” is related to the knowledge and courtesy of staff and their ability to inspire trust and confidence. “Empathy” is related to the caring, individualized attention the organization provides its patients. “Reliability” is related to the ability to perform the promised service dependably and accurately. “Responsiveness” is related to the willingness to help patients and provide prompt service.

We replaced the “Tangibles” construct that is related to the physical facilities, appearance of personnel, and the equipment of the hospital with the use of the telehealth system. The construct of “Usability” is based on the technology acceptance model,²⁸ and is related to the perceived usefulness and perceived ease of use. For the satisfaction construct, we included questions on overall experience, as well as future service use intention and word of mouth recommendation, which are highly correlated constructs.^7,29

SURVEY DEVELOPMENT

To reduce survey burden and increase response rate, we designed the survey to be easily responded within 5 min. Two to three items were included for each construct, along with questions on the challenges and the use of technology associated with telehealth to help us perform diagnostic analysis (Table 1).

DESCRIPTION ANALYSIS OF SURVEY PARTICIPANTS

Table 2 presents the sample characteristics for 1,938 patients whose telehealth visit experience were surveyed, as well as the demographic characteristics of all telehealth patients during that period. In most cases, the survey was completed by parents (85%). Patient characteristics for the sample were mostly white (75.4%), non-Hispanic or Latino (94.7%), and speaks English (96.0%). Notably, participants ranged in age from 0 to 62 years with mean age of 10 years (standard deviation [SD] 6.29). Although NCH is a pediatric hospital, patients with certain congenital conditions may continue to receive care into adulthood, in addition to a few other scenarios that adults may be cared for at NCH.

Table 2.

Telehealth Patient and Survey Participant Demographic

PATIENT CHARACTERISTIC	SURVEY RESPONDENTS		ALL TELEHEALTH PATIENTS		CHI SQUARE TEST, p -VALUE
PATIENT CHARACTERISTIC	FREQUENCY ( N = 1,938)	PERCENTAGE	FREQUENCY ( N = 18,870)	PERCENTAGE	CHI SQUARE TEST, p -VALUE
Gender
Male	992	51.2	8,898	47.2	<0.05^a
Female	946	48.8	9,970	52.8
No information	0	0	2	0.02
Race
White	1,462	75.4	13,580	72.0	<0.05^a
Black or African American	239	12.3	2,800	14.8
Multiple race	138	7.1	1,359	7.2
Asian	54	2.8	480	2.5
Unknown	31	1.5	484	2.6
Refuse to answer	8	0.4	123	0.7
Other	6	0.3	21	0.1
Ethnicity
Not Hispanic or Latino	1,836	94.7	17,866	94.7	0.88
Hispanic or Latino	77	4.0	736	3.9
No information	25	1.3	268	1.4
Primary language
English	1,859	96.0	17,901	94.9	0.08
Spanish	29	1.5	400	2.1
Other	26	1.3	256	1.4
Arabic	14	0.7	86	0.5
Somali	4	0.2	150	0.8
Nepali	6	0.3	77	0.4
Survey respondent
Parent	1,655	85.4	NA	NA	NA
Patient	168	8.7	NA	NA
Guardian (other than biological parents)	115	5.9	NA	NA
Department specialty for the encounter
Behavioral Health	863	44.5	8,487	45.0	<0.05^a
Other	512	26.4	6,420	34.0
Speech Therapy	266	13.7	2,047	10.8
Occupational Therapy	105	5.5	597	3.2
Gastroenterology	104	5.3	930	4.9
Physical Therapy	88	4.5	389	2.1

p-Value indicating statistically significant difference between the demographic characteristics of survey respondents and all telehealth patients.

The survey responses were collected from patients visiting 43 department specialties with 44.5% for Behavioral Health (including behavioral health, psychiatry and psychology) and 23.7% for the Clinical Therapy Specialties (including speech, occupational, and physical therapies). Compared to overall telehealth patient demographics, more white male patients and their families participated in the survey. In addition, we observed higher response rates from the Clinical Therapy Specialties.

DATA DISTRIBUTION

Distribution of data and weight of outliers are important properties that can affect factor analysis. We tested the univariate normality (UVN) and multivariate normality (MVN), which were necessary to ensure before implementing Minimal Residuals (MINRES) factor extraction method (reported in the next section). UVN included mean, SD, Skewness, Kurtosis, and Shapiro-Wilk Normality Test statistics (Supplementary Table S4). The mean values of the item ranged between 1.12 and 2.25 with a low SD of 0.48–1.04. The skewness and kurtosis values of the items were above the normal range (kurtosis <7 and skewness <2), indicating that our data has violated the UVN assumption.³⁰ In addition, the Shapiro-Wilk Normality Test indicated that the data were not univariately normally distributed (p < 0.05).³¹ Similarly, using the skewness and kurtosis approach (Mardia's test of skewness and kurtosis) and consistent approach (Henze-Zirkler test) to check the MVN assumption^32,33 revealed that the data were not normally distributed.

EXPLORATORY FACTOR ANALYSIS

To extract relevant dimensions as factors, we used EFA. In the factor analysis, “groups of items that tend to be inter-related with one another more strongly than they relate to other groups of items will tend to form factors.”³⁴ We used Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy test and Bartlett's test of sphericity to test the sampling adequacy and examine whether the variables are measuring common factors or not. Our data have shown a high KMO index (0.94) and a significant Bartlett's statistic (Bartlett's K-squared = 3,279, p < 0.05),³⁵ indicating its suitability for factor analysis. Since our data violate the MVN distribution assumption, MINRES was used for factor extraction instead of the default maximum likelihood extraction method that assumes the MVN distribution of data³⁶ with Direct Oblimin rotation approach.^37,38

We expect the Eigenvalue to be >1 for factor extraction threshold³⁹ and conducted scree test with parallel analysis. The results from the Scree plot and parallel analysis indicated that our items correspond to three unobserved factors (constructs).^39,40 We considered the items with factor loadings >0.30 for our analysis.⁴¹ Construct reliability was acceptable (Cronbach's alpha >0.6) (Table 3).⁴²

Table 3.

Factor Loading

FACTORS	LOADING	CRONBACH ALPHA
Factor 1—Perceived quality of service		0.93
My or my child's needs were met promptly.	0.65
The provider was able to diagnose problems and treat my or my child's condition.	0.71
I understand the plan of care, which includes why tests and procedures are needed.	0.90
I feel that I was included in the health care decisions.	0.86
I was given home care instructions and able to ask questions.	0.84
I know my care and/or my child's care is important to the staff.	0.80
The staff was polite and kind.	0.64
Overall my needs were met. I would recommend others to Nationwide Children's Hospital.	0.60
Factor 2—Telehealth satisfaction		0.83
I was satisfied with the quality of the video connection during the visit.	0.35
Telehealth improves my access to health care services.	0.89
My overall experience with NCH telehealth was good.	0.56
If given the option, I would use video visits for future appointments.	0.76
Factor 3—Admission process		0.67
It was easy to schedule my appointment.	0.58
I was happy with the check-in process for my appointment.	0.76

We removed one item (“Informed of long wait time”) as the factor loading was below the cutoff of 0.30. There were no cross loadings of items observed. Factor 1 and Factor 2 had high reliability scores (Cronbach's alpha) of 0.93 and 0.83, whereas Factor 3 score (0.67) was reasonable.⁴³ The overall reliability score of the model was high with a Cronbach's alpha score of 0.91.

DEFINING CONSTRUCT

Reviewing the items within each factor and existing frameworks and literature,^{12,24,44
–46} labels for the constructs were discussed by the authors and confirmed by consensus. Factor 1 consists of most items that reflect the patient/family's clinical encounter experience; we label it more broadly as “Perceived Quality of Service” and define it as “the patient/family's assessment of the overall excellence of the service.” The four items loaded to Factor 2 relate to telehealth, including the overall experience rating, future use, and perceived usefulness. Thus, Factor 2 closely relates to “Telehealth Satisfaction,” defined as “positive attitudes toward the use and value of telehealth.”

Factor 3 consists of two items, one about scheduling and the other about check-in to appointment, both relate to the process “before” the start of the clinical encounter. We thereby label the factor as “Admission Process” and define it as “subjective experience of the admission process that precedes receiving healthcare.” The three new factors can form a new conceptual model to be further validated.

Discussion

In this study, we describe the development of the TPSS that is generalizable across clinical disciplines. This is particularly useful for large health care organization. The advantage is manifold. First, the same survey can be delivered as an institutional effort, reducing duplication of effort and making implementation and analysis more efficient. Second, the survey allows for compilation and comparison between different service lines. Third, this centralized approach can set appropriate mechanism in place to reduce patients receiving multiple survey requests, for example, in our case, we set the logic to limit only one survey link be sent to patient within a 2-week period.

The TPSS development was operationally driven. Nevertheless, we leveraged a conceptual framework to inform our survey design as well as to reveal item relationships. While we initially matched our survey items to an adapted SERVPERF model, our factor analysis supports a new 3-factor model. We observed some degree of consistency between the two models (Tables 1 and 3): for instance, three questions on assurance and two questions on empathy were grouped into new Factor 1 (perceived quality of service), and two questions on usability were grouped into new Factor 2 (telehealth satisfaction). Several reasons may account for the difference.

First, we kept many of the questions from in-person patient experience survey as requested by clinical departments for ease of analyzing patient satisfaction regardless of mode of delivery. These different items were matched by the construct definition, but may not have adequate discriminative power to reflect different constructs. Second, we were able to use only two to three items for each construct, which may likely affect coverage of construct and the factor loadings.⁴⁷

Existing surveys on patient satisfaction, such as the Hospital Consumer Assessment of Healthcare Providers and Systems survey or survey based on SERVPERF,^27,48 focus primarily on the clinical experience and do not cover the usability aspect of telehealth care delivery. Moreover, these surveys do not consider home/nonclinical site as primary location. Often, the social determinants, including access to device, connectivity, and safe private space, are not considered.

Our survey includes a barrier question to identify the challenges associated with telehealth. The list of challenges and barriers was based on literature⁴⁹ and our clinical operational experience. Although this question is not considered part of our overall patient satisfaction conceptual model, it was deemed operationally relevant for more diagnostic analysis for future quality improvement effort.

LIMITATIONS

There are some limitations to the current survey. We recognize that some questions kept from previous internal surveys were double-barreled, which introduced ambiguity. These questions are “The provider was able to diagnose problems and treat my or my child's condition”; “I was given home care instructions and able to ask questions”; and “Overall, my needs were met. I would recommend others to Nationwide Children's Hospital.” While this approach is useful for comparison to historical data, we recommend others who plan to adopt this survey to separate the questions into two distinct questions to reduce ambiguity.

This survey delivery method, while operationally efficient through automation, imposes a “survivorship bias.” Patients who encountered sufficient challenges that prevented them from completing the telehealth visit were not surveyed. Nevertheless, for patients who are unable to initiate or conduct the telehealth visit, most of the questions in this survey will not be applicable. We recommend future studies to consider the challenges and experiences of those who declined or tried, but failed to have a telehealth visit. For example, we recently used clinical notes to study reasons of “just-in-time” (close to appointment time) canceled or rescheduled telehealth appointment, which revealed certain challenges in initiating telehealth visits.⁵⁰

The trade-off for a generic survey is that the questions may not be tailored for certain clinical service lines. A related limitation is that at the time of study, not all clinical service lines were offering telehealth visits. Since the volume of telehealth visits was different between clinical departments, the results may be skewed toward the high-volume clinical lines (e.g., behavioral health and clinical therapies). We also noted that the survey respondents were mostly White, English-speaking parents of young patients. This may affect the generalizability of the survey results. In addition, we used a single telehealth platform. Organizations using multiple platforms may need additional software-specific considerations.

FUTURE WORKS

Additional validation of survey items and conceptual model with confirmatory factor analysis and structural equation modeling is presented in a companion article.⁵¹ We have also completed a separate survey for provider's view on telehealth, and future work will include a comparison of the provider and patient perspectives.

Conclusions

We developed a telehealth satisfaction survey based on operational guidelines, tested factor loadings and internal consistency, and eventually proposed a conceptual model to understand the patient satisfaction toward telehealth. It is versatile to be used across diverse clinical disciplines and covers both the clinical experience as well as the telehealth usability aspect. With the strong adoption of telehealth during the pandemic, telehealth is becoming not just a modality, but positioned to be a new “venue” or “setting” for providing health care. Well-designed survey to capture patient experience and pinpoint challenges is important to ensure high quality of health care delivery.

Footnotes

Authors' Contributions

S.L. conceived and presented the idea. E.-J.D.L., M.G., and E.S. drafted the article. M.G., E.S., and R.G. conducted data preparation and analysis. Collectively, E.-J.D.L., Y.H., L.M., J.L., U.R., and S.L. designed and implemented the survey. All authors contributed, critically reviewed, and revised the article. The final version of the article is approved by all authors.

Acknowledgments

We thank the Nationwide Children's Hospital telehealth committee and the family advisory team for contributing feedback to the design and implementation of the survey. The authors thank Sirina Lu and Sarah Wagner for supporting the technical implementation of the survey. They also thank Dr. Megan Gregory for helpful discussion and the NCH transcription services for survey translation.

Disclosure Statement

No competing financial interests exist.

Funding Information

There is no funding information to declare.

Supplementary Material

Supplementary Table S1

Supplementary Table S2

Supplementary Table S3

Supplementary Table S4

References

Lin

CCC

, Dievler

, Robbins

, Sripipatana

, Quinn

, Nair

. Telehealth in health centers: Key adoption factors, barriers, and opportunities. Health Aff, 2018; 37:1967–1974.

Centers for Medicare & Medicaid Services. COVID-19 emergency declaration blanket waivers for health care providers. 2020. Available at https://www.cms.gov/files/document/summary-covid-19-emergency-declaration-waivers.pdf (last accessed June 10, 2021 ).

U.S. Department of Health & Human Services. Notification of enforcement discretion for telehealth remote communications during the COVID-19 nationwide public health emergency. Available at https://www.govinfo.gov/content/pkg/FR-2020-04-21/pdf/2020-08416.pdf (last accessed June 10, 2021 ).

Berg

, Picoraro

, Miller

, et al. COVID-19—A guide to rapid implementation of telehealth services: A playbook for the pediatric gastroenterologist. J Pediatr Gastroenterol Nutr, 2020; 70:734–740.

Bakken

, Grullon-Figueroa

, Izquierdo

, et al. Development, validation, and use of English and Spanish versions of the Telemedicine Satisfaction and Usefulness Questionnaire. J Am Med Informatics Assoc, 2006; 13:660–667.

Yip

, Chang

, Chan

, Mackenzie

. Development of the Telemedicine Satisfaction Questionnaire to evaluate patient satisfaction with telemedicine: A preliminary study. J Telemed Telecare, 2003; 9:46–50.

Parmanto

, Lewis

Jr ., Graham

, Bertolet

MH.

Development of the Telehealth Usability Questionnaire (TUQ). Int J Telerehabilitation, 2016; 8:3–10.

Dagger

, Sweeney

, Johnson

. A hierarchical model of health service quality: Scale development and investigation of an integrated model. J Serv Res, 2007; 10:123–142.

Teshnizi

, Aghamolaei

, Kahnouji

, Teshnizi

SMH

, Ghani

. Assessing quality of health services with the SERVQUAL model in Iran. A systematic review and meta-analysis. Int J Qual Heal Care , 2018; 30:82–89.

10.

Sixma

, Kerssens

, Campen

C van

, Peters

. Quality of care from the patients' perspective: From theoretical concept to a new measuring instrument. Health Expect, 1998; 1:82–95.

11.

Chakraborty

, Majumdar

. Measuring consumer satisfaction in health care sector: The applicability of SERVQUAL. J Art Sci Commer, 2011; 2:149–160.

12.

Batbaatar

, Dorjdagva

, Luvsannyam

, Amenta

. Conceptualisation of patient satisfaction: A systematic narrative literature review. Perspect Public Health, 2015; 135:243–250.

13.

Kruse

, Krowski

, Rodriguez

, Tran

, Vela

, Brooks

. Telehealth and patient satisfaction: A systematic review and narrative analysis. BMJ Open, 2017; 7:e016242.

14.

Parasuraman

, Zeithaml

, Berry

. SERQUAL: A multiple-item scale for measuring consumer perceptions of service quality. J Retail, 1988; 64:12–40.

15.

Jain

, Gupta

. Measuring service quality: SERVQUAL vs. SERVPERF scales. Vikalpa, 2004; 29:25–37.

16.

Serrano

, Shah

, Abràmoff

. Use of expectation disconfirmation theory to test patient satisfaction with asynchronous telemedicine for diabetic retinopathy detection. Int J Telemed Appl 2018;2018;7015272.

17.

Naidu

Factors affecting patient satisfaction and healthcare quality. Int J Health Care Qual Assur, 2009; 22:366–381.

18.

Kim

, Shin

J-S

, Lee

, et al. Quality of medical service, patient satisfaction and loyalty with a focus on interpersonal-based medical service encounters and treatment effectiveness: A cross-sectional multicenter study of complementary and alternative medicine (CAM) hospitals. BMC Complement Altern Med, 2017; 17:174.

19.

Lee

, Kim

. Assessing healthcare service quality: A comparative study of patient treatment types. Int J Qual Innov, 2017; 3:1–15.

20.

Liu

, Singer

, Sun

, Camargo

. A conceptual model for assessing quality of care for patients boarding in the emergency department: Structure-process-outcome. Acad Emerg Med, 2011; 18:430–434.

21.

Newsome

PRH

, Wright

. A review of patient satisfaction: 1. Concepts of satisfaction. Br Dent J , 1999; 186:161–165.

22.

Ahenkan

, Aduo-Adjei

. Predictors of patient satisfaction with quality of healthcare in university hospitals in Ghana. Hosp Pract Res, 2017; 2:9–14.

23.

Azmi

, Rahman

, Mustafi

MAA

, Islam

. Measurement of patient satisfaction with SERVQUAL model of private hospitals: SEM approach. Am J Manag, 2017; 17:64–76.

24.

Langbecker

, Caffery

, Gillespie

, Smith

. Using survey methods in telehealth research: A practical guide. J Telemed Telecare. 2017; 23:770–779.

25.

Goldberg

, Velicer

. Principles of exploratory factor analysis. In: Strack

, ed. Differentiating normal and abnormal personality. New York: Springer Publishing Company, 2006:209–237.

26.

Kang

The prevention and handling of the missing data. Korean J Anesthesiol, 2013; 64:402–406.

27.

Cronin

, Taylor

. Measuring service quality: A reexamination and extension. J Mark, 1992; 56:55–68.

28.

Davis

FD.

User acceptance of information technology: System characteristics, user perceptions and behavioral impacts. Int J Man Mach Stud, 1993; 38:475–487.

29.

Kitapci

, Akdogan

, Dortyol İT. The impact of service quality dimensions on patient satisfaction, repurchase intentions and word-of-mouth communication in the public healthcare industry. Procedia Soc Behav Sci, 2014; 148:161–169.

30.

Ryu

Effects of skewness and kurtosis on normal-theory based maximum likelihood test statistic in multilevel structural equation modeling. Behav Res Methods, 2011; 43:1066–1074.

31.

Srivastava

, Hui

. On assessing multivariate normality based on Shapiro-Wilk W statistic. Stat Probab Lett, 1987; 5:15–18.

32.

Mardia

K V.

Applications of some measures of multivariate skewness and kurtosis in testing normality and robustness studies. Sankhyā Indian J Stat Ser B, 1974; 36:115–128.

33.

Henze

, Zirkler

. A class of invariant consistent tests for multivariate normality. Commun Stat Theory Methods, 1990; 19:3595–3617.

34.

Spector

PE.

Summated rating scale construction—An introduction. In: Lewis-Beck MS, ed. Quantitative applications in the social sciences. Newbury Park, CA: Sage, 1992:53.

35.

Dziuban

, Shirkey

. When is a correlation matrix appropriate for factor analysis? Some decision rules. Psychol Bull, 1974; 81:358–361.

36.

Joreskog

KG.

Factor analysis by MINRES. 2003. Available at https://ssicentral.com/wp-content/uploads/2020/07/lis_minres.pdf (last accessed June 6, 2021 ).

37.

Schmitt

, Sass

. Rotation criteria and hypothesis testing for exploratory factor analysis: Implications for factor pattern loadings and interfactor correlations. Educ Psychol Meas, 2011; 71:95–113.

38.

Zygmont

, Smith

. Robust factor analysis in the presence of normality violations, missing data, and outliers: Empirical questions and possible solutions. Tutor Quant Methods Psychol, 2014; 10:40–55.

39.

Hayton

, Allen

, Scarpello

. Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organ Res Methods, 2004; 7:191–205.

40.

Cattell

RB.

The scree test for the number of factors. Multivariate Behav Res, 1966; 1:245–276.

41.

Costello

, Osborne

. Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Pract Assess Res Eval, 2005; 10:3.

42.

Steel

RGD

, Torrie

, Dickey

. Principles and procedures of statistics: A biometrical approach. New York, NY: McGraw-Hill, 1997.

43.

Taber

KS.

The use of Cronbach's alpha when developing and reporting research instruments in science education. Res Sci Educ, 2018; 48:1273–1296.

44.

Jha

, Frye

, Schlimgen

. Evaluating variables of patient experience and the correlation with design. Patient Exp J, 2017; 4:33–45.

45.

Weaver

, Lukowski

, Wichman

, Navaneethan

, Fisher

, Neumann

. Human connection and technology connectivity: A systematic review of available telehealth survey instruments. J Pain Symptom Manage, 2021; 61:1042.e2–1051.e2.

46.

Zeithaml

VA.

Consumer perceptions of price, quality, and value: A means-end model and synthesis of evidence. J Mark, 1988; 52:2–22.

47.

Hair

, Black

, Babin

, Anderson

. Multivariate data analysis, 7th ed. Upper Saddle River, NJ: Prentice Hall, 2009.

48.

Centers for Medicare & Medicaid Services. HCAHPS: Patients' perspectives of care survey. Available at https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/HospitalHCAHPS (last accessed June 10, 2021 ).

49.

Almathami

HKY

, Win

, Vlahu-Gjorgievska

. Barriers and facilitators that influence telemedicine-based, real-time, online consultation at patients' homes: Systematic literature review. J Med Internet Res, 2020; 22:e16407.

50.

Sezgin

, Huang

, Lin

, Ramtekkar

, Pauline

, Lin

. Documented reasons of cancellation and rescheduling of telehealth appointments during the pandemic. Telemed J E Health, 2021; 27:1143–1150.

51.

Guntu

, Lin

E-JD

, Sezgin

, Gregory

, Huang

, Lin

Identifying the factors influencing patients' telehealth visit satisfaction: Survey validation through a structural equation modeling approach. Telemed J E Health (Companion paper under review. Citation to be updated after acceptance).

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.03 MB

0.00 MB