Using survey methods in telehealth research: A practical guide

Abstract

Surveys are a common method for assessing patient and clinician perceptions, attitudes and outcomes of telehealth. However, inadequacies in both the conduct and reporting of survey studies are common in telehealth research. This article provides clinicians and researchers with practical guidance on the appropriate selection, use and reporting of survey tools for telehealth research. We identify common survey outcomes and instruments used in telehealth research, and methods to assess the validity and psychometric properties of survey tools. Enhancing the quality and reporting of telehealth research is important to improve our understanding of which telehealth-supported models of care improve outcomes and for which patient groups.

Keywords

Telehealth telemedicine survey research evaluation

Introduction

Surveys are commonly used in telehealth research to assess patient satisfaction, patient experiences, patient preferences and attitudes, and the technical quality of a teleconsultation.¹ The popularity of the survey as a method of measurement can be understood through three major strengths of this technique. First, confidential survey questions are well suited to capture individuals’ experiences, perceptions and attitudes. The views of patients and of clinicians can be independently and confidentially assessed and compared, without jeopardizing the clinical relationship. Second, pre-existing scales can be used across studies, enabling the comparison and replication of results. As surveys allow data to be collected from large samples for relatively low cost, they can produce generalizable results, which provide an understanding of the views or experiences of a population group. Third, the validity and reliability of survey instruments can be assessed through rigorous, transparent and well-accepted validation methods, providing the researcher with confidence that the measures tap the intended constructs, and provide an accurate measurement.

However, many methodological limitations of survey research in telehealth have been identified.^2,3 Inadequate consideration given to the selection of constructs (i.e. the attributes which we measure via survey instruments), the use of novel survey instruments or instruments that lack validity or reliability (note: these will be defined later), and poor adherence to guidelines for the conduct and reporting of research limit the usefulness of the results generated, both to the individual researchers, and to the field of telehealth. The aim of this paper is to provide practical guidance to researchers and clinicians on the selection and use of survey measures for evaluating telehealth interventions. In the next sections, we first identify the various types of outcomes that are commonly assessed using surveys in telehealth research. We then provide guidelines on how to select an appropriate survey instrument, or develop and validate a new survey, followed by a brief overview of considerations in the conduct and reporting of survey research.

Choosing the outcomes of interest in telehealth research

Surveys are commonly used in telehealth research to assess attributes such as demographic, clinical or professional characteristics, behaviours, future intentions, experiences and attitudes, including evaluations of a service.^4,5 Constructs that are evaluated using surveys in telehealth research include satisfaction, experience, technical quality, effectiveness, usefulness or effect on interaction (Table 1).

Table 1.

Patient- and clinician-reported constructs commonly reported in telehealth research.

Construct	Definition	Indicator variables
Satisfaction	Subjective evaluation about whether the user’s expectations were met	• Overall satisfaction • Willingness to use / re-use
Experience	Evaluation of the user’s experience of a healthcare service. The experience can be objective (e.g. waiting time) or subjective (e.g. level of patient-centredness)	• Patient-reported experience measures • Comfort • Patient-centredness
Technical quality	Subjective evaluation of the quality of the technology used	• Audio &/or picture (video) quality / interface quality • Reliability • Usability/ease of use • Intuitiveness or learnability • Privacy and security
Effectiveness/ perceived effectiveness	Objective or subjective assessment that a telehealth interaction/s helped improve the health status or wellbeing of a patient or client	• Quality of life • Change in health status • Measures of health or wellbeing • Patient empowerment • Patient knowledge • Patient-reported outcome measures • Diagnostic confidence
Usefulness / perceived usefulness	Objective or subjective assessment that a telehealth interaction/s produced some benefit or met the purpose of the interaction	• Convenience • Time consequences • Cost consequences • Accessibility • Effect on continuity of care • (Future) intention to use • Willingness to use / practice • Acceptability
Effect on interaction	Subjective assessment that the modality of communication affected clinician–patient or clinician–clinician interaction. The similarity of the telehealth interaction between patient and clinician to an in-person interaction⁶⁰	• Communication style • Ease of communication • Ability to undertake clinical examination • Completeness of information • Effect of visual and non-verbal cues on consultation

In survey research, patient-level constructs are used to assess the patient’s perspective or experiences resulting from receiving care via telehealth. Patient-level constructs may measure process outcomes, such as how the care was delivered, or outcomes resulting from receiving health care via telehealth, such as increased patient knowledge or empowerment, improved accessibility to health care or changes in health status. When both telehealth and face-to-face consultations are used for an episode of care, it is not possible to attribute outcomes to one particular form of consultation.⁶ For this reason, subjective measurements of the perceived value of telehealth are often used.

One commonly measured process outcome is patient satisfaction. There are a number of validated survey instruments that measure patient satisfaction, including some specific to telehealth (see Table 2), such as the Telemedicine Satisfaction Questionnaire⁷ and The Telemedicine Satisfaction and Usefulness Questionnaire,⁸ as well as generic patient satisfaction surveys, such as the Patient Satisfaction Questionnaire.⁹ However, there is widespread criticism of satisfaction as a process outcome measure for health care generally, with the concept lacking conceptual and theoretical clarity.^10-12 In telehealth research, patient satisfaction has consistently been found to be greater than 80% and frequently 100%.¹³ However, the meaningfulness of this is unclear, particularly as patients tend to rate their healthcare highly. From a patient perspective, satisfaction may be due to the fact that an appointment was available at an earlier date, or that the potential time spent travelling to an appointment was saved. From a provider perspective, high satisfaction may be related to the ability to make an accurate diagnosis or due to a higher level of remuneration being received compared with usual practice. The inability of many satisfaction surveys to identify the elements of a service desired by users, and to disentangle satisfaction with clinical outcomes, limit their usefulness in informing service design.¹⁴ Another common challenge with satisfaction surveys occurs when a high proportion of participants in a study select the maximum scores on the observed variable, otherwise referred to as the ceiling effect.² This is a common issue and many satisfaction surveys are thus not considered to be sensitive to variations in the quality of the service provided.¹⁰

Table 2.

Selected survey instruments relevant to telehealth.

Instrument	Number of items	Licensed (Y/N)	Constructs measured	Validity & reliability	Population in which validity/reliability tested
American College of Chest Physicians (ACCP) ICU Telemedicine Survey⁶¹	32	N	Staffing, structure, processes of care, & telemedicine practices in ICU	• Inter-rater reliability (excellent)⁶¹	Responders from ICUs in the US supported by a comprehensive telemedicine programme, including nurse managers, medical directors, and administrators⁶¹
Patient Assessment of Communication during Telemedicine (PACT)⁶²	45	N	Patient-centred communication, perceived physician’s competence, physician’s interpersonal skills, convenience, ability to develop rapport, shared decision-making during video consultations	• Internal consistency reliability (Cronbach’s α > 0.9 for all constructs)⁶²	Patients attending pulmonary, endocrine, and rheumatology clinics at a US Veterans Administration hospital⁶²
Telemedicine Perception Questionnaire (TMPQ)⁶³	17	Y*	Patient acceptability via perception of risk & benefits of in-home telemonitoring	• Construct validity Internal consistency reliability (Cronbach’s α = 0.8)⁶³	Residents of an assisted-living facility for the elderly and a church community in the US⁶³
Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ)⁸	26	N	Patient satisfaction, perceived utilization & frequency of utilization for hybrid in-home telemonitoring & video consultations	• Construct validity • Internal consistency reliability video visits (Cronbach’s α = 0.97) • Internal consistency reliability impact and use (Cronbach’s α = 0.92)⁸	Adults aged 55+ years receiving treatment for diabetes mellitus residing in a US federally designated medically underserved area fluent in English or Spanish⁸
Telemedicine Satisfaction Questionnaire (TSQ)⁷	14	N	Technical quality, effect on communication, ease of use, comfort, accessibility, perceived usefulness, perceived effectiveness, intention to use/re-use, overall satisfaction with in-home video consultation	• Content validity • Construct validity • Internal consistency reliability⁷	Newly diagnosed Chinese diabetic patients aged 40–70 years⁷
Whole Systems Demonstrator – Service User Technology Acceptability Questionnaire (WSD-SUTAQ)⁶⁴	22	N	Patient acceptability via measures of accessibility, comfort, usability, privacy and security, confidentiality, satisfaction, convenience & health benefits with in-home telemonitoring	• Face validity • Internal consistency reliability (per construct)⁶⁴	Patients with social care needs or long-term health conditions (diabetes, COPD or heart failure) in the UK^64,65
Telehealth Usability Questionnaire (TUQ)⁶⁰	21	N	Usefulness, ease of use, effectiveness, reliability, & satisfaction of video conferencing	• Internal consistency reliability (Cronbach’s α > 0.8 for all constructs) • Construct validity⁶⁰	US patients & clinicians with and without experience of telehealth (participants without experience took part in a simulated telehealth session)⁶⁰
Internet Attitude Scale (IAS)⁶⁶	38	N	Internet, email and computer literacy & ability	• Internal consistency reliability (Cronbach’s α = 0.81) • Convergent validity⁶⁶	US undergraduate & graduate students; employees of a US industrial enterprise⁶⁶
Technology Acceptance Model (TAM),⁶⁷ TAM-21²¹	12	N	Perceived usefulness, perceived ease of use, attitude & intention to use	• Construct validity • Face validity • Content validity • Convergent validity • Internal consistency reliability (Cronbach’s α > 0.7)⁶⁷	A number of different populations including US postgraduate students;⁶⁷ physicians in Hong Kong public tertiary hospitals;²¹ pre-service teachers in Singapore and Malaysia⁶⁸

Free for academic use

ICU: Intensive care unit

Due to the limitations of satisfaction measures, patient experience surveys may be a better option for telehealth research. Patient experience may assess the degree of patient-centred communication such as concordance to the patient’s values, needs and preferences and patient participation in health care decisions.¹⁵ Additional patient-reported experience measures include being treated with compassion, dignity and respect.¹⁶ Patient experience, as well as indicators of effectiveness or perceived effectiveness and effect on interaction, is frequently measured using tools specific to the clinical population or context.

In contrast, some tools are necessarily specific to the telehealth environment. The technical quality of the telehealth consultation (e.g. audio and visual quality of a videoconference) has been shown to influence clinicians’ willingness to practice and the satisfaction of users.^17,18 Technical quality is commonly measured using scales developed by the International Telecommunications Union, assessing listening quality, listening effort, image quality and image impairment.¹⁹ Reliability is another outcome measure of technical quality. This may be assessed by whether a video conference call was completed successfully or failed.²⁰ The usefulness or perceived usefulness of telehealth may be assessed using a variety of indicators such as perceived convenience, time or cost saved. Often a variety of attitudes towards telehealth are captured to assess this construct. For example, the technology acceptance model (TAM)^21-23 has been successfully used to capture both patients’ and clinicians’ attitudes towards telehealth. This survey measures the perceived usefulness, perceived ease of use, attitude towards and intention to use telehealth.

Many of the same constructs measured at the patient level apply equally to clinicians (e.g. technical quality). Additional constructs can be used to evaluate clinicians’ perspectives on delivering care via telehealth, such as perceived ability to conduct a telehealth consultation, the appropriateness of the technology,^24,25 and the clinician’s diagnostic confidence or confidence in patient management resulting from the use of the telehealth.²⁶

Selecting survey instruments

Once the patient and/or clinician-level constructs to be measured have been selected, the next step is to select the survey instruments to measure these constructs. Using existing validated instruments where possible, rather than developing new measures, is recommended as it helps establish a set of commonly accepted and well-validated instruments, assists the comparison of results across studies, and ultimately facilitates a more integrated body of telehealth literature.²⁷ The following questions and guidelines are provided to assist researchers in the choice of an appropriate existing survey measure.

Does the instrument match the chosen definition and conceptualization of the construct? Validity is the extent to which a data collection instrument measures what it is intended to measure.²⁸ Hence, the first step is to ensure that the chosen survey operationalizes the concept in a way that fits with the commonly accepted definition and conceptualization of the concept. For example, if patient experience is the concept of interest, how well does the survey measure align with and capture the definition and underlying dimensions of ‘patient experience’? The content and dimensions of the concept – and hence its valid survey operationalization – may differ depending on the role of the person completing the survey (i.e. the rater, such as patient vs. clinician). Hence, caution needs to be exercised when adapting or translating survey tools designed and validated with one rater to another.

Is the instrument well validated and psychometrically sound? A rigorous evaluation of an instrument’s validity includes assessment of construct validity (including divergent and convergent validity), content validity and criterion validity, as well as the instrument’s reliability, stable factor structure across studies, and responsiveness to changes over time if the instrument will be used longitudinally.^29,30 The meaning of these concepts is described in Table 3. Often researchers adopt survey measures that have been published in prior empirical studies without examining evidence of the instrument’s actual validity. The use of a scale in a published study does not necessarily mean the scale is well validated or reliable when used with the population of interest. Table 2 identifies a number of survey instruments applicable to telehealth evaluations along with their psychometric properties. We also later discuss how to develop your own survey, in the absence of a valid, reliable instrument suited to your intervention and context.

Table 3.

Psychometric properties of survey instruments.

Attribute	Meaning
Validity	The extent to which the instrument measures what it is designed to measure
Construct validity	The extent to which the instrument measures the concept being studied, based on a variety of evidence, including evidence of content and criterion validity, and that from studies of expected performance differences over time, across groups and settings
Divergent validity (also known as discriminant validity)	The extent to which measures that are not expected to be related are actually unrelated (i.e. scores for instruments measuring two different constructs do not correlate)
Convergent validity (also known as concurrent validity)	The extent to which measures that are expected to be related are actually related (i.e. scores for instruments measuring similar or overlapping constructs correlate highly)
Face validity	The extent to which the instrument appears to be a good measure of the concept (i.e. the items look appropriate to a lay person)
Content validity	The extent to which the instrument represents the full definition of the concept (i.e. the instrument contains everything it should, and nothing it should not, as assessed by experts)
Criterion validity	The extent to which the instrument can predict some future event, behaviour or outcome, or it compares well with a similar measure (‘gold standard’) of the same concept
Reliability	The extent to which the instrument is free from random error (i.e. it is consistent, stable and repeatable)
Internal consistency	The average correlation among a group of items in a questionnaire (i.e. how well a group of questions assess a particular characteristic)
Test-retest reliability	The extent to which a set of scores on a particular test for a given sample are stable over time (i.e. if there is no change over time, do we get the same score if people do the test twice?)
Inter-rater reliability	The extent of agreement among raters (i.e. do different individuals agree in their ratings in the same situation?)
Other
Factor structure	The relationship between correlated groups of items (factors) in a questionnaire
Responsiveness (also known as sensitivity to change)	An instrument’s ability to detect changes over time (i.e. if there is a change over time, do we get different scores if people do the test twice?)

Is the instrument applicable to the chosen telehealth intervention and context? A key issue here is whether survey items are worded in a manner that is applicable to the chosen research context. Often telehealth survey measures are designed to evaluate a particular clinical service (e.g. diabetes management), for a particular patient population (e.g. children) in a particular location (e.g. rural area). Hence, a judgement needs to be made as to whether the survey instrument is valid and appropriate for use in a different setting. The careful adaptation of survey items may be required to ensure their relevance to the changed setting. Adapting a survey in any way, whether it is changing the wording of individual survey items, using a selection of items rather than the entire survey, changing the order of items in the survey, using different response options, or translating a survey into a different language, may impact the validity and reliability of the instrument. If such adaptation is done, it is best practice to then pilot the revised survey instrument in the novel setting, and examine the validity and reliability of the new instrument.

Developing and validating a novel survey

If an exhaustive search of the literature reveals no suitable survey measure, then the choice is either to adapt an existing survey measure to the specific telehealth context and service, or to develop and validate a new survey. Important steps in developing a new survey include:

Review the existing literature and scales to identify common definitions, conceptualizations and dimensions of the concept.

Conduct qualitative research (e.g. interviews, focus groups, open-ended surveys) with the population of interest (e.g. patients/clinicians) to identify and confirm the ‘domain of content’ for the construct.

Develop an initial pool of survey items tapping each dimension of the construct. Ensure the language used and reading level is appropriate for the intended population (‘readability’ not exceeding the sixth³¹ to eighth³² grade levels is usually recommended; readability scores can be automatically calculated by word processing programs such as Microsoft Word).³³ Consider the characteristics of the population you wish to survey and the variation that will be present with regard to their perceptions of questions and concepts – are there any concepts which may be understood in multiple ways that may vary across cultures or communities, or sensitive topics in your pool of items? Have you considered social norms around sharing personal information or discussing certain topics in developing your item pool?

Consider the response choices you will provide for any close-ended questions. These should be consistent across survey items, for example, always going from least to most satisfied. Switching up the order across survey items can be confusing to participants and is likely to yield missing or error-laden data. Also ensure that the response options make sense given the question item, and are mutually exclusive and exhaustive. Likert scales, which allow a participant to indicate their preferences or extent of agreement with a statement or statements on a unidimensional scale (e.g. a five-point scale ranging from ‘Strongly agree’ to ‘Strongly disagree’), are often used.³⁴ Each level on such a scale is assigned a numeric value or coding (e.g. 1 for ‘Strongly agree’, 2 for ‘Agree’, 3 for ‘Neither agree nor disagree’, and so on), although it must be noted that these data are ordinal (in that the response categories have a rank order), not continuous, and should be analysed as such.³⁴ Labelling either all the response options, or at least the extremes and ‘middle’ response option when using a scale with an odd number of items (e.g. 1, 3 and 5 on a 5-point Likert scale) will help respondents to understand what each category means and also enhance the clarity of the results reported. The researcher should also consider whether having a ‘middle’ response option is useful to their study. Although theoretically a mid-point response should represent ambivalent or ‘neutral’ attitudes, research suggests some participants will select this option when it is available rather than reporting their actual opinion.³⁵ This may occur for a number of reasons, such as social desirability bias (responding in a manner that presents a favourable image of themselves, e.g. as the ‘good patient’).³⁶ Removing a mid-point response and using a scale with an even number of items may thus increase the likelihood that the participant will report their actual opinion; however, the downside of this is that it does not accommodate the participant with ambivalent or truly neutral attitudes.

At this point it is worth considering the length of the draft survey and the likely impact of this on participant propensity to complete it in full and thoughtfully. Many novice researchers devise surveys that are overly long, especially when considering the characteristics of the population of interest. In the field of telehealth, where we often survey patients with health conditions and clinicians providing care, the added participant burden and fatigue of a too-long survey may lead to lower response rates³⁷ and poorer quality data.³⁸ In contrast, surveys that are too short will be unable to capture participant views of the construct of interest fully, and may yield little variability in the data that could be explored. Whilst the ‘ideal’ length of a survey will vary depending on the topic, population and context, a positive impact on survey completion rates have been found both for the general population and among physicians of surveys of shorter length (one or two pages).^37,39

Conduct a pilot study to assess the face validity and usefulness of the survey items. Use feedback from a representative sample of respondents to delete poor performing items (i.e. items that respondents report to be ambiguous, unclear or open to different interpretations) and to ensure that all relevant elements of a construct are assessed. Ideally, conduct a second pilot study to ensure the final version of the survey works effectively. It is worth considering at this point, if this has not been done previously, the desired method of administration of the survey (more on this later), as piloting the method of administration of the survey as well as the content may be useful.

Conduct a validation study using a large representative sample to examine and report the reliability and psychometric properties of the scale. One method to establish the construct validity of a new instrument^30,40 is to examine whether the survey measure behaves as hypothesized in relation to other variables within the nomological network.⁴¹ Analyses can be conducted comparing the new instrument with other validated instruments to provide evidence of the convergent and discriminant validity of the scale. To establish criterion validity, survey responses from participants (e.g. providers and patients) can be compared with expert ratings and evaluations of the telehealth service, or other objective indicators of telehealth outcomes. The content validity of the scale can be further assessed by examining participant responses to open-ended survey questions designed to elicit the key indicators of the construct.

As evidenced by the process described above, the rigorous development and validation of a new survey instruments takes considerable time and investment. Establishing construct validity of a new instrument is an ongoing process, and future researchers adopting a scale can strengthen the assessment of the scale’s validity by reporting on its psychometric properties and links to actual telehealth outcomes.

Survey conduct and reporting

A number of decisions must be made in planning and conducting a survey study alongside the choice of constructs and survey tools. The choice of sampling frame and strategy,⁴² desired sample size, method of administration of a survey,⁴³ strategies employed to enhance response rates,⁴ data management procedures, and analytic decisions, alongside others, will influence the usefulness of the reported results, and a number of resources are available to assist in the making of these decisions.^44,45 Although it is beyond the scope of this paper to delve too deeply into each potential issue, we detail here some advice for a few concerns that commonly arise.

Firstly, it is essential that the telehealth researcher obtain ethical clearance (or a waiver if appropriate) from an institutional review board (or equivalent; outside the US these may be known by other names, e.g. ‘ethics committee’) for their survey study prior to study commencement. Many telehealth surveys are completed online using survey software (more on that later), which allow participants (a perception of) anonymity, and there appears to be a misperception among some quarters that this kind of research is not subject to the same ethical oversight as are other research studies. Regardless of whether surveys are completed online or using pen and paper, whether they are completed anonymously or if participants are fully identifiable, appropriate ethical clearance must be sought, and documentation kept. This is essential to protect the study participant, the researcher and their institution, and increasingly, journals will not publish research findings if ethical clearance was not obtained.

One consideration for the survey researcher with ethical implications is whether to offer incentives for participation. Incentives may increase response rates for patient and clinician surveys, although the evidence for this is mixed.^4,46 Incentives also have costs; higher than expected response rates could work out unexpectedly costly, and care must be taken to ensure the research budget could withstand this. The extent to which incentives influence data quality should be considered (if, for example, someone participates only to achieve the incentive, and gives little thought to their responses), although the evidence for this is again inconclusive.^47,48 Another consideration is that for research to be ethical, the value of any incentive, including non-monetary incentives such as gift vouchers or entry in a prize draw, must allow the potential participant to make a truly voluntary decision to participate – high-value ‘incentives’ may be viewed as coercive, if they persuade individuals to accept risks that they would not otherwise accept.⁴⁹ What value an incentive would need to take to be considered coercive varies by context; something of trivial value to a person of higher socioeconomic status may be viewed very differently to someone with very limited financial resources.

The method of administration of the survey should also be considered. Asking participants to complete surveys online is becoming increasingly common in many fields, not least technology-savvy areas such as telehealth. As always, care must be taken to consider the population of interest and context in which the study is being conducted. If your population of interest has fast, reliable, private internet access and is comfortable with this technology, then online surveys may be appropriate; in contrast, if your population lives in an area with unreliable internet, or it is most commonly available in shared spaces that do not offer the privacy a participant might require, then this modality may be a poorer choice. Whilst this delivery mode has advantages in terms of scalability, speed of data collection and cost, it does not prevent – and indeed may contribute to – sampling and other representational errors, such that the sample does not represent the population of interest.⁵⁰

Regardless of the method of administration of a survey, care should be given to the language used to recruit study participants and explain how the survey data will be reported. As with all research, potential participants should be provided enough information to make an informed decision as to whether or not to participate in your survey; typical information needed will include the topics to be discussed, approximately how long it will take to complete, the risks and benefits of participation, the manner in which data will be collected and how it will be used, and how their confidentiality will be maintained. Very little research is truly anonymous, even if names are not requested, as other demographic or clinical information (e.g. sex, race, location) may together potentially identify them, so rather than assuring participants of anonymity, it may be more correct to describe the methods that will be taken to preserve participant confidentiality, such as storing survey data (whether in paper form or electronically) securely with limited access only to essential research staff, and only reporting pooled data.

One point in which particular comment is due relates to small sample sizes, for which the field of telehealth has been widely criticized.^51-54 While small sample sizes are an issue across many disciplines and topics, they are particularly prevalent in telehealth reports. This may be partly due to the relative maturity of the field and rapid increase in a range of different technologies being used, which may have led to a predominance of pilot and feasibility studies, testing new technologies in new settings. So what can be done? Careful consideration of methodological issues, the use of validated survey instruments and adherence to relevant reporting guidelines^42,55-57 can facilitate the combination of outcomes for meta-analyses, to determine an overall “effect size”.⁵⁸ In the present and expected future funding climate, where adequately funded, large-scale studies are rare, collaborative research efforts to jointly collect standardized outcomes across sites may provide a way forward.^58,59

Box 1.

Questions to ask when reviewing telehealth research using survey measures.

1.	Is the research question clearly stated?
2.	Was the survey instrument appropriate to answer the research question?
3a.	Was a validated survey instrument used?	OR	3b. Has access to the adapted or novel survey instrument been provided? AND 3c. Has the validity and reliability of an adapted or novel survey instrument been described?
4.	Was an appropriate sample size used to answer the research question?
5.	Has the sampling method been reported?
6.	Has the response rate been reported?
7.	Has the method of survey administration been reported?
8.	Have the analytic techniques been clearly described?

Reporting of survey research is often inadequate in providing the reader the ability to judge the quality of the research. Box 1 lists eight simple questions that readers of telehealth survey research papers can ask when reading such studies. This framework can also be used by authors to structure the reporting of their survey research.

Conclusion

Surveys are a common method for assessing patient-reported outcomes and clinician views of technology appropriateness, but inadequacies in both the conduct and reporting of survey studies are common.¹ Improving the quality and reporting of this research is essential to improve our understanding of which telehealth-supported models of care improve outcomes for which patient groups.

Footnotes

Acknowledgement

We thank Monica Taylor for her assistance with the literature search for validated surveys relevant to telehealth.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This review was conducted with the support of the Centre of Research Excellence in Telehealth funded by Australia’s National Health and Medical Research Council (NHMRC; APP1061183). D. Langbecker is supported by a NHMRC Early Career Research Fellowship (APP1072061). The contents of the published material are solely the responsibility of the individual authors and do not reflect the views of NHMRC or other funding agencies.

References

Whitten

Johannessen

Soerensen

et al.

A systematic review of research methodology in telemedicine studies. J Telemed Telecare 2007; 13: 230–235.

Masino

Lam

. Choice of rating scale labels: implication for minimizing patient satisfaction response ceiling effect in telemedicine surveys. Telemed J E Health 2014; 20: 1150–1155.

Allemann Iseli

Kunz

Blozik

. Instruments to assess patient satisfaction after teleconsultation and triage: A systematic review. Patient Prefer Adherence 2014; 8: 893–907.

McColl

Jacoby

Thomas

et al.

Design and use of questionnaires: A review of best practice applicable to surveys of health service staff and patients. Health Technol Assess 2002; 5: 1–256.

Dillman

. Mail and telephone surveys: the total design method, New York: Wiley, 1978.

Caffery

Taylor

Lucas

et al.

Substitution rates of video consultations for traditional consultations at a tertiary public hospital. J Telemed Telecare 2016; 22: 453–458.

Yip

Chang

Chan

et al.

Development of the Telemedicine Satisfaction Questionnaire to evaluate patient satisfaction with telemedicine: A preliminary study. J Telemed Telecare 2003; 9: 46–50.

Bakken

Grullon-Figueboa

Izquierdo

et al.

Development, validation, and use of English and Spanish versions of the Telemedicine Satisfaction and Usefulness Questionnaire. J Am Med Inform Assn 2006; 13: 660–667.

Marshall

Hays

. The patient satisfaction questionnaire short-form (PSQ-18), Rand Santa Monica, CA, 1994.

10.

Carr-Hill

. The measurement of patient satisfaction. J Public Health Med 1992; 14: 236–249.

11.

Sitzia

Wood

. Patient satisfaction: A review of issues and concepts. Soc Sci Med 1997; 45: 1829–1843.

12.

Manary

Boulding

Staelin

et al.

The patient experience and health outcomes. New Engl J Med 2013; 368: 201–203.

13.

Williams

May

Esmail

. Limitations of patient satisfaction studies in telehealthcare: A systematic review of the literature. Telemed J E Health 2001; 7: 293–316.

14.

Whitten

Love

. Patient and provider satisfaction with the use of telemedicine: Overview and rationale for cautious enthusiasm. J Postgrad Med 2005; 51: 294–300.

15.

Epstein

Franks

Fiscella

et al.

Measuring patient-centered communication in patient-physician consultations: Theoretical and practical issues. Soc Sci Med 2005; 61: 1516–1528.

16.

Black

Varaganum

Hutchings

. Relationship between patient reported experience (PREMs) and patient reported outcomes (PROMs) in elective surgery. BMJ Qual Saf 2014; 23: 534–542.

17.

Wade

Eliott

Hiller

. Clinician acceptance is the key factor for sustainable telehealth services. Qual Health Res 2014; 24: 682–694.

18.

Stahl

Dixon

. Acceptability and willingness to pay for primary care videoconferencing: A randomized controlled trial. J Telemed Telecare 2010; 16: 147–151.

19.

Caffery

Smith

. Investigating the quality of video consultations performed using fourth generation (4G) mobile telecommunications. J Telemed Telecare 2015; 21: 348–354.

20.

Taylor

Morris

Pech

et al.

Home telehealth video conferencing: Perceptions and performance. Jmir Mhealth Uhealth 2015; 3.

21.

Chau

PYK

Sheng

ORL

et al.

Examining the technology acceptance model using physician acceptance of telemedicine technology. J Manage Inform Syst 1999; 16: 91–112.

22.

Holden

Karsh

. The technology acceptance model: Its past and its future in health care. J Biomed Inform 2010; 43: 159–172.

23.

Russell

Gillespie

Hartley

et al.

Exploring the predictors of home telehealth uptake by elderly Australian healthcare consumers. J Telemed Telecare 2015; 21: 485–489.

24.

Fatehi

Gray

Russell

. A clinimetric study of outpatient diabetes consultations: The potential for telemedicine substitution. Diabetes Technol Ther 2014; 16: 8–14.

25.

Luxton

Pruitt

Osenbach

. Best practices for remote psychological assessment via telehealth technologies. Prof Psychol-Res Pr 2014; 45: 27–35.

26.

Ludzik

Witkowski

Roterman-Konieczna

et al.

Improving diagnostic accuracy of dermoscopically equivocal pink cutaneous lesions with reflectance confocal microscopy in telemedicine settings: Double reader concordance evaluation of 316 cases. Plos One 2016; 11.

27.

Schilling

Kozak

Lundahl

et al.

Inaccessible novel questionnaires in published medical research: Hidden methods, hidden costs. Am J Epidemiol 2006; 164: 1141–1144.

28.

Boynton

Greenhalgh

. Selecting, designing, and developing your questionnaire. BMJ 2004; 328: 1312–1315.

29.

Hinkin

Schriesheim

. Development and application of new scales to measure the French and Raven (1959) bases of social power. J Appl Psychol 1989; 74: 561–567.

30.

Landy

. Stamp collecting versus science – validation as hypothesis-testing. Am Psychol 1986; 41: 1183–1192.

31.

Langbecker

Janda

. Quality and readability of information materials for people with brain tumours and their families. J Cancer Educ 2012; 27: 738–743.

32.

Calderon

Morales

Liu

et al.

Variation in the readability of items within surveys. Am J Med Qual 2006; 21: 49–56.

33.

Microsoft. Test your document s readability. 2017.

34.

Jamieson

. Likert scales: How to (ab)use them. Med Educ 2004; 38: 1217–1218.

35.

Edwards

Smith

. The effects of the neutral response option on the extremeness of participant responses. Incite 2011; 6: 177–192.

36.

van de Mortel

. Faking it: Social desirability response bias in self-report research. Aust J Adv Nurs 2008; 25: 40–48.

37.

Kellerman

Herold

. Physician response to surveys. Am J Prevent Med 2001; 20: 61–67.

38.

Galesic

Bosnjak

. Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opin Quart 2009; 73: 349–360.

39.

Dillman

. Mail and telephone surveys: The total design method, New York: John Wiley and Sons, 1978.

40.

Nunnally

. Psychometric theory, 2nd ed. New York: McGraw-Hill, 1978. p.xv, 701 pp .

41.

Cronbach

Meehl

. Construct validity in psychological tests. Psychol Bull 1955; 52: 281–302.

42.

Kelley

Clark

Brown

et al.

Good practice in the conduct and reporting of survey research. Int J Qual Health Care 2003; 15: 261–266.

43.

Edwards

. Questionnaires in clinical trials: Guidelines for optimal design and administration. Trials 2010; 11: 2–2.

44.

Rea

Parker

. Designing and Conducting Survey Research: A Comprehensive Guide, 4th ed. San Fransisco: Jossey-Bass, 2014.

45.

BMJ Publishing Group. Statistics Notes. 2017.

46.

Robb

Gatting

Wardle

. What impact do questionnaire length and monetary incentives have on mailed health psychology survey response? Br J Health Psychol 2017.

47.

Medway

Tourangeau

. Response quality in telephone surveys. Do prepaid cash incentives make a difference? Public Opin Quart 2015; 79: 524–543.

48.

Litman

Robinson

Rosenzweig

. The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behav Res Methods 2015; 47: 519–528.

49.

Singer

Couper

. Do incentives exert undue influence on survey participation? Experimental evidence. J Empir Res Hum Res Ethics 2008; 3: 49–56.

50.

Marcano Belisario

Jamsek

Huckvale

et al.

Comparison of self-administered survey questionnaire responses collected using mobile apps versus other methods. Cochrane Database Syst Rev 2015, pp. MR000042–MR000042.

51.

Hersh

Helfand

Wallace

et al.

Clinical outcomes resulting from telemedicine interventions: A systematic review. BMC Med Inform Decision Making 2001; 1: 5–5.

52.

Radhakrishnan

Jacelon

. Impact of telehealth on patient self-management of heart failure: A review of literature. J Cardiovasc Nurs 2012; 27: 33–43.

53.

Chi

N-C

Demiris

. A systematic review of telehealth tools and interventions to support family caregivers. J Telemed Telecare 2015; 21: 37–44.

54.

Caffery

Bradford

Wickramasinghe

et al.

Outcomes of using telehealth for the provision of healthcare to Aboriginal and Torres Strait Islander people: A systematic review. Aust N Z J Public Health 2017; 41: 48–53.

55.

Bennett

Khangura

Brehaut

et al.

Reporting guidelines for survey research: An analysis of published guidance and reporting practices. PLoS Med 2010; 8: e1001069–e1001069.

56.

Burns

Duffett

Kho

et al.

A guide for the design and conduct of self-administered surveys of clinicians. CMAJ 2008; 179: 245–252.

57.

The EQUATOR Network and UK EQUATOR Centre for Statistics in Medicine NDoO, Rheumatology and Musculoskeletal Sciences (NDORMS), University of Oxford Botnar Research Centre. The EQUATOR Home page. Oxford, UK.

58.

Grigsby

Brega

Devore

. The evaluation of telemedicine and health services research. Telemed J E Health 2005; 11: 317–328.

59.

Kahn

Hill

Lilly

et al.

The research agenda in ICU telemedicine: A statement from the Critical Care Societies Collaborative. Chest 2011; 140: 230–238.

60.

Parmanto

Nelson Lewis

Graham

et al.

Development of the Telehealth Usability Questionnaire (TUQ). Int J Telerehabil 2016; 8: 3–10.

61.

Lilly

Fisher

Ries

et al.

A national ICU telemedicine survey: Validation and results. Chest 2012; 142: 40–47.

62.

Agha

Schapira

Laud

et al.

Patient satisfaction with physician-patient communication during telemedicine. Telemed J E Health 2009; 15: 830–839.

63.

Demiris

Speedie

Finkelstein

. A questionnaire for the assessment of patients' impressions of the risks and benefits of home telecare. J Telemed Telecare 2000; 6: 278–284.

64.

Hirani

Rixon

Beynon

et al.

Quantifying beliefs regarding telehealth: Development of the Whole Systems Demonstrator Service User Technology Acceptability Questionnaire. J Telemed Telecare 2016; 23: 460–469.

65.

Bower

Cartwright

Hirani

et al.

A comprehensive evaluation of the impact of telemonitoring in patients with long-term conditions and social care needs: Protocol for the whole systems demonstrator cluster randomised trial. BMC Health Serv Res 2011; 11: 184–184.

66.

Zhang

. Development and validation of an internet use attitude scale. Compute Educ 2007; 49: 243–253.

67.

Davis

. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quart 1989; 13: 319–340.

68.

Teo

Lee

Chai

et al.

Assessing the intention to use technology among pre-service teachers in Singapore and Malaysia: A multigroup invariance analysis of the Technology Acceptance Model (TAM). Compute Educ 2009; 53: 1000–1009.