Validity of the Model for Assessment of Telemedicine: A Delphi study

Abstract

Introduction

In 2009, the Model for Assessment of Telemedicine (MAST) was developed within the MethoTelemed project as a framework for description of the effectiveness of telemedicine applications. The goal was for the assessments to be used as basis for decision-making in healthcare systems. Since then, MAST has been used in many European telemedicine studies and is now the most widely used model for assessment of telemedicine. The aim of this study was to assess the face validity of MAST.

Methods

A modified Delphi process was carried out and included a workshop with a sample of healthcare decision makers. A total of 56 decision makers and experts in telemedicine were invited and 19 persons participated in the two Delphi rounds. Thirteen hospitals or regional health authorities from 12 European countries and six research organisations were represented in the final sample. The participants were asked to assess the importance of the different domains and topics in MAST on a 0–3 Likert scale.

Results

All respondents completed the two rounds. Based on the answers, the face validity of all MAST domains was confirmed, since all domains were considered moderately or highly important by more than 80% of the respondents.

Discussion

Even though the study confirmed the validity of MAST, a number of supplements and improvements regarding study design and data collection were suggested. When considering the results it should be noticed that the sample size was small and larger studies are needed to confirm the results.

Keywords

Telemedicine telehealth health economics assessment decision making

Introduction

In 2009, the European Commission initiated the MethoTelemed project with the aim of supporting the development of a generic framework for assessment of telemedicine services that could be a guideline for consistent assessment and serve as a basis for decision making.¹

The MethoTelemed project resulted in the publication of the Model for Assessment of Telemedicine (MAST).² MAST was developed on the basis of a systematic literature review³ and the EUnetHTA Core model,⁴ through workshops with stakeholders and decision makers from a body of European healthcare organisations. The objective of MAST was to describe the effectiveness of telemedicine applications and to serve as a basis for decisions on investments in telemedicine services.² An assessment based on MAST includes a multidisciplinary process which summarises and evaluates information about the medical, social, economic and ethical issues related to the use of telemedicine in a systematic, unbiased and robust manner. Thus, in MAST the term ‘assessment’ is based on the principles of health technology assessment (HTA)⁴ and has the objective of describing the properties and effects of telemedicine for decision makers. MAST includes three steps and assessment of outcomes within seven domains as shown in Figure 1.

Figure 1.

The three steps in the Model for Assessment of Telemedicine.

Subsequently, MAST has been used in a number of European telemedicine projects, as described in Table 1, including more than 25,000 patients.^5–11 Based on this, MAST can be considered the most widely used assessment framework in studies of telemedicine in Europe.

Table 1.

European telemedicine projects using the Model for Assessment of Telemedicine (MAST).

Project	Design of studies	Number of patients
Renewing Health⁵	19 RCT studies	6600
United4Health⁶	3 Observational studies	4500
SmartCare⁷	1 Observational study	9800
InCASA⁸	5 Feasibility studies	150
MasterMind⁹	1 Observational study	5200
Connected4Health¹⁰	6 Feasibility studies	200
Beyond Silos¹¹	1 Observational study	2400
Carewell¹²	1 Observational study	900

RCT: randomised controlled trial.

Building on these and other studies, MAST has been applied in a number of publications in scientific journals in reporting of, for example, protocols,¹³ clinical outcomes,^14–18 patient perception,¹⁹ organisational aspects²⁰ and economic results^21,22 in studies of telemedicine. MAST has also been recommended for assessment of telemedicine in wound care²³ and included in a review of theoretical frameworks in telemedicine research²⁴ together with a number of other generic frameworks for assessment of telemedicine.^25–29 Thus, MAST has been considered usable for researchers who carry out assessments of telemedicine. However, the validity of MAST for decision makers who are supposed to use the results remains unclear.

Therefore, the objective of this study was to assess the validity of MAST by asking a group of European healthcare decision makers about their perception of the importance of the information included in MAST.

The type of validity tested in this study is face validity. Face validity is defined as the extent to which MAST is subjectively viewed as covering the concept it purports to measure.³⁰ Face validity of MAST can be tested empirically by examining a group of healthcare decision makers’ opinions of the model and its content. Face validity should be distinguished from the term ‘construct validity’ which is the degree to which a test measures what it claims, or purports, to be measuring.³¹

Methods

Design

The Delphi technique is used in the validation process. This technique can be used for consensus building, using a series of questionnaires to collect data from a panel of selected subjects.³²

In this study the modified Delphi technique included the following steps, similar to a recent study:³³

A. Collection of demographic information from the respondents (by use of the software SurveyXact).

B. Development of a structured questionnaire about the importance of the different domains and topics in MAST.

C. Round 1: Presentation of information about MAST to the panellists at the workshop, and subsequently asking the panel to answer the first Delphi questionnaire during the workshop.

D. The panel discussed the validity of MAST at the workshop.

E. Round 2: Submission of the second Delphi web based questionnaire (by use of SurveyXact) four weeks after the workshop including information about results from the first round to the panel.

Data collection

The structured questionnaire included questions about the importance of the seven MAST domains and the 26 topics within the domains. To assist the respondents, examples of the different topics were included in the questionnaire. See the list of domains and topics in Table 2 . In addition, seven questions were included about the importance of the transferability of information from the different domains.

Table 2.

Response to questions about importance of domains and topics in the Delphi process.

	Results from first round			Results from second round
Domains and topics	Median	Range	Proportion answering ‘moderately important’ or ‘highly important’	Median	Range	Proportion answering ‘moderately important’ or ‘highly important’
Domain 1: Health problem and description of the application	3	2–3	100%	3	2–3	100%
Health problem of the patients	3	1–3	95%	3	2–3	100%
Description of the application	3	1–3	95%	3	2–3	100%
Technical characteristics	3	0–3	84%	2	1–3	89%
Domain 2: Safety	3	2–3	100%	3	2–3	100%
Clinical safety	3	3–3	100%	3	3–3	100%
Technical safety	3	2–3	100%	3	2–3	100%
Domain 3: Clinical effectiveness	3	1–3	95%	3	2–3	100%
Effects on morbidity	3	2–3	100%	3	2–3	100%
Effects on mortality	3	2–3	100%	3	2–3	100%
Effects on quality of life	2	1–3	68%	2	1–3	84%
Behavioural outcomes	2	1–3	84%	2	1–3	95%
Use of health service	3	2–3	100%	3	2–3	100%
Domain 4: Patient perspectives	3	2–3	100%	3	2–3	100%
Patient satisfaction	3	1–3	95%	3	2–3	100%
Patients understanding of information	3	1–3	89%	3	2–3	100%
Patient acceptance	3	1–3	95%	3	2–3	100%
Patients confidence in the telemedicine treatment	2	1–3	84%	2	2–3	100%
Patients ability to use the application	3	2–3	100%	3	2–3	100%
Patients access and accessibility	2	1–3	95%	3	2–3	100%
Patients empowerment and self-efficacy	2	1–3	84%	2	2–3	100%
Domain 5: Economic aspects	3	2–3	100%	3	2–3	100%
Societal economic evaluation	2	0–3	95%	2	2–3	100%
Business case	2	1–3	84%	2	1–3	95%
Domain 6: Organisational aspects	2	2–3	100%	3	2–3	100%
Consequences for the process	2	2–3	100%	3	2–3	100%
Consequences for the structure	2	1–3	84%	2	2–3	100%
Consequences for the culture	2	1–3	89%	2	2–3	100%
Consequences for the management	2	0–3	79%	2	1–3	95%
Domain 7: Socio-cultural, ethics, legal aspects	2	0–3	89%	3	2–3	100%
Ethical issues	2	0–3	84%	2	2–3	100%
Legal issues	3	0–3	84%	3	0–3	95%
Social issues	2	0–3	79%	2	1–3	95%
Transferability of the described results to your local setting	3	1–3	95%	3	2–3	100%
Transferability of safety	3	1–3	95%	3	2–3	100%
Transferability of clinical effectiveness	3	2–3	100%	3	2–3	100%
Transferability of patient perspectives	2	2–3	100%	3	2–3	100%
Transferability of economic aspects	3	0–3	89%	2	1–3	95%
Transferability of organisational aspects	2	2–3	100%	3	2–3	100%
Transferability of socio-cultural, ethical, legal aspects	2	0–3	89%	2	2–3	100%

Finally, the questionnaire included open questions about the need for adjustments of MAST. One question concerned aspects currently missing in MAST. In another question, the respondents were asked whether they found that some outcomes within the MAST domains should be excluded. The questionnaire is presented in the Supplementary Material.

The importance of each domain and topic was assessed on a 0–3 Likert scale (0 = not important, 1 = somewhat important, 2 = moderately important, 3 = highly important).

In the second round, the respondents were presented with information about the proportion of respondents considering the domains and topics as ‘moderately important’ or ‘highly important’ during the first round.

Selection of participants

The inclusion criterion for participants was that they were potential users of the results from an assessment of telemedicine services, thus they were decision makers in the European healthcare systems. In addition, a group of researchers with knowledge of decision-making on health technologies and telemedicine was invited. In order to avoid a potentially biased view of the respondents, the goal was to recruit persons who were not involved in EU projects using MAST to assess telemedicine interventions.

Potential respondents in the network of the authors were contacted. If the persons were not interested in participating they were asked about potential respondents in their network. Finally, it was the intention to involve respondents from as many European countries as possible.

Data analysis

A descriptive analysis was carried out regarding demographic characteristics. Median score and range was estimated for the Likert scale questions in accordance with guidelines.³² STATA version 14 (StataCorp, Texas, USA) was used for the analysis of quantitative data. In the analysis of the questions, a proportion of 70% or higher viewing each domain and topic as ‘moderately important’ or ‘highly important’ was used as an indication of stability of the participants’ consensus with regard to the face validity as recommended.³⁴ During the workshop four of the authors (KK, LKJ, TK and MBH) summarised the participants discussions and based on this produced a list of common, central topics.

Approval of the study from the local ethical committee was not required because the study only collected data by questionnaire and did not involve patients.³⁵

Results

A total of 56 persons were invited from 17 European countries and of this total 19 persons from Belgium, Denmark, England, Finland, France, Germany, Greece, Netherlands, Norway, Scotland, Spain and Turkey participated in the workshop. The workshop took place in Brussels, 16 March 2016. The participants represent 13 hospitals or regional health authorities from 12 European countries. In addition six participants were from universities or other research organisations. The full list of participants is presented in the Supplementary Material.

Out of 19 respondents, 17 answered the demographic questions. Of these 17 respondents, 11 were educated in medicine and the rest in law, economics or IT. Regarding job title, 13 respondents were managers, directors or advisors from clinics, hospitals or regional health authorities. The remaining respondents were professors, IT consultants, law and ethics consultants or advisors, primarily from research institutions. Twelve respondents answered that they took part in making decisions on buying or implementing health technologies within their organisation. In addition, 12 respondents had not previously participated in EU projects where MAST was used to assess telemedicine.

Table 2 presents the participants' answers to Likert scale questions in the first and second rounds of the Delphi process. All respondents in the first round considered the seven MAST domains to be moderately or highly important, except for domain 3 on clinical effectiveness and domain 7 on ethical, legal and socio-cultural issues. These domains were considered important by 18 and 17 of the 19 respondents, respectively. With regard to the perception of the topics within the different domains, the results show more variation. Whereas both topics relating to an assessment of safety (domain 2) were considered moderately or highly important by all respondents, only 13 of the 19 participants (68%) considered the effects on quality of life in domain 3 moderately or highly important. In addition, only 15 of the 19 (79%) participants considered the organisational consequences for management (in domain 6) and the social aspects (in domain 7) as moderately or highly important. The transferability of the information included in an assessment was also considered to be moderately or highly important by 17 or more of the 19 participants.

In the second Delphi round, all respondents considered all domains to be moderately or highly important. The results regarding the topics within the domains are more diverse. The lowest level of perception of importance is with regard to effects on quality of life in domain 3. Here, 16 of the 19 participants (84%) found this information moderately or highly important. Generally, the answers indicate that the respondents consider the domains and the topics slightly more important in this final round of the Delphi process.

During the workshop the following questions were central in the discussion of the validity of MAST:

How should the preceding assessment of the level of maturity of the telemedicine service be carried out in practise?

How can data collection and analysis ensure assessment of long term effects of telemedicine, challenges for patients with comorbidity and the organisational impact of telemedicine?

How can the use of MAST form the basis for selection of key performance indicators (KPIs)?

How can information collected as part of one domain be used in the assessment of other domains?

How can health professionals perception of a telemedicine service be assessed?

Should the number of domains in MAST be reduced?

A number of suggestions and comments were made as part of the Delphi questionnaire regarding information that should be or not be the part of MAST as described in Table 3.

Table 3.

Aspects or outcomes that should be or not be part of the basis for decisions on investment in telemedicine.

	Aspects or outcomes mentioned by the respondents
Other aspects or outcomes of telemedicine that should be part of MAST	Reimbursement model and the related business case for the hospital.
	Impact on accessibility.
	Relevance of telemedicine for patients with comorbidity.
	Impact on strategy and image of the hospital.
	Integration with other devices and ICT systems.
	Key performance indicators for the telemedicine intervention.
	Impact of telemedicine on the costs of healthcare for the patients.
	Regional aspects such as inland versus island regions.
	Usability of the telemedicine device.
Other aspects or outcomes of tele-medicine that should not be part of MAST	The legal issues.
	The perception of the telemedicine service by the healthcare professionals.
	The societal economic evaluation.

MAST: Model for Assessment of Telemedicine.

Discussion

Overall, the results from the Delphi process confirm the face validity of the domains included in MAST. More than 80% of the participants in the process considered the seven MAST domains to be moderately or highly important in an assessment of telemedicine. The final Delphi round confirmed the results from the first round.

Need for improvement and supplementing

Even though the validity of the MAST domains was confirmed, the need for further improvement and description of the model was expressed in the Delphi process and during the discussions at the workshop.

One of the most important elements in MAST is the first step, called preceding assessment, which underlines the need for ensuring that a telemedicine application and the organisation using the services is mature and optimised before the multidisciplinary assessment (step two) is initiated. The importance of the first step has been mentioned in relation to design of new empirical studies.¹⁰

In practice, it can be difficult to determine when a new telemedicine intervention is sufficiently mature and how an intervention is matured and optimised. Generally, three different approaches can be used: Firstly, involvement of the users in the development process. As an example, participatory design has been applied to assess parents’ needs for telemedicine services in neonatal home care by use of observational studies, individual patient interviews and focus group interviews.³⁶ Secondly, optimisation studies can isolate the elements of a complex telemedicine intervention that are more likely to be effective if implemented in full scale . A recent review described how optimisation studies can include interview and focus group studies with collection of information about perception and acceptability of patients and healthcare professionals.³⁷ Finally, a pilot study can test the study procedures, validate tools for data collection, estimate recruitment rate and provide a basis for sample size calculation.³⁸

A number of comments on MAST were related to whether different types of study designs, including the randomised controlled trial (RCT), were able to measure the effects of telemedicine interventions. Examples include problems with assessment of organisational impact of telemedicine, if a randomised design prevents a hospital from making the necessary organisational changes and thereby increasing efficiency. This may happen if a hospital must offer conventional treatment for a control group and a telemedicine service for an intervention group at the same time during a trial. In that case, a hospital may not carry out necessary organisational changes because of the temporary nature of a trial. Thus, cluster randomisation or observational before-after studies should be considered if large organisational changes are a condition for realising the full benefits of a telemedicine service.³⁹ Problems with assessment of long term effects of e.g. telemedicine for patients with diabetes in short term studies were also mentioned. Here, a modelling approach e.g. a Markow model can be applied on the basis of results from an RCT and data from studies of the long term consequences of diabetes. Finally, challenges with assessment of the consequences of using telemedicine for patients with comorbidity were mentioned. These patients may be offered different types of telemedicine devices for different diseases. The potential practical problems related to use of several independent telemedicine services should be considered when the inclusion criteria for new studies are decided.

Identifying KPIs within each domain for quality assurance in the years after implementation of a telemedicine service was also suggested. Based on common register data in many hospitals, examples of KPIs could be the number of patients using the service, mean number of readmissions, mortality rate, reimbursement per patient, number of telemedicine contact per patient etc. By identification of KPIs hospital managers can promote successful adaptation by providing regular monitoring of the new technology.⁴⁰

As pointed out during the workshop, data collected as part of the assessment of one domain may be used in analysis of outcomes within other domains. Thus, interviews with the health professionals in organisational studies²⁰ may reveal effects on use of time and shifts in tasks that may explain outcomes within the economic domain. In addition requests were made during the workshop for guidelines on how to assess the organisational consequences of telemedicine. Examples from studies using MAST^2,20 and from studies of the whole systems demonstrator⁴¹ can be found, but a generic guideline on how to carry out organisational assessments of telemedicine services needs to be developed.

During the workshop, it was suggested that the number of domains could be reduced. For example, by combining the organisational and economic domains into one domain (on resource impact) or by combining the perception of patients, informal carers and health professionals into one domain (on user perspectives). However, it was also suggested that the seven domains should be kept because many European regions and healthcare organisations are now familiar with the seven domains. In addition, it is the experience from the development of the European network for health technology assessment (EUnetHTA) Core model that the discussion of the number of domains is never-ending. It should be noted that suggestions about changing the number of domains was not included in the answers to the final Delphi questionnaire after the workshop.

Finally, it should be emphasised that some suggestions for improvements were misunderstandings because the suggestions have already been included in the original description of MAST.² As an example, the business case for the hospital implementing a telemedicine intervention is part of the economic domain. Accessibility and usability are also mentioned in MAST as important aspects of patients’ perception of telemedicine and the ethical aspects.

Study strengths and limitations

The study has a number of limitations and the first and main problem is the selection of participants. It was the primary goal to include decision makers from the European healthcare systems and researchers with knowledge of healthcare decision-making. Of the 19 participants, 13 were managers or advisors from hospitals or regional health authorities. Thus, they were persons who participate in making decisions on investment in health technologies such as telemedicine and thereby users of assessments based on MAST. It is difficult to assess whether 19 other persons from the total group of 56 potential participants would have produced a different result. However, the high level of agreement between the 19 participants in this study could indicate that another group of participants may have given a similar result.

Secondly, the sample size can be criticised as the workshop only included 19 participants. However, when using the Delphi technique, it has been recommended to include the minimal sufficient number of subjects and to verify the results through follow-up exploration.⁴² Thus, 10–15 subjects have been suggested as sufficient if the background of the Delphi subjects is homogeneous. In accordance with this recommendation, it has been found that the majority of Delphi studies have used 15–20 respondents.⁴³

Thirdly, the criterion of 70% agreement can be criticised for being arbitrary. It has been suggested that consensus is achieved when 80% of participants fall within two categories on a seven-point scale.⁴⁴ However, the overall result of the face validity of the seven MAST domains remains the same if 80% instead of 70% is used as the criterion.

Fourthly, there is a risk that some participants felt social pressure to give answers in support of MAST, since authors behind the article describing MAST participated in the workshop as facilitators. This potential bias is similar to ‘social desirability bias’.⁴⁵ To address this, the participants were informed that their answers would be anonymous in the final Delphi round.

In addition, it may be considered a limitation that this study did not include other groups of respondents who may be impacted by telemedicine e.g. the patients. Similarly, other types of validity could have been tested, but this was considered beyond the scope of this study. Other empirical studies addressing the information needed by healthcare managers when making decisions on investment in telemedicine can be used to assess the face validity of MAST. In a recent study regarding policies for remote monitoring in the UK, Germany, Italy and Spain interviews were made with 19 policy makers, payers and clinical experts.⁴⁶ The respondents were asked which assessment criteria they would prefer to support decision making on remote monitoring. The respondents answered: clinical criteria (observational data e.g. comparing effectiveness and safety, patient mortality, real life data on effectiveness, safety and patient compliance), health economic criteria (health economic evaluation, budget impact model, reduction in healthcare resource utilisation) and other criteria (evidence on patient satisfaction and additional value e.g. early diagnostics). The assessment criteria are all included in MAST and generally only the seventh MAST domain on legal and ethical aspects is not mentioned.

In another study, the perception of the use of MAST in the EU project Renewing Health was addressed by 12 pilot managers in a questionnaire study.⁴⁷ Overall, the study shows that MAST was considered useful in the project. In addition, it was suggested to add domains regarding technological usability, responsible innovation, health literacy, behavioural change, caregiver perspectives and motivational issues of professionals. However, as noted in a comment,⁴⁸ MAST does include most of these topics.

Directions for future research

The results from the Delphi process described here need to be tested in future studies with large, representative samples of European healthcare decision makers to ensure that the small sample of respondents in this study are not biased. Similar, future studies should investigate whether information about all MAST domains are needed or if a less costly and less time consuming model could be useable in some cases. In addition it is relevant to carry out tests of other kinds of validity e.g. the construct validity of MAST in the future.

Footnotes

Acknowledgement

This work arises from the SmartCare Project (Joining up ICT and service processes for quality integrated care in Europe) which is co-funded by the European Commission within the ICT Policy Support Programme of the Competitiveness and Innovation Framework Programme (CIP) (grant agreement no. 325158).

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work is co-funded by the European Commission within the information and communication Technology (ICT) Policy Support Programme of the CIP (grant agreement no. 325158).

Supplementary material

Supplementary material for this paper can be found at .

References

Communication from the commission to the european Parliament, the council, the european economic and social Committee and the committee of the regions On telemedicine for the benefit of patients, healthcare systems and society, http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri= COM:2008:0689:FIN:EN:PDF (2008, accessed 1 August 2016).

Kidholm

Ekeland

Jensen

et al.

A model for assessment of telemedicine applications: MAST. Int J Technol Assess Health Care 2012; 28: 44–51.

Ekeland

Bowes

Flottorp

. Effectiveness of telemedicine: A systematic review of reviews. Int J Med Inform 2010; 79: 736–771.

Lampe

Makela

Garrido

et al.

The HTA core model: A novel method for producing and reporting health technology assessments. Int J Technol Assess Health Care 2009; 25: 9–20.

Renewing Health project. Regions of Europe working together for health. http://www.renewinghealth.eu/en/ (2016, accessed 1 august 2016).

United4Health project. Transforming the patient experience with telehealth in Europe. http://united4health.eu/ (2016, accessed 1 August 2016).

Smart Care project. Smartcare – Delivering integrated eCare. http://www.pilotsmartcare.eu/home/ (2016, accessed 1 August 2016).

InCASA project. Integrated network for completely assisted senior citizen's autonomy. http://www.incasa-project.eu/news.php (2012, accessed 1 August 2016).

MasterMind project. Management of mental health disorders through advanced technology and services – telehealth for the mind. http://mastermind-project.eu/ (2016, accessed 1 August 2016).

10.

European Commission. Testing e-health services in the Nordics – Connected For Health project launched. https://ec.europa.eu/digital-single-market/en/news/testing-e-health-services-nordics-connected-health-project-launched (2016, accessed 1 August 2016).

11.

Beyond Silos project. Taking integrated care one step further… http://beyondsilos.eu/project/project-overview.html (2016, accessed 1 August 2016).

12.

Carewell project. Delivering integrated care to frail patients through ICT. http://www.carewell-project.eu/home.html (2016, accessed 1 August 2016).

13.

Charrier

Zarca

Durand-Zaleski

et al.

ARS Ile de France telemedicine group. Efficacy and cost effectiveness of telemedicine for improving access to care in the Paris region: Study protocols for eight trials. BMC Health Serv Res 2016; 45: 1–8.

14.

Rasmussen

Froekjaer

Bjerregaard

et al.

A randomized controlled trial comparing telemedical and standard outpatient monitoring of diabetic foot ulcers. Diabetes Care 2015; 84: 799–807.

15.

Karhula

Vuorinen

Rääpysjärvi

et al.

Telemonitoring and mobile phone-based health coaching among Finnish diabetic and heart disease patients: Randomized controlled trial. J Med Internet Res 2015; 17: e153–e153.

16.

Torbjørnsen

Jenum

Småstuen

et al.

A low-intensity mobile health intervention with and without health counseling for persons with type 2 diabetes, Part 1: Baseline and short-term results from a randomized controlled trial in the Norwegian part of Renewing Health. JMIR Mhealth Uhealth 2014; 2: e52–e52.

17.

Sorknaes

Bech

Madsen

et al.

The effect of real-time teleconsultations between hospital-based nurses and patients with severe COPD discharged after an exacerbation. J Telemed Telecare 2013; 19: 466–474.

18.

Rosenbek Minet

Hansen

Pedersen

et al.

Early telemedicine training and counselling after hospitalization in patients with severe chronic obstructive pulmonary disease: A feasibility study. BMC Med Inform Decis Mak 2015; 15: 3–3.

19.

Dario

Luisotto

Mancin

et al.

Assessment of patients’ perception of telemedicine services using the Service User Technology Acceptability Questionnaire. Int J Integr Care 2016; 16: 1–11.

20.

Rasmussen

Jensen

Froekjaer

et al.

A qualitative study of the key factors in implementing telemedical monitoring of diabetic foot ulcer patients. Int J Med Inform 2015; 84: 799–807.

21.

Fasterholdt

Gerstrøm

Rasmussen

BSB

et al.

Cost-effectiveness of telemonitoring of diabetic foot ulcer patients. Health Infor J Epub ahead of print 16 September 2016; .

22.

Kidholm

Rasmussen

Andreasen

et al.

Cost-utility analysis of a cardiac telerehabilitation program: The Teledialog Project. Telemed J E Health 2016; 22: 553–563.

23.

Moore

Angel

Bjerregaard

et al.

Chapter 3: The model for assessment of telemedicine (MAST) – evaluation of telemedical solutions. J Wound Care 2015; 24: S10–S13.

24.

Wade

Gray

Carati

. Theoretical frameworks in telemedicine research. J Telemed Telecare Epub ahead of print 18 January 2016.

25.

Nepal

Jang-Jaccard

et al.

A framework for telehealth program evaluation. Telemed J E Health 2014; 20: 393–404.

26.

Agboola

Hale

Masters

et al.

‘Real world’ practical evaluation strategies: A review of telehealth evaluation. JMIR Res Protoc 2014; 3: e75–e75.

27.

Bashshur

Shannon

Sapci

. Telemedicine evaluation. Telemed J E Health 2005; 11: 296–316.

28.

Grigsby

Brega

Devore

. The evaluation of telemedicine and health services research. Telemed J E Health 2005; 11: 317–328.

29.

Scott

McCarthy

Jennett

et al.

National Telehealth Outcome Indicators Project. J Telemed Telecare 2007; 13: S1–S38.

30.

Gravetter

Forzano

LAB

. Research methods for the behavioral sciences, 4th ed. Belmont, CA: Wadsworth, 2012.

31.

Polit

Beck

. Nursing research: Generating and assessing evidence for nursing practice, 9th ed. Philadelphia: Wolters Klower Health, Lippincott Williams and Wilkins, 2012.

32.

Hsu

Sandford

. The Delphi technique: Making sense of consensus. Practical Assessment, Research and Evaluation 2007; 12: 1–8. . http://essentialsofmedicine.com/sites/default/files/Delphi%20Technique_%20Making%20 Sense%20Of%20Consensus.pdf (accessed 1 february 2016).

33.

Jeon

Conway

Chenoweth

et al.

Validation of a clinical leadership qualities framework for managers in aged care: A Delphi study. J Clin Nurs 2014; 24: 999–1010.

34.

Green PJ. The content of a college level outdoor leadership course. In: Conference of the Northwest District Association for the American Alliance for Health, Physical Education, Recreation and Dance, March 1982, Spokane, WA, 1982.

35.

The Regional Committees on Health Research Ethics for Southern Denmark. Forsøgstyper uden anmeldelsesplig 2016. https://komite.regionsyddanmark.dk/wm428123 (accessed 1 February 2016).

36.

Garne

Brødsgaard

Zachariassen

et al.

Telemedicine in neonatal home care: Identifying parental need through participatory design. JMIR Res Protoc 2016; 5: e100–e100.

37.

Levati

Campbell

Frost

et al.

Optimisation of complex health interventions prior to a randomised controlled trial: A scoping review of strategies used. Pilot Feasibility Stud 2016; 2: 17–17.

38.

Arain

Campell

Cooper

et al.

What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol 2010; 10: 67–67.

39.

Liu

JLY

Wyatt

. The case for randomized controlled trials to assess the impact of clinical information systems. J Am Med Inform Assoc 2011; 18: 173–180.

40.

Davies A and Newman S. Evaluating telecare and telehealth interventions. WSDAN briefing paper. Kings Fund, London, 2011.

41.

Hendy

Chrysanthaki

Barlow

et al.

An organisational analysis of the implementation of telecare and telehealth: The whole systems demonstrator. BMC Health Serv Res 2012; 12: 403–403.

42.

Delbecq

Van de Ven

Gustafson

. Group techniques for program planning, Glenview, IL: Scott, Foresman and Co., 1975.

43.

Ludwig

. Predicting the future: Have you considered using the Delphi methodology? J Ext 1997; 35: 1–4.

44.

Ulschak

. Human resource development: The theory and practise of need assessment, Reston, VA: Reston Publishing Company, Inc, 1983.

45.

Hebert

Clemow

Pbert

et al.

Social desirability bias in dietary self-report may compromise the validity of dietary intake measures. Int J Epidemiol 1995; 24: 389–398.

46.

Rojahn K, Laplante S, Sloand J, et al. Remote Monitoring of Chronic Disease: A Landscape Assessment of Policies in Four European Countries. PLOS ONE 2016; 19: 1–15.

47.

Ekeland

Grøttland

. Assessment of MAST in European patient-centered telemedicine pilots. Int J Technol Assess Health Care 2015; 31: 304–311.

48.

Kidholm

Jensen

Kotzeva

. On the use of the MAST model in assessment of telemedicine: A comment on Ekeland and Grøttland. Int J Technol Assess Health Care 2015; 31: 312–313.