Critical Appraisal of Published Systematic Reviews Assessing the Cost-Effectiveness of Telemedicine Studies

Abstract

Background: Over the last 10 years several systematic reviews have been published on the cost-effectiveness of telemedicine studies. Most reviews have concluded that there is not much difference in the cost-effectiveness when delivering health services via telemedicine or by conventional means. We are not aware of any systematic review looking at the systematic reviews of cost-effectiveness of telemedicine. This study was designed to identify published systematic reviews on the cost-effectiveness of telemedicine studies and to undertake a quality assessment of the identified systematic reviews. Materials and Methods: We searched six electronic databases, including Medline, Embase, and the NHS Economic Evaluation Database, combining “review” terms with “telemedicine” terms to identify systematic reviews. Results: We identified 4,116 potential abstracts. Nine systematic reviews met the inclusion criteria, which looked at the cost-effectiveness of telemedicine in general. All reviews were similar in terms of their stated purpose, and the objectives were clear. Three of the reviews did not use a checklist for the economic evaluation studies included in their review. The quality assessment found that five of the nine reviews had minimal flaws. Conclusions: Even though the general quality of reporting of the reviews was fine, we have found that conclusions cannot be drawn on the cost-effectiveness of telemedicine applications based on the methodological flaws in the economic analysis of the studies included in the reviews. Over time, reporting of cost-effectiveness has generally improved; however, there is still room for improvement, and authors need to use the recommended checklists for economic evaluations.

Introduction

Even though the use of telemedicine and telecare interventions in healthcare has grown over the last 30 years, clinicians, healthcare providers, and decision makers still have to make choices on which interventions to implement or use in healthcare because of financial constraints. Therefore, decisions have to be made to see whether the additional benefit of this new telemedicine/telecare intervention compared with the current alternative(s) is worth the additional cost.¹ The question remains as to whether telemedicine services could provide a significant improvement, in terms of both clinical and cost-effectiveness.

Over the last two decades, growing numbers of studies have been published on the effectiveness and cost-effectiveness of telemedicine interventions; however, the economic analyses associated with these interventions have not always been correctly reported. Several guidelines exist for assessing the quality of health economic studies, such as the standard checklists for economic evaluations by Drummond et al.,¹ Evers et al.,² and Husereau et al.³ Likewise, McIntosh and Cairns⁴ and Sisk and Sanders⁵ have provided frameworks to serve as a guide when conducting economic evaluations of telemedicine technologies. They also point out some of the challenges faced when evaluating telemedicine technologies, such as the constantly changing technology, small sample sizes, and the lack of methodological rigor.

Systematic reviews are becoming increasingly popular in evidence-based research. They summarize a vast body of evidence and can be used by policy makers to make informed decisions. Because of the growing number of studies reporting on the cost-effectiveness of telemedicine, there have also been several systematic reviews on the cost-effectiveness of telemedicine studies being published. However, there has been no general agreement in the methods and analytical results that these systematic reviews have raised, and therefore a critical appraisal of the quality of these review articles is warranted. There have been several reviews looking at systematic reviews of telemedicine: Bahaadinbeigy et al.⁶ highlighted the need for an overview of the current situation regarding systematic reviews of telemedicine, Ekeland et al.⁷ looked at the effectiveness of telemedicine with a brief section on economic analysis, and Ekeland et al.⁸ looked at the methodologies for assessing telemedicine. As far as we are aware, this is the first systematic review looking at the systematic reviews of cost-effectiveness of telemedicine.

The aim of this study was to conduct a review that was aimed at identifying published systematic reviews on the cost-effectiveness of telemedicine studies, providing a critique of these reviews in order to determine their validity, and also providing recommendations to guide future research aimed at assessing the cost-effectiveness of telemedicine studies.

Materials and Methods

Search Strategy and Data Extraction

A systematic search was undertaken to identify all published systematic reviews looking at the cost-effectiveness of telemedicine studies. This article focuses on the general field of telemedicine (i.e., on all health areas) rather than the application of telemedicine in certain health areas, such as heart failure⁹ or chronic diseases,¹⁰ to make the results more generalizable. The following databases were searched from their inception date until the last week in July 2012: Medline, Embase, ISI Social Science Index, Cochrane Database of Systematic Reviews, NHS Economic Evaluation Database, and the Database of Abstracts of Reviews of Effects. Search terms included “review” terms combined with “telemedicine” terms (Fig. 1). We did not use specific terms such as cost-effectiveness or economics as we did not want to miss any potential reviews.

Fig. 1.

Example of the search strategy in Medline.

The citation indices including abstracts for each of the search strategies were exported into the citation manager software (Endnote Windows version 9). All duplicates within Endnote were deleted. For three of the databases—NHS Economic Evaluation Database, Database of Abstracts of Reviews of Effects, and Cochrane—abstracts could not be exported to Endnote; therefore these abstracts were cross-checked against the Endnote database for duplicates by hand.

This review adheres to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) checklist, except for items that were not relevant for a systematic review of reviews.¹¹ All titles, bibliographic data, and abstracts of the results from these searches were then read and scanned by H.M. and H.G. for relevance based on the inclusion and exclusion criteria. Any disagreements were resolved by a third reviewer (R.O.). The inclusion criteria were journal articles relating to published systematic reviews that examined the cost-effectiveness of telemedicine applications. The exclusion criteria for the abstracts included reviews with no cost-effectiveness studies, in specific telemedicine areas (and not a general overview of telemedicine), studies looking at e-health, non-journal articles such as books and reports, non-English articles, and articles where one review led to multiple reporting of the same review results. The study flow was documented using a PRISMA diagram.¹¹

Data extraction forms were set up in Microsoft^® (Redmond, WA) Excel™ to help synthesize the results extracted from the reviews in the form of descriptive statistics. A comprehensive quality assessment of each review using the validated scale of Oxman and Guyatt¹² was undertaken by H.M. and H.G. The instrument contains nine items pertaining to individual aspects in the reporting of a systematic review. A 10th item collates the information from the first nine and asks the assessor to grade the review articles based on an ordinal scale ranging from 1 (minimal flaws) to 7 (extensive flaws).

Results

A detailed search of the electronic databases retrieved 5,066 abstracts; after the removal of duplicates, 4,116 abstracts remained. The full titles and abstracts of the 4,116 abstracts were read to check for relevance. After the exclusion criteria were applied, 113 abstracts remained, and the full-text articles were obtained. Nine published systematic reviews remained after reviewing the full-text articles (Fig. 2).

Fig. 2.

Preferred Reporting Items for Systematic Reviews and Meta-Analysis flow diagram showing the review selection process.

Summary of Identified Reviews

Nine reviews were published between 2001 and 2012 that assessed studies on the cost-effectiveness of general telemedicine, arranged chronologically according to publication year. All of the reviews were similar in terms of their stated purpose, and the objectives were clearly stated. Table 1 highlights the details for the literature searches used for each systematic review. The number of databases searched ranged from 4¹³ to 10.^14,15 In terms of time frame for review searches, most of the reviews were from inception of the database to just before the article was published; however, Hailey et al.¹⁶ had only a 2½-year time frame for their review searches, as this was an extension of their previous review.¹⁷ The numbers of telemedicine and economic search terms were not stated in two reviews,^16,18 so the search strategy that was undertaken could not be replicated; the number of telemedicine terms ranged from 4^13,17,19,20 to 10,²¹ and the number of economic terms ranged from 2²⁰ to 10.²¹ The number of abstracts retrieved (after removal of duplicates) ranged from 605¹⁶ to 5,219.¹⁸ In terms of the articles that were included in the reviews, the numbers ranged from 9²¹ to 80¹⁵; perhaps this higher number of articles for the latter review was because of the fact it was the latest review to be published.

Table 1.

Details of Literature Search for Each Systematic Review Including Databases and Search Strategy

REFERENCE (YEAR)	NUMBER OF DATABASES SEARCHED	TIME FRAME	NUMBER OF ABSTRACTS RETRIEVED	NUMBER OF ARTICLES THAT MET INCLUSION CRITERIA	NUMBER OF TELEMEDICINE SEARCH TERMS	NUMBER OF ECONOMIC SEARCH TERMS	GENERAL CONCLUSION(S) PROVIDED BY THE AUTHOR(S)	OXMAN AND GUYATT(12) SCORE
Roine et al.(19) (2001)	8	Different start date for each database; all searches until early 2000	1,124	50 (16 were economic analyses)	4	5	Evidence regarding the effectiveness or cost-effectiveness of telemedicine is still limited. Based on the current scientific evidence, only a few telemedicine applications can be recommended for broader use.	5
Whitten et al.(13) (2002)	4	Database inception to June 2000	612	24	4	7	There was no good evidence that telemedicine is a cost-effective means of delivering healthcare compared with standard healthcare delivery.	7
Hailey et al.(17) (2002)	8	Different start date for each database; all searches until December 2000	1,323	66 (41 included economic assessment)	4	5	Although useful clinical and economic outcomes data have been obtained for some telemedicine applications, good-quality studies are still scarce, and the generalizability of most assessment findings is rather limited.	7
Hailey et al.(16) (2004)	6	January 2000–June 2002	605	46 (25 were economic analyses)^a	Not stated	Not stated	The authors felt the results confirmed previous findings and that good-quality studies are still scarce.	6
Rojas and Gagnon(18) (2008)	8	January 1997–October 2007	5,219	23	Not stated	Not stated	This review provides knowledge on the indicators available for assessing cost-effectiveness in telehomecare projects. It appears that the specific context in which the projects take place, meaning different patients, environments, technologies, and healthcare systems, should be taken into account when selecting indicators for assessing telehomecare cost-effectiveness.	5
Wade et al.(20) (2010)	6	Database inception to June 2009	3,960	36	4	2	Delivery of health services by real-time video communication was cost-effective for homecare and access to on-call hospital specialists, showed mixed results for rural service delivery, and was not cost-effective for local delivery of services between hospitals and primary care.	7
Bergmo(14) (2009)	10	1990–2007	779	33	6	4	Most of the economic evaluations were not in accordance with standard evaluation techniques.	4
Peeters et al.(21) (2011)	7	Unsure of start date to December 2009	3,085	9^b	10	10	The review found no evidence that the cost of implementing video communication in homecare was lower than the resulting financial benefits.	7
Mistry(15) (2012)	10	Database inception to September 2010	2,583	80	5	7	Results were consistent with previous findings (i.e., there is no further conclusive evidence that telemedicine and telecare interventions are cost-effective compared with conventional healthcare).	7

Hailey et al.(16) provided 46 studies from 48 articles (that is, two articles reported the same study) and was an extension of Hailey et al.(17)

Peeters et al. (2011) reported nine studies from 20 articles.

Table 2 shows the general characteristics of studies that were included in each of the systematic reviews. All reviews except one¹³ stated the different health areas in which telemedicine was used, and only five of the nine reviews stated the different countries in which the telemedicine intervention was used.^{13,15,17,18,21} In terms of study design, the three earlier studies^13,17,19 only stated the study design for some of the articles that were included; however, the majority of reviews tended to have nonrandomized controlled trials, although Rojas and Gagnon¹⁸ and Wade et al.²⁰ had more randomized trials included in their reviews than nonrandomized trials. The majority of reviews highlighted that telemedicine was only compared with one comparator; however, from the review of Wade et al.,²⁰ over half of the included studies did not state how many comparators with which telemedicine was compared. Most of the reviews did not mention for how long the telemedicine intervention was evaluated, nor did they state the sample sizes of the included studies.

Table 2.

General Characteristics of Studies Included in the Systematic Reviews

			STUDY DESIGN				COMPARATORS			TIME FRAME
REFERENCE (YEAR)	NUMBER OF DIFFERENT HEALTH AREAS	NUMBER OF DIFFERENT COUNTRIES	RCT	NON-RCT	OTHER	NOT STATED	ONE	TWO OR MORE	NS	≤1 YEAR	>1 YEAR	NS	SAMPLE SIZE STATED
Roine et al.(19) (2001)	9	NS	6 (12.0%)	28 (56.0%)	0	16 (32.0%)	45 (90.0%)	5 (10.0%)	0	0	0	50 (100.0%)	1 (2.0%)
Whitten et al.(13) (2002)	NS	6	3 (12.5%)	0	0	21 (87.5%)	22 (91.7%)		2 (8.3%)	19 (79.2%)	5 (20.8%)	0	0
Hailey et al.(17) (2002)	9	10	7 (17.1%)	19 (46.3%)	5 (12.2%)^a	10 (24.3%)	32 (78.0%)	9 (22.0%)	0	14 (34.1%)	5 (12.2%)	22 (53.7%)	23 (56.1%)
Hailey et al.(16) (2004)	20	NS	22 (47.8%)	24 (52.2%)	0	0	46 (100.0%)	0	0	0	0	46 (100.0%)	0
Rojas and Gagnon(18) (2008)	10	6	15 (65.2%)	8 (34.8%)	0	0	23 (100.0%)	0	0	0	0	23 (100.0%)	0
Wade et al (2010)	17	NS	18 (50.0%)	6 (16.7%)	12 (33.3%)^b	0	14 (38.9%)	1 (2.8%)	21 (58.3%)	0	0	36 (100.0%)	0
Bergmo(14) (2009)	17	NS	13 (39.4%)	10 (30.3%)	10 (30.3%)^c	0	32 (97.0%)	1 (3.0%)	0	0	0	33 (100.0%)	0
Peeters et al.(21) (2011)	7	2	4 (44.4%)	3 (33.3%)	2 (22.2%)^d	0	9 (100.0%)	0	0	6 (66.6%)	0	3 (33.3%)	7 (77.7%)
Mistry(15) (2012)	33	20	28 (35.0%)	49 (61.2%)	3 (3.8%)^a	0	64 (80.0%)	16 (20.0%)	0	43 (53.8%)	26 (32.5%)	65 (81.2%)	64 (81.2%)

Before and after studies.

Five before and after studies, six diagnostic accuracy studies, and one decision modeling study.

Four before and after studies and six decision modeling studies.

Two decision modeling studies.

NS, not stated; RCT, randomized controlled trial.

With the exception of three of the reviews,^14,18,19 all other reviews used checklists for economic evaluation. One review¹³ used a checklist for assessing the quality of economic evaluations formulated by Haycox and Walley,²² three reviews^15
–17 adopted the checklist of Drummond et al.,¹ one review²¹ adopted the checklist of Evers et al.,² and the final review²⁰ assessed included articles according to the grading system of Chiou et al.²³

The quality scores obtained using the index of Oxman and Guyatt¹² varied from 4 (major/minor flaws) to 7 (minimal flaws); five of the nine reviews indicating minimal flaws in study performance of risk assessments^{13,15,17,20,21} (Table 1).

Summary of Economic Evaluation Methods Used

Table 3 shows the type of economic evaluation that was conducted and the perspective adopted for each of the included studies in the review. The four earlier reviews^13,16,17,19 included studies in which the majority were either cost analyses or cost-minimization analyses; one review did not state the type of economic evaluations that were conducted in their included studies,¹⁸ the later four reviews^14,15,20,21 tended to use the different range of economic evaluations that are available,¹ and there was a growing number of cost-utility studies. In terms of perspective adopted by the studies, Rojas et al.,¹⁸ Roine et al.,¹⁹ and Hailey et al.¹⁶ did not state in their reviews the perspective of the analysis that was adopted by any of their included studies; however, the other six reviews found that most articles did not state the perspective from which the analysis was conducted.

Table 3.

Type of Economic Evaluation and Perspective of Analysis

	TYPE OF ECONOMIC EVALUATION						PERSPECTIVE
REFERENCE (YEAR)	COST ANALYSIS	CMA	CCA	CEA	CUA	CBA	HEALTHCARE	PATIENT	SOCIETAL	HEALTHCARE AND PATIENT OR SOCIETAL	NS
Roine et al.(19) (2001)	47 (94.0%)	0	0	0	0	3 (6.0%)	NS	NS	NS	NS	50 (100.0%)
Whitten et al.(13) (2002)	5 (20.8%)	15 (62.5%)	0	4 (16.7%)	0	0	6 (25.0%)	1 (4.2%)	3 (12.5%)	1 (4.2%)	13 (54.2%)
Hailey et al.(17) (2002)	30 (73.2%)	2 (4.9%)	0	6 (14.6%)	0	3 (7.3%)	4 (9.8%)	2 (4.9%)	1 (2.4%)	0	34 (82.9%)
Hailey et al.(16) (2004)	25 (100.0%)	NS	NS	NS	NS	NS	NS	NS	NS	NS	25 (100.0%)
Rojas and Gagnon(18) (2008)	NS	NS	NS	NS	NS	NS	NS	NS	NS	NS	23 (100.0%)
Wade et al.(20) (2010)	0	13 (36.1%)	18 (50.0%)	2 (5.6%)	1 (2.8%)	2 (5.6%)	23 (63.9%)	1 (2.8%)	12 (33.3%)	0	0
Bergmo(14) (2009)	0	5 (15.2%)	19 (57.8%)	4 (12.1%)	5 (15.2%)	0	8 (24.2%)	0	2 (6.1%)	2 (6.1%)	21 (63.6%)
Peeters et al.(21) (2011)	0	0	7 (77.7%)	0	0	2 (22.2%)	4 (44.4%)	0	0	0	5 (55.5%)
Mistry(15) (2012)	0	18 (22.5%)	38 (47.5%)	15 (18.8%)	7 (8.8%)	2 (2.5%)	22 (27.5%)	1 (1.2%)	12 (15.0%)	9 (11.3%)	36 (45.0%)

CBA, cost–benefit analysis CCA, cost-consequences analysis; CEA, cost-effectiveness analysis; CMA, cost minimization analysis; CUA, cost-utility analysis; NS, not stated.

Table 4 shows the results from the economic evaluations for each of the included studies in the reviews. Hailey et al.¹⁶ and Wade et al.²⁰ did not clarify in their reviews what type of costs were included in each of the articles; the majority of studies that were included in each of the reviews had measured direct medical costs, whereas some of the reviews also included direct nonmedical and indirect costs. In terms of outcomes that were included in these studies, from the reviews we found that the majority of outcomes were either clinical or economic, and very few studies in each of the reviews included any utility-based measures as an outcome measure. These outcome measures included the generic-preference based measures such as the EQ-5D, SF-12, and SF-36, which can measure overall health-related quality of life. The majority of reviews found that the included studies did not report any incremental cost-effectiveness ratios; the latter study by Mistry¹⁵ found that 20% of the articles included in the review had reported an incremental cost-effectiveness ratio. Examples of some of the cost-effectiveness ratios included were cost per patient journey avoided, cost per additional patient treated, cost per waiting day avoided, and cost per quality-adjusted life year gained. All four ratios are useful to clinical decision makers; however, the last ratio is perhaps more informative for policy makers as this ratio allows for comparison across different diseases and treatments, and this ratio, depending on the threshold that policy makers are willing to pay for a new treatment/intervention, would enable them to decide whether this new treatment/intervention is cost-effective or not to the health service. Discounting was not conducted in most of the studies that were included in each of the nine reviews. In terms of sensitivity analyses, all but two reviews^18,21 mentioned the number of actual studies in their reviews that conducted some sensitivity analyses.

Table 4.

Costs, Outcomes, Incremental Cost-Effectiveness Ratios, Discounting, and Sensitivity Analyses

	COSTS				OUTCOMES						ICER		DISCOUNTING		SENSITIVITY ANALYSES
REFERENCE (YEAR)	DIRECT MEDICAL COSTS	DIRECT NONMEDICAL COSTS	INDIRECT COSTS	NS	CLINICAL	SATISFACTION	ECONOMIC	UTILITIES, LYGS, QALYS, DALYS	ASSUMED SAME	NS	YES	NS	YES	NS	YES	NS
Roine et al.(19) (2001)	25 (50.0%)	0	4 (8.0%)	21 (42.0%)	48 (96.0%)	0	0	2 (4.0%)	0	0	1 (2.0%)	49 (98.0%)	7 (14.0%)	43 (86.0%)	14 (28.0%)	36 (72.0%)
Whitten et al.(13) (2002)	23 (95.8%)	4 (16.7%)	3 (12.5%)	0	7 (29.2%)	0	8 (33.3%)	0	8 (33.3%)	1 (4.2%)	0	24 (100.0%)	0	24 (100.0%)	9 (37.5%)	NA
Hailey et al.(17) (2002)	36 (87.8%)	8 (19.5%)	7 (17.1%)	3 (7.3%)	8 (19.5%)	4 (9.8%)	14 (34.1%)	2 (4.9%)	2 (4.9%)	15 (36.6%)	1 (2.4%)	39 (95.1%)	3 (7.3%)	38 (92.7%)	10 (24.4%)	31 (75.6%)
Hailey et al.(16) (2004)	NS	NS	NS	25 (100.0%)	NS	NS	NS	2 (8.0%)	NS	23 (92.0%)	3 (12.0%)	22 (88.0%)	0	25 (100.0%)	7 (28.0%)	18 (72.0%)
Rojas and Gagnon(18) (2008)	23 (100.0%)	3 (13.0%)	13 (56.5%)	0	0	14 (60.9%)	23 (100.0%)	8 (34.8%)	1 (4.3%)	0	1 (4.3%)	22 (95.7%)	0	23 (100.0%)	0	23 (100.0%)
Wade et al.(20) (2010)	NS	NS	NS	36 (100.0%)	10 (27.8%)	1 (2.8%)	7 (19.4%)	0	0	22 (61.1%)	0	36 (100.0%)	0	36 (100.0%)	6 (16.7%)	30 (83.3%)
Bergmo(14) (2009)	33 (100.0%)	14 (42.4%)	8 (24.2%)	0	26 (78.8%)	7 (21.2%)	5 (15.2%)	15 (45.5%)	0	0	5 (15.2%)	28 (84.8%)	0	33 (100.0%)	13 (39.4%)	NA
Peeters et al.(21) (2011)	9 (100.0%)	0	1 (11.1%)	0	2 (22.2%)	3 (33.3%)	9 (100.0%)	2 (22.2%)	0	0	0	9 (100.0%)	0	9 (100.0%)	0	9 (100.0%)
Mistry(15) (2012)	79 (98.8%)	32 (40.0%)	15 (18.8%)	0	26 (32.5%)	13 (16.3%)	23 (28.8%)	18 (22.5%)	16 (20.0%)	0	16 (20.0%)	64 (80.0%)	8 (10.0%)	n/a	38 (47.5%)	NA

DALYs, disability-adjusted life years; ICER, incremental cost-effectiveness ratio; LYGs, life-years gained; NA, not applicable; NS, not stated; QALYs, quality-adjusted life-years.

Cost-Effectiveness of Telemedicine

In terms of conclusions provided by the different reviews, a general consensus found that there was no further evidence that telemedicine was cost-effective, and further assessment was needed.

The reviews also highlighted various issues: conclusions were influenced strongly by local conditions, thereby making it hard to generalize¹⁷; a major dearth in the methodology of the studies and its ability to produce robust and generalizable conclusions¹³; the overall reliability of a study for decision making will depend on both performance and study design¹⁶; and the challenges faced in economic evaluations in telemedicine are diverse because of nature of both study contexts and methods applied.¹⁴ Overall, there was also a general consensus that most studies were not carried out in accordance with standard economic evaluation guidelines.

Discussion

As far as we are aware, this is the first systematic review of published systematic reviews looking at the cost-effectiveness of telemedicine. Based on our specific inclusion criteria, this appraisal identified nine systematic reviews that span a 12-year period, addressing the cost-effectiveness of telemedicine applications in the delivery of healthcare. The reviews identified and addressed a wide variety of telemedicine applications in different health areas; however, the evidence with regard to its cost-effectiveness has remained inconclusive.

In summary, there were slight variations in the research question for each review; however, all reviews wanted to systematically assess the cost-effectiveness of general telemedicine interventions. Methods for literature search used are vital to ensure that the review is complete and up-to-date and also for readers to replicate these searches. For most of the reviews, the search strategies used were clear and comprehensible, and there was a more coherent flow of methodology with each succeeding review.

Most reviews reported that analyses of costs and cost-effectiveness in many of the included studies lack consistency as to how cost estimates were collected and reported. The information on benefits in all the studies ranged from clinical outcomes to cost savings from avoided travel through to quality of life measures; however, these benefits were not properly reported in most studies to enable a proper cost-effectiveness analysis.

The overall cost-effectiveness of telemedicine was inconclusive mainly because many studies reported both significant and nonsignificant findings. All the reviews expressed difficulty when drawing conclusions on the overall cost-effectiveness, as included studies were often small scale with poor designs and ill-defined outcomes. Conversely, Rojas and Gagnon¹⁸ did not provide assertions on the cost-effectiveness, presumably because of the lack of proper measures used in reporting cost-effectiveness, so therefore the authors based their conclusions on general statements rather than results presented in their review.

There are also challenges in generalizability of the cost-effectiveness of telemedicine due to the heterogeneous nature of the fields to which telemedicine was applied. Issues such as the variation in study design, setting, technologies used, and health areas to which telemedicine was applied can impact on its generalizability. However, Wade et al.²⁰ tried to deal with this by only considering real-time telemedicine. These authors suggested that grouping telemedicine applications into similar organizational settings can mitigate the problem of generalizability. Because of the heterogeneous nature of the data of the included studies, no reviews have conducted a meta-analysis; however, this is not unusual as results from cost-effectiveness studies are not usually presented as a meta-analysis or a meta-review.

We found that, over time, reporting of the cost-effectiveness has generally improved, but there is still room for a lot more improvement. As with the previous systematic reviews, Mistry¹⁵ found that there was a continual increase in use of economic tools for evaluation of telemedicine over time. However, there still remains a lag in economic studies' adherence to the general standards for health economic evaluations^1,2 and the transparency of reporting methodologies and results. In addition, researchers do not use the existing frameworks for conducting economic evaluations in telemedicine.^4,5 Zanaboni and Wootton²⁴ found that the widespread adoption of telemedicine was a major and still an underdeveloped challenge. They concluded that even though “cost-effectiveness of telemedicine is necessary, it was not a sufficient condition for the widespread adoption of telemedicine.”

The findings from this appraisal also raise questions as to whether reviews published may have included studies that reported bad-quality results. The only way to look at this in more detail would have been to compare the results reported in the reviews with the original studies included in each of the reviews. The most recent review by Mistry¹⁵ also highlighted some important considerations in assessing articles for publication in journals, such as restrictions on word counts preventing detailed reporting of methodologies and findings, and often health economists are not named as co-authors on articles, or they may not have contributed to analysis or to the write-up of the cost-effectiveness analysis.

This review did not consider applications in a specific health area, and we realize that this can be seen as a limitation. Although this review covers a wide variety of clinical specialty applications, it is quite difficult to reach reliable conclusions, as each health area and the nature of the applications are very different, but, as we have emphasized earlier, this is the first review of systematic reviews looking at the cost-effectiveness of telemedicine. Another limitation is that the conclusions in this article regarding the cost-effectiveness of telemedicine are drawn from the findings of systematic reviews that are mostly based on cost-effectiveness studies conducted that have immense methodological flaws. As Booth²⁵ reported, evidence from a critical appraisal is in itself limited by the quality of reporting of the included reviews. Furthermore, these reviews on the cost-effectiveness of telemedicine need to be continuously updated because new research is being done all the time as technologies are changing and some telemedicine technologies are becoming obsolete.

Ultimately, the systematic reviews unanimously concluded that evidence on the cost-effectiveness of telemedicine in general is inconclusive. Therefore, this review still reaffirms the knowledge of the inconclusive nature of telemedicine assessment, which is based on small sample sizes, short-term time horizons, failure to state the study perspective, and no proper calculation of costs and benefits.

Even though the quality of most the systematic reviews indicated minimal flaws, the methodological deficiencies found in the individual studies included in the systematic reviews made them more prone to bias and limit their value to aid in decision making. Therefore, decision makers should take caution when making informed decisions, and their decisions should be made based on both the methodological rigor of the research and the quality of its findings. However, caution needs to be taken as both the aspect of study context and methods applied are equally important. Ultimately, the value of systematic reviews depends on their replicability, validity, precision, and relevance.

Finally, findings from this appraisal reinforce the need for further studies targeted at economic evaluation of telemedicine to provide robust and empirically valid evidence regarding the cost-effectiveness of telemedicine.

Footnotes

Acknowledgments

This review originally formed part of the MSc dissertation thesis of H.G. and has been updated accordingly for publication.

Disclosure Statement

No competing financial interests exist.

References

Drummond

, Sculpher

, Torrance

, O'Brien

, Stoddart

. Methods for the economic evaluation of health care programmes, 3rd ed. New York: Oxford University Press, 2005.

Evers

, Goossens

, de Vet

, van Tulder

, Ament

. Criteria list for assessment of methodological quality of economic evaluations: Consensus on health economic criteria. Int J Technol Assess Health Care, 2005; 21:240–254.

Husereau

, Drummond

, Petrou

, Carswell

, Moher

, Greenberg

, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. BMJ, 2013; 346:f1049.

McIntosh

, Cairns

. A framework for the economic evaluation of telemedicine. J Telemed Telecare, 1997; 3:132–139.

Sisk

, Sanders

. A proposed framework for economic evaluation of telemedicine. Telemed J, 1998; 4:31–37.

Bahaadinbeigy

, Yogesan

, Wootton

. Gaps in the systematic reviews of telemedicine fields. J Telemed Telecare, 2010; 16:414–416.

Ekeland

, Bowes

, Flottorp

. Effectiveness of telemedicine: A systematic review of reviews. Int J Med Inform, 2010; 79:736–771.

Ekeland

, Bowes

, Flottorp

. Methodologies for assessing telemedicine: A systematic review of reviews. Int J Med Inform, 2012; 81:1–11.

Clarke

, Shah

, Sharma

. Systematic review of studies on telemonitoring of patients with congestive heart failure: A meta-analysis. J Telemed Telecare, 2011; 17:7–14.

10.

Gaikwad

, Warren

. The role of home-based information and communications technology interventions in chronic disease management: A systematic literature review. Health Informatics J, 2009; 15:122–146.

11.

Moher

, Liberati

, Tetzlaff

, Altman

; PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA statement. PLoS Med, 2009; 6:e1000097.

12.

Oxman

, Guyatt

. Validation of an index of the quality of review articles. J Clin Epidemiol, 1991; 44:1271–1278.

13.

Whitten

, Mair

, Haycox

, May

, Williams

, Hellmich

. Systematic review of cost effectiveness studies of telemedicine interventions. BMJ, 2002; 324:1434–1437.

14.

Bergmo

. Can economic evaluation in telemedicine be trusted? A systematic review of the literature. Cost Eff Resour Alloc, 2009; 7:18.

15.

Mistry

. Systematic review of studies of the cost-effectiveness of telemedicine and telecare. Changes in the economic evidence over twenty years. J Telemed Telecare , 2012; 18:1–6.

16.

Hailey

, Ohinmaa

, Roine

. Study quality and evidence of benefit in recent assessments of telemedicine. J Telemed Telecare, 2004; 10:318–324.

17.

Hailey

, Roine

, Ohinmaa

. Systematic review of evidence for the benefits of telemedicine. J Telemed Telecare, 2002; 8(Suppl 1):1–30.

18.

Rojas

, Gagnon

M-P

. A systematic review of the key indicators for assessing telehomecare cost-effectiveness. Telemed J E Health, 2008; 14:896–904.

19.

Roine

, Ohinmaa

, Hailey

. Assessing telemedicine: A systematic review of the literature. CMAJ, 2001; 165:765–771.

20.

Wade

, Karnon

, Elshaug

, Hiller

. A systematic review of economic analyses of telehealth services using real time video communication. BMC Health Serv Res, 2010; 10:233–246.

21.

Peeters

, Mistiaen

, Francke

. Costs and financial benefits of video communication compared to usual care at home: A systematic review. J Telemed Telecare, 2011; 17:403–411.

22.

Haycox

, Walley

. Pharmacoeconomics: Evaluating the evaluators. Br J Clin Pharmacol, 1997; 43:451–456.

23.

Chiou

C-F

, Hay

, Wallace

, Bloom

, Neumann

, Sullivan

, et al. Development and validation of a grading system for the quality of cost-effectiveness studies. Med Care, 2003; 41:32–44.

24.

Zanaboni

, Wootton

. Adoption of telemedicine: From pilot stage to routine delivery. BMC Med Inform Decis Mak, 2012; 12:1.

25.

Booth

. Who will appraise the appraisers?—The paper, the instrument and the user. Health Info Libr J, 2007; 24:72–76.