Economic evaluation in telemedicine – still room for improvement

Introduction

There has been a general improvement in the quality of evidence in telemedicine research in recent years. ¹ Does this apply to economic evaluation as well? It has been reported that economic evaluations of telemedicine are less adherent to methodological standards than economic evaluations in other fields. ² In view of the importance of economic evaluation in priority setting, I have recently carried out a systematic review of economic evaluation methodology. I restricted it to full economic evaluations, i.e. evaluations with a comparative analysis of both costs (resource use) and outcomes (non-resource consequences). ³

My purpose in conducting yet another review of telemedicine was to focus on methodological matters, especially the measurements and valuation of non-resource consequences. Previous reviews of telemedicine have found that most studies evaluated benefits in terms of the cost savings, with no assessment of the health benefits for patients. ^4,5 My review explored whether more recent evaluations have included patient outcomes, how these were measured, valued and compared to the costs. The objectives were to provide detailed information about evaluation approaches and applied methods, to assess the quality of the evaluations and their potential contribution to priority setting, and to discuss the applicability of best-practice methods to telemedicine settings. The main indicators used to assess the articles reviewed were clarity of study objective, adequacy of comparison, choice of study perspective and design, transparency in the measurements and valuation of costs and outcomes, reporting of data sources, handling of uncertainty and presentation of results.

I found 33 articles that measured both the costs and the non-resource consequences of using telemedicine in direct patient care. Compared to previous reviews that found most studies had evaluated benefits in terms of cost savings, this is a considerable increase. The articles analysed were highly diverse in both study context and applied methods. They covered several medical specialities including cardiology, dermatology and psychiatry, and they analysed telemedicine in home care, in primary care and in secondary care, using a variety of different technologies including videoconferencing, still-images and monitoring (store-and-forward telemedicine). Six of the evaluations were decision-modelling studies using secondary data to analyse costs and consequences, with 4–14 primary studies used as data input. The few studies of good quality may limit the use of decision modelling.

One third of the studies were published in the two specialist telemedicine journals, Journal of Telemedicine and Telecare and Telemedicine and E-health (see Table 1). Most studies used multiple outcome measures and took a disaggregated approach to costs and consequences. For example, one study measured ECG, pulse, spirometry, body mass index and quality of life (SF-36) and listed these alongside the cost without deciding on a one-dimensional measure. Multidimensional presentation of the consequences can also be difficult to interpret, especially if the different outcome dimensions move in different directions. ⁶ The benefit side of the equation appeared more systematic, consistent and well-reported than the costing. This is somewhat counter-intuitive, since measuring health outcome can be difficult. For telemedicine research this seems even more relevant because the intervention in most settings is replacing in-person consultation for diagnostic purposes.

The outcome measures reported ranged from impact on process to the final outcome. The outcome measures were for example diagnostic accuracy, medication compliance, number of days to independent pouch change after abdominal surgery and colostomy, percentage reduction in wound size, length of stay (LOS), blood glucose levels, anxiety and depression levels, physical capacity, health-related quality of life (HRQL), life years gained (LYG) and quality-adjusted life-years (QALYs) gained.

In settings where telemedicine services are replacing conventional in-person encounters between patients and health-care personnel, disease- or case-specific measures (such as blood glucose levels or measures related to skin problems) can be sufficient to estimate the relative effectiveness of telemedicine for patient management. These measurements can even be at an ordinal level: inferior, equivalent or superior. If specific outcome measures show equal or better patient outcomes than usual care, then the next step is to assess the differences in costs using standard costing techniques (one should note, however, that services could generate less benefit at lower cost and still be considered cost-effective). This approach to telemedicine evaluation will however limit generalisability and make it impossible to compare or synthesise results from evaluations with different disease-specific outcome measures.

In settings where telemedicine is used to provide new or additional services alongside traditional care, such as monitoring of chronic conditions for patients living at home, there could be potential improvements in patients' health. For example, if investing in telemedicine costs more and is more effective, the decision-maker would need information on how much more beneficial it is for the costs involved. To be able to compare this with other services, generic health status measurements such as QALYs or LYG are required.

Consistency in effectiveness measures has important implications for the usefulness of cost-effectiveness results to decision-making. ⁷ If the objective of using telemedicine in dermatology is to improve wound healing, it seems appropriate for the endpoint to measure wound size. On the other hand, it can be difficult to interpret cost-effectiveness in terms of a specific cost per reduction in wound size. While this example is acceptable for assessing technical efficiency, (i.e. how to produce a given level of health outcome at least cost) this will not help in deciding how to allocate resources across programmes. In these situations, generic health measures are required to allow for comparison between the two, such as LYG or QALYs. Few studies have used such generic health measures. One reason for this may be that they are not sensitive enough to detect the small changes in health outcomes which telemedicine services are most likely to produce. In most cases, telemedicine use will probably not affect survival. Another reason may be that most economic evaluations have been undertaken to justify decisions within clinical areas and to support reimbursement and payment systems, and not as a basis for broader decision-making.

Whether to use disease-specific or generic tools to measure the consequences should be viewed in relation to the objectives and the type of services provided. Health outcome might not be a benefit in the decision context. The purpose of employing telemedicine might be to provide consultations or episodes of care, not long-term health improvements. ² Telemedicine evaluations should however ensure that the technology is safe and generates as much benefit as conventional care before any decision about implementation is made.

Lack of transparency in costing methods was a general problem in the studies reviewed. In some studies it was impossible to tell the type and magnitude of the cost items included and others provided limited or unclear cost information and did not present unit cost data alongside resource data. The latter is important because it enables the readers to assess whether all appropriate costs are included and to judge whether the cost results could be adapted to their own setting. Measuring and valuing the costs can be a major challenge, especially if the telemedicine services involve a mix of complex delivery systems and technologies. Aspects that need extra attention include how to handle shared resources, production capacity, marginal costing and the use of salaries and charges as proxies for opportunity costs. In most health-care systems, charges or tariffs are only financial indices with no relation to actual resource consumption. In health systems with both tariffs and lump sum financing, tariffs may cover only part of the total cost of an activity. Another problem with using hospital accounting systems is that some resources that are used but not billed may be overlooked. Different costing patterns may also make results from one study less adaptable to other settings.

The studies reviewed appeared to have used some combination of an ‘ingredients’ approach and an ‘activity’ based approach to costing. The former is a costing method where every cost item is broken down into its underlying components, while the latter does not and refers to, for example, the costs of a hospital bed-day without any further information. An ‘ingredients’ approach was used to calculate the cost of providing the telemedicine service or the intervention, while ‘activity’ based costing was mainly used to calculate hospital costs and the costs of home care visits. This combination of ‘ingredients’ and ‘activity’ approaches is common in economic analyses in health care in general. ⁸

In a real clinical situation it seems unlikely that telemedicine could be a complete substitute for in-person encounters; some combination of the two will probably be required. This requires careful planning of what constitutes the different comparators in the evaluation. Similarly, analysing two different telemedicine alternatives without comparison to a current alternative will not produce relevant information if the objective is to decide whether to invest in or reimburse telemedicine services. Another challenge is the inclusion and valuation of time costs. Very few studies provided any reasons for including time costs. In practice, patients may already be off work because of their health condition, leaving the actual production loss unchanged. There is however, a low level of agreement in the health economic literature about whether to include productivity changes or not. ^9,10

In my review I found eight studies to be rigorous economic evaluations, i.e. they had addressed all the key evaluation criteria (a clear study objective, adequate comparison(s), reporting of study perspective and design, transparent measurements and valuation of costs and outcomes, reporting of data sources, addressing of uncertainty and clear presentation of the results). I also viewed most of them as high quality studies. The remaining studies however had several shortcomings primarily on technical aspects and reporting of results. Objectives, study design, and choice of comparators were mostly well reported. The majority of the studies lacked information on perspective and costing method, few used general statistics and sensitivity analysis to assess validity, and even fewer used marginal analysis. These shortcomings in economic evaluation methodology are relatively common ¹¹ and have been found in other fields of research. ^12–14

Despite the somewhat gloomy conclusion of my review, ³ there seems to have been an improvement in the conduct of evaluations which report both costs and outcomes. In future evaluation work the focus should be on making the costing more consistent and transparent, and synthesising costs and outcomes in a cost per unit of effect measure (see, for example, the BMJ's guidelines for economic evaluation ¹⁵ ).

Even though there is still room for improvement in the quality and conduct of economic evaluation of telemedicine, it is worth noting that no economic evaluation is completely free of flaws. It is therefore important that potential users assess the quality of the evaluation themselves before employing the results. It is also important to judge any economic evaluation based on whether it can potentially lead to a better decision than would have been the case had the evaluation not been available. ¹¹