Abstract
Dear Editors:
We offer a few comments regarding a recent article by Mistry et al. 1 published in this Journal entitled “Critical appraisal of published systematic reviews assessing the cost-effectiveness of telemedicine studies.”
Normally, we would not engage in comments as a rebuttal to a published article, but this article puts forth misleading and unwarranted conclusions at a critical period in the telemedicine policy debate. At issue are the merits of telemedicine as a cost-effective healthcare option. The authors repeatedly conclude or note a general consensus in their selective review regarding the lack of evidence for telemedicine cost-effectiveness, which may be interpreted as telemedicine not being cost-effective. However, a closer examination reveals the inaccuracy of this interpretation and that it stems from errors in logic and methodology. That is, in our view, the authors' conclusions about cost-effectiveness of telemedicine are not supported.
There are serious problems with the approach taken in this article: 1. There is a methodological issue as to whether an assessment on cost-effectiveness can be derived from a “review of reviews,” especially when the number of such reviews dwindles from over 4,000 initial reported abstracts to 9, only 3 of which covered the last 5 years. Basically, a clinically based methodology is used to index the quality of the reviews. The authors then presume to make a valid assessment of cost-effectiveness findings based on the determinations of review quality, such as it is. The question is whether the index of review quality provides sufficient basis for drawing conclusions about the bodies of information within the selected reviews. The authors' challenge was to establish and apply an index that captures the widely differing levels of scientific merit of reviews so as to create a valid summary of cost-effectiveness findings. This requires a measure that summarizes the study findings on cost-effectiveness as these were reported through a flawed lens of reviews applied across a broad range of study designs and sample sizes. And often, these selected reviews are themselves significantly flawed, especially in terms of internal and external validity, as well as in defining uniform units of analysis. Nonetheless, the authors of this article pursued this problem by scoring review quality and veracity, applying a scale derived from the work of Oxman and Guyatt.
2
Then, the authors further undermined a questionable assessment framework by disregarding the framer's methodology, such as assigning themselves (two of the authors) as the judges, rather than obtaining independent rankings, and by trying to score the quality and veracity of the reviews on a 7-point scale. Initially the scale had nine items (p. 610). A “10th item collates the information from the first nine items by having the assessor grade the review on an ordinal scale ranging from 1 (minimal flaws) to 7 (extensive flaws).” Based on this summary score, five reviews were rated as 7, one as 6, and two as 5, and one was rated best with a 4. Yet, strangely enough, the rating scale is inverted on page 614, with 7 being minimal flaws. 2. Second, this article is conceptually flawed in the determination of the scope of the analysis. The authors restricted their analysis to “the general field of telemedicine” (p. 609), as opposed to telemedicine studies defined by clinical application, technological configuration, provider mix, patient mix, and context. For example, teleradiology and telemonitoring may be excluded from the analysis. The stated purpose of this approach is “to make the results more generalizable” (p. 610). However, there appears to be some confusion between the “generalization of a technology” and the generalizability of results (i.e., external validity). Consequently, the selected reviews incorporated studies with technologically heterogeneous applications, which often have differing impacts, costs, and contextual suitability. And, the likely problem with evaluations focusing on “general telemedicine” appears in the concluding discussion section of the artcile. The authors' argument for reviewing “general telemedicine” seems to be undermined when they note the “… challenges in generalizability of the cost-effectiveness of telemedicine due to the heterogeneous nature of the fields to which telemedicine was applied” (p. 616). Indeed, in studying telemedicine costs, in particular, it makes little sense to view the entire field as a single or unidimensional telemedicine intervention. Precise specifications of telemedicine applications are often required to ascertain the production process that makes the cost-effectiveness impacts of telemedicine more discernible. 3. Finally, and perhaps most importantly, this article is an indictment of the inadequate methodologies used in systematic reviews of cost-effectiveness for telemedicine. Systematic reviews in Health Policy Systems Research (HPSR) provide summaries of scientific knowledge for policy and organizational decision-making. But, the findings of these systematic reviews have been incomplete because of the application of clinically oriented systematic review protocols. These clinically oriented systematic reviews have not adequately dealt with the required HPSR study information, such as findings on contextual impacts, sensitivity to self-selection, fidelity of implementation, maturation, and bundling along with the traditional methodological rigor, including research design, sample size, frequency, and duration of the study. With HPSR systematic reviews, methods have been rapidly emerging over the last several years, but alas, the conclusions from the review of reviews discussed here depended largely on applications of the older methodologies with significant limitations when applied to the cost-effectiveness of healthcare delivery innovations.
To summarize, this “review of reviews” applied weak or inappropriate tools in assessing selected studies and filtered out some of the most important ones. The defined scope of the studies for review eliminated important findings on cost-effectiveness. Moreover, although this article focused on cost-effectiveness analysis, this is not the only or optimal methodology for assessing the impacts of telemedicine applications, nor is it the only method for assessing economic efficiency. For example, numerous studies have investigated the impact of telemedicine interventions on the use of services, such as hospital admissions/re-admissions, visits to the emergency room, and length of hospital stay. Typically these studies are not categorized as cost-effectiveness studies, but they add insight into the cost issue. Finally, in the current state of methodological development, all evidence from research studies must be viewed from the perspective of the specific design features of the studies from which they derive, and systematic reviews of findings must be based on actual studies, not a derivative from other reviews.