Cost–Benefit Analysis on the Use of Telemedicine Program of Kosova for Continuous Medical Education: A Sustainable and Efficient Model to Rebuild Medical Systems in Developing Countries

Abstract

Background: The Ministry of Health of Kosova has recently announced the Telemedicine Program of Kosova (TMPK) as the official institution responsible for managing and coordinating the nation's Long-Distance Continuous Medical Education (CME) program. There are a lack of studies on cost–benefit analysis (CBA) and other economic evaluations of telemedicine programs (TMP), in particular the financial value of CME offered through such a service. In addition, there is lack of prospective studies on Monitoring & Evaluation (M&E) of TMP. Objective: The goal of this study was to conduct a retrospective CBA of prospective data collected at TMPK over a 5-year period (2005–2010) in order to determine the cost benefit as opposed to the alternative method of delivery of this model for developing countries whose healthcare systems are in disarray. Materials and Methods: We reviewed data on the number of participants in virtual lectures both at the Telemedicine Center of Kosova (TCK) as well as the number of participants at six Regional Telemedicine Centers throughout Kosova, the number of lectures broadcasted, the clinical cases reviewed and transmitted for international consultation, and other quantitative data. Results: Only in 2009, approximately 2,000 CME certificates were awarded to physicians and nurses of Kosova, 18 international teleconsultations were conducted, 138 videoconferences, lectures, and seminars were held, and there were over 9,000 visitors at the TCK e-library. Data analysis shows that the TMPK has been an efficient mechanism for CME and sustainable model for rebuilding the medical system. TMPK has been successful in offering physicians, nurses, and other medical professions access to electronic information. Conclusions: TMP is an efficient mechanism to ensure CME and rebuilding medical systems in developing countries. There is a need for prospective CBA of any TMP and the establishment of M&E programs in any future telemedicine initiatives in developing countries.

Background

The International Virtual e-Hospital (IVeH; www.iveh.org) has established a robust telemedicine program in the Republic of Kosova (TMPK; www.telemedks.org). The program is located at the University Clinic Center of Kosova (UCCK) in Prishtina, Kosova. Initiated and developed in the aftermath of war, the program applies innovative technology, which filled an unmet need. IVeH has presented the Initiate, Build, Operate, Transfer (IBOT) strategy to neighboring countries such as Albania, Macedonia, and Montenegro.¹ In addition, the Telemedicine Program of Albania has inaugurated six new telemedicine centers distributed among regional hospitals as well as the University Clinic Center of Tirana, which serves as the national center.²

Initiate Build Operate Transfer (IBOT)

IVeH is a not-for-profit 501(c)(3) organization established in 2001 (formerly known as the Kosovo Medical Development Foundation). IVeH is dedicated to bringing telemedicine and healthcare system improvements to developing countries around the world. The model implemented in Kosova to establish a sustainable telemedicine program in the Balkans is the IBOT strategy.¹ This strategy was the blueprint to build a sustainable and effective telemedicine program by exporting American technologies.

Initiate

The journey began with the First Intensive Balkan Telemedicine and e-Health Seminar, held in October 2002 in Prishtina.³ Over 300 medical and military professionals from across the globe attended the event. The conference provided IVeH with the opportunity to formally present the tremendous opportunities and capabilities of telemedicine by world-renowned experts while simultaneously showing how other telemedicine programs can be applied to Kosova's (then) current situation. The tremendous and enthusiastic support, interest, and encouragement by the medical community, government officials, and others fueled the foundation's determination to continue on to build the Telemedicine Center of Kosova (TCK).

Build

Due to the support of the Ministry of Health (MOH), the Prime Minister, and the international community present in Kosova, IVeH was given a location within the campus of the UCCK in Prishtina, on the 5th floor of the Emergency Services building and in complete shambles. IVeH had been awarded a grant for approximately 1.2 million dollars from the European Agency for Reconstruction (EAR), which would cover the high cost of reconstruction of the site, equipment, and personnel. The site was inaugurated in December of 2002.

Operate

This is likely the most challenging phase of the IBOT strategy. This phase is characterized as the capacity building phase where emphasis is placed on creating telemedicine experts and medical ambassadors. In developing countries, new institutions or new concepts are sometimes met with hesitation from those who will benefit the most: medical professionals. With the assistance of the U.S. Department of State's Bureau for Education and Cultural Exchange, IVeH selected highly motivated doctors and nurses to participate in a Telemedicine Fellowship program at operational telemedicine programs in Alaska and Arizona. During the operation phase, the TCK expanded to the five regional hospitals of Kosova and a primary care hospital in the rural town of Skenderaj. Fellows were selected to expand telemedicine expertise within the six newly created regional centers.

Transfer

This phase is ideally a time where the completed program is given to the public institution, which it serves. For TCK, the MOH became the owners of the telemedicine centers and equipment. Institutionalization of telemedicine is vital for sustainability. It is important that telemedicine become part of standard protocol and procedures. Although, the legal and technical aspects of the transfer phase were completed in 2005, TCK is still faced with institutionalization. The current program has yet to infiltrate standard medical practices; however, TCK remains diligent and optimistic in this endeavor.

Introduction

Although there is a wide perception of great benefits of telemedicine programs for developing countries, analysis of such programs is uncommon. Analysis is typically conducted as implementation evaluation: Is the program on schedule and are goals being met? Programs are individually, very dynamic, and it may be difficult to adjust the characteristics of the program to the evaluation method.⁴ In addition, due to constant new developments in technology, the ever-changing face of telemedicine, programs introduce new technology³ and adjust to the needs of the population, which can be problematic when evaluations call for long-term data and analysis.

Although the TCK has only recently become a government-run program, based on the IBOT strategy, it is important to consider the types of methods of evaluation that compliment the program. There are three types of evaluation methods: formative, process, and outcome evaluation.⁵ The TCK has surpassed the initial stages of pretesting and pilot program and has evolved into the process or implementation stage.

TMPK is particularly active in videoconferencing, broadcasting live surgeries (for educational purposes) both nationally and internationally, and conducting lectures and seminars between the TCK in Prishtina to other sites within Kosova and externally. During 2008 and 2009, TCK broadcasted over 50 lectures and seminars that were endorsed by the MOH and Continuous Medical Education (CME) program of Kosova. It is also been able to provide patients with teleconsultations upon request. The e-library is also an important part of the telemedicine platform in Kosova, which provides access to databases such as HINARI (www.who.int/hinari/en/) and PubMed (www.ncbi.nil.nih.gov/pubmed/).

Methods

For the purposes of this program evaluation, we assessed the benefits of the following intermediate processes or indicators based on the costs of alternative methods:

1. CME program participants (saved travel costs for international lecturers, time, etc.);

2. CME lectures (saved travel costs from regional centers and participation fees);

3. Teleconsultations and other similar activities

4. E-library visitors (including time spent and article downloads)

The TCK was designed with education as its main purpose and pillar. Today, education encompasses various methods and formats and is no longer restricted by a formal pedagogy in institutions or classrooms. The TCK has been actively pursuing medical education since 2002.

By our definition, educational events come in several different forms: (1) multi-point videoconference between TCK and regional hospital centers; (2) multi-point videoconference between TCK, regional hospital centers, and abroad: (3) live broadcast from UCCK operating rooms and the TCK auditorium; and (4) lectures and seminars. Figure 1 illustrates the yearly activities from 2005 to 2010, based upon the above-mentioned categories.

Fig. 1.

Yearly activities (2005–2010) using different approaches for content delivery.

The TCK's structured and approved CME program began in 2008. While the highest number of lectures were broadcasted during 2008 and part of 2009, the program continues and is now focusing on national and regional lecturers/experts. A series of broadcasted lectures for many specialties such as radiology, general surgery, pediatric surgery, and internal medicine were conducted. Topics included “Nutrition for Trauma and Critically Ill Patients,” “Evaluation and Management of Adult Trauma Patients,” “Surgical Treatment of Orbital Tumors,” and “New Digital Technologies in Radiology.” With over 50 lectures, the MOH provided physicians and nurses who attended the lectures with over 2,000 CME credits.

It would be very difficult to assess the financial benefits on the audiences' overall knowledge transfer; therefore, we are left to determine the general financial benefits of broadcasting lectures and other activities from the main TCK location to the adjoining six regional hospitals. By determining the costs of the alternative method, for example, 30 lecture participants traveling from Prizren to Prishtina (45.9 miles one way), these figures will provide the basis for comparison. The majority of cost estimations are based on U.S. Government figures, whereas the remaining costs are based on real-time costs of services (Internet, travel, hotel stay, fuel, etc.).

Results

It is important to establish basic costs such as mileage reimbursement, average hourly wage per staff member, and meals/incidental expenses. The number of participants does not accurately capture the number of participants per lecture; therefore, the data are presented by the number of participations, which means the number of participants in the entire lecture.* The total number of participations during the lecture series was 3,796. Table 1 shows the calculations for all regional hospitals.

Table 1.

Continuous Medical Education Program Benefits

REGIONAL TELEMEDICINE CENTER	NUMBER OF PARTICIPATIONS	KILOMETERS FROM TCK (MILES)	MILEAGE REIMBURSEMENT (PER PARTICIPANT—0.50 PER MILE)	MEALS AND INCIDENTALS	SALARY REIMBURSEMENT	TOTAL BENEFIT
Prizren	94	74 km (45.9 m)	$45.90	$7,143.6	$1,692	$7,134
Gjakova	1,419	86 km (53.4 m)	$53.40	$94,230	$1,428	$94,230
Peja	1,457	82 km (50.9 m)	$50.90	$32,498	$4,068	$32,076
Mitrovica	452	35 km (22.3 m)	$22.30	$32,076	$13,113	$32,498
Gjilan	164	46 km (28.5 m)	$28.50	$8,118	$1,476	$8,118
Skenderaj	210	65 km (40.4 m)	$40.40	$21,840	$3,150	$21,840
Total benefits						$195,989

TCK, Telemedicine Center of Kosova.

The results show that the CME program benefit totals $195,989 or approximately 51 U.S. dollars (USD)/participation. Other factors that were considered for this calculation include average rate of pay per hour per participants (3 USD) and meals/incidentals (12 USD/person). These rates are based on the standard rates used by U.S. State Department (http://aoprals.state.gov/web920/per_diem.asp).

The CME program delivered by the telemedicine network also allowed for a significant reduction in costs concerning the travel and reimbursement costs for the lecturers. The TCK was not obliged to pay for travel costs and overnight stay, per diem based on U.S. State Department rates, nor were the lecturers paid an honorarium. IVeH determined the average travel cost per lecturer to be between 900 USD to 1,200 USD from 2006 through 2009. In addition, lodging costs were estimated at 100 USD/night. Based on the above-mentioned costs, if the lecturers traveled to Kosova it would have cost TCK approximately 210,000 USD.

In addition, CME programs are generally implemented by private medical associations/organization. Membership fees apply to participants, as well as conference fees. If the participants were obliged to pay membership fees and conference fees, IVeH has applied the average cost of 125 USD/participation. This amounts to 474,500 USD. The total benefits (and costs) per participation is 230 USD.

E-Library

The electronic library (e-library) at the TCK is a tremendous accomplishment and benefit to the staff of the UCCK. This facility allows physicians and nurses and other medical staff access to the PubMed, HINARI, and Up to Date databases. The e-library, which consists of 15 desktop computers, is open 24 h a day and has been a small educational sanctuary in the lives of busy medical professionals. The complete e-library system (as well as equipment for teleconsultations and videoconferencing system) is maintained by the TCK's technical director, assistant information technology (IT) officer, and 2 e-library officers.

Within the parameters of the cost–benefit analysis (CBA), IVeH determined that it would be important to assess the benefits of the e-library, as it accounts for the vast majority of visitors to the TCK and compliments the CME program. In 2010, over 11,000 medical professionals entered the e-library in the TCK. Figure 2 illustrates the number of visitors since 2006.

Fig. 2.

E-library visitors.

Although the number of visitors was duly monitored, for the purposes of the CBA, additional information was needed. Therefore, IVeH conducted a small quantitative survey with visitors of the e-library to determine many key questions that would quantify or monetize e-library visitations: the number of visitations per individual each week, number of articles downloaded, and average time spent in the e-library. The results showed that among the most frequent visitors are doctors, visiting the e-library twice a week, with an average stay of 2.7 h. Although there is no way to determine the type of activity during the login period, IVeH has taken further steps to monitor the activity. In addition, visitors' viewed/downloaded one article. The cost for Internet services for the average time spent at the e-library would be 2.4 USD, also purchasing articles average at 20 USD per article; therefore, IVeH determined that the total benefit was 22.4 USD/visitor.

In February 2011, 13 Regional Telemedicine Program Coordinators of Albania attended a 2-day training program (which was endorsed by the CME program of Albania). Based on comparing costs of implementing a similar program at our partner the University of Arizona, TCK resulted in an indirect benefit of 40,000 USD. All monetized benefits are displayed in Table 2.

Table 2.

Total Benefits Are Accumulated

BENEFITS	YEAR 1	YEAR 2	YEAR 3	YEAR 4	YEAR 5
Direct benefits
CME program			$97,949	$97,949
E-library visitors	$55,664	$56,296	$73,867	$108,537	$137,392
Lecturer costs	$20,000	$16,000	$22,000	$92,000	$60,000
Indirect benefits
Conference fees			$272,250	$272,250
Telemedicine Program of Albania Training Program					$40,000
Grand total benefits					$1,433,854

CME, continuous medical education.

Costs: Tck Annual Budget

As previously discussed, the TCK was transferred to the Republic of Kosovo's MOH in 2005. All management and finances are now the responsibility of the central government. From 2002 until 2005, TCK operated on the 1.2 million USD awarded by the EAR. Approximately 40% (500,000 USD) of the total grant money awarded was spent on reconstruction of the site/location of TCK. This amount can be amortized over the next 20 years, resulting in an annual fixed cost of 25,000 USD. Therefore, for the total fixed costs from initial investment and 2006 until 2010 amount to 667,500 USD.

TCK's operating budget provided by the MOH, similar to other institutions, is approved on an annual basis and may vary (and does so) from year to year. In 2010, the Republic of Kosova's MOH allocated approximately 145,000 USD, which is an increase 40% from 2009. The MOH categorizes the TCK budget based on salaries, services, and miscellaneous costs. Within this budget, 11,000 USD is provided for Virtual Private Network and IT maintenance. The fixed costs and IT costs are displayed in Table 3.

Table 3.

Total Costs Are Accumulated

COSTS	INITIAL INVESTMENT	YEAR 1	YEAR 2	YEAR 3	YEAR 4	YEAR 5
Fixed costs
EAR grant	$542,500	$25,000	$25,000	$25,000	$25,000	$25,000
Operating costs		$129,000	$127,940	$62,805	$101,521	$143,534
Information technology cost		$11,000	$11,000	$11,000	$11,000	$11,000
Total costs	$542,500	$165,000	$163,940	$98,805	$137,521	$179,534
Grand total						$1,287,300

EAR, European Agency for Reconstruction.

As can be seen within the operating costs or MOH Annual Budget (Table 3), the limited funds allocated are not sufficient to provide TCK with the means to expand its telemedicine program to anything more than an educational platform.

Discussion

The program has not reached institutionalization where behavior or knowledge can be monitored; therefore, we must concentrate on the intermediate processes of the program. If the TCK's mission can be summarized as providing a platform to advance education, the TCK ultimately seeks to improve the quality of clinical care.

Similar to standard program evaluations, economic evaluations of telemedicine program are also uncommon. Davalos et al.⁶ state that there is a lack of concrete evidence that fully assesses the economic impact of telemedicine. Evaluation methods such as the BCA measures program efficiency based on the cost required to produce a unit of benefit.⁶ By assigning a monetary value to each program expense and benefit, a program where the benefits outweigh the costs is deemed positive. BCA is a particularly useful approach in telemedicine, where costs and outcomes can vary widely.⁶ In addition, there are no restrictions regarding indicators or values; therefore, this type of evaluation can be easily adjusted to cater to the program itself. IVeH's objective for conducting the CBA is to determine if the benefits of the intermediate processes or indicators outweigh the costs.

As discussed previously, the postconflict environment and state of the public health system allowed for the establishment of telemedicine in order to advance medical education and provide better quality healthcare.

Figure 3 illustrates the telemedicine program as the intervention, which leads to the proximal effect of general knowledge of the CME program (and other telemedicine resources), which then continues on to the midrange effect of attendance to the CME program and use of telemedicine resources such as teleconsultations. The distal effect of better clinical care is very difficult, if not impossible to assess, because telemedicine may not be the sole factors in changes in better clinical care. In addition, the data available (e.g., one lecture on Nutrition for Critically Ill Patients) is not statistically valid. As a result, the BCA will assess the midrange effect of attendance and use of telemedicine by monetizing these indicators and comparing the benefits to the costs of the program.

Fig. 3.

Intervention and effects: Telemedicine Program of Kosova (adapted from Christoffel and Gallagher⁵).

According to Kristin and Struyk,⁷ there are five core reasons for evaluating programs: feedback and improvement, accountability, funding, ownership, and policy impact. IVeH's objective for conducting the BCA is to essentially provide detailed information regarding the comparison of costs and benefits for the CME program delivered through TCK.

Conclusions

The BCA allows a detailed evaluation that provides vital statistics of program elements. This type of data allows the telemedicine program to be armed and dangerous with information and figures that can ultimately argue the case for future and continuous support of telemedicine programs.

Ultimately, BCA is a great financial assessment but it does not capture the depth and outreach of the TPK. The calculated Internal Rate of Return of 4.4% is conclusive allows the TCK to seek new opportunities for Return on Investment, better use of funds, and income generation opportunities. This figure is purely a dollar figure. Telemedicine is not the type of program that is implemented for purely financial purposes; this is understood by the high start-up costs and other obstacles usually associated with the start of any telemedicine program. BCA should be accompanied by other forms of evaluation, preceded by the implementation evaluation, and proceeded by the outcome evaluation. The 3-stage evaluation would offer a wide-range of information and results, allowing stakeholders to assess the telemedicine program accurately.

Recommendations

During the data collection period, IVeH had some issues with the data. It is difficult to retrospectively collect data, and IVeH had to assume many values and figures. This was also caused by standard protocols and documentation of the MOH of the Republic of Kosova, which was not as detailed as the BCA required.

Accurate monitoring and collection of data is very important. It is sometimes difficult for organizations to assume specific monitoring tools until the evaluation is underway and a particular set of data is needed. We suggest that different logs and other forms be as comprehensive as possible. The more information available, the more evaluation methods can be considered and implemented.

According to Kristin and Struyk's⁷ definition of outcome evaluation, this method assesses how well the project has achieved its intended goals and identifies factors affecting their achievement. Typically, outcome is measured by identifying the impact by contrasting the outcomes for an experimental group that receives a treatment and a control group that does not.⁷ If TCK was involved in more clinical applications or if the CME program ultimately targeted problem areas in healthcare delivery, then BCA would be a conclusive type of evaluation method. Our goal is to be able to conduct an outcome evaluation, and it is our observation that the behavior of physicians and nurses can be monitored if specific procedures and methods were presented/discussed in the CME program. For example, if the UCCK had a high rate of mastectomies, a series of lectures could be given on alternative methods for breast cancer treatment. Then, for a period of 5 years, the surgeons who attended the lecture can be monitored and the outcome analysis can be conducted based on the results.

Footnotes

Disclosure Statement

No competing financial interests exist.

*

This also includes that participants attended more than one lecture. On average, a participant attended at least 5 lectures during the entire CME series.

References

Latifi

, Merrell

, Doarn

, Bekteshi

, Lecaj

, Hadeed

, Boucha

, Hajdari

, Hoxha

, Koshsi

, de Leonni Stanonik

, Berisha

, Novoberdaliu

, Imeri

, Weinstein

. “Initiate-Build-Operate-Transfer” A strategy for establishing sustainable telemedicine programs in developing countries: Initial lessons from the Balkans. Telemed J E Health, 2009; 15:956–969.

Latifi

. International virtual e-hospital: The Balkans journey. Stud Health Technol Inf, 2008; 131:2–20.

Hersh

, Helfand

, Wallace

, Kraemer

, Patterson

, Shapiro

, Greenlick

. Clinical outcomes resulting from telemedicine interventions: A systematic review. BMC Med Inform Decis Mak, 2001; 1:5.

Bashshur

, Shannon

, Sapci

. Telemedicine evaluation. Telemed J E Health, 2005; 11:296–316.

Christoffel

, Gallagher

. Injury prevention and public health: Practical knowledge, skills and strategies. Gaithersburg: An Aspen Publication, 1999.

Davalos

et al. Economic evaluation of telemedicine: Review of the literature and research guidelines for benefit cost analysis. Telemed J E Health, 2009; 15:933–946.

Kristin

, Struyk

. Policy analysis for effective development. London: Lynne Rienner Publishing, 2004.