A Systematic Review of the Concepts of Efficacy,Effectiveness,and Efficiency as Applied and Measured for Virtual Health Care Delivery in Correctional Facilities

Abstract

Background:

The use of virtual care interventions in correctional facilities has increased in recent years owing to the impacts of the COVID-19 pandemic. However, the literature shows variability in the application and measurement of efficacy, effectiveness, and efficiency of virtual care interventions. This systematic review addresses this gap in evidence and provides an overview and appraisal of the methods and measures used to evaluate these aspects of virtual care interventions in correctional facilities, using a modified conceptual framework by the World Health Organization (WHO).

Methods:

We conducted a systematic review using a narrative synthesis approach. Comprehensive searches were performed in PubMed, Scopus, and Web of Science for peer-reviewed studies published in English between 2014 and 2024. The Joanna Briggs Institute Meta-Analysis of Statistical Assessment and Review Instrument was used to assess the methodological quality of included studies.

Results:

Twenty-one studies were included, and most were conducted in the United States and focused on synchronous modality for adult males. None of the studies explicitly defined the efficacy, effectiveness, and efficiency of virtual care interventions. The concept of effectiveness was the most frequently explored, and aligned best with WHO’s conceptual framework, whereas efficiency was the least explored. The most common evaluation measures were clinical effectiveness, user satisfaction, and interexaminer agreement.

Conclusions:

This review highlights the need for adopting a unified framework for evaluating virtual care in correctional facilities that can standardize evaluation metrics and improve resource allocation, ultimately enhancing patient outcomes by ensuring that virtual care interventions are efficacious, effective, and efficient.

Background

Globally, over 11.5 million people are incarcerated in correctional facilities,¹ with the highest prison population and rates of imprisonment in the Americas, Europe, and Oceania.^2,3 The number of people living in correctional facilities is increasing. For example, since 2000, Oceania has experienced an 82% increase in its overall prison population.⁴ The high number of people in custody and the increasing trend of incarceration are major concerns in the public health field.

People in custody bear a substantial disease burden compared with the general population. They experience higher rates of mental illness and substance use disorders, communicable and noncommunicable diseases, and chronic illnesses, such as major depression, hepatitis C, HIV infection, and cardiovascular disorders.^5

–10 These health disparities stem from socioeconomic and demographic disadvantages experienced throughout their life course. In particular, Indigenous, racial, and ethnic minorities are disproportionately impacted by incarceration.^9,10 Despite this increased burden of disease, people in custody have historically been marginalized with respect to health care, as health service delivery in prisons often does not meet the health needs of this group.⁹ Some correctional facilities transfer patients to local public hospitals for ongoing care. These transfers pose additional challenges and expenses, including the costs associated with patient transportation and heightened security risks. These risks affect not only the security personnel and hospital staff but also the broader community, particularly during the transfer of high-security prisoners.^11
–13 In an attempt to address the existing health inequities experienced by people in custody, a number of innovative modalities of health care delivery, including virtual care, have been implemented in correctional facilities.^14,15

Virtual care, telemedicine, digital health, and telehealth are terms that are often used interchangeably in the literature.^6,16
–18 According to the Toronto Women’s College Hospital for Health System Solutions and Virtual Care, virtual care is defined as “any interaction between patients and members of their circle of care that occurs remotely, using any form of communication or information technology, to facilitate or maximize the quality and effectiveness of patient care.”¹⁹ It is a broad term that encompasses diverse forms of health services that occur outside traditional face-to-face clinical encounters, including telephone calls, video visits, secure messaging, and email consultations. For this review, we used the term “virtual care” as an umbrella term to encompass all these modalities.^20,21

Virtual care is primarily delivered synchronously or asynchronously. Synchronous modalities involve real-time interaction through telephone, chat, or videoconferencing, whereas asynchronous communication is a store-and-forward method that collects text, audio, or video for later use by clinicians. The concept of virtual care not only covers both synchronous and asynchronous modes of patient–provider and provider-to-provider interactions but also the use of virtual reality to simulate care environments.^5,17,22

In everyday use, efficacy, effectiveness, and efficiency are often misunderstood as synonyms.^23,24 In experimental health research, however, the terms are more strictly defined: efficacy refers to the measure of how an intervention performs in ideal conditions, effectiveness is a measure of the performance of an intervention in real-world (nonresearch) settings, and efficiency is concerned with measuring the relationship between health service inputs, outputs, and outcomes to achieve the best value for money.^24
–26 In the context of virtual care settings, the World Health Organization (WHO) has provided guidance regarding the meaning, application, and evaluation of efficacy, effectiveness, and efficiency. It outlines a life-cycle approach and evaluation measures for assessing virtual care interventions. The WHO’s conceptual framework aligns the developmental cycle of the intervention with the evaluation stage and includes the assessment of the efficacy, effectiveness, and efficiency of virtual care interventions.²⁷ However, in correctional facilities, there is a variable application of the three concepts and the measures used to evaluate them. In particular, virtual care evaluations do not employ a life-cycle approach to evaluate efficacy, effectiveness, and efficiency.^6,28
–30

The use of virtual care interventions in correctional systems has increased in recent years owing to the impact of the COVID-19 pandemic.^31
–33 A study highlighted that in June 2020, the COVID-19 infection rate in correctional facilities was 5.5 times higher than in the general U.S. population.³² This heightened incidence was attributed to the challenges of effectively maintaining social distancing in overcrowded spaces, inadequate ventilation, and substandard hygiene practices. The limited health care resources within these centers, coupled with restrictions on transferring patients to external facilities, led to an increased adoption of virtual care solutions.^11,32,33 Given this increase in the use of virtual care in correctional facilities, it is vital that these interventions are evaluated to ensure their efficacy, effectiveness, and efficiency. To facilitate such evaluation, there needs to be a common understanding of the concepts of efficacy, effectiveness, and efficiency in relation to virtual health care delivery in correctional facilities, and the appropriate metrics to measure these in this context.^28,31 Hence, this systematic review aimed to appraise the literature systematically describing the evaluation of virtual health care delivery in correctional facilities to clarify the methods and measures used to elucidate the efficacy, effectiveness, and efficiency of such interventions.

Methods

A systematic narrative synthesis was conducted, and the findings were reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.³⁴ The protocol for this systematic review was registered on the International Prospective Register of Systematic Reviews (CRD42023454115).

LITERATURE SEARCH

We searched three electronic databases (PubMed, Scopus, and Web of Science) for peer-reviewed studies that evaluated virtual care interventions in correctional facilities. The databases were searched in May 2024. An additional literature search was performed using an internet search engine (Google Scholar, limited to the first 10 pages) to improve the overall coverage of the review. However, a gray literature search was not performed.

While a variety of terms are used in the literature, in this review, the major search concepts that defined the Subject Heading terms and keywords used for the search were virtual care, correctional facility, efficacy, effectiveness, and efficiency. The key search terms and Medical Subject Headings (MeSH) relevant to each concept were developed through an initial review of articles and in consultation with a health sciences librarian at the University of Newcastle. For the purposes of this review, the terms virtual care and telemedicine were used interchangeably to produce an inclusive list of studies. To ensure comprehensive coverage of all the literature, we combined key concepts using Boolean operators and applied relevant truncations (Table 1).

Table 1.

Search Strategy

SEARCH CONCEPTS	KEY SEARCH TERMS	MEDICAL SUBJECT HEADING (MESH)
Concept 1: Virtual care Telemedicine	Virtual care or telehealth or telemedicine or telemonitor* or telepractice or telenursing or telecare or ehealth or e-health or m-health or m-health or digital health or tele-referral or virtual medicine or virtual health or virtual consult* or remote consult* or videoconferenc* or video-conferenc* or video-monitor*	“Telemedicine/” or “remote consultation/” or “telenursing/” or “videoconferencing/” or “telepathology/” or “teleradiology/”
Concept 2: Correctional facility Prisoners	Prison* or jail* or incarcerat* or imprison* or correction* facilit* or prisoner* or inmate* or penitentiary	“Correctional Facility/” “Jail/” “Prisons/” “Prisoners/”
Concept 3: Efficacy, effectiveness, and efficiency	Effect* or efficacy or efficiency or outcome* or assess* or measure* or eval*	“Efficiency” “Outcome and process assessment, (health care)/” or “Outcome assessment (health care)/” “Quality assurance (health care)/methods*/”

STUDY SELECTION

Inclusion criteria

Publications were eligible for inclusion if they: (i) were published in peer-reviewed journals; (ii) described primary quantitative, qualitative, or mixed-methods research; (iii) reported on methods and measures used to evaluate the efficacy, effectiveness, or efficiency of virtual care in correctional facilities; and (iv) were published within the last 10 years (since 2014).

While no restrictions were placed on the region where the study was conducted, only articles published in English were eligible for inclusion. Publications describing synchronous or asynchronous (store-and-forward) virtual health modalities were eligible for inclusion. Publications that described studies examining virtual consultations among health professionals were also included. There were no restrictions on the age, sex, ethnicity, or health conditions of the patients included in the studies. Studies conducted in correctional facilities were included. In this review, the definition of “correctional facility” is used broadly to include a range of institutions designated for the confinement of individuals serving sentences, awaiting trial, or otherwise detained. This encompasses prisons, pretrial detention centers (such as jails and remand facilities), and juvenile correctional institutions.³⁵ We included publications that evaluated the efficacy, effectiveness, and efficiency of virtual care interventions in correctional facilities based on the terminology used by the authors (i.e., if the authors used these terms, the study was eligible for inclusion whether or not the authors’ understanding of these terms corresponded with the WHO definitions).

Feasibility and pilot studies were also included, as we interpreted these as evaluations of efficacy. Similarly, cost-effectiveness studies were included, as we interpreted them to be evaluations of efficiency. Studies that assessed the impact or quality of virtual care in correctional facilities were also included, as we interpreted these as evaluations of its effectiveness. Note that we restricted eligibility to studies published in the last 10 years due to the dynamic nature of the technological applications used in virtual health care delivery.

Exclusion criteria

We excluded literature describing virtual care interventions that were delivered to individuals or groups within the community (i.e., noncorrectional settings); people on probation, parole, or postrelease follow-up; individuals undergoing home detention monitoring; individuals and groups in immigration detention facilities; and prisoners of war. We excluded studies that were limited to a description of a virtual care intervention or model of care but did not provide a comprehensive evaluation of the intervention or clearly articulate the measures used to assess its efficacy, effectiveness, or efficiency. We also excluded studies that used simulations or focused primarily on future intentions, attitudes, and willingness to use virtual care interventions. Non-English literature, secondary research (systematic, scoping, and literature reviews), commentaries, abstracts, editorials, letters, books, posters, protocols, conference proceedings, and opinion pieces were also excluded.

Two reviewers (C.N. and A.L.) independently screened each title and abstract, and the full text of all publications deemed potentially relevant for inclusion after title/abstract screening. Consensus meetings were held after title and abstract screening and later after full-text reviews to finalize the selection of articles to be included in the review.

DATA EXTRACTION AND SYNTHESIS

The following data items were extracted from each article to a data charting form by one of the reviewers (C.N.): title; authors; year of publication; country where study was conducted; study population (age, sex, ethnicity); medical service/condition or discipline; study design; sample size; technological intervention description (synchronous/asynchronous); study aim/objective; duration of study; study timeline; evaluation method; measures of efficacy, effectiveness, and efficiency; other evaluation measures; outcomes; technology used; and additional notes.

ASSESSMENT OF STUDY QUALITY

The methodological quality of each included study was independently appraised by two reviewers using standardized critical appraisal instruments from the Joanna Briggs Institute Meta-Analysis of Statistical Assessment and Review Instrument (JBI-MAStARI).³⁶

Using the different evaluation checklists in the JBI as a guide, the studies were grouped into the following categories: cross-sectional (n = 11), pre–post (n = 2), qualitative (n = 2), cohort (n = 2), quasi-experimental (n = 1), and economic evaluations (n = 3). Each checklist criterion required a “yes,” “no,” “unclear,” or “not applicable” response. “Yes” meant the criterion had been established through the report description, “No” meant the criterion had not been applied appropriately, and “unclear” meant the criterion was not clearly identified in the report and it was not possible to acquire clarification from the author. The primary reviewer (C.N.) compared the two sets of appraisals. Discordant responses were first discussed by two reviewers. Note that we did not exclude any study based on its methodological quality.

CONCEPTUAL FRAMEWORK AND DATA SYNTHESIS

The included studies were heterogeneous with substantial variability in the measures used to evaluate the efficacy, effectiveness, and efficiency of virtual care interventions in correctional facilities. Therefore, conducting a meta-analysis was not feasible. Considering the diversity of the reported measures, we determined that a conceptual framework was necessary to synthesize the available evidence. Hence, we adapted a conceptual framework developed by the WHO to monitor and evaluate virtual care interventions creating a framework to categorize the concepts and measures described in the included studies.²⁷ The WHO conceptual framework was deemed most suitable for this systematic review, as it aligns the different stages of intervention maturity with specific evaluation stages, thereby providing a structured approach to assessing the efficacy, effectiveness, and efficiency of virtual care interventions (Table 2).²⁷

Table 2.

WHO Conceptual Framework Stages

WHO CONCEPTUAL FRAMEWORK
STAGES OF INTERVENTION MATURITY	EVALUATION STAGES
Preprototype	Feasibility/usability
Prototype	Feasibility/usability
Pilot	Efficacy
Demonstration	Effectiveness
Scale-up	Efficiency
Integration/sustainability	Implementation science

WHO, World Health Organization.

Within the WHO conceptual framework, efficacy corresponds to the pilot stage of a virtual care intervention (Table 2). Within this framework, “Efficacy” is defined as the assessment of whether a digital health intervention achieves the intended results in a controlled research setting.²⁷ The evaluation stage primarily addresses the question, “Can it work?” The five measurement targets for this stage are user satisfaction, workflow fit, learning curve, performance errors, and reliability.²⁷

Within the WHO framework, the evaluation of the effectiveness of virtual care aligns with the demonstration stage of digital health maturity (Table 2). “Effectiveness” is defined as the assessment of whether the digital health intervention achieves the intended results in an uncontrolled, nonresearch, or clinical setting.²⁷ The key question at this stage is, “Does it work?”²⁷ The two main measurement targets for this stage are changes in process (e.g., time to x), and changes in outcome (e.g., system performance).²⁷ For this review, changes in process are associated with service delivery improvements in terms of coverage and scale (availability), as well as quality and utilization of services at the client, provider, and system levels.²⁷ Changes in outcome, on the other hand, relate to improvements in system performance or health outcomes attributable to the intervention.

For this review, the scale-up stage from the WHO conceptual framework was adapted to evaluate efficiency, focusing on the cost measurement targets. “Efficiency” is defined as the measurement of whether health care resources are being used to obtain the best value for money. It addresses the question,” Has the digital health intervention reduced costs associated with the delivery of health services?” The measurement targets for this stage include changes in process and outcome, evaluation of costs (cost reduction and implementation costs), error rates, and the learning curve of users.²⁷

By employing this as “a priori” framework for our data synthesis, we aimed to assess whether the evaluations described in the included studies aligned with the criteria outlined in the adapted WHO conceptual framework. This structured analysis would determine whether the array of measures discussed in the studies is adequate to successfully evaluate virtual care programs and interventions.

For each study, we assessed whether the evaluation strategy adhered to the relevant measurement targets associated with the evaluation stage, as described in the WHO conceptual framework. For each target measure, we assigned one of the three outcomes: “yes,” “no,” or “unclear.” “Yes” indicated that the criterion had been observed in the study description, while “no” meant the criterion had not been observed in the study description. “Unclear” indicated that the criterion was not clearly identified in the study description. However, four studies^37

–40 that explored more than one concept were not scored against the WHO framework. These studies lacked clarity regarding their evaluation stage, making it difficult to determine whether they were measuring efficacy, effectiveness, or efficiency.

Results

The literature search identified 1,063 articles from electronic databases, of which 382 were duplicates. After excluding duplicates, the titles and abstracts of the remaining 681 articles were screened by two independent reviewers (C.N. and A.L.) in Covidence, and 562 studies were found to be irrelevant. Of the remaining 119 studies, the full text of one study could not be located. After screening 118 full-text studies, 97 were excluded as they did not meet the inclusion criteria for this review. Overall, 21 studies that met the inclusion criteria were included. The study selection process is summarized in the PRISMA flow diagram presented in (Fig. 1).

Fig. 1.

PRISMA flow diagram. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

STUDY CHARACTERISTICS

The 21 publications included in this review were published between July 2014 and November 2023 and described 10 different types of medical services implemented across nine countries. Most studies (n = 8) were conducted in the United States.^{39,41

–47} Three studies were from Australia,^48
–50 three from Spain,^38,51,52 and two from France.^53,54 One study each from Colombia,³⁷ China,⁵⁵ Brazil,⁵⁶ South Korea,⁵⁷ and Italy⁴⁰ was included. All publications retrieved for this review described observational primary research studies, of which 19^{37

–47,50

–57} comprised quantitative and two^48,49 comprised qualitative study designs. The most commonly identified modality was synchronous virtual care (n = 14),^{38,39,41
–43,45

–51,55,57} followed by asynchronous virtual care (n = 5)^{40,52
–54,56}; two of the studies were hybrid in nature.^37,44 The number of patients in the included studies ranged from 20 to 687, and participants were aged between 15 and 76 years old. Only one study had a female majority,⁴⁸ whereas the rest focused primarily on males. Similarly, only one study focused on young people aged 15–19 in custody.⁵⁶ The 10 different medical services described in the 21 studies were all related to the delivery of specialist consultations and are presented in (Table 3).

Table 3.

Medical Services Evaluated in the Included Studies (N = 21)

MEDICAL SERVICES	NUMBER OF STUDIES
Communicable diseases
Hepatology	4 (19%)
HIV	1 (4.8%)
Dental	3 (14.3%)
Dermatology	3 (14.3%)
Specialist Medical Consult	2 (9.5%)
Mental health	2 (9.5%)
Telepsychiatry	2 (9.5%)
Urological care	2 (9.5%)
Otolaryngology	1 (4.8%)
Diabetes management	1 (4.8%)
Total	21

Virtual care services described in the included studies primarily targeted patient care (n = 17).^{37
–39,41

–48,50
–52,54,55,57} However, two services were aimed at mentoring and advising primary care providers,^40,49 while two were aimed at providing a second diagnostic and treatment perspective in addition to face-to-face care.^53,56

Thirteen studies reported the technology and peripherals used for delivering virtual care services,^{38,42,44
–46,48,50,51,53

–57}which included videoconferencing systems and software for virtual consultations with secure video communication and administrative systems. The peripherals included cameras (intraoral, retinal, general examination, digital, high-resolution close-up cameras), high-definition monitors, personal computers (PCs), tablet PCs with lockdown software, electronic stethoscopes, digital pulse oximetry, electrocardiogram (EKG), and light-emitting diode (LED) illumination systems.

A summary of the characteristics of the included studies is shown in Table 4. The measures included in this table reflect the terminology used by the authors (whether or not it corresponds with the WHO framework).

Table 4.

Summary of the Characteristics of Studies Included in the Systematic Review (N = 21)

AUTHOR AND COUNTRY	MEDICAL SPECIALTY AND MODALITY	STUDY DESIGN	DURATION IN MONTHS	STUDY POPULATION (AGE IN YEARS)	AUTHORS’ CHARACTERIZATION OF WHAT WAS BEING EVALUATED AND MEASURE(S)	FINDINGS
Barrera-Valencia³⁷ Colombia	Telepsychiatry (Depression) Hybrid: synchronous and asynchronous	Economic evaluation Prospective data collection	NR	Age: above 18 Sex: all males N = 157n = 106	Efficiency measure Cost-effectiveness: Cost comparison between synchronous and asynchronous telepsychiatry services measured by the variability in care times for each modality. Effectiveness measure Clinical effectiveness: Comparative difference of the progression of the depression symptoms from baseline to final Hamilton Depression Rating Score (HDRS) between synchronous and asynchronous modalities.	Cost-effectiveness: Consultation costs were found to be much higher in the synchronous model. Clinical effectiveness: Results of the asynchronous model were higher than the synchronous model (p < 0.001).
Cheng⁵⁵ China	Telepsychiatry (stable out-patients) Synchronous via videoconferencing	Quasi-experimental pilot study Prospective data collection	24	Mean age: 40 Age range: 21–64Sex: all malesIG n = 86CG n = 249	Effectiveness measure Pre–post difference: C-GHQ-12 score (self-administered Chinese version, 12-item General Health Questionnaire) filled in seven days before and seven days after consultation for both groups. Patient satisfaction: psychiatric teleconsultation measured using 9-item questionnaire with 5-point Likert scale. Completed by intervention group alone.	Pre–post difference: in C-GHQ-12 score was significantly higher for the intervention group compared to the control participants (p = 0.023). Patient satisfaction score: The intervention group showed high levels of satisfaction with a mean satisfaction score above the 80th percentile.
Cuadrado³⁸ Spain	Hepatology Hepatitis C virus (HCV) Synchronous via videoconferencing	Economic evaluation Retrospective document review	18	NR IG TCP n = 75 UCP Modeled control group	Effectiveness measure Clinical effectiveness: measured through Sustained Virological Response Rates (SVR) after first treatment and after relapse. Efficiency measure Perspective of the NHS and the social perspective Cost-minimization analysis: estimating differences in average cost per patient of implementing TCP and not implementing it using a 52-week decision-tree cost model User satisfaction/acceptance: With telemedicine was evaluated using an 11-question questionnaire with 5-point Likert scale	Clinical effectiveness: SVR of 94.7% after first treatment and 100% after 4 relapses. Cost-minimization analysis: Average cost per patient with the TCP strategy was lower. TCP provided a per patient cost saving of 30.6%. Patient satisfaction with TCP was reported to be high.
Gallagher⁴¹ US	Otolaryngology Synchronous telephone visits	Cross-sectional pilot studyProspective data collection	7	Age: above 18Sex: 95% maleEthnicity: 65% Hispanic: 25% African American: 5% White: 5% mixed/other Total N = 188 patients n = 20 patients n = 12 physicians	Efficacy measure (acceptability) Patient and physician satisfaction: Evaluated using the 14-question patient Telehealth Satisfaction Questionnaire (TSQ) with a 5-point Likert scale and a modified physician satisfaction questionnaire with 5-point Likert scale. Feasibility: 1. Availability of image or audiogram prior to consultation or ordered after consultation. 2. Patients requiring follow-up in person 3. Patients requiring follow-up via telephone	Patient and physician satisfaction: With telehealth was high at 4.25 ± 0.12 and 4.65 ± 0.13 out of a score range of 1–5. Feasibility: 1. Imaging or audiogram was available prior to appointment in 60% of cases. 2. In 40% of cases image or audiogram ordered after appointment. 3. 45% of patients required follow-up in-person 4. 15% were followed up with another phone visit.
Giraudeau⁵³ France	TeledentistryAsynchronous (oral images)	Cross-sectional feasibility studyProspective data collection	1	Average age: 33.0 Sex: NR IG n = 30IS n = 1	Efficacy measure (feasibility) Diagnostic concordance: Compared dental emergency score given by in-person nurse and tele-dentist	Diagnostic concordance: Low concordance between the in-person nurse score and tele-dentist (unweighted kappa 0.24 and weighted kappa 0.53).
Halder⁵⁰ Australia	Hepatology (HCV) Synchronous via videocall	Pre–post Retrospective and prospective review of medical records	30	Mean age: 36 Age range: 20–66 Sex: 80% Male IG n = 332 CG n = 10IS n = 4	Effectiveness measure Clinical effectiveness: Sustained Virological Response rates at 12 weeks (SVR12) between telehealth-based model of care and face-to-face consultations. A retrospective comparison of per annum treatment numbers calculated using the average number of patients treated per annum in-person from 2016 to 2018 compared with the average treatment number per annum using the video call-based model of care.	Clinical effectiveness: SVR12 was high at 91%. Increased per annum treatment numbers from an average of 10–137 following introduction of teleconsultation videocalls.
Inquimbert⁵⁴ France	TeledentistryAsynchronous	Cross-sectional studyProspective data collection	12	Average age: 31.65 ± 10.69 Age range: 16–76 Sex: all males N = 1,051IG n = 567	Effectiveness measure Average evaluation time: Time taken to evaluate a patient via teledentistry, including recording and analysis time. Image accuracy: Proportion of oral teledentistry images analyzed for the Decayed, Missing, and Filled Teeth (DMFT) score. Triaging: identification of oral conditions requiring urgent care using Emergency Score. Eventual dental appointments: Proportion of patients who received a teledentistry oral exam that subsequently had an in-person dental appointment.	Average evaluation time: For recording data and analysis was 22 min. Image accuracy: 98.9% had sufficient accuracy for further DMFT analysis. Triaging: Over 40% with a high dental urgency score, which required immediate care. Eventual dental appointments: About 12–31% of patients subsequently had an in-person dental appointment.
Jiménez-Galán⁵¹ Spain	Hepatology (HCV) Synchronous via videoconferencing	Cross-sectional studyProspective data collection	∼48	Mean age (±SD): 46.7(±8.16) Sex: all males N = 1,200IG n = 163 IS n = 1 IG completing survey n = 99	Effectiveness measure Clinical effectiveness: Via Sustained Virological Response (SVR) rate: User satisfaction: Surveys delivered to patients one year after starting telemedicine (on technical quality of image, sound, comfortability, perceived professionalism, and overall assessment) using a 5-point Likert scale.	Clinical effectiveness: Overall SVR rate was 96.9% with 0 HCV prevalence at end of program. User satisfaction: The overall assessment was good or very good 67%, average 17%, poor 10%, and very poor for 4% of responses.
Kassar⁴⁴ US	Diabetes Management Hybrid: synchronous tele-visits and asynchronous	Pre–post study Retrospective chart reviews	33	Mean age: 43.7 Age range: 20–75 Sex: all malesIG n = 106 IS n = 15	Effectiveness measure Clinical effectiveness: Improvements in clinical parameters 1. Blood glucose (glycemic control) measurements 2. Blood pressure measurements3. Lipid control/cholesterol–low-density lipoprotein (LDL)	Clinical effectiveness: Improvements in glycemic, blood pressure, and lipid control observed. 1. Improvements in glycemic control (HbA1c) in >50% of patients with at least one follow-up visit. 2. Majority of patients received treatment for hypertension and hyperlipidemia (blood pressure and lipid control)
Khairat⁴⁵ US	Specialist Medical consult Synchronous via videoconferencing	Cross-sectional studyProspective data collection	6	Age: 52% 51 and above, 33.3% 35–50, 14.7% 18–34Sex: 94.6% male patients, 73% male practitioners IS n = 55IG n = 482 patients IG n = 316 practitioners IG n = 786 tele-presenters	Efficacy measures (feasibility) User satisfaction and future use of Telemedicine (TM): For patients, tele-presenters, and practitioners using a modified TSQ with two open-ended questions about the experience and preferred health delivery model. Cost-savings: Measured by transportation and implementation costs.	User satisfaction and future use of TM: Patients and practitioners who rated their TM experience as excellent indicated they would utilize TM again. Cost-savings: Estimated net financial effect after one year (savings minus implementation costs) was $683,877.
Lewis⁴² US	Mental health (Competency Restorative Services) Synchronous via videoconferencing	Pre–post study with matched historical comparisonProspective and retrospective data collection	∼60	Mean age: 38.1–39.6Sex: all maleRace: 83–90% African American, 8–17% White, 0–2% other IG n = 42CG n = 398	Effectiveness measure Time to restoration: Duration of time from admission until the patients was deemed competent to stand trial by an evaluator. Restoration rate: Proportion of patients deemed competent to stand trial by an evaluator.	Time to restoration: Length of stay between in-person and telehealth remained the same. Restoration rate: Increased for patients deemed competent to stand trial for telehealth compared to prepandemic level.
Morosini⁵⁶ Brazil	TeledentistryAsynchronous (file sharing service)	Cross-sectional censusProspective data collection	5	Average age: 16.8 Age range: 15–19 (SD = 0.941)Sex: all maleIG n = 102	Effectiveness measure Telediagnosis sensitivity: Measured through interexaminer agreement and accuracy of the DMFT score calculated between each distant consultant and the on-site examiner.	Telediagnosis sensitivity: Degree of agreement between the clinical examiner and consultants was also ‘‘almost perfect (kappa of 0.78 to 0.86).’’ Specificity ranged from 97% to 98%, sensitivity was 48–73%.
Neuhaus⁴⁹ Australia	Hepatology (HCV) telementoring Synchronous videoconferencing	Pre–post studyProspective data collection	10	Average age: 35.7 Sex: 19% female IG patients n = 127 IG staff n = 16IS n = 5	Effectiveness measure (qualitative) Perceived clinical effectiveness: Measured by reported increase in service uptake. Prescriber eligibility status: measured by reported increase in number of accredited prescribers.	Telementoring can increase the capacity to treat HCV in correctional facilities.
Perdacher⁴⁸ Australia	Well-being and Mental Health Application for Indigenous AustraliansSynchronous teleconsultations	Pre–post feasibility study Prospective data collection	21	Age: NRSex: 70% female Ethnicity: 100% Indigenous Australians Practitioners: 60% female IG clients n = 27IG practitioners n = 10IS n = 3	Efficacy measure (feasibility) Pre- and postintervention differences: In mental health and well-being measured using the Kessler Psychological Distress Scale, Warwick–Edinburgh Mental Well-being Scale, and the Growth and Empowerment Measure, at 3-month intervals. Technology acceptance: Clients and practitioners asked to evaluate the SSA and identify benefits and areas of improvement using semi-structured interviews that focused on their experiences with the app, app effectiveness, service context, and mode of delivery.	Stay Strong App was found to be a feasible and culturally safe mental health and well-being tool for Aboriginal and Torres Strait Islander people within high-security prisons.
Pitarch-Bort⁵² Spain	Teledermatology Asynchronous	Cross-sectional study Retrospective review of medical records	18	Mean age: 42.4Sex: all male IG patients n = 37IS n = 1	Effectiveness measure Establish a diagnostic suspicion via teledermatology: Proportion of cases where a diagnostic suspicion was established via teleconsultation. Availability of a complete treatment plan: Proportion of patients with a complete treatment plan comparing teleconsultations and face-to-face consultations (FFC). Patients not requiring further FFC following teleconsultation: Proportion of patients that were reviewed via teleconsultations no longer requiring FFC. Delays in responding to first teleconsultation: average time in days to respond to referrals.	Establish diagnostics suspicion via teledermatology: A diagnosis was established in 86% of teleconsultations. Availability of complete treatment plan: 86% of teleconsultations had a complete treatment plan. Patients not requiring further FFC following teleconsultation: Only 18.6% of the consultations requiring a face-to-face consultation. Delays in responding to first teleconsultation: time to respond to referrals was <3 days in 73% of the first consultations.
Rappaport⁴⁶ USA	Specialist medical care (for 10 nonurgent specialties)Synchronous via videoconferencing	Economic evaluationRetrospective and prospective review of records	30	NR IS n = 24 n = 3,356 TM visits	Efficiency measure Return on Investment (ROI): Measured as time to match the initial investment (in months). Calculated by estimating the cumulative cost-savings from avoided costs (multiplying transportation expenses by the number of telehealth encounters). Cost-savings: based completely upon costs avoided from patient transfers to FFC.	ROI: 1.2 million dollars in savings, equaling the initial capital investment, were achieved at 32 months instead of the initial estimated 48 months. TM encounters increased during the first 32 months of operation. Cost-savings: 3,356 patient transfers estimated at $368.46 avoided.
Seol⁵⁷ South Korea	Teledermatology(LITD)Synchronous teleconsultations	Cross-sectional study Retrospective review of medical records	38 months	Mean age: 39.0Age range: 17–76 Sex: 91.4% maleIG n = 406	Effectiveness measure Clinical effectiveness: Demonstrated improvement in clinical parameters in follow-up patients.	Clinical effectiveness: 86.7% showed clinical improvement, whereas 5.2% showed either no change or a worse clinical outcome.
Sherwood³⁹ USA	Urological care Synchronous via videoconferencing	Cross-sectional Study Retrospective review of medical records telemedicine visits	90	Mean age: 42.3 (SD ± 13.2) Sex: all male n = 376 unique visits n = 154 repeat visits	Effectiveness measure Diagnostic concordance of eventual in-person visit: analyzing diagnostic concordance of all patients eventually seen in-person. Efficiency Analyzing the number of in-person visits saved by using telemedicine: by looking at patients with 1) Order completion via TM prior to in-person visit2) Number of in-person visits required after initial TM visitAdditional TM visits conducted prior to or in place of follow-up in-person visits	Diagnostic concordance of 90% for 56% of patients eventually seen in-person. Analysis of number of in-person visits saved by using ^TM: 1) Order compliance achieved in 91% radiological orders and 89% pharmacological orders via TM. 2) 80% required one less in-person visit if all the orders had been completed (and 52% of population could have been managed by telemedicine alone). 3) Over 94% could have had one in-person visit saved.
Sherwood⁴³ USA	Urological care Synchronous via videoconferencing	Cross-sectional study Retrospective review of medical records	90	Mean age: 37.0 (SD ±11.1) Sex: all male n = 110	Effectiveness measure Evaluation using telemedicine (TM) alone: Number of patients who were evaluated using telemedicine alone. Eventual in-person visits following telemedicine consultation. Diagnostic concordance: Measured by comparing telemedicine and eventual in-person visit diagnosis. (in-person visits reduced by telemedicine). Evaluation using TM alone: Proportion of patients who could be completely evaluated using TM alone, without needing to be seen in-person. In-person visits avoided: patients with at least one in-person visit avoided due to telemedicine consultation. Radiological orders completed before in-person visit: Patients with radiological orders completed before in-person visit	Evaluation using TM alone: 28% of patients had complete evaluation and successful management using TM alone. Eventual in-person visits: 49% of patients seen via TM eventually attended in-person visits. Diagnostic concordance: Between TM and eventual in-person diagnosis was confirmed in 98% of consultations. Evaluation using TM alone: 85% of patients could have been completely evaluated with TM alone, without ever needing to be seen in person. In-person visits avoided: 98% of TM patients avoided at least one in-person visit. Radiological orders completed before in-person visit: 94% radiological tests (and 85% medication prescriptions) completed before in-person visits.
Young⁴⁷ USA	HIV care Synchronous via videoconferencing	Cohort study Retrospective and prospective review of medical records	12	Age: above 18Sex: NRIG n = 687 CG n = 514	Efficacy measure Virological suppression: Comparison of complete virological suppression between subspecialty management via TM and on-site physicians without subspecialty training. Community HIV Viral Load (CVL)and in-care viral load: are measures of total virologic burden of a community.	Virological suppression: The proportion of patients with complete virologic suppression during the first six visits was significantly greater in the TM group. CVL and in-care viral load: TM cohort had a lower mean CVL during the first 6 visits and a lower mean in-care HIV load at the final visit.
Zarca⁴⁰ Italy	Dermatology tele-expertise Asynchronous	Cohort study Retrospective review of medical records	12	Mean age: 34.2 Sex: 90% maleIG patients n = 450 CG patients n = 54 IS n = 10 IG physicians n = 10	Efficacy measure Median delay for obtaining a response: The delay from successfully uploaded of a request to the response among all replied requests. Picture quality: Proportion of tele-expertise lesion pictures assessed by the dermatologists as very satisfactory, satisfactory, or unsatisfactory. Recommendation for in-person consultation or hospitalization: Proportion of patients requiring in-person consultation or hospitalization for further investigations following tele-expertise. Effectiveness measure Patients with a complete treatment plan for skin lesions: Proportion of patients with a completed treatment plan for skin lesions compared to the proportion of patients without a treatment plan before implementation of tele-expertise. Provider satisfaction: using a survey adapted from MAST with 6-point Likert scale Using Model for Assessment of Telemedicine (MAST) assessing the value of telemedicine, technical problems experienced, image quality, investment, and operation costs, and provider satisfaction. Differences in mean cost per completed treatment plan: (Depending on number of patients) of implementing tele-expertise not implementing it.	Median delay for obtaining a response: Five days was the median delay for obtaining a response. Picture quality: 48% of lesion images assessed were very satisfactory and 40% were satisfactory. Recommendation for in-person consultation or hospitalization: 2.9 % of patients were recommended for further in-person consultations and hospitalization. Patients with complete treatment plan: For skin lesions available in 82% of patients compared to 35% prior to implementation of tele-expertise. Provider satisfaction: Tele-expertise was well accepted among physicians with seven out of nine willing to continue using it. Differences in mean cost per completed treatment plan: The mean cost for one completed treatment plan was 184 by tele-expertise and 315 without tele-expertise.

CG, control group/comparison group; C-GHQ-12, Chinese version of the 12-item General Health Questionnaire; CVL, community HIV viral load; DAAs, direct acting antivirals; DM, diabetes mellitus; DMFT, Decayed, Missing, and Filled Teeth score; FFC, face-to-face consultations; GP, general practitioner; HCV, hepatitis C virus; HDRS, Hamiliton Depression Rating Score; HIV, human immunodeficiency virus; IG, intervention group; IS, intervention site(s); LDL, low-density lipoprotein; LITD, live interactive teledermatology; MAST, model for assessment of telemedicine; NHS, National Health Systems; OPD, Out-Patient Department; ROI, return on investment; SD, standard deviation; SSA, Stay Strong App; SVR, sustained virological response; SVR12, sustained virological response week-12; TCP, telemedicine clinical practice; TPP, testicular pain and pathology; TSQ, Telehealth Satisfaction Questionnaire; TM, telemedicine; UCP, usual clinical practice.

EFFICACY, EFFECTIVENESS, AND EFFICIENCY OF VIRTUAL CARE INTERVENTIONS

Defining efficacy, effectiveness, and efficiency of virtual care interventions

Analysis of the 21 studies included in the review revealed a lack of consensus regarding the definitions of efficacy, effectiveness, and efficiency. Notably, none of the 21 studies provided explicit definitions of these terms within the context of their evaluation. Three studies^40,47,56 employed these terms interchangeably in the text without differentiation. In 11 studies, the concepts of effectiveness^{37,39,42,43,48

–51,57} and efficiency^38,46 were used in relation to the specific outcome measure used in the study. Another seven studies did not define the terms or provide clear explanations of their use in the methodology, but instead referred to the terms efficacy, effectiveness, and efficiency in the abstract of the study⁵² or in the discussion,^44,53
–55 or conclusion.^45,56

What included studies purported to evaluate and the methods and measures employed

Of the 21 included studies, the majority (n = 10, 47.6%) were described by their authors as evaluating the effectiveness of virtual care interventions,^{42
–44,49

–52,54,55,57} while six (28.6%) were stated to be evaluating the efficacy,^{41,45,47,48,53,56} and one (4.8%) reported on efficiency.⁴⁶ The remaining four (19.0%) studies were described by their authors as evaluations of a combination of efficacy and effectiveness,⁴⁰ as well as efficiency and effectiveness (Table 5).^37
–39

Table 5.

Concepts Observed in the Literature as per Stated by Authors (N = 21)

THEMES	STUDIES (REFERENCES)	FREQUENCY, n (%)
Efficacy	4, 5, 10, 12, 14, 20	6 (28.6)
Effectiveness	2, 6–9, 11, 13, 15, 17, 19	11 (47.6)
Efficiency	16	1 (4.8)
Efficacy and effectiveness	21	1 (4.8)
Efficiency and effectiveness	1, 3, 18	3 (14.3)

The measures used to evaluate virtual health interventions described in the included studies are presented in Table 6. The most frequently explored concept in the studies analyzed in this review was effectiveness, and the least was efficiency. Clinical effectiveness was the most common measure, followed by user satisfaction, and interexaminer agreement.

Table 6.

A Summary of the Measures Reported by the 21 Studies^a in the Order of Most Used to Least Used

MEASURE OF EFFICACY	N ^a	MEASURE OF EFFECTIVENESS	N ^a	MEASURE OF EFFICIENCY	N ^a
User satisfaction^41,45,48	3	Clinical effectiveness^{37,38,50,51,44,49,57}	7	Cost-minimization analysis³⁸	1
		User satisfaction^55,38,51,40	4	Return on investment⁴⁶	1
				Costs savings⁴⁶	1
		Evaluation using telemedicine alone^52,39	2	Cost-effectiveness³⁷	1
Interexaminer agreement^53,56	2	Interexaminer agreement^39,43	2
Picture quality⁴⁰	1	Diagnostic quality of images^54,52	2
Pre–post intervention difference⁴⁸	1	Pre–post intervention differences^55,50	2
		Test order compliance^39,43	2
		Complete treatment plan^52,40	2
Cost-savings analysis⁴⁵	1
Access to care⁴⁵	1	Time to restoration⁴²	1
Improvements in clinical parameters⁴⁷	1	Restoration rate⁴²	1
		Triage⁵⁴	1
In-person visits recommended⁴⁰	1	In-person visits saved⁴³	1	In-person visits saved³⁹	1
		Average evaluation time⁵⁴	1
		Prescriber eligibility⁴⁹	1
Time taken to respond to referrals⁴⁰	1	Time taken to respond to referrals⁵²	1
		Mean cost per completed treatment plan⁴⁰	1

Multiple studies used more than one measure and therefore the total count of measures used is more than 21.

The details of the measures used to evaluate the efficacy, effectiveness, and efficiency of interventions are presented in (Table 7) efficacy measures (Supplementary Table S1), effectiveness measures, and (Supplementary Table S2) efficiency measures. Clinical effectiveness, user satisfaction, and interexaminer agreement are the most commonly explored measures. No common measures have been explored under efficiency.

Table 7.

Description of the Measures Used to Evaluate the Efficacy of a Virtual Care Intervention

MEASURE	MEDICAL SPECIALITY	AUTHOR AND YEAR	DESCRIPTION
User satisfaction	Otolaryngology	Gallagher et al., 2023⁴¹	Assessment of patient satisfaction (teleconsultation) using the 14-question patient Telehealth Satisfaction Questionnaire (TSQ) with 5-point Likert scale. Provider satisfaction assessed using a modified physician satisfaction questionnaire with a 5-point Likert scale from (1) not at all effective to (5) extremely effective
	Specialist medical consult	Khairat et al., 2023⁴⁵	Assessment of patient and provider satisfaction using a modified Telemedicine Satisfaction Survey (TSQ) with two open-ended questions about their experience (1) what aspects of telemedicine did you like? (2) what aspects of telemedicine did you not like? And preferred health delivery model
	Mental health	Perdacher et al., 2022⁴⁸	Assessment of patient and provider technology acceptance using qualitative semi-structured feedback interview method with questions on their experience with the application, app effectiveness, service context, and mode of delivery, including the benefits and areas of improvement
Interexaminer agreement	Teledentistry	Giraudeau et al., 2017⁵³	Comparative difference between dental emergency scores assessed by in-person nurse and remote tele-dentist using intraoral videos.
Interexaminer agreement	Teledentistry	Morosini et al., 2014⁵⁶	Comparative difference between Decayed, Missing, and Filled Teeth (DMFT) score assessed by an on-site examiner (face-to-face) and two remote consultants using intraoral photographs.
Picture quality	Dermatology	Zarca et al., 2018⁴⁰	Quality of lesion images assessed as very satisfactory, satisfactory, or unsatisfactory by remote expert.
Access to care	Specialist medical consult	Khairat et al., 2023⁴⁵	Measured using health service utilization rates
Cost-savings analysis	Specialist medical consult	Khairat et al., 2023⁴⁵	An estimation of the cost and time needed to conduct the telemedicine consultation in person.
Pre–post intervention differences	Mental health	Perdacher et al., 2022⁴⁸	Comparative differences between preintervention Kessler Psychological distress scale, Warwick–Edinburgh Mental Well-being Scale, and the Growth and Empowerment Measure at intake, first follow-up, and second follow-up at 3-month intervals
Improvement in clinical parameters	HIV Care	Young et al., 2014⁴⁷	Comparison of complete HIV virological suppression rates between on-site physician without HIV subspeciality training and remote telemedicine team with HIV subspeciality management training
In-person visits recommended	Dermatology	Zarca et al., 2018⁴⁰	Patients who required an in-person consultation or hospitalization (further investigations) following tele-expertise consultation.
Time taken to respond	Dermatology	Zarca et al., 2018⁴⁰	Time taken, in days, by the remote expert to respond to a successfully uploaded request.

Assessment of “efficacy” as per WHO conceptual framework

Of the six studies that purported to evaluate efficacy, only one⁴¹ study explicitly categorized the stage of maturity as the pilot stage. Furthermore, only one study⁵³ satisfied the criteria described in the WHO conceptual framework (Table 8). The remaining five could not be classified as efficacy studies according to the WHO criteria.

Table 8.

Comparison of Included Studies’ Assessment of Efficacy to WHO Conceptual Framework

AUTHOR AND YEAR		STAGE OF MATURITY: PILOT
		STAGE OF EVALUATION: EFFICACY
		MEASUREMENT TARGETS
	SCORE OUT OF 5	USER SATISFACTION	WORKFLOW FIT	LEARNING CURVE	PERFORMANCE ERRORS	RELIABILITY
Giraudeau et al., 2017⁵³	5	Yes	Yes	Yes	Yes	Yes
Khairat et al., 2023⁴⁵	4	Yes	Yes	Yes	Yes	No
Gallagher et al., 2023⁴¹	2	Yes	Unclear	No	Yes	No
Morosini et al., 2014⁵⁶	2	No	No	No	Yes	Yes
Young et al., 2014⁴⁷	0	No	No	No	No	No
		3	2	2	4	2

N, no; UC, unclear; Y, yes.

Four of these documented performance errors,^41,45,53,56 three reported on user satisfaction,^41,45,53 and less than half reported on workflow fit,^45,53 reliability,^53,56 and learning curve.^45,53 A summary of the analyses is presented in Table 8.

Assessment of “effectiveness” as per WHO conceptual framework

Of the 11 studies^{37,42
–44,49

–52,54,55,57} that purported to investigate the effectiveness of virtual care interventions, all documented changes in outcome while six^{37,42,49,50,52,54} documented both changes in process and changes in outcome as specified in the WHO framework (Table 9). Notably, one study that explored effectiveness categorized the intervention’s developmental stage as the pilot stage.

Table 9.

Comparison of Included Studies’ Assessment of Effectiveness to WHO Conceptual Framework

AUTHOR AND YEAR		STAGE OF MATURITY: DEMONSTRATION
		STAGE OF EVALUATION: EFFECTIVENESS
		MEASUREMENT TARGETS
	SCORE OUT OF 2	CHANGES IN PROCESS (TIME TO X)	CHANGES IN OUTCOME (SYSTEM PERFORMANCE/HEALTH)
Barrera-Valencia et al., 2017³⁷	2	Yes	Yes
Lewis et al., 2023⁴²	2	Yes	Yes
Neuhaus et al., 2018⁴⁹	2	Yes	Yes
Pitarch-Bort et al., 2023⁵²	2	Yes	Yes
Halder et al., 2021⁵⁰	2	Yes	Yes
Inquimbert et al., 2021⁵⁴	2	Yes	Yes
Jimenez-Galan et al., 2019⁵¹	1	Unclear	Yes
Kassar et al., 2017⁴⁴	1	Unclear	Yes
Cheng et al., 2018⁵⁵	1	Unclear	Yes
Sherwood, Nepple & Erickson., 2018⁴³	1	Unclear	Yes
Seol et al., 2018⁵⁷	1	No	Yes
		6	11

Assessment of “efficiency” under the WHO conceptual framework

The evaluation of virtual care efficiency was the least reported concept in the studies included in this review. Only one study purported to assess the efficiency of a virtual care intervention,⁴⁶ which did not satisfy the WHO criteria, as it only assessed cost reduction and changes in process/outcome in less controlled environment components, while failing to assess the total cost of implementation, error rates, and the learning curve of users (Table 10).

Table 10.

Comparison of Included Study’s Assessment of Efficiency to WHO Conceptual Framework

STUDY AND YEAR		STAGE OF MATURITY: SCALE-UP
		STAGE OF EVALUATION: EFFICIENCY
		MEASUREMENT TARGETS
	SCORE OUT OF 5	CHANGES IN PROCESS/OUTCOME IN LESS CONTROLLED ENVIRONMENT	REDUCTION IN COST	TOTAL COST OF IMPLEMENTATION	ERROR RATES	LEARNING CURVE OF USER
Rappaport et al., 2018⁴⁶	2	Yes	Yes	Unclear	No	No

QUALITY OF STUDIES

The appraisal of methodological quality was carried out using standardized critical appraisal instruments from the Joanna Briggs Institute Meta-Analysis of Statistical Assessment and Review Instrument (JBI-MAStARI).³⁶ The studies were grouped into the following categories: cross-sectional (n = 11), pre–post (n = 2), qualitative (n = 2), cohort (n = 2), quasi-experimental (n = 1), and economic evaluations (n = 3). The majority of studies, 15 (71.4%),^{39,41

–44,47
–49,51

–57} had strong methodologies and appropriate measures that met the recommended quality standards (Supplementary Table S3-S3e). Notably, the quality of the economic analysis among the three studies,^37,38,46 categorized as economic evaluations, was low. The studies failed to meet the recommended quality standards; they did not clearly report cost adjustments for differential timing, incremental analysis of costs and consequences, sensitivity analysis, or relevant costs for each alternative identified (Supplementary Table S3f).

Discussion

Our systematic review evaluated 21 studies that assessed virtual care interventions in correctional facilities and revealed several significant insights. First, no single study included in the review provided explicit definitions of the concepts of efficacy, effectiveness, and efficiency. Second, studies evaluating the effectiveness of virtual care interventions aligned best with the evaluation stage criteria outlined in the modified WHO conceptual framework. Third, the most frequently explored concept in the studies analyzed was effectiveness, followed by efficacy and efficiency, and the most common measures employed were clinical effectiveness, user satisfaction, and interexaminer agreement.

There was no common understanding of the concepts of efficacy, effectiveness, and efficiency in the included studies, and none of the studies provided a definition of the concept. Rather, the majority used measurement parameters linked to the outcomes as an implicit definition. For example, effectiveness is typically defined by measuring improvements in mental or clinical health parameters, such as sustained virological response rates for hepatitis C and HIV management, stabilization of vital signs or symptomatic relief. Roine et al.⁵⁸ also highlighted the common use of clinical terminology to define effectiveness, and that often studies that purport to measure effectiveness are, in fact, pilot studies measuring efficacy. Other studies either applied the terms interchangeably or merely referred to them in the abstract, discussion, or conclusion of the study. The varied application of these concepts is consistent with findings from other studies examining their use within the health care sector.^23,24,28,58 For instance, Burches and Burches²³ noted that these variations are unique to health research, where explicit definitions of efficacy and effectiveness are primarily applied to treatment evaluations in clinical trials. In these trials, a stage-based approach was employed, commencing with the evaluation of efficacy, which refers to the performance of a treatment in ideal conditions, followed by the evaluation of effectiveness, which gauges an intervention’s success in real-world settings. However, they pointed out that according to dictionary definitions and common usage, these three concepts remain synonymous in an everyday context. Labrique et al.²⁸ also noted a lack of a unifying language to guide the analyses and synthesis of virtual care interventions. This varied application underscores the importance of developing standard definitions and a guiding framework to ensure consistent and comparable assessment of virtual care interventions in correctional facilities.

Studies evaluating intervention effectiveness ranked highest in terms of conformity to relevant measurement targets associated with the evaluation stage outlined in the adapted WHO conceptual framework. For instance, all studies exploring intervention effectiveness measured changes in outcomes, and close to half measured changes in the process. Only one of the six studies evaluating efficacy adhered to the outlined measurement targets, and the one study exploring efficiency did not satisfy the criteria. Additionally, despite the distinct measurement targets for effectiveness and efficacy outlined in the WHO conceptual framework, the review findings indicated that similar measures were used to assess both the effectiveness and efficacy of an intervention. For example, user satisfaction and interexaminer agreement (reliability) are common measures explored by studies purported to evaluate effectiveness, even though these are measurement targets that are classified under efficacy. Studies that evaluated more than one concept did not permit the scoring of the framework to be performed. Of the studies explored in this review, only two explicitly identified the stage of intervention maturity as the pilot stage, even then, one of these studies explored the effectiveness of the intervention instead of its efficacy.

The WHO conceptual framework employed in this review illustrates the interrelationships between efficacy, effectiveness, and efficiency, as well as chronology, with efficacy requiring investigation during the early stages of the intervention’s maturity, followed by effectiveness and efficiency. Stage-based approaches to virtual care assessment have been proposed by other studies that emphasize the importance of a life-cycle approach to capture the varying needs that need to be accounted for at each developmental stage.^{28,30,58
–60} Herbert⁵⁹ argued that a stage-based approach with distinct milestones for each evolutionary stage is necessary, as virtual care interventions often impact health care systems beyond their initial intended purpose. Roine et al.⁵⁸ highlighted this issue and acknowledged the need for studies to address not only the initial stages of virtual care interventions but also the full lifespan, system scale-up, and sustainability. Similarly, Vis and colleagues,⁶⁰ after reviewing frameworks for virtual care implementation, determined that the sector would benefit significantly by adopting phased approaches when evaluating virtual care interventions.

In correctional facilities, the effectiveness of virtual care interventions was the most frequently explored concept, whereas the least explored concept was efficiency. The measurement components used to evaluate effectiveness included clinical effectiveness, which is the most common measure, followed by user satisfaction. Other measures included interexaminer agreement, the diagnostic quality of images, pre–post intervention differences, the availability of a complete treatment, and whether radiological and medication orders were completed prior to in-person visits. Evidence on the popularity of studies on virtual care effectiveness in community and correctional facilities is mixed^58,61,62; for example, a review by Roine et al.⁵⁸ on the effectiveness and cost-effectiveness of virtual care in community settings revealed that most of the studies assessed the clinical effectiveness of virtual care interventions. However, the study also pointed out that the assessment of effectiveness was in fact, at times, an assessment of efficacy. Aoki and colleagues found clinical effectiveness to be the least reported measure. Even so, the growing popularity of studies on the effectiveness of virtual care is reportedly driven by the interests of stakeholders who prioritize real-world data on virtual care performance over controlled settings. This preference stems from the difficulty in conducting randomized controlled trials (RCTs) owing to the small sample size.^59,63,64 The efficacy of virtual care interventions is the second most frequently explored concept. Common measurement components used to evaluate efficacy include user satisfaction and interexaminer agreement (reliability).

Other measures assessed the achievement of health-related outcomes, such as improvement in clinical parameters, pre- and postintervention differences, diagnostic quality of pictures, access to care, and savings related to cost and in-person visits. The efficiency of virtual care interventions was the least explored concept; as a result, no common measurement components were identified in this review. Instead, efficiency was mainly measured using cost analysis, as the benefits were predominantly estimated as savings, particularly in relation to patient transfers, compared to in-person consultations. Additionally, only one study provided a cost perspective. Although research in community settings has noted a paucity of studies evaluating the efficiency of virtual care interventions,^30,58 a review focused on the correctional facilities revealed the availability of efficiency studies within this context.⁶

GAPS

Overall, there is a paucity of studies focusing on women, youth, asynchronous modes of delivery, and primary health care in this setting. These studies predominantly concentrated on single medical specialties, with the majority evaluating real-time modes of virtual care delivery. A clear male sex bias and adult focus were observed when the sex and age of the study population were captured. Similar findings have been reported in related systematic reviews and ongoing studies of virtual care in correctional facilities.^5,6,65,66 Although studies have demonstrated the feasibility, effectiveness, and efficiency of asynchronous virtual care^5,37 in correctional facilities, adoption of this modality remains limited.

Future research should provide explicit definitions of efficacy, effectiveness, and efficiency to reduce variability and improve harmonization in the use of these terms and the evaluation of virtual care interventions in correctional facilities. Adoption of the WHO conceptual framework would help address this gap. Future research should also incorporate more comprehensive studies on virtual care efficiency given the significant investment in resources required for their deployment. Additionally, studies should explore the benefits of asynchronous modality for this demographic, the potential of virtual care in primary health care, and the specific virtual health care needs and obstacles faced by women and youth in custody.

This systematic review aims to enhance the clarity, shared understanding, and evaluation of three key concepts within correctional facilities. This will foster improved dialogue among health technology stakeholders and decision-makers facing public and commercial pressure to implement new virtual care initiatives in these settings. Additionally, this review provides valuable insights for custodial health stakeholders, as it not only provides a comprehensive overview of the most commonly used measures, but also offers a structured approach to evaluating the efficacy, effectiveness, and efficiency of virtual care interventions using a simplified life-cycle methodology.

LIMITATIONS

This review had several limitations. A broad and systematic search of published literature was undertaken; however, gray literature, which may have provided alternative insights into the examination of virtual care in correctional facilities, was not included. Several details were not reported in some studies, including age, sex, and ethnicity of the study population, as well as the duration of the study and the technology used. Owing to the observational nature of the included studies, nine articles utilized retrospective methods to collect data, which may have introduced misclassification bias. Additionally, not only were the studies primarily conducted in developed countries, but the small sample size may not be adequate to draw meaningful comparisons or generalizations. Non-English literature was not included in this review, which may have resulted in publication bias owing to the exclusion of potentially pertinent articles that were published in other languages.

Conclusions

The potential of virtual care to alleviate the high burden of disease experienced by highly vulnerable populations within correctional settings, particularly where face-to-face model of health care are limited is a growing area of interest in research and evaluation studies. This review offers valuable insights into the evaluation of virtual health care delivery in such settings. Foremost, the lack of standardized definitions for efficacy, effectiveness, and efficiency underscores the pressing need for an international consensus on a unifying language and framework. Establishing such standards is essential to facilitate consistent and comparable evaluation across studies. To enhance the evidence base, we advocate for the adoption of the modified stage-based conceptual framework by WHO.

Furthermore, tackling the current variability in how these key concepts are evaluated, considering the limited resources in correctional facilities, requires a paradigm shift in research priorities. Specifically, there is a need to expand the research focus from solely evaluating real-world applicability of virtual care interventions to also include more studies on virtual care efficiency within these contexts. To date, the nature of virtual care research in correctional facilities has predominantly focused on specialist synchronous health care for adult men. This emphasis underscores the need to explore new frontiers in virtual health care delivery in correctional facilities. Promising areas for further study include primary health care, store-and-forward technology, and the health needs of women and young people.

Footnotes

Acknowledgments

The authors would like to thank Jessica Birchall, Research Liaison Librarian, University of Newcastle Australia, for assistance with the development of the search strategy.

Authors’ Contributions

C.N., T.M., E.S., and M.R. conceived of and designed the study. C.N. defined the search terms and performed the initial literature search. C.N. and A.L. appraised the search, screening, full-text review, and assessment of the study quality. C.N. wrote the draft article. C.N., E.S., T.M., M.R., L.H., and X.D.G. contributed to the revision, editing, and approval of the final article.

Disclosure Statement

The authors declare that they have no competing interests.

Funding Information

No funding was provided for the research.

Supplementary Material

Supplementary Data

References

Penal Reform International. Global Prison Trends Report 2024. 2024. Available from: https://www.penalreform.org/global-prison-trends-2023/ [Last accessed: October 22, 2024 ].

UNODC. Data Matters Snapshot: Global Prison Population and Trends NM International Day, 2023. 2023. Available from: https://www.unodc.org/documents/data-and-analysis/briefs/DataMatters_NM_Day_2023.pdf [Last accessed: October 22, 2024 ].

UNODC. Data Matters 1. 2021. Available from: https://www.unodc.org/documents/data-and-analysis/statistics/DataMatters1_prison.pdf [Last accessed: October 22, 2024 ].

Helen

, Walmsley

. World Prison Population List (14th Edition). 2024. https://www.prisonstudies.org/sites/default/files/resources/downloads/world_prison_population_list_14th_edition.pdf [October 22, 2024].

Sethuram

, Helmer-Smith

, Karunananthan

, et al. Electronic consultation in correctional facilities worldwide: A scoping review. BMJ Open, 2022; 12(8):e055049; doi: 10.1136/bmjopen-2021-055049

Tian

, Venugopalan

, Kumar

, et al. The impacts of and outcomes from telehealth delivered in prisons: A systematic review. PLoS One, 2021; 16(5):e0251840; doi: 10.1371/journal.pone.0251840

Wheeler

, Hinton

. Effectiveness of telehealth on correctional facility health care: A systematic review protocol. JBI Database System Rev Implement Rep, 2017; 15(5):1256–1264; doi: 10.1112/4/JBISRIR-2016-002969

Latifi

, Osmani

, Kilcommons

, et al. Telemedicine for Prisons and Jail Population: A Solution to Increase Access to Care. Springer: Cham; 2021.

Fazel

, Baillargeon

. The health of prisoners. Lancet, 2011; 377(9769):956–965.

10.

Lancet Public Health. Promoting health beyond prison walls. Lancet Public Health, 2022; 7(7):e573; doi: 10.1016/S2468-2667(22)00141-4

11.

Sabin

. Improving prisoner health for stronger public health. Lancet, 2021; 397(10274):555–555; doi: 10.1016/S0140-6736(21)00361-5

12.

Binks

. District nursing in the digital era. Br J Community Nurs, 2017; 22(10):478–483.

13.

Mikow-Porto

, Smith

. The IHSSF 2011 prisoner escape study. J Health Care Prot Manag, 2011; 27(2).

14.

Edge

, George

, Black

, et al. Using telemedicine to improve access, cost and quality of secondary care for people in prison in England: A hybrid type 2 implementation effectiveness study. BMJ Open, 2020; 10(2):e035837; doi: 10.1136/bmjopen-2019-035837

15.

Gualano

, Bert

, Andriolo

, et al. Use of telemedicine in the European penitentiaries: Current scenario and best practices. Eur J Public Health, 2017; 27(1):30–35; doi: 10.1093/eurpub/ckw145

16.

Herzer

, Pronovost

. Ensuring quality in the era of virtual care. JAMA, 2021; 325(5):429–430; doi: 10.1001/jama.2020.24955

17.

, Borycki

, Kushniruk

. Connecting the world of healthcare virtually: A scoping review on virtual care delivery. Healthcare (Basel), 2021; 9(10):1325; doi: 10.3390/healthcare9101325

18.

Schwamm

, Erskine

, Licurse

. A digital embrace to blunt the curve of COVID19 pandemic. NPJ Digit Med, 2020; 3(1):64; doi: 10.1038/s41746-020-0279-6

19.

Jamieson

, Wallace

, Armstrong

, et al. Virtual care: A framework for a patient-centric system. 2015.

20.

Babaei

, Zamanzadeh

, Valizadeh

, et al. A scoping review of virtual care in the health system: Infrastructures, barriers, and facilitators. Home Health Care Serv Q, 2023; 42(2):69–97; doi: 10.1080/01621424.2023.2166888

21.

Shaw

, Brewer

, Veinot

. Recommendations for health equity and virtual care arising from the COVID-19 pandemic: Narrative review. JMIR Form Res, 2021; 5(4):e23233.

22.

Wong

, Bhyat

, Srivastava

, et al. Patient care during the COVID-19 pandemic: Use of virtual care. J Med Internet Res, 2021; 23(1):e20621; doi: 10.2196/20621

23.

Burches

, Burches

. Efficacy, effectiveness and efficiency in the health care: The need for an agreement to clarify its meaning. Int Arch Public Health Community Med, 2020; 4(1):1–3.

24.

Patel

. Efficacy, effectiveness, and efficiency. Natl J Community Med, 2021; 12(02):33–39.

25.

Palmer

, Torgerson

. Definitions of efficiency. BMJ, 1999; 318(7191):1136.

26.

Brook

, Lohr

. Efficacy, effectiveness, variations, and quality: Boundary-crossing research. Med Care, 1985; 23(5):710–722.

27.

World Health Organization. Monitoring and Evaluating Digital Health Interventions: A Practical Guide to Conducting Research and Assessment. Geneva; 2016. Available from: https://iris.who.int/bitstream/handle/10665/252183/9789241511766-eng.pdf [Last accessed: October 22, 2024 ].

28.

Labrique

, Vasudevan

, Weiss

, et al. Establishing standards to evaluate the impact of integrating digital health into health systems. Glob Health Sci Pract, 2018; 6,(Suppl 1):S5–S17.

29.

DeChant

, Tohme

, Mun

, et al. Health systems evaluation of telemedicine: A staged approach. Telemed J E Health, 1996; 2(4):303–312.

30.

LeFevre

, Shillcutt

, Broomhead

, et al. Defining a staged-based process for economic and financial evaluations of mHealth programs. Cost Eff Resour Alloc, 2017; 15:5–16.

31.

Ross

, Wood

, Baird

, et al. Digital Service Delivery Applications in Corrections: A Scoping Review. Australian Institue of Criminology; 2023.

32.

Williams

, Singh

, Elumn

, et al. Enhancing healthcare accessibility through telehealth for justice impacted individuals. Front Public Health, 2024; 12:1401950.

33.

Hearty

, Canvin

, Bellass

, et al. Understanding the impact of Covid-19 on the delivery and receipt of prison healthcare: An international scoping review. Health Justice, 2023; 11(1):42.

34.

Page

, McKenzie

, Bossuyt

, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 2021; 88:n71; doi: 10.1136/bmj.n71

35.

New South Wales Government. NSW Legislation: Crimes Act 1999 No 93. 1999. Available from: https://legislation.nsw.gov.au/view/html/inforce/current/act-1999-093#sec.2A

36.

Institute JB. Critical Appraisal Tools. 2014. Available from: https://jbi.global/critical-appraisal-tools [Last accessed: October 22, 2024 ].

37.

Barrera-Valencia

, Benito-Devia

, Velez-Alvarez

, et al. Cost-effectiveness of synchronous vs. Asynchronous telepsychiatry in prison inmates with depression. Rev Colomb Psiquiatr, 2017; 46(2):65–73; doi: 10.1016/j.rcp.2016.04.008

38.

Cuadrado

, Cobo

, Mateo

, et al. Telemedicine efficiently improves access to hepatitis C management to achieve HCV elimination in the penitentiary setting. Int J Drug Policy, 2021; 88:103031; doi: 10.1016/j.drugpo.2020.103031

39.

Sherwood

, Han

, Nepple

, et al. Evaluating the effectiveness, efficiency and safety of telemedicine for urological care in the male prisoner population. Urol Pract, 2018; 5(1):44–51; doi: 10.1016/j.urpr.2017.01.001

40.

Zarca

, Charrier

, Mahé

, et al. Tele-expertise for diagnosis of skin lesions is cost-effective in a prison setting: A retrospective cohort study of 450 patients. PLoS One, 2018; 13(9):e0204545; doi: 10.1371/journal.pone.0204545

41.

Gallagher

, Choi

, Garcia

, et al. Telemedicine in an otolaryngology clinic serving the incarcerated population. Ann Otol Rhinol Laryngol, 2023; 132(11):1321–1329; doi: 10.1177/00034894221149547

42.

Lewis

, Ash

, Roberts

, et al. Jail-based competency restoration services in the United States: The need, the controversy, the impact of COVID-19, and implications for future treatment delivery. Crim Justice Behav, 2023; 50(2):216–234; doi: 10.1177/00938548221120280

43.

Sherwood

, Nepple

, Erickson

. Managing the high incidence of testicular pain and pathology in prisoners with telemedicine. Urol Pract, 2018; 5(5):372–376; doi: 10.1016/j.urpr.2017.09.004

44.

Kassar

, Roe

, Desimone

. Use of telemedicine for management of diabetes in correctional facilities. Telemed J E Health, 2017; 23(1):55–59; doi: 10.1089/tmj.2016.0036

45.

Khairat

, Zebrowski

, Stabile

, et al. Assessment of stakeholder perceptions and cost of implementing a telemedicine specialty program at correctional facilities in North Carolina. Perm J, 2023; 27(3):49–59; doi: 10.7812/tpp/22.167

46.

Rappaport

, Reynolds

, Baucom

, et al. Telehealth support of managed care for a correctional system: The open architecture telehealth model. Telemed J E Health, 2018; 24(1):54–60; doi: 10.1089/tmj.2016.0275

47.

Young

, Patel

, Badowski

, et al. Improved virologic suppression with HIV subspecialty care in a large prison system using telemedicine: An observational study with historical controls. Clin Infect Dis, 2014; 59(1):123–126; doi: 10.1093/cid/ciu222

48.

Perdacher

, Kavanagh

, Sheffield

, et al. Using the stay strong app for the well-being of indigenous Australian prisoners: Feasibility study. JMIR Form Res, 2022; 6(4):e32157; doi: 10.2196/32157

49.

Neuhaus

, Langbecker

, Caffery

, et al. Telementoring for hepatitis C treatment in correctional facilities. J Telemed Telecare, 2018; 24(10):690–696; doi: 10.1177/1357633X18795361

50.

Halder

, Li

, Sebastian

, et al. Use of telehealth to increase treatment access for prisoners with chronic hepatitis C. Intern Med J, 2021; 51(8):1344–1347; doi: 10.1111/imj.15451

51.

Jiménez-Galán

, Alia-Alia

, Vegue-González

, et al. The contribution of telemedicine to hepatitis C elimination in a correctional facility. Rev Esp Enferm Dig, 2019; 111(7):550–555; doi: 10.1723/5/reed.2019.6152/2018

52.

Pitarch-Bort

, Arnandis-Viñerta

, MaHerrero-Matías

. Usefulness of teledermatology in prisons. The experience in the Penitentiary Center Castellón II. Rev Esp Sanid Penit, 2023; 25(1):16–19; doi: 10.1817/6/resp.00062

53.

Giraudeau

, Inquimbert

, Delafoy

, et al. Teledentistry, new oral care tool for prisoners. Int J Prison Health, 2017; 13(2):124–134.

54.

Inquimbert

, Balacianu

, Huyghe

, et al. Applications of teledentistry in a French inmate population: A one-year observational study. PLoS One, 2021; 16(4):e0247778; doi: 10.1371/journal.pone.0247778

55.

Cheng

, Siu

BWM

, Yeung

CCYA

, et al. Telepsychiatry for stable Chinese psychiatric out-patients in custody in Hong Kong: A case-control pilot study. Hong Kong Med J, 2018; 24(4):378–383; doi: 10.1280/9/hkmj187217

56.

Morosini

, de Oliveira

, Ferreira

, et al. Performance of distant diagnosis of dental caries by teledentistry in juvenile offenders. Telemed J E Health, 2014; 20(6):584–589; doi: 10.1089/tmj.2013.0202

57.

Seol

, Park

, Kim

. Analysis of live interactive teledermatologic consultations for prisoners in Korea for 3 years. J Telemed Telecare, 2018; 24(9):623–628; doi: 10.1177/1357633X17732095

58.

Roine

, Ohinmaa

, Hailey

. Assessing telemedicine: A systematic review of the literature. CMAJ, 2001; 165(6):765–771.

59.

Hebert

. Telehealth success: Evaluation framework development. Stud Health Technol Inform, 2001; 84(Pt 2)(:1145–1149; doi: 10.3233/978-1-60750-928-8-1145

60.

Vis

, Bührmann

, Riper

, et al. Health technology assessment frameworks for eHealth: A systematic review. Int J Technol Assess Health Care, 2020; 36(3):204–216.

61.

Hersh

, Hickam

, Severance

, et al. Diagnosis, access and outcomes: Update of a systematic review of telemedicine services. J Telemed Telecare, 2006; 12(2_suppl):S3–S31.

62.

Heinzelmann

, Williams

, Lugn

, et al. Clinical outcomes associated with telemedicine/telehealth. Telemed J E Health, 2005; 11(3):329–347.

63.

Quigley

, Thompson

, Halfpenny

, et al. Critical appraisal of nonrandomized studies—a review of recommended and commonly used tools. J Eval Clin Pract, 2019; 25(1):44–52.

64.

Aoki

, Dunn

, Johnson-Throop

, et al. Outcomes and methods in telemedicine evaluation. Telemed J E Health E Health, 2003; 9(4):393–401; doi: 10.2337/diacare.27.5.1095

65.

Edge

, Black

, King

, et al. Improving care quality with prison telemedicine: The effects of context and multiplicity on successful implementation and use. J Telemed Telecare, 2021; 27(6):325–342; doi: 10.1177/1357633X19869131

66.

Senanayake

, Wickramasinghe

, Eriksson

, et al. Telemedicine in the correctional setting: A scoping review. J Telemed Telecare, 2018; 24(10):669–675.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.06 MB

0.00 MB