Closing the Digital Health Evidence Gap: Development of a Predictive Score to Maximize Patient Outcomes

Abstract

Background:

Clinical studies of telemedicine (TM) programs for chronic illness have demonstrated mixed results across settings and populations. With recent uptake in use of digital health modalities, more precise patient classification may improve outcomes, efficiency, and effectiveness.

Objective:

The purpose of the research was to develop a predictive score that measures the influence of patient characteristics on TM interventions. The central hypothesis is that disease type, illness severity, and the social determinants of health influence outcomes, including resource utilization, and can be precisely characterized.

Methods:

The retrospective study evaluated the feasibility of creating a patient “Telemedicine ImPact” (TIP) score derived from a Virginia Medicare and Medicaid claims data set. Claims were randomly selected, stratified by disease type, and matched by illness severity into a TM intervention group (N = 7,782) and a nontelemedicine “usual care” control cohort (N = 7,981). The individual records were then summarized into 15,762 cases with 80% of the cases used to develop, train, and test four predictive models (hospital utilization, readmissions, total utilization, and mortality) using 10-fold cross-validation.

Results:

Bayesian supervised machine learning achieved reference model performance index area under the curve for receiver operating characteristic (AUC/ROC) ≥0.85. Posterior probabilities for each outcome model were generated on a “hold-back” set of 3,082 cases. Robust parametric statistical methods enabled dimension reduction, model validation, and derivation of a reliable composite scaled score that quantified the overall health risk for each case. The TM intervention cohort demonstrated higher total utilization (representing the sum of inpatient, outpatient, and prescription use) and lower mean inpatient utilization than the usual standard of care. This finding suggests TM-based care may shift the composition of health resource utilization, reducing hospitalizations while increasing outpatient services, adjusted for patient differences.

Conclusions

: The creation of a patient score using machine learning to predict the effect of TM on outcomes is feasible. Adoption of the TIP score may reduce variability in results by more precisely accounting for the effects of patient characteristics on health outcomes and utilization. More consistent outcome prediction may lead to greater support for digital health.

Introduction

The results of multiple studies point to disparate health outcomes originating in the social determinants of health (SDOH).^1,2 SDOH may contribute to the prevalence of chronic illness and growing health care inequity in diverse vulnerable patient populations.^3
–5 The presence of anxiety or depression is positively correlated with increased hospitalizations.^6
–8 Depression reduces adherence to treatment in patients with chronic illness, including worsening symptoms.^9,10 Poor health literacy and poverty may limit access to care and self-efficacy, as well as serve to reduce adherence to treatment.^11
–13

Telemedicine (TM) is thought to support earlier intervention, reducing the effects of illness exacerbations and readmissions.¹⁴ Studies have documented positive effects in improving access to care, health outcomes, and self-management.^15

–18 Despite an abundance of studies, mixed results are prevalent.^19

–23 The challenge remains to devise more effective digital health strategies that reach diverse populations with chronic conditions.^24

–27

The strategic use of data to identify high-risk patients and optimize treatment strategies has a storied history, continuing to find novel applications across clinical research and practice.^28
–30 Risk classification is used to assess lifestyle factors that increase health risks with significant positive public health benefits.^31,32 A risk model in this context could be applied to screen for and tailor TM treatment to better match patient need. The model, embodied in a risk-based score, could characterize the impact of patient characteristics on TM outcomes. The development of a patient risk classifier is intended to (1) array the population to identify high-need patients, (2) inform tailoring of TM interventions through objective criteria, and (3) adjust for patient-specific differences to increase likelihood of achieving desired outcomes.

Research Questions and Hypothesis

The primary objective of this study was to create a reliable valid score to control for patient characteristics in assessment of TM effectiveness. The central hypothesis is that disease diagnosis, illness severity, and the SDOH can be used to develop a robust and valid score that controls for patient variability. Adequate study controls increase the likelihood of discerning the true treatment effects of TM-based programs for chronic illness management. The research questions are as follows:

Research Question 1: Which patient characteristics can be used to develop a Telemedicine ImPact (TIP) score for prediction of health and utilization outcomes?

Hypothesis 1: A subset of patient characteristics including disease type, illness severity, and the SDOH can be used to develop a reliable and valid score to improve patient risk classification and outcome prediction.

Research Question 2: Is telemedicine more appropriate for one chronic disease than another?

Hypothesis 2: The TIP is independent of disease factors and, therefore, generalizable. The score improves outcome prediction by identifying a patient's likely response to telemedicine-based treatment while controlling for disease type, illness severity, and comorbidities.

Research Question 3: Can group differences in hospital and outpatient resource utilization be detected between the telehealth cohort and those subject to the usual standard of care by using the TIP score to control for patient characteristics?

Hypothesis 3: Statistically significant group differences in utilization between patients using telemedicine and those subject to usual care can be discerned with the score serving as the control variable.

The research questions guided the analysis to (1) identify the most influential characteristics from a set of candidate predictor and target outcome variables, generate probabilistic models, and create a composite TIP score quantifying patient health risks, (2) determine whether the score covaries by disease, and (3) assess group differences in resource utilization by comparing outcomes between those who use TM with a “usual standard of care,” non-telemedicine (NTM) cohort. The study sought to establish the predictive validity of the TIP score that generalizes across patient characteristics, including disease type.

Materials and Methods

Design

The research was approved by Virginia Commonwealth University Office of Research Subjects Protection under the exempt category per 45 CFR 46 with the study number IRB HM 20017231. The retrospective cohort design included a TM intervention group (N = 7,782) and NTM control cohort (N = 7,981). Randomized selection into each cohort used intervention or control criteria, including stratification by disease type and matching by illness severity.

Sample

Claims from adults (21 years and older) in the Virginia Health Institute Commonwealth of Virginia All Payer Claims Database (APCD) provided the authoritative source for the data.³³ Cases included four primary chronic illnesses: heart failure (HF), chronic obstructive pulmonary disease (COPD), diabetes, and hypertension. In addition, co-occurring illness included anxiety, depression, substance use, and end-stage kidney disease.^34
–36 International Classification of Diseases (ICD) 9/10 diagnosis codes identified the select chronic illnesses.³⁷ A minimum of one chronic illness from the mentioned set of target diagnoses served as inclusion criteria. A risk adjustment factor, the Milliman Advanced Risk Adjuster for concurrent risk of hospitalization (included in the APCD), provided the illness severity cohort matching criteria.^38,39 Centers for Medicare and Medicaid “Healthcare Common Procedure Coding System” and American Medical Association Current Procedural Terminology code sets (and code modifiers) were used to select the appropriate encounters into the TM and NTM cohorts.⁴⁰

Approach

To answer research question 1, robust parametric and nonparametric techniques were used for variable selection and model creation. A bivariate correlation matrix, principal components analysis, and Bayes unsupervised learning supported the exploratory analysis and dimension reduction to select input parameters and generate outcome models from a larger set of candidate variables. Final models were created using Bayes supervised machine learning. Each outcome model was resampled using 10-fold cross-validation specifying the target model performance criteria of area under the curve for receiver operating characteristic (AUC/ROC) >80%. Posterior probabilities generated from the outcome models were scaled to derive a single composite score for each case in the hold-back set of data. Analysis of covariance (ANCOVA) tests, using the same set of Bayesian model predictors, provided confirmation of results.

To answer research question 2, an ANCOVA test used the TIP score as the outcome variable, with patient characteristics (including disease diagnosis) as covariates, and the cohort designation (TM or NTM) as the predictor variable. This test sought to identify the explained variance in the score associated with a patient's chronic illness.

To answer research question 3, a multivariate analysis of covariance (MANCOVA) test was conducted using hospital and total utilization as outcome variables. The score and disease-related variables were entered as covariates, with the intervention cohort (TM or NTM) entered as the predictor variable. The covariates eliminated the variance associated with disease, leaving the remaining variance explained by the intervention. IBM SPSS (Version 26) and Bayesia Lab (Version 8) were used for the data analysis.^41,42

Data Collection

A total of 15,763 unique cases, aged 21 years and older, were derived from ∼3,500,000 claims records. Multiple claims were aggregated to a single deidentified (patient) case with utilization data for inpatient, outpatient, and prescription medications. After data cleaning and integrity checks, the training data set sample size was 12,680 cases. After eliminating one outlier, the “hold-back” set of data comprised 3,082 cases. Four variables (two predictors and two outcome variables) with positive skew were log transformed. Tables 1 and 2 detail the set of predictors and outcome variables, respectively, included in the data analysis. The variable coding procedures for aggregation from individual claims to deidentified cases are detailed in the Supplementary Appendix A. Table 3 lists the patient attributes of the TM intervention and NTM “usual standard of care” control groups. The cohorts appeared to be reasonably matched by sample size and demographics.

Table 1.

Candidate Predictor Variables

PREDICTOR (INDEPENDENT) VARIABLES	VARIABLE TYPE	CONTROL COHORT	TM COHORT	VALUE
1. Intervention type—cohort indicator	Categorical	X	X	NTM: no = 0, TM: yes = 1
2. Gender	Categorical	X	X	Male = 0, female = 1
3. Age—age in no. of years (or Age_Group) as categorical variable for frequency analysis	Continuous	X	X	Individuals 21 years and older
4. Ethnicity/member race—ethnicity formed four binary predictors with “unspecified” as the reference category	Categorical	X	X	Unspecified = 0, white = 1, African American = 2, Other = 3, Latino = 5
5. Insurance status—Medicare or Medicaid	Categorical	X	X	0 = Medicare, 1 = Medicaid
6. TM type of service—NTM = 0 as reference category	Categorical		X	NTM = 0, 1 = synchronous, 2 = asynchronous
7. Technical experience	Continuous		X	Number of TM encounters in claims for each case
8. Federally qualified health center	Categorical	X	X	0 = no, 1 = yes
9. HF diagnosis (from ICD 9/10 codes)	Categorical	X	X	0 = no, 1 = yes
10. COPD diagnosis	Categorical	X	X	0 = no, 1 = yes
11. Diabetes diagnosis	Categorical	X	X	0 = no, 1 = yes
12. Hypertension diagnosis	Categorical	X	X	No = 0, yes = 1
13. Anxiety diagnosis	Categorical	X	X	No = 0, yes = 1
14. Depression diagnosis	Categorical	X	X	No = 0, yes = 1
15. Substance abuse	Categorical	X	X	No = 0, yes = 1
16. End-stage kidney disease	Categorical	X	X	No = 0, yes = 1
17. Other illness (if none of the above #10 to #17)	Categorical	X	X	No = 0, yes = 1
18. Chronic illness comorbidities	Categorical	X	X	No = 0, yes = 1
19. No. of comorbidities	Continuous	X	X	Number of comorbidities
20. Illness severity (MARA) risk adjuster	Continuous	X	X	Provided in the APCD data set from VHI; used as a log transform to reduce skew
21. Patient zip code-based poverty rate	Continuous	X	X	Patient zip code in claim matched to census data—used as a log transform to reduce skew
22. Provider zip code-based poverty rate	Continuous	X	X	Provider zip code in claim matched to census data
23. Patient county poverty rate	Continuous	X	X	Patient county-based poverty rate matched to census data

APCD, All Payer Claims Database; COPD, chronic obstructive pulmonary disease; HF, heart failure; ICD, International Classification of Diseases; MARA, Milliman advanced risk adjuster; NTM, nontelemedicine; TM, telemedicine; VHI, Virginia Health Institute.

Table 2.

Candidate Outcome Variables

OUTCOME (DEPENDENT) VARIABLES	VARIABLE TYPE	CONTROL COHORT	TM COHORT	VALUE
1. Hospital utilization	Continuous	X	X	Total of inpatient utilization from APCD claims (log transform)
2. Inpatient total cost	Continuous	X	X	Total cost of inpatient services in claims
3. No. of readmissions	Continuous		X	Derived from inpatient discharge date
4. Mortality (all-cause mortality)	Categorical	X	X	No = 0, Yes = 1 (“expired”)—inpatient record discharge status
5. Outpatient utilization	Continuous	X	X	Total utilization of outpatient, professional, and ancillary services from claims record
6. Outpatient cost	Continuous	X	X	Total cost of outpatient services paid
7. Total prescription utilization	Continuous	X	X	Total units of prescriptions ordered
8. Total prescription cost	Continuous	X	X	Total cost of prescriptions
9. Total prescription—days supply	Continuous	X	X	Total prescription supply
10. Total no. of prescription refills	Continuous	X	X	Total prescription refills
11. Total utilization	Continuous	X	X	Summarized utilization for all services including inpatient, outpatient, and prescriptions (log transform)
12. Total cost of all medical services	Continuous	X	X	Summarized cost across all services

Table 3.

Patient Attributes in Cohort Data

CHARACTERISTICS OF STUDY AND CONTROL GROUPS (N = 15,763)
VARIABLE	TM STUDY GROUP (DATA SET = 1)	NTM CONTROL GROUP (DATA SET = 0)
No. of patient cases, n (%)	7,782 (100%)	7981 (100%)
Men (gender = 0)	3,049 (39.18%)	3,476 (43.55%)
Women (gender = 1)	4,733 (60.82%)	4,505 (56.45%)
Age (in years), mean (SD)	61.49 (18.52)	69.43 (15.55)
Medicare (yes = 0)	6,375 (81.92%)	6,387 (80.03%)
Medicaid (no = 1)	1,407 (18.08%)	1,594 (19.97%)
Ethnicity—unspecified/unknown	246 (3.16%)	300 (3.76%)
Caucasian	4,843 (62.23%)	5,141 (64.42%)
African American	1,289 (16.56%)	948 (11.88%)
Hispanic	1,068 (13.72%)	816 (10.22%)
Other ethnicity	336 (4.32%)	776 (9.72%)
Poverty rate (by zip code), mean (SD)	15.37% (8.05%)	11.3% (8.37%)
Mortality (yes = 1)	553 (7.11%)	371 (4.65%)
Mortality (no = 0)	7,229 (92.89%)	7,610 (95.35%)
HF (yes = 1)	1,933 (24.84%)	1,292 (16.19%)
COPD (yes = 1)	2,696 (34.64%)	1,532 (19.20%)
Diabetes (yes = 1)	2,916 (37.47%)	2,183 (27.35%)
Hypertension (yes = 1)	5,460 (70.16%)	5,037 (63.11%)
Anxiety (yes = 1)	3,346 (43.00%)	975 (12.22%)
Depression (yes = 1)	4,752 (61.06%)	968 (12.13%)
Substance use (yes = 1)	3,573 (46.00%)	868 (19.5%)
End-stage kidney disease (yes = 1)	2,221 (28.54%)	1,702 (21.33%)
Comorbidities, mean (SD)	3.46 (1.74)	1.82 (1.44)
Illness severity (MARA), mean (SD)	11.34 (12.18)	7.10 (10.26)
Technical experience (no. of TM encounters), mean (SD)	7.26 (16.34)	0

SD, standard deviation; TM, Telemedicine.

Data Analysis and Results

Model Development

The first research question was informed by the results of parametric approaches to variable selection and Bayes machine learning on the set of predictors to target outcomes. Four predictive models were generated including hospital utilization, readmissions, total utilization, and mortality. Outcome variable factor loading results are detailed in the Supplementary Appendix B. A subset of 8 (of 23) candidate independent variables were found to be predictive of outcomes. Different subsets of patient characteristics served as predictors for separate outcome models as given in Table 4. One variable, the illness severity risk adjuster, served as a predictor across the four models. Five other variables, which are the intervention cohort (TM or NTM), technical experience, Medicare/Medicaid subscriber status, depression, and age, appear in more than one model. Two variables (the number of comorbidities and provider zip code poverty rate) were found to be predictors in a single Bayes outcome model each (i.e., total utilization and readmissions). The four Bayes network models including performance metrics are detailed in the Supplementary Appendices C–F. Each outcome model achieved a reference index ROC ≥0.85.

Table 4.

Summary of Model Predictors by Target Outcome

PREDICTOR VARIABLE	OUTCOME MODEL
	HOSPITAL UTILIZATION	READMISSIONS	TOTAL UTILIZATION	MORTALITY
	ROC INDEX = 0.88	ROC INDEX = 0.98	ROC INDEX = 0.85	ROC INDEX = 0.88
Illness severity risk adjuster (MARA)	X	X	X	X
Insurance status (Medicare or Medicaid)	X	X
Intervention or control cohort (TM/NTM)	X		X
Technical experience		X	X
Depression	X			X
Age			X	X
No. of comorbidities			X
Provider zip code poverty rate		X

ROC, receiver operating characteristic.

Follow-up ANCOVA tests were conducted as confirmation (and comparison) for the two continuous outcome variables, inpatient and total utilization. The intervention cohort served as the predictor in each test. Parameter estimates are given in Table 5. The only difference between the Bayes model and the ANCOVA results was the age variable for total utilization, which was not significant.

Table 5.

Parameter Estimates for Inpatient and Total Utilization

	INPATIENT UTILIZATION^a	TOTAL UTILIZATION^a
	ROC = 0.88	ROC = 0.85
Intervention cohort (TM and NTM)^b (independent variable)	p < 0.001, partial η² = 0.01 (−0.141 to −0.09)	p < 0.001, partial η² = 0.07 (−0.44 to −0.37)
Illness severity score (covariate)	p < 0.001, partial η² = 0.36 (0.92 to 0.98)	p < 0.001, partial η² = 0.156 (0.75 to 0.83)
No. of comorbidities (covariate)	—	p < 0.001, partial η² = 0.06 (0.09 to 0.11)
Payer type (Medicare or Medicaid subscriber) (covariate)	p < 0.001, partial η² = 0.02 (−0.21 to −0.15)	—
Technology experience (by no. of encounters) (covariate)	—	p < 0.001, partial η² = 0.01 (0.002 to 0.004)
Depression (covariate)	p < 0.001, partial η² = 0.01 (0.059 to 0.118)	—
Age (covariate)	—	p = 0.13, partial η² = 0.00 (−0.001 to 0.000)

ANCOVA confirmatory results provided for the two continuous Bayes outcome models.

TM and NTM cohorts.

ANCOVA, analysis of covariance; CI, confidence interval.

Note: Parameter estimates provided with p-values, partial η² effect size, and robust standard errors (99% CI).

TIP (Composite Score) Development

A batch utility generated posterior probabilities for each of the four outcomes in the hold-back set of 3,082 cases. Using an adjustment factor to uniformly discount for the differing risk probabilities across the discrete categorical distributions for each Bayes model outcome, the individual item probabilities were then scaled and summed into a single composite TIP score for each case. A robust estimate of the low and high range in percentiles for the TIP was distributed from a low score of 1.61 in the 5th percentile to a high score of 3.89 in the 95th percentile. The distribution is negatively skewed with lower scores corresponding to higher risk patients (scores of ≤2.56) and higher scores corresponding to healthier patients (≥3.70). A perfect score (lowest risk possible) equals 4.00 and the highest health risk scores could asymptotically approach 0 in theory. Using test construction evaluation measures, including item test analysis, the TIP score was found to be reliable (Cronbach's alpha = 0.654) and demonstrated criterion validity.

Cohort Differences in Mean TIP Scores

The second research question provides validation of the score across disease-related factors including comorbidities. ANCOVA was conducted using the TIP as the outcome variable. Covariates included the number of comorbidities, illness severity, HF, COPD, diabetes, and hypertension diagnoses. The TM intervention served as the predictor of interest. The test was significant for the intervention (F (1,3074) = 1,218.14, p < 0.001, partial η² = 0.280) with an adjusted R ² = 0.85. The significant control variables included the number of comorbidities (β = −0.137, p < 0.001, partial η² = 0.158) and the illness severity risk adjustment factor (β = −1.0, p < 0.001, partial η² = 0.521). There was a statistically significant mean difference in the TIP score between the TM intervention and the usual standard of care. The score was lower for TM cases (mean = 2.834) than for usual care (NTM) cases (mean = 3.256) (99% confidence interval [CI]). Robust parameter estimates are given in Table 6. TIP appears to improve outcome prediction by identifying a patient's likely response to TM, while controlling for diagnosis, illness severity, and comorbidities. One covariate in the model, a hypertension diagnosis, was found significant with a small effect size (p < 0.001, partial η² = 0.011). With that exception, the score appears to be independent of disease category and, therefore, generalizable.

Table 6.

TIP Score (Outcome Variable) Analysis of Covariance Summary of Parameter Estimates

PARAMETER	B	STANDARD ERROR	T	SIG.	99% CI		PARTIAL η²
PARAMETER	B	STANDARD ERROR	T	SIG.	LOWER	UPPER	PARTIAL η²
Comorbidities	−0.137	0.006	−24.040	0.000	−0.148	−0.126	0.158
Illness_Severity	−1.000	0.017	−57.843	0.000	−1.034	−0.966	0.521
HF	−0.045	0.017	−2.681	0.007	−0.077	−0.012	0.002
COPD	0.040	0.015	2.702	0.007	0.011	0.069	0.002
Diabetes	0.038	0.013	3.006	0.003	0.013	0.064	0.003
Hypertension	0.075	0.013	5.931	0.000	0.050	0.099	0.011
Intervention cohort	0.422	0.012	34.534	0.000	0.398	0.446	0.280

ESTIMATED MARGINAL MEANS
TIP SCORE
INTERVENTION COHORT	MEAN	STANDARD ERROR	99% CI
INTERVENTION COHORT	MEAN	STANDARD ERROR	LOWER	UPPER
Nontelemedicine	3.256^a	0.008	3.236	3.277
Telemedicine	2.834^a	0.008	2.813	2.855

TIP = telemedicine impact.

Bolded rows are significant at p < 0.001 and effect size ≥0.01 with robust standard errors. Covariates appearing in the model are evaluated at the following values: comorbidities = 2.62, illness severity = 0.8002, HF = 0.20, COPD = 0.27, diabetes = 0.31, and hypertension = 0.66.

Resource use in TM

The third research question was informed by statistically significant group differences in resource use found between patients using TM and the usual standard of care. MANCOVA identified a significant effect for hospital and total utilization with the intervention cohort as the predictor variable (TM or NTM) (λ = (F (2,3070) = 85.49, p < 0.001, partial η² = 0.053). TIP and disease diagnoses were entered as the set of control variables. The total utilization adjusted R ² = 0.43 and the hospital utilization adjusted R ² = 0.48. Table 7 summarizes the robust parameter estimates that met the threshold significance level for p < 0.001 and an effect size (partial η²) ≥0.01 for a large sample. The results highlight the mean differences for each utilization measure. Total utilization is significantly higher for the TM intervention (mean = 3.226) and lower for the NTM group (mean = 2.878), (p < 0.001, 99% CI). By contrast, hospital utilization (figure 36) is significantly lower in the TM cohort (mean = 0.356) and higher in the NTM cohort (mean = 0.479) (p < 0.001, 99% CI). The findings suggest a significant effect on resource utilization associated with the TM intervention, adjusted by the TIP. A “shifting” effect in use of TM was found in reduced inpatient services and increased prescriptions and outpatient services for the management of chronic illness.

Table 7.

Parameter Estimates for Multivariate Analysis of Covariance with Two Outcomes (Hospital and Total Utilization) and Estimated Marginal Means

DEPENDENT VARIABLE	PARAMETER	B	STANDARD ERROR	T	SIG.	99% CI		PARTIAL η²
DEPENDENT VARIABLE	PARAMETER	B	STANDARD ERROR	T	SIG.	LOWER	UPPER	PARTIAL η²
Total utilization (log)	TIP score	−0.320	0.026	−12.243	0.000	−0.388	−0.253	0.047
	Diabetes	0.143	0.026	5.525	0.000	0.076	0.210	0.010
	Hypertension	0.210	0.026	8.156	0.000	0.144	0.277	0.021
	Intervention (TM)	−0.348	0.030	−11.592	0.000	−0.426	−0.271	0.042
Hospital utilization (log)	TIP score	−0.597	0.020	−29.913	0.000	−0.648	−0.545	0.226
	End-stage kidney	0.129	0.023	5.651	0.000	0.070	0.187	0.010
	Intervention (TM)	0.123	0.023	5.371	0.000	0.064	0.182	0.009
Threshold for significance p < 0.001 and effect size ≥0.01
Remaining error variance for total utilization = 0.352 and hospital utilization = 0.223.

ESTIMATED MARGINAL MEANS	DEPENDENT VARIABLE	INTERVENTION	MEAN^A	STANDARD ERROR	99% CONFIDENCE INTERVAL
ESTIMATED MARGINAL MEANS	DEPENDENT VARIABLE	INTERVENTION	MEAN^A	STANDARD ERROR	LOWER	UPPER
	Total utilization	NTM	2.878^a	0.019	2.83	2.926
	Total utilization	TM	3.226^a	0.019	3.178	3.275
	Hospital utilization	NTM	0.479^a	0.014	0.443	0.516
	Hospital utilization	TM	0.356^a	0.014	0.319	0.393

Covariates appearing in the model are evaluated at the following values: TIP score = 3.0483, HF = 0.20, COPD = 0.27, diabetes = 0.31, hypertension = 0.66, anxiety = 0.28, depression = 0.36, substance use = 0.29, end-stage kidney disease = 0.24, comorbidities = 2.62.

Discussion

Cohort Differences

The differences in TIP scores indicated patients with higher health risks in the TM cohort. The summary of ANCOVA results for TIP (outcome) and the intervention (predictor) with the disease covariates demonstrated effective control for group differences with large proportions of variation explained by TM (adjusted R ² = 0.85). The MANCOVA for the two outcome variables (hospital and total utilization) by the intervention, with the TIP score and the individual set of diseases as covariates, resulted in an adjusted R ² = 0.42 and 0.48, respectively. This showed a significant share of explained variation in resource use for the intervention, adjusted for disease factors.

The TIP as shown in the two utilization models demonstrated a useful control and highlighted a key effect. The use of hospital services was lower in the TM group than the control. At the same time, the use of outpatient and prescription services was higher than the control. This makes sense in the context of TM programs that deliver care in geographically dispersed rural areas lacking robust medical services infrastructure.

Control Variables and Effect Sizes

The small effect sizes found in this study suggest that smaller experimental studies may not adequately control for subject variation. This could account for mixed results in the form of type 2 errors that mask true treatment effects. Controlling for traditional demographic factors may not adequately cover differences in the patient mix between treatment groups, thereby increasing the potential for selection bias. Other adjustments may be necessary, including better characterization of the patient's individual level of health needs, as suggested by the TIP score distribution. More precise characterization of the impact of the SDOH, including factors relevant to vulnerable underserved populations, may improve accuracy of outcome prediction in planning TM programs. Although the full scope of health determinants is not available in claims data, a significant effect was revealed in the composition of resources in the TM group as compared with usual care.

TM Adoption Policy Barriers

A patient's predisposing, enabling, and need characteristics are important factors in the prediction of health outcomes.^43,44 This study suggests a role for the empirical assessment of a patient's health risks and needs when considering digital health programs to deliver care. The TM intervention cohort was concentrated in distinct areas of the Commonwealth of Virginia lacking the full scope of medical resources. This conforms to current TM reimbursement policy for the years under study (2016–2017).^45,46 Policy rules through 2019 proscribed the use of TM to designated low medically resourced rural areas. Yet, even in these areas, barriers to adoption remained.^47
–49 Until 2020, TM covered a small set of services and encounters in a limited range of health care settings.

Interim changes in reimbursement rules (during the 2020 pandemic) have increased adoption, since TM encounters protect both provider and patient while providing access to care for a wide range of chronic and acute conditions. It is uncertain whether the short-term changes will revert once the current crisis ends. A future large-scale study, using data from the 2020 interim policy changes as a natural experiment, could complement these findings by increasing understanding of digital health effectiveness beyond low-resourced regions. An assessment of the impact of patient characteristics using the TIP score could discern effect of TM use across the broader population.

Borne out by the findings in this study and from previous research, TM serves as a distinguishing factor, as well as an essential bridge to delivering medical care in underserved patient populations. The prevalence of co-occurring behavioral health diagnoses (depression, anxiety, and substance use) lends additional credence to the need to identify and stratify the treatment of patients with both chronic illness and additional complex needs to achieve expected outcomes.^50,51 The TIP score could assist by providing an empirical basis for tailoring treatment, including indicating resources and strategies needed to reduce health disparities.

Conclusions

A subset of patient characteristics contributes to the creation of reliable and valid TIP score to improve patient treatment and consistency of TM program results. The predictive score demonstrated the association of a few risk factors with resource use. Predictive modeling could lead to improved screening and digital health practice innovations, including development of targeted participation criteria. Classification methods can provide patient-specific risk stratification supporting care planning and resource allocation decisions for large patient case loads. Busy care coordinators and nurse case managers may find the simple indicator provided by the TIP a helpful input to identify patients needing more intensive therapy or more suitable approaches to treatment.

A TIP score may provide an important indicator supporting diverse patient populations with chronic illness. Matching treatment intensity (and support) to patient need using an empirically derived indicator of risk may reduce the high variability in TM programs across settings and populations by adjusting for patient characteristics. More consistent outcome prediction may lead to greater support for digital health.

Footnotes

Acknowledgments

This article is based on the dissertation published in the VCU Scholar's Compass on February 18, 2020 (Crump, 2020).⁵² Thank you to Virginia Commonwealth University, The College of Health Professions, the Dissertation Committee, and the staff at the Virginia Health Institute for their amazing support for the research.

Disclosure Statement

No competing financial interests exist.

Funding Information

No funding was received for this article.

Supplementary Material

Supplementary Appendix A

Supplementary Appendix B

Supplementary Appendix C

Supplementary Appendix D

Supplementary Appendix E

Supplementary Appendix F

References

Gourevitch

, Curtis

, Durkin

, et al. The emergence of population health in US Academic Medicine: A qualitative assessment. JAMA Netw Open, 2019; 2:e192200.

Phelan

, Link

, Tehranifar

. Social conditions as fundamental causes of health inequalities: Theory, evidence, and policy implications. J Health Soc Behav, 2010; 51 Suppl:S28–S40.

Artiga

, Hinton

Beyond health care: The role of social determinants in promoting health and health equity. 2018. Available at www.kff.org/disparities-policy/issue-brief/beyond-health-care-the-role-of-social-determinants-in-promoting-health-and-health-equity/ (last accessed January 18, 2020 ).

Marmot

, Friel

, Bell

, Houweling

, Taylor

. Closing the gap in a generation: Health equity through action on the social determinants of health. Lancet, 2008; 372:1661–1669.

Shi

, Singh

. Delivering health care in America: A systems approach, 6th ed. Burlington MA: Jones & Bartlett, 2015:54–55, 415–452.

Musey Jr

, Patel

, Fry

, Jimenez

, Koene

, Kline

. Anxiety associated with increased risk for emergency department recidivism in patients with low-risk chest pain. Am J Cardiol 2018.122:1133–1141.

Wallace

, Stuart

, Vaughan

, Bennett

, Fahey

, Smith

. Risk prediction models to predict emergency hospital admission in community-dwelling adults: A systematic review. Med Care, 2014; 52:751–765.

de Boer

AGEM

, Wijker

, de Haes

. Predictors of health care utilization in the chronically ill: A review of the literature. Health Policy, 1997; 42:101–115.

Heiman

, Lim

, Bao

, Drenkard

. Depressive Symptoms Are Associated with Low Treatment Adherence in African American Individuals with Systemic Lupus Erythematosus. J Clin Rheumatol, 2018; 24:368–374.

10.

Simon

GE.

Treating depression in patients with chronic disease: Recognition and treatment are crucial; depression worsens the course of a chronic illness. West J Med, 2001; 175:292–293.

11.

Chen

, Yehle

, Albert

, et al. Relationships between health literacy and heart failure knowledge, self-efficacy, and self-care adherence. Res Social Adm Pharm, 2014; 10:378–386.

12.

Nicholas

, Kalantar-Zadeh

, Norris

. Socioeconomic disparities in chronic kidney disease. Adv Chronic Kidney Dis, 2015; 22:6–15.

13.

Sharma

, Colvin-Adams

, Yancy

. Heart failure in African Americans: Disparities can be overcome. Cleve Clin J Med, 2014; 81:301–311.

14.

Smith

, Holland

, McDonald

. Beyond forest plots: Clinical gestalt and its influence on COPD telemonitoring studies and outcomes review. BMJ Open, 2019; 9:e030779.

15.

Darkins

, Kendall

, Edmonson

, Young

, Stressel

. Reduced cost and mortality using home telehealth to promote self-management of complex chronic conditions: A retrospective matched cohort study of 4,999 veteran patients. Telemed J E Health, 2015; 21:70–76.

16.

Greenwood

, Young

, Quinn

. Telehealth remote monitoring systematic review: Structured self-monitoring of blood glucose and impact on A1C. J Diabetes Sci Technol, 2014; 8:378–389.

17.

Rush

, Hatt

, Gorman

, Janicki

, Polasek

, Shay

. Planning telehealth for older adults with atrial fibrillation in rural communities: understanding stakeholder perspectives. Clin Nurs Res, 2019; 28:130–149.

18.

Totten

, Womack

, Eden

, et al. Telehealth: Mapping the evidence for patient outcomes from systematic reviews [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US), 2016. Technical Briefs No. 26. Available at https://www.ncbi.nlm.nih.gov/books/NBK379320/ (last accessed November 15, 2019 ).

19.

Chaudhry

, Mattera

, Curtis

, et al. Telemonitoring in patients with heart failure [published correction appears in N Engl J Med. 2011 Feb 3;364:490] [published correction appears in N Engl J Med 2013 Nov 7;369:1869]. N Engl J Med , 2010; 363:2301–2309.

20.

Clarke

, Fursse

, Brown-Connolly

, Sharma

, Jones

. Evaluation of the National Health Service (NHS) direct pilot telehealth program: Cost-effectiveness analysis. Telemed J E Health, 2018; 24:67–76.

21.

Pekmezaris

, Tortez

, Williams

, et al. Home telemonitoring in heart failure: A systematic review and meta-analysis. Health Aff (Millwood), 2018; 37:1983–1989.

22.

Takahashi

, Pecina

, Upatising

, et al. A randomized controlled trial of telemonitoring in older adults with multiple health issues to prevent hospitalizations and emergency department visits. Arch Intern Med, 2012; 172:773–779.

23.

Upatising

, Wood

, Kremers

, et al. Cost comparison between home telemonitoring and usual care of older adults: A randomized trial (Tele-ERA). Telemed J E Health, 2015; 21:3–8.

24.

Evans

, Papadopoulos

, Silvers

, et al. Remote health monitoring for older adults and those with heart failure: adherence and system usability. Telemed J E Health, 2016; 22:480–488.

25.

Lorig

, Sobel

, Stewart

, et al. Evidence suggesting that a chronic disease self-management program can improve health status while reducing hospitalization: A randomized trial. Med Care, 1999; 37:5–14.

26.

Portz

, LaMendola

. Participation, retention, and utilization of a web-based chronic disease self-management intervention among older adults. Telemed J E Health, 2019; 25:126–131.

27.

Robbins

, Krebs

, Jagannathan

, Jean-Louis

, Duncan

. Health app use among us mobile phone users: Analysis of trends by chronic disease status. JMIR Mhealth Uhealth, 2017; 5:e197.

28.

Dawber

TR.

The Framingham study: The epidemiology of atherosclerotic disease. Cambridge, MA: Harvard University Press, 1980.

29.

Schatzkin

, Cupples

, Heeren

, Morelock

, Kannel

. Sudden death in the Framingham Heart Study. Differences in incidence and risk factors by sex and coronary disease status. Am J Epidemiol , 1984; 120:888–899.

30.

Sherman

, D'Agostino

, Cobb

, Kannel

. Does exercise reduce mortality rates in the elderly? Experience from the Framingham Heart Study. Am Heart J, 1994; 128:965–972.

31.

Bates

, Saria

, Ohno-Machado

, Shah

, Escobar

. Big data in health care: Using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood), 2014; 33:1123–1131.

32.

Golas

, Shibahara

, Agboola

, et al. A machine learning model to predict the risk of 30-day readmissions in patients with heart failure: A retrospective analysis of electronic medical records data. BMC Med Inform Decis Mak, 2018; 18:44.

33.

Virginia Health Institute. All payer claims database. Available at https://www.vhi.org/APCD/ (last accessed April 4, 2019 ).

34.

Gelberg

, Andersen

, Leake

. The Behavioral Model for Vulnerable Populations: Application to medical care use and outcomes for homeless people. Health Serv Res, 2000; 34:1273–1302.

35.

Nicholas

, Kalantar-Zadeh

, Norris

. Socioeconomic disparities in chronic kidney disease. Adv Chronic Kidney Dis, 2015; 22:6–15.

36.

Sharma

, Colvin-Adams

, Yancy

. Heart failure in African Americans: disparities can be overcome. Cleve Clin J Med, 2014; 81:301–311.

37.

Centers for Disease Control and Prevention. International classification of diseases, tenth edition. Available at https://www.cdc.gov/nchs/icd/icd10.htm (last accessed December 8, 2019 ).

38.

Milliman. Milliman Advanced Risk Technologies. Available at https://us.milliman.com/en/products/mara (last accessed July 30, 2020 ).

39.

Virginia Health Institute. All Payer Claims Database. Available at https://www.vhi.org/APCD/ (last accessed April 4, 2019 ).

40.

Centers for Medicare & Medicaid. CMS proposes to modernize medicare advantage, expand telehealth access for patients. CMS. Available at https://www.cms.gov/newsroom/press-releases/cms-proposes-modernize-medicare-advantage-expand-telehealth-access-patients (last accessed January 27, 2019 )

41.

IBM. IBM SPSS Software. 2020. Available at https://www.ibm.com/analytics/spss-statistics-software (last accessed November 16, 2019 ).

42.

Conrady

, Jouffe

Bayesian Networks & BayesiaLab. Franklin, TN: Bayesia USA, 2015. Available at www.bayesia.us (last accessed November 16, 2019 ).

43.

Andersen

, Aday

. Access to medical care in the U.S.: Realized and potential. Med Care 1978;16:533–546.

44.

Salisbury

, Thomas

, O'Cathain

, et al. TElehealth in CHronic disease: Mixed-methods study to develop the TECH conceptual model for intervention design and evaluation. BMJ Open, 2015; 5:e006448.

45.

Centers for Medicare & Medicaid. Centers for Medicare & Medicaid services chronic care management services. Available at https://www.cms.gov/Outreach-and-Education/Medicare-Learning-Network-MLN/MLNProducts/Downloads/ChronicCareManagement.pdf (last accessed January 27, 2019 ).

46.

Kwong

. State Telehealth Laws and Reimbursement

Policies

. Center for Connected Health Policy

Fall

. 2019. Available at https://www.cchpca.org/sites/default/files/2019-10/50%20State%20Telehalth%20Laws%20and%20Reibmursement%20Policies%20Report%20Fall%202019%20FINAL.pdf (last accessed July 30, 2020 )

47.

Fairchild

, Ferng-Kuo

, Laws

, Rahmouni

, Hardesty

. Telehealth decreases rural emergency department wait times for behavioral health patients in a group of critical access hospitals. Telemed J E Health, 2019; 25:1154–1164.

48.

Kim

, Gellis

, Bradway

, Kenaley

. Key determinants to using telehealth technology to serve medically ill and depressed homebound older adults. J Gerontol Soc Work, 2019; 62:451–474.

49.

Speyer

, Denman

, Wilkes-Gillan

, et al. Effects of telehealth by allied health professionals and nurses in rural and remote areas: A systematic review and meta-analysis. J Rehabil Med, 2018; 50:225–235.

50.

Greenhalgh

, A'Court

, Shaw

. Understanding heart failure; explaining telehealth—A hermeneutic systematic review. BMC Cardiovasc Disord, 2017; 17:156.

51.

Woo

, Dowding

. Factors affecting the acceptance of telehealth services by heart failure patients: An integrative review. Telemed J E Health, 2018; 24:292–300.

52.

Crump

. A risk-based telehealth needs scoring model for patient outcome prediction. Scholar Compass Dissertations and Theses. 2020. Available at https://scholarscompass.vcu.edu/etd/6117/ (last accessed July 13, 2020 ).

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

39.52 MB

0.00 MB