Abstract
We compare multiple machine learning algorithms and develop models to predict future hospitalization among Home- and Community-Based Services (HCBS) Users. Furthermore, we calculate feature importance, the score of input variables based on their importance to predict the outcome, to identify the most relevant variables to predict hospitalization. We use the 2012 national Medicaid Analytic eXtract data and Medicare Provider Analysis and Review data. Predicting any hospitalization, Random Forest appears to be the most robust approach, though XGBoost achieved similar predictive performance. While the importance of features varies by algorithm, chronic conditions, previous hospitalizations, as well as use of services for ambulance, personal care, and durable medical equipment were generally found to be important predictors of hospitalization. Utilizing prediction models to identify those who are prone to hospitalization could be useful in developing early interventions to improve outcomes among HCBS users.
• This is the first study to apply a machine learning approach to predict hospitalization among Medicaid Home- and Community-Based Services (HCBS) users. • Our results indicate that machine learning offers distinct improvements in predictive accuracy over other approaches. • This study shows that specific chronic conditions, previous hospitalizations, and use of ambulance/personal care/durable medical equipment services are important variables in predicting future hospitalization among HCBS users across different algorithms.
• Our study supports the feasibility of using prediction models to identify HCBS users early on who are prone to adverse health outcomes. • Current interventions in other care settings that seek to reduce the risk of hospitalizations can provide helpful information in designing interventions to specifically serve high-risk HCBS users. • Considering the importance of previous hospitalizations based on our feature importance analysis, strengthening discharge and transitional care for HCBS users based on current interventions would improve patient care and might prevent future hospitalizations.What this paper adds
Applications of study findings
Introduction
Medicaid home- and community-based services (HCBS) support older adults to receive long-term care (LTC) in non-institutional settings. Medicaid policymakers have promoted HCBS to potentially reduce LTC costs and to satisfy older adults’ preferences to stay home for as long as possible (Gruman & Menne, 2020). The percentage of total Medicaid LTC expenditures going to HCBS increased from 39% in 2006 to 56% in 2018 (Eiken et al., 2018; Murray et al., 2021). The momentum behind expanding HCBS is likely to continue, given recent attention by the Biden Administration to the need for investment in this sector (Musumeci, 2021). Yet as currently configured, Medicaid HCBS does not always result in beneficial patient outcomes. Older Medicaid-Medicare dual beneficiaries receiving HCBS appear more prone to hospital admission than their counterparts receiving institutional care (Konetzka et al., 2020), an unintended negative consequence of the shift in care toward home-based settings. The accompanying human and economic costs might be reduced through improved targeting of appropriate services and intensity of care. Identifying opportunities to improve health among HCBS users is essential in ensuring appropriate expansion of home-based services.
Older Medicaid beneficiaries with LTC needs are a major subgroup targeted by HCBS. Older beneficiaries who are eligible and enrolled for both Medicare and Medicaid—generally referred to as “dual eligible” or “duals”—are more likely to have multiple chronic conditions, physical disabilities, and cognitive impairments as well as higher use of health care services and cost than non-duals (Kane et al., 2013; Sommers et al., 2016). Thus, while dual-eligibles accounted for 15% and 20% of Medicaid and Medicare enrollment, respectively, they accounted for 32% and 34% of Medicaid and Medicare expenditures in 2013 (Medicare Payment Advisory Commission (MedPAC), Medicaid and CHIP Payment and Access Commission (MACPAC), 2018). Further, more than two-thirds of the Medicaid spending on older duals was allocated to LTC (Kaiser Family Foundation, 2016). The projected increase in older adults with the aging of the baby boomers will likely induce greater reliance on long-term care (Watts et al., 2020), increasing Medicaid’s programmatic and fiscal challenges. HCBS for dual-eligible seniors thus present key opportunities to improve patient care, to support the efforts and well-being of family and other caregivers, and potentially to reduce spending and improve cost-effectiveness of care.
While there is little definitive evidence, several factors may explain why HCBS users are prone to poor outcomes. Receiving LTC in home-based settings often implies a low intensity of care. Home environments may not be safe or appropriately designed to accommodate patient needs (National Research Council (U.S.). Committee on the Role of Human Factors in Home Health Care., 2013); home care workers may face challenges implementing high-intensity treatments in the home environment (Silver & Wellman, 2002). Furthermore, in contrast to institutional settings, HCBS relies on largely untrained family or friends to fill in critical gaps of care intensity (Konetzka et al., 2020). Informal caregivers are rarely thoroughly trained to handle clinical issues, yet typically operate with limited observation or support from medical professionals (Burgdorf et al., 2019). Combined, these factors may result in poor control of health conditions and higher propensity for hospitalization once a worrisome condition develops (Konetzka et al., 2020). Developing opportunities for early identification for HCBS users at high risk for adverse health outcomes could open new windows to provide needed and timely care.
Machine learning (ML) approaches may provide opportunities to improve the health of HCBS users by identifying high-risk individuals early on. ML is a powerful tool to derive predictive insights about future outcomes based on past data (Naimur Rahman et al., 2016). One stream of ML studies seeks to predict adverse health outcomes. For example, previous studies used statistical and ML tools to identify Medicare patients with high risk of hospital readmission (Garcia-Arce et al., 2018; Li et al., 2020). Similarly, to predict risk and health outcomes, other studies utilized ML models in Medicaid, which could be used to prioritize individual patients for interventions and screening (Holcomb et al., 2022; Lo-Ciganic et al., 2019).
While ML methods are used in other care settings to predict adverse health outcomes to apply early target interventions and screenings, these efforts have been limited in Medicaid HCBS. Considering the characteristics and home environment of Medicaid HCBS users mentioned above, efforts to utilize ML methods to early identify high-risk HCBS users hold great promise to benefit the HCBS program.
Thus, in this paper, we aim to develop and compare multiple ML algorithms to predict hospitalization among HCBS users by using Medicaid and Medicare claims data. Our study provides opportunities to develop early interventions for high-risk HCBS users, hopefully improving health and quality of life by slowing the progression of salient risk factors and preventing predictable adverse outcomes.
Methods
Our study first develops a conventional logistic regression model to predict hospitalization among dual-eligible seniors who are HCBS users. We then compare the predictive performance of this simple model with several ML algorithms (lasso regression, random forest, and XGBoost). After evaluating the different models’ performance, we use these models to describe HCBS users who face the highest hospitalization risk.
Data Sources
We use the 2012 national Medicaid Analytic eXtract (MAX) and Medicare Provider Analysis and Review (MedPAR) to develop our analysis dataset. These 2012 data were the latest MAX available nationally at the time of study initiation. MAX data are a set of individual-level enrollment and claims data files for Medicaid enrollees. This data set is created by the Centers for Medicare and Medicaid Services (CMS), derived from data submitted quarterly by states (Ruttner et al., 2015). We use MAX data to identify HCBS users, type of service use, and expenditures. The MedPAR data contain information on Medicare beneficiaries using hospital inpatient services, which we use to identify hospitalizations by month. We draw upon the Medicare Master Beneficiary Summary File (MBSF) for demographic characteristics, Medicare managed care enrollment, original reason for Medicare eligibility, and health status, including indicators of chronic conditions. Rural-Urban Commuting Area (RUCA) Codes were used to define beneficiaries’ geographic location.
Study Sample
We focus on older-adult (age 65 or older) Medicare and Medicaid dual beneficiaries who received LTC under HCBS. Following previous literature (Ruttner et al., 2015), we identify HCBS users as those who have (1) a 1915(c) HCBS enrollment record, or (2) fee-for-service claims for use of waiver services or state plan LTC services during the baseline period. Using the MAX Long-Term Care files, we limit our sample to HCBS users who have not received any institutional care during the input period (will use the term “input period” as the length of data that is fed into the model). We also exclude beneficiaries with missing demographic information or who died during the input period. We exclude HCBS users enrolled in Medicare Advantage, as we identify hospitalizations based on fee-for-service claims. Finally, we exclude beneficiaries who were hospitalized and were not discharged before the end of input period. Our final study sample includes 1,217,019 HCBS users. Supplementary Appendix Table 1 provides sample size by each exclusion criteria.
Outcome and Predictors
Hospital admission is used as our main outcome. We focus on hospitalization because it is costly and can reduce a patient’s quality of life (Centers for Medicare & Medicaid Services, 2021; Kimmey & Verdier, 2015), and is sometimes a marker of poor care quality or missed opportunities for preventive interventions. Hospitalizations impose significant burden on patients and families (Alzahrani, 2021; Bassett et al., 2020; Belayachi et al., 2014), while exposing HCBS users to health risks such as nosocomial infection (Haque et al., 2018; Monegro et al., 2022). By definition, frequent hospitalizations detract from the potential social-emotional benefits of home-based care (AARP Public Policy Institute, 2006).
We use data on HCBS users’ demographic characteristics, health status, and previous service use to predict future hospitalization. We include demographic characteristics such as age, gender, and geography (state of residence). Also, we include health status information including 24 indicators of chronic conditions and the count and spending of previous hospitalizations by month during the baseline period. We also include HCBS spending and use by month, and type of services used, such as personal care and home health services. (The full list is provided in Supplementary Appendix Table 2).
Model Performance
We use area under curve (AUC) for both the precision-recall (PR) curve and the receiver operating characteristic (ROC) curve to evaluate and compare algorithms. The precision-recall curve shows the estimates of precision and recall at all possible threshold scores (Boyd et al., 2012).
Precision (predictive value positive) is the probability that subjects with a positive screening indication actually experience the outcome one seeks to predict. Recall (sensitivity), in contrast, represents the probability that someone who actually experiences that outcome will display a positive screening indication (Ramola et al., 2019). Though high precision and high recall are both preferred, they are generally inversely related. Implementing a given screening test so as to maximize precision will reduce recall, and vice-versa.
The area under precision-recall curve (AUPRC) summarizes a classifier’s performance as a single score which ranges from 0 to 1 (Saito & Rehmsmeier, 2015). The value of AUPRC for a purely random classifier is equal to the actual positive rate. Precision-recall curves are known to be more informative than ROC curves when evaluating binary classifiers on imbalanced data (Saito & Rehmsmeier, 2015). We therefore primarily focus on AUPRC, as our outcomes are imbalanced classes, in other words, unequal distribution in outcome variables.
We also use AUROC as a model evaluation tool. The ROC is created by plotting the true positive rate against the false positive rate at different cutoffs. This not only measures the level of classification accuracy but also reaches a balance between the true positive and false positive rates (Fawcett, 2006). The AUROC is a measure of how well a parameter can distinguish between two diagnostic groups, in which 1.0 means that the model perfectly distinguishes between positive and negative class, and 0.5 indicates purely random guessing (Hanley & McNeil, 1982).
Model Development
We develop three different types of models—LASSO regression, random forest, and XGBoost—and compare these models with logistic regression. Each approach brings advantages and disadvantages. LASSO retains some of the transparency and interpretability associated with standard regression (Balabaeva & Kovalchuk, 2021); while RF and XGBoost typically display greater predictive performance (Boehmke & Greenwell, 2019; Chang et al., 2019).
For model development, we divide the dataset into a training set and test set, using a 70:30 ratio, to compare algorithms and reduce over-fitting (Tan et al., 2018). We use the 10-fold cross-validation method to tune hyperparameters based on the best average precision.
LASSO (Least Absolute Shrinkage and Selection Operator) Regression is a shrinkage method that imposes a penalty proportional to the sum of the absolute coefficients (the L1 norm) (Göbl et al., 2015). By penalizing the regression model in this way, Lasso shrinks the coefficients with minor contributions to the model toward zero. As such, Lasso regression is used not only to prevent over-fitting but also for feature selection (Pavlou et al., 2016). We tune the hyperparameter—generally called lambda—that sets the weighting of the penalty to the loss function. Higher lambda produces a simpler model that is less vulnerable to over-fitting, but also provides lower predictive accuracy within a given dataset.
Random Forest (RF) is an ensemble of Classification and Regression Trees where each individual tree predicts the outcome. The ensemble averages the predicted values. RF uses modified bagged decision trees to prevent highly correlated trees where all variables are considered at every split of every tree. Hence, RF uses bootstrapping, a randomly selected sample, so each tree is developed based on a resampled data set. Also, by using a random subset of variables at each split of every tree, RF reduces the correlation between trees. We use grid search to find the best hyperparameters (number of trees, maximum depth of each tree, minimum number of samples required to split an internal leaf node, minimum number of samples required to be at a leaf node) for RF.
eXtreme Gradient Boosting (XGBoost) is based on gradient-boosted decision trees designed for computational speed and model performance, particularly for large datasets (Liu et al., 2020). XGBoost use an ensemble technique where new models are added to correct the errors made by previous models. Models are added and updated sequentially with increased weights for misclassified observations until no further improvements can be made. In particular, we tune the following hyperparameters: maximum depth of a tree, learning rate, subsample percentage, subsample ratio of variables, and maximum tree depth.
Analysis
First, we compare characteristics of HCBS users stratified by hospitalization during the August 2012 outcome period. Next, as our main analysis, we develop and compare four different algorithms to predict these August hospitalizations using May and June 2012 input period data. We focus on hospitalization 2 months later to ensure that the service use and health status were precursors and not immediate consequences of hospitalization.
We further explore feature importance, based on the best performing model. Feature importance is the value assigned to a specific measure based on its relevance or importance in relation to the outcome. Higher scores mean greater relevance to the outcome (Brick et al., 2017).
To achieve better understanding of how the models are affected by time, we conduct secondary analyses using different input and outcome periods. Specifically, we use either four or 6 months of HCBS users’ data to predict the health outcome in August. Also, we use the same 2 months (May and June) of data to predict the outcome across 2 months (August and September). Finally, we test the model developed from the main analysis (use 2 months of data to predict the outcome in August) to data from different input periods as a sensitivity analysis. For example, we apply the model developed from the main analysis to a test set with data from July to August predicting October outcomes (Supplementary Appendix Table 3).
Results
Sample Characteristics by Hospitalization in August.a
Note. COPD = Chronic Obstructive Pulmonary Disease; RA OA = Rheumatoid Arthritis/Osteoarthritis; Equip, Tech, & Mod = Equipment, Technology, and Modifications.
aWe only displayed diseases and services with >5%.
Figure 1 displays the main results for predicting any August hospitalization using May-to-June data. Figure 1(a) shows the results for AUPRC and Figure 1(b) shows the results for AUROC. Based on the AUPRC, RF (AUPRC = 0.348, confidence interval (CI): [0.342, 0.354]) and XGBoost (AUPRC = 0.339, CI: [0.333, 0.345]) show higher AUPRC than did LASSO (AUPRC = 0.280, CI: [0.275, 0.285]) or logistic regression (AUPRC = 0.271, CI: [0.266, 0.276]). XGBoost shows highest estimated AUROC of 0.901 (CI: [0.900, 0.903]) followed by RF (AUROC = 0.898, CI: [0.896, 0.899]), LASSO (AUROC = 0.834, CI: [0.831, 0.837]), and logistic regression (AUROC = 0.829, CI: [0.827, 0.832]). Performance of algorithms for predicting hospitalization in August with test set based on area under precision-recall curve (a) and area under receiver operating characteristic curve (b). Note. AUPRC = Area Under Precision-Recall Curve; AUROC = Area Under Receiver Operating Characteristic Curve; TPR = True Positive Ratio; FPR = False Positive Ratio. *Square dots indicate the precision and recall based on the highest F-1 value for each algorithm.
Figure 2 displays the feature importance of the model based on our high-performance algorithms, XGBoost and RF. Length of stay during HCBS users’ previous hospitalizations and use of HCBS services such as DME and personal care services were highly predictive. Specific chronic conditions such as chronic kidney disease, COPD, and diabetes also displayed high feature importance in predicting hospitalization. Feature importance by using random forest and XGBoost. Note. COPD = Chronic Obstructive Pulmonary Disease; RA OA = Rheumatoid Arthritis/Osteoarthritis; CHF = Chronic Heart Failure; LOS = Length of stay during previous hospitalizations. *Feature importance values of LOS_m5 and LOS_m6 from the XGB model are 0.21 and 0.19, respectively.
Performance of Algorithms for Hospitalization Based on Area under Precision-Recall Curve by Different Data Time Periods. (Accompanying 95% Confidence Intervals Shown in Parentheses).
aUse 2 months (May–June) of HCBS users’ data to predict hospitalization in August.
bUse 4 months (March–June) of HCBS users’ data to predict hospitalization in August.
cUse 6 months (January–June) of HCBS users’ data to predict hospitalization in August.
dUse 2 months (May–June) of HCBS users’ data to predict hospitalization in August and/or September.
Our sensitivity analyses using different time periods indicate that our study results are robust. Similarity to our main analysis, we can see that RF produces highest estimated AUROC compared other algorithms across different time periods (September, AUPRC = 0.330, CI: [0.321, 0.339]; October, AUPRC = 0.330, CI: [0.324, 0.336]; November, AUPRC = 0.328, CI: [0.323, 0.334]; Supplementary Appendix Table 3).
Discussion
This novel study uses and compares ML algorithms applied to Medicare and Medicaid claims data to predict hospitalization among older duals using Medicaid HCBS. The 1-month hospitalization rate was 6.6%, and in our main analysis model, which uses 2 months of claims data as our input data, RF showed the best model performance based on both AUPRC, with XGBoost displaying roughly similar results. Longer periods of input and outcome data produced even better model performance.
Our results show that RF and XGBoost produce higher model performance compared to other algorithms based on AUPRC. This study suggests the viability of applying ML methods to develop predictive models for target interventions among HCBS users in Medicaid. The high performance of RF and XGBoost is not surprising, as tree-based ensemble algorithms are well known for producing a high-performance prediction model and obtaining winning solutions in various data competitions (Chang et al., 2019). Yet, in selecting the most suitable algorithm to develop a prediction model for particular applications and settings, other characteristics such as simplicity, transparency, speed and training time, and scalability and deployment should be considered.
The feature importance based on Figure 2 highlights the characteristics related to future hospitalization, providing insight on the dataset and model, and on how interventions could be designed or applied. While feature importance varies by algorithm (i.e., XGBoost and RF), length of stay during previous hospitalizations and indictors for specific chronic conditions are some of the most important variables across different algorithms in predicting future hospitalization. Previous studies have shown higher number of prior hospitalizations and longer lengths of previous hospitalizations are associated with increased risk of subsequent hospitalizations (Arundel et al., 2021; Lin et al., 2019). Also, indicators for specific chronic conditions, such as chronic kidney disease and diabetes, are known risk factors of hospitalization.
While the calculated feature importance metrics do not show the direction of association with future hospitalization, based on previous studies, we can infer the direction of the relationship between these factors and future hospitalization. For example, our study highlights the importance of length of stay during previous hospitalizations in predicting future hospitalizations. While feature importance does not provide the direction of association, we could expect longer length of stay during previous hospitalizations to be associated with higher risk of hospitalization based on previous literature (Arundel et al., 2021). The simultaneous use of our findings with previous evidence can help identify people who are more likely in need, further facilitating the design and delivery of interventions to improve health and prevent adverse health outcomes among HCBS users.
Furthermore, use of specific services such as personal care and DME were found to be important markers in predicting future hospitalization. Given the non-causal framework of our models, such service receipt is best-regarded as an important marker for hospitalization risk rather than as a causal agent. HCBS users who need and receive specific services likely do so because they face other underlying risk factors for hospitalization. Indeed, the receipt of such services may dampen the likelihood of hospitalization among HCBS users with certain conditions. Further studies to clarify the role of services for patients at hospitalization risk may be especially beneficial to improve care for HCBS users.
Based on prediction models and study results, policy makers could develop and apply target interventions for HCBS users to prevent hospitalizations and worsening health symptoms early on. While the HCBS setting remains understudied, current interventions in other care settings aiming to reduce the risk of hospitalizations can provide helpful information in designing interventions targeting high-risk HCBS users (Delta Health Technologies et al., 2012; Purdy et al., 2012). These interventions contain strategies such as medication management, patient education, disease and case management, and fall reduction programs. To further facilitate the effectiveness and development of these interventions and strategies, we suggest consideration of the characteristics and environment of Medicaid-enrolled older-adult HCBS users. For instance, strategies such as medication management are important intervention components to reduce hospitalizations. Medication management programs provide a comprehensive approach to assess and monitor each patient’s ability and willingness to accurately and safely maintain their prescribed medication regimen (Delta Health Technologies et al., 2012).
A majority of older-adult HCBS users receive long-term care at their home and heavily rely on caregivers for daily self-care and household activities. Highlighting and incorporating the role of caregivers into these strategies is thus critical in focusing services on high-risk HCBS users. Similarly, given the importance of particular chronic conditions based on our feature importance analysis and the high prevalence of these chronic conditions among HCBS users, it may be fruitful to incorporate the characteristics of these chronic conditions in interventions. For example, for high-risk HCBS users with chronic kidney disease, including characteristics of medical interventions aimed to slow down the progression, such as strategies to control blood pressure and use medications to lower urine albumin, should be considered.
Moreover, compared to institutional LTC users, HCBS users receive limited monitoring and care from medical professionals. This creates further challenges in recognizing and managing acute conditions early on, before these become severe. Based on prediction models, we could develop and utilize periodic medical professional home or telemedicine visits for high-risk HCBS users for medical check-up/to assess health conditions which could facilitate early disease management plans to appropriately manage acute conditions after diagnosis. Emphasizing education and providing materials to caregivers related to cautious symptoms or reporting protocols could also be important in identifying and managing acute conditions.
Given the prominent position of previous hospitalizations in our feature importance analysis, strengthening the discharge and transitional care for HCBS users based on current interventions could help to prevent future hospitalizations. Repeated hospitalizations force HCBS users to leave their home and community. This reduces the quality of life of HCBS users; it also increases overall health care spending. Several interventions are known to improve discharge care and transitional care including Re-Engineered Discharge (RED) and Project Better Outcomes for Older Adults through Safer Transitions (BOOST) (Jarrett et al., 2020; Josephson, 2016; Mansukhani et al., 2015; Mitchell et al., 2017). For instance, Project BOOST is an intervention that aims to improve transitional care. It supports clinicians with proven resources and expert mentoring for a better discharge transition process. It improves patient and family education, and facilitates information communication among inpatient and outpatient providers (Josephson, 2016; Mansukhani et al., 2015). Utilizing these interventions for high-risk HCBS patients based on the prediction model could be helpful to reduce future hospitalization risk and to otherwise improve care to this patient group. Yet as mentioned above, efforts to incorporate the characteristics of HCBS users in these strategies would be needed to optimize the benefits of these strategies and interventions.
Our study contains several limitations that must be considered in evaluating or applying our findings. First, this study is based on 2012 data due to substantial lags in MAX availability at the time of study initiation. We believe that the underlying relationships between HCBS use and hospitalization are unlikely to have changed significantly. Second, this study is based on claims data, which lack lab reports and other relevant clinical detail. For instance, people with long-term care generally require assistance with activities of daily living due to functional and/or cognitive limitations, well known to be risk factors for adverse health outcomes (Fuino et al., 2020; Greysen et al., 2015). Although more clinical detail could be helpful, we have no reason to believe that the lack of such data introduced bias into our results. Third, this study excludes beneficiaries in Medicare and/or Medicaid managed care, limiting the study results’ generalizability to FFS Medicare-Medicaid duals. Finally, our ML models should not be interpreted as identifying causality. While our study shows that we could utilize ML algorithms to identify HCBS users at high risk of hospitalization and highlights key characteristics related to future hospitalization, this study could not develop causal connections between identified risk factors and hospitalization. Future randomized trials and other study designs are essential to explore causal hypotheses concerning hospitalization and specific HCBS services.
This study supports the feasibility of using prediction models to identify those who are prone to adverse health outcomes early on among Medicaid HCBS users. Prediction models hold considerable promise to inform development of early interventions that enhance health and quality of life among HCBS users, and that are essential to support the policy movement from institutional care to HCBS in Medicaid long-term care.
Supplemental Material
Supplemental Material - Predicting Hospitalization among Medicaid Home- and Community-Based Services Users Using Machine Learning Methods
Supplementary Material for Predicting Hospitalization Among Medicaid Home- and Community-Based Services Users Using Machine Learning Methods by Daniel Jung, R Tamara Konetzka, and Harold A. Pollack in Journal of Applied Gerontology.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Institute on Aging, Grant/Award Number: RF1AG054071.
Ethical Approval
IRB: University of Chicago, Protocol Number: IRB17-0865.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
