Abstract
The Centers for Medicare and Medicaid Services (CMS) implemented hierarchical condition category (HCC) models in 2004 to adjust payments to Medicare Advantage (MA) plans to reflect enrollees’ expected health care costs. We use Verisk Health’s diagnostic cost group (DxCG) Medicare models, refined “descendants” of the same HCC framework with 189 comprehensive clinical categories available to CMS in 2004, to reveal 2 mispricing errors resulting from CMS’ implementation. One comes from ignoring all diagnostic information for “new enrollees” (those with less than 12 months of prior claims). Another comes from continuing to use the simplified models that were originally adopted in response to assertions from some capitated health plans that submitting the claims-like data that facilitate richer models was too burdensome. Even the main CMS model being used in 2014 recognizes only 79 condition categories, excluding many diagnoses and merging conditions with somewhat heterogeneous costs. Omitted conditions are typically lower cost or “vague” and not easily audited from simplified data submissions. In contrast, DxCG Medicare models use a comprehensive, 394-HCC classification system. Applying both models to Medicare’s 2010-2011 fee-for-service 5% sample, we find mispricing and lower predictive accuracy for the CMS implementation. For example, in 2010, 13% of beneficiaries had at least 1 higher cost DxCG-recognized condition but no CMS-recognized condition; their 2011 actual costs averaged US$6628, almost one-third more than the CMS model prediction. As MA plans must now supply encounter data, CMS should consider using more refined and comprehensive (DxCG-like) models.
Introduction
In 2013, the US Medicare program provided health insurance coverage to 52 million beneficiaries entitled by age greater than 64, disability, or end stage renal disease (ESRD). 1 Medicare spending accounted for 16% (US$536 billion) of the federal budget and is projected to double by 2023 due to increasing numbers of beneficiaries and costs per person.2,3
Medicare beneficiaries can enroll in a private sector option called Medicare Advantage (MA) rather than receive the traditional fee-for-service (FFS) benefit. In 2013, 28% of Medicare beneficiaries were enrolled in MA. 4 Historically, MA plan premiums were linked to FFS expenditures by geographic area, with payments set at 95% of an enrollee’s county’s adjusted average per capita cost. Adjustments to the county average were purely demographic, and explained very little variation in expenditures; in particular, MA plans were not paid more for enrolling sicker people. 5 Thus, the Centers for Medicare and Medicaid Services (CMS), which administers Medicare, sought a health-risk-based model for paying MA plans. It considered using self-reported status for risk adjustment, as well as several methods that extract diagnoses from medical claims data, including the Adjusted Clinical Groups (ACGs) system, 6 the Chronic Disease and Disability Payment System (CDPS), 7 Clinical Risk Groups (CRGs), 8 the clinically detailed risk information system for cost (CD-RISC), 9 and the diagnostic cost group/hierarchical condition categories (DCG/HCCs). 10 Kanika Kapur, a researcher at the RAND Corporation wrote, “CMS chose the DCG/HCC model for Medicare risk adjustment, largely on the basis of transparency, ease of modification, and good clinical coherence.” 11
The HCC models calculate payments to MA plans based on enrollee’s age, gender, and diagnoses. The HCC framework requires classifying all coded diagnoses into condition categories (CCs) and using hierarchies to eliminate redundant recognition of a single underlying medical problem. First implemented in 2004, the CMS-HCC models are periodically updated.
Verisk Health, a private for-profit health analytics firm, estimates and supports DxCG Medicare HCC models, originally relying on CMS’ 189 CCs. 10
For implementation, the CMS-HCC payment models have omitted and consolidated many CCs, now recognizing approximately 80 distinct conditions. In contrast, today’s DxCG Medicare (Version 7) models exploit the full detail of a comprehensive classification system with 394 HCCs.
In this article, we use DxCG Medicare models to illustrate and quantify both lack of available precision and mispricing—that is, differences between actual and CMS-model-predicted costs—in CMS payment model implementation. We hypothesized that CMS models would “underprice” people whose costly diagnoses were recognized only by the DxCG model and “overprice” those with no medical problems detected by either model.
HCC Models
Both CMS-HCC and DxCG Medicare risk adjustment models are linear regression models using demographic information (age, sex, Medicaid dual eligibility, and reasons for Medicare eligibility) as well as the profiles of major medical conditions in a base year to predict “costs” in the following, or target, year. Costs are payments for services covered by Medicare’s hospital insurance (part A) and supplementary medical insurance (part B) benefit. Medicare has yet another model for its ESRD program. MA CMS payments (but not DxCG predictions) also consider whether the beneficiary is institutionalized (eg, living in a nursing home) or “new,” that is, enrolled for less than 12 full months.
Both models first classify all (approximately 16 000) International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) diagnosis codes into CCs. Each CC contains clinically related groups of diagnoses, such as colon cancer and rectal cancer, with similar cost implications. Hierarchies are imposed so that a person is coded for only the most severe manifestation among related diseases (eg, someone with cystic fibrosis would not also be coded for either “chronic obstructive lung disease” or cough). This converts CCs into HCCs. Both models also include interactions between disease groups (eg, diabetes and congestive heart failure) and between diseases and disability status (eg, disability and congestive heart failure) so long as they make sense to clinicians and strongly predict additional costs. 12
Prior to adopting the HCC modeling framework, CMS explored and rejected using nonlinear models with interactions for all diseases. The overall R2s for such models were only slightly larger than the basic linear model. And although their predictions were more accurate for people with expected low costs, they mispriced people in categories defined by age, sex, and other variables. After substantial model testing, CMS decided to add selected interaction terms (eg, between 2 or more HCCs) to a linear model.
In contrast to linear models, nonlinear models are more cumbersome to estimate, more difficult to explain to stakeholders, and more strongly incentivize diagnostic upcoding (laying claim to both more, and more serious forms of, diseases than actually present) because of the large marginal increase in predicted expenditures for individuals with many diagnoses. Including interaction terms in a linear model also incentivizes upcoding, but the HCCs used in interactions can be restricted to medical conditions that are less subject to discretionary coding variations, and the effects of interactions are more transparent. 12 Despite their theoretical appeal—and undisputed advantages for hypothesis testing—more complex models can be inferior to linear models for calculating payments in samples exceeding a million observations.13-15 Econometrician Andrew Jones reviewed the evidence, concluding that “the simple linear model, estimated by OLS, performs quite well across all of the criteria.” 16
CMS-HCC Models
CMS implements distinct risk models for beneficiaries entitled by age, disability, or ESRD, and for community-residing versus long-term institutional (nursing home) enrollees. Unlike the CMS-HCC models for “continuing” MA members, the payment formula for “new” Medicare enrollees (enrolled for less than 12 months in the base year) uses no diagnostic information.
Prior to the Affordable Care Act (2010), MA plans were exempted from submitting encounter records. Although CMS calibrated its models on FFS data with full ICD-9-CM coding, risk-adjusted MA payments are calculated from a short list, submitted by the plan, of the CCs present for each person. Model recalibration for 2014 used 2010 100% FFS claims data to predict 2011 costs. 17
The original CMS-HCC payment models included 70 HCCs; even the 2014 CMS models include only 79 HCCs (87 HCCs for its ESRD models). Newly added HCCs are either previously unrecognized conditions among the 189 HCCs available or splits/collapses of previously included HCCs. 17
The simplified method for capturing information on a small number of well-reimbursed CCs makes upcoding both easy and profitable for MA plans. The US Government Accountability Office report of 2012 estimated that the more aggressive diagnostic coding in MA plans than in Medicare FFS caused as much as US$5.8 billion overpayments to MA plans in 2010, of which only US$2.7 billion was recouped by CMS’ adjustments for this difference. 18
DxCG Medicare Models
As described, DxCG Medicare models share the same basic HCC structure as CMS models, but consider up to 394 HCCs for prediction. Altogether 138 of the DxCG HCCs have 0 weight in the Medicare models, with the omitted set chosen using statistical criteria, clinical judgment, and practical considerations, balancing the desire for greater accuracy against the principle that difficult-to-verify distinctions in medical problems should not result in large distinctions in payment. Like the CMS models, the DxCG Medicare models exclude many low-cost, vague, and discretionary conditions—including hypertension and high cholesterol—to reduce opportunities for manipulating payments by aggressive upcoding. Regression coefficients were estimated using FFS claims data for beneficiaries with both parts A and B insurance in Medicare’s 2005-2006 5% sample. Unlike CMS’ implementation, prior-year diagnoses are used in all enrollees’ predictions, not just for continuing enrollees. Finally, DxCG modelers use second-stage regression splining to ensure that mean predictions closely approximate actual spending within each subgroup defined by age and gender across the spectrum of low-, middle-, and high-risk individuals. In this way, the models use nonlinear fine-tuning to stabilize and tailor outputs from a straightforward, underlying linear modeling structure.
The Data
The data pertain to 1.5 million enrollees from Medicare’s 2010-2011 FFS 5% sample: enrolled exclusively in FFS, present and eligible for parts A and B coverage for at least 1 month in each year, and not currently entitled to the ESRD program (see Table A1 of the appendix).
Medicare “allowed costs” are those covered by the combined parts A and B benefit. We summed these for each beneficiary, during all “eligible months” in 2011: that is, months of nonhospice, parts A and B enrollment in FFS Medicare. We annualized these sums by dividing by the fraction of the year that the person was eligible, and conducted all analyses using this fraction as a weight. Thus, data for a person with US$10 000 of costs over 6 eligible months is treated as ½ an observation at US$20 000 per year; this leads to correct monthly cost estimates for each beneficiary, including those who die. 19
Both DxCG and CMS models use 2010 data to predict eligibility-weighted, annualized costs in 2011. The only distinction is that CMS models are designed to predict “paid” costs whereas the DxCG models “allowed” amounts. The amount paid equals the amount allowed minus deductibles and copayments. In this article, and in our modeling, we use allowed amounts because they are more closely linked to actual resource use and less subject to variation in plan-level cost-sharing features. Nonetheless, annual paid and allowed amounts are highly correlated (in our sample, ρ = .998), and predictions evaluated with either outcome should perform similarly.
Both CMS and DxCG models generate relative risk scores (RRSs). Indeed, the CMS-HCC software automatically generates 3 RRSs for each person: new enrollee RRS, continuing enrollee (or “community model”) RRS, and institutional model RRS (usually for enrollees living in a nursing home). Users must choose the appropriate RRS for each enrollee: either new, or if continuing, whether community dwelling or institutionalized. We have not examined the institutional model. Instead, we evaluate the CMS model in 2 ways: first, following CMS’ approach of using the new enrollee model RRS for members enrolled for less than 12 months in 2010, and using the risk score from the community model for everyone else; second, we used the RRS from the community model for everyone. We will call the first method “CMS implemented” (as of 2014) and the second, “CMS improved.”
To generate predicted costs from any HCC model, risk scores must be converted by applying a multiplier—the payment associated with next year’s average cost—prior to knowing that cost. We ignore the forecasting error associated with that unknown, and level the playing field among models, by choosing multiplicative factors to make each model’s weighted mean predictions exactly match weighted mean actual (allowed) cost in the 2011 sample, separately for new and continuing enrollees.
All models in this study rely on the same basic HCC structure, use identical demographic predictors, and identify diagnoses from the same claims records. We call Verisk Health’s DxCG Medicare Model the “DxCG model”; its principal distinguishing characteristic is its reliance on a refined, comprehensive classification of up to 394 CCs.
Results
First we compared overall accuracy by examining each model’s R2 when using its “off-the-shelf” (unmodified, as fit to other data) RRSs to predict cost (cost = a + b × RRS) in Medicare 2010-2011 FFS 5% file data.
Table 1 examines and compares the performance of 3 such models: the CMS implemented and improved 2014 models, and the DxCG model. Each model was previously developed on Medicare FFS data: the CMS-HCC models on 2010-2011 100% files, and the DxCG model on 2005-2006 5% files; thus, each model uses just 1 degree of freedom to predict in this article’s “validation set,” and there is no concern about overfitting. We separately inspected 3 groups: the full population, new enrollees (those with less than 12 months of eligibility in the base year), and continuing (ie, non-new) enrollees, to explore the extent to which, by ignoring diagnostic information for new enrollees, CMS-HCC models lead to mispayments within this subgroup.
Off-the-Shelf R2 for Predicting Next Year’s Medicare Cost: CMS-HCC Versus DxCG Models.
Source. Medicare FFS 5% sample, present in both 2010 and 2011, excluding those with 2010 ESRD (N = 1 487 628). All models use 2010 information to predict 2011 Medicare cost.
Note. Each so-called “off-the-shelf” models have 1 degree of freedom; each regresses cost on a formula-based risk score: cost = a + b × (risk score). The CMS-HCC 2014 models were calibrated on 100% FFS 2010-2011 data; DxCG models were calibrated on the 2005-2006 Medicare FFS 5% sample. Both models predict next year’s costs from beneficiary age, sex, Medicaid dual eligibility, original reason for Medicare entitlement, and diagnoses from the previous year’s inpatient, outpatient, and carrier-file claims. CMS-HCC = Centers for Medicare and Medicaid Services hierarchical condition category; FFS = fee-for-service; ESRD = end stage renal disease; RRS = relative risk score.
“Implemented” means using the new enrollee model RRS for members enrolled for less than 12 months in 2010, and using the risk score from the community model for everyone else.
Improved means using the RRS from the community model for every enrollee.
DxCG, Version 7, Model 121.
The contrast between the CMS implemented and improved columns shows that the CMS could increase its predictive accuracy (R2) simply by applying its own community model to new enrollees. The R2 improvement within the new enrollee population itself is huge (from 2.0% to 17.2%), but because few members are new, this only increases the whole-population R2 from 13.8% to 14.2%. The whole-population R2 for the DxCG model is 16.5%.
Other performance measures involve comparing a model’s predicted payments for groups of people to their actual costs. For example, plans that enroll members with serious, high-cost-generating conditions should receive funds adequate to care for them; more generally, with a good model, most moderately large, prospectively identifiable subgroups will have similar predicted payments and actual costs. We will examine mispayment, that is, differences between mean model-predicted payments and actual costs, and compute overpayment percentages (predicted payment minus actual cost, divided by actual cost) for various subgroups and models.
Mispayments by Model-Predicted Risk Quantiles
We evaluated model discrimination by sorting the population into quantiles of increasing CMS-model-predicted cost and calculating mean (observed) Medicare cost and percent mispricing for quantile-based groups. Table 2 shows the actual year 2 costs by prediction quantiles from the CMS implemented model and associated overpayment percentages. Note that the model makes large distinctions among beneficiaries; average costs of those with the 1% highest predictions are nearly 20 times as much as for the bottom 20%. However, we also care about “calibration”—Do the plans pay correctly across the spectrum of expected costs? The last column of Table 2 shows the percent over- or underpayment within each subpopulation; CMS underpays both those in the top 5% and those in the bottom 20% of expected costs. For example, while those in the bottom 20% actually cost about US$4000, the CMS model would have paid out 12% less, only about US$3500.
Mean Medicare Cost and Mispricing by 2014 CMS-HCC Implemented Model-Predicted Percentile Groups.
Source. Medicare FFS 5% sample, present in both 2010 and 2011, excluding those with 2010 ESRD (N = 1 487 628). All models use 2010 information to predict 2011 Medicare cost.
Note. CMS-HCC = Centers for Medicare and Medicaid Services hierarchical condition category; FFS = fee-for-service; ESRD = end stage renal disease.
Using the “as implemented” algorithm—that is, ignoring all diagnoses for new enrollees.
Percentages are calculated as (predicted payment − actual cost) / actual cost. For example, −5 means that what the model expects (and what a payment system based on it would pay) is 5% less than the actual cost.
Mispayments by the Presence/Absence of Various Kinds of Diagnoses
We further examine means and mispricing separately for members who have and do not have any diagnoses recognized by the CMS classification system. Here we contrast mispricing under both the CMS implemented and DxCG models, for everyone and separately among new enrollees. As shown in the top half of Table 3 (and in Figure 1), both models allocate the correct total payment to the 66% of members with at least one clinical condition recognized by CMS’ implementation. For the remaining 34%, with no HCCs identified by CMS, there is substantial mispricing within subgroups. For members with no HCC in either system (7%), CMS overpays by 44% whereas the DxCG model gets the average right. The remaining 27% of members can be split into the 13% of members with at least one higher cost DxCG condition not recognized by CMS, and those with only lower cost conditions. The CMS model underpays the higher cost DxCG conditions by 25% and overpays those with only low-cost DxCG HCCs by 17%. Table 3 also examines the same information for new enrollees. Because CMS ignores diagnostic information for new enrollees, it underpays those with CMS conditions by 35%, and overpays the rest, especially the healthiest 25% of members costing less than US$4000 each, for whom it pays over US$8000. In contrast, the DxCG model’s expected costs are close to observed costs across these subgroups.
2011 Mean Costs, Model-Based Payments, and Percent Over- and Underpayments for Subgroups of People by Types of Conditions.
Source. Medicare FFS 5% sample, present in both 2010 and 2011, excluding those with 2010 ESRD (N = 1 487 628). Both models use 2010 information to predict 2011 Medicare cost. The CMS model uses its 2014 update calibrated on 2010-2011 data; the DxCG model, Version 7, was calibrated on 2005-2006 data.
Note. CMS = Centers for Medicare and Medicaid Services; HCC = hierarchical condition category; FFS = fee-for-service; ESRD = end stage renal disease.
Error is calculated as (payment − cost) / cost. For example, −6% means that what the model expects (and what a payment system based on it would pay) is 6% less than the actual cost.
The conditions with the highest 100 coefficients in the DxCG model from the subgroup after excluding people with any conditions classified in the CMS-HCC.
All DxCG-HCC conditions not previously classified.
Percent overpayment (underpayment) by the presence of HCC type: CMS versus DxCG models.
Discussion and Conclusions
We examined CMS and DxCG Medicare models in Medicare FFS data, finding 2 changes that Medicare could implement to predict more accurately. These are as follows: use whatever diagnoses are present to distinguish among “new” enrollees with less than 12 months of base year data, and adopt a more refined and comprehensive predictive tool, such as Verisk Health’s DxCG model. The first change requires only an administrative decision to implement. By ignoring differences in diagnosed disease among members enrolled for less than 12 months, CMS-HCC’s current implementation creates incentives for MA plans to enroll healthier Medicare enrollees and “penalties” (at least in the first year) for plans enrolling people who are already sick. The second change takes more work. The CMS-HCC model seeks to include CCs based on their association with next year’s costs for Medicare parts A and B benefits. CCs with small coefficients, low t-values, so few beneficiaries that the coefficient is unstable, or composed of poorly specified diagnostic codes are excluded, from both CMS 10,12 and DxCG models. However, in 2004, CMS had an additional, political, reason to limit the number of categories recognized in its initial model: Many managed care organizations (MCOs) balked at supplying the detailed diagnostic information from encounter records, which automatically populate models recognizing all diseases that drive costs. Thus, the CMS model dropped 88 of the 189 existing HCCs in its payment model and merged others; plans were only asked to certify annually the presence/absence of each of approximately 70 to 80 remaining medical conditions for each of its enrolled beneficiaries. The reduction in explanatory power (as compared with a fuller model) was viewed as “acceptably small,” in the context of the MCO industry’s assertions that existing, long-term subcapitation contracts meant that they could not supply full encounter data. This compromise led to CMS adopting a simpler, easier to implement, but more easily manipulated, less easily audited, and—as we confirm here—notably less accurate, model.
As shown in this article, the DxCG Medicare model, despite being developed on an earlier, smaller data set (Medicare 2005-2006 5% FFS) than CMS’ current model (developed on Medicare 2010-2011 100% FFS), predicts costs more accurately due to its more refined and granular HCCs. More accurate predictions help reduce incentives for selection, improve payment fairness for the included rarer, high-cost conditions, and reduce financial risk for MA plans.
Now, with about 30% of Medicare beneficiaries enrolled in MA, with physicians coding with more specificity, and with Affordable Care Act (ACA) requiring that MA plans submit encounter data that include all diagnoses, additional research is warranted to explore the sensitivity of simplified versus refined, comprehensive models to aggressive, hard-to-audit upcoding. Reduced sensitivity to upcoding may require further refinements. Addressing behavioral responses to risk adjustment, as many researchers have discussed, is another area for further research.20,21 Our suggested improvements to the current CMS-HCC models corrects some, although likely not all, of the troublesome payment and incentives problems related to under- and overprediction of costs for large groups of prospectively identifiable people. Given that models similar to the CMS-HCCs for MA are also used for part D, for health insurance exchanges and for diverse research evaluations, improving the classification and modeling approaches seems especially worthwhile.
Footnotes
Appendix
Characteristics of the 2010-2011 Medicare FFS, Non-ESRD a 5% Sample.
| Mean | SD | |
|---|---|---|
| Annualized 2010 Medicare cost | $10 153 | 22 907 |
| Annualized 2011 Medicare cost | $11 943 | 29 453 |
| Age in 2010 | 71.4 | 12.6 |
| N | % | |
| Aged 65+ on December 31, 2010 | 1 229 140 | 82.6 |
| Female | 831 378 | 55.9 |
| Continuing (enrolled for 12 months in 2010) b | 1 418 862 | 95.4 |
Source. Medicare FFS 5% sample, present in both 2010 and 2011, excluding those with 2010 ESRD (N = 1 487 628).
Note. FFS = fee-for-service; ESRD = end stage renal disease.
Even after removing members with ESRD as their current reason for entitlement in 2010, the study sample still contains 10 428 members with an ESRD diagnosis in 2010, probably those newly diagnosed with ESRD who are not yet eligible for this program, or with diagnoses or renal disease durations that do not meet ESRD program eligibility criteria.
“New” members, enrolled for <12 months, account for ~0.4% for each number of months of eligibility, from 1 to 11.
Acknowledgements
We thank Ron Russell, Matt Siegel, Deb Bradley, Jo Anne Lutz, Tim Layton, and Wenjia Zhu for their comments, and Timur Turkdogan and Punam Mahajan for preparing the data for this analysis.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Chen is a former research scientist at Verisk Health, Inc. Ash and Ellis are senior scientists at Verisk Health, Inc., where they consult on developing health-based predictive models; neither has any ownership interest in Verisk Health, Inc. Toro is a former director of product management at Verisk Health, Inc.
Funding
Ash was partially supported by NIH grants U01HL05268 and UL1TR000161. All four authors received financial support from Verisk Health for researching and writing this paper, in the form of salary support (Chen and Toro), consulting fees (Ash), and a research contract to Boston University (Ellis).