Did the Hospital Readmissions Reduction Program Reduce Readmissions? An Assessment of Prior Evidence and New Estimates

Abstract

In this article, we provide a comprehensive, empirical assessment of the hypothesis that the Hospital Readmissions Reduction Program (HRRP) affected hospital readmissions. In doing so, we provide evidence as to the validity of prior empirical approaches used to evaluate the HRRP and we present results from a previously unused approach to study this research question—a regression-kink design. Results of our analysis document that the empirical approaches used in most prior research assessing the efficacy of the HRRP often lack internal validity. Therefore, results from these studies may not be informative about the causal consequences of the HRRP. Results from our regression-kink analysis, which we validate, suggest that the HRRP had little effect on hospital readmissions. This finding contrasts with the results of most prior studies, which report that the HRRP significantly reduced readmissions. Our finding is consistent with conceptual considerations related to the assumptions underlying HRRP penalty: in particular, the difficulty of identifying preventable readmissions, the highly imperfect risk adjustment that affects the penalty determination, and the absence of proven tools to reduce readmissions.

Keywords

pay-for-performance hospital readmissions hospital quality

Introduction

The fee-for-service payment approach of traditional Medicare has long been criticized for its inherent incentive to provide too much, low-valued care, and there is a widespread belief that this problem can be solved by linking provider payments to performance (outcomes) instead of the quantity of care (Cromwell et al., 2011; Institute of Medicine, 2007; Mendelson et al., 2017). It was this logic that led to the inclusion of several pay-for-performance initiatives in the 2010 Patient Protection and Affordable Care Act (ACA). One of the most prominent of the ACA pay-for-performance initiatives was the Hospital Readmissions Reduction Program (HRRP). The goal of the HRRP was to reduce avoidable hospital readmissions that were thought to be substantial, for example, 27% of all admissions (Auerbach et al., 2016; Van Walraven et al., 2011)—and costly—$17 billion in 2016 (Bailey et al., 2019).

Given the visibility of the ACA and the wide scope of the HRRP, which applies to over 3000 hospitals, there has been substantial interest in evaluating the consequences of the HRRP. Prior studies have assessed whether the HRRP has reduced readmissions and affected other outcomes (e.g., mortality). However, many of these studies are single, or two, state analyses (e.g., Carey & Lin, 2015; Chen & Grabowski, 2019; Demiralp et al., 2018; Zingmond et al., 2018; Mellor et al., 2017);¹ many do not identify the impact of being penalized, but instead analyze differences between hospitals that differ along a dimension likely to be correlated with being penalized (e.g., McGarry et al., 2016; Chen & Grabowski, 2019; Ibrahim et al., 2018); and a few studies including some of the most well-cited national analyses use an interrupted time-series approach that has acknowledged limitations with respect to identifying causal relationships (e.g., Zuckerman et al., 2016; Chahabra et al., 2019). In addition, the prior literature has not produced a consensus finding. While it is clear that hospital readmissions have declined over the last decade, it is uncertain whether the HRRP had led to this decline (Lisk & Stensland, 2018).² Therefore, it remains largely unknown whether the HRRP achieved its intended goal and, more generally, whether the type of pay-for-performance mechanism embedded in the HRRP may be a productive way to improve the efficiency of health care.

In this article, we provide a comprehensive, empirical assessment of the hypothesis that the HRRP affected hospital readmissions. In doing so, we provide evidence as to the validity of prior empirical approaches used to evaluate the HRRP, and we present results from a previously unused approach to study this research question—a regression-kink design. We also provide a discussion of conceptual issues related to how hospitals may respond to the HRRP, which has largely been missing from previous studies of the HRRP. The conceptual framework suggests that the effect of the HRRP, if any, is likely to be small and to differ by the cause of hospitalization (and for readmission after that hospitalization) and by hospital characteristics, for example, the share of patients in penalized conditions of total Medicare patients.

Results of our analysis document that the empirical approaches used in most prior research assessing the efficacy of the HRRP lack internal validity. Therefore, results from these studies may not be informative about the causal consequences of the HRRP. Results from our novel, regression-kink approach, which we validate, suggest that the HRRP had little effect on hospital readmissions. This finding contrasts with the conclusions of most prior studies, which report that the HRRP reduced readmissions. However, our finding is consistent with conceptual considerations related to the assumptions underlying the HRRP penalty: in particular, the difficulty of identifying preventable readmissions, the highly imperfect risk adjustment that affects the penalty determination, and the absence of proven tools to reduce readmissions.

Our article contributes to the literature on pay-for-performance incentives in health care settings (Doran et al., 2016; Mendelson et al., 2017). The HRRP is a particularly important pay-for-performance program because of its prominence and wide coverage, and because the financial incentive is a penalty, which differs from the more common approach of using bonuses to reward performance (Cromwell et al., 2011). There is also a dynamic aspect to the incentive structure of the HRRP that is unique. Hospitals compete against each other to be in the lower half of the distribution of (adjusted) readmissions, and hospitals have potentially different ways (actions) of competing. For example, an obvious and hoped for response is for a hospital to improve the quality of care it provides. The fact that we find little effect of the HRRP on performance (i.e., readmissions) suggests that this type of pay-for-performance incentive may not be efficacious or that the implementation details of such a performance incentive need to be carefully considered.

Our article also contributes modestly to applied research in causal methods. We develop a novel test of the validity of a difference-in-differences-in-differences research design that is easily applied and straightforward to assess. We show that the test is consistent with an event-study specification that is typically used to assess difference-in-differences models. However, the test we propose is more parsimonious, which is its main strength.

Conceptual Considerations

As noted, most prior studies of the HRRP are empirical assessments, which is reasonable given that ultimately it is the consequences of the HRRP that are important in terms of assessing the efficacy of the program. However, it is helpful to consider some of the conceptual issues underlying the HRRP because doing so helps distinguish between consequences of the HRRP that are more or less scientifically plausible and because a conceptual model can highlight empirical predictions that can be assessed.

Structure of HRRP Incentive

Before discussing some of the conceptual issues, it is helpful to review the structure of the HRRP. The HRRP imposes financial penalties on hospitals with excess readmissions of patients admitted for one of several conditions (e.g., heart failure).³ Excess readmissions are defined as having more than the average (expected) number of readmissions for a condition after adjusting for patient characteristics (comorbidities) and, more recently, patient socioeconomic status (dual Medicaid-Medicare eligibility).⁴ A hospital is penalized if it has excess readmissions for any of the specified HRRP conditions and the penalty increases with the number of excess readmissions and/or the number of conditions for which the hospital has excess readmissions. An important feature of the HRRP is that the financial penalty imposed applies to all Medicare payments and not just payments for targeted conditions. The penalty is substantial—currently up to 3% of Medicare payments.⁵ In 2017 and 2018, the average penalty was over $500 million for penalized hospitals.⁶

Will a Hospital Respond?

Given the structure of the HRRP incentive, it is reasonable to assume that hospitals will respond by assessing whether the cost of responding, for example, taking actions to reduce readmissions, is less than the benefit of responding, which is the savings from not being penalized. This simple cost-benefit calculus has several implications about hospital behavior that we now discuss.

Perhaps the most important implication is that hospitals will only respond to the HRRP if in fact they can identify and prevent readmissions. Identifying a potentially preventable readmission is inherently a subjective exercise, and while there are undoubtedly some readmissions that are preventable, the share of readmissions classified as such (determined ex post) is uncertain according to two systematic reviews (Vest et al., 2010; van Walraven et al., 2011) and small (11%) according to MEDPAC (2013). Even if the share of preventable hospitalizations is sizable and larger than calculated by MEDPAC (2013), it is still necessary to be able to identify these cases ex ante to respond in an efficient manner. However, common approaches to identifying potentially preventable readmissions ex ante are often not very accurate (Alimadadi et al., 2020; Kansagara et al., 2011). The uncertainty over the share of readmissions that are preventable and the difficulty of identifying them ex ante suggests that hospitals have limited ability to target admissions amenable to interventions to reduce potentially preventable readmissions. So, the cost of reducing a readmission includes the resources spent on admissions that are in fact not preventable, which are the large majority of admissions. The higher the cost of reducing preventable readmissions, the less likely it is that a hospital will expend the resources to avoid a HRRP penalty.

In addition to being difficult to identify a preventable readmission, there is, at best, mixed evidence of demonstrably effective interventions to reduce readmissions, particularly interventions that can be effectively implemented by the hospital to influence readmissions occurring after 7 days post-discharge Graham et al., 2018). These delayed readmissions are the least responsive to the quality of care of the inpatient stay directly controlled by the hospital and depend on patient behavior and outpatient care.

Some quotes from recent reviews make the point about the lack of effective interventions.⁷

"Of interest (and concern), the authors found no consistent evidence from RCTs that any one intervention by itself significantly reduced hospital readmission. Of the 16 RCTs, only 5 yielded significant reductions in hospital readmission.” (Kripalani et al., 2014, p. 3)

“Burke and colleagues recently performed an updated systematic review… They included 61 interventions, 42 of which have been studied in RCTs. ... Just under half (47.5%) of interventions demonstrated a statistically significant reduction in readmissions. Consistent with prior reviews, no singular intervention component significantly reduced readmissions, though a trend was present for patient education and engaging social and community supports (p = 0.06 for each).” (Kripalani et al., 2014, p. 5)

"Controlled trials^6-13 and cross-sectional studies^14-16 have identified different practices that may be effective, such as patient education, discharge planning, and telephone follow-up; however, no specific clinical practice has been shown to consistently reduce readmissions.⁵ Furthermore, longitudinal studies have failed to link specific hospital practices with subsequent reduction in risk-standardized readmission rates (RSRRs), although employment of a greater number of these practices seems to be helpful.” (Brewster et al., 2016, p. 600)

To recap, prior evidence has demonstrated that it is difficult to identify and target preventable readmissions. This implies that to reduce readmission, a hospital would have to take actions, and presumably spend resources, on many admissions that are not preventable raising the cost of responding. Moreover, there is limited evidence that there are effective actions that a hospital can take to reduce readmissions—even if it could identify them. These considerations suggest a limited response by hospitals to the HRRP and, therefore, a small if there is any change in readmissions as a result of the HRRP.

Another aspect of the HRRP that may limit the ability of a hospital to respond is the 30-day period that is used to define a readmission. A study by Graham et al. (2018) reported that readmissions within 7 days of discharge are more likely to be amenable to changes in hospital practices than admissions after this period that are more likely due to the care received from other providers. Chin et al. (2016) came to a similar conclusion. However, most readmissions (two-thirds) during the 30-day widow occur after 7 days (Fingar et al., 2017).⁸ This evidence, again, suggests that it may be very costly for the hospital to respond to the HRRP because any action the hospital may take will, at best, affect only a portion of the readmissions that will be attributed to it.

A third aspect of the HRRP that may limit the ability of hospitals to respond is that the definition of excess readmissions is based on an extensive, but still limited set of observable comorbidities of patients. Hospitals with patients that are more (less) likely to be readmitted for unmeasured reasons will be more (less) likely to be penalized even if they are providing optimal care. This feature of the HRRP has been one of the most heavily criticized (American Hospital Association, 2015; Joynt et al., 2016; Soumerai & Sullivan, 2019). Even the recent addition of the share of patients that are dual eligible for Medicaid and Medicare (i.e., crude socioeconomic indicator) to the calculation of adjusted readmissions is only a partial solution to the problem of unmeasured confounders. Indeed, the models used by the Centers for Medicare and Medicaid Services (CMS) to adjust readmissions for comorbidities and that is intended to put hospitals on an equal footing for comparison (and penalty) purposes explain relatively little of the total variance in readmissions for some conditions (Krumholz et al., 2008). The inability of the CMS models to explain variation in readmissions across hospitals using patient characteristics is consistent with the studies noted earlier about the inability of hospitals to identify potentially preventable readmissions.

The implication of this measurement issue is much of the variation in readmissions, and therefore, the determination of the HRRP penalty is likely due to omitted differences in patient-level factors. If so, then hospitals have little incentive to change practices because it will have little effect on the probability of being penalized.⁹ Evidence that the HRRP consistently penalizes the same hospitals is provided by Thompson et al. (2017) who report that 53% (65%) of hospitals received penalties in every year (four) of the first five years of the HRRP program and 11% (15%) of hospitals were never penalized (or penalized just once). Overall, it is likely that only about 20% of hospitals face much uncertainty over whether or not they will be penalized by the HRRP. Most hospitals know that they will or will not be penalized, and it is largely because of unmeasured patient risk factors and not the standard of care.

Finally, there is the dynamic aspect of the HRRP that may influence hospital behavior. Assuming, perhaps heroically, hospitals have some ability to influence readmissions in a cost-effective way and that the calculation of excess readmissions effectively puts hospitals on an equal footing in terms of patient severity, then the fact that every year 50% of hospitals are penalized for each condition makes the response of one hospital depend on the responses of other hospitals. Consider a simple case where there are just two hospitals. One hospital is providing optimal (less substandard) care, and the other is providing substandard care. The substandard hospital will be penalized because it will have more (adjusted) readmissions. If it responds by providing the optimal (less substandard) level of care, then in the next HRRP period, it will be random which of the two hospitals will be penalized.

This simple logic implies that, ideally, if the HRRP was effective, over time the distribution of (adjusted) readmissions would narrow (coalescing around an optimal level of care), the mean number of readmissions would be decreasing, and the probability of being penalized would become increasingly random. Evidence, however, does not suggest this has occurred. As noted, the probability of being penalized is highly persistent even after 5 years of the program (Thompson et al., 2017) and the distribution of adjusted readmissions has not narrowed over time. It is true that the mean of readmissions has been declining, but this is true for all types of readmissions and not just those targeted by the HRRP (Lisk & Stensland, 2018).

Condition-Specific Responses

While there are reasons to expect that the HRRP may have a limited effect on hospital behavior and readmissions, if hospitals do respond to the HRRP, it is unlikely that the response will be the same across HRRP conditions or across hospitals. First, the ability to identify, ex ante, potentially preventable readmissions differs by condition (Chamberlain et al., 2018; Siracuse & Chamberlain, 2016; Smith et al., 2018; Weinreich et al., 2016). Thus, conditions for which potentially preventable readmissions can be most easily identified are the ones that are expected to have the largest change. For example, there is some evidence that readmissions are more easily predicted ex ante for heart failure patients than pneumonia and AMI patients. Therefore, all else equal, hospitals would be more likely to respond to HRRP by altering practices related to heart failure, but the all else equal caveat is not to be taken lightly. There still needs to be an effective intervention even if patients likely to be readmitted can be reasonably well identified.

Second, there may be significant differences in the cost of effective interventions across conditions. Hospitals may have to use different amounts of resources to reduce readmissions for one condition (e.g., AMI) than for another condition (e.g., pneumonia). Third, even if the marginal costs of intervention are the same across conditions, for example, because a common post-discharge protocol is used, the cost of the interventions to reduce readmissions may differ because of the relative number of patients with a condition. Consider two hospitals that were penalized only because they had excess readmissions for AMI patients. In one hospital, AMI patients represent 10% of all Medicare admissions and in the other hospital AMI patients represent 5% of all Medicare admissions. The latter hospital can avoid the penalty by spending fewer additional resources to prevent readmissions of AMI patients than the former hospital. Therefore, the hospital with relatively fewer AMI admissions will be more likely to respond; every dollar invested by the hospital with relatively fewer AMI patients to reduce AMI readmissions will bring a greater benefit (reduced penalty on all Medicare admissions) compared to a hospital with relatively more AMI admissions.

The upshot of this discussion is that it is unlikely that the HRRP will have similar effects on each of the targeted conditions. Differences in the ability to detect and the cost to prevent avoidable readmissions are very likely to differ by condition. Of course, there may be a set of circumstances that would lead to this result (similar effects), but it seems improbable.

An Empirical Assessment of the Evidence of Research Designs Used in Prior Empirical Studies

There is a substantial body of research assessing the effects of the HRRP. Prior studies use four research designs: interrupted time series; difference-in-differences (DD); difference-in-differences-in-differences (DDD); and instrumental variables (IV). We assess the validity of each approach. Data for all of the analyses comes primarily from the 100% sample of Medicare administrative inpatient records reported in the MEDPAR files from 2006 to 2014. This period covers the periods, or much of the periods, analyzed by most prior studies. From here onward, we refer to the data year 2006–2007 as 2007, and so on. In several of our analyses, we use the baseline 2007 (2006–2007) readmissions and utilize 2008 (2007–2008) to 2014 (2013–2014) in our regression models. The MEDPAR files contain detailed information on all inpatient episodes of care for fee-for-service Medicare enrollees. The outcome we examine is 30-day readmissions. In addition, we use the Inpatient Prospective Payment System (IPPS) files that provide information about what hospitals were penalized and the excess readmission ratios for each HRRP targeted condition that are used to determine the penalty. We also extracted various hospital level covariates from the IPPS files including the number of beds and the hospital’s teaching status.

Our sample is limited to hospitals that were assessed and not exempt from the penalty by CMS. A total of 3128 hospitals were assessed for the HRRP penalties. This represents a major share of hospitals¹⁰. We focus on the first-round penalty, as has much of the prior research. In the first round, 2241 hospitals received penalties, for at least one condition. Of those 2241 hospitals, 1910 received a penalty below the maximum 1 percent cap. Another 887 hospitals received no penalties for any condition.

The Validity of the DD Design and Prior Evidence Using It

Several studies have used a DD design, or the closely related interrupted time series, to study the effect of the HRRP (e.g., Desai et al., 2016; Wasfy et al., 2017; Zuckerman et al., 2016). The basic DD design is well known. In the context of the HRRP, the DD approach compares changes in readmissions (other outcome) before and after the HRRP of hospitals that were penalized, or more likely to be penalized, for example, because of higher baseline admissions (Demiralp et al., 2018), to changes in readmissions before and after the HRRP of hospitals that were not penalized, or less likely to be penalized. The key assumption of the DD design used in these prior studies is parallel trends—that in the absence of the HRRP, the time-series pattern of readmissions would be the same in hospitals that were and were not penalized, or hospitals that were more or less likely to be penalized. We assess the validity of this critical assumption.

The regression model analog of the DD approach is

\begin{array}{l} R E A D M I T_{j t} = a_{j} + δ_{t} + X_{j t} Γ + β P E N A L T Y_{j t} + e_{j t} \\ j = 1, ..., N \\ t = 2008, ..., 2014 \end{array}

(1)

In equation (1), j in an index of hospitals and t is an index of years. The model indicates that the share of admissions readmitted within 30 days (READMIT) in hospital j and year t depends on hospital fixed effects (α_j), year fixed effects (δ_t), demographic characteristics (age, race/ethnicity, gender) of patients (X) in hospital j in year t, and an indicator for whether the hospital was penalized by the HRRP (PENALTY). The demographic characteristics, which vary over time for each hospital, are indicators for age under 65, age 65–70, age 70–75, age 80–85, age over 85, share Hispanic, share Black, share White, share other race, and share female.

For studies that used proxies for being penalized (e.g., Demiralp et al., 2018; McGarry et al., 2016), the variable PENALTY is replaced with an indicator for having that proxy (e.g., high-baseline readmissions). These studies estimate differences in outcomes for hospitals that have different amount of exposure and probability of being penalized and not differences between hospitals that were and were not treated, or what may be referred to as the treatment-on-treatment effect (i.e., effect of penalty on readmissions). It is necessary to note that the DD model assumes that unpenalized hospitals do not respond. However, this may not be the case, as we noted in the discussion of the conceptual model. If unpenalized hospitals are responding, then the DD model yields an estimate of the effect of the HRRP for penalized hospitals versus unpenalized hospital and not the total effect of the HRRP. If so, then the estimate will be smaller than the true effect.

The first year that the HRRP penalty was imposed was 2012 (officially October 1, 2011 was the start of penalty period, but we date the start to August 1, 2011 when it was announced which hospitals would be penalized).¹¹ So, the variable PENALTY is zero prior to 2012 for all hospitals and equal to one in 2012 and subsequent years for hospitals that were penalized.

We estimate equation (1) for all targeted conditions, all non-targeted conditions and for each of the three conditions targeted at the beginning of the HRRP: AMI, heart failure (HF), and pneumonia (PN). Estimates are obtained using weighted least squares where the weight is the number of patients admitted to the hospital in a year. Standard errors are constructed using a robust-cluster approach that accounts for potential non-independence of observations within hospital.

We also estimate an event-study specification of equation (1) that allows there to be separate year effects for penalized and unpenalized hospitals. The regression model for this analysis is

R E A D M I T_{j t} = a_{j} + δ_{t} + X_{j t} Γ + \sum_{t = 2008}^{2014} β_{t} P E N A L T Y_{j t} + e_{j t}

(2)

Equation (2) is identical to equation (1) except that we allow the difference between penalized an unpenalized hospitals to differ by year. Estimation proceeds as described above (weighted least squares and robust-cluster standard errors).

Figure 1 describes trends in 30-day readmissions by the HRRP penalty status in round 1 for the years 2006–2007 to 2013–2014 for each of the targeted conditions and for all targeted conditions aggregated. Across AMI, heart failure, and pneumonia, the figure shows that readmissions were decreasing among hospitals penalized and unpenalized prior to the HRRP. Additionally, there were diverging trends between the two groups—the unpenalized group in some years decreased readmissions by more than the penalized group. In Figure 2, we show trends in readmission rates before and after the HRRP, by tertiles of baseline readmissions in 2006–2007, which has been used to classify hospitals more or less likely to be penalized. Here too, there is evidence of divergence in trends prior to the actual HRRP period. These simple time-series graphs highlight the parallel trend assumption and suggest that it may be violated. We turn next to regression estimates.

Figure 1.

Medicare hospital 30-day readmission rates for AMI, heart failure, pneumonia and all targeted conditions between July 2006 and July 2014 by the condition-specific penalty status in the first year of the HRRP.

Figure 2.

Medicare hospital 30-day readmission rates for AMI, heart failure, pneumonia, and all targeted conditions between July 2007 and July 2014 by the baseline condition-specific readmissions tertile in year 2006–2007.

Table 1 report estimates of equation (1). We show results for each of the three HRRP conditions that were targeted in the first round of the HRRP program—AMI, heart failure, and pneumonia—and for all targeted and all non-targeted conditions. We show estimates from models that use an indicator of whether a hospitalized was penalized for a specific condition (i.e., excess readmissions for that condition above average), which we refer to as condition-specific penalty, and for an indicator that the hospital was penalized for any (or all) conditions, which we refer to as hospital-wide penalty. We show simple DD estimates in the top row and estimates from an event-study specification in the bottom part of the table.

Table 1.

Difference-in-differences estimates of the effect of HRRP penalty on readmissions, by HRRP condition.

	Condition-specific penalty			Hospital-wide penalty
	AMI	HF	PN	AMI	HF	PN	All targeted	All non-targeted
Penalty	−0.017***	−0.018***	−0.017***	−0.008***	−0.009***	−0.008***	−0.010***	−0.003***
	(0.002)	(0.001)	(0.001)	(0.002)	(0.001)	(0.001)	(0.001)	(0.000)
Event-study estimates
Penalty X 2008	−0.009** (0.003)	−0.004* (0.002)	−0.015*** (0.002)	0.001 (0.004)	−0.001 (0.002)	−0.011*** (0.002)	−0.003* (0.002)	−0.000 (0.001)
Penalty X 2009	0.008** (0.003)	0.007*** (0.002)	0.005* (0.002)	0.007* (0.003)	0.003 (0.002)	−0.003 (0.002)	0.002 (0.002)	0.001 (0.001)
Penalty X 2010	0.006 (0.004)	0.008*** (0.002)	0.003 (0.002)	0.009* (0.004)	0.004* (0.002)	0.000 (0.002)	0.003* (0.002)	0.001 (0.000)
Penalty X 2012	−0.014*** (0.003)	−0.013*** (0.002)	−0.018*** (0.002)	−0.003 (0.004)	−0.005* (0.002)	−0.011*** (0.002)	−0.007*** (0.001)	−0.001* (0.000)
Penalty X 2013	−0.016*** (0.004)	−0.016*** (0.002)	−0.020*** (0.002)	−0.007 (0.004)	−0.009*** (0.002)	−0.012*** (0.002)	−0.010*** (0.002)	−0.003*** (0.001)
Penalty X 2014	−0.016*** (0.004)	−0.019*** (0.002)	−0.020*** (0.002)	−0.003 (0.004)	−0.010*** (0.002)	−0.013*** (0.002)	−0.011*** (0.002)	−0.004*** (0.001)
Mean dep. variable	0.182	0.215	0.162	0.182	0.215	0.162	0.195	0.181
Number of obs.	15,203	21,401	21,508	19,147	22,105	22,245	22,279	22,279

Notes: The unit of observation is the hospital-year level, and the dependent variable is 30-day readmissions. The regressions include patients admitted for each relevant condition between July 2008 and July 2014. AMI = acute myocardial infarction; HF = heart failure; and PN = pneumonia. Columns 1–3 present DD and event study estimates using an indicator for being penalized for the specific condition. Columns 4–8 present DD and event study estimates using an indicator of a hospital-wide instead of the condition-specific penalty. All regressions include hospital fixed effects, year fixed effects, and a set of controls for patient characteristics that vary over time for each hospital (indicators for age under 65, age 65–70, age 70–75, age 80–85, age over 85, share Hispanic, share Black, share White, share other race, share female). Robust standard errors are constructed allowing for non-independence at the hospital level. Each regression is weighted by the number of patients admitted for the relevant condition (AMI, HF, and PN) in the hospital year. * 0.05 < p-value ≤0.10, ** p ≤ 0.05, *** p ≤ 0.01.

There are a few notable points revealed by estimates in Table 1. First, as indicated by the simple DD estimates in the top row, hospitals that were penalized by the HRRP, either measured by condition-specific or hospital-wide penalty, had significantly lower readmissions post-penalty than hospitals that were not penalized. The declines were on the order of 5% (hospital-wide penalty) to 10% (condition-specific penalty). Second, the declines in readmissions were very similar across targeted conditions, and there was a significant, although smaller (2%) decline in readmissions for non-targeted conditions. Third, the magnitude of the decline in readmissions was approximately 100% larger when the condition-specific penalty was used instead of the hospital-wide penalty. The larger effects associated with the condition-specific penalty suggests that hospitals respond largely on a condition-specific basis in response to a condition-specific penalty.

How do these results fit with the conceptual issues raised earlier? Consider the magnitudes of the DD estimates. Is it plausible that a hospital could decrease readmissions by 10%? If we assume that 27% of readmissions are preventable (e.g., van Walraven et al., 2011), a 10% decline in readmissions suggests that a penalized hospital was able to intervene and prevent 37% of the preventable readmissions. This is a relatively large impact, perhaps too large given the weak evidence related to effective interventions to reduce readmissions and the other considerations described earlier that suggest that HRRP is likely to have small effects. It is also somewhat surprising that readmissions declined by approximately the same magnitude across all three targeted conditions. Given that the ability to identify and effectively intervene to prevent a readmission likely differs across conditions, as does the likely cost of intervention, we do not expect this to be the case.

The event-study estimates suggest the answer to our skepticism regarding these DD estimates. As indicated by estimates of the effect of the condition-specific penalty, it is clear that trends in readmissions between penalized and unpenalized hospitals across all three conditions were diverging from 2008 onward, which is prior to passage of the ACA in 2010 and prior to any hospital actually receiving a penalty. The fact that the divergence occurred before the ACA and since 2008 casts doubt on the assumption that the decline in hospital readmissions that occurred in what is often referred to as the “anticipation” period between passage of ACA (i.e., 2010) and the date when penalties were actually imposed and highlighted by previous studies (e.g., Zuckerman et al., 2016) is due to the HRRP. In addition, Ody et al. (2019) show that much of the post-2011 decline in risk-adjusted readmissions was due to a change in electronic coding of comorbidities and not due to the HRRP. When the hospital-wide penalty is used, there is less, although still substantial, evidence of divergent trends in readmissions between penalized and unpenalized hospitals. Note that in these models, approximately 80% of hospitals were penalized and the comparison is between hospitals that were penalized for any or all conditions and hospitals that were not penalized at all.

In sum, the DD approach for studying the effect of the HRRP on readmissions and other outcomes does not appear to be valid because the underlying assumption of parallel trends between penalized and non-penalized hospitals does not seem to be applicable, at least in studies using national data (as opposed to single-state studies) such as Desai et al. (2016) and Wasfy et al. (2017). Indeed, evidence of divergent trends was reported in Desai et al., 2016) but not highlighted. Trends in readmissions between penalized and non-penalized hospitals were diverging prior to 2010 when the HRRP was announced as part of the ACA and prior to when the first HRRP penalty was imposed in 2012. The magnitude of the divergence in trends prior to the HRRP was often as large as the post-HRRP divergence, and while the direction of the diverging trends was not always the same, it is the fact that there is divergence that is critical and not the direction of the divergence. It is not appropriate to use the ex-post results from a likely flawed research design to infer the true estimate because the counterfactual outcome remains unknown. While it is clear that readmissions were falling over the period of analysis, they were falling for targeted and non-targeted conditions and falling at a differential rate for hospitals that were and were not penalized up to 3 years before the policy became effective. The article by Ody et al. (2019) proposed one explanation for this uniform decline post-2011 in adjusted readmissions—a change in electronic coding of transactions. Based on these findings, estimates of the effect of the HRRP from DD analyses are not likely to be informative of the causal effect of the HRRP.

Finally, while estimates in Table 1 are from a DD approach that compares penalized and unpenalized hospitals (i.e., effect of treatment-on-treated), we also present evidence below that DD models that rely on proxies for being penalized, specifically, baseline readmissions (e.g., Demiralp et al., 2018) are also likely invalid. We show this as part of the analysis of the Gupta (2021) study because that study incorporates a DD approach that relies on the parallel trend assumption for hospitals with low- and high-baseline readmissions. Similarly, we show next that studies that compared changes over time in targeted and non-targeted conditions (e.g., Zuckerman et al., 2016) are also unlikely to be valid because of differential pre-trends between targeted and non-targeted conditions.

The Validity of the DDD Design and Prior Evidence Using It

A few studies have used a triple difference research design (Chen & Grabowski, 2019; Gupta, 2021; Mellor et al., 2017; Ody et al., 2019). The DDD approach can be viewed as the difference in two separate DD designs where one DD model is estimated for a treated group and another DD model is estimated for an untreated group. Studies that use the DDD approach to study the HRRP consider admissions for HRRP targeted conditions as treated group and non-targeted conditions (e.g., GI admissions) as the untreated group. Viewed this way, we can illustrate the DDD model approach using two DD models. The first DD model is for the treated group (TREAT = 1) and is similar to equation (1)

R E A D M I T_{j t} (T R E A T = 1) = a_{j} + δ_{t} + X_{j t} Γ + γ Z_{j t} + β P E N A L T Y_{j t} + e_{j t}

(3)

To denote that it is the treated group of conditions, we use the TREAT = 1 notation shown in equation (2). The only difference between equations (1) and (3) is that we have included another variable (Z) in equation (3) to represent unmeasured factors that affect readmissions. It is the presence of such factors that biases DD estimates, for example, because it is correlated with the penalty and, therefore, will cause trends in readmissions to diverge between treated and untreated hospitals, as demonstrated above.

An analogous DD model for the untreated group (i.e., TREAT = 0) is

R E A D M I T_{j t} (T R E A T = 0) = {a^{'}}_{j} + {δ^{'}}_{t} + X_{j t} Γ^{'} + γ Z_{j t} + e_{j t}

(4)

Equation (4) omits the penalty indicator because the penalty is assumed to have no effect on admissions for conditions that are not targeted by the HRRP. We also denote new parameters using the superscript ´. We explore shortly the consequences if this assumption is violated, but for now we proceed as if the assumption is valid. The key aspect of equation (4) is that the coefficient on the unmeasured factor (Z) is the same as that in equation (3) for the treated group—that is, parallel trends between targeted and non-targeted conditions.

Taking the difference between equations (3) and (4), or the DDD, yields

R E A D M I T_{j t} (T R E A T = 1) - R E A D M I T_{j t} (T R E A T = 0) = {\ddot{α}}_{j} + {\ddot{δ}}_{t} + X_{j t} \ddot{Γ} + β P E N A L T Y_{j t} + v_{j t}

(5)

In equation (5), the dependent variable is the difference in readmissions for treated and untreated conditions in hospital j and year t. The DDD model includes the two-way interactions between hospital and treated conditions and year and treated conditions, as indicated by the inclusion of hospital and year effects in the difference model, which are differentiated by the use of the (..) superscript. The effect of unmeasured factors, which was assumed to be the same for both treated and untreated conditions, is eliminated by taking the difference between equations (3) and (4). If the assumption of common unmeasured effects holds, the DDD estimate yields the true effect of the HRRP penalty on readmissions.

A simple test of the validity of the DDD approach is to estimate a slightly different version of equation (5)

R E A D M I T_{j t} (T R E A T = 1) = ρ R E A D M I T_{j t} (T R E A T = 0) + {\ddot{α}}_{j} + {\ddot{δ}}_{t} + X_{j t} \ddot{Γ} + β P E N A L T Y_{j t} + v_{j t}

(6)

The difference between equations (5) and (6) is that we do not restrict the coefficient on readmissions for non-treated conditions to be equal to one, as does the standard DDD model. As we show in the Appendix (see Appendix A), the validity of the standard DDD approach depends on whether ρ = 1, which is a test of whether the effect of unmeasured factors on readmissions is the same for treated and untreated conditions.

Before presenting estimates of DDD models, we review the bias of a DDD approach if the policy actually affects untreated conditions, for example, because the hospital responds to the HRRP by reallocating resources toward treated conditions that are part of the performance indicator and away from untreated conditions. In this case, the model for the untreated units (i.e., TREAT = 0) is

R E A D M I T_{j t} (T R E A T = 0) = {a^{'}}_{j} + {δ^{'}}_{t} + X_{j t} Γ^{'} + γ Z_{j t} + π β P E N A L T Y_{j t} + e_{j t}

(7)

In equation (7), we have allowed the HRRP penalty to affect untreated conditions and show it as a multiplicative factor (π) of the effect of the HRRP penalty on treated conditions (β) to simplify the algebra that comes next. Now, taking the difference between equations (3) and (7) yields

Δ R E A D M I T_{j t} = {\ddot{α}}_{j} + {\ddot{δ}}_{t} + X_{j t} {\ddot{Γ}}_{k} + β (1 - π) P E N A L T Y_{j t} + v_{j t}

(8)

In this case, even if the unmeasured factor has the same impact on the treated and untreated conditions, the DDD estimate will be biased because it measures the difference in the treatment effects for treated and untreated (differentially treated) conditions. The test of the validity of the DDD approach remains the same, although this is just testing whether unmeasured factors affect readmissions for treated and untreated conditions the same and not whether there is an effect of the HRRP on untreated conditions.¹²

Table 2 reports DDD estimates that use two different sets of conditions as untreated: all non-targeted readmissions, as was used, for example, in Desai et al. (2016), Zuckerman et al. (2016), and Ody et al. (2019), and readmissions for GI conditions, which were used by Carey and Lin (2015) and Gupta (2021). We report estimates from equations (5) and (6) with the latter providing an explicit test of the validity of the DDD approach. For equation (5), which is the standard DDD specification, we show estimates only for the specification that used all non-targeted conditions as untreated group.

Table 2.

Difference-in-differences-in-differences estimates of the effect of HRRP penalty on readmissions, by condition.

	AMI			HF			PN
Condition-specific penalty	−0.013***	−0.013***	−0.017***	−0.014***	−0.014***	−0.018***	−0.013***	−0.014**	−0.017***
	(0.002)	(0.002)	(0.002)	(0.001)	(0.001)	(0.001)	(0.001)	(0.001)	(0.001)
Readmissions non-targeted		1.052***			1.025***			0.824***
		(0.055)			(0.029)			(0.026)
Readmissions GI			0.032*			0.031***			0.021***
			(0.013)			(0.006)			(0.006)
Mean dep. variable	0.182	0.182	0.182	0.215	0.215	0.215	0.162	0.162	0.162
Number of obs.	15,203	15,203	15,203	21,401	21,401	21,401	21,508	21,508	21,508
Hospital-wide penalty	−0.005***	−0.005***	−0.008***	−0.006***	−0.006***	−0.009***	−0.005***	−0.005***	−0.008***
	(0.002)	(0.002)	(0.002)	(0.001)	(0.001)	(0.001)	(0.001)	(0.001)	(0.001)
Readmissions non-targeted		1.068***			1.054***			0.844***
		(0.053)			(0.029)			(0.026)
Readmissions GI			0.036**			0.031***			0.022***
			(0.012)			(0.006)			(0.006)
Mean dep. variable	0.182	0.182	0.182	0.215	0.215	0.215	0.162	0.162	0.162
Number of obs.	19,147	19,147	19,147	22,105	22,105	22,105	22,245	22,245	22,245

Notes: The unit of observation is the hospital-year level. In the first column of results for each condition, the dependent variable is the difference in readmissions between the condition indicated and readmissions for all non-targeted conditions. In the other two columns, the dependent variable is 30-day readmissions for that condition. The regressions include patients admitted for each relevant condition between July 2008 and July 2014. AMI = acute myocardial infarction; HF = heart failure; and PN = pneumonia. All regressions include hospital fixed effects, year fixed effects and a set of controls for patient characteristics that vary over time for each hospital (indicators for age under 65, age 65–70, age 70–75, age 80–85, age over 85, share Hispanic, share Black, share White, share other race, share female). Robust standard errors are constructed allowing for non-independence at the hospital level. Each regression is weighted by the number of patients admitted for the relevant condition (AMI, HF, and PN) in the hospital year. * 0.05 < p-value ≤0.10, ** p ≤ 0.05, *** p ≤ 0.01.

DDD estimates in Table 2 are similar to the DD estimates reported in Table 1 and suggest that an HRRP penalty reduced readmissions significantly. While the DDD estimates obtained using all non-targeted conditions as untreated conditions are smaller than DD estimates, they remain relatively large and indicate that hospitals that were penalized were able to reduce readmissions by between 5% and 8%. As noted, for several reasons we expect the HRRP to have modest effects, which this effect size may be if the share of preventable readmissions is larger than most believe, or if there are more efficacious interventions than widely believed. And it is also the case that these DDD estimates are near identical across conditions, which we have argued is unlikely unless there is a common intervention to reduce readmissions across conditions. Estimates using the condition-specific penalty indicator (top panel) are again approximately twice the size of estimates from models that use the hospital-wide penalty indicator (bottom panel). DDD estimates are also relatively similar whether we use all non-targeted readmissions or GI readmissions, as untreated conditions, but the DDD estimates obtained from models using GI readmissions are larger and almost identical to DD estimates in Table 2.

Coefficients on readmissions for untreated conditions (equation (6) specification) indicate that the DDD model that uses GI readmissions as untreated admissions and a control is not valid, as estimates of the effect of GI readmissions on HRRP targeted conditions is significantly different from one in all cases. It is notable that DDD estimates from these models are larger than from models using all non-targeted conditions and near identical to DD estimates. In analyses that use all non-targeted conditions as the control, we cannot reject the hypothesis that estimates of the effect of these other conditions on targeted readmissions is equal to one except for the case of pneumonia. However, as shown in the Appendix, in some circumstances, the underlying assumption of the DDD specification may still be violated even if we cannot reject ρ = 1. While rejecting ρ = 1 always indicates an invalid research design, not rejecting this equality does not always indicate that the design is valid.

To further asses the common trends assumption of the DDD approach, we also conducted event-study analyses in the DDD context to assess the validity of the DDD specification. Table 3 reports the event-study results in the context of the DDD analysis. For this analysis, we use all non-targeted conditions as the non-treated control because it was clear from Table 2 that analyses that use GI as the control condition are not valid. First, we note that for five of the seven analyses, there is clear evidence that the DDD design is not valid, as supported by significant estimates associated with the pre-2012 penalty-by-year interactions.¹³ A similar finding of diverging trends between targeted and non-targeted conditions was reported by Zucherman et al. (2016) in their interrupted time-series analyses, but the divergence was not highlighted. Ody et al. (2019) also reported differential trends in pre-HRRP period between targeted and non-targeted conditions.

Table 3.

Difference-in-differences-in-differences estimates of the effect of HRRP penalty on readmissions, by condition event study design.

	Condition-specific penalty			Hospital wide penalty
	AMI	HF	PN	AMI	HF	PN	All Targeted
Penalty X 2008	−0.010*** (0.003)	−0.004** (0.002)	−0.015*** (0.002)	0.0009 (0.003)	−0.0002 (0.002)	−0.010*** (0.002)	−0.003** (0.001)
Penalty X 2009	0.006** (0.003)	0.005*** (0.002)	0.003* (0.002)	0.006* (0.003)	0.002 (0.002)	−0.004*** (0.002)	0.0005 (0.001)
Penalty X 2010	0.004 (0.003)	0.007*** (0.002)	0.002 (0.002)	0.008** (0.004)	0.002 (0.002)	−0.001*** (0.002)	0.002 (0.001)
Penalty X 2012	−0.013*** (0.003)	−0.010*** (0.002)	−0.015*** (0.002)	−0.002 (0.002)	−0.003* (0.002)	−0.010*** (0.002)	−0.006*** (0.001)
Penalty X 2013	−0.014*** (0.003)	−0.011*** (0.002)	−0.016*** (0.002)	−0.004 (0.004)	−0.006*** (0.002)	−0.008*** (0.002)	−0.007*** (0.001)
Penalty X 2014	−0.013*** (0.003)	−0.014*** (0.002)	−0.015*** (0.002)	0.0004 (0.004)	−0.005*** (0.002)	−0.008*** (0.002)	−0.006*** (0.001)
Mean dependent variable	0.182	0.215	0.162	0.182	0.215	0.162	0.195
Number of observations	15,203	21,401	21,508	19,147	22,105	22,245	22,279

Notes: The unit of observation is the hospital-year level dependent variable is 30-day readmissions. In all regressions, the control conditions are all non-targeted readmissions. The regressions include patients admitted for each relevant condition between July 2008 and July 2014. AMI = acute myocardial infarction; HF = heart failure; and PN = pneumonia. All regressions include hospital fixed effects, year fixed effects and a set of controls for patient characteristics that vary over time for each hospital (indicators for age under 65, age 65–70, age 70–75, age 80–85, age over 85, share Hispanic, share Black, share White, share other race, share female) Robust standard errors are constructed allowing for non-independence at the hospital level. Each regression is weighted by the number of patients admitted for the relevant condition (AMI, HF, PN, and targeted readmissions) in the hospital year. * 0.05 < p-value ≤0.10, ** p ≤ 0.05, *** p ≤ 0.01.

Overall, estimates in Tables 2 and 3 demonstrate that the DDD design to study the HRRP is unlikely to be valid. Although, our analysis applies to only studies using national data and not single-state studies (e.g., Chen & Grabowski, 2019 and Mellor et al., 2017). The evidence in Tables 2 and 3 also implies that DD analyses that use targeted and non-targeted conditions in an intention-to-treat framework, which measures the difference in readmissions between conditions more or less likely to be penalized (and not whether a hospital was penalized) are also unlikely to be valid (e.g., Demiralp et al., 2018; Ody et al., 2019). Moreover, the DDD relies on the untested assumption that the control conditions are not affected by the HRRP. For both reasons, DDD estimates of the effect of the HRRP are not likely to be informative of the causal effect of the HRRP.

The Validity of the IV Design and Evidence Using It

One previous paper used an IV approach (Gupta, 2021). The way that the IV approach was implemented by Gupta (2021) is closely related to the DD approach. Specifically, Gupta (2021) instruments for the HRRP penalty (actually a forecasted penalty, which we discuss further below) using baseline hospital-specific readmissions in 2007 (2006–2007) interacted with a post-HRRP penalty indicator. This approach is similar in spirit to McGarry et al. (2016) and others that use baseline characteristics to divide hospitals by whether they are more or less likely to be penalized and conduct a DD analysis (this would be the reduced form estimate in Gupta, 2021).

Little theoretical justification was provided for the validity of this instrument (e.g., exclusion restriction), which is a Bartik (1991)-like instrument. The idea underlying this approach is that baseline characteristics make it more or less likely that a hospital will be penalized when the HRRP comes into effect and, therefore, examining changes in readmissions pre-to-post HRRP for hospitals stratified by baseline characteristics can be used to obtain an estimate of the effect of the HRRP. Of course, this approach still relies on the assumption that absent the HRRP, the pre-to-post changes in readmissions of hospitals stratified by baseline characteristics would be the same.

Algebraically, the IV model used by Gupta (2021) can be illustrated using the following regression model specifications

R E A D M I T_{j t} = a_{j} + δ_{t} + X_{j t} Γ + β (P E \hat{N} A L T Y_{j} * P O S T_{t}) + e_{j t}

(9)

Equation (9) is the model of interest and identical to equation (1), although in this case treatment is indicated using an interaction between a hospital-specific indicator of whether the hospital was penalized in the first round of the HRRP and an indicator that the period is 2012 or after (POST). Also note that instead of using the actual HRRP penalty indicator a predicted penalty (i.e., predicted penalty-by-post) is used. The predicted penalty is obtained by the following first-stage regression

P E N A L T Y_{j} * P O S T_{t} = {\overset{⌣}{a}}_{j} + {\overset{⌣}{δ}}_{t} + X_{j t} \overset{⌣}{Γ} + π (R E A D M I T 0607_{j} * P O S T_{t}) + u_{j t}

(10)

In equation (10), we have used the superscript (^‿) to denote new parameters. The key aspect of this model is the use of baseline (2006–2007) readmissions interacted with a dummy variable indicating the post-penalty period (2012) as an instrument. As Gupta (2021) shows, this instrument is a strong predictor of the HRRP penalty (or forecasted penalty). Hospitals with relatively more admissions in 2006–2007 are more likely to be penalized by the HRRP in 2012, as presumed by McGarry et al. (2016) and others.

The critical issue for the IV approach to be valid is whether the exclusion restriction is valid. Validity requires that baseline readmissions affect readmissions in subsequent years (e.g., post-HRRP) only because it makes it more likely that a hospital will be penalized. In the absence of the penalty, hospital readmissions would be the same for hospitals with different levels of baseline admissions, which is just the parallel trend assumption of a DD analysis. Given that the model described by equations (9) and (10) is just identified, there is no way to definitively test this assumption. However, evidence of the likely validity of the exclusion restriction can be obtained in a variety of ways. One way is to conduct the IV analysis using an event-study specification (e.g., Jaeger et al., 2020). In this case, the outcome equation is

R E A D M I T_{j t} = a_{j} + δ_{t} + X_{j t} Γ + \sum_{t = 2008}^{2014} β_{t} (P E \hat{N} A L T Y_{t} * Y E A R_{t}) + e_{j t}

(11)

Equation (11) is an event-study specification often used in a DD analysis. It allows the difference in readmissions between penalized and unpenalized hospitals to differ in each year from 2008 to 2014 (leaving 1 year omitted as reference). The only difference between this specification and the specification of equation (9) is that the POST dummy variable is replaced with a set of year dummy variables. The analogous first-stage equation to predict the penalty variables is

P E N A L T Y_{j} * Y E A R_{t = k} = {\overset{⌣}{a}}_{j} + {\overset{⌣}{δ}}_{t} + X_{j t} \overset{⌣}{Γ} + \sum_{t = 2008}^{2014} π_{t} (R E A D M I T 0607_{j} * Y E A R_{t}) + u_{j t}

(12)

Equation (12) is used to predict the interaction between the HRRP penalty and year indicators. In this equation too, the POST dummy variable (of equation (10)) has been replaced with a set of year dummy variables. There will be six (k = 6) first-stage equations for each of the six (7 years with one omitted as reference) interactions shown in equation (11). For each first-stage equation, there are six instruments (1 year omitted) constructed by interacting year dummy variables with the baseline readmissions, but the only instrument that, in practice, matters in terms of significance is the interaction between baseline readmissions (READMIT0607) and year when the year of this interaction equals the year of the interaction with the penalty (PENALTY*YEAR).¹⁴

Estimates from equation (11) can be used to test the validity of the IV approach. Estimates of the coefficients on the interactions between the penalty indicator and year (PENATY*YEAR) in years prior to the HRRP (2012) should be zero. If not, then it suggests that trends in hospital readmissions between hospitals with different levels of baseline admissions differ prior to the penalty, which implies that such differences would persist in years post-penalty even without the penalty.

A second way to assess the validity of the IV approach is to exploit the multiplicity of potential instruments stemming from the nature of the IV approach (Goldsmith-Pinkham et al., 2018). In his application, Gupta (2021) used baseline readmissions to stratify hospitals more or less likely to be penalized, but any number of baseline variables could be used as instruments and using these additional instruments makes the model an over identified one that can be used to test the likely validity of the exclusion restriction.

An expanded IV model that uses multiple potential instruments is the following

R E A D M I T_{j t} = a_{j} + δ_{t} + X_{j t} Γ + \sum_{k = 1}^{K} {λ^{'}}_{k} (W_{j k} * P O S T_{t}) + β P E \hat{N} A L T Y_{j t} + e_{j t}

(13)

and

P E N A L T Y_{j t} = {\overset{⌣}{a}}_{j} + {\overset{⌣}{δ}}_{t} + X_{j t} \overset{⌣}{Γ} + \sum_{k = 1}^{K} {π^{'}}_{k} (W_{j k} * P O S T_{t}) + u_{j t}

(14)

Equations (13) and (14) represent the overidentified IV model. The variable W is a vector representing several hospital-specific baseline characteristics including readmissions, which is the instrument used by Gupta (2021). An overidentification test can be conducted by including all of the baseline characteristics except readmissions in the second stage (equation (13)) and seeing whether the coefficient estimates on the interactions in equation (13) are statistically significant. Significant estimates suggest that changes in readmissions before and after the HRRP penalty differ by baseline hospital characteristics conditional on being penalized, which is evidence against the exclusion restriction in the case of baseline readmissions.

Table 4 presents OLS (DD) estimates in the top row and IV estimates in panels 2 and 3.¹⁵ Estimates from the just identified model are presented in the middle panel and estimates from the model given by equations (12) and (13) are presented in the bottom panel. We also show estimates when the actual penalty is used as an indicator of treatment (left side of table) and when the actual penalty is replaced by a forecasted, or expected penalty. As noted above, Gupta (2021) did not use the actual penalty as an indicator of treatment, but instead used the forecasted penalty, which is a function of readmissions in the period before the penalty period and used by CMS to determine the penalty.¹⁶ The assumption for using a forecasted penalty is that hospitals used the previous period’s readmissions to forecast their probability of being penalized and responded according to the probability of being penalized and not the actual penalty.¹⁷ The instrument remains the same in this case.

Table 4.

OLS, just identified IV, and event-study IV estimates of the effect of HRRP condition-specific penalty on condition-specific readmissions 2008–2014.

	Actual penalty			Expected penalty
OLS (difference-In-Difference)	AMI	HF	PN	AMI	HF	PN
Condition-specific penalty	−0.017*** (0.002)	−0.018*** (0.0012)	−0.017*** (0 0.001)	−0.024*** (0.002)	−0.034*** (0.001)	−0.039*** (0.001)
IV—Just identified model
Condition-specific penalty	−0.057*** (0.011)	−0.029*** (0.0037)	−0.024*** (0.003)	−0.041*** (0.006)	−0.023*** (0.003)	−0.030*** (0.004)
IV—Event-study IV
Condition-specific penalty X 2008	0.036*** (0.013)	0.041*** (0.006)	0.007 (0.005)	0.060*** (0.020)	0.043*** (0.006)	0.008 (0.007)
Condition-specific penalty X 2009	0.037*** (0.012)	0.016*** (0.005)	0.006 (0.005)	0.060*** (0.018)	0.017*** (0.005)	0.008 (0.007)
Condition-specific penalty X 2010	0.007 (0.012)	0.012** (0.005)	0.003 (0.005)	0.015*** (0.021)	0.012*** (0.006)	0.005 (0.008)
Condition-specific penalty X 2012	−0.023* (0.013)	0.002 (0.005)	−0.016*** (0.005)	−0.037 (0.021)	0.001 (0.005)	−0.020*** (0.007)
Condition-specific penalty X 2013	−0.034*** (0.013)	−0.010** (0.005)	−0.021*** (0.006)	−0.056*** (0.022)	−0.012*** (0.005)	−0.026*** (0.008)
Condition-specific penalty X 2014	−0.038*** (0.012)	−0.025*** (0.005)	−0.022*** (0.006)	−0.053*** (0.020)	−0.027*** (0.006)	−0.028*** (0.007)
Mean dependent variable	0.182	0.215	0.162	0.182	0.215	0.162
Number of observations	15,203	21,099	21,706	15,203	21,099	21,706

Notes: The unit of observation is the hospital-year level. The regressions include patients admitted for each relevant condition between July 2008 and July 2014. AMI = acute myocardial infarction; HF = heart failure; and PN = pneumonia. All regressions include hospital fixed effects, year fixed effects and a set of controls for patient characteristics that vary over time for each hospital (indicators for age under 65, age 65–70, age 70–75, age 80–85, age over 85, share Hispanic, share Black, share White, share other race, share female). The instrument(s) for IV estimates is (are) the interaction(s) between 2006 and 2007 condition-specific readmissions and indicator(s) for post-HRRP period (year). Robust standard errors are constructed allowing for non-independence at the hospital level. Each regression is weighted by the number of patients admitted for the relevant condition (AMI, HF, and PN) in the hospital year. * 0.05 < p-value ≤0.10, ** p ≤ 0.05, *** p ≤ 0.01.

OLS estimates in Table 4 have already been discussed earlier in the section on DD analyses and, therefore, we will not discuss them again. The just identified IV estimates in the middle panel are relatively large. When the actual penalty is used, the condition-specific HRRP penalty is associated with: a 31% reduction in AMI readmissions; a 13% reduction in heart failure readmissions; and a 15% reduction in pneumonia readmissions. We again note that effect sizes this large may be implausible if the actual number of preventable readmissions is 27% or even less. Just identified IV estimates obtained using the forecasted penalty are similar in magnitude to those that use the actual penalty. When the predicted penalty is used, the condition-specific HRRP penalty is associated with a 22% reduction in AMI readmissions; a 9% reduction in heart failure readmissions; and a 18% reduction in pneumonia readmissions. These IV estimates are very close to what Gupta (2021) reported, although the IV estimate here pertaining to heart failure is larger than in Gupta (2021. Specifically, Table 2 in Gupta (2021) reports the following coefficients (standard errors) for AMI, HF, and PN, respectively, −0.0349 (0.006), −0.01 (0.003), and −0.028 (0.003). Our estimates for this analysis (reported in Table 4) for AMI, HF, and PN, respectively, are −0.041(0.006), −0.023(0.003), and −0.030 (0.004).

If the forecasted penalty was the correct measure to use instead of the actual penalty, one would expect the estimate to be larger when the expected penalty is used because it better classifies hospitals to treated and untreated groups, but this is not the case two out of three times.

The last set of estimates in Table 4 is from the event-study specification of the IV model and these are shown in the bottom panel. The key estimates here are those on the interactions between the penalty indicator and year in years prior to 2012. A valid IV design would require that these estimates be zero. For both AMI and heart failure, estimated coefficients on these interaction terms are statistically significant and large suggesting that the exclusion restriction of the IV approach is unlikely to be valid. Only in the case of pneumonia does the IV design appear to hold up. Again, very similar results are obtained when the forecasted penalty is used as an indicator of treatment instead of the actual penalty. In Appendix Table A.1, we replicate the analysis of Table 4, but use as the measure of treatment an indicator of whether the hospital was penalized for any condition (i.e., hospital-wide penalty). Estimates in that table are qualitatively the same as in Table 4 and indicate the same problems with using the IV approach. It is important to note that the lack of validity of the IV approach that uses baseline readmissions as an instrument also provides evidence that DD intention-to-treat analyses based on classifying hospitals using baseline readmissions are also invalid (e.g., Demiralp et al., 2018; Ody et al., 2019).

Table 5 presents IV estimates from the model of equations (13) and (14).¹⁸ This is a just identified model with the excluded instrument of baseline readmissions interacted with a dummy variable indicating the post-penalty period. Several other interactions between baseline characteristics (shares of race/ethnicity of patients; bed size of hospital; whether hospital is a teaching institution; and share of patients with diabetes and pneumonia) and a post-penalty dummy variable are included in both the first and second stages. For continuous baseline measures (e.g., diabetes, pneumonia, bed size), we used dummy variables indicating tertiles of the distribution of the baseline characteristic.¹⁹ Because any or all of these baseline characteristics could plausibly be used as instruments, including them in the second stage allows for tests of over identification.

Table 5.

Just identified and over identified IV estimates of the effect of HRRP of HRRP condition-specific penalty on condition-specific readmissions 2008–2014.

	Actual penalty			Expected penalty
IV—Just identified model	AMI	HF	PN	AMI	HF	PN
Condition-specific penalty	−0.057*** (0.011)	−0.029*** (0.0037)	−0.024*** (0.003)	−0.041*** (0.006)	−0.023*** (0.003)	−0.030*** (0.004)
IV—Over identified model
Condition-specific penalty	−0.060*** (0.013)	−0.032*** (0.005)	−0.025*** (0.004)	−0.108*** (0.018)	−0.063*** (0.009)	−0.080*** (0.013)
Share Black 2006–2007 X post	0.027*** (0.014)	0.006 (0.004)	0.006 (0.006)	0.017+ (0.010	0.004 (0.004)	0.002 (0.006)
Share Hispanic 2006–2007 X post	0.024 (0.026)	0.005 (0.020)	−0.002 (0.011)	0.033 (0.024)	−0.005 (0.016)	−0.011 (0.013)
Share dual eligible 2006–2007 X post	0.016 (0.015)	0.009 (0.007)	0.008 (0.005)	0.012 (0.012)	0.015** (0.007)	0.024*** (0.007)
Medium group diabetes 2006–2007 X post	−0.002 (0.003)	0.004*** (0.002)	0.002 (0.001)	−0.002 (0.002)	0.003*** (0.001)	<0.000 (0.001)
High group diabetes 2006–2007 X post	<0.000 (0.004)	0.006 (0.002)	0.001 (0.002)	−0.002 (0.003)	0.005*** (0.002)	0.001 (0.002)
Medium group pneumonia 2006–2007 X post	<0.000 (0.003)	<0.000 (0.001)	<0.000 (0.001)	0.0004 (0.002)	<0.000 (0.014)	<0.000 (0.001)
High group pneumonia 2006–2007 X post	−0.003 (0.003)	<0.000 (0.002)	<0.000 (0.002)	−0.0008 (0.003)	0.002 (0.002)	<0.000 (0.002)
Teaching 2006–2007 X post	0.001 (0.002)	−0.001 (0.001)	−0.002 (0.001)	0.0006 (0.002)	−0.0005 (0.001)	−0.002 (0.001)
Medium group bed size 2006–2007 X post	0.013*** (0.005)	0.002 (0.002)	−0.002 (0.002)	0.010** (0.004)	−0.001 (0.002)	−0.006*** (0.001)
High group bed size 2006–2007 X post	0.016*** (0.005)	0.005** (0.002)	−0.004** (0.002)	0.006 (0.004)	−0.003 (0.002)	−0.013*** (0.002)
Mean dependent variable	0.182	0.215	0.162	0.182	0.215	0.162
Number of observations	15,203	21,099	21,706	15,203	21,099	21,706

Notes: The unit of observation is the hospital-year level. The regressions include patients admitted for each relevant condition between July 2008 and July 2014. AMI = acute myocardial infarction; HF = heart failure; and PN = pneumonia. All regressions include hospital fixed effects, year fixed effects and a set of controls for patient characteristics that vary over time for each hospital (indicators for age under 65, age 65–70, age 70–75, age 80–85, age over 85, share Hispanic, share Black, share White, share other race, share female). The instrument(s) for IV estimates is (are) the interaction(s) between 2006 and 2007 condition-specific readmissions and indicator(s) for post-HRRP period (year). Also added to model are interactions of baseline 2006-2007 characteristics (share Black, share Hispanic, share dual eligible, share with diabetes comorbidity, share with pneumonia comorbidity, teaching status, and bed count) and post-HRRP indicator. Robust standard errors are constructed allowing for non-independence at the hospital level. Each regression is weighted by the number of patients admitted for the relevant condition (AMI, HF, and PN) in the hospital year. * 0.05 < p-value ≤0.10, ** p ≤ 0.05, *** p ≤ 0.01.

The IV estimates from this model are similar to those in Table 4 (shown in top row of Table 5) and indicate that the HRRP penalty (actual) is associated with a 33% reduction in AMI readmissions; a 15% reduction in heart failure readmissions and a 15% reduction in pneumonia readmissions. Notably the IV estimate for this model when the expected penalty is used are much larger than the corresponding estimates in Table 4 suggesting declines in AMI readmissions of 61% and pneumonia admissions of 50%. Effects of this magnitude seem implausible given our earlier discussion. More importantly, for each outcome, there is evidence that the exclusion restriction may not be valid as some of the interactions between baseline characteristics and a dummy variable indicating the post-penalty period are statistically significant. In the case of AMI, estimates associated with interactions between share of patients who are black and bed size of hospital are statistically significant; and for heart failure and pneumonia the significant estimates pertain to diabetes and bed size. Estimates in Appendix Table A.2, which use an indicator for a hospital-wide penalty, or the forecasted hospital-wide penalty, provide similar evidence that the IV approach using baseline readmissions (in this case all readmissions) interacted with a post-HRRP dummy variable is likely invalid.

Overall, estimates in Tables 4 and 5 suggest that the IV approach is unlikely to be valid. There is substantial evidence that the exclusion restriction associated with the use of interactions between one or more baseline characteristics and time is unlikely to be valid. Baseline (2006–2007) readmissions that predict the penalty, predict reductions in readmissions up to 3 years before the HRRP was implemented. Also, characteristics that make the hospital more or less likely to be penalized by the HRRP also influence readmissions directly (perhaps because of being correlated with confounders) and not just because of the HRRP penalty. This finding is consistent with the DD analysis that showed that penalized and unpenalized hospitals had different trends in readmissions prior to the HRRP (and prior to ACA even). So, as with the DD and DDD analyses, the IV estimates are unlikely to be informative about the causal effect of the HRRP on readmissions.

A Novel Approach to Study the Effect of the HRRP: The Regression-Kink Design

The HRRP penalizes hospitals with excess readmissions, which are defined using the ratio of actual readmissions to expected readmissions, which are the predicted number of readmissions given a hospital’s patient’s comorbidities.²⁰ The excess readmission ratio is measured in a prior 3-year period. A hospital with an excess readmission ratio in the prior 3-year period of less than or equal to one is not penalized, but a hospital with an excess readmission rate greater than one is penalized, and penalties grow with the excess readmission ratio. Therefore, there is a “kink” in the relationship between the size of the HRRP penalty and the excess readmission ratio at the value of one. The penalty is zero when the excess readmission ratio is below one, and the penalty increases with the excess readmission ratio for values greater than one. Accordingly, if the HRRP penalty causes hospitals to reduce readmissions, then there should be a similar “kink” in the relationship between readmissions and the excess readmission ratio. The “kink” structure of the HRRP penalty determination lends itself to the use of a regression-kink (RK) research design (Card et al., 2015).

Identification of the effect of the HRRP in the RK design comes from a change in the effect (slope) of the excess readmission ratio on readmissions at the threshold between when the penalty is zero or becomes positive (Card et al., 2015). The main advantage of the RK approach is that, in contrast to previous studies, the RK uses variation in the penalty amount due to a kink in the penalty formula and does not rely on any type of parallel trend assumption, as do the DD, DDD, and IV approaches that have been used in the past and shown above to be problematic.

The RK approach also allows us, in contrast to all prior studies, to obtain an estimate of the effect of the size of the penalty on readmissions. Estimating the effect of the size of the penalty is arguably a more interesting parameter given that a large majority (e.g., 83%) of hospitals are penalized. The large share of hospitals penalized suggest that the effect of size of the penalty is the important parameter to know and not simply whether a hospital was penalized or not. This fact has largely been ignored in prior studies that almost uniformly analyze the effect of being penalized, or the effect of the likelihood of being penalized.

The regression-kink design is implemented using regression methods and model specifications as follows (Card et al., 2015). The equation of interest is

R E A D M I T_{j t} = b_{0} + b_{1} R A T I O_{j (t - 1)} + b_{2} P E N A L T Y_{j t} + n_{j t}

(15)

Equation (15) indicates that readmissions of hospital “j” in year “t” depends on the hospital’s excess readmission ratio (RATIO) and the amount of penalty (PENALTY) that hospital “j” in year “t” incurred. Estimates of equation (15) are likely biased because the amount of the penalty is likely correlated with unmeasured factors that influence both the penalty and readmissions, such as hospital size and patient characteristics. To address the possible bias, we instrument for the penalty using a RK specification

P E N A L T Y_{j t} = {b^{'}}_{0} + {b^{'}}_{1} R A T I O_{j (t - 1)} + {b^{'}}_{2} (R A T I O_{j (t - 1)} * A B O V E_{j}) + v_{j t}

(16)

In equation (16), the size of the readmission penalty (PENALTY) of hospital “j” in year “t” depends on the excess readmission ratio (RATIO) in year “t-1” and the interaction between a dummy variable indicating that the excess readmission ratio is greater than one (ABOVE). This regression model mimics the CMS formula that mechanically determines the readmission penalty. The readmission penalty is zero when the excess readmission ratio is less than or equal to one, and then the penalty is a linear function of the excess readmission ratio after the threshold of one.²¹ The IV approach in this context is valid as long as the interaction term in equation (16) is excludable—that is, does not belong in equation (15). We believe this is a reasonable assumption because there is no reason (theoretical or empirical) to expect a change in the relationship between the excess readmission ratio at the threshold (RATIO = 1) other than because of the change in the penalty.

Table 6 reports the estimates from equation (16) for each of the three HRRP conditions and verifies that the regression mimics the penalty formula; estimates show that the coefficient on the excess readmission ratio

({b^{'}}_{1})

is virtually zero, which is expected because the penalty is zero prior to the excess readmission threshold. After the threshold, the HRRP penalty grows with the excess readmissions. Estimates indicate that every 0.10 unit increase in the excess readmission ratio, which is about a one-standard deviation change, is associated with approximately a 0.3%–0.4% increase in the penalty (e.g., from 0 to 0.4%). The relationship between the change in excess readmission ratio and penalty is relative similar across conditions. Estimates indicate that the maximum penalty, which was 1% (1.0) in 2012, is reached when the excess readmission ratio is approximately 0.3 units above the penalty threshold.

Table 6.

Estimates of the effect of the HRRP excess readmissions ratio on the round 1 HRRP penalty.

	AMI	HF	PN
Excess readmissions ratio 2009–2011	<0.000*** (<0.000)	<0.000*** (<0.000)	<0.000*** (<0.000)
Excess readmissions ratio*Penalty threshold	0.031*** (0.000)	0.044*** (0.002)	0.035*** (0.003)
Number of observations	640	708	720
Mean dependent variable	0.0005	0.046	0.040

The reduced form effect of the HRRP on readmissions is then obtained using the following regression

R E A D M I T_{j t} = {\ddot{b}}_{0} + {\ddot{b}}_{1} R A T I O_{j (t - 1)} + {\ddot{b}}_{2} (R A T I O_{j (t - 1)} * A B O V E_{j}) + v_{j t}

(17)

In equation (17), the share of admissions that are readmitted (READMIT) in hospital “j” in year “t” depends on the excess readmission ratio and the interaction between the indicator of the threshold and the excess readmission ratio. If there is a causal effect of the HRRP penalty on outcomes, the coefficient $({\ddot{b}}_{2})$ should be negative.

Note that the dependent variable is measured in year “t” (or after) which refers to the first year after the penalty was announced by CMS and known with certainty by the hospital (August 2011). In round 1, CMS announced the penalty in August 2011 based on an analysis of data from 2008 to 2011 (denoted as t-1 in equations (14)–(16)), but penalties did not start until October 2012. Given this announcement and known penalty status, we use years spanning Aug 2011 to Aug 2014 as the post-penalty period. However, as we noted earlier, there may be some uncertainty about the likelihood of being penalized, which would affect the interpretation of the RK estimates. We discuss this below.

One complication in applying the RK design is that the readmissions penalty is a function of the excess readmission ratio for three conditions: heart attack (AMI), heart failure (HF), and pneumonia (PN). Thus, a hospital can be penalized if it has an excess readmission ratio greater than one on any, or all, of these conditions. This circumstance makes it difficult to identify the appropriate counterfactual hospital. For example, consider a hospital with excess readmission ratios of 0.9, 1.01, and 1.3 for AMI, HF, and PN, respectively. For this hospital, the ideal counterfactual hospital might be one with excess readmission ratios of 0.9, 0.99, and 1.3 for AMI, HF, and PN, respectively. This example reveals the dimensionality problem in defining appropriate comparison hospitals if we used all three excess readmission ratios.

To address this issue, we stratify the sample and focus on the effect of one cause of a penalty at a time. For example, to estimate the effect of a hospital incurring a penalty due to excess AMI readmissions, we limit the sample to hospitals with excess readmission ratios less than one for HF and PN (i.e., not penalized for HF and PN). Thus, we have a sample of hospitals that we can order with respect to the AMI excess readmission ratio that all have excess readmission ratios for HF and PN that are less than one.²² One advantage of this approach is that it is straightforward. It allows for the use of one excess readmission ratio as the running variable and, therefore, relies on a standard regression-kink design. Second, we are making a particularly interesting comparison between a penalized hospital and what may be considered good performing hospitals—those that were not penalized for any condition. Out of the 2569 penalized hospitals by CMS in round 1 (FY2013), 234 hospitals were penalized for only having excess AMI readmissions, 362 hospitals were penalized for only having excess HF readmissions, and 315 hospitals were penalized for only having excess pneumonia readmissions. Therefore, the RK analysis includes 35% of hospitals that were penalized. The fact that we only use a portion of penalized hospitals, and that they are only penalized for one condition, may limit the generalizability of our findings.

Table 7 presents the results of the reduced form RK regressions for readmissions. We show results for several years: two prior years (2010 and 2011) and three post-HRRP penalty years (2012–2014). In each panel, the 2009–2011 excess readmission ratio is used as the running variable. This is the excess readmission ratio used to determine the round 1 HRRP penalty. We include analyses of readmissions in years prior to the actual implementation to assess whether we observe any significant effects in those years, which would suggest an invalid research design (Landais, 2015). In fact, we do not find any effects in these years.

Table 7.

Reduced form regression kink estimates of the effect of the HRRP on readmissions pre- and post-HRRP round 1 penalty.

Pre-HRRP 2010	AMI	HF	PN
Excess readmissions ratio 2009–2011	0.002*** (0.0005)	0.004*** (0.0002)	0.003*** (0.0002)
Excess readmissions ratio*Penalty threshold	0.002 (0.001)	0.001 (0.001)	−0.0004 (0.0006)
Pre-HRRP 2011
Excess readmissions ratio 2009–2011	0.002*** (0.0006)	0.004*** (0.0002)	0.003*** (0.0003)
Excess readmissions ratio*Penalty threshold	0.002 (0.002)	−0.001 (0.001)	0.0002 (0.0006)
Post-HRRP 2012
Excess readmissions ratio 2009–2011	0.0003 (0.0006)	0.002*** (0.0003)	0.001*** (0.0003)
Excess readmissions ratio*Penalty threshold	0.001 (0.002)	−0.002 (0.001)	−0.0004 (0.0007)
Post-HRRP 2013
Excess readmissions ratio 2009–2011	0.0009 (0.0005)	0.001*** (0.0003)	0.002*** (0.0003)
Excess readmissions ratio*Penalty threshold	0.0003 (0.001)	−0.001 (0.001)	−0.001 (0.0007)
Post-HRRP 2014
Excess readmissions ratio 2009–2011	0.0006 (0.0006)	0.001*** (0.0003)	0.002*** (0.0003)
Excess readmissions ratio*Penalty threshold	0.0006 (0.001)	−0.002* (0.001)	−0.001 (0.001)
Mean dependent variable	0.178	0.202	0.145
Number of observations	640	708	720

The most noteworthy finding revealed in Table 7 is that all estimates associated with the interaction between the excess readmission ratio and the indicator for being above the penalty threshold are small and statistically insignificant (at 0.05 level). There is only one marginally significant estimate and that pertains to heart failure in 2014. The magnitude of this estimate indicates that a 0.1 change in the excess readmission ratio, which is about a one-standard deviation change, is associated with a two percentage point (10%) decline in heart failure readmissions. This is the largest estimate in Table 7. It is also noteworthy that not all estimates are negative. Estimates pertaining to AMI are positive, but small and insignificant.

Table 8 presents the IV estimates of the effect of the HRRP penalty on readmissions. One advantage of the IV is it provides estimates of the effect of the size of the penalty on readmissions and not just whether a hospital was penalized. Prior studies have not provided such an estimate. As in the reduced form, we show estimates from periods before and after the implementation of the HRRP. We expect estimates from earlier periods to be statistically insignificant, which they are.

Table 8.

IV estimates of the effect of the HRRP round 1 penalty on readmissions.

Pre-HRRP 2010	AMI	HF	PN
Penalty	0.051 (0.044)	0.017 (0.021)	−0.012 (0.017)
Pre-HRRP 2011
Penalty	0.055 (0.051)	−0.023 (0.017)	0.003 (0.020)
Post-HRRP 2012
Penalty	0.064 (0.050)	−0.035 (0.024)	−0.011 (0.020)
Post-HRRP 2013
Penalty	0.012 (0.046)	−0.028 (0.022)	−0.029 (0.021)
Post-HRRP 2014
Penalty	0.024 (0.046)	−0.043* (0.023)	−0.010 (0.022)
Mean dependent variable	0.178	0.202	0.145
Number of observations	640	708	720

As our model is just identified, the IV estimates are similar to the reduced form estimates, but scaled by first-stage estimate.²³ IV estimates measure the change in readmissions for a hospital that received the maximum penalty of 1% versus an unpenalized hospital. All IV estimates in Table 8 are small to moderate in magnitude and not statistically significant at 0.05 level. Consistent with reduced form estimates, IV estimates pertaining to AMI are positive. Estimates related to pneumonia are small and insignificant. Estimates of the effect of the round 1 penalty on heart failure readmissions are negative and the 2014 estimate is marginally significant. Consider the magnitude of this estimate. It implies that receiving the maximum HRRP penalty is associated with a 4.3 percentage point (21%) reduction in heart failure readmissions.

How does this estimate compare to the magnitudes of estimates from other approaches (DD, DDD, IV) reported earlier? Estimates for heart failure from these other approaches are in the range of −0.02 and −0.03, which is 25%–50% smaller than the IV estimate for heart failure in Table 8. However, these other estimates refer to the effect of the average penalty and not the maximum penalty of 1%, as does the IV estimate. The average hospital penalty for heart failure was approximately 0.2% and using this figure to scale the IV estimate implies an effect size of −0.009 (or 0.9 percentage points). Thus, putting our IV estimate on a comparable basis to the previously reported estimates indicates that the IV estimate in Table 8 is less than half the size of the previous estimates and quite small suggesting a 4% reduction in heart failure readmissions for penalized hospitals. We acknowledge that the linear interpolation used to get comparable estimates may introduce substantial bias, but such a comparison highlights that the IV estimates in Table 8 are quite small (across all conditions) relative to estimates obtained by other approaches. Overall, we conclude that the IV estimates do not show much evidence that the HRRP penalty reduced readmissions for any of the conditions. At most, there is weak evidence that the HRRP penalty reduced readmissions for heart failure.

We have also validated the regression-kink design in a couple of ways. As noted, we find no evidence of effects in prior periods when the HRRP was not in effect (Landais, 2015). We also find (see Appendix Table A.3) that the addition of several baseline (i.e., 2007) hospital-specific covariates to the model has virtually no effect on reduced form or IV estimates.

Interpreting the Regression-Kink Results

What may account for the null results we find? One answer is that hospitals do not respond to the penalty or respond ineffectively. Earlier, we provided several reasons why this may be the case: it is difficult and costly to identify preventable readmissions; there is, at best, only weak evidence that common interventions are effective, particularly with respect to readmissions after 7 days (the majority); and the penalty formula is flawed providing little incentive for hospitals to respond. Another explanation is that the RK obscures a true effect. We consider this issue next.

One way the RK may obscure a true effect is if some unpenalized hospitals close to being penalized respond to avert future, expected penalties. If so, then we expect that the true “kink” point is to the left of the threshold that is set using the excess readmissions threshold of one. The opposite behavior—some penalized hospitals close to the threshold, do not respond—would produce a “kink” to the right of the threshold that is set using the excess readmissions threshold of one. It is unlikely that both types of behavior are occurring simultaneously. If unpenalized hospitals just to the left of the threshold are responding to avert future penalties, then it suggests that penalized hospitals to the right of the threshold are responding similarly. And if penalized hospitals just to the right of the threshold are not responding, then it suggests that unpenalized hospitals to the left of the threshold are not responding. If whether a hospital was penalized or not, or the hospital lies in the excess readmissions distribution, does not provide a signal to the hospital about future penalties, as just described, then the penalty cannot bring forth a response (and is consistent with our null finding).

We assess the extent of this potential problem by testing for additional kink points to the left and right of the threshold. We use a randomization procedure that produces a distribution of placebo estimates in the region around the actual HRRP penalty threshold (Ganong & Jager, 2018; Landais, 2015). The procedure is carried out by changing the excess readmission threshold a large number of times from above to below the actual HRRP penalty threshold (0.9–1.06), For each pseudo threshold, we re-estimate the regression-kink model for each pseudo threshold to construct a distribution of estimates. We chose 0.9 (appx 10th percentile) to 1.1 (90th percentile) because we maintain a bandwidth of +/− 0.1 either side of each randomized threshold that we test and 100% of the hospitals have an excess readmission ratio between 0.8 and 1.2. Thus using 0.9 to 1.1, allows us to test whether the kink in hospital response to the HRRP, is located as early as the 10th percentile of the excess readmission ratio and up to the 90th percentile of the excess readmission ratio.

We plot the distribution of estimates and the distribution of the R-square statistic from each regression. If the true threshold was to the left (or right) of the actual threshold, we would expect to see this reflected in the R-square (a better fit).

Figures 3 –5 show the randomized estimates for 3 years 2012, 2013, and 2014: Figure 3 shows estimates for AMI; Figure 4 shows estimates for heart failure; and Figure 5 shows estimates for pneumonia. The visual evidence in Figures 3 –5 is compelling. For AMI and pneumonia, none of the estimates shown (Figures 3 and 5) are statistically significant or significantly different from the estimate obtained using the threshold of 1.0. All estimates are also very small and are within a very tight range. For heart failure, 2012 estimates are all statistically insignificant, virtually zero, and within a narrow range. In 2013 and 2014, about one-third of the randomized estimates are statistically significant, but all remain small and are not statistically different from the estimate obtained using the threshold of 1.0. Overall, the randomized estimates in Figures 3 –5 provide no evidence that there was a different threshold and that the insignificant results we report in Table 7 are due to using the wrong threshold. This point is further evidenced in Figure 6, which graphs the R-square statistics from the randomized regressions for each condition. There is no evidence that the R-square statistic increases at thresholds away from 1.0. Therefore, we conclude that the null findings in Tables 7 and 8 are not due to using the wrong threshold, but more likely because hospitals were either not responding to the HRRP penalty or not responding effectively.

Figure 3.

Randomized RK estimates 2012–2014 for AMI.

Figure 4.

Randomized RK estimates 2012–2014 for heart failure.

Figure 5.

Randomized RK estimates 2012–2014 for pneumonia.

Figure 6.

R-square statistics from randomized RK models, by condition.

Conclusion

The HRRP is one of the most significant and salient pay-for-performance programs used by the federal government to improve the quality of care provided to people covered by Medicare. It has received much attention for some of the imperfections associated with its implementation and for its consequences. In this article, we revisited the evidence on the consequences of the HRRP for the primary outcome targeted—hospital readmissions. Past studies have generally shown that the HRRP reduced hospital readmissions. However, our assessment of the validity of the empirical approaches used in past studies suggests that the research designs used are unlikely to be valid, which limits the usefulness of the evidence produced by those studies.

We also provide novel evidence of the effect of the HRRP on hospital readmissions that is based on a regression-kink design. We established the validity of this approach. In contrast to past research, our results suggest that the HRRP had no clinically or statistically significant effects on hospital readmissions. While our findings differ from prior research, they are consistent with implementation imperfections that characterize the HRRP (e.g., imperfect risk adjustment method), the known difficulty of identifying ex ante hospital readmissions that are preventable, and the absence of consistent evidence on interventions that prevent readmissions. Therefore, our findings are not surprising while the relatively large findings in past studies are.

Footnotes

Acknowledgments

We thank Atul Gupta for sharing information about the analysis in .

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Partial funding for this research was provided by the Agency for Health Research and Quality (AHRQ) under award R01 HS025586.

ORCID iD

Robert Kaestner

Notes

Algebra Underlying Specification Test of DDD

The model is estimated on aggregate data at state (j) and year (t) level. For the treated group (18)

Y_{j t} = α_{t} + β_{j} + γ P o l i c y_{j t} + λ Z_{j t} + u_{j t}, T R E A T = 1

Data on Z are unavailable (or only partly available). So observed model is (19)

Y_{j t} = {\tilde{α}}_{t} + {\tilde{β}}_{j} + \tilde{γ} P o l i c y_{j t} + {\tilde{u}}_{j t}, T R E A T = 1

The coefficient on policy is biased by omitted variable. (20a)

\begin{array}{l} Y_{j t} = {\tilde{α}}_{t} + {\tilde{β}}_{t} + \tilde{γ} P o l i c y_{j t} + {\tilde{u}}_{j t}, T R E A T = 1 \\ \tilde{γ} = γ + λ δ \\ P o l i c y_{j t} = π_{j} + r_{t} + δ Z_{j t} + u_{j t} \end{array}

For the untreated group (21)

\begin{array}{l} Y_{j t} = {α^{'}}_{t} + {β^{'}}_{j} + k P o l i c y_{j t} + σ Z_{j t} + ν_{j t}, T R E A T = 0 \\ k = 0 \end{array}

Note that policy is included in this specification, but it has a 0 coefficient because policy has no effect on control group. Also note that I allow the effect of unobservable to be different in untreated and treated groups (σ ≠ λ).

The observed model is (21a)

\begin{array}{l} Y_{j t} = {α^{″}}_{t} + {β^{″}}_{j} + k^{'} P o l i c y_{j t} + ν_{j t}, T R E A T = 0 \\ k^{'} = 0 + σ δ \\ P o l i c y_{j t} = π_{j} + r_{t} + δ Z_{j t} + u_{j t} \end{array}

Taking the difference (DDD) between treated (DD) and untreated (DD) groups and estimating the model yields (22)

\begin{array}{l} Y_{j t} (T = 1) - Y_{j t} (T = 0) = ({\tilde{α}}_{t} - {α^{″}}_{t}) + ({\tilde{β}}_{j} - {β^{″}}_{j}) + (\tilde{γ} - k^{'}) P o l i c y_{j t} + ϖ_{j t} \\ \tilde{γ} - k^{'} = γ + (λ - σ) δ \\ \tilde{γ} - k^{'} = γ, i f, (λ = σ) \end{array}

This is well known result that the DDD estimate is correct if the unobservable variable has same effect on Y for treated and untreated groups.

If we go back to first-difference specification, but use the true model, then we have (23)

Y_{j t} (T = 1) - Y_{j t} (T = 0) = ({\tilde{α}}_{t} - {α^{'}}_{t}) + ({\tilde{β}}_{j} - {β^{'}}_{j}) + γ P o l i c y_{j t} + (λ - σ) Z_{j t} + ϖ_{j t}

and if we rewrite (6) moving Y(T = 0) to the right-hand side, then we have (24)

\begin{array}{l} Y_{j t} (T = 1) = ρ Y_{j t} (T = 0) + ({\tilde{α}}_{t} - {α^{'}}_{t}) + ({\tilde{β}}_{j} - {β^{'}}_{j}) + γ P o l i c y_{j t} + (λ - σ) Z_{j t} + ϖ_{j t} \\ ρ = 1 \end{array}

The observed version of this model is (25)

\begin{array}{l} Y_{j t} (T = 1) = \ddot{ρ} Y_{j t} (T = 0) + π_{t} + ϕ_{j} + \ddot{γ} P o l i c y_{j t} + ϖ_{j t} \\ P o l i c y_{j t} = π_{j} + r_{t} + δ Z_{j t} + c Y_{j t} (T = 0) + u_{j t} \\ P o l i c y_{j t} = π_{j} + r_{t} + δ Z_{j t} + u_{j t}, c = 0 \\ \ddot{γ} = γ + (λ - σ) δ \\ Y_{j t} (T = 0) = π_{j} + r_{t} + σ Z_{j t} + k P o l i c y_{j t} + u_{j t} \\ Y_{j t} (T = 0) = π_{j} + r_{t} + σ Z_{j t} + u_{j t}, k = 0 (see equation 3) \\ \ddot{ρ} = 1 + (λ - σ) σ \end{array}

The main result from (25) is that if the coefficient $(\ddot{ρ})$ on Y(T = 0) is not equal to 1, then we reject the underlying assumption of the DDD model.

One limitation of this test of validity of the DDD model is that the bias term $(λ - σ)$ is scaled by $σ$ , which is the effect of the unmeasured confounder on the outcome for the untreated group (TREAT = 0). Consider an extreme case: $σ = 0$ and $(λ \neq σ)$ . The DDD estimate is biased $(bias is λ δ)$ , but the coefficient on $\ddot{ρ} = 1 + (λ - σ) σ$ is equal to one, which suggests no bias. Thus, rejecting $(\ddot{ρ} = 1)$ is evidence that DDD is not valid, but failure to reject may not indicate that it is valid. Table A.1.

OLS, Just Identified IV, and Event-Study IV Estimates of the Effect of HRRP Hospital-Wide Penalty on Condition-Specific Readmissions 2008–2014.

	Actual penalty					Expected penalty
OLS (difference-in-difference)	AMI	HF	PN	All targeted	Non-targeted	AMI	HF	PN	All targeted	Non-targeted
Hospital-wide penalty	−0.008*** (0.002)	−0.009*** (0.0014)	−0.008*** (0.0012)	−0.009*** (0.001)	−0.003*** (0.0006)	−0.034*** (0.005)	−0.028*** (0.004)	−0.010*** (0.004)	−0.026*** (0.003)	−0.031*** (0.002)
IV—just identified model
Hospital-wide penalty	−0.045*** (0.005)	−0.039*** (0.004)	−0.013*** (0.004)	−0.031*** (0.004)	−0.037*** (0.004)	−0.087*** (0.014)	−0.038*** (0.005)	−0.064*** (0.010)	−0.070*** (0.007)	−0.041*** (0.042)
IV—Event-study IV
Hospital-wide penalty X 2008	0.044*** (0.017)	0.041*** (0.006)	0.009 (0.007)	0.039*** (0.005)	0.033*** (0.003)	0.058*** (0.019)	0.096*** (0.014)	0.018 (0.015)	0.075*** (0.008)	0.036*** (0.003)
Hospital-wide penalty X 2009	0.045*** (0.015)	0.016*** (0.005)	0.01 (0.008)	0.020*** (0.004)	0.021*** (0.022)	0.058*** (0.017)	0.037*** (0.0122)	0.018 (0.016)	0.039*** (0.008)	0.024*** (0.002)
Hospital-wide penalty X 2010	0.010 (0.015)	0.011*** (0.005)	0.005 (0.008)	0.012*** (0.004)	0.008*** (0.002)	−0.013 (0.020)	0.025*** (0.012)	0.009 (0.015)	0.021*** (0.008)	0.009*** (0.002)
Hospital-wide penalty X 2012	−0.029+ (0.017)	0.001 (0.005)	−0.024*** (0.008)	−0.017*** (0.004)	−0.008*** (0.002)	−0.036* (0.020)	0.002 (0.012)	−0.044*** (0.015)	−0.032*** (0.008)	−0.009*** (0.002)
Hospital-wide penalty X 2013	−0.041*** (0.018)	−0.010*** (0.005)	−0.030*** (0.009)	−0.021*** (0.005)	−0.026*** (0.003)	−0.053*** (0.021)	−0.026*** (0.012)	−0.057*** (0.016)	−0.041*** (0.008)	−0.029*** (0.003)
Hospital-wide penalty X 2014	−0.043*** (0.017)	−0.025*** (0.005)	−0.032*** (0.009)	−0.032*** (0.005)	−0.031*** (0.004)	−0.050*** (0.020)	−0.061*** (0.013)	−0.060*** (0.017)	−0.063*** (0.009)	−0.035*** (0.003)
Mean dependent variable	0.182	0.215	0.162	0.195	0.181	0.182	0.215	0.162	0.195	0.181
Number of observations	15,203	21,099	21,706	22,279	22,279	19,150	21,768	21,731	22,279	22,279

Notes: The unit of observation is the hospital-year level. The regressions include patients admitted for each relevant condition between July 2008 and July 2014. AMI = acute myocardial infarction; HF = heart failure; and PN = pneumonia. All regressions include hospital fixed effects, year fixed effects and a set of controls for patient characteristics that vary over time for each hospital (indicators for age under 65, age 65–70, age 70–75, age 80–85, age over 85, share Hispanic, share Black, share White, share other race, share female). The instrument(s) for IV estimates is (are) the interaction(s) between 2006 and 2007 hospital-wide readmissions and indicator(s) for post-HRRP period (year). Robust standard errors are constructed allowing for non-independence at the hospital level. Each regression is weighted by the number of patients admitted for the relevant condition (AMI, HF, and PN) in the hospital year. * 0.05 < p-value ≤0.10, ** p ≤ 0.05, *** p ≤ 0.01.

Table A.2.

Just Identified and Over Identified IV Estimates of the Effect of HRRP of HRRP Hospital-wide Penalty on Condition-Specific Readmissions 2008–2014.

	Actual penalty					Expected penalty
IV—Just identified model	AMI	HF	PN	All Targeted	Non-targeted	AMI	HF	PN	All Targeted	Non-targeted
Hospital-wide penalty	−0.045*** (0.005)	−0.039*** (0.004)	−0.013*** (0.004)	−0.031*** (0.004)	−0.037*** (0.004)	−0.087*** (0.014)	−0.038*** (0.005)	−0.064*** (0.010)	−0.070*** (0.007)	−0.041*** (0.042)
IV—Over identified model
Hospital-wide penalty	−0.073*** (0.020)	−0.048*** (0.008)	−0.037*** (0.007)	−0.048*** (0.006)	−0.029*** (0.003)	−0.038*** (0.007)	−0.043*** (0.006)	−0.021*** (0.006)	−0.051*** (0.004)	−0.031*** (0.002)
Share Black 2006–2007 X post	0.013 (0.012)	−0.003 (0.005)	−0.003 (0.006)	−0.004 (0.005)	0.0001 (0.003)	−0.001 (0.01)	−0.0007 (0.004)	−0.003 (0.006)	0.006** (0.003)	0.006*** (0.002)
Share Hispanic 2006–2007 X post	0.005 (0.03)	−0.007 (0.020)	−0.001 (0.013)	−0.012 (0.013)	−0.001 (0.008)	0.023 (0.024)	0.005 (0.015)	0.0003 (0.012)	<0.000 (0.01)	0.009 (0.006)
Share dual eligible 2006–2007 X post	0.026 (0.020)	0.020*** (0.010)	0.021*** (0.007)	0.025*** (0.007)	0.009** (0.004)	−0.005 (0.011)	0.003 (0.006)	0.007 (0.008)	0.012*** (0.004)	0.0002 (0.002)
Medium group diabetes 2006–2007 X post	<0.000 (0.003)	0.005*** (0.002)	0.002 (0.002)	0.0015 (0.002)	0.002+ (0.001)	−0.006*** (0.002)	0.004*** (0.001)	0.0003 (0.001)	0.001 (0.001)	0.001** (0.0006)
High group diabetes 2006–2007 X post	<0.000 (0.004)	0.007*** (0.002)	0.002 (0.002)	0.003 (0.002)	0.002** (0.001)	−0.003 (0.002)	0.004*** (0.001)	<0.000 (0.002)	0.002 (0.001)	0.002*** (0.0008)
Medium group pneumonia 2006–2007 X post	<0.000 (0.003)	<0.000 (0.002)	<0.000 (0.002)	<0.000 (0.002)	−0.001 (0.001)	−0.002 (0.002)	−0.0009 (0.001)	−0.002 (0.001)	<0.000 (0.001)	−0.001*** (0.0006)
High group pneumonia 2006–2007 X post	<0.000 (0.004)	<0.000 (0.002)	−0.002 (0.002)	<0.000 (0.002)	−0.002+ (0.001)	−0.003 (0.002)	0.0002 (0.002)	−0.002 (0.016)	<0.000 (0.001)	−0.002*** (0.0007)
Teaching 2006–2007 X post	0.001 (0.003)	<0.000 (0.002)	−0.001 (0.001)	<0.000 (0.002)	<0.000 (0.001)	−0.001 (0.002)	−0.0007 (0.001)	−0.002+ (0.001)	−0.001 (0.001)	−0.001 (0.0006)
Medium group bed size 2006–2007 X post	0.012** (0.005)	0.007*** (0.002)	<0.000 (0.002)	0.002 (0.002)	0.004*** (0.001)	−0.0002 (0.003)	−0.001 (0.002)	−0.008 (0.001)	<0.000 (0.001)	0.003*** (0.0007)
High group bed size 2006–2007 X post	0.0131** (0.005)	0.007*** (0.003)	−0.003 (0.002)	0.001 (0.002)	0.005*** (0.001)	0.003 (0.003)	−0.001 (0.002)	−0.003 (0.006)	<0.000 (0.001)	0.004*** (0.0008)
Mean dependent variable	0.182	0.215	0.162	0.195	0.181	0.182	0.215	0.162	0.195	0.181
Number of observations	19,150	21,768	21,731	22,279	22,279	19,150	21,768	21,731	22,279	22,279

Notes: The unit of observation is the hospital-year level. The regressions include patients admitted for each relevant condition between July 2008 and July 2014. AMI = acute myocardial infarction; HF = heart failure; and PN = pneumonia. All regressions include hospital fixed effects, year fixed effects and a set of controls for patient characteristics that vary over time for each hospital (indicators for age under 65, age 65–70, age 70–75, age 80–85, age over 85, share Hispanic, share Black, share White, share other race, share female). The instrument(s) for IV estimates is (are) the interaction(s) between 2006 and 2007 hospital-wide readmissions and indicator(s) for post-HRRP period (year). Also added to model are interactions of baseline 2006–2007 characteristics (share Black, share Hispanic, share dual eligible, share with diabetes comorbidity, share with pneumonia comorbidity, teaching status, and bed count) and post-HRRP indicator. Robust standard errors are constructed allowing for non-independence at the hospital level. Each regression is weighted by the number of patients admitted for the relevant condition (AMI, HF, and PN) in the hospital year. * 0.05 < p-value ≤0.10, ** p ≤ 0.05, *** p ≤ 0.01.

Table A.3.

Reduced Form Regression-Kink Estimates of the Effect of the HRRP on Readmissions Pre- and Post-HRRP Round 1 Penalty With and Without Additional Controls.

Pre-HRRP 2010	AMI		HF		PN
Excess readmissions ratio 2009–2011	0.002*** (0.0005)	0.002*** (0.0006)	0.004*** (0.0002)	0.004*** (0.0003)	0.003*** (0.0002)	0.003*** (0.0003)
Excess readmissions ratio*Penalty threshold	0.002 (0.001)	0.002 (0.002)	0.001 (0.001)	0.001 (0.001)	−0.0004 (0.0006)	−0.0004 (0.0007)
Pre-HRRP 2011
Excess readmissions ratio 2009–2011	0.002*** (0.0006)	0.002*** (0.0007)	0.004*** (0.0002)	0.004*** (0.0003)	0.003*** (0.0003)	0.003*** (0.0003)
Excess readmissions ratio*Penalty threshold	0.002 (0.002)	0.002 (0.002)	−0.001 (0.001)	−0.001 (0.001)	0.0002 (0.0006)	0.0002 (0.0006)
Post-HRRP 2012
Excess readmissions ratio 2009–2011	0.0003 (0.0006)	0.0005 (0.0007)	0.002*** (0.0003)	0.002*** (0.0004)	0.001*** (0.0003)	0.001*** (0.0003)
Excess readmissions ratio*Penalty threshold	0.001 (0.002)	0.001 (0.002)	−0.002 (0.001)	−0.002 (0.001)	−0.0004 (0.0007)	−0.0002 (0.0007)
Post-HRRP 2013
Excess readmissions ratio 2009–2011	0.0009 (0.0005)	0.001 (0.0006)	0.001*** (0.0003)	0.001*** (0.004)	0.002*** (0.0003)	0.002*** (0.0003)
Excess readmissions ratio*Penalty threshold	0.0003 (0.001)	−0.0004 (0.002)	−0.001 (0.001)	−0.001 (0.001)	−0.001 (0.0007)	−0.001 (0.0007)
Post-HRRP 2014
Excess readmissions ratio 2009–2011	0.0006 (0.0006)	0.001 (0.0008)	0.001*** (0.0003)	0.001*** (0.0004)	0.002*** (0.0003)	0.001*** (0.0004)
Excess readmissions ratio*Penalty threshold	0.0006 (0.001)	−0.0005 (0.002)	−0.002* (0.001)	−0.002* (0.001)	−0.001 (0.001)	−0.003 (0.001)
Additional controls	No	Yes	No	Yes	No	Yes
Mean dependent variable	0.178	0.178	0.202	0.202	0.145	0.145
Number of observations	640	640	708	708	720	720

Notes: Each column represents a separate regression. For each condition (AMI, heart failure, and pneumonia), the sample of hospitals differs and includes hospitals that were penalized because of the condition (e.g., AMI) and hospitals who were not penalized at all. For each condition, we report estimates from a model with a modest level of hospital baseline (2007) level controls (see Table 7); share in each age bracket (5 age brackets), race/ethnicity (share white, share Hispanic, share black) and gender (share female). In a second model, we add the hospital baseline (2007) level share of patients with pneumonia comorbidities (tertiles), share of patients with diabetes comorbidities (tertiles), share dual eligible (in tertiles), and number of beds (in tertiles). Hospitals with less than 50 cases throughout the 3-year performance period (June 2008 to July 2011, i.e., 2009–2011) are excluded. Coefficient estimates show the effect of a 0.01 change in the excess readmission ratio and the effect is allowed to differ before and after the HRRP penalty threshold. * 0.05 < p-value ≤0.10, ** p ≤ 0.05, *** p ≤ 0.01.

References

Alimadadi

Abbasinia

Mohammadbeigi

Abbasi

(2020). Risk factors of readmission after coronary artery bypass graft surgery: A case-control study. Nursing Practice Today, 7(4), 295-301.

American Hospital Association (2015). Rethinking the Hospital Readmissions Reduction Program.

Auerbach

A. D.

Kripalani

Vasilevskis

E. E.

Sehgal

Lindenauer

P. K.

Metlay

J. P.

Fletcher

Ruhnke

G. W.

Flanders

S. A.

Kim

Williams

M. V.

Thomas

Giang

Herzig

S. J.

Patel

Boscardin

W. J.

Robinson

E. J.

Schnipper

J. L.

(2016). Preventability and causes of readmissions in a national cohort of general medicine patients. JAMA Internal Medicine, 176(4), 484–493.

Bailey

(IBM Watson Health) Weiss

(IBM Watson Health) Barrett

(M.L. Barrett, Inc.) Jiang

(AHRQ) . (2019). Characteristics of 30-Day Readmissions, 2010-2016. HCUP Statistical Brief #248. Agency for Healthcare Research and Quality. www.hcup-us.ahrq.gov/reports/statbriefs/sb248-Hospital-Readmissions-2010-2016.pdf.

Bartik

T. J.

(1991). Who benefits from state and local economic development policies? W.E. Upjohn Institute for Employment Research.

Brewster

A. L.

Cherlin

E. J.

Ndumele

C. D.

Collins

Burgess

J. F.

Jr. Charns

M. P.

Bradley

E. H.

Curry

L. A.

(2016). What works in readmissions reduction. Medical Care, 54(6), 600–607

Card

Lee

D. S.

Pei

Weber

(2015). Inference on causal effects in a generalized regression kink design. Econometrica, 83, 2453–2483.

Carey

Lin

M.-Y.

(2015). Readmissions to New York hospitals fell for three target conditions from 2008 to 2012, consistent with Medicare goals. Health Affairs, 34(6), 978–985.

Chamberlain

R. S.

Sond

Mahendraraj

Lau

C. S.

Siracuse

B. L.

(2018). Determining 30-day readmission risk for heart failure patients: the readmission after heart failure scale. International Journal of General Medicine, 11, 127–141.

10.

Chen

Grabowski

D. C.

(2019). Hospital readmissions reduction program: Intended and unintended effects. Medical Care Research and Review, 76(5), 643–660.

11.

Chhabra

K. R.

Ibrahim

A. M.

Thumma

J. R.

Ryan

A. M.

Dimick

J. B.

(2019). Impact of medicare readmissions penalties on targeted surgical conditions. Health Affairs, 38(7), 1207–1215.

12.

Chin

D. L.

Bang

Manickam

R. N.

Romano

P. S.

(2016). Rethinking thirty-day hospital readmissions: Shorter intervals might be better indicators of quality of care. Health Affairs, 35(10), 1867–1875.

13.

Cromwell

Trisolini

M. G.

Pope

G. C.

Mitchell

J. B.

Greenwald

L. M.

(Eds.). (2011). Pay for performance in health care: Methods and approaches. RTI Press.

14.

Desai

N. R.

Ross

J. S.

Kwon

J. Y.

Herrin

Dharmarajan

Bernheim

S. M.

Krumholz

H. M.

Horwitz

L. I.

(2016). Association between hospital penalty status under the hospital readmission reduction program and readmission rates for target and nontarget conditions. JAMA, 316(24), 2647–2656.

15.

Demiralp

Koenig

(2018). Further evidence on the system‐wide effects of the hospital readmissions reduction program. Health Services Research, 53(3), 1478–1497.

16.

Doran

Maurer

K. A.

Ryan

A. M.

(2016). Impact of provider incentives on quality and cost of health care. Annual Review of Public Health, 38, 449–465.

17.

Fingar

K. R.

Barrett

M. L.

Jiang

H. J.

(2017). A comparison of all-cause 7-day and 30-day readmissions, 2014: statistical brief #230. Healthcare Cost and Utilization Project (HCUP).

18.

Finkelstein

Zhou

Taubman

Doyle

(2020). Health care hotspotting -- A randomized, controlled trial. New England Journal of Medicine, 382(2), 152–162.

19.

Ganong

Jäger

(2018). A permutation test for the regression kink design. Journal of the American Statistical Association, 113(522), 494–504

20.

Goldsmith-Pinkham

Sorkin

Swift

(2018). Bartik instruments: What, when, why, and how. (No. w24408). National Bureau of Economic Research.

21.

Graham

K. L.

Auerbach

A. D.

Schnipper

J. L.

Flanders

S. A.

Kim

C. S.

Robinson

E. J.

Ruhnke

G. W.

Thomas

L. R.

Kripalani

Vasilevskis

E. E.

Fletcher

G. S.

Sehgal

N. J.

Lindenauer

P. K.

Williams

M. V.

Metlay

J. P.

Davis

R. B.

Yang

Marcantonio

E. R.

Herzig

S. J.

(2018). Preventability of early versus late hospital readmissions in a national cohort of general medicine patients. Annals of Internal Medicine, 168(11), 766–774.

22.

Gupta

(2021). Impacts of performance pay for hospitals: The readmissions reduction program. American Economic Review, 111(4), 1241–1283.

23.

Ibrahim

A. M.

Dimick

J. B.

Sinha

S. S.

Hollingsworth

J. M.

Nuliyalu

Ryan

A. M.

(2018). Association of coded severity with readmission reduction after the hospital readmissions reduction program. JAMA Internal Medicine, 178(2), 290–292.

24.

Institute of Medicine . (2007). Rewarding provider performance: Aligning incentives in medicare. Washington, DC: The National Academies Press.

25.

Jaeger

D. A.

Joyce

T. J.

Kaestner

(2020). A cautionary tale of evaluating identifying assumptions: Did reality TV really cause a decline in teenage childbearing? Journal of Business & Economic Statistics, 38(2), 317–326

26.

Joynt

K. E.

Figueroa

J. E.

Oray

Jha

A. K.

(2016). Opinions on the hospital readmission reduction program: Results of a national survey of hospital leaders. The American Journal of Managed Care, 22(8), e287.

27.

Kansagara

Englander

Salanitro

Kagen

Theobald

Freeman

Kripalani

(2011). Risk prediction models for hospital readmission: A systematic review. JAMA, 306(15), 1688–1698.

28.

Kripalani

Theobald

C. N.

Anctil

Vasilevskis

E. E.

(2014). Reducing hospital readmission rates: Current strategies and future directions. Annual Review of Medicine, 65, 471–485. doi:10.1146/annurev-med-022613-090415

29.

Krumholz

H. M.

Normand

S. T.

Keenan

P. S.

Desai

M. M.

Lin

Drye

E. E.

Bhat

K. R.

Schreiner

G. C.

(2008). Hospital 30-day acute myocardial infarction readmission measure: methodology. Report prepared for the Centers for Medicare & Medicaid Services.

30.

Landais

(2015). Assessing the welfare effects of unemployment benefits using the regression kink design. American Economic Journal: Economic Policy, 7(4), 243–278.

31.

Lisk

Stensland

(2018). Mandated report: Effects of the hospital readmissions reduction program. In Advising the Congress on Medicare Issues. S

32.

McGarry

B. E.

Blankley

A. A.

(2016). The impact of the medicare hospital readmission reduction program in New York state. Medical Care, 54(2), 162–171.

33.

MedPAC . 2013. Report to the congress: Medicare and the health care delivery system; chapter 4, refining the hospital readmissions reduction program (pp. 91–110). MedPAC. Available at http://www.medpac.gov/documents/reports/jun13_entirereport.pdf?sfvrsn=0 (accessed May 28 2014)

34.

Mellor

Daly

Smith

(2017). Does it pay to penalize hospitals for excess readmissions? Intended and unintended consequences of medicare’s hospital readmissions reductions program. Health Economics, 26(8), 1037–1051.

35.

Mendelson

Kondo

Damberg

Low

Motúapuaka

Freeman

O’Neil

Relevo

Kansagara

(2017). The effects of pay-for-performance programs on health, health care use, and processes of care: A systematic review. Annals of Internal Medicine, 166(5), 341–353.

36.

Ody

Msall

Dafny

L. S.

Grabowski

D. C.

Cutler

D. M.

(2019). Decreases in readmissions credited to Medicare’s program to reduce hospital readmissions have been overstated. Health Affairs, 38(1), 36–43.

37.

Siracuse

B. L.

Chamberlain

R. S.

(2016). A preoperative scale for determining surgical readmission risk after total hip replacement. JAMA Surgery, 151(8), 701–709.

38.

Smith

L. N.

Makam

A. N.

Darden

Mayo

Das

S. R.

Halm

E. A

Nguyen

O. K.

(2018). Acute myocardial infarction readmission risk prediction models: A systematic review of model performance. Circulation: Cardiovascular Quality and Outcomes, 11(1), e003885.

39.

Soumerai

Sullivan

(2019). Ignoring evidence begot Medicare’s dangerous hospital readmissions penalty. STAT News.

40.

Thompson

M. P.

Waters

T. M.

Kaplan

C. M.

Cao

Bazzoli

G. J.

(2017). Most hospitals received annual penalties for excess readmissions, but some fared better than others. Health Affairs, 36(5), 893–901.

41.

Van Walraven

Bennett

Jennings

Austin

P. C.

Forster

A. J.

(2011). Proportion of hospital readmissions deemed avoidable: A systematic review. Canadian Medical Association Journal, 183(7), E391–E402.

42.

Vest

J. R.

Gamm

L. D.

Oxford

B. A.

Gonzalez

M. I.

Slawson

K. M.

(2010). Determinants of preventable readmissions in the United States: A systematic review. Implementation Science, 5, 88.

43.

Wasfy

J. H.

Zigler

C. M.

Choirat

Wang

Dominici

Yeh

R. W.

(2017). Readmission rates after passage of the hospital readmissions reduction program: A pre–post analysis. Annals of Internal Medicine, 166(5), 324–331.

44.

Weinreich

Nguyen

O. K.

Wang

Mayo

Mortensen

E. M.

Halm

E. A.

Makam

A. N.

(2016). Predicting the risk of readmission in pneumonia. A systematic review of model performance. Annals of the American Thoracic Society, 13(9), 1607–1614.

45.

Ziedan

(2018) (In preparation). The Intended and Unintended Consequences of the Hospital Readmission Reduction Program. SSRN.

46.

Zingmond

D. S.

Liang

L. J.

Parikh

Escarce

J. J.

(2018). The impact of the hospital readmissions reduction program across insurance types in California. Health Services Research, 53(6), 4403–4415.

47.

Zuckerman

R. B.

Sheingold

S. H.

Orav

E. J.

Ruhter

Epstein

A. M.

(2016). Readmissions, observation, and the hospital readmissions reduction program. New England Journal of Medicine, 374(16), 1543–1551.