Propensity score estimators for the average treatment effect and the average treatment effect on the treated may yield very different estimates

Abstract

Objective

Propensity score matching is typically used to estimate the average treatment effect for the treated while inverse probability of treatment weighting aims at estimating the population average treatment effect. We illustrate how different estimands can result in very different conclusions.

Study design

We applied the two propensity score methods to assess the effect of continuous positive airway pressure on mortality in patients hospitalized for acute heart failure. We used Monte Carlo simulations to investigate the important differences in the two estimates.

Results

Continuous positive airway pressure application increased hospital mortality overall, but no continuous positive airway pressure effect was found on the treated. Potential reasons were (1) violation of the positivity assumption; (2) treatment effect was not uniform across the distribution of the propensity score. From simulations, we concluded that positivity bias was of limited magnitude and did not explain the large differences in the point estimates. However, when treatment effect varies according to the propensity score (E[Y(1)–Y(0)|g(X)] is not constant, Y being the outcome and g(X) the propensity score), propensity score matching ATT estimate could strongly differ from the inverse probability of treatment weighting-average treatment effect estimate. We show that this empirical result is supported by theory.

Conclusion

Although both approaches are recommended as valid methods for causal inference, propensity score-matching for ATT and inverse probability of treatment weighting for average treatment effect yield substantially different estimates of treatment effect. The choice of the estimand should drive the choice of the method.

Keywords

propensity score matching inverse probability weighting average treatment on the treated average treatment effect positivity assumption

Get full access to this article

View all access options for this article.

References

Gayat

Pirracchio

Resche-Rigon

. Propensity scores in intensive care and anaesthesiology literature: a systematic review. Intensive Care Med 2010; 36: 1993–2003.

Rosenbaum

Rubin

. The central role of the propensity score in observational studies for causal effects. Biometrika 1983; 70: 41–55.

Lunceford

Davidian

. Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study. Stat Med 2004; 23: 2937–2960.

Austin

. The performance of different propensity score methods for estimating marginal odds ratios. Stat Med 2007; 26: 3078–3094.

Austin

. The performance of different propensity-score methods for estimating relative risks. J Clin Epidemiol 2008; 61: 537–545.

Austin

. The performance of different propensity-score methods for estimating differences in proportions (risk differences or absolute risk reductions) in observational studies. Stat Med 2010; 29: 2137–2148.

Austin

. The relative ability of different propensity score methods to balance measured covariates between treated and untreated subjects in observational studies. Med Decis Making 2009; 29: 661–677.

Gayat E, Resche-Rigon M, Mary J-Y, et al. Propensity score applied to survival data analysis through proportional hazards models: a Monte Carlo study. Pharm Stat. Epub ahead of print 12 March 2012. DOI: 10.1002/pst.537.

Austin

. The performance of different propensity score methods for estimating marginal hazard ratios. Stat Med 2013; 32: 2837–2849.

10.

Rosenbaum

. Model-Based Direct Adjustment. J Am Stat Assoc 1987; 82: 387–394.

11.

Hirano

Imbens

. Estimation of causal effects using propensity score weighting: an application to data on right heart catheterization. Health Serv Outcomes Res Methodol 2001; 2: 259–278.

12.

Hirano

Imbens

Ridder

. Efficient estimation of average treatment effects using the estimated propensity score. Econometrica 2003; 71: 1161–1189.

13.

Imbens

. Nonparametric estimation of average treatment effects under exogeneity: a review. Rev Econ Stat 2004; 86: 4–29.

14.

Heckman

Vytlacil

. Policy-relevant treatment effects. Am Econ Rev 2001; 91: 107–111.

15.

Stürmer

Joshi

Glynn

. A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. J Clin Epidemiol 2006; 59: 437–447.

16.

Sato

Matsuyama

. Marginal structural models as a tool for standardization. Epidemiology 2003; 14: 680–686.

17.

Westreich

Cole

. Invited commentary: positivity in practice. Am J Epidemiol 2010; 171: 674–677. discussion 678–681.

18.

Kurth

Walker

Glynn

. Results of multivariable logistic regression, propensity matching, propensity adjustment, and propensity-based weighting under conditions of nonuniform effect. Am J Epidemiol 2006; 163: 262–270.

19.

Nieminen

Böhm

Cowie

. Executive summary of the guidelines on the diagnosis and treatment of acute heart failure: the Task Force on Acute Heart Failure of the European Society of Cardiology. Eur Heart J 2005; 26: 384–416.

20.

Mebazaa

Parissis

Porcher

. Short-term survival by treatment among patients hospitalized with acute heart failure: the global ALARM-HF registry using propensity scoring methods. Intensive Care Med 2011; 37: 290–301.

21.

Follath

Yilmaz

Delgado

. Clinical presentation, management and outcomes in the acute heart failure global survey of standard treatment (ALARM-HF). Intensive Care Med 2011; 37: 619–626.

22.

Schechtman

. Odds ratio, relative risk, absolute risk reduction, and the number needed to treat—which of these should we use? Value in Health 2002; 5: 431–436.

23.

Brookhart

Schneeweiss

Rothman

. Variable selection for propensity score models. Am J Epidemiol 2006; 163: 1149–1156.

24.

Abadie

Imbens

. Large sample properties of matching estimators for average treatment effects. Econometrica 2006; 74: 235–267.

25.

Petersen

Porter

Gruber

. Diagnosing and responding to violations in the positivity assumption. Stat Methods Med Res 2012; 21: 31–54.

26.

Austin

. A data-generation process for data with specified risk differences or numbers needed to treat. Commun Stat—Simul Comput 2010; 39: 563–577.

27.

Greenland

. Interpretation and choice of effect measures in epidemiologic analyses. Am J Epidemiol 1987; 125: 761–768.

28.

Austin

. Optimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studies. Pharm Stat 2011; 10: 150–161.

29.

Rosenbaum

Rubin

. Reducing bias in observational studies using subclassification on the propensity score. J Am Stat Assoc 1984; 79: 516–524.

30.

Bersten

Holt

Vedig

. Treatment of severe cardiogenic pulmonary edema with continuous positive airway pressure delivered by face mask. N Engl J Med 1991; 325: 1825–1830.

31.

Lin

Yang

Chiang

. Reappraisal of continuous positive airway pressure therapy in acute cardiogenic pulmonary edema. Short-term results and long-term follow-up. Chest 1995; 107: 1379–1386.

32.

Plaisance

Pirracchio

Berton

. A randomized study of out-of-hospital continuous positive airway pressure for acute cardiogenic pulmonary oedema: physiological and clinical effects. Eur Heart J 2007; 28: 2895–2901.

33.

Weng

C-L

Zhao

Y-T

Liu

Q-H

. Meta-analysis: noninvasive ventilation in acute cardiogenic pulmonary edema. Ann Intern Med 2010; 152: 590–600.

34.

Gray

Goodacre

Newby

. Noninvasive ventilation in acute cardiogenic pulmonary edema. N Engl J Med 2008; 359: 142–151.

35.

Lee

Lessler

Stuart

. Weight trimming and propensity score weighting. PLoS One 2011; 6: e18174–e18174.

36.

Robins

Hernán

Brumback

. Marginal structural models and causal inference in epidemiology. Epidemiology 2000; 11: 550–560.