Quantifying Sensitivity to Selection on Unobserved Covariates: Recasting the Coefficient of Proportionality Within a Correlational Framework

Abstract

Sensitivity analyses can inform evidence-based policy by quantifying the hypothetical conditions necessary to change an inference. Perhaps the most prevalent index used for sensitivity analyses is Oster’s Coefficient of Proportionality (COP) which expresses how strong selection on unobserved covariates would have to be relative to selection on observed covariates to nullify an estimated effect. But Oster has been critiqued based on its two-stage conceptualization of the COP and its estimation of the COP based on coefficient stability across estimated models. In this article, we reconceptualize the COP as a function of unobserved covariates’ correlations with the focal predictor (e.g., treatment) and with the outcome. Our correlation-based approach addresses the critiques of Oster while preserving the comparison of selection on unobserved covariates to selection on observed covariates. As importantly, our expressions do not depend on analysts’ subjective choices of covariates to include in a baseline model, are adapted to a threshold for inference based on statistical significance, and can be directly calculated from conventionally reported quantities (e.g., estimated effect, standard error) through the Konfound packages in R or Stata or the R-shiny app https://konfound-project.shinyapps.io/konfound-it/. Thus, for most published studies in the social sciences our correlation-based COP index can be easily applied and intuitively interpreted.

Keywords

sensitivity analysis coefficient of proportionality correlation

Get full access to this article

View all access options for this article.

References

Alberg

A. J.

Shopland

D. R.

Cummings

K. M.

(2014). The 2014 Surgeon General’s report: Commemorating the 50th Anniversary of the 1964 Report of the Advisory Committee to the US Surgeon General and updating the evidence on the health consequences of cigarette smoking. American Journal of Epidemiology, 179(4), 403–412.

Altonji

J. G.

Elder

T. E.

Taber

C. R.

(2005). Selection on observed and unobserved variables: Assessing the effectiveness of Catholic schools. Journal of Political Economy, 113(1), 151–184.

Barnow

B. S.

Cain

G. G.

Goldberger

A. S.

(1980). Issues in the analysis of selectivity bias (Vol. 4). University of Wisconsin, Inst. for Research on Poverty.

Basu

(2023). A critical assessment of a popular econometric method for sensitivity analysis. Available at SSRN 4540836.

Berridge

(2006). The policy response to the smoking and lung cancer connection in the 1950s and 1960s. The Historical Journal, 49(4), 1185–1209.

Bonett

D. G.

(2008). Meta-analytic interval estimation for bivariate correlations. Psychological Methods, 13(3), 173–181. https://doi.org/10.1037/a0012868

Breslau

DelDotto

J. E.

Brown

G. G.

Kumar

Ezhuthachan

Hufnagle

K. G.

Peterson

E. L.

(1994). A gradient relationship between low birth weight and IQ at age 6 years. Archives of Pediatrics & Adolescent Medicine, 148(4), 377–383.

Carnegie

N. B.

Harada

Hill

J. L.

(2016). Assessing sensitivity to unmeasured confounding using a simulated potential confounder. Journal of Research on Educational Effectiveness, 9(3), 395–420.

Cinelli

Hazlett

(2020). Making sense of sensitivity: Extending omitted variable bias. Journal of the Royal Statistical Society: Series B, 82(1), 39–67.

10.

Cohen

(1983). Applied multiple regression. Correlation Analysis for the Behavioral Sciences. Lawrence Erlbaum.

11.

Cornfield

Haenszel

Hammond

E. C.

Lilienfeld

A. M.

Shimkin

M. B.

Wynder

E. L.

(1959). Smoking and lung cancer: Recent evidence and a discussion of some questions. Journal of the National Cancer Institute, 22(1), 173–203.

12.

Diegert

Masten

M. A.

Poirier

(2022). Assessing omitted variable bias when the controls are endogenous. arXiv. arXiv preprint arXiv:2206.02303.

13.

Edmunds

J. A.

Unlu

Furey

Glennie

Arshavsky

(2020). What happens when you combine high school and college? The impact of the early college model on postsecondary performance and completion. Educational Evaluation and Policy Analysis, 42(2), 257–278.

14.

Edmunds

J. A.

Unlu

Phillips

Mulhern

Hutchins

B. C.

(2024). CTE-focused dual enrollment: Participation and outcomes. Education Finance and Policy, 19(4), 612–633.

15.

Fisher

R. A.

(1958). Lung cancer and cigarettes? Nature, 182(4628), 108.

16.

Frank

K. A.

(2000). Impact of a confounding variable on the inference of a regression coefficient. Sociological Methods and Research, 29(2), 147–194.

17.

Frank

K. A.

Lin

Maroulis

S. J.

(2025). Causal inferences from observational studies in education policy: Towards pragmatic social science. In Cohen-Vogel

Scott

Youngs

(Eds.), Handbook on education policy research (pp. 479–510). The American Educational Research Association.

18.

Frank

K. A.

Lin

Maroulis

Mueller

A. S.

Rosenberg

J. M.

Hayter

C. S.

Mahmoud

R. A.

Kolak

Dietz

Zhang

(2021). Hypothetical case replacement can be used to quantify the robustness of trial results. Journal of Clinical Epidemiology, 134, 150–159.

19.

Frank

K. A.

Lin

Maroulis

S. J.

Mueller

(2023). Quantifying the robustness of causal inferences: Sensitivity analysis for pragmatic social science. Social Science Research, 110, 102815.

20.

Frank

K. A.

Maroulis

Duong

Kelcey

(2013). What would it take to change an inference?: Using Rubin’s causal model to interpret the robustness of causal inferences. Education, Evaluation and Policy Analysis, 35, 437–460.

21.

Franks

D’Amour

Feller

(2019). Flexible sensitivity analysis for observational studies without observable implications. Journal of the American Statistical Association, 115, 1730–1746.

22.

Frisch

Waugh

F. V.

(1933). Partial time regressions as compared with individual trends. Econometrica: Journal of the Econometric Society, 1, 387–401.

23.

Gastwirth

J. L.

Krieger

A. M.

Rosenbaum

P. R.

(1998). Dual and simultaneous sensitivity analysis for matched pairs. Biometrika, 85(4), 907–920.

24.

Gross

R. T.

Spiker

Haynes

C. W.

(1997). Helping low birth weight, premature babies: The infant health and development program. Stanford University Press.

25.

Hirano

Imbens

G. W.

(2001). Estimation of causal effects using propensity score weighting: An application to data on right heart catheterization. Health Services and Outcomes Research Methodology, 2(3), 259–278.

26.

Holland

P. W.

(1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–960.

27.

Hong

Qin

Yang

(2018). Weighting-based sensitivity analysis in causal mediation studies. Journal of Educational and Behavioral Statistics, 43(1), 32–56.

28.

Hong

Yang

Qin

(2021). Did you conduct a sensitivity analysis? A new weighting-based approach for evaluations of the average treatment effect for the treated. Journal of the Royal Statistical Society: Series A (Statistics in Society), 184(1), 227–254.

29.

Hosman

C. A.

Hansen

B. B.

Holland

P. W.

(2010). The sensitivity of linear regression coefficients’ confidence limits to the omission of a confounder. The Annals of Applied Statistics, 4(2), 849–870.

30.

Hsu

J. Y.

Small

D. S.

(2013). Calibrating sensitivity analyses to observed covariates in observational studies. Biometrics, 69(4), 803–811.

31.

Ichino

Mealli

Nannicini

(2008). From temporary help jobs to permanent employment: What can we learn from matching estimators and their sensitivity? Journal of Applied Econometrics, 23(3), 305–327.

32.

Imai

Keele

Yamamoto

(2010). Identification, inference and sensitivity analysis for causal mediation effects. Statistical Science, 25(1), 51–71.

33.

Imbens

(2003). Sensitivity to exogeneity assumptions in program evaluation. American Economic Review, 93(2), 126–132.

34.

Johnson

R. A.

Wichern

D. W.

(2002). Applied multivariate statistical analysis. Prentice Hall.

35.

Knaeble

Dutter

(2017). Reversals of least-square estimates and model-invariant estimation for directions of unique effects. The American Statistician, 71(2), 97–105.

36.

Knaeble

Osting

Abramson

M. A.

(2020). Regression analysis of unmeasured confounding. Epidemiologic Methods, 9(1), 20190028.

37.

Kramer

M. S.

(1987). Intrauterine growth and gestational duration determinants. Pediatrics, 80(4), 502–511.

38.

Lin

D. Y.

Psaty

B. M.

Kronmal

R. A.

(1998). Assessing the sensitivity of regression results to unmeasured confounders in observational studies. Biometrics, 54, 948–963.

39.

Lonati

Wulff

J. N.

(2024). Hic Sunt Dracones: On the risks of comparing the ITCV with control variable correlations. Journal of Management, 52, 0149206324 1293126.

40.

Lovell

M. C.

(1963). Seasonal adjustment of economic time series and multiple regression analysis. Journal of the American Statistical Association, 58(304), 993–1010.

41.

Marcotte

D. E.

(2019). The returns to education at community colleges: New evidence from the Education Longitudinal Survey. Education Finance and Policy, 14(4), 523–547.

42.

Masten

M. A.

Poirier

(2022). The effect of omitted variables on the sign of regression coefficients. arXiv. arXiv preprint arXiv:2208.00552.

43.

Mauro

(1990). Understanding LOVE (left out variables error): A method for estimating the effects of omitted variables. Psychological Bulletin, 108(2), 314.

44.

Middleton

J. A.

Scott

M. A.

Diakow

Hill

J. L.

(2016). Bias amplification and bias unmasking. Political Analysis, 24(3), 307–323.

45.

National Research Council. (2000). From neurons to neighborhoods: The science of early childhood development. National Academies Press.

46.

Oster

(2019). Unobservable selection and coefficient stability: Theory and evidence. Journal of Business & Economic Statistics, 37(2), 187–204.

47.

Paredes

(2022). Mixed but not scrambled: Gender gaps in coed schools with single-sex classrooms. Journal of Research on Educational Effectiveness, 15(2), 330–366.

48.

Park

Esterling

K. M.

(2021). Sensitivity analysis for pretreatment confounding with multiple mediators. Journal of Educational and Behavioral Statistics, 46(1), 85–108.

49.

Pearl

(2009). Causality. Cambridge University Press.

50.

Redding

Grissom

J. A.

(2021). Do students in gifted programs perform better? Linking gifted program participation to achievement and nonachievement outcomes. Educational Evaluation and Policy Analysis, 43, 01623737211008919.

51.

Rice

J. A.

(2006). Mathematical statistics and data analysis. Cengage Learning.

52.

Robins

J. M.

Rotnitzky

Scharfstein

D. O.

(2000). Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models. In M. E. Halloran & D. A. Berry (Eds.), Statistical models in epidemiology, the environment, and clinical trials (pp. 1–94). Springer.

53.

Rosenbaum

P. R.

(2002). Observational studies. Springer.

54.

Rosenbaum

P. R.

(1986). Dropping out of high school in the United States: An observational study. Journal of Educational Statistics, 11(3), 207–224.

55.

Rosenbaum

P. R.

(2005). Sensitivity analysis in observational studies. Encyclopedia of Statistics in Behavioral Science, 4, 1809–1814.

56.

Rosenbaum

P. R.

Rubin

D. B.

(1983). Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome. Journal of the Royal Statistical Society: Series B (Methodological), 45(2), 212–218.

57.

Rosseel

(2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48, 1–36.

58.

Strawson

P. F.

(2008). Freedom and resentment and other essays. Routledge.

59.

U.S. Department of Health, Education, and Welfare. (1964). Smoking and Health: Report of the Advisory Committee to the Surgeon General of the Public Health Service. Washington, DC: US Department of Health, Education, and Welfare, Public Health Service (Public Health Service Publication No. 1103).

60.

US Public Health Service. Office of the Surgeon General, United States. Office on Smoking, Center for Chronic Disease Prevention, & Health Promotion (US). Office on Smoking. (1989). Reducing the Health Consequences of Smoking: 25 Years of Progress: A Report of the Surgeon General, Centers for Disease Control. Rockville, Maryland.

61.

VanderWeele

T. J.

Ding

(2017). Sensitivity analysis in observational research: introducing the E-value. Annals of Internal Medicine, 167(4), 268–274.

62.

Veitch

Zaveri

(2020). Sense and sensitivity analysis: Simple post-hoc analysis of bias due to unobserved confounding. Advances in Neural Information Processing Systems, 33, 10999–11009.

63.

Zachrisson

H. D.

Dearing

Borgen

N. T.

Sandsør

A. M. J.

Karoly

L. A.

(2023). Universal early childhood education and care for toddlers and achievement outcomes in middle childhood. Journal of Research on Educational Effectiveness, 17, 1–29.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

1.64 MB

0.00 MB