Empirical Treatment Effectiveness Models for Binary Outcomes

Randomized trials provide strong evidence regarding efficacy of interventions but are limited in their capacity to address potential heterogeneity in effectiveness within broad clinical populations. For example, a treatment that on average is superior may be distinctly worse in certain patients. We propose a technique for using large electronic health registries to develop and validate decision models that measure—for distinct combinations of covariate values—the difference in predicted outcomes among 2 alternative treatments. We demonstrate the methodology in a prototype analysis of in-hospital mortality under alternative revascularization treatments. First, we developed prediction models for a binary outcome of interest for each treatment. Decision criteria were then defined based on the treatment-specific model predictions. Patients were then classified as receiving concordant or discordant care (in relation to the model recommendation), and the association between discordance and outcomes was evaluated. We then present alternative decision criteria and validation methodologies, as well as sensitivity analyses that investigate 1) the imbalance between treatments on observed covariates and 2) the aggregate impact of unobserved covariates. Our methodology supplements population-average clinical trial results by modeling heterogeneity in outcomes according to specific covariate values. It thus allows for assessment of current practice, from which cogent hypotheses for improved care can be derived. Newly emerging large population registries will allow for accurate predictions of outcome risk under competing treatments, as complex functions of predictor variables. Whether or not the models might be used to inform decision making depends on the extent to which important predictors are available. Further work is needed to understand the strengths and limitations of this approach, particularly in relation to those based on randomized trials.