Medical Decision Making

Decision-analytic models must often be informed using data that are only indirectly related to the main model parameters. The authors outline how to implement a Bayesian synthesis of diverse sources of evidence to calibrate the parameters of a complex model. A graphical model is built to represent how observed data are generated from statistical models with unknown parameters and how those parameters are related to quantities of interest for decision making. This forms the basis of an algorithm to estimate a posterior probability distribution, which represents the updated state of evidence for all unknowns given all data and prior beliefs. This process calibrates the quantities of interest against data and, at the same time, propagates all parameter uncertainties to the results used for decision making. To illustrate these methods, the authors demonstrate how a previously developed Markov model for the progression of human papillomavirus (HPV-16) infection was rebuilt in a Bayesian framework. Transition probabilities between states of disease severity are inferred indirectly from cross-sectional observations of prevalence of HPV-16 and HPV-16–related disease by age, cervical cancer incidence, and other published information. Previously, a discrete collection of plausible scenarios was identified but with no further indication of which of these are more plausible. Instead, the authors derive a Bayesian posterior distribution, in which scenarios are implicitly weighted according to how well they are supported by the data. In particular, we emphasize the appropriate choice of prior distributions and checking and comparison of fitted models.

Decision-analytic measures to assess clinical utility of prediction models and diagnostic tests incorporate the relative clinical consequences of true and false positives without the need for external information such as monetary costs. Net Benefit is a commonly used metric that weights the relative consequences in terms of the risk threshold at which a patient would opt for treatment. Theoretical results demonstrate that clinical utility is affected by a model’;s calibration, the extent to which estimated risks correspond to observed event rates. We analyzed the effects of different types of miscalibration on Net Benefit and investigated whether and under what circumstances miscalibration can make a model clinically harmful. Clinical harm is defined as a lower Net Benefit compared with classifying all patients as positive or negative by default. We used simulated data to investigate the effect of overestimation, underestimation, overfitting (estimated risks too extreme), and underfitting (estimated risks too close to baseline risk) on Net Benefit for different choices of the risk threshold. In accordance with theory, we observed that miscalibration always reduced Net Benefit. Harm was sometimes observed when models underestimated risk at a threshold below the event rate (as in underestimation and overfitting) or overestimated risk at a threshold above event rate (as in overestimation and overfitting). Underfitting never resulted in a harmful model. The impact of miscalibration decreased with increasing discrimination. Net Benefit was less sensitive to miscalibration for risk thresholds close to the event rate than for other thresholds. We illustrate these findings with examples from the literature and with a case study on testicular cancer diagnosis. Our findings strengthen the importance of obtaining calibrated risk models.

Background. To identify best-fitting input sets using model calibration, individual calibration target fits are often combined into a single goodness-of-fit (GOF) measure using a set of weights. Decisions in the calibration process, such as which weights to use, influence which sets of model inputs are identified as best-fitting, potentially leading to different health economic conclusions. We present an alternative approach to identifying best-fitting input sets based on the concept of Pareto-optimality. A set of model inputs is on the Pareto frontier if no other input set simultaneously fits all calibration targets as well or better. Methods. We demonstrate the Pareto frontier approach in the calibration of 2 models: a simple, illustrative Markov model and a previously published cost-effectiveness model of transcatheter aortic valve replacement (TAVR). For each model, we compare the input sets on the Pareto frontier to an equal number of best-fitting input sets according to 2 possible weighted-sum GOF scoring systems, and we compare the health economic conclusions arising from these different definitions of best-fitting. Results. For the simple model, outcomes evaluated over the best-fitting input sets according to the 2 weighted-sum GOF schemes were virtually nonoverlapping on the cost-effectiveness plane and resulted in very different incremental cost-effectiveness ratios ($79,300 [95% CI 72,500–87,600] v. $139,700 [95% CI 79,900–182,800] per quality-adjusted life-year [QALY] gained). Input sets on the Pareto frontier spanned both regions ($79,000 [95% CI 64,900–156,200] per QALY gained). The TAVR model yielded similar results. Conclusions. Choices in generating a summary GOF score may result in different health economic conclusions. The Pareto frontier approach eliminates the need to make these choices by using an intuitive and transparent notion of optimality as the basis for identifying best-fitting input sets.

The intervals between screens for the early detection of diseases such as breast and colon cancer suggested by screening guidelines are typically based on the average population risk of disease. With the emergence of ever more biomarkers for cancer risk prediction and the development of personalized medicine, there is a need for risk-specific screening intervals. The interval between successive screens should be shorter with increasing cancer risk. A risk-dependent optimal interval is ideally derived from a cost-effectiveness analysis using a validated simulation model. However, this is time-consuming and costly. We propose a simplified mathematical approach for the exploratory analysis of the implications of risk level on optimal screening interval. We develop a mathematical model of the optimal screening interval for breast cancer screening. We verified the results by programming the simplified model in the MISCAN-Breast microsimulation model and comparing the results. We validated the results by comparing them with the results of a full, published MISCAN-Breast cost-effectiveness model for a number of different risk levels. The results of both the verification and validation were satisfactory. We conclude that the mathematical approach can indicate the impact of disease risk on the optimal screening interval.

Background. Risk factors increase the incidence and severity of chronic disease. To examine future trends and develop policies addressing chronic diseases, it is important to capture the relationship between exposure and disease development, which is challenging given limited data. Objective. To develop parsimonious risk factor models embeddable in chronic disease models, which are useful when longitudinal data are unavailable. Design. The model structures encode relevant features of risk factors (e.g., time-varying, modifiable) and can be embedded in chronic disease models. Calibration captures time-varying exposures for the risk factor models using available cross-sectional data. We illustrate feasibility with the policy-relevant example of smoking in India. Methods. The model is calibrated to the prevalence of male smoking in 12 Indian regions estimated from the 2009–2010 Indian Global Adult Tobacco Survey. Nelder-Mead searches (250,000 starting locations) identify distributions of starting, quitting, and restarting rates that minimize the difference between modeled and observed age-specific prevalence. We compare modeled life expectancies to estimates in the absence of time-varying risk exposures and consider gains from hypothetical smoking cessation programs delivered for 1 to 30 years. Results. Calibration achieves concordance between modeled and observed outcomes. Probabilities of starting to smoke rise and fall with age, while quitting and restarting probabilities fall with age. Accounting for time-varying smoking exposures is important, as not doing so produces smaller estimates of life expectancy losses. Estimated impacts of smoking cessation programs delivered for different periods depend on the fact that people who have been induced to abstain from smoking longer are less likely to restart. Conclusions. The approach described is feasible for important risk factors for numerous chronic diseases. Incorporating exposure-change rates can improve modeled estimates of chronic disease outcomes and of the long-term effects of interventions targeting risk factors.

Background. Control of Clostridium difficile infection (CDI) is an increasingly difficult problem for health care institutions. There are commonly recommended strategies to combat CDI transmission, such as oral vancomycin for CDI treatment, increased hand hygiene with soap and water for health care workers, daily environmental disinfection of infected patient rooms, and contact isolation of diseased patients. However, the efficacy of these strategies, particularly for endemic CDI, has not been well studied. The objective of this research is to develop a valid, agent-based simulation model (ABM) to study C. difficile transmission and control in a midsized hospital. Methods. We develop an ABM of a midsized hospital with agents such as patients, health care workers, and visitors. We model the natural progression of CDI in a patient using a Markov chain and the transmission of CDI through agent and environmental interactions. We derive input parameters from aggregate patient data from the 2007–2010 Wisconsin Hospital Association and published medical literature. We define a calibration process, which we use to estimate transition probabilities of the Markov model by comparing simulation results to benchmark values found in published literature. Results. In a comparison of CDI control strategies implemented individually, routine bleach disinfection of CDI-positive patient rooms provides the largest reduction in nosocomial asymptomatic colonization (21.8%) and nosocomial CDIs (42.8%). Additionally, vancomycin treatment provides the largest reduction in relapse CDIs (41.9%), CDI-related mortalities (68.5%), and total patient length of stay (21.6%). Conclusion. We develop a generalized ABM for CDI control that can be customized and further expanded to specific institutions and/or scenarios. Additionally, we estimate transition probabilities for a Markov model of natural CDI progression in a patient through calibration.

Background. International guidelines recommend HIV treatment expansion in resource-limited settings, but funding availability is uncertain. We evaluated the performance of a model that forecasts lives saved through continued HIV treatment expansion in Haiti. Methods. We developed a computer-based, mathematical model of HIV disease and used incidence density analysis of patient-level Haitian data to derive model parameters for HIV disease progression. We assessed the internal validity of model predictions and internally calibrated model inputs when model predictions did not fit the patient-level data. We then derived uncertain model inputs related to diagnosis and linkage to care, pretreatment retention, and enrollment on HIV treatment through an external calibration process that selected input values by comparing model predictions to Haitian population-level data. Model performance was measured by fit to event-free survival (patient level) and number receiving HIV treatment over time (population level). Results. For a cohort of newly HIV-infected individuals with no access to HIV treatment, the model predicts median AIDS-free survival of 9.0 years precalibration and 6.6 years postcalibration v. 5.8 years (95% confidence interval, 5.1–7.0) from the patient-level data. After internal validation and calibration, 16 of 17 event-free survival measures (94%) had a mean percentage deviation between model predictions and the empiric data of <6%. After external calibration, the percentage deviation between model predictions and population-level data on the number on HIV treatment was <1% over time. Conclusions. Validation and calibration resulted in a good-fitting model appropriate for health policy decision making. Using local data in a policy model-building process is feasible in resource-limited settings.

Background. A nationwide skin cancer screening (SCS) was implemented in Germany in 2008. It aims at improving early detection of melanoma in order to reduce melanoma mortality. While the idea of early detection is compelling, demonstrating the effectiveness of melanoma screening is crucial. Since it is not feasible to conduct epidemiological studies to investigate the impact of factors such as screening interval or targeted age groups, we developed a microsimulation model that is able to predict melanoma mortality in Germany under several SCS conditions. Methods. Using German cancer registry data, population data, and other published data on melanoma progression and screening participation, we developed a stochastic microsimulation model. With this model, 10,000 populations of 100,000 persons and their melanoma-related life histories were simulated separately for women and for men and calibrated to observed melanoma incidence. In a second step, life histories of melanoma were changed by an SCS. Results. Compared with a nonscreening scenario (age-standardized mortality rate: 1.6 and 2.9 per 100,000 for women and men), a biennial SCS of the 35- to 85-year-old population with a participation probability of 20% for each screening-eligible individual and a test sensitivity of 80% reduced mortality by up to 0.7 and 1.4 deaths per 100,000 person-years for women and men, respectively, corresponding to a relative reduction of melanoma mortality by 45% in women and men. Conclusions. We developed a microsimulation model of melanoma mortality that can be used to evaluate the effectiveness of population-based skin cancer screening. The simulation provided plausible melanoma mortality predictions and is a tool for comparing different SCS scenarios. However, it does not provide estimates on total costs of SCS.

Background. Cancer is a major cause of morbidity and mortality, and colorectal cancer (CRC) is the third most common cancer in the world. The estimated costs of CRC treatment vary considerably, and if CRC costs in a model are based on empirically estimated total costs of stage I, II, III, or IV treatments, then they lack some flexibility to capture future changes in CRC treatment. The purpose was 1) to describe how to model CRC costs and survival and 2) to validate the model in a transparent and reproducible way. Methods. We applied a semi-Markov model with 70 health states and tracked age and time since specific health states (using tunnels and 3-dimensional data matrix). The model parameters are based on an observational study at Oslo University Hospital (2049 CRC patients), the National Patient Register, literature, and expert opinion. The target population was patients diagnosed with CRC. The model followed the patients diagnosed with CRC from the age of 70 until death or 100 years. The study focused on the perspective of health care payers. Results. The model was validated for face validity, internal and external validity, and cross-validity. The validation showed a satisfactory match with other models and empirical estimates for both cost and survival time, without any preceding calibration of the model. Conclusions. The model can be used to 1) address a range of CRC-related themes (general model) like survival and evaluation of the cost of treatment and prevention measures; 2) make predictions from intermediate to final outcomes; 3) estimate changes in resource use and costs due to changing guidelines; and 4) adjust for future changes in treatment and trends over time. The model is adaptable to other populations.