Performance of CRASH and IMPACT Prognostic Models for Traumatic Brain Injury at 12 and 24 Months Post-Injury

Abstract

The Corticoid Randomization after Significant Head Injury (CRASH) and International Mission for Prognosis and Analysis of Clinical Trials (IMPACT) prognostic models are the most reported prognostic models for traumatic brain injury (TBI) in the scientific literature. However, these models were developed and validated to predict 6-month unfavorable outcome and mortality, and growing evidence supports continuous improvements in functional outcome after severe TBI up to 2 years post-injury. The purpose of this study was to evaluate CRASH and IMPACT model performance beyond 6 months post-injury to include 12 and 24 months post-injury. Discriminative validity remained consistent over time and comparable to earlier recovery time points (area under the curve = 0.77–0.83). Both models had poor fit for unfavorable outcomes, explaining less than one quarter of the variation in outcomes for severe TBI patients. The CRASH model had significant values for the Hosmer-Lemeshow test at 12 and 24 months, indicating poor model fit past the previous validation point. There is concern in the scientific literature that TBI prognostic models are being used by neurotrauma clinicians to support clinical decision making despite the goal of the models' development being to support research study design. The results of this study indicate that the CRASH and IMPACT models should not be used in routine clinical practice because of poor model fit that worsens over time and the large, unexplained variance in outcomes.

Introduction

Traumatic brain injury (TBI) is the leading cause of death and disability for persons <40 years of age worldwide.¹ Early identification of patients with favorable recovery trajectories is a central focus of clinical care as well as research clinical trial design. The Corticoid Randomization after Significant Head Injury (CRASH) and International Mission for Prognosis and Analysis of Clinical Trials (IMPACT) prognostic models are the most reported and validated models in the scientific literature.^2–5 These models report a favorable performance when measured with receiver operating characteristic (ROC) area under the curve (AUC), ranging from 76% to 95%, but rarely report accuracy (4–8% of articles) or sensitivity/specificity (3–5% of articles), limiting model interpretability.⁴ More important, both CRASH and IMPACT models were developed and validated to predict outcomes at 6 months post-TBI. Since the release of these models, a growing body of research suggests that 6 months is a less relevant time point for assessing long-term TBI recovery.^2,3

Recovery from severe TBI is a continuum of improvement.⁶ Among patients discharged from the hospital comatose, 80% regain consciousness during inpatient rehabilitation and 40% regain semi- of full-functional independence.^7–10 The multi-center TRACK-TBI (Transforming Research and Clinical Knowledge in TBI) consortium found that outcomes for patients with moderate-to-severe TBI continue to improve from 2 weeks to 1 year post-injury.¹¹ Nearly one half of patients with severe TBI recovered the ability to function independently by 12 months post-injury.¹¹ We previously demonstrated that nearly 40% of patients with unfavorable outcomes at 1 year progressed to favorable outcomes by 2 years.⁹ Recognizing the vast recovery potential of severe TBI patients, we sought to extend our understanding of CRASH and IMPACT model performance beyond 6 months post-injury to include 12 and 24 months post-injury.

Methods

Design and participants

The current study is a secondary analysis of a prospectively collected database of severe TBI patients treated at a level 1 trauma center in [blinded for review] from November 2002 to December 2018. The database utilized includes consecutive severe TBI patients (Glasgow Coma Scale [GCS] < = 8), excluding those with age <16, pregnancy, penetrating trauma, or impending death (GCS = 3 with non-reactive enlarged pupils bilaterally).¹² The Glasgow Outcomes Scale (GOS) was assessed at 12 and 24 months by a trained researcher. This study was approved by the University of [blinded for review] Institutional Review Board for human subjects research. Patients were fully informed before providing written informed consent to obtain access to their electronic medical health record. All data are available to qualified researchers upon request. We followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist.¹³

Prognostic models

Corticoid Randomization after Significant Head Injury (CRASH)

We utilized the computed tomography (CT) model for unfavorable outcome at 6 months and CT model for mortality at 14 days. The coefficients utilized were those validated for high-income countries.² The CRASH score for unfavorable outcome and mortality was converted into percent risk for analysis to ease interpretation as described by the previously published CRASH protocol.^2,14

International Mission on Prognosis and Analysis of Clinical Trials (IMPACT)

We utilized the full prediction model for unfavorable outcome and mortality at 6 months using coefficients provided by the original IMPACT publication.¹⁴ The IMPACT score for both mortality at 6 months and unfavorable outcome was converted into percent risk for analysis to ease interpretation as described by the previously published IMPACT protocol.

Outcomes

Mortality was determined in two ways: 1) the patient's electronic health record or 2) a search of the Social Security Death Index. Days until mortality was operationally defined as the difference in days between date of TBI and date of death. GOS score was used to identify 12- and 24-month outcome post-TBI. Outcomes are classified from 1 to 5, where 1 corresponds to death and 5 corresponds to good recovery. Consistent with the original CRASH and IMPACT publications, we defined unfavorable outcome as a GOS score of 1–3 each time point.^2,14

Statistical analysis

Sample descriptive statistics were calculated and provided. We reported the AUC to identify the discriminative validity of the percent-risk models. Binary logistic regression models were built using the percent-risk models as predictors of their corresponding outcome. To assess model fit, we performed a post hoc analyses using the Hosmer-Lemeshow chi-squared statistic with a p value ≤0.05 indicating poor model calibration and R². This statistic is important because it provides a measure of the overall precision of the model and is not often reported in studies of prognostic models for TBI.⁴ We reported overall sensitivity, specificity, positive predictive value, negative predictive value, accuracy, and false-positive rates at each time point. Statistical significance was p < 0.05. For all analyses, we used SPSS (v28.0.1; SPSS, Inc., Chicago, IL) and Stata software (SEv17; StataCorp LLC, College Station, TX).

Results

We identified 598 patients who were included in the study. Our follow-up rates with complete data for GOS at 12 (84.3%; n = 504) and 24 months (71.4%; n = 427) post-trauma are consistent or better than historical norms for clinical trials.¹⁵ Descriptive statistics for our patient cohort can be viewed in Table 1. Overall, 153 patients in the total sample died by 14 days post-injury (25.6%), 199 died by 6 months (33.3%), 234 died by 12 months (46.4%), and 238 died by 24 months (55.7%).

Table 1.

Descriptive Statistics for the Overall Sample Presented as Median and Interquartile Range or Count and Percentage of Sample

	Median (no.)	Interquartile range (%)
Age, years	39	25, 53.3
Days until death (n = 240)	8	2, 28.75
Glasgow Coma Scale	6	4, 7
Glasgow Outcome Scale at 12 months 1 2 3 4 5	234 8 104 82 76	46.4 1.6 20.6 16.3 15.1
Glasgow Outcome Scale at 24 months 1 2 3 4 5	238 2 65 63 59	55.7 0.5 15.2 14.8 13.8

Discriminative validity and model performance of CRASH and IMPACT models for unfavorable outcomes over time

AUC and clinical characteristics can be viewed in Table 2 and Figure 1. The CRASH model had R² values of 0.15–0.18 over time, with AUCs ranging from 0.77 to 0.80. Hosmer-Lemeshow chi-square values were statistically significant for the 12- and 24-month time points, indicating poor model calibration. Specificity for the CRASH models was poor across all time points (51.9–53.0%). Accuracy ranged from 71.5% to 74.0% with a high rate of false positives (21.4–23.9%).

FIG. 1.

Receiver operating characteristic area under the curve outputs for CRASH and IMPACT models to predict unfavorable outcomes (GOSE = 1–4) at 12 months (A) and 24 months (B). Receiver operating characteristic area under the curve outputs for CRASH and IMPACT models to predict mortality in severe TBI patients at 12 months (C) and 24 months (D). GOSE, Glasgow Outcome Score-Extended; TBI, traumatic brain injury.

Table 2.

Model Performance Characteristics for CRASH and IMPACT Prognostic Models for Unfavorable Outcomes Over Time

	Time	R²	Hosmer-Lemeshow p	AUC	Sensitivity	Specificity	PPV	NPV	Accuracy	False positives
CRASH unfavorable outcomes	12 months (n = 435)	0.15	0.02^*	0.77	82.0	51.9	76.1	60.7	71.5	23.9
CRASH unfavorable outcomes	24 months (n = 386)	0.18	0.01^*	0.80	84.1	53.0	78.6	62.0	74.0	21.4
IMPACT unfavorable outcomes	12 months (n = 435)	0.20	0.18	0.80	83.7	51.6	76.1	63.2	72.4	23.9
IMPACT unfavorable outcomes	24 months (n = 386)	0.23	0.18	0.82	85.3	53.9	78.9	64.5	74.9	21.2

Statistically significant at p ≤ 0.05, indicating poor model fit.

AUC, area under the curve; GOSE, Glasgow Outcome Scale-Extended; PPV, positive predictive value; NPV, negative predictive value.

The IMPACT models had R² values of 0.20–0.23, with AUCs ranging from 0.80 to 0.82. IMPACT models also had poor specificity (51.6–53.9%), moderate accuracy (72.4–74.9%), and high rates of false positives (21.2–23.9%).

Discriminative validity and model performance of CRASH and IMPACT models for mortality over time

AUC and clinical characteristics can be viewed in Table 3 and Figure 2. The CRASH models had R² values of 0.20–0.21, with AUCs of 0.80. Sensitivity was moderate, with values ranging from 61.5% to 68.5%. Accuracy was moderate (70.7–73.2%), with a high rate of false positives (23.5–23.8%).

Table 3.

Model Performance Characteristics for CRASH and IMPACT Prognostic Models for Mortality Over Time

	Time	R²	Hosmer-Lemeshow p	AUC	Sensitivity	Specificity	PPV	NPV	Accuracy	False positives
CRASH mortality	12 months (n = 504)	0.21	0.40	0.80	61.5	83.3	76.2	71.4	73.2	23.8
CRASH mortality	24 months (n = 426)	0.20	0.74	0.80	68.5	73.4	76.5	64.8	70.7	23.5
IMPACT mortality	12 months (n = 501)	0.26	0.97	0.82	69.1	79.1	74.2	74.7	74.5	25.8
IMPACT mortality	24 months (n = 426)	0.27	0.72	0.83	76.8	73.5	78.5	71.7	75.4	21.6

AUC, area under the curve; PPV, positive predictive value; NPV, negative predictive value.

The IMPACT models had R² values of 0.26–0.27, with AUCs ranging from 0.82 to 0.83. Sensitivity for the CRASH models was moderate across time points (69.1–76.8%). Accuracy was moderate (74.5–75.4%) with a high rate of false positives (21.6–25.8%).

Discussion

The CRASH and IMPACT prognostic models for TBI were evaluated at 12 and 24 months post-injury in a prospective cohort of severe TBI patients. The discriminative validity for each model remained consistent over time and comparable to earlier recovery time points (AUC = 0.77–0.83).¹⁶ Both models had poor fit for unfavorable outcomes (R² = 0.15–0.23), explaining less than one quarter of the variation in outcomes for severe TBI patients. The CRASH model had significant values for the Hosmer-Lemeshow test at 12 and 24 months, indicating poor model calibration past 6 months post-TBI. Despite IMPACT and CRASH maintaining AUCs through 2 years post-injury (i.e., good discriminative validity between groups), both models should not be used in routine clinical practice because of poor model fit that worsens over time and the large, unexplained variance in outcomes.

The nature of TBI is complex and heterogenous, which has hampered the ability of past work to develop accurate prognostic models. This inability is concerning, given that trauma specialists overestimate the risk for poor outcomes and underestimate the risk for favorable outcomes in patients with severe injuries, which biases decision making in early patient care.^17–19 Previous work has demonstrated that neurosurgeons tend to be more nihilistic on long-term outcomes than general surgeons or other trauma specialists.²⁰ In clinical trials in TBI, the leading cause of death is attributable to withdrawal of life-sustaining therapies, often occurring early within 72 h of injury.²¹ CRASH and IMPACT lack the discriminatory ability for individual patients to guide life-or-death decisions, given that they incorrectly predicted that nearly 1 in 5 patients would have an unfavorable outcome or die. Despite poor predictive ability, many neurotrauma clinicians do use the IMPACT or CRASH calculator to inform clinical decision making, including withdrawal-of-care decisions.^20,22

Positive predictive value and rate of false positives may be the most important features to assess prognostic model performance given the heterogeneity in functional outcomes.²³ Approximately 1 in 4 patients in this TBI cohort were incorrectly predicted to have unfavorable outcomes at 12 and 24 months post-injury. Lower positive predictive values are typically accompanied by higher false-positive rates, which was the case in this analysis. The high rate of misclassifying patients (∼20%) of having an unfavorable prognosis may be related to the tendency of these prognostic models to rely heavily on early post-injury clinical characteristics, which do not update based upon changes in these characteristics during recovery.²⁴ Future work should investigate additional factors that may improve the positive predictive value and false-positive rate of these models.

This study has limitations. Information about withdrawal of care or cause of mortality was not available for this analysis and would have strengthened the findings. Missing data attributable to loss of follow-up could have affected the findings. We report the overall model characteristics, but certain cutoffs along the ROC curve may be clinically useful (e.g., cutoffs that maximize specificity). Though the data were prospectively collected in consecutive patients, this was a retrospective analysis, which could introduce bias. The present work was also a single-center study, and the results should be externally validated across multiple level 1 trauma centers in future research.

Conclusion

The CRASH and IMPACT prognostic models for TBI had poor performance in key metrics for predicting unfavorable outcomes and mortality in severe TBI patients at 12 and 24 months. Positive predictive value, false positive rate, sensitivity, and specificity revealed inaccuracies associated with patient outcomes, suggesting that the current CRASH and IMPACT models are not applicable as primary predictive tools in clinical practice.

Footnotes

Authors' Contributions

S.E.: Conceptualization, writing, editing, formal analysis (equal leads). E.N.: Reviewing and editing (supporting). J.E.: Reviewing and editing, contribute to analysis (supporting). H.D.: Reviewing and editing (supporting). D.O.O.: Review and editing (supporting). M.P.: Conceptualization, reviewing, editing (equal support).

Funding Information

No financial or funding support was provided for this work.

Author Disclosure Statement

No competing financial interests exist.

Abbreviations Used

References

Nelson

, Temkin

, Dikmen

, et al. Recovery after mild traumatic brain injury in patients presenting to US Level I trauma centers: a Transforming Research and Clinical Knowledge in Traumatic Brain Injury (TRACK-TBI) Study. JAMA Neurol, 2019; 76(9):1049–1059; doi: 10.1001/jamaneurol.2019.1313

MRC CRASH Trial Collaborators. Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patients. BMJ, 2008; 336(7641):425–429; doi: 10.1136/bmj.39461.643438.25

Roozenbeek

, Lingsma

, Lecky

, et al.; International Mission on Prognosis Analysis of Clinical Trials in Traumatic Brain Injury (IMPACT) Study Group; Corticosteroid Randomisation After Significant Head Injury (CRASH) Trial Collaborators; Trauma Audit and Research Network (TARN). Prediction of outcome after moderate and severe traumatic brain injury: external validation of the International Mission on Prognosis and Analysis of Clinical Trials (IMPACT) and Corticoid Randomisation After Significant Head injury (CRASH) prognostic models. Crit Care Med, 2012; 40(5):1609–1617; doi: 10.1097/CCM.0b013e31824519ce

Dijkland

, Foks

, Polinder

, et al. Prognosis in moderate and severe traumatic brain injury: a systematic review of contemporary models and validation studies. J Neurotrauma, 2020; 37(1):1–13; doi: 10.1089/neu.2019.6401

Panczykowski

, Puccio

, Scruggs

, et al. Prospective independent validation of IMPACT modeling as a prognostic tool in severe traumatic brain injury. J Neurotrauma, 2012; 29(1):47–52; doi: 10.1089/neu.2010.1482

Deng

, Nwachuku

, Wilkins

, et al. Time to following commands in severe traumatic brain injury survivors with favorable recovery at 2 years. Neurosurgery, 2022; 91(4):633–640; doi: 10.1227/neu.0000000000002087

Kowalski

, Hammond

, Weintraub

, et al. Recovery of consciousness and functional outcome in moderate and severe traumatic brain injury. JAMA Neurol, 2021; 78(5):548–557; doi: 10.1001/jamaneurol.2021.0084

Puffer

, Yue

, Mesley

, et al. Recovery trajectories and long-term outcomes in traumatic brain injury: a secondary analysis of the phase 3 citicoline brain injury treatment clinical trial. World Neurosurg, 2019; 125:e909–e915; doi: 10.1016/j.wneu.2019.01.207

Wilkins

, Beers

, Borrasso

, et al. Favorable functional recovery in severe traumatic brain injury survivors beyond six months. J Neurotrauma, 2019; 36(22):3158–3163; doi: 10.1089/neu.2018.6153

10.

Kowalski

, Hammond

, Weintraub

, et al. Recovery of consciousness and functional outcome in moderate and severe traumatic brain injury. JAMA Neurol, 2021; 78(5):548–557; doi: 10.1001/jamaneurol.2021.0084

11.

McCrea

, Giacino

, Barber

, et al. Functional outcomes over the first year after moderate to severe traumatic brain injury in the prospective, longitudinal TRACK-TBI Study. JAMA Neurol, 2021; 78(8):982–992; doi: 10.1001/jamaneurol.2021.2043

12.

Pease

, Nwachuku

, Goldschmidt

, et al. Complications from multi-modal monitoring do not affect long-term outcomes in severe traumatic brain injury. World Neurosurg, 2022; 161:e109–e117; doi: 10.1016/j.wneu.2022.01.059

13.

Von Elm

, Altman

, Egger

, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med, 2007; 147(8):573–577; doi: 10.7326/0003-4819-147-8-200710160-00010

14.

Steyerberg

, Mushkudiani

, Perel

, et al. Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med, 2008;5(8):e165; discussion, e165; doi: 10.1371/journal.pmed.0050165

15.

Richter

, Stevenson

, Newman

, et al. Handling of missing outcome data in traumatic brain injury research: a systematic review. J Neurotrauma, 2019; 36(19):2743–2752; doi: 10.1089/neu.2018.6216

16.

Eagle

, Pease

, Nwachuku

, et al. Prognostic models for traumatic brain injury have good discrimination but poor overall model performance for predicting mortality and unfavorable outcomes. Neurosurgery, 2023; 92(1):137–143; doi: 10.1227/neu.0000000000002150

17.

Williamson

, Ryser

, Ubel

, et al. Withdrawal of life-supporting treatment in severe traumatic brain injury. JAMA Surg, 2020; 155(8):723–731; doi: 10.1001/jamasurg.2020.1790

18.

Kaufmann

, Buchmann

, Scheidegger

, et al. Severe head injury: should expected outcome influence resuscitation and first-day decisions?. Resuscitation, 1992; 23(3):199–206; doi: 10.1016/0300-9572(92)90003-u

19.

Izzy

, Compton

, Carandang

, et al. Self-fulfilling prophecies through withdrawal of care: do they exist in traumatic brain injury, too?. Neurocrit Care, 2013; 19(3):347–363; doi: 10.1007/s12028-013-9925-z

20.

Letsinger

, Rommel

, Hirschi

, et al. The aggressiveness of neurotrauma practitioners and the influence of the IMPACT prognostic calculator. PLoS One, 2017; 12(8):e0183552; doi: 10.1371/journal.pone.0183552

21.

Tisherman

, Schmicker

, Brasel

, et al. Detailed description of all deaths in both the shock and traumatic brain injury hypertonic saline trials of the Resuscitation Outcomes Consortium. Ann Surg, 2015; 261(3):586–590; doi: 10.1097/SLA.0000000000000837

22.

Hirschi

, Rommel

, Hawryluk

. Should we have a guard against therapeutic nihilism for patients with severe traumatic brain injury?. Neural Regen Res, 2017; 12(11):1801–1803; doi: 10.4103/1673-5374.219037

23.

Trevethan

. Sensitivity, specificity, and predictive values: foundations, pliabilities, and pitfalls in research and practice. Front Public Health, 2017; 5:307; doi: 10.3389/fpubh.2017.00307

24.

Giacino

, Sherer

, Christoforou

, et al. Behavioral recovery and early decision making in patients with prolonged disturbance in consciousness after traumatic brain injury. J Neurotrauma, 2020; 37(2):357–365; doi: 10.1089/neu.2019.6429