GRACE Score Validation in Predicting Hospital Mortality: Analysis of the Role of Sex

Abstract

Background:

The GRACE (Global Registry of Acute Coronary Events) risk score is recommended for risk stratification in acute coronary syndrome (ACS). It does not include sex, a variable strongly associated with ACS prognosis. The aim of this study was to examine if sex adds prognostic information to the GRACE score in a contemporary population.

Materials and Methods:

Analysis of discrimination and calibration of GRACE score in the validation population, derived from the ARIAM-SEMICYUC registry (2012–2015). Outcome was hospital mortality. The uniformity of fit of the score was tested in predefined subpopulations: with and without ST-segment elevation myocardial infarction (STEMI and NSTEMI).

Results:

A total of 9781 patients were included: 4598 with NSTEMI (28% women) and 5183 with STEMI (23% women). Discriminative capacity of the GRACE score was significantly lower in women with STEMI compared to men (area under the receiver operating characteristic curve [AUC] 0.82, 95% CI 0.78–0.86 vs. AUC 0.90, 95% CI 0.88–0.92, p = 0.0006). In multivariate analysis, female sex predicted hospital mortality independently of GRACE in STEMI (p = 0.019) but not in NSTEMI (p = 0.356) (interaction p = 0.0308). However, neither the AUC nor the net reclassification index (NRI) improved by including female sex in the STEMI subpopulation (NRI 0.0011, 95% CI −0.023 to 0.025; p = 0.928).

Conclusions:

Although female sex was an independent predictor of hospital mortality in the STEMI subpopulation, it does not substantially improve the discriminative ability of GRACE score.

Introduction

Current clinical practice guidelines on diagnosis and treatment of acute coronary syndrome (ACS) recommend early ischemic risk stratification using prognostic tools such as GRACE (Global Registry of Acute Coronary Events), a score derived from a multinational registry, developed in a heterogeneous population without bias due to predetermined inclusion criteria¹ and Thrombolysis in Myocardial Infarction (TIMI),^2,3 derived from clinical trial cohorts in the thrombolytic era.^4

–7

In ACS, the electrocardiographic pattern of ST elevation is associated with acute thrombotic occlusion of an epicardial artery (ST-elevation myocardial infarction [STEMI]). When the occlusion is not complete or there is collateral flow in their distal segments, no ST segment occurs in the electrocardiogram (non-ST-elevation myocardial infarction [NSTEMI]). Treatment of STEMI addresses the urgent opening of the culprit artery, either using pharmacological thrombolysis or percutaneous coronary intervention (PCI). Where PCI is not available or primary PCI cannot be achieved within 120 minutes of diagnosis, thrombolytic therapy should be given.⁶ In high-risk NSTEMI, treatment is based on percutaneous revascularization in the first 24–72 hours and is not an immediate emergency, with some exceptions.^5,7 In recent years, ACS treatment has undergone many changes, especially the increasing use of percutaneous revascularization.^8,9

The GRACE score applies equally to both types of ACS. Although sex was considered in the development of the GRACE score, it was not a statistically significant predictor associated with hospital mortality.¹ Therefore, GRACE does not consider this parameter, even though the literature continues to present data supporting significant differences in diagnosis and clinical outcomes between men and women, both in NSTEMI¹⁰ and STEMI^11,12 and also data suggesting that they do not respond equally to the same treatments.^13
–15 Despite these facts, the same prognostic tool for risk stratification is used for both sexes. Thus, there may be opportunities for adjustment based on this variable.¹⁶

This study aims (1) to validate the prognostic accuracy of the GRACE score (predicting in-hospital mortality) in a contemporary Spanish cohort; (2) to explore possible differences in the prognostic accuracy of the GRACE score in women compared to men; and (3) to determine whether the inclusion of sex as a predictor improves discrimination of the GRACE score.

Materials and Methods

The validation population derives from the national, hospital-based registry ARIAM-SEMICYUC (Análisis del Retraso en el infarto Agudo de Miocardio de la Sociedad Española de Medicina Intensiva y Unidades Coronarias). It includes consecutive adult patients (≥18 years) admitted with ACS within 48 hours of evolution to coronary/intensive care units in Spain (63 centers over 16 regions), from January 1, 2012, to August 31, 2015. We excluded patients whose final diagnosis did not confirm ACS.

The analysis was performed following the TRIPOD recommendations (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis).¹⁷

Complete data analysis was performed to hospital discharge. We analyzed variables included in the GRACE score as well as sex. The registry website (https://ariam.investigacion-intensivos.org) calculates the GRACE score based on epidemiological, laboratory, and electrocardiographic data collected in the registry, according to published nomograms that assign a value to each variable.¹ Data are anonymized, and the database conforms to the data protection regulations in Spain.

The outcome was hospital mortality from all causes. For ischemic risk stratification, the GRACE score (predicting in-hospital mortality) was used.

All data were collected retrospectively, including the GRACE score, independent of clinical management. Patients were managed according to standard clinical practice, following the recommendations of clinical practice guidelines.

A descriptive analysis was performed according to the type of ACS (NSTEMI and STEMI). STEMI was defined as persistent ST-segment elevation in at least two contiguous leads or new (or presumably new) left bundle branch block. NSTEMI was defined as ST-segment depression, transient ST-segment elevation, T wave changes in at least two contiguous leads, or electrocardiogram unchanged. These definitions are in line with the current universal definition of myocardial infarction.¹⁸ In the original design of the score, cases were grouped into these same subgroups.¹ Univariate analysis was performed using the chi-squared test, chi-squared test for trends, and Mann–Whitney or Kruskal–Wallis tests, as appropriate. For multivariate analyses, we constructed various models of binomial logistic regression. Hospital mortality was the response variable; the predictor variables were the GRACE score, sex, type of ACS, and the interaction term between sex and type of ACS.

The discriminative ability (ability of the score to distinguish subjects with different outcomes) was assessed by calculating the area under the ROC (receiver operating characteristic) curve (AUC). Calibration (agreement between observed probabilities and predicted probabilities of hospital death) was tested using a linear regression model, dividing the GRACE score in 20 intervals, as in the original study, and comparing the observed mortality with the predicted probability of hospital death.¹ In addition, we calculated the goodness of fit (Hosmer–Lemeshow test), the Brier score, and Nagelkerke's R² statistic.¹⁹ Finally, we analyzed uniformity of fit in predefined subpopulations (NSTEMI and STEMI) according to sex, comparing the AUCs using nonparametric methods.²⁰ The effect of sex on the reclassification was analyzed by calculating the net reclassification index (NRI).¹⁹

Analyses were performed with StatsDirect (version 3.0.167. StatsDirect Ltd; Altrincham, Cheshire, United Kingdom), Stata (version 13.1, Stata Corp, College Station, Texas), and PredictABEL, an R package designed for the evaluation of risk prediction models.²¹

Results

Validation population

During the study period, we collected data on 11,036 patients who met the inclusion criteria: 5188 patients (47%) with NSTEMI and 5848 (53%) with STEMI. We ultimately analyzed data on 9781 patients with available mortality data at hospital discharge: 4598 patients with NSTEMI (28% women) and 5183 patients with STEMI (23% women). Cases of unstable angina accounted for 19.2% (847/4598) of cases of NSTEMI, 18% (596/3326) in males and 19.76% (251/1272) in women.

Although we could not make a direct comparison with the original population described by Granger et al., there were some large differences in the validation population, as a higher percentage of STEMI (53% vs 35%), a higher percentage of percutaneous reperfusion (>65% in our population, compared to 15%), and more severe disease (almost 10% of the patients in this series had Killip class >2 vs. 4% of the original population) (Table 1).

Table 1.

Descriptive Analysis of the Validation Population

	NSTEMI (n = 4598), n %	STEMI (n = 5183), n %
Age (years) median (IQR)	69 (58–78)	64 (54–75)
Female sex	1272/4598 (27.66)	1206/5183 (23.27)
Diabetes	1657/4566 (36.29)	1285/4566 (24.98)
Tobacco use
Ex-smokers	1247/3892 (32.04)	1103/4513 (46.58)
Current smokers	1247/3892 (32.04)	2102/4513 (24.44)
Hypertension	3112/4569 (68.11)	2789/5140 (54.26)
Dyslipidemia	2674/4536 (58.95)	2435/5111 (47.64)
Prior MI	1228/4587 (26.77)	598/5173 (11.56)
Chronic renal disease	439/4595 (9.55)	195/5180 (3.76)
Heart failure	342/4589 (7.45)	125/5180 (2.41)
SBP (mmHg) median (IQR)	138 (120–155)	130 (110–150)
Heart rate (lpm) median (IQR)	76 (65–90)	77 (65–90)
Creatinine (mg/dL) median (IQR)	0.96 (0.8–1.2)	0.90 (0.79–1.1)
Elevated cardiac enzyme levels	4150/4570 (90.81)	4973/5117 (97.19)
ST-segment deviation	1763/3878 (45.46)	5124/5124 (100)
Cardiac arrest at admission	83/4587 (1.81)	291/5175 (5.62)
Killip Class>II	488/4596 (10.62)	480/5179 (9.27)
Reperfusion (invasive strategy^a)	2594/4597 (72.55)	4293/5183 (82.89)
Hospital mortality	239/4598 (5.2)	376/5183 (7.25)

Values are numbers (percentage) unless stated otherwise.

In NSTEMI: realization of urgent (in the first 24 hours) and early (within the first 72 hours) coronary angiography. In STEMI, reperfusion using primary percutaneous coronary intervention or thrombolysis.

IQR, interquartile range; MI, myocardial infarction; SBP, systolic blood pressure; NSTEMI, non-ST-elevation myocardial infarction; STEMI, ST-elevation myocardial infarction.

The differences between men and women are shown in Table 2. Women with NSTEMI were older (73 years vs. 67 years, p < 0.0001) and presented with more heart failure at admission (12.67% vs. 9.83%, p < 0.0001), but no differences were found in cardiac catheterization (performed in 73.12% men and 71% women, p = 0.2083) or in hospital mortality (5.03% vs. 5.26%, p = 0.7532). In patients with STEMI, differences were found in reperfusion therapy (77.51% in women vs. 84.52% in men, p < 0.0001). Cardiac catheterization for PCI was performed in 2292/3332 men (68.72%) and 663/996 women (66.56%), p < 0.001. Women also had higher hospital mortality (11.85% vs. 5.87%, p < 0.0001).

Table 2.

Descriptive Analysis of STEMI and NSTEMI Subpopulations, According to Sex

	STEMI			NSTEMI
	Men (n = 3977), n %	Women (n = 1206), n %	p	Men (n = 3326), n %	Women (n = 1272), n %	p
Age (years) median (IQR)	61 (52–72)	73 (59–80)	<0.0001	67 (57–77)	73 (64–80)	<0.0001
SBP (mmHg) median (IQR)	130 (112–150)	130 (110–150)	0.233	136 (120–153)	140 (120–160)	0.0036
HR (lpm) median (IQR)	76 (65–89)	80 (65–90)	0.003	75 (65–88)	78 (67–90)	<0.0001
Creatinine (mg/dL) median (IQR)	0.94 (0.80–1.11)	0.80 (0.67–1)	<0.0001	1 (0.82–1.22)	0.85 (0.7–1.1)	<0.0001
Elevated cardiac enzyme levels	3828/3929 (97.43)	1145/1188 (96.38)	0.055	2999/3300 (90.88)	1151/1270 (90.63)	0.7942
ST-segment deviation	3934/3934 (100)	1190/1190 (100)	>0.9999	1299/2799 (46.41)	464/1079 (43)	0.0566
Cardiac arrest at admission	223/3971 (5.62)	68/1204 (5.65)	0.966	68/3316 (2.05)	15/1271 (1.18)	0.0478
Killip Class>II	317/3974 (7.98)	163/1205 (13.53)	<0.0001	327/3325 (9.83)	161/1271 (12.67)	0.0053
Reperfusion^a	3359/3974 (84.52)	934/1205 (77.51)	<0.0001	1926/2634 (73.12)	668/941 (70.99)	0.2083
Hospital mortality	233/3977 (5.86)	143/1206 (11.86)	<0.0001	175/3326 (5.26)	64/1272 (5.03)	0.7532

Values are numbers (percentage) unless stated otherwise.

In STEMI: reperfusion using primary percutaneous coronary intervention or thrombolysis. In NSTEMI: urgent (in the first 24 hours) and early (within the first 72 hours) coronary angiography.

HR, heart rate.

Discrimination and calibration of GRACE score in the validation population

The GRACE score had an AUC of 0.87 (95% CI 0.85–0.88 (Fig. 1). The calibration plot (Fig. 2) shows correlation between observed and predicted results; with correlation coefficient (r) = 0.99 (95% CI 0.98–0.99). The Brier score (mean differences between predicted and observed values) was 0.044; with R² Nagelkerke (proportion of the variation in mortality explained by the GRACE) of 0.34, indicating a moderate overall performance. The Hosmer–Lemeshow goodness of fit test showed significant differences (p = 0.0218) between observed and predicted results.

FIG. 1.

Discriminative ability of the GRACE score in the validation population.

FIG. 2.

Calibration of the GRACE score in the validation population.

Uniformity of fit of GRACE score in STEMI and NSTEMI subpopulations

Considered together with the GRACE score in multivariate analysis, sex was only an independent predictor of hospital mortality in patients with STEMI (p = 0.0197), not in those with NSTEMI (p = 0.3563), with an interaction p-value of 0.0308. Discriminative capacity of the GRACE score was significantly lower in women with STEMI compared to men (AUC 0.82, 95% CI 0.78–0.86 vs. AUC 0.90, 95% CI 0.88–0.92, p = 0.0006). In the NSTEMI subpopulation, we did not observe significant differences (AUC in women 0.83, 95% CI 0.78–0.88 vs. AUC in men 0.87, 95% CI 0.85–0.90, p = 0.096) (Fig. 3).

FIG. 3.

Discriminative ability of the GRACE score according to sex.

To determine the incremental value of sex beyond the GRACE score, we again compared AUC adding the variable sex to GRACE score. The discriminative ability of the GRACE score did not improve substantially, with superimposable ROC curves in both subpopulations (Fig. 4). Adding the variable sex was not associated with significant improvement in reclassifying patients with STEMI (NRI 0.0011, 95% CI −0.0236 to 0.0258; p = 0.92869).

FIG. 4.

Added value of the variable sex in the discriminative ability of GRACE.

Discussion

This study provides information on the prognostic accuracy of the GRACE score in the era of percutaneous reperfusion in ACS. We did not use the TIMI risk score, developed in populations treated with thrombolysis, as it was not considered representative of the clinical spectrum of patients analyzed, mostly reperfused by PCI (Table 1).³ As our objective was to analyze the potential role of sex, an additional reason not to choose a risk score based on clinical trials is the underrepresentation of women in these trials.²²

The first conclusion of this external validation is the acceptable performance of the score in the validation population, based on the results of global measures of model performance (Brier score = 0.044 and Nagelkerke R² statistic = 0.34), ROC curves, and the calibration plot (Figs. 1 and 2). AUC is very sensitive to sample size and event numbers, but although the number of events in women is less than men, we think that the possible underestimation representing the number of events is negligible. The lack of adjustment according to the Hosmer–Lemeshow test was expected given the sample size and cannot be considered as evidence of miscalibration.²³ Therefore, after more than 10 years of the development of the score and following the generalization of percutaneous reperfusion in ACS, despite the differences presented regarding the population in which the score was developed (mainly a higher percentage of STEMI patients¹), the GRACE risk score proved to be a robust tool for predicting hospital mortality in our population. These results are similar to other external validations in contemporary populations.^24,25 Despite persistent differences in the era of reperfusion between men and women,²⁶ the improvement in the NRI resulting from adding the sex variable to the GRACE risk score was negligible (less than 1%).

Our result also supports the validity of the GRACE risk score in the assessment of hospital mortality in both STEMI and NSTEMI patients, although results suggest underperformance of the GRACE risk score in women compared to men in the STEMI subpopulation (Fig. 3). However, although female sex is an independent predictor of hospital mortality in STEMI patients (with some evidence of subgroup effect), the discriminative ability of the GRACE score did not substantially improve by adding the variable sex (Fig. 4). The clinical usefulness of this finding is limited. In STEMI patients, decision-making about treatment does not depend on the systematic use of the GRACE scoring system (as in NSTEMI) but on the delay from onset of symptoms and on electrocardiographic characteristics.⁶ In both regards, there are significant differences between men and women.^12,27 Prehospital delay was considered in the development of GRACE score but was not included in the final model.¹

The aim of this analysis is not to explore the causal relationship between sex and hospital mortality, but to explore whether the addition of the variable sex can improve the predictive ability of a widely used score. Therefore, our conceptual framework, in line with this objective, fits with a predictive (and not explanatory) modeling, without focusing on the causal relationship between the variables. Our results may be limited to the population analyzed, with 11.3% of missing hospital mortality data. This fact may introduce a risk of bias to our findings due to the loss of information attributable to incomplete cases. However, other studies from our group have shown these losses to be random and not associated with hospital mortality.²⁸ Another aspect that limits the results refers to nonobstructive coronary disease. The predictability of the GRACE score in patients with nonobstructive CAD is not well established and might explain, at least partially, the superiority of GRACE at predicting death in men, compared to women, in whom the rate of nonobstructive CAD is usually higher.²⁹ Unfortunately, we lack quality data to include this in the analysis. Also, the analysis of patients with unstable angina along with NSTEMI patients may be a limitation. However, we consider that the risk of this patient group is adjusted for the elevation of biomarkers in the overall calculation of the score.

Conclusions

External validation of the GRACE risk score in a contemporary Spanish cohort with ACS shows that it is still useful to predict hospital mortality in both STEMI and NSTEMI populations. GRACE risk score has a worse diagnostic performance in women than in men, but adding the variable sex to the score does not substantially improve its diagnostic accuracy. It would be desirable to consider new specific scores that adequately reflect the risk of women with STEMI.

Footnotes

Acknowledgments

We are grateful to all the investigators of the ARIAM-SEMICYUC registry.

Author Disclosure Statement

No competing financial interests exist.

References

Granger

, Goldberg

, Dabbous

, et al. Predictors of hospital mortality in the global registry of acute coronary events. Arch Intern Med, 2003; 163:2345–2353.

Antman

, Cohen

, Bernink

, et al. The TIMI risk score for unstable angina/non-ST elevation MI: A method for prognostication and therapeutic decision making. JAMA, 2000; 284:835–842.

Morrow

, Antman

, Charlesworth

, et al. TIMI risk score for ST-elevation myocardial infarction: A convenient, bedside, clinical score for risk assessment at presentation: An intravenous nPA for treatment of infarcting myocardium early II trial substudy. Circulation, 2000; 102:2031–2037.

Sanz

. Risk stratification in acute coronary syndromes: An unresolved issue. Rev Esp Cardiol, 2007; 60 Suppl 3:23–30.

Roffi

, Patrono

, Collet

, et al. 2015 ESC Guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation: Task force for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation of the European Society of Cardiology (ESC). Eur Heart J, 2016; 37:267–315.

O'Gara

, Kushner

, Ascheim

, et al. 2013 ACCF/AHA guideline for the management of ST-elevation myocardial infarction: Executive summary: A report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines. Circulation, 2013; 127:529–555.

Amsterdam

, Wenger

, Brindis

, et al. 2014 AHA/ACC guideline for the management of patients with non-ST-elevation acute coronary syndromes: Executive summary: A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation, 2014; 130:2354–2394.

, Mozaffarian

, Roger

, et al. Heart disease and stroke statistics—2014 update: A Report From the American Heart Association. Circulation, 2014; 129:e28–e292.

Garcia Del Blanco

, Hernandez Hernandez

, Rumoroso Cuevas

, Trillo Nouche

. Spanish cardiac catheterization and coronary intervention registry. 24th Official Report of the Spanish Society of Cardiology Working Group on Cardiac Catheterization and Interventional Cardiology (1990–2014). Rev Esp Cardiol, 2015; 68:1154–1164.

10.

Poon

, Goodman

, Yan

, et al. Bridging the gender gap: Insights from a contemporary analysis of sex-related differences in the treatment and outcomes of patients with acute coronary syndromes. Am Heart J, 2012; 163:66–73.

11.

D'Onofrio

, Safdar

, Lichtman

, et al. Sex differences in reperfusion in young patients with ST-segment-elevation myocardial infarction: Results from the VIRGO study. Circulation, 2015; 131:1324–1332.

12.

de-Miguel-Balsa

, Latour-Perez

, Baeza-Roman

, Llamas-Alvarez

, Ruiz-Ruiz

, Fuset-Cabanes

. Accessibility to reperfusion therapy among women with acute myocardial infarction: Impact on hospital mortality. J Womens Health (Larchmt), 2015; 24:882–888.

13.

Gulati

, Shaw

, Bairey Merz

. Myocardial ischemia in women: Lessons from the NHLBI WISE study. Clin Cardiol, 2012; 35:141–148.

14.

Dolor

, Patel

, Melloni

, et al. AHRQ future research needs papers, in treatment strategies for women with coronary artery disease: future research needs: identification of future research needs from comparative effectiveness review No. 66. Rockville (MD): Agency for Healthcare Research and Quality (US), 2013.

15.

Mehta

, Beckie

, DeVon

, et al. Acute myocardial infarction in women: A scientific statement from the American Heart Association. Circulation, 2016; 133:916–947.

16.

Agrawal

, Van Eyk

, Sobhani

, Wei

, Bairey Merz

. Sex, myocardial infarction, and the failure of risk scores in women. J Womens Health (Larchmt), 2015; 24:859–861.

17.

Collins

, Reitsma

, Altman

, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement. Ann Intern Med, 2015; 162:55–63.

18.

Thygesen

, Alpert

, Jaffe

, et al. Third universal definition of myocardial infarction. J Am Coll Cardiol, 2012; 60:1581–1598.

19.

Steyerberg

, Vickers

, Cook

, et al. Assessing the performance of prediction models: A framework for traditional and novel measures. Epidemiology, 2010; 21:128–138.

20.

DeLong

, DeLong

, Clarke-Pearson

. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 1988; 44:837–845.

21.

Kundu

, Aulchenko

, van Duijn

, Janssens

. PredictABEL: An R package for the assessment of risk prediction models. Eur J Epidemiol, 2011; 26:261–264.

22.

Grady

, Chaput

, Kristof

. Results of systematic review of research on diagnosis and treatment of coronary heart disease in women. Evid Rep Technol Assess (Summ), 2003; 80:1–4.

23.

Steyerberg

, Vergouwe

. Towards better clinical prediction models: Seven steps for development and an ABCD for validation. Eur Heart J, 2014; 35:1925–1931.

24.

Aragam

, Tamhane

, Kline-Rogers

, et al. Does simplicity compromise accuracy in ACS risk prediction? A retrospective analysis of the TIMI and GRACE risk scores. PLoS One, 2009; 4:e7947.

25.

Abu-Assi

, Ferreira-Gonzalez

, Ribera

, et al. Do GRACE (Global Registry of Acute Coronary events) risk scores still maintain their performance for predicting mortality in the era of contemporary management of acute coronary syndromes?. Am Heart J, 2010; 160:826–834.

26.

Isorni

, Blanchard

, Teixeira

, et al. Impact of gender on use of revascularization in acute coronary syndromes: The national observational study of diagnostic and interventional cardiac catheterization (ONACI). Catheter Cardiovasc Interv, 2015; 86:E58–E65.

27.

Nguyen

, Gore

, Saczynski

, et al. Age and sex differences and 20-year trends (1986 to 2005) in prehospital delay in patients hospitalized with acute myocardial infarction. Circ Cardiovasc Qual Outcomes, 2010; 3:590–598.

28.

Latour-Pérez

, Gomez-Tello

, de-Miguel-Balsa

, et al. Routine invasive strategy in acute coronary syndrome patients with renal dysfunction. Results of the ARIAM-SEMICYUC registry. Med Intensiva, 2016; 40:280–288.

29.

Garcia

, Mulvagh

, Merz

, Buring

, Manson

. Cardiovascular disease in women: Clinical perspectives. Circ Res, 2016; 118:1273–1293.