Abstract
Background:
Identifying women at highest or lowest risk of perinatal intensive care unit (ICU) admission may enable clinicians to risk stratify women antenatally so that enhanced care or elective admission to ICU may be considered or excluded in birthing plans. We aimed to develop a statistical model to predict the risk of maternal ICU admission.
Methods:
We studied 762,918 pregnancies between 2005 and 2018. Predictive models were constructed using multivariable logistic regression. The primary outcome was ICU admission. Additional analyses were performed to allow inclusion of delivery-related factors. Predictors were selected following expert consultation and reviewing literature, resulting in 13 variables being included in the primary analysis: demographics, prior health status, obstetric history and pregnancy-related factors. A complete case analysis was performed. K-fold cross validation was used to mitigate against overfitting.
Results:
Complete data were available for 578,310 pregnancies, of whom 1087 were admitted to ICU (0.19%). Model performance was fair (area under the ROC curve = 0.66). A comparatively high cut-point of ⩾0.6% for ICU admission risk resulted in a negative predictive value (NPV) of 99.8% (specificity 97.8%) but positive predictive value (PPV) of 0.8% (sensitivity 9.1%). Models including delivery-related factors demonstrated superior discriminative performance.
Conclusions:
Our model for maternal ICU admission has an acceptable discriminative performance. The low frequency of ICU admission and resulting low PPV indicates that the model would be unlikely to be useful as a ‘rule-in’ test for pre-emptive consideration of ICU admission. Its potential for improving efficiency in screening as a ‘rule-out’ test remains uncertain.
Introduction
Maternal death in the UK is rare. Efforts to improve quality of care have therefore focussed on reducing ‘near-miss’ morbidity events in the perinatal period. 1 Admission to an intensive care unit (ICU) is an example of one such severe maternal morbidity event. 2 Indeed, in population-level research of maternal deaths in the perinatal period, 45% were admitted to the ICU. 3 Previous work has demonstrated that severe maternal morbidity may not be accurately derived from routine healthcare data. 2 In contrast, ICU admission is reliably and consistently recorded in routine databases. However, there are no specified thresholds for admission to ICU, despite UK-wide guidance on management of maternal critically ill patients.4,5
While it is well recognised that certain obstetric conditions such as placenta accreta will present greatly increased risk of maternal morbidity, 6 it is less well known how common maternal and obstetric factors combine to affect a woman’s likelihood of ICU admission. Early recognition of critical illness and rapid transfer to an appropriate environment to initiate therapies has been highlighted by MBRRACE-UK as a key message for informing maternity care. 7 Identifying women at highest risk of perinatal ICU admission may enable clinicians to risk stratify women antenatally so that enhanced care, for example planning for delivery in a setting co-located with an ICU, or elective admission to ICU, may be considered in birthing plans.
The UK’s National Institute for Health and Care Excellence (NICE) guidance recommends that antenatal risk screening should be undertaken in all pregnant women. 8 Methods of identifying women at highest risk rely on history taking, examination and early investigation during routine antenatal appointments. This may be time consuming, and therefore a low cost, data-driven automated method, leveraging data collected from previous routine health care contact, may enable a more efficient screening process.
There is limited research on the development and performance of risk prediction models for maternal ICU admission, particularly in the UK context. We aimed to develop a statistical model for the prediction of risk of maternal admission to ICU in the perinatal period.
Methods
Study population and data sources
We studied women with a live birth, stillbirth or late second trimester loss (from 20 weeks gestation) from 01/01/2005 to 31/12/2018, using data from Scottish Morbidity Records, National Records of Scotland and the Scottish Intensive Care Society Audit Group database. Details relating to database coverage and accuracy have been previously reported. 2 This captured 762,918 pregnancies.
Outcomes
The primary outcome was ICU admission during pregnancy or within 42 days of delivery. Other outcomes were admission to ICU or standalone general HDU (secondary); and admission to either ICU, standalone general HDU or obstetric HDU (tertiary). Note that ‘ICU admission’ refers to admission to a unit capable of providing level 3 intensive care; this includes standalone ICUs as well as units which collocate level 3 and level 2 beds within the same integrated unit (combined ICU/HDU). Thus ‘ICU admission’ will include patients who received level 2 care in a combined ICU/HDU. Availability of obstetric HDU admission data was restricted by year and geographical location, and so analyses with the tertiary outcome were restricted to the population for whom these data were available. Therefore, the following three additional analyses were undertaken: (1 and 2) replacing the primary outcome with secondary and tertiary outcomes respectively; (3) restricting the primary outcome occurrence to either on or after day of delivery to allow the study of delivery-related factors.
Predictors
Predictors were selected following expert consultation, reviewing the literature and any additional variables with a univariate p-value < 0.2 (see Supplement). Following this, the likelihood ratio test was employed to select variables to give the best model fit. Predictors with greater than 50% of values missing were omitted. A resultant 13 variables were included in the primary model which would have been available early in pregnancy, for example, at the time of an antenatal booking appointment: maternal age, maternal country of birth, Scottish Index of Multiple Deprivation (SIMD) quintile, 9 residence in rural versus urban area, 10 body mass index (BMI), smoking status (at antenatal booking), comorbidities, 2 parity, multiple gestation, previous pregnancy loss, previous neonatal death, previous Caesarean section (C-section), and previous inpatient admissions (see Table 1). These variables were also used in additional analyses 1 and 2 to evaluate the secondary and tertiary outcomes respectively. A further five delivery-related variables were studied in additional analysis 3: analgesia required during delivery, mode of delivery, antenatal steroids required, induction of labour, and hospital admission during current pregnancy. A complete case analysis was performed.
Baseline characteristics for pregnancies which resulted in maternal ICU admission versus no ICU admission.
Data are presented as n (%) or median (lower, upper quartiles). Categorical variables were compared using a Chi-squared analysis. Body Mass Index (BMI), calculated as ratio of maternal weight (kg) and the square of maternal height (cm). Scottish Index of Multiple Deprivation (SIMD) is a marker of social deprivation using the post-code of patient’s residence.
Statistical methods
A risk prediction model was developed for each outcome using multivariable logistic regression. Derivation and internal validation were achieved using bootstrapping with optimism adjusted measures of model performance. We used robust variance estimation to account for clustering due to women with multiple, sequential pregnancies during the study period.
Model discrimination was assessed by area under the receiving operating characteristic (ROC) curve (AUC); overall performance was evaluated using the Brier score. Calibration graphs were drawn to investigate the goodness of fit of models. K-fold cross validation was used to mitigate against overfitting. The utility of the primary outcome model as a screening tool to identify women at risk of maternal ICU admission was examined by evaluating the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), false positive rate, and false negative rate at different probability cut points.
This study was reported in accordance with the TRIPOD statement. 11 Analyses were conducted using SAS version 9.4, Stata version 16.1 and R version 4.2.0 within NHS Scotland’s safe haven environment. Data were pseudonymised and access to data was granted following approval by the Public Benefit and Privacy Panel (ref 1617-0283).
Results
Baseline characteristics are shown in Table 1 for the full cohort of 762,918 pregnancies. For the primary outcome, complete data were available for 578,310 pregnancies, of which 1087 resulted in ICU admission (0.19%). In general, rates of missing data were low. With the exception of BMI, smoking status, steroid therapy, and required analgesia, all variables had less than 5% missingness (Supplemental Table S1). Notably, BMI was missing for a fifth of pregnancies.
Women who were admitted to ICU were on average more likely to be older, be from a deprived area, have a higher BMI, have at least one comorbidity, and have previously been admitted to hospital (Table 1). Women originating from the British Isles and Europe accounted for 92% of the total population but just 89% of those admitted to ICU. In contrast, women who were born in Africa, the Middle East or Asia accounted for 6% of the total population but 8% of ICU admissions. Interestingly, former smokers comprised 18% of the total population but 22% of ICU admissions, whereas current smokers accounted for 11% of the total population and 10% of those admitted to ICU. No significant difference was observed in likelihood of ICU admission for residence in a rural compared with an urban area.
Multiple gestation, previous pregnancy loss, previous neonatal death and previous Caesarean section increased the likelihood of maternal ICU admission. First pregnancies accounted for 45.5% of ICU admissions but 44.5% of the total population, whereas second pregnancies were responsible for just 27.1% of ICU admissions and 34.3% of the total population. With increasing parity after the second pregnancy, there was an increased likelihood of ICU admission.
With respect to delivery-related factors, those who were admitted to ICU were more likely to have had a preterm delivery, delivered by emergency C-section, received antenatal steroids, and had at least one hospital admission during their current pregnancy. Induction of labour did not significantly affect likelihood of ICU admission.
The ability of our model to discriminate between ICU admission and no admission was fair (Figure 1; AUC = 0.66 (95% CI 0.65, 0.68)). The Brier score of 0.0019 (95% CI 0.0018, 0.0020) was low, in keeping with predicting a rare outcome. A calibration plot (Figure 2) demonstrated plotted observations to closely follow the reference line, indicating good concordance between the predicted probability of and observed proportion of ICU admission. The optimism of K-fold cross validation was near zero at −0.017, indicating low concern about overfitting of the model. The model including all coefficients is presented in Supplemental Table S2.
Receiver operating characteristic curve for risk prediction model for ICU admission.
Calibration curve for risk prediction model for ICU admission.
The negative predictive value (NPV) of the model was high across a range of predicted probability thresholds (Table 2). A comparatively high cut-point of 0.6% or more for risk of ICU admission, in which 2.2% of patients screened positive, resulted in NPV of 99.8% and specificity 97.8%; however, this gave a positive predictive value (PPV) of just 0.8% and sensitivity of 9.1%. Using a lower cut-point of 0.3% risk, where 10.6% of patients screened positive, still only one quarter of events correctly screened positive and PPV was just 0.5%.
Performance of risk prediction model as a screening tool to identify patients at risk of ICU admission.
A model for risk prediction of the secondary outcome (additional analysis 1) performed similarly to the primary outcome model (Supplemental Table S4). Of the 578,310 pregnancies, 2121 (0.37%) resulted in admission to either an ICU or standalone general HDU. For this model, an AUC of 0.65 (95% CI 0.64, 0.66) and Brier score of 0.0036 (95% CI 0.0035, 0.0038) were achieved (Supplemental Figures S1 and S2). For the tertiary outcome (additional analysis 2), among 68,553 for whom data on obstetric HDU admission were available, 1890 (2.76%) were admitted to either ICU, standalone general HDU, or obstetric HDU. The prediction model in this group achieved an AUC of 0.77 (95% CI 0.77, 0.78) with Brier score of 0.026 (95% CI 0.25, 0.27) (Supplemental Figures S3 and S4).
Models produced in additional analysis 3 had better discriminative performance than the primary analysis (Supplemental Table S5). Complete data pertaining to delivery-related factors were available for 553,164 pregnancies, 978 (0.18%) of which involved admission to ICU. This prediction, which included five additional delivery-related variables, achieved an AUC 0.78 (95% CI 0.76, 0.80) and Brier score 0.0018 (95% CI 0.0017, 0.0019) (Supplemental Figures S5 and S6). Odds ratios, ROC curves, calibration curves and performance as screening tools (Supplemental Tables S6–S8) for the risk prediction models for the secondary/tertiary outcomes and additional analyses can be found in the Supplemental Material.
Discussion
We have identified multiple independent risk factors for maternal intensive care admission. Improving identification of women at risk of becoming critically ill during pregnancy has the potential to improve outcomes through pre-emptive care and better targeting of resources. Our primary model has a moderate discriminative performance with an AUC of 0.66. Both the model predicting risk of admission to critical care including obstetric HDU, and the model predicting ICU admission including delivery-related factors had superior discriminative ability to the primary outcome model, most likely due to the higher event frequency in the former and including factors more likely to influence admission in the latter. All models showed a favourable level of calibration. However, despite reasonable model performance, the low frequency of ICU admission and the associated low positive predictive value means that the utility of the model in clinical practice remains uncertain.
Our study has a number of strengths. Women cared for in an obstetric HDU setting are often not captured by current national measures of critical illness in pregnancy. The inclusion of these patients in our additional analysis increased the external validity of our study. We were able to include information about women before their pregnancy from routinely collected health data, as well as information related to the current pregnancy which would be available earlier in pregnancy. This allowed the prediction model to be based on information that would realistically be available at the time of booking or soon thereafter.
There are several limitations to this study. In performing a complete-case analysis and omitting observations with missing values, there is a possibility that the resultant estimates are biased as there were some minor differences in baseline characteristics between those included and excluded from analyses (Supplemental Table S1). Furthermore, the high proportion of patients with data missing for BMI may have led to the loss of discriminating data, rendering our model less accurate and producing estimates of model performance with lower precision. However, baseline characteristics between those included in the complete case analysis and those excluded did not demonstrate substantial variation in distribution of variables, indicating that a complete case analysis is less likely to produce biased estimates of model performance. An additional limitation is the time that has elapsed since the end of the study population eligibility period (2018). Secular trends in maternity populations include older age of first pregnancy, increasing BMI and increasing maternal comorbidity. 12 This may mean that performance metrics for our model applied to current cohorts of pregnant women may perform less well than those reported in this study. This stresses the importance of external validation being undertaken in a more up-to-date population to ensure model performance is retained.
In addition, our choice of ICU admission as the primary outcome captures one end of the spectrum of severe maternal morbidity. Identifying women at risk of severe maternal morbidity would potentially be of greater clinical relevance to target pre-emptive planning. However, our previous work demonstrated that defining severe maternal morbidity in routine data overestimated severe maternal morbidity, largely due to miscoding of maternal infection as sepsis. 2 Our secondary analyses broadening the outcome to include general and obstetric HDU admissions may go some way to mitigate this limitation. Whilst internal validation was undertaken using cross-validation, our model would require external validation to evaluate generalisability if it were found to be of benefit in a clinical context. Of note, the Brier score is a relatively insensitive measure of model accuracy in the context of predicting a rare event, such as ICU admission. 13
Our findings are in agreement with others who have developed risk prediction models for maternal critical care admission. Among 600,000 women in Ontario, Canada, Dayan et al. identified those experiencing severe maternal morbidity as being older, more likely to have a comorbidity, and more likely to have undergone infertility treatment. Akin to our analysis, the authors found prior pregnancy factors to enhance the discriminative ability of their model. 14 Using data for 18 million live births in the United States from 2012 to 2016, Rossi et al. 15 identified 14 variables associated with maternal ICU admission and developed a model for the prediction of this outcome, achieving an AUC of 0.81. Despite their model demonstrating better discriminative ability, the authors found that the PPV of their model remained unacceptably low across all cut-points. For example, a cut-point of 5% predicted risk missed most (98%) of the women who would go on to be admitted to ICU. This is echoed by Grobman et al. in their analysis of a predictive model for SMM. While they demonstrated an AUC of 0.80, there was no cut-point that reached an acceptable balance between PPV and sensitivity. 16
This is also true of our model. Although the AUC of 0.66 is indicative that the model has acceptable discriminative ability, as ICU admission incidence is rare, the majority of women who would go on to be admitted would not be identified by the model using even a comparatively low probability threshold.
Prediction models are often either implemented as a ‘rule-in’ test – maximising specificity often to the detriment of sensitivity – or a ‘rule-out’ test – maximising sensitivity over specificity. Prediction models for a rare outcome such as maternal ICU admission, therefore, are unlikely to be useful as a ‘rule-in’ test. At best, their role would be as a ‘rule-out’ test to exclude women from pre-emptive consideration of ICU admission, or at least as a first step in automated screening, before further screening, often a more labour-intensive clinical risk assessment. For example, using a probability threshold of 0.1%, almost one in five women would be ‘screened out’ after application of the prediction model, leaving 478,775 (82.8%). This threshold would maximise sensitivity (91.6%), resulting in only 91/1087 (8.4%) false negatives (i.e. women predicted to not need ICU, when they were ultimately admitted). In contrast, if no model were applied and a random 82.8% sample of women were included, this would result in 187/1087 (17.2%) false negatives, demonstrating some degree of improvement in classification through application of the model.
The UK’s National Institute for Health and Care Excellence (NICE) guidance recommends all pregnant women should have ‘risk assessments on the health and wellbeing of the woman and her baby’ at the booking appointment and any subsequent antenatal appointments. 8 Operationalising such risk assessments can be time-consuming for both pregnant women and health care professionals. If an automated risk prediction algorithm could be implemented as the first stage in a two (or more) stage screening process, this would have the potential to allow a more abbreviated risk assessment for 20% of women who were ‘screened out’. Such relatively modest potential improvements in efficiency have a greater impact when implemented at scale on a population level. Careful evaluation of current antenatal risk assessment processes is needed to understand if such hypothetical benefits could be realised in a real-world setting. Furthermore, the risk factors identified in our statistical models do not necessarily translate into potential targets for intervention. Coefficients associated with individual covariates in a risk prediction model are not grounded in a covariate-specific causal framework, which requires aetiological, rather than predictive, model development. 17 A wider understanding of causal literature relating to each individual covariate would identify established interventions or future research that would be required to develop interventions for each individual covariate to improve outcomes.
Conclusion
Our model for maternal ICU admission has an acceptable discriminative performance and calibration. The low frequency of ICU admission and resulting low PPV indicates that the model would be unlikely to be useful as a ‘rule-in’ test for pre-emptive consideration of ICU admission. Its potential for improving efficiency in screening as a ‘rule-out’ test remains uncertain.
Supplemental Material
sj-docx-1-inc-10.1177_17511437251313700 – Supplemental material for Predicting risk of maternal critical care admission in Scotland: Development of a risk prediction model
Supplemental material, sj-docx-1-inc-10.1177_17511437251313700 for Predicting risk of maternal critical care admission in Scotland: Development of a risk prediction model by Lorna M Cowan, Imad Adamestam, John A Masterson, Monika Beatty, James P Boardman, Louis Chislett, Pamela Johnston, Judith Joss, Heather Lawrence, Kerry Litchfield, Nicholas Plummer, Stella Rhode, Timothy S Walsh, Arlene Wise, Rachael Wood, Christopher J Weir and Nazir I Lone in Journal of the Intensive Care Society
Footnotes
Acknowledgements
We are grateful to Roz Pollock for deriving the cohort and linking the datasets. We are grateful to the late Prof. Denison who contributed to study design, analysis and securing funding. We thank the eDRIS Team (Public Health Scotland) for its involvement in obtaining approvals, provisioning and linking data and the use of the secure analytical platform within the National Safe Haven.
Authors’ note
*Presented at European Society of Intensive Care Medicine 36th Annual Congress in Milan, October 2023.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Obstetric Anaesthetists’ Association.
ORCID iDs
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
