External Validation of the Post-Concussion Symptoms Rule for Predicting Mild Traumatic Brain Injury Outcome

Abstract

Persistent symptoms are common after a mild traumatic brain injury (mTBI). The Post-Concussion Symptoms (PoCS) Rule is a newly developed clinical decision rule for the prediction of persistent post-concussion symptoms (PPCS) 3 months after an mTBI. The PoCS Rule includes assessment of demographic and clinical characteristics and headache presence in the emergency department (ED), and follow-up assessment of symptoms at 7 days post-injury using two thresholds (lower/higher) for symptom scoring. We examined the PoCS Rule in an independent sample. We analyzed a clinical trial that recruited participants with mTBI from EDs in Greater Vancouver, Canada. The primary analysis used data from 236 participants, who were randomized to a usual care control group, and completed the Rivermead Postconcussion Symptoms Questionnaire at 3 months. The primary outcome was PPCS, as defined by the PoCS authors. We assessed the overall performance of the PoCS rule (area under the receiver operating characteristic curve [AUC]), sensitivity, and specificity. More than 40% of participants (median age 38 years, 59% female) reported PPCS at 3 months. Most participants (88%) were categorized as being at medium risk based on the ED assessment, and a majority were considered as being at high risk according to the final PoCS Rule (81% using a lower threshold and 72% using a higher threshold). The PoCS Rule showed a sensitivity of 93% (95% confidence interval [CI], 88–98; lower threshold) and 85% (95% CI, 78–92; higher threshold), and a specificity of 28% (95% CI, 21–36) and 37% (95% CI, 29–46), respectively. The overall performance was modest (AUC 0.61, 95% CI 0.59, 0.65). In conclusion, the PoCS Rule was sensitive for PPCS, but had a low specificity in our sample. Follow-up assessment of symptoms can improve risk stratification after mTBI.

Introduction

At least one in four adults experiences persistent post-concussion symptoms (PPCS) 3–6 months after a mild traumatic brain injury (mTBI).¹ Identifying who is at increased risk of persistent symptoms can support clinical decision making and proactive targeted treatment.² Socio-demographic (e.g., gender) and clinical (e.g., pre-injury mental health problems) variables have shown consistent associations with PPCS.³ In particular, early post-concussion symptoms (measured with a standardized checklist at 7–14 days following injury) emerged as strong predictors of PPCS in different mTBI cohorts and prognostic models worldwide.^4

–8 However, there is no accepted risk stratification system to support the clinical management of adults with mTBI.

The Post-Concussion Symptoms (PoCS) Rule (Fig. 1) is a newly developed clinical decision rule for the early prediction of PPCS, and is intended to help clinicians streamline follow-up care after mTBI.⁹ It includes stratification based on the characteristics assessed in the emergency department (ED) into low/medium/high risk. Medium-risk patients are further stratified based on early symptoms at 1 week follow-up. The final PoCS prediction rule encompasses ED and follow-up assessments, and classifies patients into low risk, for whom scheduled follow-up may not be necessary, and high risk, who may benefit from a follow-up intervention. The authors of the PoCS Rule reported sensitivity of 91% and specificity of 54% in the derivation cohort, and somewhat lower values in their validation cohort (90% and 44%, respectively).⁹

FIG. 1.

The Post-Concussion Symptoms (PoCS) Rule. Reproduced with permission from Le Sage et al.⁹

Before considering the widespread implementation of a clinical prediction rule, its performance should be examined in independent cohorts to determine its generalizability (external validity).^10
–12 External validation studies examine if a prediction rule or model is transportable to other geographical regions, time periods, data collection methods, and/or teams of investigators at other sites.¹¹ Although the authors of the PoCS Rule validated the rule in a similar cohort recruited subsequently from the same trauma centers,⁹ further external validation studies are needed. The present study aimed to perform an external validation of the PoCS Rule in an independent sample.

Methods

Study population

The present study was a secondary analysis of a clinical trial examining the effectiveness of a practice guideline implementation tool for mental health screening and treatment initiation in primary care, the Mobilizing Early Management of Mental Health Complications After Mild Traumatic Brain Injury (M4) study.¹³ Study procedures were approved by the University of British Columbia's research ethics board (H20-00562). The M4 study recruited participants who presented to an ED or urgent care center in Greater Vancouver, British Columbia (8 sites in total) within 72 h of mTBI. The trial included patients 18–69 years of age with the diagnosis of mTBI confirmed with a structured interview (based on World Health Organization Neurotrauma Task Force diagnostic criteria).¹⁴ Patients were excluded if they had had a prior mTBI within 6 months preceding the index injury, or a pre-existing severe and unstable medical (e.g., cancer) or psychiatric (e.g., psychotic disorder requiring hospital admission in the previous year) condition.

Participants were randomized (1:1) to usual care (“control group”) or an experimental intervention that involved providing the participant and their general practitioner with tailored information about their mental health screening test results and associated recommendations from a clinical practice guideline.¹⁵ Details of the M4 study protocol can be found in the study by Silverberg and coworkers.¹³ The PoCS cohort included patients with mTBI according to the same definition,¹⁴ but who were ≥14 years of age and presented to ED within 24 h of injury.

Out of 537 participants enrolled in the M4 study, we included the 499 who completed the 3-month Rivermead Post-concussion Symptom Questionnaire (RPQ). The remainder were unreachable (n = 31), withdrew (n = 4), or did not complete the RPQ (n = 3). Because of the possibility that the intervention may have influenced 3-month outcomes, the primary analysis was performed on 236 participants randomized to the control group. A secondary analysis was performed that included all 499 participants.

Assessment of the PoCS Rule components

The PoCS Rule derivation study assessed participants in the ED during their visit and administered the RPQ¹⁶ by telephone at 7 days post-injury (Fig. 1).⁹ In contrast, the M4 study extracted information from ED charts retrospectively and had participants complete the RPQ online (self-report as a Research Electronic Data Capture [REDCap] survey) at 14 days post-injury. All PoCS Rule components except for “prior TBI <1 year” were assessed in the M4 study. Although the time of previous TBI was not measured, the study excluded participants who had had a prior mTBI in the preceding 6 months or a moderate-severe TBI that they were still recovering from; therefore, this predictor effect was set to 0 for all included participants. A history of mental health disorder was defined as a self-reported history of diagnosis or treatment for depression or anxiety. A cervical sprain was defined as a self-reported whiplash injury at the time of mTBI. Computed tomography of the head was performed as clinically indicated.¹⁷ Any abnormal (trauma-related intracranial, skull fracture, or pre-existing) findings were reviewed by a board-certified neurologist (W.P.) and coded according to the National Institute of Neurological Disorders and Stroke (NINDS) TBI Neuroimaging Common Data Elements.¹⁸

The RPQ measured the severity of 16 common symptoms within the last 24 h following mTBI, on a scale from 0 (“never experienced”) to 4 (“severe problem”). The total RPQ score was calculated as the sum of all RPQ items. The rule included a dichotomized RPQ total score (< 21 vs. ≥21) and dichotomized RPQ item scores for symptoms (headaches, sleep disturbance, fatigue, and light sensitivity; Fig. 1). Two thresholds were used for scoring individual symptoms: with the lower threshold, symptoms were considered present if their item scores on the RPQ were ≥2; the higher threshold used RPQ cutoff values ≥3.⁹

Assessment of 3-month outcomes

In the PoCS Rule derivation study, outcome assessments (including the RPQ) were completed by telephone. The primary outcome (PPCS) was defined as having post-concussion symptoms at 3 months that were: (1) spontaneously described (responded positively to an interview prompt “I would like to know whether you still have any symptoms from your head trauma?”), (2) persistent (present since follow-up at 7 days), and (3) severe (at least one severe symptom or two moderate symptoms on the RPQ), and that (4) have an impact on patient's life (non-return to work or non-return to normal activities as a result of the brain injury). Secondary outcomes were defined as “moderate to severe symptoms” (at least 3 symptoms with item scores ≥3 on the RPQ), and “non-return to work.”

In the M4 study, interview-based outcome assessment measures (e.g., Glasgow Outcome Scale Extended [GOSE]) were completed in a live videoconference meeting, and self-reported outcomes (including the RPQ) were administered online as a REDCap survey, typically on the same day. The GOSE is a core NINDS Common Data Element for mTBI and a structured interview measuring the change in functioning from pre-injury status.¹⁹ It provides eight categories of outcome ranging from dead (1) to complete return to a pre-injury level of functioning (8). We created a primary outcome (PPCS) that was as similar as possible to that used in the PoCS derivation study. We defined PPCS as reporting post-concussion symptoms on the RPQ at 3 months that were persistent (present since follow-up at 2 weeks) or severe (at least one severe symptom or two moderate symptoms on the RPQ), and that have an impact on the patient's life (GOSE <8). We defined the secondary outcome “moderate to severe symptoms” as in the PoCS derivation study, and “non-return to work” as a negative response to question 5 on the GOSE: “Have you been able to resume employment and similar roles?”

Statistical analysis

Patient characteristics, the PoCS components, and outcome variables were described using frequencies and medians (with interquartile interval [Q1–Q3]). Missing values in components and 3-month RPQ items²⁰ were imputed using multiple imputation (multivariate imputation by chained equations: mice²¹) with 10 imputations. The imputation model contained all PoCS components, RPQ items, GOSE and auxiliary variables (Glasgow Coma Scale score, results of computed tomography (CT), and Patient Health Questionnaire- 9 and General Anxiety Disorder- 7 at 2 weeks and 3 months). Predictive mean matching was used for imputing missing item scores.²⁰

Diagnostic accuracy

We examined the performance of the PoCS Rules in terms of sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Sensitivities, specificities, PPV, NPV, and standard errors were pooled over imputed data sets and combined by Rubin's rules.²² The areas under the receiver operating characteristics curve (AUC) were also reported. Their 95% confidence intervals (CI) were calculated using bootstrapping (500 bootstrap samples)

Supplementary analyses

We also examined the performance of the regression models on which the PoCS Rule was based: a model with predictors assessed at ED (“ED model”) and a model with predictors assessed at follow-up (“follow-up model,” Supplementary Table 4 in Le Sage⁹; corrected coefficients via personal communication). The performance of the follow-up model was examined in the medium-risk ED strata, as in the PoCS derivation study. We assessed discriminative ability and calibration slope. The intercept of the underlying models was not reported in the derivation study; therefore, the calibration-in-the-large was not assessed. Discrimination refers to the ability of a model to accurately discriminate between patients with and without persistent symptoms at 3 months, and it was measured by the AUC. Calibration slope was used to assess the overall validity of the regression coefficients, with ideal slope being equal to 1. To disentangle the effects of case-mix heterogeneity and slope on model performance, we calculated the model-based concordance (mbc)²³ that is affected only by case-mix heterogeneity and not the validity of coefficients

To examine whether the model could be improved even without adding new predictors, we assessed the maximum performance of the follow-up model in the M4 study, quantified by refitted AUC. Specifically, we refitted the original logistic follow-up model from the PoCS study (with dichotomized RPQ total and item scores as predictors) in the M4 study. Subsequently, we fitted a logistic model with the RPQ total score as the only predictor, and calculated the AUC. Supplementary analyses were performed in the control group and in the entire cohort using one imputed data set. Overall, analyses were performed in R 4.2.1., using rms²⁴ and mice²¹ packages, and val.prob.ci.2²⁵ and mb.c²⁶ functions.

Results

The characteristics of the study population, components of the PoCS rule, risk stratification, and the outcomes in the control group and the full sample of the M4 study are presented in Table 1. The median age in the M4 control group was 38 years (Q1–Q3: 29–52), and the slight majority of participants were female (59%). A substantial proportion of participants reported pre-injury depression or anxiety (42%). The majority of participants sustained their mTBI as a result of falls or sport activities, and approximately 25% had documented loss of consciousness and/or post-traumatic amnesia in the ED. Four participants had hemorrhage on CT. A third to half of participants had a poor 3-month outcome based on the primary or secondary outcome definitions (Table 1).

Table 1.

Characteristics of the M4 Study Population: ED Evaluation, Follow-up Assessment, and Outcomes

	Control group	All participants	Missing %
n	236	499
Sociodemographic and pre-injury variables
Age (median [Q1, Q3])	38 [29, 52]	36 [27, 50.5]	0
Age 18-24 (%)	35 (14.8)	90 (18.0)	0
Age 25-34 (%)	59 (25.0)	142 (28.5)	0
Age 35-44 (%)	52 (22.0)	89 (17.8)	0
Age 45-64 (%)	82 (34.7)	162 (32.5)	0
Age 65-69 (%)	8 (3.4)	16 (3.2)	0
Sex = F (%)	139 (58.9)	278 (55.7)	0
Prior multiple TBI (%)	50 (21.2)	108 (21.6)	0
Prior mental health disorder (%)	98 (41.5)	204 (40.9)	0
Injury variables (ED)
Mechanism of injury (%)			0.2
Assault	15 (6.4)	33 (6.6)
Fall	93 (39.4)	186 (37.3)
Fall height	4 (1.7)	8 (1.6)
MVA	26 (11.0)	65 (13.1)
Sport	64 (27.1)	130 (26.1)
Other	34 (14.4)	76 (15.3)
Loss of consciousness (%)
Yes, <30 min	61 (25.8)	146 (29.3)
Absent	121 (51.3)	244 (48.9)
Queried, unclear	34 (14.4)	70 (14.0)
Not documented	20 (8.5)	39 (7.8)
Post-traumatic amnesia (%)
Yes, <24 h	53 (22.5)	117 (23.4)
Absent	26 (11.0)	67 (13.4)
Queried, unclear	6 (2.5)	8 (1.6)
Not documented	151 (64.0)	307 (61.5)
Headache (%)	150 (63.6)	303 (60.7)	0
GCS first hospital (%)			13.6
13	1 (0.5)	1 (0.2)
14	12 (6.0)	27 (6.3)
15	187 (93.5)	403 (93.5)
Whiplash/ sprain (%)	5 (2.1)	14 (2.8)	1
Hemorrhage on CT (%)	4 (1.7)	7 (1.4)	0
Follow-up at 2 weeks
Headache ≥2^a (%)	173 (73.6)	370 (74.3)	0.2
Sleep disturbances ≥2^a (%)	134 (57.5)	285 (57.5)	0.6
Fatigue ≥2^a (%)	189 (80.4)	389 (78.4)	0.6
Light sensitivity ≥2^a (%)	116 (49.4)	244 (49.0)	0.2
RPQ ≥21^b (%)	133 (61.0)	290 (61.6)	5.6
Outcomes at 3 months
Persistent symptoms (primary) (%)	88 (41.3)	195 (41.9)	6.8
Moderate to severe symptoms (%)	102 (45.7)	209 (44.2)	5.2
Non-return to work (%)	85 (36.2)	187 (37.6)	0.4

Item on the Rivermead Postconcussion Symptoms Questionnaire (RPQ)

Sum of all items on RPQ.

M4, Mobilizing Early Management of Mental Health Complications After Mild Traumatic Brain Injury; ED, emergency department; TBI, traumatic brain injury; MVA, motor vehicle accident; GCS, Glasgow Coma Scale score; CT, computed tomography.

Based on the ED assessment of the PoCS Rule, most participants (204 of 233, 88%) were categorized as being at “medium risk” for persistent symptoms. Out of these 204 participants, 170 (89%) were in the “high risk” category based on the follow-up assessment of individual symptoms (headaches, sleep disturbances, fatigue, light sensitivity) and total RPQ score using a lower threshold, and 151 (79%) were placed in the “high risk” category using a higher threshold. Overall, the PoCS Rule was positive for 81% of participants in the control group using a lower threshold for scoring individual symptoms and for 72% participants using a higher threshold. The PoCS Rule positivity was somewhat lower in the entire cohort compared to the control group (77% and 70%, respectively).

Diagnostic accuracy

The PoCS Rule was sensitive for predicting persistent symptoms, particularly using a lower threshold for scoring symptoms at follow-up (≥ 2; Table 2). The specificities were 28% and 37%, and the overall performance measured by the AUC was modest (AUC = 0.61, Table 2). The NPV was higher than the PPV (Table 2). The results were consistent for the secondary outcome moderate to severe symptoms. For the secondary outcome non-return to work, lower sensitivities and specificities were obtained (Tables S1 and S2). In the overall M4 cohort, the results were comparable with those of the control group, with slightly better performance (higher AUC, sensitivity, and specificity) of the PoCS Rule with a higher threshold (Table S1).

Table 2.

Performance of the Post-Concussion Symptoms (PoCS) Rule for Persistent Post-Concussion Symptoms (PPCS) After Mild Traumatic Brain Injury

	M4 study: control group n = 236, 105 ^a with PPCS		PoCS study: derivation n = 471, 71^* with PPCS		PoCS study: validation n = 602, 96^* with PPCS
	PoCS with follow-up ^b ≥2	PoCS with follow-up ≥3	PoCS with follow-up ≥2	PoCS with follow-up ≥3	PoCS with follow-up ≥2	PoCS with follow-up ≥3
Sensitivity 95% CI	93 (88, 98)	85 (78, 92)	91 (85, 98)	87 (79, 95)	90 (84, 96)	87 (80, 93)
Specificity 95% CI	28 (21, 36)	37 (29, 46)	54 (49, 59)	67 (62, 72)	45 (40, 49)	56 (52, 61)
NPV 95% CI	84 (73, 95)	75 (65, 86)	97 (95, 99)	97 (94, 99)	96 (93, 98)	96 (93, 98)
PPV 95% CI	51 (44, 58)	52 (45, 60)	27 (21, 32)	32 (26, 39)	24 (19, 28)	27 (22, 32)
AUC 95% CI	0.61 (0.59, 0.65)	0.61 (0.59, 0.66)	Not reported	Not reported	Not reported	Not reported

Original values reported in Le Sage.⁹

Missing items (7%) imputed using multiple imputation, indices averaged across 10 imputed data sets.

Threshold for scoring individual symptoms.

M4, Mobilizing Early Management of Mental Health Complications After Mild Traumatic Brain Injury; CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value; AUC, area under the curve.

Supplementary analyses

Discriminative abilities of the models with ED predictors and follow-up predictors were lower than in the derivation study (Table 3), which can (partly) be explained by lower predictor effects in the validation study (slope <1, Table 3). For the ED model, the case-mix of the validation study was more homogeneous (i.e. lower mbc compared with AUC in the derivation study; Table 3), contributing to the lower discriminative ability in the validation.

Table 3.

Discriminative Ability (AUC) of the Post-Concussion Symptoms (PoCS) Models in the M4 Control Group (n = 236)^a

	PoCS derivation study AUC	Model-based Concordance (mbc)	Validation AUC 95%	Slope 95% CI
ED model	0.75	0.71	0.66 (0.59-0.73)	0.71 (0.36- 1.05)
Follow-up model^b	0.85	0.85	0.72 (0.65-0.79)	0.60 (0.38-0.83)

Missing data imputed using multiple imputation and analyses performed using one imputed data set.

Only in the medium risk group based on ED assessment (n = 207/236).

M4, Mobilizing Early Management of Mental Health Complications After Mild Traumatic Brain Injury; CI, confidence intervals; AUC, area under the curve.

When we refitted the follow-up model with (dichotomized) predictors to obtain the maximum performance in our data, we obtained higher discriminative ability (AUC = 0.74, 95% CI [0.67, 0.80]). The inclusion of the 2-week RPQ total score as the only predictor yielded better discriminative ability (AUC = 0.79, 95% CI [0.73, 0.85]). In the entire M4 cohort, consistent patterns were obtained but with better discrimination and slope (Table S3).

Discussion

We examined the performance of the PoCS Rule for risk stratification of patients with mTBI in an independent study. The PoCS Rule indicated high risk (clinical follow-up or interventions advised) for the majority of participants, using both the lower and higher threshold. The sensitivity for predicting PPCS at 3 months was relatively high (93% and 85%) and comparable to the PoCS study. However, a lower specificity (28% and 37%) was obtained compared with both the PoCS derivation cohort (54% and 67%) and the PoCS validation cohort (45% and 56%). Relative to the PoCS study, the NPV was lower and PPV was higher in this study, which was expected given the substantially higher prevalence of PPCS. In both studies, the NPV was considerably higher than the PPV. When the aim is to identify or triage patients who are unlikely to develop PPCS, the PoCS Rule appears effective. For example, it may be used to enrich clinical trials⁶ by screening out patients who are very unlikely to develop PPCS. However, the low specificity obtained in both our, and the PoCS validation cohort, can be a concern in contexts in which it is important to reduce the number of patients referred to follow-up care.

Our results confirm the value of follow-up assessment of 1–2-week post-concussion symptoms for patient stratification and prediction of PPCS after mTBI, as also demonstrated by other recent studies and available prognostic models.^5,6,27,28 Most participants were categorized as being at “medium risk” based on ED assessment, meaning that their final risk stratification in the PoCS Rule was determined based on the follow-up symptom assessment. It should be noted that the PoCS Rule includes only headache as a symptom assessed at ED. Assessing a fuller range of symptoms in the ED might have led to better stratification based on the ED assessment.^8,29,30 However, based on previous studies,^5,27,28,31 we would still expect symptom assessment at 1–2 weeks to outperform ED assessment of symptoms in predicting outcomes 3–6 months post-injury or later. The inclusion of other pre-injury comorbidities and social determinants of health may also lead to more efficient ED stratification.^{5,27,31
–33}

Moreover, our analyses showed that the discriminative ability was higher for the follow-up assessment than for the ED assessment, particularly when the total score was treated as continuous. Dichotomizing continuous and ordinal predictors, and using a binary prediction rule over a prediction model usually leads to a loss of information³⁴ and negatively affects performance in other cohorts.¹² Using the continuous RPQ total score as a predictor may maximize the identification of patients at risk for PPCS. If there is a need for a binary classification, the risk categories can be assigned post-hoc (e.g., a patient with a predicted probability of PPCS of 60% classified as high risk). Additionally, assessing other prominent predictors of PPCS that were not considered in the development of the PoCS Rule, such as mental health symptoms (e.g., post-traumatic stress and depression)³⁵ and psychological risk factors (e.g., maladaptive coping)^31,36 may further improve identification of patients at risk and referral to interventions.

Both the PoCS derivation study and M4 study used for validation recruited participants from EDs in Canada, using the same definition of mTBI.³⁷ However, there are some differences between our study and the original PoCS study that could have led to discrepancies in performance. First, our study had a substantially higher prevalence of PPCS at 3 months. The RPQ in our study used a self-administered online questionnaire, whereas in the PoCS study research assistants administered the RPQ over the telephone. Our definition of persistent symptoms could not incorporate the criterion of “spontaneity” (i.e., asking patients about their symptoms only if they responded affirmatory to: “I would like to know whether you still have any symptoms from your head trauma?”). These differences in the assessment could have resulted in a higher prevalence of persistent symptoms in our study.³⁸

In addition, although the PoCS Rule was derived in an observational cohort, the validation was performed in a trial that might have recruited patients at higher risk of symptoms because of self-selection bias. As this trial was explicitly about mental health management, patients with prior mental health problems and/or post-injury emotional distress might have been more motivated to participate. This may partly explain why 41% of participants in the M4 study had prior mental health problems and >40% had persistent symptoms at 3 months post-injury in comparison with 20% and >15%, respectively, in the PoCS derivation study. In line with previous studies, pre-injury mental health disorder was associated with persistent symptoms in the M4 study (Supplementary Table S4). As pre-injury mental health disorder was a component of the PoCS Rule, patients with depression or anxiety would be more likely to be (accurately) considered high risk. Some components were differently defined between the two studies, such as cervical sprain (that was self-reported in our study). A lower case-mix heterogeneity of the ED assessment in our study could have also contributed to lower performance.³⁹ For example, our study had a very low prevalence of CT abnormalities and cervical sprain. Further, it is plausible that the two studies differed in the distributions of important unobserved predictors, such as psychological risk factors (e.g., pessimistic beliefs about recovery) and social determinants of health, which would influence the performance of the rule.¹¹ Moreover, there are some methodological limitations of the PoCS study that may be associated with differences in performance.³⁹ In the development of the rule, the number of candidate predictors was higher than recommended⁹ for the effective sample size, and continuous and ordinal predictors were dichotomized, potentially resulting in a loss of information.^11,34 Finally, the PoCS Rule has a relatively low specificity in both the PoCS study and our study. To reduce the number of patients unnecessarily referred to further follow-up and interventions, in addition to using a full scale of post-concussion symptom severity, prediction may be improved by the inclusion of additional ED and early follow-up variables.

The present study has its own limitations. Most notably, we did not measure all rule components and outcomes identically to the PoCS derivation study, as described. Further external validation studies are necessary for examining the generalizability of the PoCS rule,¹¹ particularly using data from observational cohorts. Our validation study may inform future trials for the treatment of symptoms after mTBI. In conclusion, the PoCS Rule was sensitive for predicting 3-month PPCS in our sample, but the specificity was low. Follow-up assessment of symptoms can improve risk stratification after mTBI. Using a full range of post-concussion symptom severity (e.g., continuous RPQ total scores instead of dichotomized scores) and assessing early post-injury psychological factors may further improve the prediction of persistent symptoms after mTBI. To support feasibility, routine symptom assessment after ED discharge could be implemented with remote technologies such as smart phone applications or questionnaires pushed out from electronic medical record systems.

Transparency, Rigor, and Reproducibility Summary

This study is a secondary analysis of a clinical trial. The trial was pre-registered at ClinicalTrials.gov (Identifier: NCT04704037). The analysis plan for this secondary analysis was not formally pre-registered, but it was pre-specified, including mapping the components of the examined PoCS Rule to the derivation study. The number of participants recruited (537 in total), the number of participants who completed the 3-month assessments (499 in total, 237 in the control group) and the reasons for non-completion of the 3-month assessments are reported in the article. Outcome assessments were performed by research assistants blinded to relevant characteristics (e.g., the PoCS Rule prognostic score) of the participants, and not involved in the data analysis. Missing data have been handled using multiple imputation, as reported in the text. Confidence intervals have been reported in the abstract for primary outcome(s) and in the article for all outcomes. The algorithm of the analytical code is described or referenced in the article, and more detailed information is available by contacting the corresponding author.

Footnotes

Authors' Contributions

N.D.S. and A.M. were responsible for conceptualization. A.M., P.M.A.B., and N.D.S. were responsible for methodology. A.M. was responsible for formal analysis. J.R.B., F.X.S., and A.K. were responsible for data collection (recruitment). A.M. was responsible for writing – original draft. P.M.A.B., J.R.B., F.X.S., W.P., P.A., A.K., and N.D.S. were responsible for writing – review and editing. N.D.S. was responsible for supervision. P.M.A.B., J.R.B., F.X.S., W.P., P.A., A.K., and N.D.S. were responsible for funding acquisition.

Funding Information

This study was funded by the Canadian Institutes for Health Research (CIHR) RN399383-426544. Vancouver General Hospital (VGH) and University of British Columbia (UBC) Hospital Foundation provided salary support for A.M.

Author Disclosure Statement

No competing financial interests exist.

Supplementary Material

Supplementary Table S1

Supplementary Table S2

Supplementary Table S3

Supplementary Table S4

References

Cancelliere

, Verville

, Stubbs

, et al. Post-concussion symptoms and disability in adults with mild traumatic brain injury: a systematic review and meta-analysis. J Neurotrauma, 2023; 40(11-12):1045–1059; doi: 10.1089/neu.2022.0185

Maas

AIR

, Menon

, Manley

, et al. Traumatic brain injury: progress and challenges in prevention, clinical care, and research. Lancet Neurol, 2022; 21(11):1004–1060; doi: 10.1016/s1474-4422(22)00309-x

Silverberg

, Gardner

, Brubacher

, et al. Systematic review of multivariable prognostic models for mild traumatic brain injury. J Neurotrauma, 2015; 32(8):517–526; doi: 10.1089/neu.2014.3600

Brett

, Kramer

, Whyte

, et al. Latent profile analysis of neuropsychiatric symptoms and cognitive function of adults 2 weeks after traumatic brain injury: findings From the TRACK-TBI study. JAMA network open, 2021; 4(3):e213467–e213467; doi: 10.1001/jamanetworkopen.2021.3467

Mikolić

, Steyerberg

, Polinder

, et al. Prognostic models for global functional outcome and post-concussion symptoms following mild traumatic brain injury: a Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI) study. J Neurotrauma, 2023; 40(15–16):1651–1670; doi: 10.1089/neu.2022.0320

Temkin

, Machamer

, Dikmen

, et al. Risk factors for high symptom burden three months after traumatic brain injury and implications for clinical trial design: a Transforming Research and Clinical Knowledge in Traumatic Brain Injury study. J Neurotrauma, 2022; 39(21–22):1524–1532; doi: 10.1089/neu.2022.0113

Ponsford

, Cameron

, Fitzgerald

, et al. Predictors of postconcussive symptoms 3 months after mild traumatic brain injury. Neuropsychology, 2012; 26(3):304–313; doi: 10.1037/a0027888

Falk

, Bechtold

, Peters

, et al. A prognostic model for predicting one-month outcomes among emergency department patients with mild traumatic brain injury and a presenting Glasgow Coma Scale of fifteen. J Neurotrauma, 2021; 38(19):2714–2722; doi: 10.1089/neu.2021.0137

Le Sage

, Chauny

, Berthelot

, et al. Post-Concussion Symptoms Rule: derivation and validation of a clinical decision rule for early prediction of persistent symptoms after a mild traumatic brain injury. J Neurotrauma, 2022; 39(19–20):1349–1362; doi: 10.1089/neu.2022.0026

10.

Dijkland

, Retel Helmrich

IRA

, Steyerberg

. Validation of prognostic models: challenges and opportunities. J Emerg Crit Care Med, 2018; 2(91):1–4; doi: 10.2103/7/jeccm.2018.10.10

11.

Steyerberg

EW.

Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer International Publishing: Cham; 2019.

12.

Cowley

, Farewell

, Maguire

, et al. Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature. Diagn Progn Res, 2019; 3(1):16; doi: 10.1186/s41512-019-0060-y

13.

Silverberg

, Otamendi

, Brasher

, et al. Effectiveness of a guideline implementation tool for supporting management of mental health complications after mild traumatic brain injury in primary care: protocol for a randomised controlled trial. BMJ Open, 2022; 12(6):e062527; doi: 10.1136/bmjopen-2022-062527

14.

Holm

, Cassidy

, Carroll

, et al. Summary of the WHO collaborating centre for neurotrauma task force on mild traumatic brain injury. J Rehabil Med, 2005; 37(3):137–141; doi: 10.1080/16501970510027321

15.

Marshall

, Lithopoulos

, Curran

, et al. Living Concussion Guidelines: Guideline for Concussion & Prolonged Symptoms for Adults 18 years of Age or Older. 2023. Available from: https://concussionsontario.org [Last accessed: 12/1/2023].

16.

King

, Crawford

, Wenden

, et al. The Rivermead Post Concussion Symptoms Questionnaire: a measure of symptoms commonly experienced after head injury and its reliability. J Neurol, 1995; 242(9):587–592; doi: 10.1007/BF00868811

17.

Stiell

, Wells

, Vandemheen

, et al. The Canadian CT Head Rule for patients with minor head injury. Lancet, 2001; 357(9266):1391–1396; doi: 10.1016/s0140-6736(00)04561-x

18.

Duhaime

, Gean

, Haacke

, et al. Common data elements in radiologic imaging of traumatic brain injury. Arch Phys Med Rehabil, 2010; 91(11):1661–1666; doi: 10.1016/j.apmr.2010.07.238

19.

Wilson

, Boase

, Nelson

, et al. A Manual for the Glasgow Outcome Scale–Extended Interview. J Neurotrauma, 2021; 38(17):2435–2446; doi: 10.1089/neu.2020.7527

20.

Eekhout

, de Vet

HCW

, Twisk

JWR

, et al. Missing data in a multi-item instrument were best handled by multiple imputation at the item score level. J Clin Epidemiol, 2014; 67(3):335–342; doi: https://doi.org/10.1016/j.jclinepi.2013.09.009

21.

van Buuren

, Groothuis-Oudshoorn

. mice: Multivariate imputation by chained equations in R. J Stat Softw, 2011; 45(3):1–67; doi: 10.1863/7/jss.v045.i03

22.

Rubin

DB.

Multiple imputation for nonresponse in surveys. John Wiley & Sons: New. York; 2004.

23.

van Klaveren

, Gönen

, Steyerberg

, et al. A new concordance measure for risk prediction models in external validation settings. Stat Med, 2016; 35(23):4136–4152; doi: https://doi.org/10.1002/sim.6997

24.

Harrell

FEJ.

rms: Regression Modeling Strategies 6.7.1. 2023. Available from: https://cran.r-project.org/package=rms [Last accessed: 9/12/2023].

25.

Van Calster

, Nieboer

, Vergouwe

, et al. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol, 2016; 74:167–176; doi: 10.1016/j.jclinepi.2015.12.005

26.

van Klaveren

, Steyerberg

, Gönen

, et al. The calibrated model–based concordance improved assessment of discriminative ability in patient clusters of limited sample size. Diagn Progn Res, 2019; 3(1):11; doi: 10.1186/s41512-019-0055-8

27.

Cnossen

, van der Naalt

, Spikman

, et al. Prediction of persistent post-concussion symptoms after mild traumatic brain injury. J Neurotrauma, 2018; 35(22):2691–2698; doi: 10.1089/neu.2017.5486

28.

Stulemeijer

, van der Werf

, Borm

, et al. Early prediction of favourable recovery 6 months after mild traumatic brain injury. J Neurol Neurosurg Psychiatry, 2008; 79(8):936–942; doi: 10.1136/jnnp.2007.131250

29.

Nelson

, Furger

, Ranson

, et al. Acute clinical predictors of symptom recovery in emergency department patients with uncomplicated mild traumatic brain injury or non-traumatic brain injuries. J Neurotrauma, 2018; 35(2):249–259; doi: 10.1089/neu.2017.4988

30.

Varner

, Thompson

, de Wit

, et al. Predictors of persistent concussion symptoms in adults with acute mild traumatic brain injury presenting to the emergency department. Can J Emerg Med, 2021; 23:365–373; doi: 10.1007/s43678-020-00076-6

31.

van der Naalt

, Timmerman

, de Koning

, et al. Early predictors of outcome after mild traumatic brain injury (UPFRONT): an observational cohort study. Lancet Neurol, 2017; 16(7):532–540; doi: 10.1016/S1474-4422(17)30117-5

32.

Yue

, Cnossen

, Winkler

, et al. Pre-injury comorbidities are associated with functional impairment and post-concussive symptoms at 3- and 6-months after mild traumatic brain injury: a TRACK-TBI study. Front Neurol, 2019; 10:343; doi:10.3389/fneur.2019.00343

33.

Johnson

, Diaz

. Exploring the social determinants of health and health disparities in traumatic brain injury: a scoping review. Brain Sci, 2023; 13(5):707; doi: 10.3390/brainsci13050707

34.

Wynants

, van Smeden

, McLernon

, et al. Three myths about risk thresholds for prediction models. BMC Med, 2019; 17(1):192; doi: 10.1186/s12916-019-1425-3

35.

de Neeling

, Liessens

, Depreitere

. Relationship between psychosocial and psychiatric risk factors and poor long-term outcome following mild traumatic brain injury: a systematic review. Eur J Neurol, 2023; 30(5):1540–1550; doi: https://doi.org/10.1111/ene.15713

36.

Silverberg

, Panenka

, Iverson

. Fear avoidance and clinical outcomes from mild traumatic brain injury. J Neurotrauma, 2018; 35(16):1864–1873; doi: 10.1089/neu.2018.5662

37.

Holm

, Cassidy

, Carroll

, et al. Summary of the WHO Collaborating Centre for Neurotrauma Task Force on Mild Traumatic Brain Injury. J Rehabil Med, 2005; 37(3):137–141; doi:10.1080/16501970510027321

38.

Iverson

, Brooks

, Ashton

, et al. Interview versus questionnaire symptom reporting in people with the postconcussion syndrome. J Head Trauma Rehabil, 2010; 25(1):23–30; doi: 10.1097/HTR.0b013e3181b4b6ab

39.

Helmrich

IRAR

, Mikolić

, Kent

, et al. Does poor methodological quality of prediction modeling studies translate to poor model performance? An illustration in traumatic brain injury. Diagn Progn Res, 2022; 6(1):8; doi: 10.1186/s41512-022-00122-0

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.39 MB

0.40 MB