A Data-Driven Approach to Unlikely,Possible,Probable,and Definite Acute Concussion Assessment

Abstract

Kutcher and Giza suggested incorporating levels of certainty in concussion diagnosis decisions. These guidelines were based on clinical experience rather than objective data. Therefore, we combined data-driven optimization with predictive modeling to identify which athletes are unlikely to have concussion and to classify remaining athletes as having possible, probable, or definite concussion with diagnostic certainty. We developed and validated our framework using data from the Concussion Assessment, Research, and Education (CARE) Consortium. Acute concussions had assessments at <6 h (n = 1085) and 24–48 h post-injury (n = 1413). Normal performances consisted of assessments at baseline (n = 1635) and the time of unrestricted return to play (n = 1345). We evaluated the distribution of acute concussions and normal performances across risk categories and identified inter-class and intra-class differences in demographics, time-of-injury characteristics, the Standard Assessment of Concussion (SAC), Sport Concussion Assessment Tool (SCAT) symptom assessments, and Balance Error Scoring System (BESS). Our algorithm accurately classified concussions as probable or definite (sensitivity = 91.07–97.40%). Definite and probable concussions had higher SCAT symptom scores than unlikely and possible concussions (p < 0.05). Definite concussions had lower SAC and higher BESS scores (p < 0.05). Baseline to post-injury change scores for the SAC, SCAT symptoms, and BESS were significantly different between acute possible and probable concussions and normal performances (p < 0.05). There were no consistent patterns in demographics across risk categories, although a greater proportion of concussions classified as unlikely were reported immediately compared with definite concussions (p < 0.05). Although clinical interpretation is still needed, our data-driven approach to concussion risk stratification provides a promising step toward evidence-based concussion assessment.

Introduction

Concussion, a type of traumatic brain injury, is an important public health issue that has been associated with potential long-term health consequences.¹ Accurate diagnosis and proper post-injury management of concussion are pragmatic steps for mitigating possible consequences.² However, the clinical diagnosis of concussion is challenging for many reasons,^3,4 especially in the sporting environment, which requires rapid injury assessment and injury management decisions.

Currently, no diagnostic marker or clinical assessment has been designed for perfectly identifying concussion. To this end, existing guidelines call for the evaluation of concussion along multiple domains, including symptom presentation, neurocognitive status, and a physical examination.^5
–7 Previous studies analyzing such multi-dimensional testing batteries for acute concussion assessment have found that symptom evaluations were the most sensitive component of these batteries.^8

–12 However, an over-reliance on symptom presentation for concussion diagnosis is troublesome for several reasons. First, symptoms may be under-reported or go completely unrecognized.^13
–15 Further, common symptoms (e.g., headache, dizziness, fatigue) which are indicative of concussion are not necessarily specific to concussion. Finally, rapid changes in symptom and neurocognitive presentation within the acute stages of concussion result in highly variable clinical presentation of concussion across multiple patients.^16,17

To address these challenges, Kutcher and Giza recommended incorporating diagnostic certainty to the assessment of concussion.⁴ That is, rather than a binary diagnosis paradigm (i.e., concussion or no concussion), Kutcher and Giza suggested that concussion diagnosis should be relayed across a spectrum of risk categories (e.g., possible, probable, and definite concussion), with each category reflecting the degree to which a concussion diagnosis is certain. Similar risk-based categories have been used for classifying diagnosis decisions for other diseases, including multiple sclerosis,¹⁸ Alzheimer's disease,¹⁹ and diabetes.²⁰ Compared with traditional binary diagnosis, risk-based diagnosis frameworks account for the evolution of the injury over time and allow for more flexibility in the post-injury management of concussion. Specifically, incorporating certainty in the assessment of concussion can help to determine whether an athlete should be managed as if he or she had a concussion, ultimately improving the quality of patient care. However, the guidelines developed by Kutcher and Giza were based on clinical experience rather than objective data.

Therefore, the goal of our study is to create a data-driven modeling framework to identify concussed and non-concussed athletes who are unlikely to have a concussion and classify the remaining athletes as having possible, probable, or definite concussion, with each category reflecting increasing diagnostic certainty. Although experienced clinicians may be able to quickly synthesize the likelihood of concussion and ultimately identify a post-injury management plan for athletes, our data-driven framework provides a more objective approach, which can ultimately benefit those clinicians who may be inexperienced in managing concussion. We then aim to validate our framework to identify how athletes are distributed across each risk classification and identify differences in demographics, time-of-injury characteristics, and standard assessment scores among athletes under each risk classification.

Methods

Study population and design

To develop our methodology for classifying athletes as having unlikely, possible, probable, or definite concussion, we used data from the Concussion Assessment, Research, and Education (CARE) Consortium.²¹ The CARE Consortium defines concussion as “a change in brain function following a force to the head, which may be accompanied by temporary loss of consciousness, but is identified in awake individuals with measures of neurologic and cognitive dysfunction.”²² These acquired data contain 33,271 player-seasons collected during the 2014–2018 academic years from 29 National College Athletic Association (NCAA) universities and military service academies. Player-season data were collected across male (57.58%) and female (42.42%) participants from 27 sports, including 19.8% from football, 12.1% from cross country/track, and 9.6% from soccer. The data include 24,561 athletes with pre-season baseline evaluations and 1950 concussions across 1755 athletes. For student-athletes (hereafter, referred to as just athletes) who were diagnosed with concussion by the local institution's medical staff (e.g., team physicians and athletic trainers), additional post-injury data were collected within 6 h of the injury (< 6 h), 24–48 h post-injury (24–48 h), when he or she was identified as asymptomatic, when he or she was cleared for unrestricted return to play (RTP), and 6 months post-RTP. We note that some athletes may not have completed a post-injury assessment at every time point. Therefore, there is some missingness in the data leading to unequal sample sizes across study cohorts. However, these imbalances are not to a degree that has significant effects on the methodologies employed in this research. All participants provided written consent that was approved by their local institutional review board and the United States Army Human Research Protection Office.

Sample selection

In our analysis, we focused on the time points at baseline, <6 h, 24–48 h, and unrestricted RTP. We only included baseline data that could be matched with post-injury data. The assessments at <6 h and 24–48 h were denoted “acute concussion” and those from baseline and unrestricted RTP were denoted “normal performance.” We consider those from the unrestricted RTP time point to demonstrate a normal performance because the subjects had been cleared for RTP by each institution's local medical staff. We analyzed data from <6 h and 24–48 h separately.

Study variables

For each participant in the study data, we obtained demographic information along with raw scores on the Standard Assessment of Concussion (SAC), Sport Concussion Assessment Tool (SCAT) symptom survey, and the Balance Error Scoring System (BESS) at baseline. For those diagnosed with concussion, we obtained time-of-injury characteristics along with raw scores for SAC, SCAT symptoms, and BESS scores at each post-injury evaluation time point. We computed the change score for these athletes by subtracting the raw score at baseline from the raw score at each post-injury time point. A positive change score indicated an increase in the measure compared with baseline, whereas a negative change score indicated a decrease compared with baseline. We also filled missing data elements using multiple imputation by chained equations.²³ We describe our study variables in more detail subsequently.

Demographic information

We aimed to identify differences in each risk classification by age, sex, and the number of previous concussions. Previous studies have suggested that younger athletes, females, and those with greater concussion history are at increased risk for concussion.^24

–29

Time-of-injury characteristics

In our analysis, we included whether the athlete experienced loss of consciousness (LOC), post-traumatic amnesia (PTA), or retrograde amnesia (RGA); whether the athlete was removed from play immediately; and whether the injury was reported immediately, as these variables have been suggested to impact concussion risk.^30

–33

SAC

The SAC is a neurocognitive assessment that measures orientation, immediate memory, concentration, and delayed recall.³⁴ In our analysis, we focused on the SAC total score and change score, both of which summarize the SAC assessment.

SCAT symptom survey

Symptom presentation has been shown, in numerous studies, to be highly associated with acute concussion.^8

–12 The SCAT symptom evaluation includes 22 symptoms, each of which is rated on a scale of 0–6 based on severity.³⁵ In our analysis, we included the total symptom severity and the total number of symptoms, in addition to their respective change scores.

BESS

The BESS is a physical examination that measures postural stability by assessing the number of “movement errors” committed by an athlete while attempting to hold different stances.³⁶ Balance has been noted to be affected by concussion, and we included the BESS total score (across all six stances) and change score in our analysis.

Data analysis

Our overall framework for classifying unlikely, possible, probable, or definite concussion is summarized in Figure 1. To create and evaluate our models, we divided our post-injury data into a training set and a validation set. The training set consisted of all data collected between January 23, 2014 and November 29, 2016, whereas the validation data consisted of all data collected after that date (i.e., November 30, 2016 to October 2, 2017). The CARE Consortium protocol for concussion diagnosis along with assessments performed at baseline and post-injury remained unchanged during this period and, therefore, rater drift, if there was any, was minimal. We used our training set to develop the models to determine which athletes should be classified under each risk category. Then, we applied our models to the validation data to evaluate and analyze our framework. We describe each of the steps in our methodology in more detail subsequently.

FIG. 1.

Illustration of methodological framework for developing data-driven models that were used to classify athletes as having unlikely, possible, probable, or definite concussion based on certainty of acute concussion. CART, Classification Tree; ADASYN, Adaptive Synthetic Sampling.

Model calibration

Using a randomly chosen 40% of the training data, we created a logistical regression model for estimating risk scores. For any athlete, risk scores are a scalar between 0 and 1, where greater risk scores indicate higher likelihood of acute concussion. We used a previously published and validated multivariate logistical regression model (i.e., the raw score model) to estimate risk scores associated with athletes at <6 h and 24–48 h.¹² We used the raw score models, because change scores could not be computed for baseline data and may not always be available for acute concussion assessments in clinical settings. Because time-of-injury characteristics were not available for baselines but were part of these previous models, we assumed in this logistical regression analysis only that for baseline data, injuries were reported immediately and that participants were removed from play immediately.

With the remaining 60% of the training data, we determined risk score thresholds to identify unlikely and definite concussions. We first applied our logistical regression models to this subset of training data to obtain risk scores for each athlete. Then, we used these risk scores as the input for a previously developed data-driven optimization algorithm to determine risk score thresholds (Garcia, G.-G.P., Lavieri, M.S., Jiang, R., McCrea, M., McAllister, T.W., Broglio, S.P., and CARE Consortium Investigators. Data-driven stochastic optimization approaches to determine decision boundaries for medical diagnosis. Submitted for publication.) This algorithm identifies an upper and lower risk threshold by maximizing sensitivity and specificity while limiting false-positive and false-negative rates. Athletes with risk scores below the lower threshold represent those whose concussion probability was low, and who therefore would be identified as having unlikely concussion. Similarly, athletes with risk scores above the upper threshold are most likely to have a concussion and would be classified as having a definite concussion. In designing these thresholds, we favored higher sensitivity over lower false-positive rates and lower false-negative rates over higher specificity.

After determining the upper and lower risk score thresholds, we identified athletes in the training set with risk scores between the unlikely and definite thresholds and used them to determine how athletes should be classified as possible or probable, as these cases could not be easily distinguished by our logistical regression model. Categorization of these cases was approached using a classification and regression tree (CART)³⁷ analysis. CART is a non-parametric statistical modeling technique that produces a decision tree for prediction and is capable of handling categorical variables and continuous variables. Compared with other predictive modeling methods (e.g., generalized linear models), CART is advantageous in its interpretability and ability to model highly non-linear relationships between variables. Because of the higher proportion of normal performances to acute concussions in our data, we applied adaptive synthetic sampling (ADASYN) to mitigate data imbalance issues before creating a CART.³⁸ Additionally, for this CART, we restricted the resulting decision tree to include only variables that were available for all time points. That is, the resulting decision tree did not include time-of-injury characteristics and change scores for the SAC, SCAT symptom assessments, and the BESS, as they were not available for baseline data. Athletes who were predicted to have acute concussions by this CART were classified as having probable concussions, whereas those who were predicted to be normal performances were classified as having possible concussions.

Model validation

To implement our models, we applied our logistical regression models to the validation data to obtain risk scores for each athlete. Then, we compared these risk scores to the upper and lower thresholds we generated using our optimization algorithm in the model calibration phase. Athletes with risk scores below the lower threshold were classified as having unlikely concussions, whereas athletes with risk scores above the upper threshold were classified as having definite concussions. We then applied our CART to any athlete with a risk score between these thresholds to classify them as having a possible or probable concussion.

Model evaluation

After implementing our models on the validation data, we performed additional analysis to evaluate the performance of our classification framework. The goals of this analysis were to (1) analyze how our models classified acute concussions and normal performance throughout each risk category and (2) identify inter-class differences (i.e., differences across different risk classifications) and intra-class differences (i.e., differences within the same risk classification) in demographics, time-of-injury characteristics, and standard assessment scores for acute concussions and normal performances among the risk classifications.

To achieve the first goal, we determined the percentage of acute concussions and normal performances within each risk classification at both <6 h and 24–48 h. Ideally, data captured in the acute post-injury state should place the athlete in greater risk classifications (i.e., definite or probable), whereas data captured at baseline should place the athlete in lower risk classifications (i.e., unlikely or possible). We compared the distribution of acute concussions and normal performances using the Kolmogorov–Smirnov test. A significant p value for this test implies that the distribution of acute concussions and normal performances among the risk classifications is dissimilar.

Because our diagnosis scheme consisted of four risk categories instead of two, we also computed a modified sensitivity and specificity. Our modified computation was founded in recommendations by Kutcher and Giza who indicated that probable and definite concussions should be managed as concussions, whereas possible concussions should be managed based on clinical judgment. Further, we assume that unlikely concussions are managed as non-concussions. Therefore, we provide a sensitivity range where the lower bound reflects the proportion of acute concussions that are correctly classified as probable and definite, and the upper bound reflects the proportion of acute concussions correctly classified as possible, probable, and definite. We also provide a range for specificity, where the lower bound reflects a situation in which no patients with possible concussions are treated as non-concussed, and where the upper bound reflects a situation in which all patients with possible concussions are treated as non-concussed. In practice, the true sensitivity and specificity should fall between these bounds, depending on how possible concussions are managed.

To achieve the second goal, we first identified inter-class differences in the study variables across each risk classification for acute concussions and normal performances using analysis of variance (ANOVA) tests with Tukey's post-hoc comparisons. For example, we determined if athletes with acute concussions who were classified as having a probable concussion had any differences in SAC, SCAT symptoms, or BESS compared with those with acute concussions who were classified as having definite concussions. Next, using Student's t test, we identified intra-class differences in the study variables between acute concussions and normal performances within each risk classification. All models were created and analyzed using Python 3.5.2 (Python Software Foundation, Beaverton, OR).

Results

In Table 1, we summarize the study data at each time point with respect to the study variables. Across all time points, there were significant differences between training and validation data in height (p = 0.0082–0.047), weight (p = 0.012–0.047), and number of previous concussions (p < 0.001 for all). There were also significant differences in age at baseline (p = 0.0012) and 24–48 h (p = 0.021) and the proportion of males with unrestricted RTP (p = 0.013). Among post-injury assessments, there were significant differences between SAC raw scores at baseline (p = 0.00085), <6 h (p = 0.038), and at unrestricted RTP (p = 0.0038), SCAT total symptoms raw score at unrestricted RTP (p = 0.027), and BESS raw score at <6 h (p = 0.048) and 24–48 h (p = 0.00098).

Table 1.

Data Characteristics of Training and Validation Set with Respect to Each Time Point

	Baseline		<6 h		24–48 h		Unrestricted RTP
	Training	Validation	Training	Validation	Training	Validation	Training	Validation
n	751	884	546	539	719	694	716	629
Height in meters (SD)	1.79 (0.11)^**	1.78 (0.12)	1.80 (0.12)^**	1.79 (0.11)	1.79 (0.12)^*	1.78 (0.12)	1.79 (0.12)^*	1.78 (0.12)
Weight in kg (SD)	83.75 (21.65)^**	81.95 (21.25)	86.01 (22.45)^**	83.32 (21.39)	83.94 (22.08)^**	81.33 (20.85)	83.33 (21.4)^**	80.74 (21.01)
Age in years (SD)	19.43 (1.30)^*	19.24 (1.32)	19.36 (1.32)	19.23 (1.36)	19.34 (1.27)^**	19.2 (1.33)	19.37 (1.30)	19.27 (1.32)
Male sex (% yes)	60.80%	57.58%	64.47%	61.6%	60.08%	56.34%	60.06%^**	54.05%
Number of previous concussions (SD)	0.73 (1.0)^*	0.59 (0.83)	0.78 (1.03)^*	0.57 (0.78)	0.74 (1.03)^*	0.58 (0.82)	0.70 (0.93)^*	0.52 (0.77)
Report injury immediately? (% yes)¹	NA	NA	55.31%^**	60.67%	40.47%^**	46.54%	41.2%^**	47.22%
Removed from play immediately? (% yes)¹	NA	NA	57.51%	56.77%	46.18%	47.26%	48.6%	45.31%
LOC? (% yes)¹	NA	NA	5.86%	5.01%	4.45%	4.76%	5.45%	4.77%
PTA? (% yes)¹	NA	NA	12.27%^**	9.09%	11.4%^**	8.65%	11.45%^*	7.63%
RGA? (% yes)¹	NA	NA	5.86%	5.57%	5.84%	5.33%	5.59%	4.13%
SAC change score (SD)¹	NA	NA	−0.82 (3.19)^*	−1.76 (3.58)	−0.4 (2.61)^*	−1.11 (2.72)	0.97 (2.13)^*	0.51 (1.99)
SAC raw score (SD)	27.05 (2.01)^*	27.38 (1.96)	26.18 (2.94)^**	25.83 (3.33)	26.63 (2.41)	26.48 (2.77)	27.93 (1.75)^*	28.19 (1.76)
SCAT symptom severity change score (SD)¹	NA	NA	23.22 (20.75)	23.45 (22.76)	19.48 (21.68)	20.62 (23.27)	−4.87 (8.8)	−5.12 (9.71)
SCAT symptom severity raw score (SD)	5.16 (8.54)	5.25 (9.71)	28.64 (20.83)	28.58 (21.52)	25.21 (21.54)	26.04 (21.93)	0.63 (1.97)	0.46 (1.67)
SCAT total symptoms change score (SD)¹	NA	NA	8.01 (5.93)	8.03 (6.43)	7.49 (6.58)	8.05 (6.88)	−2.48 (4)	−2.55 (3.83)
SCAT total symptoms raw score (SD)	2.81 (3.85)	2.77 (3.89)	10.86 (5.42)	10.77 (5.57)	10.53 (6.03)	10.91 (6.18)	0.47 (1.39)^**	0.34 (1.09)
BESS change score (SD)¹			3.58 (8.64)	4.02 (8.52)	1.5 (7.48)^**	2.58 (8.19)	−2.35 (6.32)	−2.39 (5.9)
BESS raw score (SD)	12.7 (6.32)	12.78 (6.21)	16.35 (8.78)^**	17.36 (8.38)	14.42 (7.87)^*	15.82 (8.1)	10.46 (5.81)	10.84 (5.36)

Change score at a time point is computed as: raw score at time point - raw score at baseline.

Significantly different from validation data at same time point based on Student's t test (p < 0.01).

Significantly different from validation data at same time point based on Student's t test (p < 0.05).

The variable was not available for baseline data.

RTP, return to play; LOC, loss of consciousness; PTA, post-traumatic amnesia; RGA, retrograde amnesia; SAC, Standard Assessment of Concussion; SCAT, Sport Concussion Assessment Tool; BESS, Balance Error Scoring System; NA, variable not available for baseline data.

Multivariate logistical regression

The model variables and corresponding coefficient values for the multivariate logistical regression models at <6 h and 24–48 h are shown in Table 2. At <6 h, all variables were significant except for whether the injury was reported immediately (p = 0.16), SAC raw score (p = 0.13), and BESS raw score (p = 0.080). At 24–48 h, all variables were significant except for SAC raw score (p = 0.23) and BESS raw score (p = 0.94).

Table 2.

Multivariate Logistical Regression Coefficients at <6 h and 24–48 h

	<6 h			24–48 h
Study variable	Coefficient	SE	p value	Coefficient	SE	p value
Intercept	−0.57	1.71	0.74	−0.37	1.46	0.80
Male sex	0.71	0.27	0.01	0.45	0.22	0.04
Report injury immediately?	−0.46	0.33	0.16	−1.37	0.22	<0.01
Removed from play immediately?	−0.91	0.35	0.01	NA	NA	NA
SAC raw score	−0.09	0.06	0.13	−0.06	0.05	0.23
SCAT symptom severity raw score	0.06	0.03	0.02	−0.05	0.02	0.02
SCAT total symptoms raw score	0.28	0.07	<0.01	0.47	0.05	<0.01
BESS raw score	0.03	0.02	0.08	0.00	0.02	0.94

SE, standard error; SAC, Standard Assessment of Concussion; SCAT, Sport Concussion Assessment Tool;

BESS, Balance Error Scoring System; NA, variable not included in this model.

Classifying unlikely, possible, probable, and definite concussion

We obtained risk score thresholds after applying the training data to our optimization algorithm. At <6 h, the lower threshold was 0.047 and the upper threshold was 0.33. At 24–48 h, the lower threshold was 0.07 and the upper threshold was 0.46. The CARTs we developed for <6 h and 24–48 h are shown in Figure 2.

FIG. 2.

Classification tree for determining possible and probable concussions at <6 h (A) and 24–48 h (B).

We now provide an example to illustrate how these risk thresholds and CART can be used to determine whether an athlete should be classified as having an unlikely, possible, probable, or definite concussion.

Consider the case of a 19-year-old female athlete who is being assessed for acute concussion 24–48 h after injury. She did not report the injury immediately and in her post-injury assessments, obtained total scores of 30 and 12 on the SAC and BESS, respectively. On the SCAT symptom assessment, she reported four total symptoms with a total severity of 6. Using the logistical regression model for 24–48 h, her risk score is equal to 0.36. Because her risk score is less than the upper threshold of 0.46 and greater than the lower threshold of 0.07 at 24–48 h, she is not classified as having a definite or unlikely concussion. To determine if she has a possible or probable concussion, one would refer to the CART for 24–48 h. Because her SCAT symptom severity raw score is not 0, her SAC raw score is >25, and her SCAT total symptoms raw score is >1, she would be classified as having a probable concussion.

To provide an additional example, consider the case of a 21-year-old male athlete who was assessed for concussion within 6 h of a suspected injury. His injury was not reported immediately and he was not removed from play immediately. His SAC raw score and BESS raw score were 24 and 12, respectively. He also reported one symptom with a severity of 1. Based on these values, his risk score is equal to 0.22. Because his risk estimate is between the lower and upper thresholds of 0.047 and 0.33 at <6 h, respectively, then he must either have a possible or a probable concussion. Because his SCAT symptom severity raw score is ≤4, the CART analysis at <6 h would classify him as having a possible concussion.

Distribution of acute concussions and normal performances

The distribution of acute concussions and normal performances within each risk classification is shown in Table 3. At <6 h, 434 (80.52%) of acute concussions were classified as definite concussion whereas only 14 (2.60%) were classified as unlikely concussion. Among the remaining acute concussions, 31 (5.75%) were classified as possible concussion and 60 (11.13%) were classified as probable concussion. When the <6 h algorithm was applied to normal performance data (i.e., baseline and unrestricted RTP), 696 (46.00%), 526 (34.77%), 189 (12.49%), and 102 (6.74%) were classified as unlikely, possible, probable, and definite concussion respectively. With the 24–48 h algorithm, 522 (75.22%) acute concussions were classified as definite concussion whereas 21 (3.03%) were classified as unlikely concussion. There were 41 (5.91%) and 110 (15.85%) acute concussions classified as possible and probable concussion, respectively. Among the normal performances, 714 (47.19%), 397 (26.24%), 309 (20.42%), and 95 (6.15%) were classified as unlikely, possible, probable, and definite concussion, respectively. With both <6 h and 24–48 h algorithms, the distributions among risk classifications were different between acute concussions and normal performances based on the Kolmogorov–Smirnov test (p < 0.001). Additionally, the distribution of baselines and unrestricted RTP across the risk classifications was also significantly different at both <6 h and 24–48 h (p < 0.001).

Table 3.

Distribution of Acute Concussions and Normal Performances among Risk Classifications at <6 h and 24–48 h

	Unlikely		Possible		Probable		Definite
	n	%	n	%	n	%	n	%
<6 h
Acute concussion^*	14	2.60%	31	5.75%	60	11.13%	434	80.52%
Normal performance	696	46.00%	526	34.77%	189	12.49%	102	6.74%
(Unrestricted RTP)^**	282	44.83%	329	52.31%	15	2.38%	3	0.48%
(Baseline)	414	46.83%	197	22.29%	174	19.68%	99	11.20%
Total	710	34.60%	557	27.14%	249	12.13%	536	26.12%
24–48 h
Acute concussion^*	21	3.03%	41	5.91%	110	15.85%	522	75.22%
Normal performance	714	47.19%	397	26.24%	309	20.42%	93	6.15%
(Unrestricted RTP)^**	275	43.72%	312	49.60%	36	5.72%	6	0.95%
(Baseline)	439	49.66%	85	9.62%	273	30.88%	87	9.84%
Total	735	33.30%	438	19.85%	419	18.99%	615	27.87%

Distributions of acute concussions and normal performances within risk classifications are significantly different at p < 0.001 using Kolmogorov–Smirnov test.

Distributions of unrestricted return to play (RTP) and baseline time points within risk classifications are significantly different at p < 0.01 using Kolmogorov–Smirnov test.

Using our modified calculation for sensitivity and specificity, we obtained a sensitivity range of 91.65–97.40% with the <6 h algorithm and 91.06–97.00% with the 24–48 h algorithm. We also obtained a specificity range of 46.00–80.77% with the <6 h algorithm and 47.19–73.43% with the 24–48 h algorithm, respectively.

As an ancillary analysis, we performed our analysis without the unrestricted RTP data (data not shown). The resulting logistical regression model, risk score thresholds, and CART models led to a distribution with a sensitivity of 89.42–98.52% and a specificity of 23.21–71.60% at <6 h. At 24–48 h, the sensitivity and specificity ranged from 85.73% to 95.67% and from 41.52% to 71.60%, respectively.

Inter-class differences

The inter-class differences for acute concussions and normal performances are shown in Tables 4 and 5, respectively. Among acute concussions, all mean raw and change scores for SCAT symptom assessments of unlikely and possible concussions are significantly different from definite concussions at both <6 h and 24–48 h (p < 0.001 for all). Among the SAC and BESS at <6 h, only the SAC change score is not significantly different between definite and unlikely concussions. In contrast, at 24–48 h, only the BESS raw score is significantly different between definite and unlikely concussions (p = 0.021). Possible and probable concussions are significantly different in SCAT total symptoms raw score at <6 h and 24–48 h (p = 0.0027–0.0082). They are also significantly different in SAC change score (p = 0.013), SAC raw score (p < 0.001), and SCAT total symptoms change score (p = 0.012) at 24–48 h.

Table 4.

Comparison of Study Variables for Acute Concussions Classified as Unlikely, Possible, Probable, and Definite Concussion at <6 h and 24–48 h

	<6 h				24–48 h
	Unlikely	Possible	Probable	Definite	Unlikely	Possible	Probable	Definite
n	14	31	60	434	21	41	110	522
Age in years (SD)	19.93 (0.80)^*	19.42 (1.48)	19.23 (1.28)	19.19 (1.37)^*	19.38 (1.29)	19.20 (1.40)	19.15 (1.29)	19.20 (1.33)^*
Male sex (% yes)	78.57%^*	74.19%	43.33%	62.67%	61.90%	73.17%^{* b}	49.09%	56.32%
Number of previous concussions (SD)	0.43 (0.62)	0.55 (0.87)	0.60 (0.82)	0.56 (0.77)	0.52 (0.73)	0.66 (0.95)	0.53 (0.72)	0.58 (0.82)^*
Report injury immediately? (% yes)	100.00%^{* a}	67.74%^{* a}	83.33%^*	55.76%	100.00%^{* a}	51.22%^* a,b	76.36%^*	37.74%
Removed from play immediately? (% yes)	100.00%^{* a}	64.52%^{* a}	76.67%^*	52.07%	90.48%^{* a}	51.22%^{* a}	62.73%^*	41.95%
LOC? (% yes)	14.29%	6.45%^*	5.00%	4.61%^*	9.52%	4.88%^*	3.64%	4.79%^*
PTA? (% yes)	7.14%	3.23%^*	3.33%	10.37%^*	9.52%	7.32%	5.45%	9.39%
RGA? (% yes)	21.43%	6.45%^*	0.00%	5.76%^*	9.52%^*	2.44%	7.27%	4.98%^*
SAC change score (SD)	0.14 (1.46)	−0.97 (2.86)^a	−0.55 (2.10)	−1.76 (3.51)	−0.52 (2.32)	−1.59 (2.56)^* a,b	−0.15 (1.63)	−1.28 (2.76)^*
SAC raw score (SD)	28.57 (1.18)^a	26.61 (2.88)^a	27.48 (1.80)	25.49 (3.42)^*	27.29 (1.72)	25.66 (2.69)^* a,b	27.76 (1.33)	26.24 (2.90)^*
SCAT symptom severity change score (SD)	−2.79 (5.83)^a	−0.48 (9.08)^a	5.18 (7.17)^{* a}	28.49 (21.94)^*	−3.81 (7.40)^a	−1.00 (10.92)^a	4.38 (8.4)^{* a}	26.42 (23.33)^*
SCAT symptom severity raw score (SD)	0.71 (1.33)^a	2.74 (1.24)^{* a}	8.70 (3.00)^{* a}	34.01 (20.29)^*	0.90 (1.72)^{* a}	3.24 (4.11)^{* a}	7.15 (5.47)^a	32.88 (20.95)^*
SCAT total symptoms change score (SD)	−1.21 (3.41)^a	0.42 (3.98)^a	2.60 (3.51)^{* a}	9.63 (5.85)^*	−2.05 (2.87)^a,b	−0.54 (3.79)^a,b	2.63 (3.47)^{* a}	10.21 (6.14)^*
SCAT total symptoms raw score (SD)	0.57 (0.90)^a,b	2.06 (1.08)^* a,b	4.73 (1.42)^a	12.57 (4.53)^*	0.33 (0.56)^* a,b	1.59 (1.56)^* a,b	4.28 (1.79)^{* a}	13.47 (4.74)^*
BESS change score (SD)	−1.93 (8.92)^a	3.97 (7.45)	2.87 (6.39)^*	4.94 (9.41)	0.62 (5.60)	0.83 (5.92)	1.46 (6.73)^*	3.52 (8.40)^*
BESS raw score (SD)	11.64 (6.00)^a	15.00 (6.51)^a	13.83 (7.34)	18.40 (9.02)^*	11.62 (5.62)^a	13.17 (6.29)^a	14.07 (7.18)^a	16.89 (8.61)

Change score at a time point is computed as: raw score at time point - raw score at baseline.

Significantly different (p < 0.05) from normal performances in the same risk classification and timepoint based on Student's t test

Significantly different (p < 0.05) from definite concussion at the same timepoint based on Tukey's post-hoc pairwise comparisons.

Significantly different (p < 0.05) from probable concussion at the same timepoint based on Tukey's post-hoc pairwise comparisons.

LOC, loss of consciousness; PTA, post-traumatic amnesia; RGA, retrograde amnesia; SAC, Standard Assessment of Concussion; SCAT, Sport Concussion Assessment Tool; BESS, Balance Error Scoring System

Table 5.

Comparison of Study Variables for Normal Performances Classified as Unlikely, Possible, Probable, and Definite Concussion At <6 h and 24–48 h

	<6 h				24–48 h
	Unlikely	Possible	Probable	Definite	Unlikely	Possible	Probable	Definite
n (% baseline)	696 (59.48%)	526 (37.45%)	189 (92.06%)	102 (97.06%)	714 (61.48%)	397 (21.41%)	309 (88.35%)	93 (93.55%)
Age in years (SD)	19.28 (1.27)^*	19.33 (1.37)	19.13 (1.32)	18.86 (1.30)^{* a}	19.36 (1.31)	19.16 (1.30)	19.24 (1.35)	18.86 (1.29)^{* a}
Male sex (% yes)	48.13%^*	72.24%^a	41.80%^b	53.92%	56.16%	59.19%^*	54.05%	49.46%
Number of previous concussions (SD)	0.55 (0.81)	0.53 (0.76)	0.58 (0.86)	0.70 (0.84)	0.54 (0.78)	0.51 (0.80)	0.61 (0.82)	0.76 (0.88)^*
Report injury immediately? (% yes)^**	84.75%^*	14.89%^*	53.33%^*	33.33%	100.00%^*	2.88%^*	30.56%^*	33.33%
Removed from play immediately? (% yes)^**	84.40%^*	11.85%^*	46.67%^*	33.33%	77.09%^*	18.59%^*	33.33%^*	50.00%
LOC? (% yes)^**	7.45%	2.43%^*	6.67%	0.00%^*	8.73%	0.96%^*	8.33%	0.00%^*
PTA? (% yes)^**	7.09%	7.90%^*	13.33%	0.00%^*	8.73%	6.41%	8.33%	16.67%
RGA? (% yes)^**	6.03%	2.43%^*	6.67%	0.00%^*	5.09%^*	3.21%	5.56%	0.00%^*
SAC change score (SD)	0.85 (1.93)	0.54 (2.13)	0.13 (1.82)	−0.33 (1.89)	0.85 (2.03)	0.54 (2.07)^*	0.28 (1.74)	1.17 (2.19)^*
SAC raw score (SD)	28.11 (1.60)	27.35 (2.25)^a	27.40 (1.90)^a	27.27 (1.93)^a	27.88 (1.82)	27.30 (2.39)^{* a}	27.92 (1.40)^b	27.34 (2.02)^*
SCAT symptom severity change score (SD)	−4.98 (8.37)	−5.13 (10.12)	−8.33 (18.17)^*	−8.67 (14.27)^*	−5.09 (8.96)	−4.99 (8.71)	−5.75 (17.90)^*	−12.83 (16.08)^*
SCAT symptom severity raw score (SD)	0.41 (1.02)	1.02 (1.41)^*	8.22 (3.23)^a,b	25.65 (15.51)^* a,b	0.34 (0.96)^*	0.99 (2.29)^*	6.64 (5.50)^a,b	24.75 (16.6)^* a,b
SCAT total symptoms change score (SD)	−2.63 (3.51)	−2.54 (3.90)	−1.73 (5.96)^*	−1.67 (4.64)^*	−2.61 (3.64)	−2.68 (3.73)	−1.06 (4.52)^*	−2.50 (6.85)^*
SCAT total symptoms raw score (SD)	0.28 (0.64)	0.81 (1.16)^{* a}	4.84 (1.59)^a,b	11.41 (3.99)^* a,b	0.20 (0.49)^*	0.66 (1.37)^{* a}	3.85 (1.82)^* a,b	11.87 (3.85)^* a,b
BESS change score (SD)	−2.72 (6.03)	−1.64 (5.68)	−1.60 (5.55)^*	−0.33 (3.68)^*	−2.06 (6.19)	−2.08 (5.50)	−3.06 (6.42)^*	−0.83 (3.02)^*
BESS raw score (SD)	10.74 (5.14)	12.96 (5.97)^a	13.20 (7.05)^a	16.41 (7.77)^a,b	11.76 (5.68)	11.28 (5.46)	13.38 (6.76)^a,b	15.56 (7.67)^* a,b

Change score at a time point is computed as: raw score at time point - raw score at baseline.

Significantly different (p < 0.05) from acute concussions in the same risk classification and time point based on Student's t-test.

Variable not available for baseline data.

Significantly different (p < 0.05) from no concussion at the same time point based on Tukey's post-hoc pairwise comparisons.

Significantly different (p < 0.05) from possible concussion at the same time point based on Tukey's post-hoc pairwise comparisons.

LOC, loss of consciousness; PTA, post-traumatic amnesia; RGA, retrograde amnesia; SAC, Standard Assessment of Concussion; SCAT, Sport Concussion Assessment Tool; BESS, Balance Error Scoring System.

For normal performances, the mean raw scores for the SAC, SCAT symptom severity, SCAT total symptoms, and the BESS among definite and probable concussions were significantly different from those for unlikely concussion (p < 0.001 for all), except for SAC raw score at 24–48 h. At <6 h and 24–48 h, possible and probable concussions were significantly different in SCAT symptom severity raw score and SCAT total symptoms raw score (p < 0.001 for all). At 24–48 h, possible and probable concussions were also significantly different in SAC raw score and BESS raw score (p < 0.001 for all).

Intra-class differences

The intra-class differences are highlighted in Tables 3 and 4. Among those classified as having possible concussions, acute concussions and normal performances are significantly different in SCAT symptom severity (p < 0.001 at <6 h, p = 0.016 at 24–48 h) and SCAT total symptoms raw score (p < 0.001 at <6 h, p = 0.0019 at 24–48 h). There are also significant differences in SAC raw change scores (p = 0.0026) and raw scores (p = 0.046) for acute concussions and normal performances classified as Possible concussion at 24–48 hours. Among probable concussions at <6 h and 24–48 h, acute concussions and normal performances are significantly different in change scores for SCAT symptom severity (p = 0.0012–0.0077), SCAT total symptoms (p = 0.0093 at <6 h, p < 0.001 at 24–48 h), and BESS (p = 0.0074 at <6 h, p < 0.001 at 24–48 h). They are also significantly different in SCAT symptom severity raw score at <6 h (p < 0.001) and SCAT total symptoms raw score at 24–48 h (p < 0.001).

To illustrate how these intra-class differences can be used to inform clinical decision making, we revisit the examples from the Classifying unlikely, possible, probable, and definite concussion subsection. If one considers the first athlete (19-year-old female) and supposes that her change scores for the SAC, SCAT symptom severity, SCAT total symptoms, and BESS are 0, 6, 4, and −5, respectively, based on intra-class differences identified in this study for 24–48 h, there were significant differences between acute concussions and normal performances for the SCAT symptom severity change score, SCAT total symptoms change score, SCAT total symptoms raw score, and the BESS change score. Comparing this athlete's assessments with the mean values for probable concussions presented in Tables 4 and 5, we find that the athlete's situation is more comparable to acute concussion in terms of change scores for the SCAT symptom severity and total number of symptoms. Conversely, her situation is more comparable to the normal performances in terms of the BESS change score. Following the conservative decision-making approaches that are recommended for concussion management, one could treat this athlete as if she had an acute concussion.

If one considers the 21-year-old male athlete and supposes that his change scores for the SAC and BESS were 0 and 5 respectively, and that additionally, his SCAT symptom severity and total symptoms both decreased by 4 compared with baseline, based on intra-class differences identified in this study for the possible concussion group at <6 h, there were significant differences between acute concussions and normal performances in the SCAT symptom severity and SCAT total symptom raw scores. Comparing this athlete's values in these measures (of one symptom reported with a severity of 1) to the mean values obtained in our analysis, we find that this athlete's situation more closely resembles the normal performances within the possible concussion group despite the low SAC raw score and high BESS change score. These results could potentially indicate that additional assessments should be performed on this athlete to confirm the possibility that this athlete is not concussed.

Discussion

Kutcher and Giza proposed a risk-based classification framework for diagnosing acute concussion developed from clinical experience.⁴ Compared to traditional binary diagnosis, this framework allows the assessment of acute concussion to reflect the physician's diagnostic certainty. Further, taking this approach allows the injury diagnosis to evolve as the injury evolves and more information becomes available. However, although these authors provided clinical guidelines for each risk classification, they did not provide specific criteria with respect to commonly recommended and implemented concussion assessment tools. In this research, we designed and evaluated a novel data-driven method for classifying athletes evaluated for acute concussion as having either unlikely, possible, probable, or definite concussion. The major contributions of our research are as follows:

We develop an objective and data-driven framework that stratifies acute concussion assessment by diagnostic certainty. These risk categories lay the foundation for guiding post-injury management decisions.

We identify key characteristics that can be used to differentiate between acute concussions and normal performances in each risk category.

We provide additional, quantitative support for the value of a multi-dimensional battery, the use of change scores in acute concussion assessment, and the potential implications for several demographic factors and time-of-injury characteristics in acute concussion assessment.

The variables used in our logistical regression and CART models are parts of standard concussion assessment batteries, giving foundation for our framework to be used in sporting environments. To our knowledge, we are the first to combine predictive modeling techniques (i.e., logistical regression and CART) and optimization algorithms to classify athletes into concussion risk categories. Erring in the direction of minimizing false negatives, our framework classified most acute concussions (91.07–91.65%) into the higher risk categories (i.e., probable and definite concussion) and most normal performances (73.43–80.77%) into the lower risk categories (i.e., unlikely and possible concussion). Additionally, few acute concussions were classified as unlikely concussion (2.60–3.03%) and few normal performances were classified as definite concussion (6.15–6.74%).

Our most important finding was that athletes classified as having definite concussion had lower SAC, higher SCAT symptom, and higher BESS scores than the other risk categories. In comparing these risk groups, definite concussions exhibited noticeably more symptoms and greater symptom severity than the other risk categories, whereas the Unlikely concussions exhibited mean symptom severities and mean total symptoms close to 0. Definite concussions also had much higher BESS raw scores and lower SAC scores than unlikely concussions. These findings demonstrate the ability of our framework to separate the “easy” cases from the “hard” cases, and are consistent with previous research demonstrating that symptoms are typically the most sensitive to acute concussion.^8

–12 Our findings also provide support for the utility of using neurocognitive assessments and postural control measures for acute concussion assessment, as demonstrated by previous research.^{8,26,39

–46}

However, among those classified as possible or probable concussions, raw scores for SCAT symptom severity and total symptoms are significantly less for acute concussions in the possible concussion group compared with all baselines (p < 0.01 for both measures at <6 h and 24–48 h using Student's t test). Additionally, there are no significant differences in the SCAT symptom severity or total symptoms between acute concussions and normal performances in the probable risk category. These findings demonstrate the difficulty in identifying all acute concussions using symptom raw scores alone. Fortunately, there were some significant differences between acute concussions and normal performances in the possible and probable risk categories for change scores in the SAC, SCAT symptom severity, SCAT total symptoms, and BESS. This result suggests that change scores, which require baseline assessments, have added value when evaluating possible and probable concussions, and are an important finding regarding the utility of the baseline assessment.

In our analysis, we also sought out to identify differences in athlete demographics and time-of-injury characteristics across and within risk classifications. There were statistically significant differences in age, sex, and number of previous concussions between acute concussions and normal performances within some risk categories. For example, among those classified as having definite concussions, the athletes were, on average, older than those providing normal performances at both <6 h and 24–48 h. Outside of age, there were no other consistent demographic differences and risk categories. For time-of-injury variables among acute concussions, a larger proportion of those classified as having unlikely concussion reported the injury immediately and were removed from play immediately compared with those who were classified as having definite concussions. This result suggests that those athletes who were removed from play immediately or assessed immediately after injury may have been in the earliest stages of an evolving injury wherein neurocognitive declines, increased symptoms, and worsening postural control emerge over time.^16,17,47 However, we note that very few acute concussions were classified as unlikely concussion, and because of this small sample size, this point may require further investigation.

We also found that baselines comprised most normal performances within the probable and definite risk categories. Despite varying parameter settings to balance the sensitivity and specificity of our results, we were unable to drastically improve on the proportion of baselines in these upper risk categories. This finding may be the result of performance differences between baseline and unrestricted RTP time points. Specifically, the baseline data showed lower SAC scores, higher SCAT symptom assessment scores, and higher BESS scores than unrestricted RTP data (p < 0.001 for all measures and in both training and validation sets). As a result, our logistical regression model categorized the baseline performance of some athletes into the higher risk categories. The performance discrepancy between baseline and unrestricted RTP time points is consistent with previous studies^{12,16,48
–50} and may be attributed to comorbidities⁵¹ or learning effects from multiple assessments prior to return to play.^41,52 Future works may be able to address this shortcoming by incorporating individual items from the SAC, SCAT symptom assessments, and the BESS instead of using total scores. Regardless, this finding highlights the need for clinicians to interpret the administered assessments in the context of the injury, such as an observed mechanism, and differentiate from other injuries and conditions with similar signs and symptoms.^1,4,33,53

To account for clinical judgment in our methodology, we used a modified range-based computation for sensitivity and specificity, which provided a 6% sensitivity increase and >34% specificity increase in possible concussion management. Furthermore, the sensitivity of our algorithm mirrors those reported in previous studies (80.0–100.0%) evaluating concussion testing batteries for acute concussion assessment.^8,9,39,54 Methodological differences between our study and these aforementioned studies account for some differences, including the test battery assessments. For example, both Broglio and colleagues³⁹ and Resch and colleagues⁹ used the Sensory Organization Test (SOT) for balance assessment instead of the BESS. Whereas both the SOT and BESS reveal similar post-concussion trends in postural control deficits,⁴⁰ the SOT has less clinical applicability given its size and cost. Additionally, the diagnosis criteria differed greatly across each study. McCrea and colleagues,⁸ Broglio and colleagues,³⁹ and Putukian and colleagues⁵⁴ used different measures of significant change to indicate concussion whereas Resch and colleagues⁹ used both predictive discriminant analyses and clinical interpretation guidelines. In comparison, we paired a data-driven optimization framework with predictive modeling methods (i.e., logistical regression and CART) to classify athletes into risk categories. By using predictive modeling methods, we were able to simultaneously incorporate demographic information and time-of-injury characteristics, along with SAC, SCAT symptoms, and BESS results. Finally, the concussed sample used in the present study (n = 1085 for <6 h and n = 1413 for 24–48 h) is much larger than those in the aforementioned studies (n = 32–166).

From a clinical perspective, previous studies have discussed the importance and value of taking a heterogeneous and targeted approach to concussion management.^55
–57 However, because the focus of this study was on identifying acute concussion, it does not address injury heterogeneity by accounting for potential concussion subtypes or clinical profiles. However, our work lays the foundation for doing so using clustering or clinically determined approaches.

Our study is not without limitations. First, we acknowledge that our framework does not provide a recommendation for post-injury management for athletes classified in each risk category. These post-injury decisions are beyond the scope of our study and are an important topic for future research. To this end, clinicians can still benefit from knowing the degree of certainty in a diagnosis decision before determining the next course of action. Second, our study treats all concussions in the CARE data as true concussions, regardless of the medical staff certainty. Therefore, there is the possibility that our models were trained and validated on athletes who were not actually concussed but were labeled so. Third, the differences between our training and validation data in important clinical measures such as PTA or LOC may have caused differences in the presentation of concussion between those two groups, potentially explaining some of the prediction errors in our models. Determining training and validation data using random subset selection instead of by a time-based cutoff could lead to a more homogeneous division in data and ultimately, improved modeling results. Fourth, because our study data only included athletes 18–22 years of age, we cannot directly apply our results to populations beyond this group. Therefore, future studies should focus on other population groups, such as those engaged in youth sports and professional athletes, to determine the generalizability of our results beyond our study population. Fifth, we were limited in our ability to include change scores and time-of-injury characteristics in our models, as these measures were not available for baseline data. Finding ways to incorporate such variables in future analysis may improve our results. Further, our analysis focused on the SAC, SCAT symptom assessments, and BESS. Data limitations precluded our ability to include assessments such as the SOT, computer-based neurocognitive testing, the King–Devick test, and/or the Vestibular/Ocular Motor Screening Assessment, which have shown promise in other investigations.^58

–61 Finally, as there is no gold standard for concussion diagnosis, we did not have a comparative mechanism for our results.

The objective, algorithmic approach we proposed and developed for risk-based classification of athletes undergoing acute concussion assessment extends the original framework proposed by Kutcher and Giza.⁴ By applying predictive modeling and optimization methods, our work provides a promising first step in taking an evidence-based approach to acute concussion assessment stratification. Although the clinical examination remains the gold standard for concussion diagnosis, the models we have designed and analyzed have the potential to provide valuable decision support for clinicians.

Footnotes

Acknowledgments

The material is based on work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE 1256260. This publication was made possible, in part, with support from the Grand Alliance CARE Consortium, funded, in part, by the NCAA and the Department of Defense (DoD). The United States Army Medical Research Acquisition Activity, 820 Chandler Street, Fort Detrick MD 21702-5014 is the awarding and administering acquisition office. This work was supported by the Office of the Assistant Secretary of Defense for Health Affairs through the Psychological Health and Traumatic Brain Injury Program under Award No. W81XWH-14-2-0151. Opinions, interpretations, conclusions, and recommendations are those of the author(s) and are not necessarily endorsed by the Department of Defense (DoD) (Defense Health Program [DHP] funds). We thank April Marie (Reed) Hoy, MS, ATC (Azusa Pacific University), Joseph B. Hazzard Jr., EdD, ATC (Bloomsburg University), Louise A. Kelly, PhD (California Lutheran University), Justus D. Ortega, PhD (Humboldt State University), Nicholas Port, PhD (Indiana University), Margot Putukian, MD (Princeton University), Gerald McGinty, DPT and Jonathan C. Jackson, PhD (United States Air Force Academy), Kenneth L. Cameron, PhD, MPC, ATC (United States Military Academy), Christopher Giza, MD (University of California Los Angeles), Holly J. Benjamin, MD (University of Chicago), Thomas Buckley, EdD, ATC and Thomas W. Kaminski, PhD, ATC (University of Delaware), James R. Clugston, MD, MS (University of Florida), Julianne D. Schmidt, PhD, ATC (University of Georgia), Louis A. Feigenbaum, DPT, ATC (University of Miami), James T. Eckner, MD, MS (University of Michigan), Kevin M. Guskiewicz, PhD, ATC and Jason P. Mihalik, PhD, ATC (University of North Carolina), Jessica Dysart Miles, PhD, ATC (University of North Georgia), Scott Anderson, ATC (University of Oklahoma), Christina L. Master, MD (University of Pennsylvania), Anthony P. Kontos, PhD and Micky Collins, PhD (University of Pittsburgh), Sara P.D. Chrisman, MD, MPH (University of Washington), Alison Brooks, MD, MPH (University of Wisconsin), Steven Rowson, PhD (Virginia Tech), Christopher M. Miles, MD and Laura J. Lintner, DO (Wake Forest University), and Brian H. Dykhuizen, MS, ATC, LAT (Wilmington College).

Author Disclosure Statement

No competing financial interests exist.

References

McCrory

, Meeuwisse

, Dvorak

, Aubry

, Bailes

, Broglio

, Cantu

R.C.

, Cassidy

, Echemendia

R.J.

, Castellani

R.J.

, et al. (2017). Consensus statement on concussion in sport—the 5th International Conference on Concussion in sport held in Berlin, October 2016. Br. J. Sports Med. 51, 838–847.

Meehan

S.K.

, Mirdamadi

J.L.

, Martini

D.N.

, and Broglio

S.P.

(2017). Changes in cortical plasticity in relation to a history of concussion during adolescence. Front. Hum. Neurosci. 11, 1–13.

Makdissi

, Davis

, and McCrory

(2015). Clinical challenges in the diagnosis and assessment of sports-related concussion. Neurol. Clin. Pract. 5, 2–5.

Kutcher

J.S.

, and Giza

C.C.

(2014). Sports concussion diagnosis and management. Continuum (N.Y). 20, 1552–1569.

McCrory

, Meeuwisse

W.H.

, Aubry

, Cantu

R.C.

, Dvorák

, Echemendia

R.J.

, Engebretsen

, Johnston

K.M.

, Kutcher

J.S.

, Raftery

, Sills

, Benson

B.W.

, Davis

. a., Ellenbogen

, Guskiewicz

K.M.

, Herring

. a., Iverson

G.L.

, Jordan

B.D.

, Kissick

, McCrea

, McIntosh

A.S.

, Maddocks

D.L.

, Makdissi

, Purcell

, Putukian

, Schneider

, Tator

C.H.

, and Turner

(2013). Consensus Statement on Concussion in Sport-The 4th International Conference on Concussion in Sport Held in Zurich, November 2012. Br. J. Sports Med. 47, 255–279.

Broglio

S.P.

, Cantu

R.C.

, Gioia

G.A.

, Guskiewicz

K.M.

, Kutcher

, Palm

, and McLeod

T.C.V.

(2014). National Athletic Trainers' Association Position Statement: Management of Sport Concussion. J. Athl. Train. 49, 245–265.

Harmon

K.G.

, Drezner

J.A.

, Gammons

, Guskiewicz

K.M.

, Halstead

, Herring

S.A.

, Kutcher

J.S.

, Pana

, Putukian

, and Roberts

W.O.

(2013). American Medical Society for Sports Medicine Position Statement: Concussion in Sport. Br. J. Sports Med. 47, 15–26.

McCrea

M.A.

, Barr

W.B.

, Guskiewicz

K.M.

, Randolph

, Marshall

S.W.

, Cantu

R.C.

, Onate

J.A.

, and Kelly

J.P.

(2005). Standard regression-based methods for measuring recovery after sport-related concussion. J. Int. Neuropsychol. Soc. 11, 58–69.

Resch

J.E.

, Brown

C.N.

, Schmidt

, Macciocchi

S.N.

, Blueitt

, Cullum

C.M.

, and Ferrara

M.S.

(2016). The sensitivity and specificity of clinical measures of sport concussion: three tests are better than one. BMJ Open Sport Exerc. Med. 2, e000012.

10.

J.K.

, Guskiewicz

K.M.

, Mihalik

J.P.

, Schmidt

J.D.

, Kerr

Z.Y.

, and McCrea

. a. (2013). Reliable change, sensitivity, and specificity of a multidimensional concussion assessment battery. J. Head Trauma Rehabil. 28, 274–283.

11.

Chin

E.Y.

, Nelson

L.D.

, Barr

W.B.

, McCrory

, and McCrea

M.A.

(2016). Reliability and validity of the sport concussion assessment tool-3 (SCAT3) in high school and collegiate athletes. Am. J. Sports Med. 44, 2276–2285.

12.

Garcia

G.-G.P.

, Broglio

S.P.

, Lavieri

M.S.

, McCrea

, and McAllister

(2018). Quantifying the value of multidimensional assessment models for acute concussion: an analysis of data from the NCAA-DoD Care Consortium. Sport. Med. 48, 1739–1749.

13.

Williamson

I.J.S.

, and Goodman

(2006). Converging evidence for the under-reporting of concussions in youth ice hockey. Br. J. Sports Med. 40, 128–132.

14.

McCrea

, Hammeke

, Olsen

, Leo

, and Guskiewicz

(2004). Unreported concussion in high school football players: implications for prevention. Clin. J. Sport Med. 14, 13–17.

15.

Llewellyn

, Burdette

G.T.

, Joyner

A.B.

, and Buckley

T.A.

(2014). Concussion reporting rates at the conclusion of an intercollegiate athletic career. Clin. J. Sport Med. 24, 76–79.

16.

McCrea

, Guskiewicz

K.M.

, Marshall

S.W.

, Barr

, Randolph

, Cantu

R.C.

, Onate

J. A.

, Kelly

J.P.

, and Page

(2004). Acute effects and recovery time following concussions in collegiate football players. JAMA, 290, 2556–2563.

17.

Makdissi

, Darby

, Maruff

, Ugoni

, Brukner

, and McCrory

P.R.

(2010). Natural history of concussion in sport. Am. J. Sports Med. 38, 464–471.

18.

McDonald

W.I.

, Compston

, Edan

, Goodkin

, Hartung

H.-P.

, Lublin

F.D.

, McFarland

H.F.

, Paty

D.W.

, Polman

C.H.

, Reingold

S.C.

, Sandberg-Wollheim

, Sibley

, Thompson

, Van Den Noort

, Weinshenker

B.Y.

, and Wolinsky

J.S.

(2001). Recommended diagnostic criteria for multiple sclerosis: guidelines from the International Panel on the Diagnosis of Multiple Sclerosis. Ann. Neurol. 50, 121–127.

19.

McKhann

G.M.

, Knopman

D.S.

, Chertkow

, Hyman

B.T.

, Jack

C.R.

, Kawas

C.H.

, Klunk

W.E.

, Koroshetz

W.J.

, Manly

J.J.

, Mayeux

, et al. (2011). The diagnosis of dementia due to Alzheimer's disease: Recommendations from the National Institute on Aging-Alzheimer's Association Workgroups on Diagnostic Guidelines for Alzheimer's Disease. Alzheimer's & Dement. 7, 263–269.

20.

American Diabetes Society (2016). Classification and diagnosis of diabetes. Diabetes Care, 39, S13–S22.

21.

Broglio

S.P.

, McCrea

, McAllister

, Harezlak

, Katz

, Hack

, and Hainline

(2017). A national study on the effects of concussion in collegiate athletes and US Military Service Academy members: the NCAA–DoD Concussion Assessment, Research and Education (CARE) Consortium Structure and Methods. Sport Med. 47, 1437–1451.

22.

Carney

, Ghajar

, Jagoda

, Bedrick

, Davis-O'Reilly

, Du Coudray

, Hack

, Helfand

, Huddleston

, Nettleton

, and Riggio

(2014). Concussion guidelines step 1: systematic review of prevalent indicators. Neurosurgery, 75, 3–15.

23.

Royston

(2004). Multiple imputation of missing values. Stata J. 4, 224–241.

24.

Covassin

, Buz Swanik

, and Sachs

M.L.

(2003). Sex differences and the incidence of concussions among collegiate athletes. J. Athl. Train. 38, 238–244.

25.

Broshek

D.K.

, Kaushik

, Freeman

J.R.

, Erlanger

, Webbe

, and Barth

J.T.

(2005). Sex differences in outcome following sports-related concussion. J. Neurosurg. 102, 856–863.

26.

Covassin

, Schatz

, and Swanik

C.B.

(2007). Sex differences in neuropsychological function and post-concussion symptoms of concussed collegiate athletes. Neurosurgery, 61, 345–350.

27.

Covassin

, Elbin

R.J.

, Harris

, Parker

, and Kontos

(2012). The role of age and sex in symptoms, neurocognitive performance, and postural stability in athletes after concussion. Am. J. Sports Med. 40, 1303–1312.

28.

Kutcher

J.S.

, and Eckner

J.T.

(2010). At-risk populations in sports-related concussion. Curr. Sports Med. Rep. 9, 16–20.

29.

Gessel

L.M.

, Fields

S.K.

, Collins

C.L.

, Dick

R.W.

, and Comstock

R.D.

(2007). Concussions among United States high school and collegiate athletes. J. Athl. Train. 42, 495–503.

30.

Asken

B.M.

, McCrea

M.A.

, Clugston

J.R.

, Snyder

A.R.

, Houck

Z.M.

, and Bauer

R.M.

(2016). “Playing through it”: delayed reporting and removal from athletic activity after concussion predicts prolonged recovery. J. Athl. Train. 51, 329–335.

31.

Elbin

R.J.

, Sufrinko

A.M.

, Schatz

, French

, Henry

, Burkhart

, Collins

M.W.

, and Kontos

A.P.

(2016). Removal from play after concussion and recovery time. Pediatrics, 138, e20160910.

32.

Echemendia

R.J.

, Bruce

J.M.

, Meeuwisse

, Hutchison

M.G.

, Comper

, and Aubry

(2018). Can visible signs predict concussion diagnosis in the National Hockey League?. Br. J. Sports Med. 52, 1149–1154.

33.

Bruce

J.M.

, Echemendia

R.J.

, Meeuwisse

, Hutchison

M.G.

, Aubry

, and Comper

(2018). Development of a risk prediction model among professional hockey players with visible signs of concussion. Br. J. Sports Med. 52, 1143–1148.

34.

McCrea

, Kelly

J.P.

, Randolph

, Kluge

, Bartolic

, Finn

, and Baxter

(1998). Standardized assessment of concussion (SAC): on-site mental status evaluation of the athlete. J. Head Trauma Rehabil. 13, 27–35.

35.

Concussion in Sport Group (2013). Sport Concussion Assessment Tool - 3rd Edition. Br. J. Sports Med. 47, 259.

36.

Riemann

B.L.

, Guskiewicz

K.M.

, and Shields

E.W.

(1999). Relationship between clinical and forceplate measures of postural stability. J. Sport Rehabil. 8, 71–82.

37.

Breiman

, Friedman

, Stone

C.J.

, and Olshen

R.A.

(1984). Classification and Regression Trees. CRC Press, New York, NY.

38.

, Bai

, Garcia

E.A.

, and Li

(2008). ADASYN: Adaptive synthetic sampling approach for imbalanced learning. Proc. Int. Jt. Conf. Neural Networks. 1322–1328.

39.

Broglio

S.P.

, Macciocchi

S.N.

, and Ferrara

M.S.

(2007). Sensitivity of the concussion assessment battery. Neurosurgery, 60, 1050–1057.

40.

Guskiewicz

K.M.

, Ross

S.E.

, and Marshall

S.W.

(2001). Postural stability and neuropsychological deficits after concussion in collegiate athletes. J. Athl. Train. 36, 263–273.

41.

Valovich McLeod

T.C.

, Perrin

D.H.

, Guskiewicz

K.M.

, Shultz

S.J.

, Diamond

, and Gansneder

B.M.

(2004). Serial administration of clinical concussion assessments and learning effects in healthy young athletes. Clin. J. Sport Med. 14, 287–295.

42.

Guskiewicz

K.M.

(2001). Postural stability assessment following concussion: one piece of the puzzle. Clin. J. Sport Med. 11, 182–189.

43.

Riemann

B.L.

, and Guskiewicz

K.M.

(2000). Effects of mild head injury on postural stability as measured through clinical balance testing. J. Athl. Train. 35, 19–25.

44.

Buckley

T.A.

, Munkasy

B.A.

, and Clouse

B.P.

(2018). Sensitivity and specificity of the modified balance error scoring system in concussed collegiate student athletes. Clin. J. Sport Med. 28, 174–176.

45.

Barr

W.B.

, and McCrea

(2001). Sensitivity and specificity of standardized neurocognitive testing immediately following sports concussion. J. Int. Neuropsychol. Soc. 7, 693–702.

46.

Sufrinko

, McAllister-Deitrick

, Womble

, and Kontos

(2017). Do sideline concussion assessments predict subsequent neurocognitive impairment after sport-related concussion?. J. Athl. Train. 52, 676–681.

47.

Guskiewicz

K.M.

, McCrea

, Marshall

S.W.

, Cantu

R.C.

, Randolph

, Barr

, Onate

J.A.

, and Kelly

J.P.

(2003). Cumulative effects associated with recurrent concussion in collegiate football players. JAMA, 290, 2549.

48.

McCrea

, Guskiewicz

, Randolph

, Barr

W.B.

, Hammeke

. a., Marshall

S.W.

, Powell

M.R.

, Woo Ahn

, Wang

, and Kelly

J.P.

(2013). Incidence, clinical course, and predictors of prolonged recovery time following sport-related concussion in high school and college athletes. J. Int. Neuropsychol. Soc. 19, 22–33.

49.

Shehata

, Wiley

J.P.

, Richea

, Benson

B.W.

, Duits

, and Meeuwisse

W.H.

(2009). Sport concussion assessment tool: baseline values for varsity collision sport athletes. Br. J. Sports Med. 43, 730–734.

50.

Piland

S.G.

, Ferrara

M.S.

, Macciocchi

S.N.

, Broglio

S.P.

, and Gould

T.E.

(2010). Investigation of baseline self-report concussion symptom scores. J. Athl. Train. 45, 273–278.

51.

Lovell

M.R.

, Iverson

G.L.

, Collins

M.W.

, Podell

, Johnston

K.M.

, Pardini

, Norwig

, and Maroon

J.C.

(2006). Measurement of symptoms following sports-related concussion: reliability and normative data for the post-concussion scale. Appl. Neuropsychol. 13, 166–174.

52.

Moreau

M.S.

, Langdon

, and Buckley

T. A.

(2014). The lived experience of an in-season concussion amongst NCAA Division I student-athletes. Int. J. Exerc. Sci. 7, 62–74.

53.

Zuckerman

S.L.

, Totten

D.J.

, Rubel

K.E.

, Kuhn

A.W.

, Yengo-Kahn

A.M.

, and Solomon

G.S.

(2016). Mechanisms of injury as a diagnostic predictor of sport-related concussion severity in football, basketball, and soccer. Neurosurgery, 63, 102–112.

54.

Putukian

, Echemendia

, Dettwiler-Danspeckgruber

, Duliba

, Bruce

, Furtado

J.L.

, and Murugavel

(2015). Prospective clinical assessment using sideline concussion assessment tool-2 testing in the evaluation of sport-related concussion in college athletes. Clin. J. Sport Med. 25, 36–42.

55.

Collins

M.W.

, Kontos

A.P.

, Reynolds

, Murawski

C.D.

, and Fu

F.H.

(2014). A comprehensive, targeted approach to the clinical care of athletes following sport-related concussion. Knee Surg. Sports Traumatol. Arthrosc. 22, 235–246.

56.

Collins

M.W.

, Kontos

A.P.

, Okonkwo

D.O.

, Almquist

, Bailes

, Barisa

, Bazarian

, Bloom

O.J.

, Brody

D.L.

, Cantu

, Cardenas

, Clugston

, Cohen

, Echemendia

, Elbin

R.J.

, Ellenbogen

, Fonseca

, Gioia

, Guskiewicz

, Heyer

, Hotz

, Iverson

G.L.

, Jordan

, Manley

, Maroon

, McAllister

, McCrea

, Mucha

, Pieroth

, Podell

, Pombo

, Shetty

, Sills

, Solomon

, Thomas

D.G.

, Valovich McLeod

T.C.

, Yates

, and Zafonte

(2016). Statements of Agreement from the Targeted Evaluation and Active Management (TEAM) Approaches to Treating Concussion Meeting Held in Pittsburgh, October 15–16, 2015. Neurosurgery, 79, 912–929.

57.

Ellis

M.J.

, Ryner

L.N.

, Sobczyk

, Fierstra

, Mikulis

D.J.

, Fisher

J.A.

, Duffin

, and Mutch

W.A.C.

(2016). Neuroimaging assessment of cerebrovascular reactivity in concussion: current concepts, methodological considerations, and review of the literature. Front. Neurol. 7, 1–16.

58.

Mucha

, Collins

M.W.

, Elbin

R.J.

, Furman

J.M.

, Troutman-Enseki

, DeWolf

R.M.

, Marchetti

, and Kontos

A.P.

(2014). A brief Vestibular/Ocular Motor Screening (VOMS) assessment to evaluate concussions. Am. J. Sports Med. 42, 2479–2486.

59.

Pearce

K.L.

, Sufrinko

, Lau

B.C.

, Henry

, Collins

M.W.

, and Kontos

A.P.

(2015). Near point of convergence after a sport-related concussion. Am. J. Sports Med. 43, 3055–3061.

60.

Kontos

A.P.

, Sufrinko

, Elbin

R.J.

, Puskar

, and Collins

M.W.

(2016). Reliability and associated risk factors for performance on the Vestibular/Ocular Motor Screening (VOMS) tool in healthy collegiate athletes. Am. J. Sports Med. 44, 1400–1406.

61.

Anzalone

A.J.

, Blueitt

, Case

, McGuffin

, Pollard

, Garrison

J.C.

, Jones

M.T.

, Pavur

, Turner

, and Oliver

J.M.

(2017). A positive Vestibular/Ocular Motor Screening (VOMS) is associated with increased recovery time after sports-related concussion in youth and adolescent athletes. Am. J. Sports Med. 45, 474–479.