Use of Support Vector Machines Approach via ComBat Harmonized Diffusion Tensor Imaging for the Diagnosis and Prognosis of Mild Traumatic Brain Injury: A CENTER-TBI Study

Abstract

The prediction of functional outcome after mild traumatic brain injury (mTBI) is challenging. Conventional magnetic resonance imaging (MRI) does not do a good job of explaining the variance in outcome, as many patients with incomplete recovery will have normal-appearing clinical neuroimaging. More advanced quantitative techniques such as diffusion MRI (dMRI), can detect microstructural changes not otherwise visible, and so may offer a way to improve outcome prediction. In this study, we explore the potential of linear support vector classifiers (linearSVCs) to identify dMRI biomarkers that can predict recovery after mTBI. Simultaneously, the harmonization of fractional anisotropy (FA) and mean diffusivity (MD) via ComBat was evaluated and compared for the classification performances of the linearSVCs. We included dMRI scans of 179 mTBI patients and 85 controls from the Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI), a multi-center prospective cohort study, up to 21 days post-injury. Patients were dichotomized according to their Extended Glasgow Outcome Scale (GOSE) scores at 6 months into complete (n = 92; GOSE = 8) and incomplete (n = 87; GOSE <8) recovery. FA and MD maps were registered to a common space and harmonized via the ComBat algorithm. LinearSVCs were applied to distinguish: (1) mTBI patients from controls and (2) mTBI patients with complete from those with incomplete recovery. The linearSVCs were trained on (1) age and sex only, (2) non-harmonized, (3) two-category-harmonized ComBat, and (4) three-category-harmonized ComBat FA and MD images combined with age and sex. White matter FA and MD voxels and regions of interest (ROIs) within the John Hopkins University (JHU) atlas were examined. Recursive feature elimination was used to identify the 10% most discriminative voxels or the 10 most discriminative ROIs for each implementation. mTBI patients displayed significantly higher MD and lower FA values than controls for the discriminative voxels and ROIs. For the analysis between mTBI patients and controls, the three-category-harmonized ComBat FA and MD voxel-wise linearSVC provided significantly higher classification scores (81.4% accuracy, 93.3% sensitivity, 80.3% F1-score, and 0.88 area under the curve [AUC], p < 0.05) compared with the classification based on age and sex only and the ROI approaches (accuracies: 59.8% and 64.8%, respectively). Similar to the analysis between mTBI patients and controls, the three-category-harmonized ComBat FA and MD maps voxelwise approach yields statistically significant prediction scores between mTBI patients with complete and those with incomplete recovery (71.8% specificity, 66.2% F1-score and 0.71 AUC, p < 0.05), which provided a modest increase in the classification score (accuracy: 66.4%) compared with the classification based on age and sex only and ROI-wise approaches (accuracy: 61.4% and 64.7%, respectively). This study showed that ComBat harmonized FA and MD may provide additional information for diagnosis and prognosis of mTBI in a multi-modal machine learning approach. These findings demonstrate that dMRI may assist in the early detection of patients at risk of incomplete recovery from mTBI.

Introduction

Traumatic brain injury (TBI) is a global public health problem, with nearly 70,000,000 TBI cases worldwide every year, and over 2,500,000 cases in Europe.¹ The majority of TBIs (> 85%) may be classified as mild. The term “mild” underestimates the effect of this brain injury, with up to 50% of the patients having persisting symptoms, which may include cognitive, psychological, and somatic problems that can last for months to years after injury.^2
–4 The development of accurate, data-driven methods for classifying and characterizing (image-based brain) neuroimaging features of mild TBI (mTBI) could help identify those at risk of developing deficits that affect functional outcome, stratify for early treatment, and design future clinical trials.

The presence and type of lesions visualized on conventional computed tomography (CT) and structural magnetic resonance imaging (MRI) scans do not explain all variance in outcome observed after mTBI.^5

–8 Indeed scans may appear to be normal even in patients with persisting symptoms. The more advanced diffusion MRI (dMRI) has been shown to better detect subtle abnormalities associated with mTBI, because dMRI is susceptible to microstructural changes in particularly in brain white matter (WM).^5,9
–11 Detecting these WM alterations may offer the potential to improve outcome prediction for mTBI patients.

Fractional anisotropy (FA) and mean diffusivity (MD) are the most commonly used diffusion parameters, representing the fraction of the total diffusion attributable to anisotropic diffusion and the overall extent of diffusivity along the direction of WM tracts, respectively. Abnormalities in FA and MD in mTBI patients with incomplete recovery, compared to patients with complete recovery and controls have been reported to be primarily located in the corpus callosum, right anterior thalamic radiations, superior longitudinal fasciculus, and inferior longitudinal, fronto-occipital fasciculi and cerebellum.^12

–15 Some mTBI studies have reported that abnormalities in diffusion parameters are associated with clinically relevant outcomes, including cognitive and functional impairment.^5,16,17

Many studies on recovery from mTBI rely on small sample sizes, leading to low reproducibility of results. Nonetheless, research is evolving toward large multi-center studies with the aim of increasing statistical power and generalizability of results. The success of a joint analysis is highly dependent on the implementation of harmonization procedures to increase the comparability of the multi-scanner data.¹⁸ It has been shown that the variability in diffusion metrics in the corpus callosum among controls, mTBI, and moderate TBI patients, are of the same order of magnitude as intra-scanner changes.¹⁹ Therefore, it is crucial to reduce the variability of diffusion data across multiple scanners before classification implementations. Hence, there is a substantial need for robust harmonization techniques.^20,21 The overall concept of harmonization is to apply mathematical concepts to reduce unwanted site variability while maintaining the biological information to further evaluate outcome and recovery of mTBI patients from their imaging data.

Many prediction models have been developed to investigate and anticipate the outcome of patients who have sustained an mTBI, predominantly using regression techniques.^22
–24 However, none of these models has both good discrimination and prediction and/or are not yet suitable for use in clinical practice.²³ In recent years, there has been a shift in clinical prediction models toward the study of dynamic models, whereas in the case of imaging prediction models, because of the static nature of imaging, the exploration of new indices using machine learning (ML) techniques offers a new promising diagnostic path to improve outcome prognostication.^25,26 ML methods implemented with multiple dMRI-derived indices may help to infer the pathophysiological features of WM changes and provide more specific biomarkers for WM neuropathology in mTBI patients.

Support vector machines (SVMs) are popular ML models that have demonstrated considerable potential for robustly identifying neuroimaging biomarkers of neurological dysfunctions and diseases.^27
–29 SVMs can achieve reliable performance by determining a hyper-plane that divides the samples into two groups. Compared with conventional regression-based methods, ML-based approaches are able to reveal subtle and complex patterns that can be used to classify various psychiatric and neurodegenerative disorders, such as schizophrenia,²⁸ dementia,³⁰ and Parkinson's disease.³¹ In particular, support vector classifiers (SVCs) have been applied to non-harmonized dMRI and resting-state functional connectivity data to detect mTBI, identifying biomarkers to characterize mTBI and suggesting that ML can reveal underlying mTBI-related neuropathology.^32

–35 Nonetheless, harmonization has been an important step that has been shown to remove unwanted scanner biases and improve the image analysis, which is recommended in multi-site investigations.¹⁸

The aim of the study was to explore whether linearSVCs can aid in classification of mTBI patients and predict recovery. Simultaneously, the effect of ComBat harmonization was evaluated on the performance of linearSVCs. Two key analyses were performed. First, an analysis to investigate whether the use of linearSVCs allows for distinguishing between mTBI patients and controls; and second whether linearSVCs trained on ComBat harmonized dMRI metrics obtained within 21 days of injury are able to predict the clinical outcome of mTBI patients at 6 months.

Methods

Participants

All eligible subjects were included from the prospective observational study Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI) (December 19, 2014, to December 17, 2017; https://clinicaltrials.gov/ct2/show/NCT02210221).^3,36 CENTER-TBI was accessed using the Neurobot platform (RRID/SCR_017004, core data, version 3.0; International Neuroinformatics Coordinating Facility; released November 24, 2020). Ethical approval for CENTER-TBI was obtained in accordance with all relevant laws and regulations for each recruiting site. Details may be found at: https://www.center-tbi.eu/project/ethical-approval. Informed consent from the patient or legal representative/next of kin was obtained for all participants. Reporting of this study follows Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD).³⁷ TRIPOD Checklist available in the Supplementary Material.

Inclusion criteria for this analysis were being ≥16 years of age who had sustained an mTBI (defined as Glasgow Coma Score [GCS] on presentation of 13–15), requiring a head CT according to local criteria on initial presentation, and having had an MRI within 21 days of the injury. Outcome at 6 months was measured using the Extended Glasgow Outcome Scale (GOSE), a classification based on function, independence, and participation.³⁸ The preferred method for assessment was by interview in person. However, to maximize the follow-up rate, postal and Web-based questionnaires and administration by telephone were also allowed. These different methods for assessment of GOSE were found to have good agreement in the larger CENTER-TBI core data set.³⁹ Quantitative imaging analysis did not occur until after follow-up had been completed. Hence, raters were blinded to the neuroimaging results. GOSE was used to dichotomize between complete (GOSE = 8) and incomplete (GOSE <8) recovery.

Image acquisition and analysis

The initial cohort consisting of 194 eligible patients (15 scanners, 10 sites) underwent MRI at 3 Tesla with each scanner using similar acquisition protocols. The full scanning protocol details for all sites and scanners can be found at https://www.center-tbi.eu/project/mri-study-protocols. The same imaging protocol was obtained on 89 eligible controls (12 scanners, 9 sites). Sequences included volumetric T1-weighted, volumetric fluid-attenuated inversion recovery, T2-weighted, and susceptibility-weighted imaging, and diffusion tensor imaging (DTI). Base values of DTI were 2-mm isotropic voxels, 32 non-collinear directions, and a b value of 1000 sec/mm². MRI scans were reported centrally by CENTER-TBI investigators blinded to patient outcome based on Common Data Elements (CDEs)⁴⁰ and using all available sequences to identify any intracranial traumatic abnormality. This variable indicates whether any of the following CDEs is present (detailed description available in Table S1 mass lesion, extra axial hematoma, epidural hematoma, acute subdural hematoma, subacute or chronic subdural hematoma, subdural collection mixed density, contusion, traumatic axonal injury, traumatic subarachnoid hemorrhage, intraventricular hemorrhage, midline shift, or cisternal compression.

Sequences were processed using a TBI-specific pipeline. All DTI data were corrected for noise,^41,42 Gibbs ringing artifacts,⁴³ head motion and eddy current artifacts,⁴⁴ and inhomogeneities in the magnetic field.⁴⁵ Diffusion tensors were obtained by weighted least squares fitting to derive MD and FA maps using the FMRIB Software Library.⁴⁶ After diffusion tensor model fitting, each FA map was spatially normalized to the FA template of the John Hopkins University (JHU ICBM-DTI-81) Atlas⁴⁷ via non-linear registration using advanced normalization tools (ANTs).^48,49 The transformations were subsequently used to project corresponding MD maps to the same space.

Image data and pipeline outputs for controls and patients were visually inspected to ensure that no scans with artifacts were included. The diffusion parameter maps (FA and MD) were visually inspected and two mTBI patients and one control were excluded because of blurring in the dMRI parameter maps caused by excessive head motion artifacts. Zero mean normalized cross-correlation (ZNCC) was computed between the JHU FA template and the aligned FA maps to quantify registration accuracy. Subjects were excluded from further analysis if the ZNCC similarity metric fell below a pre-defined threshold (ZNCC <0.8). Therefore, 13 mTBI patients and three controls were excluded based on this criterion. After identifying the eligible diffusion maps, data were retained and considered to reflect true variation or pathology. The final data cohort consisted of 179 patients (15 scanners, 10 sites) and 85 controls (12 scanners, 9 sites; comparable age and sex), with a minimum of two controls and/or mTBI patients per scanner, maximizing the number of subjects for the present study. Patients were dichotomized according to their GOSE scores at 6 months into those with complete (n = 92; GOSE = 8) and those with incomplete (n = 87; GOSE <8) recovery. A flow chart of patients can be seen in Supplementary Figure S1.

WM voxels of the spatially normalized FA and MD maps were harmonized on a voxelwise level using ComBat.⁵⁰ ComBat facilitates adjustment of quantitative diffusion maps while taking into account the effect of possible confounding factors. In this study, age, sex, and disease status served as biological covariates while fitting the harmonization model, which should be protected during the removal of scanner/site effects. FA and MD maps were ComBat harmonized in two ways: (1) using two categories for disease status (controls and mTBI patients), and (2) using three categories for disease status (controls, mTBI patients with complete recovery, and mTBI patients with incomplete recovery). The categories were chosen to identify the optimal use of ComBat harmonization to address the overall aims of the study.

Machine learning

LinearSVCs were applied to differentiate mTBI patients from controls and mTBI patients with complete from those with incomplete recovery using the dMRI maps, sex, and age. Sex and age were included as they have previously been shown to affect both the quantitative metrics obtained with DTI (FA and MD), as well as outcome after mild TBI.^51

–54 The linearSVCs were either trained on the voxel- or ROI-wise FA and MD images (1) non-harmonized, (2) ComBat harmonized with two categories, and (3) ComBat harmonized with three categories combined with age and sex. FA and MD maps were examined on a voxel- and ROI-wise level within the areas of the JHU ICBM-DTI-81 Atlas (Fig. 1). For the voxelwise approach, each FA and MD voxel intensity within the atlas served as individual features. For the ROI-wise approach, FA or MD intensities were averaged within each of the 48 JHU regions, such that we end up with 96 (2 x 48) image derived regional features to train the linearSVC.

FIG. 1.

Overlap between Montreal Neurological Institute (MNI) brain template and the Johns Hopkins University (JHU) white matter (WM) atlas regions (GCC, genu corpus callosum; BCC, body corpus callosum; SCC, splenium corpus callosum; Fx, fornix; mCP, middle cerebellar peduncle; CP, right and left [R/L] cerebral peduncle; iCP, inferior cerebellar peduncle; sCP, superior cerebellar peduncle; CST, corticospinal tract; ACR, anterior corona radiata; PCR, posterior corona radiata; SLF, superior longitudinal fasciculus; EC, external capsule; pIC, posterior limb of internal capsule; cCg, cingulate gyrus part of the cingulum; hCg, hippocampus part of the cingulum; PTR, posterior thalamic radiation; and Tp, tapetum).

Analyses were conducted with the ‘sklearn’ package of Python (version 3.5.0)⁵⁵ with the linear kernel SVC and default regularization (C = 1). To estimate the linearSVCs generalizability, a stratified fourfold cross-validation (CV) was performed (for an overview of the proposed classification framework see Fig. 2). For this, the data set was divided into four subsets, each including all disease categories (e.g., controls and mTBI patients with different recovery). For each fold, three subsets were chosen to train the model, which was then validated on the remaining subset.

FIG. 2.

Flow chart of the analysis and prediction framework.

Recursive feature elimination (RFE)⁵⁶ is an iterative feature selection algorithm, which ranks features (e.g., voxels or ROIs within the JHU Atlas) in a training data set based on their relevance for predicting the target variable (e.g., controls and mTBI patients or mTBI recovery at 6 months post-injury). RFE was used to identify the 10% most discriminative JHU voxels or the ten most discriminative JHU ROIs to classify mTBI separately from controls, and to predict mTBI recovery. For the voxelwise approach, the 10% most discriminative WM voxels within the JHU regions were identified in each of the four folds. Similarly, for the ROI-wise approach, from the 48 JHU regions, the 10 most discriminative ROIs were selected in each of the folds in the cross-validation method. Next, for each fold, the classification scores (accuracy, sensitivity, specificity, and F1-score) of the different linearSVC models were calculated based on the results obtained from the RFE-selected 10% most discriminative JHU voxels or the RFE-selected 10 most discriminative JHU ROIs exclusively, while the remaining voxels or ROIs were excluded from the analysis.

Statistical analysis

Analysis of variance (ANOVA) was performed to compare the demographics among the groups (controls, mTBI complete and incomplete recovery).⁵⁷ Groups were considered statistically different if p < 0.05.

The linearSVCs were compared based on accuracy, sensitivity, specificity, and F1-score, calculating the mean and standard error across the fourfold cross-validation based on the positive and negative predicted values (Table 1).

Table 1.

Meaning of Positive and Negative Predictions

Classification	Prediction
Classification	Negative (class 0)	Positive (class 1)
Controls vs. mTBI patients	Controls	mTBI patients
mTBI complete vs. incomplete recovery	mTBI complete recovery	mTBI incomplete recovery

mTBI, mild traumatic brain injury.

The voxels (voxelwise approach) or brain regions (ROI-wise approach) that significantly contributed to the classification between the two groups identified via RFE were used for further analysis. For the voxelwise approach, discriminative voxels identified in at least two folds were mapped to the JHU Atlas. The relative voxel count was calculated as the percentage of discriminative voxels in the JHU regions relative to all voxels in the WM structure.

Additionally, the mean value across those discriminative FA and MD voxels was calculated and compared between groups via Student's t test. For the ROI-wise approach, the 10 most discriminative JHU ROIs were selected via the RFE algorithm in each of the four folds and retained as the final discriminative ROIs if identified in at least two folds. The average FA and MD for the 48 JHU regions were compared between groups using Student's t test. The diffusion metrics between groups were considered statistically different if p < 0.05, after controlling for false discovery rate (FDR).⁵⁸

Lastly, to evaluate the classification and prediction performances for the different linearSVCs, the receiver operating characteristic (ROC) curves were obtained. Classification accuracy was also assessed using the AUC. The paired non-parametric DeLong test was used to compare the difference AUCs among the implemented linearSVC methods across the four folds.⁵⁹

Results

Demographic data are listed in Table 2. The cohort consists of 85 controls and 179 mTBI patients, 92 with complete recovery (GOSE = 8) and 87 with incomplete recovery (GOSE <8). Age differed significantly among the three groups: controls and mTBI patients with complete and incomplete recovery (p < 0.05). Two thirds of each patient group were male (66 male and 26 female for complete recovery and 56 male and 31 female for incomplete recovery). There was no significant difference for the time between injury and scan for the patient cohorts (mean number of days was 8.5 for complete recovery and 9.0 for incomplete recovery). For both complete and incomplete recovery, the majority of mTBI patients had an initial GCS of 15 (76 out of 92 mTBI with complete recovery and 64 out of 87 with incomplete recovery). The most prevalent mechanisms of injury for both patient groups were road collisions and falls (40 and 32 for complete and 40 and 36 for incomplete recovery, respectively). Patients with incomplete recovery after an mTBI had a (prevailing) GOSE score of 7 after 6 months (42 out of 87 mTBI patients).

Table 2.

Demographic and Clinical Data for the Included Patients

			mTBI (n = 179)
		Controls (n = 85)	Complete recovery (n = 92)	Incomplete recovery (n = 87)
Age (mean±SD)		42.9 ± 11.7	37.9 ± 15.5	46.2 ± 17.1
Sex	Male	49	66	56
Sex	Female	35	26	31
GCS	15	85	76	64
	14	-	11	17
	13	-	5	6
Scan time post-injury (days mean±SD)		-	8.5 ± 7.4	9.0 ± 7.3
Mechanism of injury	Road collision	-	40	40
	Fall	-	32	36
	Cycling accident	-	9	3
	Violence	-	6	5
	Unknown	-	1	2
	Other	-	4	1
GOSE	8	85	92	-
	7	-	-	42
	6	-	-	26
	5	-	-	13
	4	-	-	3
	3	-	-	3
	2	-	-	-
	1	-	-	-
Intracranial abnormalities	Present	-	24	52
Intracranial abnormalities	Absent	85	68	35

mTBI, mild traumatic brain injury; SD, standard deviation; GCS, Glasgow Coma Scale Score; GOSE, Extended Glasgow Outcome Scale.

Intracranial abnormalities are described in Supplementary Table S1. The number of subjects per scanner was comparable across 14 of the 15 scanners (Supplementary Table S2, average of seven controls, eight complete, and six incomplete recovery per scanner), with exception of scanner #1 which scanned the highest number of mTBI patients.

The results will be presented separately for the classification between controls and mTBI patients and the prediction between complete (GOSE = 8) and incomplete (GOSE <8) recovery of the mTBI patients.

Classification of mTBI patients versus controls

Classifying mTBI patients versus controls based on age and sex yielded a classification accuracy of 59.8 (± 1.0)%. The addition of imaging features FA/MD yielded a higher classification performance than using age and sex alone. The voxelwise approaches provided a classification accuracy of 73.5 (± 3.5)% for non-harmonized FA and MD, 80.7 (± 2.1)% for the maps harmonized with ComBat using two categories, and 81.4 (± 1.7)% for the maps harmonized with ComBat using three categories (Table 3). For the ROI-wise approach, the classification scores obtained were 65.2 (± 3.9)% for non-harmonized diffusion maps, 62.9 (± 2.2)% for the maps harmonized with ComBat using two categories and 64.8 (± 2.9)% for the maps harmonized with ComBat using three categories for the ROI-wise approach (Table 3). Comparing the multiple classification features listed in Table 3 and shown in Figure 3, the implantation based on ComBat 3 categories voxelwise approach including age and sex was the one that provided the highest performance scores when classifying controls and mTBI patients (93.3% sensitivity, 81.4% accuracy, 80.3% F1-score, and 0.88 AUC).

FIG. 3.

Receiver operating characteristic (ROC) curves for the fourfold cross-validation linearSVCs classification between controls and mild traumatic brain injury (mTBI) patients based on: (A) age and sex, without imaging input; (B) non-harmonized fractional anisotropy (FA) and mean diffusivity (MD) combined with age and sex voxelwise (top row) and region of interest (ROI)-wise (bottom row); (C) ComBat harmonized two categories FA and MD combined with age and sex voxelwise (top row) and ROI-wise (bottom row); and (D) ComBat harmonized three categories FA and MD combined with age and sex voxelwise (top row) and ROI-wise (bottom row).

Table 3.

Averaged Performance Statistics of the Fourfold Cross-Validation for the Classification Between mTBI and Controls (Average ± Standard Error)

mTBI vs. controls: Fourfold cross-validation
Mean across four folds	Sensitivity (%)	Specificity (%)	Accuracy (%)	F1-score (%)	AUC
Age & sex	68.2 ± 1.4	42.4 ± 5.2	59.8 ± 1.0	60.2 ± 1.2	0.56 ± 0.03
Non-harmonizedFA + MD + age/sex10% voxels JHU	88.3 ± 1.6	42.1 ± 8.2	73.5 ± 3.5	71.6 ± 4.1	0.78 ± 0.04
ComBat 2 categoriesFA + MD + age/sex10% voxels JHU	93.3 ± 1.8	54.0 ± 6.8	80.7 ± 2.1	79.3 ± 2.5	0.87 ± 0.02
ComBat 3 categoriesFA + MD + age/sex10% voxels JHU	93.3 ± 1.8	56.4 ± 5.4	81.4 ± 1.7	80.3 ± 1.9	0.88 ± 0.02
Non-harmonizedFA + MD + age/sex10 ROIs JHU	67.6 ± 6.9	59.8 ± 5.2	65.2 ± 3.9	65.7 ± 3.5	0.68 ± 0.05
ComBat 2 categoriesFA + MD + age/sex10 ROIs JHU	65.3 ± 3.0	57.6 ± 11.4	62.9 ± 2.2	63.4 ± 2.7	0.67 ± 0.04
ComBat 3 categoriesFA + MD + age/sex10 ROIs JHU	69.3 ± 1.2	55.4 ± 6.8	64.8 ± 2.9	65.4 ± 3.1	0.69 ± 0.03

mTBI, mild traumatic brain injury; AUC, area under the curve; FA, fractional anisotropy; MD, mean diffusivity; JHU, Johns Hopkins University; ROI, region of interest.

Based on the paired DeLong test (Table 4), comparing the AUCs across the CV folds for each of the linearSVCs implementations, comparing age and sex alone as input, and using age and sex combined with FA and MD voxel- or ROI-wise, the use of diffusion imaging significantly improves (p < 0.05) the classification between controls and mTBI patients. When looking at the voxelwise comparison among harmonization approaches, there is a significant improvement (p < 0.05) in the use of ComBat harmonization for the use of two or three categories in the disease status, but no significant differences for the ROI-wise approach. Finally, when comparing the same harmonization approaches for both voxel- and ROI-wise implementations, there are significant differences for both ComBat harmonization approaches but none for the non-harmonized FA/MD. Therefore, comparing the linearSVC implementations, ComBat harmonized using three disease status categories FA and MD maps in the voxel-wise linearSVC significantly improves the differentiation between mTBI and controls (p < 0.05).

Table 4.

DeLong Test Comparison Between Implementations for the Classification Between Controls and mTBI Patients: Delong z Value, p Value, and Statistical Significance

mTBI vs. controls: DeLong test
LinearSVC comparisons	Z	p value	Statistical significance
Age/Sex vs. non-harmonizedvoxelwise FA + MD + age/sex	-3.92	8.8 E-5	^***
Age/Sex vs. ComBat 2 categoriesvoxelwise FA + MD + age/sex	-6.91	4.7 E-12	^***
Age/Sex vs. ComBat 3 categoriesvoxelwise FA + MD + age/sex	-6.65	1.8 E-11	^***
Age/Sex vs. non-harmonizedROI-wise FA + MD + age/sex	-2.80	5.1 E-3	^**
Age/Sex vs. ComBat 2 categoriesROI-wise FA + MD + age/sex	-2.31	2.1 E-2	^*
Age/Sex vs. ComBat 3 categoriesROI-wise FA + MD + age/sex	-2.74	6.2 E-3	^**
Non-harmonizedvoxelwise FA + MD + age/sexvs. ComBat 2 categoriesvoxelwise FA + MD + age/sex	-2.86	4.2 E-3	^**
Non-harmonizedvoxelwise FA + MD + age/sexvs. ComBat 3 categoriesvoxelwise FA + MD + age/sex	-3.38	8.8 E-4	^***
ComBat 2 categoriesvoxelwise FA + MD + age/sexvs ComBat 3 categoriesvoxelwise FA + MD + age/sex	-0.28	7.8 E-1
Non-harmonizedROI-wise FA + MD + age/sexvs. ComBat 2 categoriesROI-wise FA + MD + age/sex	0.50	6.2 E-1
Non-harmonizedROI-wise FA + MD + age/sexvs. ComBat 3 categoriesROI-wise FA + MD + age/sex	0.01	9.9 E-1
ComBat 2 categoriesROI-wise FA + MD + age/sexvs. ComBat 3 categoriesROI-wise FA + MD + age/sex	-0.56	5.7 E-1
Non-harmonizedROI-wise FA + MD + age/sexvs. non-harmonizedvoxelwise FA + MD + age/sex	-1.42	1.5 E-1
ComBat 2 categoriesROI-wise FA + MD + age/sexvs. ComBat 2 categoriesvoxelwise FA + MD + age/sex	-4.34	1.5 E-5	^***
ComBat 3 categoriesROI-wise FA + MD + age/sexvs ComBat 3 categoriesvoxelwise FA + MD + age/sex	-4.31	1.7 E-5	^***

p < 0.05, ^** p < 0.01, ^*** p < 0.001.

mTBI, mild traumatic brain injury; FA, fractional anisotropy; MD, mean diffusivity; ROI, region of interest.

The most predictive voxels were consistent across the voxels selected when using ComBat harmonization using two or three categories or non-harmonized data. On the other hand, for the ROI-wise classification, most predictive ROIs selected based on ComBat harmonized FA and MD maps using two or three categories were consistent, but for non-harmonized maps, the areas identified were not similar to those for the harmonized data.

As can be seen in Figure 4, for the three-category-harmonized ComBat voxelwise approach (highest classification scores) the most predictive brain tracts were identified in the middle cerebellar peduncle, corpus callosum, external capsule, superior longitudinal fasciculus, and posterior thalamic radiation. In the three-category-harmonized ComBat ROI-wise approach, as can be seen in Figure 5, the most predictive brain tracts identified were the corpus callosum, anterior corona radiata, external capsule, posterior thalamic radiation, superior longitudinal fasciculus, pontine crossing fibers, cerebral peduncles, and superior corona radiata.

FIG. 4.

Location of the 10% most predictive three-category-harmonized ComBat fractional anisotropy (FA) and mean diffusivity (MD) voxels identified via a recursive feature elimination (RFE) algorithm. Voxels identified in at least two folds were mapped to the Johns Hopkins University (JHU) atlas (A) and the relative voxel count for the discriminative voxels (B) for the control versus mild traumatic brain injury (mTBI) classification. The most discriminative voxels identified were middle cerebellar (mCP), splenium corpus callosum (SCC), body corpus callosum (BCC), genu corpus callosum (GCC), right external capsule (EC-R), left external capsule (EC-L), right superior longitudinal fasciculus (SLC-R), left posterior thalamic radiation (pTR-L), left posterior limb of internal capsule (plIN-L), and right superior corona radiata (SCR-R).

FIG. 5.

Location of the most predictive three-category harmonized ComBat fractional anisotropy (FA) and mean diffusivity (MD) Johns Hopkins University (JHU) regions of interest (ROIs) for control versus mild traumatic brain injury (mTBI) classification. The most predictive JHU tracts identified were: genu of the corpus callosum (GCC), body of the corpus callosum (BCC), splenium of the corpus callosum (SCC), anterior corona radiata (ACR), external capsule (EC), posterior thalamic radiation (pTR), superior longitudinal fasciculus (SLF), pontine crossing fibers (PCF), cerebral peduncles (CP), and superior corona radiata (SCR).

Figure 6 shows a violin plot of the mean values and standard deviations of ComBat using three categories FA and MD over the discriminative voxels and ROIs for TBI patients and healthy control groups. For the linearSVCs based on dMRI metrics combined with age and sex, mTBI patients displayed higher MD and lower FA values for the most discriminative voxels and ROIs identified through RFEs. The analysis of the most discriminative voxels and ROIs of the DTI diffusion indices demonstrated possible differences in the WM microstructure in the TBI patient group compared with healthy controls. Significantly decreased FA and increased MD were present in several major WM tracts in the TBI group compared with the healthy control group (FDR corrected p value <0.05).

FIG. 6.

Mean three-category-harmonized ComBat fractional anisotropy (FA) and mean diffusivity (MD) for the most discriminative voxels (A and B) and discriminative region of interest (ROI) - genu of corpus callosum (C and D) for control versus mild traumatic brain injury (mTBI) classification for the Johns Hopkins University (JHU) Atlas analysis.

Prediction between mTBI patients with complete (GOSE = 8) and incomplete (GOSE <8) recovery

The use of age and sex for the differentiation between mTBI patients with complete versus those with incomplete recovery yielded a classification accuracy of 61.4 (± 5.2)%. The addition of imaging features (FA/MD) provided higher classification performance than age and sex alone, when the imaging maps were optimally harmonized, yielding a classification accuracy of 66.2 (± 6.8)% for the three-category-harmonized ComBat FA and MD voxelwise. In contrast, for the non-harmonized FA and MD voxelwise an accuracy of 56.0 (± 7.7)% was obtained, and for the two-category- harmonized ComBat FA and MD voxelwise, 44.7 (± 4.1)% was obtained, demonstrating the impact of harmonization on the prediction results. A similar result was obtained when using the ROI-wise approach: 60.1 (± 4.4)% for the non-harmonized FA and MD ROI-wise, 54.9 (± 3.7)% for the two-category-harmonized ComBat FA and MD ROI-wise, and 64.8 (± 3.3)% for the three-category- harmonized ComBat FA and MD ROI-wise, Table 5. Similarly to the previous classification between controls and patients, comparing the multiple classification features listed in Table 5 and shown in Figure 7, the three-category-harmonized voxelwise approach combined with age and sex was the one that provided the highest performance scores when predicting between mTBI patients with complete and those with incomplete recovery (71.8% specificity, 66.4% accuracy, 66.2% F1-score, and 0.71 AUC).

FIG. 7.

Receiver operating characteristic (ROC) curves for the fourfold cross-validation linearSVCs classification between mild traumatic brain injury (mTBI) patients with complete and incomplete recovery based on: (A) age and sex, without imaging input; (B) non-harmonized fractional anisotropy (FA) and mean diffusivity (MD) combined with age and sex voxelwise (top row) and region of interest (ROI)-wise (bottom row); (C) ComBat harmonized two categories FA and MD combined with age and sex voxelwise (top row) and ROI-wise (bottom row); and (D) ComBat harmonized three categories FA and MD combined with age and sex voxelwise (top row) and ROI-wise (bottom row).

Table 5.

Averaged Performance Statistics of the Fourfold Cross-Validation for the Prediction Between mTBI Complete and Incomplete Recovery (Average ± Standard Error)

mTBI vs. controls: Fourfold cross-validation
Mean across four folds	Sensitivity (%)	Specificity (%)	Accuracy (%)	F1-score (%)	AUC
Age & sex	63.1 ± 5.9	59.8 ± 6.2	61.4 ± 5.2	61.3 ± 5.1	0.50 ± 0.10
Non-harmonizedFA + MD + age/sex10% voxels JHU	49.2 ± 7.2	63.0 ± 9.1	56.3 ± 7.8	56.0 ± 7.7	0.59 ± 0.08
ComBat 2 categoriesFA + MD + age/sex10% voxels JHU	40.1 ± 8.4	50.0 ± 2.8	45.2 ± 4.0	44.7 ± 4.1	0.49 ± 0.05
ComBat 3 categoriesFA + MD + age/sex10% voxels JHU	60.8 ± 7.4	71.8 ± 8.0	66.4 ± 6.8	66.2 ± 6.8	0.71 ± 0.07
Non-harmonizedFA + MD + age/sex10 ROIs JHU	63.0 ± 6.3	57.6 ± 5.7	60.3 ± 4.4	60.1 ± 4.4	0.65 ± 0.06
ComBat 2 categoriesFA + MD + age/sex10 ROIs JHU	56.2 ± 5.3	53.4 ± 6.8	55.3 ± 3.7	54.9 ± 3.7	0.60 ± 0.05
ComBat 3 categoriesFA + MD + age/sex10 ROIs JHU	63.2 ± 4.4	66.3 ± 4.1	64.8 ± 3.3	64.7 ± 3.3	0.68 ± 0.05

mTBI, mild traumatic brain injury; AUC, area under the curve; FA, fractional anisotropy; MD, mean diffusivity; JHU, Johns Hopkins University; ROI, region of interest.

The paired DeLong test, Table 6, revealed significant differences (p < 0.05) between the approaches: (1) age and sex alone and the ComBat harmonized with three categories FA and MD voxelwise combined with age and sex linearSVCs, (2) age and sex alone and ComBat harmonized with three categories FA and MD ROI-wise combined with age and sex linearSVCs, (3) non-harmonized and ComBat harmonized with three categories FA and MD voxelwise combined with age and sex linearSVCs, (4) ComBat harmonized with two or three categories FA and MD voxelwise combined with age and sex linearSVCs, and (5) Combat harmonized with two categories FA and MD ROI- and voxelwise combined with age and sex linearSVCs. Thus, the combination of age and sex in addition to the use of ComBat harmonized with three categories FA and MD in the voxelwise approach provided a statistically significant improvement in the prediction between mTBI patients with complete and those with incomplete recovery.

Table 6.

Delong Test Comparison Between Implementations for the Classification Between mTBI Patients With Complete and Those with Incomplete Recovery: Delong Z-Value, p Value, and Statistical Significance

mTBI complete vs. incomplete recovery: DeLong test
LinearSVC comparisons	Z	p value	Statistical significance
Age/Sex vs. non-harmonizedvoxelwise FA + MD + age/sex	-0.72	4.69 E-1
Age/Sex vs. ComBat 2 categoriesvoxelwise FA + MD + age/sex	0.87	3.82 E-1
Age/Sex vs. ComBat 3 categoriesvoxelwise FA + MD + age/sex	-2.90	3.74 E-3	^**
Age/Sex vs. non-harmonizedROI-wise FA + MD + age/sex	-1.64	1.00 E-1
Age/Sex vs. ComBat 2 categoriesROI-wise FA + MD + age/sex	-1.19	2.35 E-1
Age/Sex vs. ComBat 3 categoriesROI-wise FA + MD + age/sex	-2.33	1.97 E-2	^*
Non-harmonizedvoxelwise FA + MD + age/sexvs. ComBat 2 categoriesvoxelwise FA + MD + age/sex	1.62	1.03 E-1
Non-harmonizedvoxelwise FA + MD + age/sexvs. ComBat 3 categoriesvoxelwise FA + MD + age/sex	-2.41	1.58 E-2	^*
ComBat 2 categoriesvoxelwise FA + MD + age/sexvs. ComBat 3 categoriesvoxelwise FA + MD + age/sex	-4.59	4.53 E-6	^***
Non-harmonizedROI-wise FA + MD + age/sexvs. ComBat 2 categoriesROI-wise FA + MD + age/sex	0.47	6.36 E-1
Non-harmonizedROI-wise FA + MD + age/sexvs. ComBat 3 categoriesROI-wise FA + MD + age/sex	-0.88	3.74 E-1
ComBat 2 categoriesROI-wise FA + MD + age/sexvs. ComBat 3 categoriesROI-wise FA + MD + age/sex	-1.31	1.90 E-1
Non-harmonizedROI-wise FA + MD + age/sexvs. Non-harmonizedvoxelwise FA + MD + age/sex	0.99	3.19 E-1
ComBat 2 categoriesROI-wise FA + MD + age/sexvs. ComBat 2 categoriesvoxelwise FA + MD + age/sex	2.06	3.98 E-2	^*
ComBat 3 categoriesROI-wise FA + MD + age/sexvs. ComBat 3 categoriesvoxelwise FA + MD + age/sex	-0.38	7.06 E-1

p < 0.05, ^** p < 0.01, ^*** p < 0.001.

mTBI, mild traumatic brain injury; FA, fractional anisotropy; MD, mean diffusivity; ROI, region of interest.

Similar discriminative voxels and ROIs were identified independently of the harmonization approach. Moreover, the distributions of the discriminative FA and MD as measured with voxelwise or ROI methods in complete and incomplete mTBI recovery were similar when compared with the findings for discriminating between controls and mTBI for the harmonization approaches (Figs. 8 and 9). With the ComBat harmonization with three categories voxelwise approach, the most predictive brain tracts identified were in the middle cerebellar peduncle, corpus callosum, superior longitudinal fasciculus, external capsule, and anterior corona radiata (Fig. 8). Whereas for the ComBat harmonization with three categories ROI-wise approach, the most predictive brain tracts identified were the corpus callosum, external capsule, superior corona radiata, sagittal stratum, middle cerebellar peduncle, medial lemniscus, and cingulum (Fig. 9).

FIG. 8.

Location of the 10% most predictive three-category harmonized ComBat fractional anisotropy (FA) and mean diffusivity (MD) voxels identified via recursive feature elimination (RFE) algorithm. Voxels identified in at least two folds were mapped to the Johns Hopkins University (JHU) Atlas (A) and the relative voxel count for the discriminative voxels (B) for the mild traumatic brain injury (mTBI) complete versus incomplete recovery prediction. The most discriminative voxels identified were middle cerebellar (mCP), splenium corpus callosum (SCC), genu corpus callosum (GCC), body corpus callosum (BCC), right superior longitudinal fasciculus (SLC-R), left superior longitudinal fasciculus (SLC-L), right external capsule (EC-R), left external capsule (EC-L), anterior corona radiata (ACR-R), and left anterior corona radiata (ACR-L).

FIG. 9.

Most predictive three-category-harmonized ComBat fractional anisotropy (FA) and mean diffusivity (MD) Johns Hopkins University (JHU) regions of interest (ROIs) for the mild traumatic brain injury (mTBI) complete versus incomplete recovery prediction. The most predictive identified JHU tracts were: genu of the corpus callosum (GCC), body of the corpus callosum (BCC), external capsule (EC), superior corona radiata (SCR), sagittal stratum (SS), middle cerebellar peduncles (mCP), medial lemniscus (ML), and cingulum (Cg).

Significantly increased FA and decreased MD were present in some major WM JHU tracts in patients with complete versus those with incomplete recovery (FDR corrected p value <0.05, Fig. 10). These differences for complete and incomplete recovery are expected to be related to microstructural neuropathological alterations in WM bundles in the semi-acute phase.⁶⁰

FIG. 10.

Mean ComBat harmonization with three categories fractional anisotropy (FA) and mean diffusivity (MD) for the most discriminative voxels (A and B) and discriminative region of interest (ROI) - the body of corpus callosum (C and D) for the mild traumatic brain injury (mTBI) complete versus incomplete recovery prediction for the Johns Hopkins University (JHU) Atlas analysis.

Discussion

In this study, ComBat harmonized FA and MD were demonstrated to provide additional information for recovery prediction in mTBI using a multi-modal ML approach, based on dMRI acquired within the first 21 days post-injury.

The results of the predictions based on the dMRI quantitative metrics were compared with the baseline predictions based on age and sex only. Age and sex are factors that are likely associated with recovery after an mTBI. Both variables interact and are related to cortical maturation, biological response, and social modifiers.^52,60,61 Therefore, using age and sex only in a predictive model is a useful baseline comparison for research and provides an early assessment of injury severity, but it is not sufficiently accurate to guide decision making in the clinical setting.

We applied ComBat harmonization to remove unwanted sources of variability, while preserving variations caused by other biologically relevant covariates, such as age, sex, and disease status, in the data, because the method accounts for systemic intensity variations caused by inter-scanner biases.¹⁸ Nonetheless, the choice of parameters for the harmonization, such as disease status, influences the accuracy of the classification groups. It should be noted that ComBat harmonization is a retrospective harmonization tool, and that therefore our data set could be regarded as a best-case simulation of how SVCs would perform when using perfectly harmonized data sets.

The linear SVC trained on age and sex and ComBat three-category-harmonized voxels of FA and MD to differentiate mTBI patients from controls yielded an accuracy of 81.4 (± 3.4)% and an AUC of 0.88 (± 0.04). Distinguishing between mTBI patients with complete and those with incomplete recovery was demonstrated to be more challenging, yielding a prediction accuracy of 66.4 (± 13.7)% and an AUC of 0.71 (± 0.12) for the age and sex and ComBat three-category-harmonized FA and MD voxelwise approach (Fig. 7). Our results are comparable to the UPFRONT model, which is the current gold standard for the prediction of outcome for complete (GOSE = 8) and incomplete (GOSE <8) recovery in mTBI, providing an of AUC 0.7.⁶²

The discriminative voxels identified on the DTI maps that differed between mTBI patients and controls were predominantly located in the corpus callosum, middle cerebral peduncle, right and left cerebral peduncles, external capsule, fornix, and right and left tapetum. The identified discriminative brain regions associated with mTBI with complete or incomplete recovery based on the GOSE 6 months post-injury were mainly the corpus callosum, middle cerebral peduncles, external capsule, and the right and left anterior corona radiata. These areas mostly overlapped with the regions identified by the linearSVC that differentiates mTBI patients from controls.

Many JHU regions that showed different diffusion metrics in mTBI and control subjects were also shown to be discriminative when classifying patients with different outcomes. This implies that the DTI differences found to be associated with mTBI are also important for the assessment of recovery post-injury. Further, the cerebellar peduncle, brainstem, and corpus callosum have previously been found to have prognostic value in moderate and severe TBI,^63
–65 and their identification as discriminative for mTBI in this analysis adds weight to the biological plausibility of our findings. This is also consistent with the hypothesis that the pathological and biomechanical mechanisms in mTBI may be similar (at least in part) to those in severe TBI; albeit that differences may be more subtle, hence harder to detect.

The regions identified as predictive in the current study have been previously associated with mTBI outcome. Reduced FA is the most common finding in the acute and semi-acute phases of mTBI when comparing patients and controls.¹¹ However, some studies in the acute/early subacute phase of mTBI have shown significantly increased FA in major WM tracts, consistent with our findings of an increased FA and reduced MD in some major WN JHU tracts in patients with complete versus incomplete recovery (Fig. 10).^66,67 An investigation using whole-brain voxelwise non-parametric statistical comparison evaluated mTBI patients with complete or incomplete recovery, and found higher MD values voxelwise in patients with incomplete recovery than in patients with complete recovery and controls, in the corpus callosum, right anterior thalamic radiations, superior longitudinal fasciculus, and inferior longitudinal and fronto-occipital fasciculi at 7–28 days after injury.¹² Ling and colleagues found increased FA and decreased radial diffusivity voxelwise within the genu of the corpus callosum, in a cohort of 28 mTBI patients with complete or incomplete recovery who underwent MRI 15.6–4.3 days after injury.¹³ In contrast, an ROI approach found no significant difference in FA or MD in the genu, body, or splenium of the corpus callosum in 60 mTBI patients with complete and incomplete recovery (on the more severe end of the mTBI spectrum), in comparison with 34 controls.¹⁴ Yuh and colleagues found that WM FA was significantly reduced in mTBI patients who had positive acute traumatic intracranial abnormality on conventional MRI, but not in negative MRI mTBI patients, compared with controls.¹⁵ In addition, regions of reduced FA in mTBI patients were modest, but statistically significant, predictors of unfavorable 3- and 6-month outcomes. The FA alterations are likely related to microstructural neuropathological changes in the WM major tracts, which may include cytotoxic edema, changes in water content within the myelin sheath, and inflammation.^66
–68

Using the mean FA/MD measurement across all discriminative voxels (Figs. 6A/B and 10A/B), and specific discriminative ROIs (Figs. 6C/D and 10C/D), we found significant (p < 0.05) between-group differences based on the Student's t test calculated with the average FA and MD values. Thus, the differences in the WM dMRI metrics for the discriminative areas identified by the method were demonstrated to be significantly associated with the classification between mTBI patients and controls and with the prediction between complete or incomplete recovery for those mTBI patients. Moreover, the distributions of FA and MD, even if visually similar between the groups, showed significant p values (p < 0.05) in the Student's t test in such comparisons in the neuroanatomical regions of WM known to be vulnerable to axonal injury.

The DeLong test comparisons between the linearSVC models are demonstrated in Tables 4 and 6, for the classification between controls and mTBI patients and the classification between mTBI patients with complete and those with incomplete recovery, respectively. Model performance is related to the use of either single intensities (predictive voxels) or the average voxel intensity in selected regions (predictive ROIs), as well as to the choice of harmonization approach (no harmonization or ComBat-harmonized FA and MD maps with either two or three disease status categories). Based on the paired DeLong test, the ComBat harmonized with three categories FA and MD voxelwise combined with age and sex linearSVC implementation provides a significantly better classification between mTBI and controls than do the other methods (Table 4, p < 0.05), which is not an unexpected result, because the optimal harmonization of single voxels seems to highlight the group differences by preserving the underlying biological changes caused by trauma.⁶⁹ Moreover, a similar trend is demonstrated in the prediction between mTBI patients with complete and those with incomplete recovery. The combination of age and sex in addition to the use of ComBat harmonized with three categories FA and MD in the voxelwise approach provided a statistically significant improvement compared with other models (Table 6, p < 0.05), providing evidence that the single-voxel ComBat harmonization successfully removes scanner effects in diffusion data, preserving the mTBI effects in the brain.⁶⁹

The findings provide evidence for the use of DTI to aid identification of patients at risk of incomplete recovery after mTBI. The prediction of recovery remains an extremely challenging and complex question, which is influenced by multiple pre-injury factors.⁷⁰ DTI significantly improved performance of models predicting complete versus incomplete recovery, showing that there is promise in combining this advanced imaging technology with linearSVCs. However, even the best DTI model only achieved an accuracy of 66%, which is insufficient for clinical practice. Future research therefore needs to address the challenges for outcome prediction after mTBI generally (such as having large enough samples to control for a wide range of pre-morbid factors) and the use of DTI in particular (identifying the optimal timing for imaging and optimal acquisition parameters and harmonization strategies).

Therefore, the results of this study should be considered in the context of certain limitations. First, partial volume effects can lead to abnormal DTI indices in the case of inaccurate registration of individual images into standard space. To reduce the likelihood of such an error, we registered the diffusion maps using the JHU FA map and performed visual checks and correlation evaluations as a strict inclusion criterion for the quantitative dMRI maps in common space. Despite our diligent processing steps and quality control, subtle misregistration between diffusion maps and the atlas cannot be ruled out entirely.

It is important to validate an automated method based on advanced MRI techniques, such as dMRI, which can be used as a predictive tool for a wide spectrum of mild TBI outcomes. Therefore, future work with even larger mTBI patient data sets would be beneficial to increase cohort size for a comprehensive and generalizable predictive model. Additionally, the use of ComBat harmonization should be thought through carefully, as it can be tricky to decide which biological covariates to use with the intention of keeping image alterations related to the mTBI disorder. In our case, we used outcome categories as a biological covariate, but other possibilities are to include variables that become available immediately after the patient is scanned, such as the presence of any intracranial abnormalities including microhemorrhages. In addition, other biomarkers may add prognostic information; for example, proteomic biomarkers including glial fibrillary acid protein (GFAP) or ubiquitin C-terminal hydrolase L1 (UCH-L1).^71,72 The development of more complex prognostic models including this additional information would be best explored in future studies with larger numbers of patients. Given that GOSE may miss more subtle deficits, such studies should also explore more granular outcomes including cognitive and mental health outcomes, the latter being particularly salient in patients at the higher end of functional outcome as may be expected after mTBI.⁷³ Finally, for our study we have only used two biological features (age and sex), which were not removed as confounding factors in the analysis. Although we demonstrated that age and sex seem predictive and that the diffusion metrics FA and MD improve such predictions, further investigation of multiple diffusion metrics (e.g., radial diffusivity, axial diffusivity, mean kurtosis) and their association with direct alterations caused by mTBI would be beneficial for better understanding the underlying neuropathological changes that occur after mTBI. Based on the available measurements from the CENTER-TBI database, we did not include the scan time post-injury, severity of extracranial injury, education, or a history of mental health problems as variables in our model for the reasons outlined in the following paragraphs.

The time between injury and imaging ranged from 1 to 21 days in our study, and it is known that diffusion metrics may change during this time frame.⁶⁶ Because the time to imaging was not significantly different between patients with complete and those with incomplete recovery in our cohort, it would not affect our conclusion that DTI has some prognostic value after TBI. However, it is likely that DTI would perform better if all patients were imaged at the same time point, ideally closer to the time of injury.++⁵

Further, at present there is no accepted framework for handling missing data when applying machine learning algorithms, such as the SVM used in the present analysis. We therefore had to restrict the covariates in our model to data that were available for all patients. Future studies should strive to assess if DTI adds prognostic value also, when other variables are included such as severity of extracranial injury, education, or a history of mental health problems.

Despite the mentioned limitations, our model still adds value to the investigation of mTBI recovery and provides important insights that may be built on in bigger cohorts.

Conclusion

This study shows that ComBat-harmonized dMRI metrics provide additional information related to WM differences, which are demonstrated to be relevant for the classification of mTBI patients, suggesting that DTI may be a predictive marker of recovery in mTBI. We also demonstrated the potential utility of ComBat harmonization on FA and MD maps voxelwise combined with age and sex for mTBI recovery prediction using linearSVCs. Predictive models capable of identifying mTBI patients with the potential for incomplete recovery could facilitate the design of future clinical trials and stratification for treatment planning.

Transparency, Rigor, and Reproducibility Summary

This study was pre-registered at the CENTER-TBI study site (https://clinicaltrials.gov/ct2/show/NCT02210221). The analysis plan was registered after beginning data collection but before data analysis at https://www.center-tbi.eu/data/study. The lead author (M.S.P.) with primary responsibility for the analysis certifies that the analysis plan was pre-specified. A sample size of 194 mTBI patients and 89 controls was planned based on availability of dMRI and GOSE evaluation at 6 months post-injury. Two mTBI patients and one control were excluded because of blurring in the dMRI parameter maps, and 13 mTBI patients and 3 controls were excluded because of poor registration accuracy. Imaging quality control decisions and analyses were performed by investigators who were aware of relevant characteristics of the participants. Actual sample size was 179 mTBI patients, 92 with complete recovery (GOSE = 8) and 87 with incomplete recovery (GOSE <8), and 85 controls. Imaging was collected using multiple 3T MRI scanners and imaging of participants in the relevant groups were distributed across 15 scanners. Variability among scanners was reduced using ComBat harmonization. All equipment and software used to perform imaging and pre-processing are widely available from commercial sources. The primary clinical outcome measure and evaluations are established standards in the field. Data from this study are available in a protected archive: https://www.center-tbi.eu/data. Data can be obtained after approval of a study plan proposal, submitted through the online system. For the purpose of open access, the author has applied a Creative Commons Attribution (CC BY) license to any author accepted manuscript version arising from this submission.

Footnotes

Acknowledgments

We thank Dr. Ella Roelant and Dr. Erik Fransen from StatUa (University of Antwerp, Belgium) for providing statistical advice.

Authors' Contributions

M.S.P. was responsible for investigation, formal analysis and writing – original draft (lead); S.W. was responsible for methodology, and writing – review and editing; E.N.K. was responsible for data curation, methodology, and writing – review and editing; S.R. was responsible for methodology and writing – review and editing; R.P. was responsible for methodology; M.M.C. was responsible for conceptualization, methodology, and writing – review and editing; B.G. was responsible for methodology; G.W. was responsible for conceptualization; A.V. was responsible for conceptualization and writing – review and editing; J.P.P. was responsible for conceptualization and writing – review and editing; A.H. was responsible for conceptualization and writing – review and editing; JS was responsible for conceptualization and writing – review and editing; PJG was responsible for conceptualization, supervision and writing – review and editing; AJD was responsible for conceptualization, supervision and writing – review and editing; DKM was responsible for conceptualization; J.S. was responsible for funding, conceptualization, supervision, and writing – review and editing; P.V.D. was responsible for funding, resources, supervision, and writing – review and editing; and V.F.J.N. was responsible for funding, conceptualization, supervision, and writing – review and editing.

Data Sharing

Data access is conditional on an approved study proposal; there are no end dates to the availability. The CENTER-TBI data used in this study are available to researchers who provide a methodologically sound study proposal that is approved by the CENTER-TBI management committee to achieve the aims in the approved proposal. Proposals may be submitted online to CENTER-TBI (https://www.center-tbi.eu/data). A data access agreement is required, and all access must comply with regulatory restrictions imposed on the original study. No patient-identifiable information is made available, and all data have been anonymized. Study protocols and additional information for CENTER-TBI about data collection, recruitment, and participating centers are available online.

Funding Information

The CENTER-TBI study was supported by the European Union 7th Framework Programme (EC grant 602150), with additional project support from OneMind, the Hannelore Kohl Foundation, NeuroTrauma Sciences. and Integra Neurosciences. This project has received funding from the European Union's Horizon 2020 research and innovation programme under Marie Skłodowska-Curie grant agreement No 764513. Individual sources of funding were the Engineering and Physical Sciences Research Council (EP/R511547/1, to B.G.), a Wellcome Trust PhD Fellowship (S.R.), the Academy of Finland (Grant 17379 to J.P.P.), the National Institute for Health Research UK (D.K.M. and V.F.J.N), and the Academy of Medical Sciences/The Health Foundation (V.F.J.N.).

Author Disclosure Statement

No competing financial interests exist.

Supplementary Material

Supplementary Figure S1

Supplementary Table S1

Supplementary Table S2

Supplementary Author List

TRIPOD Checklist

References

Dewan

, Rattani

, Gupta

, et al. Estimating the global incidence of traumatic brain injury. J Neurosurg, 2018; 130(4):1080–1097; doi: 10.3171/2017.10.JNS17352.

Carroll

, Outtrim

, Forsyth

, et al. Mild traumatic brain injury recovery: a growth curve modelling analysis over 2 years. J Neurol, 2020; 267(11):3223–3234; doi: 10.1007/s00415-020-09979-x.

Steyerberg

, Wiegers

, Sewalt

, et al. Case-mix, care pathways, and outcomes in patients with traumatic brain injury in CENTER-TBI: a European prospective, multicentre, longitudinal, cohort study. Lancet Neurol, 2019; 18(10):923–934; doi: 10.1016/S1474-4422(19)30232-7.

Nelson

, Temkin

, Dikmen

, et al. Recovery after mild traumatic brain injury in patients presenting to US level i trauma centers: a Transforming Research and Clinical Knowledge in Traumatic Brain Injury (TRACK-TBI) study. JAMA Neurol, 2019; 76(9):1049–1059; doi: 10.1001/jamaneurol.2019.1313.

Richter

, Winzeck

, Kornaropoulos

, et al. Neuroanatomical substrates and symptoms associated with magnetic resonance imaging of patients with mild traumatic brain injury. JAMA Netw Open, 2021; 4(3):e210994; doi: 10.1001/jamanetworkopen.2021.0994.

Yuh

, Mukherjee

, Lingsma

, et al. Magnetic resonance imaging improves 3-month outcome prediction in mild traumatic brain injury. Ann Neurol, 2013; 73(2):224–235; doi: 10.1002/ana.23783.

Yue

, Winkler

, Puffer

, et al. Temporal lobe contusions on computed tomography are associated with impaired 6-month functional recovery after mild traumatic brain injury: a TRACK-TBI study. Neurol Res, 2018; 40(11):972–981; doi: 10.1080/01616412.2018.1505416.

Einarsen

, Moen

, Håberg

, et al. Patients with mild traumatic brain injury recruited from both hospital and primary care settings: a controlled longitudinal magnetic resonance imaging study. J Neurotrauma, 2019; 36(22):3172–3182; doi: 10.1089/neu.2018.6360.

Y-C

, Mustafi

, Harezlak

, et al. Hybrid diffusion imaging in mild traumatic brain injury. J Neurotrauma, 2018; 35(20):2377–2390; doi: 10.1089/neu.2017.5566.

10.

Yin

, Li

D-D

, Huang

, et al. Longitudinal changes in diffusion tensor imaging following mild traumatic brain injury and correlation with outcome. Front Neural Circuits, 2019; 13:28; doi: 10.3389/fncir.2019.00028.

11.

Wallace

, Mathias

, Ward

. Diffusion tensor imaging changes following mild, moderate and severe adult traumatic brain injury: a meta-analysis. Brain Imaging Behav, 2018; 12(6):1607–1621; doi: 10.1007/s11682-018-9823-2.

12.

Messé

, Caplain

, Paradot

, et al. Diffusion tensor imaging and white matter lesions at the subacute stage in mild traumatic brain injury with persistent neurobehavioral impairment. Hum Brain Mapp, 2011; 32(6):999–1011; doi: 10.1002/hbm.21092.

13.

Ling

, Peña

, Yeo

, et al. Biomarkers of increased diffusion anisotropy in semi-acute mild traumatic brain injury: a longitudinal perspective. Brain, 2012; 135(Pt 4):1281–1292; doi: 10.1093/brain/aws073.

14.

Lange

, Iverson

, Brubacher

, et al. Diffusion tensor imaging findings are not strongly associated with postconcussional disorder 2 months following mild traumatic brain injury. J Head Trauma Rehabil, 2012; 27(3):188–198; doi: 10.1097/HTR.0b013e318217f0ad.

15.

Yuh

, Cooper

, Mukherjee

, et al. Diffusion tensor imaging for outcome prediction in mild traumatic brain injury: a TRACK-TBI study. J Neurotrauma, 2014; 31(17):1457–1477; doi: 10.1089/neu.2013.3171.

16.

Borja

, Chung

, Lui

. Diffusion MR imaging in mild traumatic brain injury. Neuroimaging Clin N Am, 2018; 28(1):117–126; doi: 10.1016/j.nic.2017.09.009.

17.

Grassi

, Conceição

DM da

, Leite C da

, et al. Current contribution of diffusion tensor imaging in the evaluation of diffuse axonal injury. Arq Neuropsiquiatr, 2018; 76(3):189–199; doi: 10.1590/0004-282x20180007.

18.

Pinto

, Paolella

, Billiet

, et al. Harmonization of brain diffusion MRI: concepts and methods. Front Neurosci, 2020; 14:396; doi: 10.3389/fnins.2020.00396.

19.

Kumar

, Gupta

, Husain

, et al. Comparative evaluation of corpus callosum DTI metrics in acute mild and moderate traumatic brain injury: its correlation with neuropsychometric tests. Brain Inj, 2009; 23(7):675–685; doi: 10.1080/02699050903014915.

20.

Jenkins

, Chang

L-C

, Hutchinson

, et al. Harmonization of Methods to Facilitate Reproducibility in Medical Data Processing: Applications to Diffusion Tensor Magnetic Resonance Imaging. In: 2016 IEEE International Conference on Big Data. IEEE: Washington, DC, USA; 2016; pp. 3992–3994; doi: 10.1109/bigdata.2016.7841086.

21.

Jovicich

, Barkhof

, Babiloni

, et al. Harmonization of neuroimaging biomarkers for neurodegenerative diseases: a survey in the imaging community of perceived barriers and suggested actions. Alzheimers Dement, 2019; 11:69–73; doi: 10.1016/j.dadm.2018.11.005.

22.

Lingsma

, Yue

, Maas

AIR

, et al. Outcome prediction after mild and complicated mild traumatic brain injury: external validation of existing models and identification of new predictors using the TRACK-TBI pilot study. J Neurotrauma, 2015; 32(2):83–94; doi: 10.1089/neu.2014.3384.

23.

Mikolić

, Polinder

, Steyerberg

, et al. Prediction of global functional outcome and post-concussive symptoms after mild traumatic brain injury: external validation of prognostic models in the Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI) study. J Neurotrauma, 2021; 38(2):196–209; doi: 10.1089/neu.2020.7074.

24.

Kaplan

, Cheng

, Mohan

, et al. Mixture model framework for traumatic brain injury prognosis using heterogeneous clinical and outcome data. IEEE J Biomed Health Inform, 2022; 26(3):1285–1296; doi: 10.48550/ARXIV.2012.12310.

25.

Liu

, Salinas

. Machine learning for predicting outcomes in trauma. Shock, 2017; 48(5):504–510; doi: 10.1097/SHK.0000000000000898.

26.

Gravesteijn

, Nieboer

, Ercole

, et al. Machine learning algorithms performed no better than regression models for prognostication in traumatic brain injury. J Clin Epidemiol, 2020; 122:95–107; doi: 10.1016/j.jclinepi.2020.03.005.

27.

Cui

, Xia

, Su

, et al. Disrupted white matter connectivity underlying developmental dyslexia: a machine learning approach. Hum Brain Mapp, 2016; 37(4):1443–1458; doi: 10.1002/hbm.23112.

28.

Cao

, Cho

, Chen

, et al. Treatment response prediction and individualized identification of first-episode drug-naïve schizophrenia using brain functional connectivity. Mol Psychiatry, 2020; 25(4):906–913; doi: 10.1038/s41380-018-0106-5.

29.

Ten Kate

, Dicks

, Visser

, et al. Atrophy subtypes in prodromal Alzheimer's disease are associated with cognitive decline. Brain, 2018; 141(12):3443–3456; doi: 10.1093/brain/awy264.

30.

Sørensen

, Nielsen

. Alzheimer's disease neuroimaging initiative. Ensemble support vector machine classification of dementia using structural MRI and mini-mental state examination. J Neurosci Methods, 2018; 302:66–74; doi: 10.1016/j.jneumeth.2018.01.003.

31.

Zhang

, Gao

, He

, et al. Identifying Parkinson's disease with mild cognitive impairment by using combined MR imaging and electroencephalogram. Eur Radiol, 2021; 31(10):7386–7394; doi: 10.1007/s00330-020-07575-1.

32.

Vergara

, Mayer

, Kiehl

, et al. Dynamic functional network connectivity discriminates mild traumatic brain injury through machine learning. Neuroimage Clin, 2018; 19:30–37; doi: 10.1016/j.nicl.2018.03.017

33.

Harrington

, Hsu

P-Y

, Theilmann

, et al. Detection of chronic blast-related mild traumatic brain injury with diffusion tensor imaging and support vector machines. Diagnostics (Basel), 2022; 12(4);987; doi: 10.3390/diagnostics12040987.

34.

Bruschetta

, Tartarisco

, Lucca

, et al. Predicting outcome of traumatic brain injury: is machine learning the best way?. Biomedicines, 2022; 10(3):686; doi: 10.3390/biomedicines10030686.

35.

Puybasset

, Perlbarg

, Unrug

, et al. Prognostic value of global deep white matter DTI metrics for 1-year outcome prediction in ICU traumatic brain injury patients: an MRI-COMA and CENTER-TBI combined study. Intensive Care Med, 2022; 48(2):201–212; doi: 10.1007/s00134-021-06583-z.

36.

Maas

AIR

, Menon

, Adelson

, et al. Traumatic brain injury: integrated approaches to improve prevention, clinical care, and research. Lancet Neurol, 2017; 16(12):987–1048; doi: 10.1016/S1474-4422(17)30371-X.

37.

Collins

, Reitsma

, Altman

, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med, 2015; 13:1; doi: 10.1136/bmj.g7594.

38.

Wilson

, Pettigrew

, Teasdale

. Structured interviews for the Glasgow Outcome Scale and the extended Glasgow Outcome Scale: guidelines for their use. J Neurotrauma, 1998; 15(8):573–585; doi: 10.1089/neu.1998.15.573.

39.

Horton

, Rhodes

, Menon

, et al. Questionnaires vs. interviews for the assessment of global functional outcomes after traumatic brain injury. JAMA Netw Open, 2021; 4(11):e2134121; doi: 10.1001/jamanetworkopen.2021.34121.

40.

Vande Vyvere

, Wilms

, Claes

, et al. Central versus local radiological reading of acute computed tomography characteristics in multi-center traumatic brain injury research. J Neurotrauma, 2019; 36(7):1080–1092; doi: 10.1089/neu.2018.6061.

41.

Manjón

, Coupé

, Concha

, et al. Diffusion weighted image denoising using overcomplete local PCA. PLoS One, 2013; 8(9):e73021; doi: 10.1371/journal.pone.0073021.

42.

Veraart

, Novikov

, Christiaens

, et al. Denoising of diffusion MRI using random matrix theory. Neuroimage, 2016; 142:394–406; doi: 10.1016/j.neuroimage.2016.08.016.

43.

Veraart

, Fieremans

, Jelescu

, et al. Gibbs ringing in diffusion MRI. Magn Reson Med, 2016; 76(1):301–314; doi: 10.1002/mrm.25866.

44.

Andersson

JLR

, Sotiropoulos

. An integrated approach to correction for off-resonance effects and subject movement in diffusion MR imaging. Neuroimage, 2016; 125:1063–1078; doi: 10.1016/j.neuroimage.2015.10.019.

45.

Jeurissen

, Tournier

J-D

, Dhollander

, et al. Multi-tissue constrained spherical deconvolution for improved analysis of multi-shell diffusion MRI data. Neuroimage, 2014; 103:411–426; doi: 10.1016/j.neuroimage.2014.07.061.

46.

Jenkinson

, Beckmann

, Behrens

TEJ

, et al. FSL. Neuroimage, 2012; 62(2):782–790; doi: 10.1016/j.neuroimage.2011.09.015.

47.

Mori

, Oishi

, Jiang

, et al. Stereotaxic white matter atlas based on diffusion tensor imaging in an ICBM template. Neuroimage, 2008; 40(2):570–582; doi: 10.1016/j.neuroimage.2007.12.035.

48.

Avants

, Epstein

, Grossman

, et al. Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Med Image Anal, 2008; 12(1):26–41; doi: 10.1016/j.media.2007.06.004.

49.

Avants

, Tustison

, Song

, et al. A reproducible evaluation of ANTs similarity metric performance in brain image registration. Neuroimage, 2011; 54(3):2033–2044; doi: 10.1016/j.neuroimage.2010.09.025.

50.

Fortin

J-P

, Parker

, Tunç

, et al. Harmonization of multi-site diffusion tensor imaging data. Neuroimage, 2017; 161:149–170; doi: 10.1016/j.neuroimage.2017.08.047.

51.

Eikenes

, Visser

, Vangberg

, et al. Both brain size and biological sex contribute to variation in white matter microstructure in middle-aged healthy adults. Hum Brain Mapp, 2023; 44(2):691–709; doi: 10.1002/hbm.26093.

52.

Levin

, Temkin

, Barber

, et al. Association of sex and age with mild traumatic brain injury-related symptoms: a TRACK-TBI study. JAMA Netw Open, 2021; 4(4):e213046; doi: 10.1001/jamanetworkopen.2021.3046.

53.

Mikolic

, Groeniger

, Zeldovich

, et al. Explaining outcome differences between men and women following mild traumatic brain injury. J Neurotrauma, 2021; 38(23):3315–3331; doi: 10.1089/neu.2021.0116.

54.

Starkey

, Duffy

, Jones

, et al. Sex differences in outcomes from mild traumatic brain injury eight years post-injury. PLoS One, 2022; 17(5):e0269101; doi: 10.1371/journal.pone.0269101.

55.

Pedregosa

, Varoquaux

, Gramfort

, et al. Scikit-learn: machine learning in Python. J Mach Learn Res, 2011; 12:2825–2830; doi: 10.48550/ARXIV.1201.0490.

56.

Guyon

, Weston

, Barnhill

, et al. Gene selection for cancer classification using support vector machines. Mach Learn, 2002; 46(1/3):389–422; doi: 10.1023/A:1012487302797.

57.

Kim

H-Y

. Analysis of variance (ANOVA) comparing means of more than two groups. Restor Dent Endod, 2014; 39(1):74–77; doi: 10.5395/rde.2014.39.1.74.

58.

Benjamini

, Hochberg

. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc, 1995; 57(1):289–300; doi: 10.1111/j.2517-6161.1995.tb02031.x.

59.

DeLong

, DeLong

, Clarke-Pearson

. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 1988; 44(3):837–845; doi: 10.2307/2531595.

60.

Yue

, Levin

, Suen

, et al. Age and sex-mediated differences in six-month outcomes after mild traumatic brain injury in young adults: a TRACK-TBI study. Neurol Res, 2019; 41(7):609–623; doi: 10.1080/01616412.2019.1602312.

61.

Biegon

. Considering biological sex in traumatic brain injury. Front Neurol, 2021; 12:576366; doi: 10.3389/fneur.2021.576366.

62.

van der Naalt

, Timmerman

, de Koning

, et al. Early predictors of outcome after mild traumatic brain injury (UPFRONT): an observational cohort study. Lancet Neurol, 2017; 16(7):532–540; doi: 10.1016/S1474-4422(17)30117-5.

63.

Cole

, Jolly

, de Simoni

, et al. Spatial patterns of progressive brain volume loss after moderate-severe traumatic brain injury. Brain, 2018; 141(3):822–836; doi: 10.1093/brain/awx354.

64.

Graham

NSN

, Jolly

, Zimmerman

, et al. Diffuse axonal injury predicts neurodegeneration after moderate-severe traumatic brain injury. Brain, 2020; 143(12):3685–3698; doi: 10.1093/brain/awaa316.

65.

Tjerkaski

, Nyström

, Raj

, et al. Extended analysis of axonal injuries detected using magnetic resonance imaging in critically ill traumatic brain injury patients. J Neurotrauma, 2022; 39(1-2):58–66; doi: 10.1089/neu.2021.0159.

66.

Mayer

, Ling

, Mannell

, et al. A prospective diffusion tensor imaging study in mild traumatic brain injury. Neurology, 2010; 74(8):643–650; doi: 10.1212/WNL.0b013e3181d0ccdd.

67.

Croall

, Cowie

CJA

, He

, et al. White matter correlates of cognitive dysfunction after mild traumatic brain injury. Neurology, 2014; 83(6):494–501; doi: 10.1212/WNL.0000000000000666.

68.

Needham

, Helmy

, Zanier

, et al. The immunological response to traumatic brain injury. J Neuroimmunol, 2019; 332:112–125; doi: 10.1016/j.jneuroim.2019.04.005.

69.

Richter

, Winzeck

, Correia

, et al. Validation of cross-sectional and longitudinal ComBat harmonization methods for magnetic resonance imaging data on a travelling subject cohort. Neuroimage Rep, 2022; 2(4):100136; doi: 10.1016/j.ynirp.2022.100136

70.

Skandsen

, Stenberg

, Follestad

, et al. Personal factors associated with postconcussion symptoms 3 months after mild traumatic brain injury. Arch Phys Med Rehabil, 2021; 102(6):1102–1112; doi: 10.1016/j.apmr.2020.10.106.

71.

Korley

, Jain

, Sun

, et al. Prognostic value of day-of-injury plasma GFAP and UCH-L1 concentrations for predicting functional recovery after traumatic brain injury in patients from the US TRACK-TBI cohort: an observational cohort study. Lancet Neurol, 2022; 21(9):803–813; doi: 10.1016/S1474-4422(22)00256-3.

72.

Helmrich

IRAR

, Czeiter

, Amrein

, et al. Incremental prognostic value of acute serum biomarkers for functional outcome after traumatic brain injury (CENTER-TBI): an observational cohort study. Lancet Neurol, 2022; 21(9):792–802; doi: 10.1016/S1474-4422(22)00218-6.

73.

Wilson

, Horton

, Kunzmann

, et al. Understanding the relationship between cognitive performance and function in daily life after traumatic brain injury. J Neurol Neurosurg Psychiatry, 2020; doi: 10.1136/jnnp-2020-324492; doi: 10.1136/jnnp-2020-324492.[Epub ahead of print].

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.07 MB

0.01 MB

0.02 MB

0.08 MB