Graph Theory Analysis of Functional Connectivity Combined with Machine Learning Approaches Demonstrates Widespread Network Differences and Predicts Clinical Variables in Temporal Lobe Epilepsy

Abstract

Understanding how global brain networks are affected in epilepsy may elucidate the pathogenesis of seizures and its accompanying neurobehavioral comorbidities. We investigated functional changes within neural networks in temporal lobe epilepsy (TLE) using graph theory analysis of resting-state connectivity. Twenty-seven TLE presurgical patients (age 41.0 ± 12.3 years) and 85 age, gender, and handedness equivalent healthy controls (HCs; age 39.7 ± 16.9 years) were enrolled. Eyes-closed resting-state functional magnetic resonance image scans were analyzed to compare network properties and functional connectivity (FC) changes. TLE subjects showed significantly higher global efficiency, lower clustering coefficient ratio, and lower shortest path lengths ratio than HCs, as an indication of a more synchronized, yet less segregated network. A trend of functional reorganization with a shift of network hubs to the contralateral hemisphere was noted in TLE subjects. Support vector machine (SVM) with linear kernel was trained to separate between neural networks in TLE and HC subjects based on graph measurements. SVM analysis allowed separation between TLE and HC networks with 80.66% accuracy using eight features of graph measurements. Support vector regression (SVR) was used to predict neurocognitive performance from graph metrics. An SVR linear predictor showed discriminative prediction accuracy for four key neurocognitive variables in TLE (absolute R value range: 0.61–0.75). Despite TLE, our results showed both local and global network topology differences that reflect widespread alterations in FC in TLE. Network differences are discriminative between TLE and HCs using data-driven analysis and predicted severity of neurocognitive sequelae in our cohort.

Introduction

Long-term seizure freedom after surgical treatment of temporal lobe onset epilepsy (TLE) is <50% at 10 years (Spencer and Huh, 2008). A common reason for surgical failure is the inability to identify and resect epileptogenic neural connections that extend beyond the electrographic seizure onset zone (Harroud et al., 2012). Although hippocampal sclerosis is most commonly observed in TLE, magnetic resonance image (MRI)-based structural analysis has indicated widespread gray and white matter abnormalities that extend to temporolimbic and frontocentral regions, and to the contralateral side (Bernhardt et al., 2008; Concha et al., 2005; Keller et al., 2015; Seidenberg et al., 2005). These structural changes may negatively impact seizure outcomes (Keller et al., 2015) and also form the basis of cognitive impairment across multiple domains (Dabbs et al., 2009). Yet, we have an incomplete understanding of how focal neurotransmission abnormalities within the ictal onset zone impact global network topology and cognitive dysfunction, which have distributive network representation. Hence, there is a need to evaluate functional abnormalities within large-scale brain organization and identify system-wide changes in brain connectivity (Bernhardt et al., 2008; Concha et al., 2005) that extend beyond the seizure-onset zone as defined by semiology and electrographic features.

Neuroimaging the temporal covariance of blood-oxygen-level-dependent imaging (BOLD) signals in different brain regions, under resting conditions, can identify and characterize functional connectivity (FC) networks (Bullmore and Sporns, 2009; Guye et al., 2010). This approach provides an analytical framework for determining the potential role of neural network properties in the pathogenesis of epilepsy at ictal onset, during seizure propagation (He et al., 2009), and for prognostication of seizure outcomes (Bernhardt et al., 2010; Doucet et al., 2015; He et al., 2017; Munsell et al., 2015). Characterizing these aberrant network connectivity patterns could address current limitations in understanding disease heterogeneity between patients and longitudinal changes in the natural history of epilepsy.

Previous studies have utilized graph theory analysis of structural MRI-based features to identify network alterations (Bernhardt et al., 2011) and prognosticate surgical outcomes (Munsell et al., 2015) in epilepsy. Widespread alterations in neural network topology have also been evidenced in a right TLE cohort utilizing resting-state connectivity studies (Liao et al., 2010). However, there is limited understanding regarding the biological implications of the functional network connectivity dynamics observed in TLE. The role of network markers in discriminating epileptogenic from nonepileptogenic networks and improving detection of seizure onset zone remains to be investigated and validated in clinical cohorts.

Our study aim was to evaluate the extent of functional abnormalities, in whole-brain neural organization in TLE, using graph theory analysis of functional network properties derived from resting-state functional MRI (rs-fMRI) studies. Our study hypothesis was that epileptic activity is an emergent property of abnormal neural transmission within distributed neural networks, which extends from the seizure onset zone and involves distant brain regions. We further hypothesized that the distributive neural representation of the range of neurocognitive dysfunction, frequently observed in TLE, is indicative of a widespread FC disorder rather than a focal impairment of neural networks within the epileptogenic temporal lobe. We, therefore, focused on the application of functional network features in examining connectivity changes in temporal onset epileptic networks and prediction of neurocognitive dysfunction, a pervasive clinical feature in TLE.

Materials and Methods

Subject selection

We utilized an institutional database for adult (age >18 years) TLE subjects, diagnosed through standard clinical, neurophysiological, and imaging diagnostic criteria between the years 2011 and 2018. Twenty-seven TLE subjects with presurgical rs-fMRI scans were identified for the study cohort. Of these, 18 subjects had left-sided onset TLE (LTLE) and 9 had right-sided onset TLE (RTLE), demonstrated by continuous video electroencephalogram (EEG) monitoring of spontaneous seizures. Exclusion criteria included patients with a symptomatic etiology for seizures including prior traumatic brain injury, infections, suspected or confirmed autoimmune or inflammatory conditions, and concurrent neurodegenerative or neurological disorders. An institutional database of age, gender, and handedness-matched healthy controls (HCs; n = 85, 48 female) was used for statistical comparison. Study approval was obtained from the Health Sciences Institutional Review Board. Subject demographics are summarized in Table 1.

Table 1.

Clinical and Demographic Data for the Study Cohort

Sample group	RTLE (n = 9)	LTLE (n = 18)	Healthy controls (n = 85)	F/t/χ ²	p
Age, mean ± SD, years	39.1 ± 12.4	42.0 ± 12.0	39.7 ± 16.9	0.342	0.71
Gender (M/F), n	3/6	8/10	37/48	0.364	0.84
Age at epilepsy onset, mean ± SD, years	21.0 ± 11.4	20.1 ± 13.0	NA	0.042	0.85
Duration of epilepsy onset, mean ± SD, years	15.0 ± 8.2	21.0 ± 14.5	NA	1.544	0.23

For age and epilepsy duration, single factor analysis of variance was carried out and gender comparison between groups was carried out by chi-square test.

F, female; LTLE, left-sided onset temporal lobe epilepsy; M, male; NA, not applicable; RTLE, right-sided onset temporal lobe epilepsy; SD, standard deviation.

Neuropsychological assessment

Standardized neuropsychological tests were used for objective comparison between TLE and HCs, focusing on cognitive deficits in memory. The California Verbal Learning Test assessed rote verbal learning ability and delayed free recall. The dependent measures of interest were (1) verbal learning ability, defined as the total number of words recalled across trials 1–5, California Verbal Learning Test (CVLT) TT standard score (SS) and (2) long delayed free recall defined as the total number of words recalled after a 20-min delay, long delayed free recall CVLT standard score (CVLT LDFR SS) (Hermann et al., 1992). The Brief Visuospatial Memory Test—Revised (BVMT-R) was used to evaluate visuospatial memory. The BVMT-R learning total across trials 1–3 BVMT standard score (BVMT TT SS), and the 25-min delayed recall trial, BVMT delayed recall SS, were used as measures of visuospatial learning and memory, respectively.

MRI acquisition, data preprocessing, and network construction

T1-weighted structural imaging along with eyes closed 5-min rs-fMRI data was collected from all participants. MRI data were collected on 3 Tesla GE MR750 scanners equipped with high-speed gradients (Sigma GE Healthcare, Milwaukee, Wisconsin) using an eight-channel head coil. Five-minute rs-fMRI data were collected using a T2*-weighted gradient-echo planar imaging (EPI) pulse sequence sensitive to BOLD contrast. Technical parameters used to acquire these EPI scans were as follows: field of view 224 mm, matrix 64 × 64, TR 2600 ms, TE 22 ms, flip angle 60°, and 40 axial plane slices of 3.5 mm thickness with 3.5 mm spacing between slices. A T1-weighted high-resolution anatomical image was also obtained for each subject using a BRAVO FSPGR pulse sequence. Technical parameters used to acquire these scans are as follows: field of view 256 mm, matrix 256 × 256, TR 8.16 ms, TE 3.18 ms, flip angle 12°, and 156 axial plane slices of 1 mm thickness with 1 mm spacing between slices.

rs-fMRI data were processed using the AFNI package (Cox, 1996). Images were despiked, slice time corrected, motion corrected, aligned with the anatomical scan, normalized to Montreal Neurological Institute (MNI) space, resampled to 3.5 mm, and spatially smoothed with a 4 mm full width half maximum Gaussian kernel. Motion censoring (per TR motion >1 mm or 1°), nuisance regression, and band-pass filtering (0.009–0.08 Hz) were performed simultaneously in one regression model. Nuisance signals that were regressed out included six motion estimates and their temporal derivatives, the voxel-wise locally averaged white matter signal, and the cerebrospinal fluid signal. Finally, motion censoring of two groups was compared with each other to see whether two groups differ significantly in motion and motion censoring. First the Euclidean norm value of motion derivatives (first difference) in all directions for each TR time-point was averaged across all time-points for each patient within groups as the average motion censoring of each patient. These average motion censorings were then tested between two groups by two-sample t-test with null hypothesis that these values are equal between two groups with p value = 0.1178.

Graph theory analysis

Supplementary Figure S1 illustrates the algorithm for graph theory analysis applied to the fMRI data (Achard and Bullmore, 2007; Bernhardt et al., 2011; Bullmore and Sporns, 2009; Song et al., 2014; van Wijk et al., 2010). Regions of interest (ROI) within the network were first identified through automated anatomical labeling (AAL) percolations. AAL was originally designed with 90 ROIs that did not include the cerebellum. We included the cerebellum using a total of 100 ROIs as nodes for our network analysis (Supplementary Table S1) (Tzourio-Mazoyer et al., 2002). FC matrices between these ROIs were determined using temporal correlations among all ROIs' rs-fMRI time series. The absolute values (by converting negative FCs to positive) of these matrices were used as a representation of FC strength between different regions.

Next, graph thresholding was used to identify network edges. A minimum spanning tree (MST) method was first applied as a backbone network that connected all 100 nodes with 99 edges, to make all network completed and connected for each participant (Alexander-Bloch et al., 2010; Iyer et al., 2018; Song et al., 2014). Proportional thresholding was then applied to exclude weak or irrelevant FCs from graph analysis. These edges acquired from the proportional thresholding step were then added to MST of each patient, resulting in a series of connected networks with connection density (also known as network sparsity) ranging from 2% to 50% in increments of 2% at the whole-network level from which network topological properties were evaluated (Alexander-Bloch et al., 2010; Song et al., 2014), to allow a proper estimation of global parameters while minimizing the number of spurious edges in each network (He et al., 2017; Rutter et al., 2013; Vaessen et al., 2014). A threshold value (or network sparsity) in the context of proportional thresholding was defined as the number of correlations that were considered as connections in the final graph, divided by the number of all possible correlations within the correlation matrix (Achard et al., 2012). Finally, these matrices were binarized (i.e., each nonzero entry in the matrix was set to be 1) to get undirected binary matrices that represent a sparse, connected, and biologically meaningful graph for each patient. This thresholding approach would ensure that networks in all groups had the same number of edges, or wiring cost, and that the between-group differences reflected alterations in topological organization rather than differences in low-level correlations (Bernhardt et al., 2011).

For each connection density, four global property measurements were estimated: (1) global efficiency; (2) average shortest path length (L), to explore graph integration at the whole-brain level (Rubinov and Sporns, 2010); (3) global clustering (C), as a measure of graph segregation that reflects the degree to which nodes tend to cluster together (Watts and Strogatz, 1998); and (4) small-worldness (σ), to test clustering of neural networks in comparison with a random network. The clustering coefficient (C) is a measure of the degree to which nodes in a graph tend to cluster together (Watts and Strogatz, 1998). The characteristic path length (L) reflects the level of global integration in the network. A shortest path between two nodes A and B is the path between A and B with the smallest number of edges.

Global efficiency is a measure of network's capacity for parallel information transfer and integrated processing, (Bullmore and Sporns, 2012) and was calculated at each range of thresholds, as described earlier (Bernhardt et al., 2011).

Compared with random networks, small-world networks have similar characteristic path lengths, but higher characteristic clustering, that is γ = C/C _rand >1, while λ = L/L _rand <1 (Watts and Strogatz, 1998). The small-world index σ = γ/λ is thus >1 in small-world networks (Humphries and Gurney, 2008). C_rand and L_rand were defined as the mean clustering coefficient across 1000 randomly generated networks that had the same number of nodes, edges, and degree distribution as the real network (Sporns and Zwi, 2004).

Local metrics also evaluated for each group include measures of hubness: (1) degree centrality (DC; richness of connections); (2) betweenness centrality (BC; importance for mediating information between other nodes) (Rubinov and Sporns, 2010). A mapping of hub regions within each group was also performed. Hub regions were defined as those with a nodal BC that was 1 standard deviation above the mean nodal BC of all cortical regions.

DC is a measure of the number of neighbors of each node. Nodes with higher DC would have more functional connections with other parts of the network and is, therefore, assessed to be more involved in network communication. BC is an index for the nodal influence over information flow between all other nodes within the network. It is determined by counting the number of times that a shortest pathway between each two nodes in the network passes that node (Girvan and Newman, 2002). BC of each node was normalized by dividing it by the mean BC across all nodes. Brain Network Connective toolbox in MATLAB was used to calculate all these measurements.

Local metrics were calculated for each regional node—to identify the most important nodes during graph analysis (Brandes, 2001)—at a connection density of 17%. This threshold value is the lowest density that guaranteed the existence of all significant connections within the graphs of all TLE and HC patients (Bernhardt et al., 2011; He et al., 2007). For this step, statistical significance (α = 0.05) was determined after correction for multiple comparison for each patient's FC matrix using false discovery rate (FDR) (Benjamini and Hochberg, 1995), thus minimizing the number of false positive paths (Achard et al., 2012).

Classification analysis

We investigated whether the overall FC changes observed in our study cohort can be used to discriminate between epileptiform networks in TLE subjects from normal networks in HCs. A support vector machine (SVM) with a linear kernel classifier was utilized to test the predictive value of the connectome profile in discriminating between epileptiform networks in TLE subjects and baseline networks in HCs. Multivariate features consisting of clinical variables and regional and global connectivity characteristics were extracted from HCs and TLE subjects. The SVM was then trained using feature vectors consisting of (1) regional centrality measures of regional nodes, (2) global measurements of the whole network (global efficiency, clustering coefficient, and shortest path length), and (3) clinical variables (age, gender, handedness, and left/right hemispheric seizure onset; HC subjects were classified as none). To match the number of subjects in each group, 27 control patients were randomly chosen from the total HC cohort (n = 85). The model was then tested on the study data set. Performance of the SVM-based network classification was determined using fivefold cross-validation for accuracy testing (Mohanty et al., 2018). In brief, the data set is randomly divided into five groups. The classification model is trained using four folds/groups and is then validated on the fifth group. This is repeated till every group serves as the test set. The mean score for each round yields the performance metric. The spider toolbox in MATLAB was used for classification analysis.

We also analyzed the predictive ability of neural connectivity patterns in identifying neurocognitive dysfunction. The same feature vectors that were used for SVM analysis were used to train support vector regression (SVR) models with a linear kernel on standardized neurocognitive scores. Individual SVR models were used for two verbal (trials 1–5 CVLT standard score [CVLT T1–5 SS] and CVLT LDFR SS) and two visuospatial (BVMT TT SS and delayed recall trial, BVMT standard score [BVMT recall SS]) neuropsychological variables, respectively. Associations between each neurocognitive measure and graph measures were tested using a linear kernel SVR analysis with nonparametric permutation tests based on root mean squared error. To test accuracy of the data-driven model, a fivefold cross-validation approach was utilized (Mohanty et al., 2018).

Statistical analysis

The single factor analysis of variance was used for age comparison between groups and gender comparison was carried out by chi-square test. Differences in network parameters global efficiency C, L, γ, λ, and σ as well as local measurements were assessed separately between each TLE group (i.e., LTLE and RTLE) and controls using a nonparametric permutation test with 100,000 repetitions (Bullmore et al., 1999). Significances were thresholded at p < 0.05. For BC and DC measurements, p values were corrected for multiple comparison using FDR. Significant tests are marked with “*” in figures and tables. Tests with p values <0.07 were considered to “trend toward” significance and marked with “^+” in figures and tables.

Results

Clinical and demographic data for the study cohort (n = 27) are summarized in Table 1. There were no significant differences in baseline clinical characteristics between LTLE (n = 18/27) and RTLE (n = 9/27, 33%) subjects with respect to HCs (n = 85).

Comparison of functional network characteristics

Global network properties

Global network properties did not differ significantly between LTLE and RTLE subjects (Supplementary Fig. S2). Hence, these two groups were combined as TLE and compared with HCs. Shortest path length and clustering coefficient of functional networks in TLE subjects were significantly lower than HCs across different ranges of network sparsity (p < 0.05, Fig. 1A,B). Global efficiency was significantly higher in TLE subjects than in HCs (Fig. 1C). Brain networks in both groups followed the paradigm of small-world network, with small-world index, σ > 1 (Table 2). However, TLE subjects had network characteristics closer to the random network paradigm with a lower shortest path length (p value = 0.014) and a less clustered network (lower clustering coefficient, p value = 0.013, Table 2).

FIG. 1.

Comparison of global network properties between TLE subjects and HCs in (A) clustering coefficient, (B) average shortest path length, and (C) global efficiency as a function of network sparsity. Blue circle line shows difference between mean TLE and HCs for each measurement at each specific network sparsity. Although Δ = 0 indicates no difference, Δ > 0/Δ < 0 indicates an increase/decrease in the network density of TLE subjects relative to HCs. The blue dash line shows 95% confidence interval of the null distribution of number between group difference obtained from 100,000 permutation tests at each density value. Asterisks (red) indicate significant between-group differences (p < 0.05). HCs, healthy controls; TLE, temporal lobe epilepsy.

Table 2.

Comparison of Global Properties and Small-Worldness Index Between Temporal Lobe Epilepsy and Healthy Control Subjects

Parameter	Average control	Average TLE	p
Shortest path length, L	1.71	1.52	0.014^*
Characteristic path lengths, λ	1.09	1.07	0.051⁺
Clustering coefficient, C	0.48	0.43	0.013^*
Characteristic clustering coefficient, γ	1.65	1.69	0.052⁺
Small-world coefficient, σ	1.51	1.57	0.075

Results reported at 17% network density (the minimal threshold at which all functional connections were significant after correction for multiple comparison).

L, shortest path length; C, clustering coefficient; λ, γ, and σ are small-world parameters. Note that γ > 1, λ ≈ 1, and σ > 1 in the small-world network.

Statistical significance.

p value <0.07 trends toward significance.

TLE, temporal lobe epilepsy.

Hub distribution and regional centrality

Regional hubness analysis showed that network hubs (nodes with highest BC values) in HCs were located in frontal (n = 5) and cerebellar (n = 4) regions with bilateral representation within these regions (Table 3). Overall, nine hubs were found within the left, and three hubs in right hemisphere. However, network hubs in RTLE were mainly located contralateral to the epileptic hemisphere (12 of 13 hubs). In comparison, 6 of 15 hubs were located within the contralateral hemisphere in LTLE. Overall, the mean normalized BC for all hubs in HCs was significantly higher than LTLE (Wilcoxon rank sum test, p = 0.0007) and was higher than in RTLE subjects (p = 0.0203; see Supplementary Fig. S3 for brain diagram representation of network hub distribution).

Table 3.

Distribution of Hub Regions in All Groups

Group	Brain region	Mean BC	Mean DC
Control
	L_Olfact	5.034	48.753
	L_Amyg	3.327	24.729
	L_Hesch	3.109	44.753
	L_Cereb1	2.379	10.329
	L_Pallid	2.328	22.518
	R_Cereb1	2.296	8.965
	L_SupOrbF	2.230	38.941
	L_Cereb6	2.191	7.588
	R_Olfact	2.119	32.494
	L_Precent	2.044	41.729
	R_Parac	1.872	39.765
	L_Cereb7	1.803	5.282
RTLE
	L_Olfact	3.814	45.111
	L_Amyg	2.704	23.222
	L_Cereb6	2.478	13.444
	L_Precent	2.463	37.778
	L_SMA	2.275	39.333
	L_Rectal	2.088	34.556
	L_PoleT	1.898	19.000
	L_Hesch	1.802	35.444
	L_SupOrbF	1.663	32.111
	L_MidCing	1.654	38.333
	L_ParsOp	1.650	31.667
	R_Olfact	1.590	33.667
	L_Parahhip	1.583	16.556
LTLE
	L_SupOrbF	2.844	36.222
	L_Olfact	2.735	36.722
	L_Amyg	1.964	18.222
	R_Fusif	1.963	27.000
	L_Hesch	1.811	32.167
	R_Cereb2	1.768	10.222
	L_MidT	1.745	29.389
	L_Precent	1.743	35.111
	R_Parahipp	1.682	9.222
	R_PoleT	1.677	20.000
	R_Cereb5	1.561	11.556
	L_Cereb5	1.561	12.444
	L_MedOrbF	1.542	28.222
	L_Pallid	1.495	16.056
	R_MidCing	1.491	29.556
	L_Rectal	1.479	31.556

List of nodes and their abbreviation can be found in Supplementary Table S1.

Hub regions were defined as those with a nodal BC that was 1 SD above the mean nodal BC of all cortical regions.

BC, betweenness centrality; DC, degree centrality.

Network hubs were localized to the left hemisphere in 75% of HC subjects. In comparison, 92% of the network hubs in RTLE subjects were localized to the contralateral left side. For LTLE subjects, 38% of the network hubs were localized to the contralateral right side. To examine the relative contributions of various brain regions toward global neural networks, group comparisons were made between TLE and HC subjects through regional centrality analysis using degree and BC measures. Regional network comparison of LTLE with HCs (Fig. 2A) showed increased DC in the contralateral middle frontal, mesial temporal (hippocampus, parahippocampus, and amygdala), caudate and thalamus, and bilateral putamen regions. Significantly lower DC was noted in ipsilateral frontal and temporal and contralateral paracentral regions. BC analysis showed a similar pattern with increased BC in ipsilateral temporal and trends toward significance in contralateral insular and mesial temporal regions (Fig. 2B). Decreased BC was observed within ipsilateral frontal regions.

FIG. 2.

Group differences between LTLE and HC cohorts for (A) DC and (B) BC. Blue dots represent the difference between the mean LTLE and mean HC measurement value at each specific network sparsity. Although Δ = 0 indicates no difference, Δ > 0/Δ < 0 indicates an increase/decrease in the network property for LTLE subjects relative to HCs. Red lines represent 95% confidence intervals for the null distribution of no between-group differences obtained from 100,000 permutation tests at each density value. Asterisk (*) symbols indicate significant between-group differences (p < 0.05). Plus (+) symbols indicate trend toward significance for between-group differences (p < 0.07). Corrected for multiple comparisons using FDR. BC, betweenness centrality; DC, degree centrality; FDR, false discovery rate; LTLE, left-sided onset TLE.

Regional network comparison for RTLE was relatively consistent with LTLE (Fig. 3A,B). In comparison with HCs, a trend toward significant increase in DC was observed in the bilateral parahippocampus (p = 0.05) and cerebellar (p = 0.056) regions. A trend toward a significant decrease was observed in bilateral paracentral regions (p = 0.065). A trend toward a significant increase for BC for contralateral midtemporal and ipsilateral amygdala was also observed.

FIG. 3.

Group differences between RTLE and HC cohorts for (A) DC and (B) BC. Black dots represent the difference between the mean RTLE and mean HC measurement value at each specific network sparsity. Although Δ = 0 indicates no difference, Δ > 0/Δ < 0 indicates an increase/decrease in the network property for RTLE subjects relative to HCs. Red lines represent 95% confidence intervals for the null distribution of no between-group differences obtained from 100,000 permutation tests at each density value. Asterisk (*) symbols indicate significant between-group differences (p < 0.05). Plus (+) symbols indicate trend toward significance for between-group differences (p < 0.07). Corrected for multiple comparisons using FDR. RTLE, right-sided onset TLE.

Classification analysis

We next investigated whether FC patterns can be used to discriminate between epileptiform networks in TLE and baseline networks in control subjects. We utilized a classification approach based on SVM analysis. A classifier was first estimated utilizing multivariate features including clinical demographics and rs-FMRI–based FC indices. SVM-based network prediction was able to discriminate between neural networks in TLE and HCs with an accuracy of 80.66% (tuning parameters C = 0.02 and epsilon = 1; sensitivity = 82.50% and specificity = 81.67%), using eight feature vectors. Seizure laterality was not significant for discriminating neural networks between TLE and HC cohorts.

Prediction of clinical variables

To investigate clinical implications of the composite FC changes observed in TLE, an SVR linear predictor was applied to predict four key neurocognitive variables (Fig. 4). Highest prediction accuracy was reached utilizing 11 discriminating graph measurements for total verbal learning (CVLT T1–5 SS; Pearson correlation coefficient R = −0.61; normalized mean square error [NMSE] = −0.62); and 10 graph measurements for delayed verbal memory (CVLT LDFR SS; R = −0.61; NMSE = −0.62). Highest prediction accuracy for visuospatial memory measures was reached utilizing 5 graph measurements for total visual learning (BVMT TT SS; R = 0.75; NMSE = 0.23) and 10 graph measurements for delayed visual recall (BVMT recall SS; R = −0.64; NMSE = −0.42). In comparison with null hypothesis of no correlation between SVR linear predictor and neurocognitive measures, the SVR prediction accuracy was significant for all neurocognitive variables, with R values ranging from 0.624 to 0.761 (Fig. 4; see Supplementary Fig. S4 for brain diagram representation of ROI distribution).

FIG. 4.

Predictive ability of neural connectivity patterns in identifying neurocognitive dysfunction. An SVR model with a linear kernel was trained using neural connectivity-based graph measurements to predict verbal (CVLT T1–5 SS, top left and CVLT LDFR SS, top right) and visuospatial (BVMT TT SS, bottom left and BVMT recall SS, bottom right) neuropsychological variables, respectively. Circles indicate each subject's coordinate of actual (X-axis) and predicted (Y-axis) neurocognitive variable. Solid lines represent the best-fit line as predicted by SVR. Purple lines represent 95% prediction accuracy intervals. BVMT, Brief Visuospatial Memory Test; BVMT recall SS, delayed recall trial, BVMT standard score; BVMT TT SS, trials 1–3 BVMT standard score; CVLT, California Verbal Learning Test; CVLT LDFR SS, long delayed free recall CVLT standard score; CVLT T1–5 SS, trials 1–5 CVLT standard score; SS, standard score; SVR, support vector regression.

Table 4 lists features identified in all five folds of cross-validation for prediction of neurocognitive variables. Centrality measures for the right paracentral lobule and right calcarine gyrus correlated with verbal memory performance (CVLT) in the SVR model. Similarly, visual–spatial memory performance (BVMT) showed correlation with FC indices within the left paracentral lobule, right angular gyrus, and right basal frontal regions.

Table 4.

Graph Features Common in All Five Folds of Cross-Validation in Training the Support Vector Regression Model for Prediction of Neurocognitive Variables

Clinical variable	Region name	Graph measurement	Average weight of all folds
CVLT T1-5 SS
	R_Parac	BC	0.3765
CVLT LDFR SS
	R_Calcar	BC	−0.7813
BVMT TT SS
	L_Parac	DC	0.8313
	R_Ang	DC	0.7903
BVMT recall SS
	R_MidOrbF	DC	0.5571
	R_SupOrbF	BC	0.5283
	R_Precent	BC	−0.7162

CVLT T1–5 SS, The California Verbal Learning Test-II assesses learning trial 1, CVLT T1–5 SS, and Long Delay Free Recall CVLT LDFR SS; the Brief Visuospatial Memory Test BVMT TT SS, and BVMT recall SS. List of nodes and their abbreviations are listed in Supplementary Table S1.

BVMT, Brief Visuospatial Memory Test; BVMT recall SS, delayed recall trial, BVMT standard score; BVMT TT SS, trials 1–3 BVMT standard score; SS, standard score.

Discussion

Our study investigated organization of functional neural networks at the global and regional level in TLE through graph theory analysis of rs-FC, using a network model. Study results demonstrate (1) widespread differences in global topological properties, notably decreased segregation and hub reorganization, and the ability of FC metrics to (2) discriminate between epileptiform networks from HCs, and (3) predict clinical variables like neurocognitive performance for epilepsy subjects. The observed alterations in macroscopic whole brain network organization in TLE support our central hypothesis that epilepsy is a neural network disorder, defined by altered network organization and connectivity, which extends beyond the traditionally defined seizure onset zone, based on semiology and electrographic characteristics. Our results demonstrate the contribution of FC features in assessment of neurocognitive performance for TLE subjects. Data-driven analysis using a classification approach with neuroimaging indices suggests a role for incorporating neural network features in diagnosis and stratification of disease variables and outcomes.

Graph theory analysis of neural networks in TLE

Our neural network analysis in TLE indicated that the characteristic shortest path length (λ) was significantly decreased at most thresholds (K). Reduced shortest path lengths may allow faster and more efficient transmission between different brain regions (Achard et al., 2012; Asadi-Pooya et al., 2016; Iyer et al., 2018; van Wijk et al., 2010). The resultant rapid information flow within neural networks may be the pathogenic basis for sustaining pathological networks responsible for seizure propagation from the epilepsy onset zone in TLE. For most thresholds, the absolute clustering coefficients (Cnet) were significantly lower in TLE, implying relatively sparse local connectedness and a less segregated network structure. Similar local connectivity changes have been observed in random networks (Alexander-Bloch et al., 2010). A combination of decreased path length and clustering has been shown in mesial TLE (Liao et al., 2010). These random network characteristics may, therefore, have an adverse effect on directed synchronized information flow and processing in TLE, which may, in turn, impact cognitive functions.

Reorganization of network hubs

Basal FC in HCs was asymmetrically organized with left predominance—network hubs were localized to the left hemisphere in 75% of HCs. This is consistent with previous demonstration of asymmetric fiber density within the left temporal lobe and may underlie the basis of language lateralization (Nucifora et al., 2005).

In our RTLE cohort, network hubs were predominantly (92%) located with the contralateral left hemisphere, spared by seizures. This may reflect a functional compensatory mechanism for limiting the detrimental effect of seizures on neurological function.

In contrast, distribution of networks hubs to the contralateral unaffected right hemisphere was 62% in LTLE subjects. This observation may be explained by the baseline asymmetric fiber density and increased baseline connectivity within the left hemisphere (as seen in HCs) that may conversely limit the contralateral shift of network function in left hemisphere onset TLE.

These results indicate that (1) functional network connectivity changes in TLE may reflect compensatory mechanisms of increased contralateral shift in network hubs and (2) a seizure laterality-specific susceptibility of neural networks to undergo these functional changes.

Regional centrality comparison identifies the relative contributions of individual brain regions toward global neural networks. The contralateral (right-sided) regions with high DC thus identified in the LTLE cohort were located within the mesial temporal, frontal, and basal ganglia (caudate and putamen). Whether these network differences are compensatory as a consequence of aberrant network organization or a reflection of a more pervasive network disorder remains to be determined.

Cerebellar hubs were also demonstrable in TLE subjects (Figs. 2 and 3). Through synaptic relays via the thalamus (Bostan et al., 2010), the cerebellum has been hypothesized to affect the wider cortical and subcortical networks that may be pervasively disrupted or reorganized in TLE. The cerebellar inhibitory output has also been postulated to influence seizure propagation and control (Englot et al., 2016).

Our graph theory analysis, therefore, identified alterations in topological features that provide insight into the functional architecture of neural networks in TLE. These network alterations suggest a mechanism for altered excitability that may impact the pathogenesis and progression of epilepsy. Reorganization of neural information transfer may have potential clinical implications for seizure treatment by informing improved target selection for treatments like neuromodulation.

Classification analysis of neural networks in TLE

Standard clinical workflows in epilepsy lack a practical framework for incorporating neural network features in clinical decision-making. An important unanswered question in connectome analysis is the identification of connectivity changes that are significantly associated with epileptiform networks (as in TLE) versus baseline FC changes in healthy subjects. We, therefore, investigated whether rs-fMRI–based FC patterns can be used to discriminate between epileptiform networks in TLE and baseline networks in control subjects.

We used a machine learning algorithm to assess the role of neural network features in (1) differentiating connectivity patterns in epileptogenic TLE networks versus HCs through classification analysis and (2) prognosticating behavioral variables using the SVR approach.

Clinical variables and FC indices for network nodes were incorporated into a classifier model using SVM analysis. The SVM-based network classification was trained and validated on the study cohort using fivefold cross-validation. Eight feature vectors were identified in the model that discriminated epileptiform TLE networks from nonepileptiform HC networks with an accuracy of 80%. Our results, therefore, indicate that fMRI features can be used to identify epileptiform networks in epilepsy subjects with reasonable accuracy. The combination of features that were associated with highest accuracy varied in each fold of cross-validation. A larger study cohort with separate training and validation data sets may allow improved classification accuracy using different feature combinations. The evolving role of FC indices as neuroimaging markers may improve the diagnostic yield of noninvasive evaluations of epilepsy subjects and potentially reduce the need for invasive diagnostic interventions.

Various methods of classification analysis have been reported in the literature (Antel et al., 2002). Seed-based connectivity analyses require an a priori hypothesis based on specific ROIs. Results from seed-based FC analyses are, therefore, prone to selection bias, are specific to selected seed regions, and do not provide information on whole brain network topology. An SVM approach is aptly suited for simultaneous analysis of a large number of features and yields a prediction model based on significant features that enable discrimination between two comparison groups.

FC models for prediction of clinical variables

We hypothesized that the range of neurocognitive dysfunction frequently observed in TLE (Lin et al., 2012) has an underlying distributive neural representation, which is indicative of a widespread FC disorder rather than focal impairment within the epileptogenic temporal lobe. Prior investigations of the neurobiological substrate of verbal and visual memory dysfunction in TLE have focused on regional and global structural anomalies or FC changes (Bonilha et al., 2007; Doucet et al., 2016; Englot et al., 2016). Memory impairment, a pervasive clinically debilitating consequence of TLE has been less often examined using resting state connectivity metrics, particularly by analysis of graph theory metrics. We, therefore, examined the role of FC in assessing the secondary impact of epilepsy on neurocognitive functions. A data modeling approach (SVR) was utilized, derived from characteristics of neural network topology, to establish a predictive model for neurobehavioral end-points. These end-points focused on verbal and nonverbal memory dysfunction, a significant component of neuropsychological detriment associated with TLE (Lin et al., 2012). Discriminative prediction accuracy of SVR for each cognitive outcome was reasonable with R value ranging from 0.61 to 0.75 (Fig. 4).

Centrality measures for nodal regions within the right hemisphere correlated with verbal memory performance. Implication of right hemispheric nodes in verbal memory may reflect their compensatory involvement in language processing more typically associated with the left hemisphere. These connectivity changes may reflect a broader potentially adaptive reorganization of neural networks (Bonilha et al., 2007). Consistent with this, our results also indicated a trend toward functional reorganization within TLE networks, with a shift of network hubs to the contralateral hemisphere. The SVR model also predicted visual–spatial memory performance through FC changes within the right frontal (orbital), angular gyrus, and left paracentral regions. In line with these results, visual memory tasks have previously been demonstrated to activate a broad frontal parietal network along with subcortical regions such as the pulvinar and cerebellum (Pessoa and Ungerleider, 2004). Whether the association between memory impairment and FC is causal or associative remains to be determined. Interestingly, these broader widespread changes in neural connectivity challenge our traditional understanding of lateralized temporal lobe functions with verbal memory impairments in left- and visual memory impairment in right onset TLE.

Our results, therefore, support the application of neuroimaging parameters based on functional neural connectivity measures, to discriminate epileptiform networks, and prognosticate disease variables and outcomes. We demonstrate the application of data modeling, derived from characteristics of neural network topology, in clinical decision-making. As with all mathematical models, the efficacy of the data model depends upon data sources (network variables and analytical approach) and the model assumptions. Further validation of classification models with prospective patient data set studies will aid clinical application.

Study limitations

Our analysis should be interpreted with the following limitations. First, a cross-sectional determination of connectivity changes does not take into account the dynamic alterations between regions of the brain that can be significantly modified from interictal to ictal and postictal states (Guye et al., 2006). BOLD signal measurements and FC analysis may be affected by interictal epileptiform discharges that may be assessed through simultaneous EEG-fMRI recordings (Bullmore and Sporns, 2009; He et al., 2017; Sporns and Zwi, 2004; Watts and Strogatz, 1998). Second, physiological noise in the form of respiratory and cardiac fluctuations may confound analysis of fMRI time series, by reducing the specificity of low-frequency fluctuations to functional connected regions (Bullmore et al., 1999). Third, statistical comparison of FC was accomplished by using two-sample two-tailed test with FDR corrected for multiple comparisons, which is a less stringent approach and can introduce statistical bias. Fourth, a mixed patient sample with differing durations of epilepsy, medication burden, and varying natural history was used in this study. The sample size did not allow for stratification on all these measures that may have a variant effect on performance and localization of neurocognitive functions (Bonilha et al., 2006). There is scant consensus in the literature on the relationship between these clinical factors and the observed functional and structural connectivity changes. Alternative functional parcellation approaches may further elucidate reorganization of functional neural networks and will be the focus of future work (Schaefer et al., 2018).

We determined feature selection for the SVR prediction model by optimizing maximum accuracy. Hence, identification of significant nodal regions (for neurocognitive score prediction) can be limited by this computational approach. A small sample size also precludes interpretation of machine learning analyses. Finally, whether these changes are a consequence of the underlying epileptiform disorder or an intrinsic aberrant network structure that leads to epilepsy cannot be determined by our study. A longitudinal approach with serial imaging assessments, a focus of our future studies, hopes to answer this question.

Conclusions

A graph theoretical approach for investigating neural network differences in TLE is presented. Our results showed both local and global topology changes reflect underlying alterations in neural network connectivity in TLE. These changes in functional network characteristics are discriminative between epilepsy subjects and HCs. In addition, we show the utility of data-driven machine learning approaches to predict the severity of key neurobehavioral variables. Our study demonstrates that graph properties of the network derived from fMRI data can be used to (1) characterize local and global neural network differences, (2) differentiate network topology in TLE, and (3) have prognostic potential in predicting epilepsy-related cognitive comorbidities.

Footnotes

Acknowledgments

The authors thank all the study subjects and their families for participating in the study, and all the magnetic resonance technologists for their support in collection of the data.

Author Disclosure Statement

No competing financial interests exist. The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Funding Information

Research reported in this publication was supported by National Institute of Neurological Disorders and Stroke of the National Institutes of Health under award no. NIH R01NS105646.

Supplementary Material

Supplementary Figure S1

Supplementary Figure S2

Supplementary Figure S3

Supplementary Figure S4

Supplementary Table S1

References

Achard

, Bullmore

. 2007. Efficiency and cost of economical brain functional networks. PLoS Comput Biol, 3:e17.

Achard

, Delon-Martin

, Vertes

, Renard

, Schenck

, Schneider

, et al. 2012. Hubs of brain functional networks are radically reorganized in comatose patients. Proc Natl Acad Sci U S A, 109:20608–20613.

Alexander-Bloch

, Gogtay

, Meunier

, Birn

, Clasen

, Lalonde

, et al. 2010. Disrupted modularity and local connectivity of brain functional networks in childhood-onset schizophrenia. Front Syst Neurosci, 4:147.

Antel

, Li

, Cendes

, Collins

, Kearney

, Shinghal

, Arnold

. 2002. Predicting surgical outcome in temporal lobe epilepsy patients using MRI and MRSI. Neurology, 58:1505–1512.

Asadi-Pooya

, Nei

, Sharan

, Sperling

. 2016. Historical risk factors associated with seizure outcome after surgery for drug-resistant mesial temporal lobe epilepsy. World Neurosurg, 89:78–83.

Benjamini

, Hochberg

. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Ser B Methodol, 57:289–300.

Bernhardt

, Bernasconi

, Concha

, Bernasconi

. 2010. Cortical thickness analysis in temporal lobe epilepsy: reproducibility and relation to outcome. Neurology, 74:1776–1784.

Bernhardt

, Chen

, He

, Evans

, Bernasconi

. 2011. Graph-theoretical analysis reveals disrupted small-world organization of cortical thickness correlation networks in temporal lobe epilepsy. Cereb Cortex, 21:2147–2157.

Bernhardt

, Worsley

, Besson

, Concha

, Lerch

, Evans

, Bernasconi

. 2008. Mapping limbic network organization in temporal lobe epilepsy using morphometric correlations: insights on the relation between mesiotemporal connectivity and cortical atrophy. Neuroimage, 42:515–524.

10.

Bonilha

, Rorden

, Appenzeller

, Coan

, Cendes

, Li

. 2006. Gray matter atrophy associated with duration of temporal lobe epilepsy. Neuroimage, 32:1070–1079.

11.

Bonilha

, Rorden

, Halford

, Eckert

, Appenzeller

, Cendes

, Li

. 2007. Asymmetrical extra-hippocampal grey matter loss related to hippocampal atrophy in patients with medial temporal lobe epilepsy. J Neurol Neurosurg Psychiatry, 78:286–294.

12.

Bostan

, Dum

, Strick

. 2010. The basal ganglia communicate with the cerebellum. Proc Natl Acad Sci U S A, 107:8452–8456.

13.

Brandes

. 2001. A faster algorithm for betweenness centrality. J Math Sociol, 25:163–177.

14.

Bullmore

, Sporns

. 2009. Complex brain networks: graph theoretical analysis of structural and functional systems. Nat Rev Neurosci, 10:186–198.

15.

Bullmore

, Sporns

. 2012. The economy of brain network organization. Nat Rev Neurosci, 13:336–349.

16.

Bullmore

, Suckling

, Overmeyer

, Rabe-Hesketh

, Taylor

, Brammer

. 1999. Global, voxel, and cluster tests, by theory and permutation, for a difference between two groups of structural MR images of the brain. IEEE Trans Med Imaging, 18:32–42.

17.

Concha

, Beaulieu

, Gross

. 2005. Bilateral limbic diffusion abnormalities in unilateral temporal lobe epilepsy. Ann Neurol, 57:188–196.

18.

Cox

. 1996. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput Biomed Res, 29:162–173.

19.

Dabbs

, Jones

, Seidenberg

, Hermann

. 2009. Neuroanatomical correlates of cognitive phenotypes in temporal lobe epilepsy. Epilepsy Behav, 15:445–451.

20.

De Vico Fallani

, Richiardi

, Chavez

, Achard

. 2014. Graph analysis of functional brain networks: practical issues in translational neuroscience. Philos Trans R Soc Lond B Biol Sci, 369:20130521.

21.

Doucet

, He

, Sperling

, Sharan

, Tracy

. 2015. Frontal gray matter abnormalities predict seizure outcome in refractory temporal lobe epilepsy patients. Neuroimage Clin, 9:458–466.

22.

Doucet

, He

, Sperling

, Sharan

, Tracy

. 2016. Gray matter abnormalities in temporal lobe epilepsy: relationships with resting-state functional connectivity and episodic memory performance. PLoS One, 11:e0154660.

23.

Englot

, Konrad

, Morgan

. 2016. Regional and global connectivity disturbances in focal epilepsy, related neurocognitive sequelae, and potential mechanistic underpinnings. Epilepsia, 57:1546–1557.

24.

Girvan

, Newman

MEJ

. 2002. Community structure in social and biological networks. Proc Natl Acad Sci U S A, 99:7821.

25.

Guye

, Bettus

, Bartolomei

, Cozzone

. 2010. Graph theoretical analysis of structural and functional connectivity MRI in normal and pathological brain networks. MAGMA, 23:409–421.

26.

Guye

, Regis

, Tamura

, Wendling

, McGonigal

, Chauvel

, Bartolomei

. 2006. The role of corticothalamic coupling in human temporal lobe epilepsy. Brain, 129:1917–1928.

27.

Harroud

, Bouthillier

, Weil

, Nguyen

. 2012. Temporal lobe epilepsy surgery failures: a review. Epilepsy Res Treat, 2012:201651.

28.

, Doucet

, Pustina

, Sperling

, Sharan

, Tracy

. 2017. Presurgical thalamic “hubness” predicts surgical outcome in temporal lobe epilepsy. Neurology, 88:2285–2293.

29.

, Chen

, Evans

. 2007. Small-world anatomical networks in the human brain revealed by cortical thickness from MRI. Cereb Cortex, 17:2407–2419.

30.

, Wang

, Chen

, Yan

, Yang

, et al. 2009. Uncovering intrinsic modular organization of spontaneous brain activity in humans. PLoS One, 4:e5226.

31.

Hermann

, Wyler

, Bush

, Tabatabai

. 1992. Differential effects of left and right anterior temporal lobectomy on verbal learning and memory performance. Epilepsia, 33:289–297.

32.

Humphries

, Gurney

. 2008. Network ‘small-world-ness’: a quantitative method for determining canonical network equivalence. PLoS One, 3:e0002051.

33.

Iyer

, Mathis

, Ustine

, Nair

, Rozman

, McMillan

, et al. 2018. Altered frontal lobe network function in temporal lobe epilepsy revealed by graph theory analysis (P1.279). Neurology. 90.

34.

Keller

, Richardson

, O'Muircheartaigh

, Schoene-Bake

, Elger

, Weber

. 2015. Morphometric MRI alterations and postoperative seizure control in refractory temporal lobe epilepsy. Hum Brain Mapp, 36:1637–1647.

35.

Liao

, Zhang

, Pan

, Mantini

, Ding

, Duan

, et al. 2010. Altered functional connectivity and small-world in mesial temporal lobe epilepsy. PLoS One, 5:e8525.

36.

Lin

, Mula

, Hermann

. 2012. Uncovering the neurobehavioural comorbidities of epilepsy over the lifespan. Lancet, 380:1180–1192.

37.

Mohanty

, Sinha

, Remsik

, Dodd

, Young

, Jacobson

, et al. 2018. Machine learning classification to identify the stage of brain-computer interface therapy for stroke rehabilitation using functional connectivity. Front Neurosci, 12:353.

38.

Munsell

, Wee

, Keller

, Weber

, Elger

, da Silva

, et al. 2015. Evaluation of machine learning algorithms for treatment outcome prediction in patients with epilepsy based on structural connectome data. Neuroimage, 118:219–230.

39.

Nucifora

, Verma

, Melhem

, Gur

. 2005. Leftward asymmetry in relative fiber density of the arcuate fasciculus. Neuroreport, 16:791–794.

40.

Pessoa

, Ungerleider

. 2004. Neural correlates of change detection and change blindness in a working memory task. Cereb Cortex, 14:511–520.

41.

Rubinov

, Sporns

. 2010. Complex network measures of brain connectivity: uses and interpretations. Neuroimage, 52:1059–1069.

42.

Rutter

, Nadar

, Holroyd

, Carver

, Apud

, Weinberger

, Coppola

. 2013. Graph theoretical analysis of resting magnetoencephalographic functional connectivity networks. Front Comput Neurosci, 7:93.

43.

Schaefer

, Kong

, Gordon

, Laumann

, Zuo

, Holmes

, et al. 2018. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI. Cereb Cortex, 28:3095–3114.

44.

Seidenberg

, Kelly

, Parrish

, Geary

, Dow

, Rutecki

, Hermann

. 2005. Ipsilateral and contralateral MRI volumetric abnormalities in chronic unilateral temporal lobe epilepsy and their clinical correlates. Epilepsia, 46:420–430.

45.

Song

, Birn

, Boly

, Meier

, Nair

, Meyerand

, Prabhakaran

. 2014. Age-related reorganizational changes in modularity and functional connectivity of human brain networks. Brain Connect, 4:662–676.

46.

Spencer

, Huh

. 2008. Outcomes of epilepsy surgery in adults and children. Lancet Neurol, 7:525–537.

47.

Sporns

, Zwi

. 2004. The small world of the cerebral cortex. Neuroinformatics, 2:145–162.

48.

Tzourio-Mazoyer

, Landeau

, Papathanassiou

, Crivello

, Etard

, Delcroix

, et al. 2002. Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage, 15:273–289.

49.

Vaessen

, Jansen

, Braakman

, Hofman

, De Louw

, Aldenkamp

, Backes

. 2014. Functional and structural network impairment in childhood frontal lobe epilepsy. PLoS One, 9:e90068.

50.

van Wijk

, Stam

, Daffertshofer

. 2010. Comparing brain networks of different size and connectivity density using graph theory. PLoS One, 5:e13701.

51.

Watts

, Strogatz

. 1998. Collective dynamics of ‘small-world’ networks. Nature, 393:440–442.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.11 MB

0.17 MB

0.11 MB

0.07 MB

0.02 MB