Abstract
Background:
White matter hyperintensities (WMH) that occur in the setting of vascular cognitive impairment and dementia (VCID) may be dynamic increasing or decreasing volumes or stable over time. Quantifying such changes may prove useful as a biomarker for clinical trials designed to address vascular cognitive-impairment and dementia and Alzheimer’s Disease.
Objective:
Conducting multi-site cross-site inter-rater and test-retest reliability of the MarkVCID white matter hyperintensity growth and regression protocol.
Methods:
The NINDS-supported MarkVCID Consortium evaluated a neuroimaging biomarker developed to track WMH change. Test-retest and cross-site inter-rater reliability of the protocol were assessed. Cognitive test scores were analyzed in relation to WMH changes to explore its construct validity.
Results:
ICC values for test-retest reliability of WMH growth and regression were 0.969 and 0.937 respectively, while for cross-site inter-rater ICC values for WMH growth and regression were 0.995 and 0.990 respectively. Word list long-delay free-recall was negatively associated with WMH growth (p < 0.028) but was not associated with WMH regression.
Conclusions:
The present data demonstrate robust ICC validity of a WMH growth/regression protocol over a one-year period as measured by cross-site inter-rater and test-retest reliability. These data suggest that this approach may serve an important role in clinical trials of disease-modifying agents for VCID that may preferentially affect WMH growth, stability, or regression.
Keywords
INTRODUCTION
White matter hyperintensities (WMH) found on magnetic resonance imaging (MRI) brain scans are a widely-recognized imaging correlate for the assessment of small vessel-type cerebrovascular disease (CVD) burden in vascular cognitive impairment and dementia (VCID) and Alzheimer’s disease (AD) in both cohort and population studies [1 –3]. WMH in late life are associated with small vessel ischemic disease and vascular risk factors such as hypertension, hyperlipidemia, and diabetes, among others [4 –7].
WMH are dynamic, often increasing in volume over time, but can also be stable and even regress [8 –12]. Despite a limited understanding of specific risk factor contributions to WMH and the longitudinal nature of such lesions, WMH volumes are associated with deficits in cognitive function, including both memory and executive function [13 –15]. As such, WMH volumes represent an important potential biomarker for assessing WMH-related VCID as a part of entry criteria for large scale multi-site clinical trials for vascular dementia and AD dementia, as well as a potential surrogate outcome of treatment mechanisms. Measuring such changes in WMH over periods as short as one year is critical for both mechanistic discoveries and for the informed future use of biomarkers to track disease modification in clinical trials of VCID and AD.
Although several studies measured and/or predicted WMH change over time, these have focused on total WMH volume measures rather than assessing regional WMH changes [12 , 16–19], and/or lacked data showing that the protocol could be performed reliably across MRI analysts and between centers [20 –23]. In this setting, total WMH volume changes over time have been defined as the arithmetic subtraction of summed whole brain WMH volumes across two or more time points [10, 12]. Such methods assume that only global WMH burden is important, which ignores the possibility that mixed WMH changes (i.e., growth, regression, and/or stable volumes) may occur concomitantly within a given individual in different anatomic regions and or even within discrete WMHs.
Understanding the relative contributions of such dynamic changes in the progression of VCID and how discrete growth, regression, and stability of WMH lesions may be altered by specific risk factors or therapeutic interventions may provide valuable insights into both potential mechanisms of disease and future therapeutic interventions. Major obstacles to the development of dynamic WMH protocols include difficulties in co-registration of fluid attenuated inversion recovery (FLAIR) images, which is often complicated by structural volume changes and fluctuations in WMH intensity between image acquisitions, despite using standardized acquisition protocols.
We developed a protocol that provides a reliable measure of WMH changes over periods as short as one-year [24]. This protocol was assessed previously for single-site inter-rater reliability (i.e., it was tested by two independent raters at one site), but not for cross-site inter-rater or test-retest reliability. In the current study, the NINDS-supported MarkVCID Consortium tested this protocol for cross-site inter-rater and test-retest reliability. In addition, we tested the hypothesis that measurement of discrete longitudinal change in WMH has a meaningful clinical correlation (i.e., construct validity).
METHODS
Study design
The MarkVCID approach for assessing MRI-based candidate biomarkers has been reported previously [25]. Inter-rater assessments used a standardized set of MRI acquisitions at baseline and one-year follow-up that were processed by independent raters across six sites of the MarkVCID consortium.
Test-retest reliability assessment involved comparisons of discrete WMH growth (i.e., conversion of normal appearing WM (NAWM) to WMH) and regression (i.e., conversion of WMH to NAWM) measures between each of two baseline scans, obtained 7–14 days apart, in relation to a single one-year follow-up scan for each participant. Figure 1A is a schematic of the study. Briefly, the test-retest reliability assessment was performed in two steps. First, each of MarkVCID sites acquired two scans (7–14 days between the first and second scans) and a single scan at one-year follow-up visit. Then, the test-retest reliability analysis was completely performed at a single site (University of Kentucky) utilizing the scans from the MarkVCID sites including the scans from the University of Kentucky.
An exploratory investigation of construct validity was performed in a subset of cross-site MarkVCID participants in relation to supraspan word list long delayed free recall (LDFR) [26] based on prior data demonstrating that supraspan word list delayed free recall was correlated with total WMH volume changes over a one-year interval [12, 27].
Participants
Participants, comprising persons with low and high CVD risk, with WMH Fazekas scores ≥1 at baseline were recruited into the MarkVCID cohort across the MarkVCID sites. The complete inclusion/exclusion criteria for the MarkVCID participants have been published [28]. In general, each site followed specific recruitment criteria to create a generalizable MarkVCID validation data set including cognitive status, vascular risk factors, age, gender, etc., that could be used for future studies. MarkVCID was approved by the Institutional Review Boards (IRBs) at each participating site and all participants provided informed consent.
For the test-retest reliability analysis, Data from 18 participants who completed test-retest and follow-up visits were identified for test-retest assessment. Data from 30 participants were randomly selected from the group of the initial participants (n = 97 by May 2022) completing all baseline and one-year follow-up clinical and imaging acquisitions for inter-rater reliability assessments as specified in the a priori analysis plan for this study. Prior to reliability assessments, all cases were reviewed by the University of Kentucky as the kit leading site to ensure an adequate distribution of clinical and WMH volume scores and measurements and to exclude any inconsistent scans for example with motion artifacts or any possible changes to the scan sequence parameters prior to the follow-up scans. The selection of cases was otherwise arbitrary regarding participant characteristics and scanner vendors and models including specific acquisition parameters (which vary based on scanner type), and site staff characteristics.
For the exploratory investigation of clinical correlates, Data of additional group of participants with longitudinal imaging and clinical data from the MarkVCID consortium were also collected (n = 53). The imaging and clinical data from these participants were used for exploratory evaluation of the construct validity of the WMH growth/regression protocol.
Image acquisition
Multi-echo 3D T1-weighted and 3D FLAIR MRI sequences [25] were acquired longitudinally for the WMH growth/regression protocol at the baseline and 1-year follow-up visits. Participants at all sites were scanned using a harmonized protocol that was optimized to generate similar image contrast irrespective of scanner type [25]. Scanners were from two different vendors and three different models (Siemens, TRIO & PRISMA and Philips ACHIEVA), but all had a magnetic field strength of 3.0T [25]. The image resolutions for both FLAIR and 3D T1-weighted images were 1×1×1 mm, and the field of view was 256×256 mm. Bandwidth was matched as closely as possible to match the image distortions to increase the accuracy of the image registration between the two scan types. All sites were asked to use the same head coil for all scans for each participant to ensure identical scans for each individual site. Also, instructions were given to each site for reducing head movement, consistency of head coil placement, and consistency in selecting the field of view in an identical manner for each time point.
WMH growth/regression longitudinal protocol
The details of the longitudinal protocol have been published previously [24]. Briefly, the longitudinal protocol was based on quantitative WMH protocols that were developed at the University of Kentucky’s Sanders-Brown Center on Aging [29, 30]. 3D anatomical T1-weighted and 3D FLAIR images were acquired for each participant at baseline and at the one-year follow-up visit. Intensity correction (N3-correction) was performed for all images to correct the intensity non-uniformity [31 –33]. The T1-weighted image was registered to the FLAIR image, using a 6-parameter rigid-body registration, for each time point utilizing SPM12 (http://www.fil.ion.ucl.ac.uk/spm). The resultant two time-point FLAIR images were registered again using SPM12 longitudinal registration tool to generate a midpoint image and deformation field maps for each time-point to be used to register the two time-point images to the midpoint image as previously described [24]. Images with motion artifacts were excluded from the study. The midpoint images were skull-stripped using the FSL-BET FMRIB software library (FSL v5.0.8) Brain Extraction Tool (http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/BET) and segmented utilizing the SPM12 unified segmentation tool to generate a WM mask [24 , 34]. The histogram distribution curve for each FLAIR WM image was fitted to the Gaussian model curve to compute the maximum and minimum threshold values to generate the WMH mask [24]. The processing pipeline was scripted within a Singularity container (https://singularity.lbl.gov) for consistent usage across sites.
As any WMH mask may include artifacts (false-positive voxels) due to the intensity overlaps between the normal-appearing WM and the WMH, a manual editing step was performed independently by all raters [29]. All sites received uniform training in manual editing procedures [29]. Using the same protocol, scripts within the Singularity container allowed the generation of binary growth and regression WMH masks based on a simple subtraction between the total WMH masks of the two-time points. Positive voxels represent WMH growth, and negative voxels represent WMH regression (Fig. 1B).
Strategy for determining test-retest repeatability for this longitudinal protocol and the source of image acquisition variability. A) This figure schematically illustrates the test-retest study design that utilized a comparison of two unique baseline scans with a single follow-up visit scan. B) Dynamic WMH changes within an individual subject are shown, (Red: Stable, Blue: Growth, Green: Regression).
Data analysis
Reliability
Test-retest reliability analyses were performed at a single site using images (n = 18) acquired across the MarkVCID consortium sites using standard ICC analysis that included a two-way (raters and participants) mixed effects model [35]. For the ICC analyses, rho≥0.85 [36, 37] was set as the minimally acceptable agreement for test-retest and cross-site inter-rater reliability. The Levene Statistic (test of Homogeneity of Variances) was used to test the homogeneity of variances across sites (p-values<0.05 were required to reject the null hypothesis).
Exploratory clinical construct validity
To evaluate construct validity, a preliminary analysis on a small sample of MarkVCID participants (n = 53) was conducted using an a priori statistical analysis plan that used the supraspan word list LDFR as the clinical correlate. For these analyses, WMH growth and regression volumes were normalized to intracranial volume, and then log-transformed to improve the plausibility of the linear model assumptions.
Raw LDFR scores were downloaded from MarkVCID1 database. The MarkVCID protocol allowed sites to use their choice of California Verbal Learning Test-second edition (CVLT-II; target words = 16), Hopkins Verbal Learning Test (HVLT; 12), and CVLT-II Short Form (CVLT-II-SF; 9). Due to the word lists differing in maximum target words, all raw LDFR scores were standardized to z-scores using published reference data for the respective word lists. For the CVLT-II and CVLT-II-SF we used the lookup tables in Delis et al. (2000) [38]. For the HVLT, we used the means and SDs from Ryan et al. (2021) using Table 3 [39]. The form version for CVLT-II (i.e., Standard or Alternate) is not captured in the MarkVCID1 database so we used the Standard Form reference tables from Delis et al. (2000) for all instances of CVLT-II. The form version for HVLT is also not captured in the MarkVCID1 database, but Ryan et al. (2021) used Form 6 in developing their norms [39]. The CVLT-II and CVLT-II-SF reference norms account for age at visit and sex. The HVLT reference norms account for sex, education, ethno-racial group, and age at visit. Despite these discrepancies in the segmentation of reference norms for the HVLT compared to CVLT-II and CVLT-II-SF, all reference data were derived from large, nationally representative samples. The standardized LDFR data at 12-month follow-up were regressed on measures of WMH change (growth and regression), controlling for the LDFR z-scores at the baseline.
RESULTS
Participant demographics, including age, education, and sex at birth, as well as general clinical status measured using the Montreal Cognitive Assessment test scores are provided in Table 1.
Demographic and clinical variables for the participants analyzed for cross-site inter-rater and test-retest reliability/reproducibility assessments
SD, standard deviation; SE, Standard error; MoCA, Montreal Cognitive Assessment; WMH, white matter hyperintensities.
Test-retest reliability
Test-retest ICCs of the two-way mixed consistency and absolute agreement models for WMH growth were identical at 0.969 (95% confidence interval CIrange 0.919–0.988). The two-way mixed consistency ICC was 0.939 (95% CI range 0.838–0.977) for WMH regression test-retest, and the two-way mixed absolute agreement model was 0.937 (95% CI range 0.835–0.976; Fig. 2).
Test-Retest Repeatability scatter plots for WMH A) Growth, and B) Regression volumes in mm3. The figure demonstrates highly significant ICC values (p < 0.001) for both WMH growth and regression comparing the two repeat baseline scans (test and retest) to the single one-year follow-up scan.
Cross-site inter-rater reliability
The mean ICC for growth was 0.995 (CI of 0.991–0.997), and the mean ICC for regression was 0.990 (CI of 0.984–0.995; Fig. 3). Moreover, the ICC of the WMH masks before editing was tested to check the reliability of the kit fully automated part and the mean ICC value was 1 at each time point. The Levene Statistic test provided no evidence of heterogeneity across sites. Levene coefficients for WMH (baseline), WMH (follow-up), WMH growth and WMH regression were 0.003 (p = 1), 0.007 (p = 1), 0.094 (p = 0.997), and 0.183 (p = 0.981), respectively. Thus, these results provided no evidence of heterogeneity across-sites, supporting the ICC test for cross-site inter-rater reliability.
Intraclass correlation coefficient (ICC) matrix demonstrating robust correlations across raters both at the University of Kentucky (two raters) and across 5 other MarkVCID consortium sites for WMH A) Growth and B) Regression. UKY1, UKY2, University of Kentucky, raters 1 and 2; RUSH, Rush University Medical Center; JHU, Johns Hopkins University School of Medicine; USC, University of South California; UTHSCSA, University of Texas Health San Antonio; UCD, University of California, Davis.
Variability in WMH growth, stability, and regression over a one-year period
Figure 4 shows participant-specific WMH volume changes (increasing (growth), decreasing (regression), and stability) over a one-year follow-up for the 53 participants included in the analysis of construct validity. Although WMH volume stability is the dominant result over the year of follow-up, within-participant WMH volume changes were notable and varied significantly among participants irrespective of total WMH change.
Histogram depiction of within-subject WMH change for the 53 participants that were analyzed for the preliminary assessment of clinical meaningfulness. WMH volume stability is dominant, while WMH volume growth and regression are variable across subjects and ranges of WMH volumes.
Preliminary analysis of construct validity
Adjusted linear regression analyses demonstrated that a 1-unit increase in log WMH growth was associated with about a 75% of standard deviation lower mean LDFR z-score at follow-up (95% confidence interval: –1.39, –0.093; p = 0.028); model R 2 = 0.59 (Fig. 5). In contrast, WMH regression was not significantly associated with mean LDFR z-score (p = 0.54), but the association was in the same direction (about 25% of standard deviation lower z-score with a 1-unit increase in log WMH regression.
Scatter plot of log WMH volume growth normalized to the total intercranial volume versus LDFR z-score one-year follow-up scores, analyzed using a linear regression model adjusted for baseline LDFR z-score. The results demonstrate that WMH volume growth is significantly, negatively associated with CVLT-LDFR change (p = 0.028).
DISCUSSION
In this study, we evaluated test-retest and cross-site inter-rater reliability of a novel neuroimaging biomarker that measures WMH growth, regression, and stability. These data, obtained from the MarkVCID consortium, demonstrated strong reliability for the assessment of longitudinal WMH change over one-year and may be a useful biomarker for future interventional studies of VCID, AD and for other targeted diseases and interventions in which the burden of WM lesions is important. Exploratory analysis using WMH growth and regression to predict memory performance demonstrated that isolating WMH growth may be a more powerful for clinical investigation than relying on the global WMH difference volume that has mixed effect of WMH growth and regression.
Advancing clinical trials for VCID is a research priority that is in part dependent on the development of pharmacodynamic biomarkers, which are currently lacking [40 –42]. Although whole-brain WMH, DTI, and ASL changes applied cross-sectionally as repeated measures show some promise, these imaging features are highly dependent on a complex interplay of both tissue destructive and healing/reparative factors. A biomarker based on WMH dynamic changes may provide a window into new processes in the development and evolution of VCID.
Optimal management of CVD risk factors, increased angiogenesis, and reductions in central nervous system inflammation may facilitate WMH regression, whereas poorly controlled CVD risk factors, reduced angiogenesis, and increased central nervous system inflammation may preferentially increase WMH growth. Evaluating WMH growth and regression using this protocol or similar methods may allow investigators to choose the VCID cohort (inclusion/exclusion) criteria most likely to be selectively in need of reduced WMH growth, stability and or augmented regression, most likely to be influenced by a candidate intervention. Such a discovery could yield insights into basic mechanisms responsible for VCID that are essential for progress in the field. Therefore, this protocol could be used to track the effectiveness of an intervention, pharmaceutical, etc., on the progression of a disease. For example, the SPRINT MIND trial found that lowering systolic blood pressure to less than 120 mmHg was associated with a reduction in the growth of WMH [43]. It remains unclear if such aggressive blood pressure management may also augment regression of WMH, and thus, whether WMH regression can serve as a marker of clinical benefit. Further understanding of these relationships, using refined biomarkers for VCID, is critical for optimal scientific discovery into disease mechanisms and interpretation of data stemming from interventional trials.
Limitations of the current study include the use of a participant group that is largely White/Caucasian, highly educated, and that has agreed to participate in an observational study for biomarker development in VCID. Further evaluation in diverse cohorts that can assess the generalizability of the current findings is essential. Although tested in a relatively small cohort of patients, the robustness of our findings calls into question the need for larger reliability studies. It should also be recognized that the MarkVCID consortium of sites represent a faculty that is expert in the assessment of biomarkers of VCID and it remains unclear how robust the protocol might be in settings with less expertise in the assessment of VCID biomarkers. Further work in centers that are not part of the MarkVCID consortium is needed to assess the reliability and validity of widespread use of the WMH growth and regression protocol. The Singularity Container for this protocol has been designed to advance knowledge of WMH longitudinal changes irrespective of the specific expertise or skills of the faculty and or staff performing such analyses at individual sites.
The need for a manual editing step can also be considered a limitation of the present protocol. The requirement for final manual editing is a critical issue for use of this protocol as the artifacts inherent in 3D FLAIR acquisition and post-processing steps are substantial. All site staff across all centers engaged in our study had specific training in the use of the protocol with a focus on the final manual editing step. The data presented attest to the success of such training and the usability of the MarkVCID WMH growth and regression protocol. Ongoing work seeks to fully automate the protocol using machine learning algorithms to bypass this manual process, but until such time, human analysis remains an essential part of the process. Indeed, data such as that derived from the current protocol may well serve as the “gold-standard” for machine learning algorithms that require such data to refine more automated processes.
In summary, despite the caveats presented above, the present data demonstrate that the MarkVCID WMH Growth/Regression protocol is reliable for use as a potentially valuable tool for future discoveries and as an outcome measure for clinical trials targeting VCID and AD in an aging population. Therefore, the protocol can be used to process the data despite the brand of the MRI scanner; however, for group comparison, it is important for the scan parameters to be as close as possible for different MRI scanners. Further biologic and clinical validation of this protocol is currently in progress.
Footnotes
ACKNOWLEDGMENTS
The authors wish to thank the many research participants who volunteered their time to make these discoveries possible. The authors also acknowledge Roderick A. Corriveau and Linda McGavern from the NINDS without whose vision and administrative oversight of the MarkVCID program this work would not have been possible, and also Baljeet Singh from University of California Davis, and the many staff and faculty engaged in the MarkVCID consortium at each of the performing sites.
FUNDING
This work was funded by NIH UH3 NS100599, U24 NS100591, NIGMS S10 OD023573, S10 OD025313, UH3 NS100588, UH3 NS100608, P30 AG010129, P30 AG072946, P30 AG066546, UH3 NS100605, UH3 NS100599 and UH 3NS100606.
CONFLICT OF INTEREST
Erin Abner, Claudia L. Satizabal, and Konstantinos Arfanakis are Editorial Board Members of this journal but were not involved in the peer-review process nor had access to any information regarding its peer-review. All other authors have no conflicts of interest to disclose.
DATA AVAILABILITY
The data are not publicly available. To request a copy of the raw data, please contact the MarkVCID consortium.