Temporal Order of Alzheimer’s Disease-Related Cognitive Marker Changes in BLSA and WRAP Longitudinal Studies

Abstract

Investigation of the temporal trajectories of currently used neuropsychological tests is critical to identifying earliest changing measures on the path to dementia due to Alzheimer’s disease (AD). We used the Progression Score (PS) method to characterize the temporal trajectories of measures of verbal memory, executive function, attention, processing speed, language, and mental status using data spanning normal cognition, mild cognitive impairment, and AD from 1,661 participants with a total of 7,839 visits (age at last visit 77.6 SD 9.2) in the Baltimore Longitudinal Study of Aging (BLSA) and 1510 participants with a total of 3,473 visits (age at last visit 59.5 SD 7.4) in the Wisconsin Registry for Alzheimer’s Prevention (WRAP). This method aligns individuals in time based on the similarity of their longitudinal measurements to reveal temporal trajectories. As a validation of our methodology, we explored the associations between the individualized cognitive progression scores (Cog-PS) computed by our method and clinical diagnosis. Digit span tests were the first to show declines in both data sets, and were detected mainly among cognitively normal individuals. These were followed by tests of verbal memory, which were in turn followed by Trail Making Tests, Boston Naming Test, and Mini-Mental State Examination. Differences in Cog-PS across the clinical diagnosis and APOE ɛ4 groups were statistically significant, highlighting the potential use of Cog-PS as individualized indicators of disease progression. Identifying cognitive measures that are changing in preclinical AD can lead to the development of novel cognitive tests that are finely tuned to detecting earliest changes.

Keywords

Alzheimer’s disease cognition longitudinal studies neuropsychological tests preclinical

Introduction

Cognitive changes in Alzheimer’s disease (AD), particularly declines in episodic memory, are detectable on neuropsychological testing up to 15 years prior to clinical diagnosis [1 –4]. However, cognition has only limited cross-sectional association with cerebral amyloid burden [5 –11], which marks the beginning of preclinical AD, according to the National Institute on Aging–Alzheimer’s Association (NIA–AA) criteria [12]. Longitudinal studies have consistently shown that amyloid levels are associated with greater rates of decline on tests of episodic memory [13 –21], suggesting that amyloid changes precede episodic memory declines. These findings indicate that despite the fifteen-year period prior to diagnosis for detecting cognitive change, currently used neuropsychological tests fall short of detecting changes in the earliest disease stages where therapeutic intervention is hypothesized to be most promising. Therefore, there is a need to develop cognitive tests and batteries that are more sensitive to changes in early preclinical AD and that correlate better with AD-related neuroimaging measures. Cognitive measures that are dynamic in preclinical disease can facilitate clinical trials aimed at this early stage, as they can serve as outcome measures that are non-invasive and cheaper to administer than neuroimaging evaluations.

To develop such cognitive tests or batteries, it is necessary to study the neuroimaging correlates and temporal trajectories of currently used tests to identify earliest changing measures. Various studies have investigated correlations between cognitive measures and early AD-related neuroimaging markers [22 –30]. In this work, we focus on characterizing the trajectories of a collection of cognitive markers widely used in studies of aging and AD. The ideal study of preclinical AD markers would follow cognitively normal individuals until they are diagnosed with dementia due to AD and retrospectively analyze the time courses of the markers of those who developed AD. However, currently available sample sizes do not allow for cross-validation or yield adequate statistical power to conduct such a study given that preclinical stages span a decade or longer [19]. To overcome this limitation, several statistical analysis approaches have been developed [31 –36]. An important concept in a subset of these approaches is the time-alignment of individuals. Here, we used the Progression Score Model [36, 37], a multivariate nonlinear mixed effects model, to construct cognitive marker trajectories spanning normal cognition, mild cognitive impairment (MCI), and AD stages by aligning individuals in time based on the similarity of their marker profiles. The method incorporates longitudinal information in performing this alignment and accounts for inter-individual differences in baseline measurement levels and rate of progression. The main premise of the method is that rather than using age as a proxy for disease progression, we can obtain better disease progression indicators as well as temporal trajectories for the biomarkers by aligning individuals in time based on the similarity of their biomarker profiles. Time alignment of individuals allows us to study the long-term trajectories of cognitive measures despite the availability of only a small number of individuals who have traversed large extents of the cognitive trajectories. We conducted separate analyses on two well-characterized longitudinal studies, the Baltimore Longitudinal Study of Aging (BLSA) and the Wisconsin Registry for Alzheimer’s Prevention (WRAP), to delineate the trajectories of measures of verbal memory, executive function, attention, processing speed, language, and mental status among individuals who are cognitively normal, have MCI, or have dementia due to AD.

The main aims of the analyses were to: 1) develop a standardized template of cognitive changes against which individuals can be quantified; 2) present evidence of the validity of the individualized scores obtained using the standardized cognitive template by exploring their associations with clinical diagnosis, APOE ɛ4 status, and time to dementia diagnosis; 3) identify the order in which detectable changes begin to appear across cognitive tests; and 4) validate this identified ordering using an independent approach at the individual level.

Methods

Statistical methods used in the following analyses were applied separately to two ongoing longitudinal cohort studies of human aging: the Baltimore Longitudinal Study of Aging (BLSA) [38], initiated in 1958 and conducted by the Intramural Research Program of the National Institute on Aging, and the Wisconsin Registry for Alzheimer’s Prevention (WRAP) [39], initiated in 2001 and conducted by the University of Wisconsin Alzheimer’s Institute.

Participants

BLSA participants

BLSA analyses were based on data from 1,661 participants (Table 1). We included visits where participants were cognitively normal or had a clinical diagnosis of MCI or dementia due to AD. Baseline and last visit were defined for each participant as the first and last BLSA visit between 1/1992 and 11/2015 where they were aged ≥60 and met the inclusion criteria based on clinical diagnosis and number of available cognitive scores (see below). A total of 7,839 visits were selected for analysis. The Institutional Review Board associated with the NIA Intramural Research Program approved the research protocol for this study, and informed consent was obtained at each visit from all participants.

Table 1

Participants

Final cognitive status	BLSA			WRAP
	Normal	MCI	AD	Whole sample	Normal	CI^*	Whole sample
	(n = 1428)	(n = 146)	(n = 87)	(n = 1661)	(n = 1275)	(n = 233)	(n = 1510^**)
Age at last visit, mean (SD), y	76.4 (9.0)	84.3 (6.4)	85.8 (6.7)	77.6 (9.2)	59.2 (7.4)	61.2 (7.3)	59.5 (7.4)
Range	60.0–101.0	68.0–99.3	66.4–98.6	60.0–101.0	38–78	40–76	38–78
Male, no. (%)	687 (48%)	58 (40%)	36 (41%)	781 (47%)	346 (27%)	100 (43%)	447 (30%)
White, no. (%)	1082 (76%)	128 (88%)	76 (87%)	1286 (77%)	1159 (91%)	194 (83%)	1355 (90%)
Education, mean (SD), y	16.7 (2.7)	16.2 (2.9)	17.2 (2.4)	16.7 (2.7)	16.1 (2.8)	15.9 (3.1)	16.1 (2.8)
Range	8–21	8–20	9–21	8–21	9–29	10–26	9–29
Number of visits, mean (SD)	4.5 (3.8)	5.5 (4.4)	6.3 (4.5)	4.7 (3.9)	2.3 (1.0)	2.1 (1.0)	2.3 (1.0)
Range	1–22	1–23	1–22	1–23	1–5	1–5	1–5
Follow-up duration, mean (SD), y	7.1 (6.5)	7.6 (6.2)	9.1 (6.5)	7.3 (6.5)	5.7 (3.8)	5.9 (4.1)	5.7 (3.8)
Range	0.0–23.4	0.0–23.0	0–22.9	0–23.4	0–13	0–12	0–13
Trails A at last visit, mean (SD)^***	38.8 (19.6)	54.5 (20.8)	87.1 (56.7)	42.3 (25.0)	26.0 (8.4)	32.3 (10.4)	26.9 (9.0)
Range	11–300	21–145	29–300	11–300	9–88	13–69	9–88
Trails B	98.4 (50.2)	168.6 (70.2)	198.8 (73.9)	106.2 (57.8)	61.8 (24.6)	93.6 (49.7)	66.8 (32.0)
	26–300	51–300	78–300	26–300	24–300	29–300	24–300
Digit span forward	7.7 (2.5)	7.2 (2.3)	6.2 (2.6)	7.6 (2.5)	10.5 (2.2)	9.9 (2.3)	10.4 (2.2)
	0–14	0–13	0–12	0–14	4–16	5–16	4–16
Digit span backward	6.7 (2.4)	5.8 (2.1)	4.7 (2.2)	6.5 (2.4)	7.2 (2.2)	6.3 (2.3)	7.1 (2.2)
	0–14	0–13	0–11	0–14	2–14	2–13	2–14
CVLT/AVLT^# immediate recall	49.5 (12.4)	33.1 (9.3)	21.4 (10.6)	46.8 (14.1)	52.2 (7.5)	41.1 (8.1)	50.5 (8.6)
	0–80	15–61	0–47	0–80	24–72	21–62	21–72
CVLT/AVLT delayed recall	10.2 (3.6)	5.2 (3.3)	1.7 (2.1)	9.5 (4.1)	11.0 (2.6)	7.2 (3.1)	10.4 (3.1)
	0–16	0–13	0–8	0–16	1–15	0–15	0–15
Boston Naming test	53.6 (6.4)	48.2 (8.0)	42.3 (12.2)	52.6 (7.4)	57.2 (3.4)	55.3 (4.6)	56.9 (3.7)
	6–60	18–60	5–58	5–60	34–60	40–60	34–60
MMSE	28.3 (1.8)	26.4 (2.7)	22.3 (5.9)	27.8 (2.6)	29.4 (0.9)	29.0 (1.2)	29.4 (1.0)
	13–30	13–30	0–30	0–30	24–30	24–30	23–30

^*Cognitively impaired (CI) group in WRAP includes both clinical MCI (n = 22) as well as individuals who were determined to have early MCI based on internal norms for cognitive measures (n = 211). ^*^*Two participants were diagnosed with dementia due to Alzheimer’s disease at last visit. ^***All cognitive measure statistics are computed using available values at last visit and reported as mean (SD); Range. ^#Verbal memory is measured using CVLT in BLSA and AVLT in WRAP. BLSA, Baltimore Longitudinal Study of Aging; WRAP, Wisconsin Registry for Alzheimer’s Prevention; CI, cognitive impairment; MCI, mild cognitive impairment; AD, Alzheimer’s disease; SD, standard deviation; CVLT, California verbal learning test; AVLT, Auditory Verbal Learning Test; MMSE, Mini-Mental State Exam.

WRAP participants

WRAP analyses were based on data from 1,510 participants (Table 1), 72.6% of whom had a family history of AD. We included all visits where participants were cognitively normal or had a clinical diagnosis of MCI (including an “early MCI” diagnosis, defined below) or dementia due to AD. Any visit that had not been assigned a consensus diagnosis at the time of the analyses was excluded. Baseline and last visit were defined for each participant as the first and last WRAP visit up until Wave 5 (inclusive) where they met the inclusion criteria based on clinical diagnosis and number of available cognitive scores (see below). A total of 3,473 visits were selected for analysis. All activities for this study were approved by the Institutional Review Board and completed in accordance with the Helsinki Declaration.

Cognitive measures

Eight cognitive measures were selected for our analyses: Trail Making Tests (Trails) A and B [40] to assess processing speed and executive function, Wechsler Adult Intelligence Scale (WAIS) digit span forward and backward (revised edition [41] in BLSA, 3rd edition [42] in WRAP) to assess attention and executive function, California Verbal Learning Test (CVLT) [43] in BLSA and Rey Auditory Verbal Learning Test (AVLT) [44] in WRAP measuring immediate recall (sum of total recall across five learning trials), 20-min delayed recall to assess verbal memory, Boston Naming Test [45] to assess language, and Mini-Mental State Examination (MMSE) [46] to assess mental status. Trails A and B were truncated at 300 s and log-transformed. We included visits where at least four of these eight measures were available. We did not include WAIS digit symbol substitution [41, 42] or animal fluency [47] in our analyses even though they were common to both data sets, since they were introduced in later stages in BLSA and/or WRAP.

CVLT was added to the BLSA battery in 3/1993 and digit span was added in 1/1992, while the remainder of the measures were added between 5/1984 and 1/1990. BLSA participants under age 60 did not receive the Trail Making Test until 2005. MMSE was added to the WRAP battery in Wave 2, while the remaining eight measures were available starting with Wave 1 in 2001.

Clinical diagnoses

BLSA methods

All BLSA participants are reviewed for cognitive impairment at a consensus case conference if they have a score ≥4 on the Blessed Information–Memory–Concentration Test [48], if their Clinical Dementia Rating (CDR) score [49] is ≥0.5 on the subject or informant report or if they screen abnormal on the Dementia Questionnaire. Consensus diagnoses are determined by a research team that includes neurologists, psychiatrists, neuropsychologists, research nurses, and research assistants. Diagnoses of dementia and AD are based on Diagnostic and Statistical Manual of Mental Disorders 3rd edition–revised [50] and National Institute of Neurological and Communication Disorders and Stroke–AD and Related Disorders Association (NINCDS-ADRDA) [51] criteria, respectively. MCI is diagnosed based on the Petersen criteria [52] when cognitive impairment is evident for a single domain or multiple domains without significant functional loss in activities of daily living. Out of the eight cognitive measures described above, only Trails A and B, Boston Naming, and MMSE are used in clinical diagnosis determination.

WRAP methods

WRAP participant visits are reviewed at a consensus case conference if they meet one or more of the following criteria: 1) cognitive abnormalities relative to WRAP peers (i.e., at least 1.5 standard deviations below expected relative to robust internal norms adjusting for age, sex, and literacy-level on the most recent assessment for factor scores or individual measures of memory, executive function, language, working memory, or attention [53, 54]; 2) cognitive performance on one or more tests below values used in other studies as cutoff points for MCI diagnoses (e.g., WMS-R Logical Memory II [55] story A score <9, Alzheimer’s Disease Neuroimaging Initiative [56]); or 3) an abnormal informant report indicating subjective cognitive or functional decline. Consensus diagnoses are determined by a research team that includes physicians, clinical neuropsychologists, and clinical nurse practitioners. Cognitive status includes cognitively normal, early MCI, clinical MCI, other cognitive impairment (e.g., due to other medical conditions), or dementia. The diagnosis of clinical MCI is based on NIA-AA criteria [57] and includes a) concern regarding change in cognition, b) impairment in one or more cognitive domains, c) preservation of functional abilities, and d) does not meet criteria for dementia. The status of early MCI was developed to identify individuals in the cohort who exhibit lower than expected objective performance in one or more cognitive domains (relative to internal robust norms), but may not yet report subjective cognitive complaints. This experimental construct is thought to represent a phenotype of early cognitive decline expected to precede a clinical diagnosis of MCI [54, 58]. Dementia diagnosis is based on NINCDS-ADRDA criteria [51]. Both demented WRAP individuals included in these analyses had dementia due to AD. The consensus review process was initiated in late 2012.

Statistical analyses

Removal of cognitive effects of aging and other covariates in the absence of neurodegenerative disease

To estimate the cognitive effects of aging in the absence of neurodegenerative disease, we selected a subset of visits corresponding to individuals we refer to as the “robust normal group”. For inclusion in this group, individuals needed to have at least 2 years of cognitive follow-up in which they remained cognitively normal. Individuals whose clinical diagnosis reverted from any type of cognitive impairment to cognitively normal were excluded. All visits prior to impairment, excluding last 2 years of cognitively normal follow-up, were included in the subset.

Linear mixed effects models were fitted separately for each cognitive measure using this subset. Fixed effects included baseline age, sex, race, years of education, time from baseline in years, and interactions of time with baseline age and sex. These variables were selected based on previous reports demonstrating associations with cognition [59]. Random effects included intercept per individual. Estimated fixed effects were used to make predictions for each individual and visit in the larger sample, and these predictions were subtracted from observed cognitive measures to yield residuals.

Progression score model

We used an improved version [37] of the Progression Score (PS) method [36] to compute cognitive progression scores (Cog-PS) for individuals in BLSA and WRAP in separate models using residuals for the eight cognitive tests described above (see Supplementary Material for model details). The PS method is based on a multivariate model that enables the computation of a score for each visit using a collection of longitudinal biomarker measures to reflect the state of the visit relative to the rest of the sample. Cog-PS is an affine transformation of age, and these transformations are allowed to vary across individuals via subject-specific variables that model inter-subject differences in rate and baseline levels of progression. The scale of the Cog-PS at the end of the model fitting procedure is arbitrary (i.e., a Cog-PS of 0 does not convey any meaning on its own, but compared to a Cog-PS of 1, it indicates better overall performance on the cognitive measures included in the model). To give meaning to the Cog-PS scale, we shifted all Cog-PS values such that a Cog-PS of 0 corresponds to the mean value across cognitively normal individuals, and then scaled all Cog-PS values such that the standard deviation of Cog-PS among cognitively normal individuals is 1. The Cog-PS calibration, when combined with the appropriate transformations of the model parameters, does not affect the likelihood of the model (i.e., the calibrated and uncalibrated models are identical in terms of their fit to the observed data). This calibration is intended to make Cog-PS values computed on separate data sets comparable, with the underlying assumption that cognitively normal groups are comparable. At the end of model fitting, we obtained Cog-PS for each visit as well as estimates of cognitive measure trajectories as a function of Cog-PS. The overall procedure is summarized in Fig. 1.

Fig.1

Diagram illustrating progression score (PS) model fitting, PS calibration, and cognitive measurement scaling to obtain standardized space of cognitive markers. Lower values for the illustrated cognitive markers indicate lower cognitive performance. PS values are calibrated such that lower progression scores indicate better overall cognitive performance.

Temporal ordering of cognitive measures

To compare the estimated trajectories for the eight cognitive measures, we linearly scaled them such that their estimated values at the minimum and maximum Cog-PS observed in the sample were 0 and 1, respectively. In this standardized space, we refer to a scaled cognitive value of 0 as “normal” and a scaled cognitive value of 1 as “abnormal”. We visually evaluated the estimated trajectories to make determinations about temporal ordering of cognitive changes. To complement this qualitative evaluation, we developed a quantitative procedure based on bootstrapping. The number of bootstrap experiments was limited to 20 due to the time-consuming model fitting procedure. For each bootstrap, we computed the PS value at which the estimated trajectory attained the midway point of 0.5 in the standardized normal-to-abnormal scale for each cognitive measure. We compared the range of these PS values across biomarkers to make quantitative assessments about ordering.

Validation of Cog-PS results

To show evidence for the validity of the individualized Cog-PS, we first investigated differences in Cog-PS cross-sectionally (at last visit) across diagnosis and APOE ɛ4 (carrier of at least one ɛ4 allele versus non-carrier) groups using two-sided Wilcoxon rank-sum tests. APOE ɛ4 is the most influential known genetic risk factor for AD, with each ɛ4 allele leading to an earlier onset of the disease by about a decade [60, 61]. Since this variable was not used in the PS model, and since it is not dependent on the cognitive measures that went into our analyses, it served as an unbiased validator of Cog-PS as a disease stage indicator. Finally, using data for 87 BLSA individuals who had an AD diagnosis at their last visit, we investigated the relationship between Cog-PS at baseline and years to dementia diagnosis using partial Pearson correlation, adjusting for age.

To provide further evidence for the temporal ordering determined by the Cog-PS model, we performed an analysis using linear mixed effects models that was independent from the Cog-PS approach. These models allowed us to assess the ordering of a pair of cognitive measures. For a pair of measures, both of which decline over time, we expected to find a negative association between baseline values of the later changing cognitive measure and the rate of change of the earlier changing measure. We also expected to find a positive association between baseline values of the earlier changing measure and the rate of change of the later changing measure. These concepts are illustrated in Supplementary Figure 1. Longitudinal measures on one of the two cognitive tests being compared constituted the outcome in the linear mixed effects model. Covariates were included for baseline age, time (measured from baseline), sex, years of education, interaction of baseline age with time, and interaction of sex with time. Baseline measure on the other cognitive test, as well as its interaction with time, were included as predictors. Random effects included intercept per individual. We used the estimates and p-values for the time×baseline cognitive measure interaction term to assess our hypothesis about temporal ordering.

Results

Temporal ordering of cognitive measures

Estimated cognitive marker trajectories as function of Cog-PS are presented in Fig. 2. While digit span tests have a marked downward slope in the left-hand side of the figure at a Cog-PS of –1, other cognitive measures do not begin exhibiting marked declines until later along the Cog-PS scale. CVLT/AVLT measures are next to exhibit declines. These are followed by Trails, Boston Naming, and MMSE in BLSA, and MMSE, Trails, and Boston Naming in WRAP.

Fig.2

Cognitive measure trajectories in (a) BLSA and (b) WRAP. Gray, blue, and red dots indicate cognitively normal, MCI or CI, and AD visits, respectively.

The standardized cognitive template, obtained by scaling the cognitive measures based on fitted values corresponding to the minimum and maximum Cog-PS values in the sample, as well as the bootstrap results, are presented in Fig. 3. In both BLSA and WRAP, digit span forward and backward were the first measures to attain the midway point of the biomarker abnormality scale, followed by CVLT/AVLT immediate and delayed recall.

Fig.3

Scaled cognitive trajectories obtained using (a) BLSA and (b) WRAP data. Dotted line in the top panel corresponds to the midway point between Normal and Abnormal. Line segments in the bottom panel indicate the range of the Cog-PS values that attain the midway point along the standardized normal-to-abnormal y-axis across 20 bootstrap experiments.

Validation of Cog-PS results

Cog-PS at last visit was associated with concurrent diagnosis, with cognitively normal individuals having lower Cog-PS compared to impaired individuals (Fig. 4). All pairwise group comparisons (normal versus CI/MCI, MCI versus AD, and normal versus AD) were significant within each data set (two-sided Wilcoxon rank-sum test, all p < 0.0004). Cog-PS was higher in the APOE ɛ4+ group compared to non-carriers in both BLSA (ɛ4+ mean 0.320 SD 1.089 versus ɛ4- mean 0.161 SD 1.077, two-sided Wilcoxon rank-sum test p = 0.015) and WRAP (ɛ4+ mean 0.235 SD 0.995 versus ɛ4- mean 0.116 SD 1.020, p = 0.02). Years to dementia diagnosis was negatively correlated with Cog-PS at baseline in BLSA (partial Pearson correlation given age = –0.61, p≤0.00001).

Fig.4

Box plot of Cog-PS by clinical diagnosis at last visit in BLSA (left) and WRAP (right). Central mark is the median, edges of the box correspond to the interquartile range, and whiskers extend to the range of non-outlier Cog-PS values. Outliers are plotted individually. All pairwise group comparisons were significant within each data set (two-sided Wilcoxon rank-sum test, p < 0.0004).

We used linear mixed effects models to further assess the finding that digit span measures are the earliest measures to change among of the eight considered in this study. To this end, we investigated their temporal relationship with the CVLT/AVLT measures. Model results are presented in Supplementary Table 1. As we expected, in BLSA, we found that higher baseline scores on digit span forward were associated with attenuated declines on CVLT immediate (time×baseline digit span forward estimate = 0.027, SE = 0.011, p = 0.012) as well as delayed recall (estimate = 0.0066, SE = 0.0030, p = 0.026), and that higher baseline CVLT delayed recall scores were associated with steeper declines on digit span forward (time×baseline CVLT delayed recall estimate = –0.0043, SE = 0.0021, p = 0.043). We did not find any associations with digit span backward or with AVLT in WRAP.

Discussion

We applied the progression score model to align individuals in time based on the similarity of their cognitive marker profiles to learn long-term cognitive trajectories from shorter-term longitudinal measurements. From this model, we obtained individualized scores, termed Cog-PS, which we validated by demonstrating their association with clinical diagnosis, APOE ɛ4 status, and time to dementia diagnosis. Our analyses were conducted separately on data from two well-characterized longitudinal studies of aging, BLSA and WRAP, and revealed similar results.

Cognitive measures reflect not only disease-related changes but also changes due to aging. While we used residualized measures in our analyses to distill effects due to disease, changes due to aging may not have been completely removed. Therefore, our results regarding the ordering of cognitive measure changes may be confounded by changes due to aging rather than due to disease.

Using the estimated trajectories and the approximate confidence intervals obtained via bootstrapping, we showed that digit span tests exhibit their most extensive dynamic range while individuals are cognitively normal, and that these changes precede those observed on verbal memory tests, which in turn precede changes on Trails and Boston naming. MMSE was the last to change among the eight measures in BLSA, but in WRAP, changes in MMSE followed those in AVLT. We must consider the limited range of MMSE values observed in WRAP data set when interpreting this discrepancy. As evident in the MMSE panel in Fig. 2b, the MMSE residual attained at highest values of Cog-PS in WRAP is about –1, whereas in BLSA, MMSE exhibits a much wider range, with highest values of Cog-PS corresponding to an MMSE residual of about –5. This means when we standardize the cognitive measures to visualize their trajectories on a common y-axis as presented in Fig. 3, we base the standardization of the MMSE trajectory on very different values in the two data sets. Therefore, the standardized MMSE trajectory reflects the path to a MMSE residual of –1 in WRAP, and to –5 in BLSA. This explains the difference between WRAP and BLSA in the temporal order of MMSE relative to the other cognitive measures.

Our finding that digit span changes occur prior to verbal memory changes was further supported by our linear mixed effects modeling approach. Neither the progression score model nor the linear mixed effects model is a perfect representation of the disease process. There are different sets of underlying assumptions in each approach. However, results of these different techniques, taken collectively, provide complementary perspectives through which we can build confidence regarding our results about temporal ordering.

The progression score approach enabled the computation of an individualized indicator of cognitive performance based on a collection of longitudinal measurements. We demonstrated that there are statistically significant differences in progression score across cognitively normal, CI/MCI, and AD groups. Since the progression score model is agnostic to clinical diagnosis, this analysis served as a validation of the computed scores. However, it should be noted that this is a partial validation since there is a degree of circularity as four out of the eight cognitive measures were used in diagnosis determination in BLSA, and eight were used in WRAP. Our demonstration of associations between Cog-PS and time to dementia diagnosis as well as APOE ɛ4 status avoided this circularity and provided further evidence that Cog-PS is indicative of the disease process.

While we used a calibration approach to render the progression scores computed on BLSA and WRAP comparable, it is important to note that due to differences in the composition of cognitively normal groups in the two data sets, Cog-PS values do not convey the same meaning across the studies. Despite this limitation of our methodology, we found similar temporal cognitive patterns in both studies. Cognitively normal as well as the CI/MCI groups in BLSA had lower cognitive performance on all tests compared to WRAP. As mentioned previously in the discussion of the MMSE trajectories, our scaling procedure for establishing correspondences across cognitive markers is sensitive to the range of cognitive measurements present in the study. Since this scaling procedure is intended to identify normal and abnormal cognitive test values, it is important that the data set contain measurements spanning the entire normal-to-abnormal range for each cognitive test. This condition is not fully satisfied in either data set we considered in this work, particularly in WRAP since WRAP participants are about 18 y younger than BLSA participants on average. Therefore, the scaling procedure may be less accurate in WRAP than in BLSA, especially for later-changing measures such as Boston Naming and MMSE.

Several previous studies have reported that changes in digit span tests are detectable prior to a clinical diagnosis of cognitive impairment. For example, a study that evaluated non-demented individuals with subjective memory complaints found that those with normal digit span scores (defined using age- and education-adjusted neuropsychological test scores) at baseline did not exhibit significant declines on verbal memory, visual memory, or executive function after a mean follow-up of 6.6 y, but had significant declines on the sum score of digit span forward and backward [62]. On the other hand, age-, sex-, and education-matched individuals with impaired digit span scores at baseline had significant declines on tests of verbal learning and animal fluency but not on any other cognitive test. These findings agree with our estimated ordering of cognitive trajectories on the path to cognitive impairment, with digit span measures declining first, followed by measures of verbal memory. Another study that investigated changes in cognition prior to autopsy in a sample of individuals who remained cognitively normal found that while longitudinal changes were not significant when assessed separately among individuals with and without AD neuropathology, longitudinal decline in the attention/working memory domain (assessed using digit span forward and backward tests) was greater among those with neuropathology compared to those without [63]. Other domains, including episodic memory, language, and executive function, did not show statistically significant longitudinal differences between the two groups [63].

A limitation of our method is that it does not indicate whether the detected cognitive changes are due to AD-related mechanisms, and further studies are needed to delineate normal aging versus disease. Therefore, our results cannot be interpreted as evidence for using digit span to predict individualized diagnoses. Digit span is not a good predictor of concurrent diagnosis; computing cognitive measure thresholds based on our Cog-PS model and using these thresholds to classify individuals as normal versus CI/MCI/AD would yield many false positives. This lack of specificity of digit span for dementia has been documented previously [64, 65]. What our results demonstrate is that changes in digit span are most evident early on in cognitively normal stages. Detectability of these changes and their associations with future outcomes at the individual-level, including AD diagnosis in following years or decades, remain to be fully elucidated; however, prior studies have reported that sensitivity to change on digit span is small [66]. Another limitation of our method is that it treats discrete cognitive measures such as digit span as continuous variables and does not account for practice effects seen on serial administrations of neuropsychological tests. These factors may introduce bias into the characterization of the longitudinal cognitive trajectories. Despite these limitations, understanding the association of digit span tests with brain changes can be informative for designing novel cognitive tests or batteries that are more sensitive to changes in preclinical AD and that correlate with functional and structural brain changes over the course of disease.

Several studies of a small number of healthy individuals found that higher performance on digit span backward is associated with activation of the right dorsolateral prefrontal cortex (DLPFC) [67 –69], bilateral inferior parietal lobule, anterior cingulate, left DLPFC, and Broca’s area, with a subset of these regions also implicated in relation to digit span forward [68, 69]. DLPFC is one of the amyloid accumulating regions, and therefore it may be possible to demonstrate associations between digit span performance and DLPFC amyloid levels among cognitively normal individuals in studies with large sample sizes.

Analyses conducted in parallel in multiple data sets or in combined samples will accelerate efforts to further elucidate the relationships among cognitive measures implicated in preclinical AD. Identifying cognitive measures that are dynamic in preclinical AD can lead to the development of novel cognitive tests that are finely tuned to detecting earliest changes. Such measures will facilitate clinical trials aimed at this early stage by serving as outcome measures that are non-invasive and cheaper to administer than neuroimaging.

Footnotes

Acknowledgments

We would like to thank the BLSA and WRAP participants and staff for their efforts and commitment. Without them this research would not have been possible.

This research was supported in part by the Intramural Research Program of the National Institute on Aging (National Institutes of Health) and the Michael J. Fox Foundation for Parkinson’s Research, MJFF Research Grant ID: 9310 (BMJ). WRAP is supported by NIA grant R01AG27161 (SCJ; Wisconsin Registry for Alzheimer Prevention: Biomarkers of Preclinical AD). WRAP is also supported by the Clinical and Translational Science Award (CTSA) program, through the NIH National Center for Advancing Translational Sciences (NCATS), grant UL1TR000427. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Authors’ disclosures available online ().

The supplementary material is available in the electronic version of this article: .

References

Bäckman

, Jones

, Berger

, Laukka

, Small

(2005) Cognitive impairment in preclinical Alzheimer’s disease: A meta-analysis. Neuropsychology 19, 520–531.

Grober

, Hall

, Lipton

, Zonderman

, Resnick

, Kawas

(2008) Memory impairment, executive dysfunction, and intellectual decline in preclinical Alzheimer’s disease. J Int Neuropsychol Soc 14, 266–278.

Kawas

, Corrada

, Brookmeyer

, Morrison

, Resnick

, Zonderman

, Arenberg

(2003) Visual memory predicts Alzheimer’s disease more than a decade before diagnosis. Neurology 60, 1089–1093.

Elias

, Beiser

, Wolf

, Au

, White

, D’Agostino

(2000) The preclinical phase of Alzheimer disease: A 22-year prospective study of the Framingham Cohort. Arch Neurol 57, 808–813.

Hedden

, Oh

, Younger

, Patel

(2013) Meta-analysis of amyloid-cognition relations in cognitively normal older adults. Neurology 80, 1341–1348.

Johnson

, Christian

, Okonkwo

, Oh

, Harding

, Xu

, Hillmer

, Wooten

, Murali

, Barnhart

, Hall

, Racine

, Klunk

, Mathis

, Bendlin

, Gallagher

, Carlsson

, Rowley

, Hermann

, Dowling

, Asthana

, Sager

(2014) Amyloid burden and neural function in people at risk for Alzheimer’s Disease. Neurobiol Aging 35, 576–584.

Perrotin

, Mormino

, Madison

, Hayenga

, Jagust

(2012) Subjective cognition and amyloid deposition imaging: A Pittsburgh compound B positron emission tomography study in normal elderly individuals. Arch Neurol 69, 223–229.

Song

, Insel

, Buckley

, Yohannes

, Mezher

, Simonson

, Wilkins

, Tosun

, Mueller

, Kramer

, Miller

, Weiner

(2015) Brain amyloid-β burden is associated with disruption of intrinsic functional connectivity within the medial temporal lobe in cognitively normal elderly. J Neurosci 35, 3240–3247.

Amariglio

, Becker

, Carmasin

, Wadsworth

, Lorius

, Sullivan

, Maye

, Gidicsin

, Pepin

, Sperling

, Johnson

, Rentz

(2012) Subjective cognitive complaints and amyloid burden in cognitively normal older individuals. Neuropsychologia 50, 2880–2886.

10.

Doraiswamy

, Sperling

, Coleman

, Johnson

, Reiman

, Davis

, Grundman

, Sabbagh

, Sadowsky

, Fleisher

, Carpenter

, Clark

, Joshi

, Mintun

, Skovronsky

, Pontecorvo

(2012) Amyloid-β assessed by florbetapir F 18 PET and 18-month cognitive decline: A multicenter study. Neurology 79, 1636–1644.

11.

Sperling

, Johnson

, Doraiswamy

, Reiman

, Fleisher

, Sabbagh

, Sadowsky

, Carpenter

, Davis

, Lu

, Flitter

, Joshi

, Clark

, Grundman

, Mintun

, Skovronsky

, Pontecorvo

(2013) Amyloid deposition detected with florbetapir F 18 (18F-AV-45) is related to lower episodic memory performance in clinically normal older individuals. Neurobiol Aging 34, 822–831.

12.

Sperling

, Aisen

, Beckett

, Bennett

, Craft

, Fagan

, Iwatsubo

, Jack

Jr , Kaye

, Montine

, Park

, Reiman

, Rowe

, Siemers

, Stern

, Yaffe

, Carrillo

, Thies

, Morrison-Bogorad

, Wagster

, Phelps

(2011) Toward defining the preclinical stages of Alzheimer’s disease: Recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimers Dement 7, 280–292.

13.

Lim

, Maruff

, Pietrzak

, Ames

, Ellis

, Harrington

, Lautenschlager

, Szoeke

, Martins

, Masters

, Villemagne

, Rowe

, AIBL Research Group (2014) Effect of amyloid on memory and non-memory decline from preclinical to clinical Alzheimer’s disease. Brain 137, 221–231.

14.

Lim

, Villemagne

, Pietrzak

, Ames

, Ellis

, Harrington

, Snyder

, Martins

, Masters

, Rowe

, Maruff

(2015) APOE ɛ4 moderates amyloid-related memory decline in preclinical Alzheimer’s disease. Neurobiol Aging 36, 1239–1244.

15.

Mormino

, Betensky

, Hedden

, Schultz

, Ward

, Huijbers

, Rentz

, Johnson

, Sperling

(2014) Amyloid and APOE ɛ4 interact to influence short-term decline in preclinical Alzheimer disease. Neurology 82, 1760–1767.

16.

Pietrzak

, Lim

, Neumeister

, Ames

, Ellis

, Harrington

, Lautenschlager

, Restrepo

, Martins

, Masters

, Villemagne

, Rowe

, Maruff

(2015) Amyloid-β, anxiety, and cognitive decline in preclinical Alzheimer disease: A multicenter, prospective cohort study. JAMA Psychiatry 72, 284–291.

17.

Roe

, Fagan

, Grant

, Hassenstab

, Moulder

, Dreyfus

, Sutphen

, Benzinger

, Mintun

, Holtzman

, Morris

(2013) Amyloid imaging and CSF biomarkers in predicting cognitive impairment up to 7.5 years later. Neurology 80, 1784–1791.

18.

Stonnington

, Chen

, Lee

, Locke

DEC

, Dueck

, Liu

, Roontiva

, Fleisher

, Caselli

, Reiman

(2014) Fibrillar amyloid correlates of preclinical cognitive decline. Alzheimers Dement 10, e1–e8.

19.

Villemagne

, Burnham

, Bourgeat

, Brown

, Ellis

, Salvado

, Szoeke

, Macaulay

, Martins

, Maruff

, Ames

, Rowe

, Masters

(2013) Amyloid β deposition, neurodegeneration, and cognitive decline in sporadic Alzheimer’s disease: A prospective cohort study. Lancet Neurol 12, 357–367.

20.

Yotter

, Doshi

, Clark

, Sojkova

, Zhou

, Wong

, Ferrucci

, Resnick

, Davatzikos

(2013) Memory decline shows stronger associations with estimated spatial patterns of amyloid deposition progression than total amyloid burden. Neurobiol Aging 34, 2835–2842.

21.

Resnick

, Sojkova

, Zhou

, An

, Ye

, Holt

, Dannals

, Mathis

, Klunk

, Ferrucci

, Kraut

, Wong

(2010) Longitudinal cognitive decline is associated with fibrillar amyloid-beta measured by [11C]PiB. Neurology 74, 807–815.

22.

Besson

, La Joie

, Doeuvre

, Gaubert

, Mezenge

, Egret

, Landeau

, Barre

, Abbas

, Ibazizene

, de La Sayette

, Desgranges

, Eustache

, Chetelat

(2015) Cognitive and brain profiles associated with current neuroimaging biomarkers of preclinical Alzheimer’s disease. J Neurosci 35, 10402–10411.

23.

Toledo

, Bjerke

, Chen

, Rozycki

, Jack

Jr , Weiner

, Arnold

, Reiman

, Davatzikos

, Shaw

, Trojanowski

, Alzheimer’s Disease Neuroimaging Inititaive (2015) Memory, executive, and multidomain subtle cognitive impairment: Clinical and biomarker findings. Neurology 85, 144–153.

24.

Wirth

, Villeneuve

, Haase

, Madison

, Oh

, Landau

, Rabinovici

, Jagust

(2013) Associations between Alzheimer disease biomarkers, neurodegeneration, and cognition in cognitively normal older people. JAMA Neurol 70, 1512–1519.

25.

Doré

, Villemagne

, Bourgeat

, Fripp

, Acosta

, Chetélat

, Zhou

, Martins

, Ellis

, Masters

, Ames

, Salvado

, Rowe

(2013) Cross-sectional and longitudinal analysis of the relationship between Aβ deposition, cortical thickness, and memory in cognitively unimpaired individuals and in Alzheimer disease. JAMA Neurol 70, 903–911.

26.

Buckner

, Snyder

, Shannon

, LaRossa

, Sachs

, Fotenos

, Sheline

, Klunk

, Mathis

, Morris

, Mintun

(2005) Molecular, structural, and functional characterization of Alzheimer’s disease: Evidence for a relationship between default activity, amyloid, and memory. J Neurosci 25, 7709–7717.

27.

Fleisher

, Sherzai

, Taylor

, Langbaum

, Chen

, Buxton

(2009) Resting-state BOLD networks versus task-associated functional MRI for distinguishing Alzheimer’s disease risk groups. Neuroimage 47, 1678–1690.

28.

Kennedy

, Rodrigue

, Devous

, Hebrank

, Bischof

, Park

(2012) Effects of beta-amyloid accumulation on neural function during encoding across the adult lifespan. Neuroimage 62, 1–8.

29.

, Madison

, Villeneuve

, Markley

, Jagust

(2014) Association of gray matter atrophy with age, β-amyloid, and cognition in aging. Cereb Cortex 24, 1609–1618.

30.

Ahn

, Seo

, Chin

, Suh

, Lee

, Kim

, Im

, Lee

, Heilman

, Na

(2011) The cortical neuroanatomy of neuropsychological deficits in mild cognitive impairment and Alzheimer’s disease: A surface-based morphometric analysis. Neuropsychologia 49, 3931–3945.

31.

Schiratti

, Allassonniere

, Colliot

, Durrleman

(2015) Learning spatiotemporal trajectories from manifold-valued longitudinal data. In, Advances in Neural Information Processing Systems 28, Cortes

, Lawrence

, Lee

, Sugiyama

, Garnett

, eds., pp. 2395–2403.

32.

Schulam

, Wigley

, Saria

(2015) Clustering longitudinal clinical marker trajectories from electronic health data: Applications to phenotyping and endotype discovery. In AAAI Conference on Artificial Intelligence, pp. 2956–2964.

33.

Donohue

, Jacqmin-Gadda

, Le Goff

, Thomas

, Raman

, Gamst

, Beckett

, Jack

Jr , Weiner

, Dartigues

, Aisen

, (2014) Alzheimer’s Estimating long-term multivariate progression from short-term data. Alzheimers Dement 10, 400–S410.

34.

Schmidt-Richberg

, Guerrero

, Ledig

, Molina-Abril

, Frangi

, Rueckert

, Alzheimer’s Disease Neuroimaging Initiative (2015) Multi-stage biomarker models for progression estimation in Alzheimer’s disease. Inf Process Med Imaging 24, 387–398.

35.

Young

, Oxtoby

, Daga

, Cash

, Fox

, Ourselin

, Schott

, Alexander

(2014) A data-driven model of biomarker changes in sporadic Alzheimer’s disease. Brain 137, 2564–2577.

36.

Jedynak

, Lang

, Liu

, Katz

, Zhang

, Wyman

, Raunig

, Jedynak

, Caffo

, Prince

(2012) A computational neurodegenerative disease progression score: Method and results with the Alzheimer’s Disease Neuroimaging Initiative cohort. Neuroimage 63, 1478–1486.

37.

Bilgel

, Prince

, Wong

, Resnick

, Jedynak

(2016) A multivariate nonlinear mixed effects model for longitudinal image analysis: Application to amyloid imaging. Neuroimage 134, 658–670.

38.

Shock

, Greulich

, Andres

, Arenberg

, Costa

Jr , Lakatta

, Tobin

(1984) Normal human aging: The Baltimore Longitudinal Study of Aging, U.S. Government Printing Office, Washington, DC.

39.

Sager

, Hermann

, La

Rue A

(2005) Middle-aged children of persons with Alzheimer’s disease: APOE genotypes and cognitive function in the Wisconsin Registry for Alzheimer’s Prevention. J Geriatr Psychiatry Neurol 18, 245–249.

40.

Reitan

(1992) Trail making test: Manual for administration and scoring, Reitan Neuropsychological Laboratory, Tucson, AZ.

41.

Wechsler

(1981) Wechsler adult intelligence scale-revised, Psychological Corporation, San Antonio, TX.

42.

Wechsler

(1997) Wechsler adult intelligence scale, The Psychological Corporation, San Antonio, TX.

43.

Delis

, Kramer

, Kaplan

, Ober

(1987) The California verbal learning test, Psychological Corporation, San Antonio, TX.

44.

Lezak

, Howieson

, Loring

(2004) Neuropsychological assessment, Oxford University Press, London.

45.

Kaplan

, Goodglass

, Weintraub

(1983) Boston naming test, Lea & Febiger, Philadelphia, PA.

46.

Folstein

, Folstein

, McHugh

(1975) “Mini-mental state”: A practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res 12, 189–198.

47.

Newcombe

(1969) Missile wounds of the brain: A study of psychological deficits, Oxford University Press, Oxford, UK.

48.

Fuld

(1978) Psychological testing in the differential diagnosis of the dementias. In Alzheimer’s disease: Senile dementia and related disorders, Katzman

, Terry

, Bick

, eds. Raven Press, New York, NY, pp. 185–193.

49.

Morris

(1993) The Clinical Dementia Rating (CDR): Current version and scoring rules. Neurology 43, 2412–2414.

50.

American Psychiatric Association (1987) Diagnostic and Statistical Manual of Mental Disorders: DSM-III-R, American Psychiatric Association, Washington, DC.

51.

McKhann

, Drachman

, Folstein

, Katzman

, Price

, Stadlan

(1984) Clinical diagnosis of Alzheimer’s disease: Report of the NINCDS-ADRDA work group under the auspices of Department of Health and Human Services Task Force on Alzheimer’s Disease. Neurology 34, 939–944.

52.

Petersen

(2004) Mild cognitive impairment as a diagnostic entity. J Intern Med 256, 183–194.

53.

Clark

, Koscik

, Nicholas

, Okonkwo

, Engelman

, Bratzke

, Hogan

, Mueller

, Bendlin

, Carlsson

, Asthana

, Sager

, Hermann

, Johnson

(2016) Mild Cognitive Impairment in late middle age in the Wisconsin Registry for Alzheimer’s Prevention (WRAP) study: Prevalence and characteristics using robust and standard neuropsychological normative data. Arch Clin Neuropsychol 31, 675–688.

54.

Koscik

, La Rue

, Jonaitis

, Okonkwo

, Johnson

, Bendlin

, Hermann

, Sager

(2014) Emergence of mild cognitive impairment in late middle-aged adults in the Wisconsin Registry for Alzheimer’s Prevention. Dement Geriatr Cogn Disord 38, 16–30.

55.

Wechsler

(1987) Wechsler memory scale-revised, Psychological Corporation, San Antonio, TX.

56.

Petersen

, Aisen

, Beckett

, Donohue

, Gamst

, Harvey

, Jack

Jr , Jagust

, Shaw

, Toga

(2010) Alzheimer’s Disease Neuroimaging Initiative (ADNI) clinical characterization. Neurology 74, 201–209.

57.

Albert

, DeKosky

, Dickson

, Dubois

, Feldman

, Fox

, Gamst

, Holtzman

, Jagust

, Petersen

, Snyder

, Carrillo

, Thies

, Phelps

(2011) The diagnosis of mild cognitive impairment due to Alzheimer’s disease: Recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimers Dement 7, 270–279.

58.

Clark

, Racine

, Koscik

, Okonkwo

, Engelman

, Carlsson

, Asthana

, Bendlin

, Chappell

, Nicholas

, Rowley

, Oh

, Hermann

, Sager

, Christian

, Johnson

(2016) Beta-amyloid and cognitive decline in late middle age: Findings from the Wisconsin Registry for Alzheimer’s Prevention study. Alzheimers Dement 12, 805–814.

59.

McCarrey

, An

, Kitner-Triolo

, Ferrucci

, Resnick

(2016) Sex differences in cognitive trajectories in clinically normal older adults. Psychol Aging 31, 166–175.

60.

Corder

, Lannfelt

, Bogdanovic

, Fratiglioni

, Mori

(1998) The role of APOE polymorphisms in late-onset dementias. Cell Mol Life Sci 54, 928–934.

61.

Liu

, Kanekiyo

, Xu

, Bu

(2013) Apolipoprotein E and Alzheimer disease: Risk, mechanisms and therapy. Nat Rev Neurol 9, 106–118.

62.

Kurt

, Yener

, Oguz

(2011) Impaired digit span can predict further cognitive decline in older people with subjective memory complaint: A preliminary result. Aging Ment Health 15, 364–369.

63.

Monsell

, Mock

, Hassenstab

, Roe

, Cairns

, Morris

, Kukull

(2014) Neuropsychological changes in asymptomatic persons with Alzheimer disease neuropathology. Neurology 83, 434–440.

64.

Kiewel

, Wisdom

, Bradshaw

, Pastorek

, Strutt

(2012) A retrospective review of digit span-related effort indicators in probable Alzheimer’s disease patients. Clin Neuropsychol 26, 965–974.

65.

Loring

, Goldstein

, Chen

, Drane

, Lah

, Zhao

, Larrabee

(2016) False-positive error rates for reliable digit span and auditory verbal learning test performance validity measures in amnestic mild cognitive impairment and early Alzheimer disease. Arch Clin Neuropsychol 31, 313–331.

66.

Bossers

, van der Woude

, Boersma

, Scherder

, van Heuvelen

(2012) Recommended measures for the assessment of cognitive and physical performance in older patients with dementia: A systematic review. Dement Geriatr Cogn Dis Extra 2, 589–609.

67.

Hoshi

, Oda

, Wada

, Ito

, Yamashita

, Oda

, Ohta

, Yamada

, Tamura

(2000) Visuospatial imagery is a fruitful strategy for the digit span backward task: A study with near-infrared optical tomography. Cogn Brain Res 9, 339–342.

68.

Gerton

, Brown

, Meyer-Lindenberg

, Kohn

, Holt

, Olsen

, Berman

(2004) Shared and distinct neurophysiological components of the digits forward and backward tasks as revealed by functional neuroimaging. Neuropsychologia 42, 1781–1787.

69.

Aleman

, van’t Wout

(2008) Repetitive transcranial magnetic stimulation over the right dorsolateral prefrontal cortex disrupts digit span task performance. Neuropsychobiology 57, 44–48.