Digital Technology Differentiates Graphomotor and Information Processing Speed Patterns of Behavior

Abstract

Background:

Coupling digital technology with traditional neuropsychological test performance allows collection of high-precision metrics that can clarify and/or define underlying constructs related to brain and cognition.

Objective:

To identify graphomotor and information processing trajectories using a digitally administered version of the Digit Symbol Substitution Test (DSST).

Methods:

A subset of Long Life Family Study participants (n = 1,594) completed the DSST. Total time to draw each symbol was divided into ‘writing’ and non-writing or ‘thinking’ time. Bayesian clustering grouped participants by change in median time over intervals of eight consecutively drawn symbols across the 90 s test. Clusters were characterized based on sociodemographic characteristics, health and physical function data, APOE genotype, and neuropsychological test scores.

Results:

Clustering revealed four ‘thinking’ time trajectories, with two clusters showing significant changes within the test. Participants in these clusters obtained lower episodic memory scores but were similar in other health and functional characteristics. Clustering of ‘writing’ time also revealed four performance trajectories where one cluster of participants showed progressively slower writing time. These participants had weaker grip strength, slower gait speed, and greater perceived physical fatigability, but no differences in cognitive test scores.

Conclusion:

Digital data identified previously unrecognized patterns of ‘writing’ and ‘thinking’ time that cannot be detected without digital technology. These patterns of performance were differentially associated with measures of cognitive and physical function and may constitute specific neurocognitive biomarkers signaling the presence of subtle to mild dysfunction. Such information could inform the selection and timing of in-depth neuropsychological assessments and help target interventions.

Keywords

Aging bayesian approach boston process approach digit symbol substitution test executive function graphomotor performance neuropsychological tests

INTRODUCTION

Neurodegenerative processes underlying Alzheimer’s disease and other related dementias begin years, if not decades, before they are clinically evident [1, 2]. It is, therefore, imperative that clinical neuropsychologists have assessment tools that are able to detect emergent neurodegenerative change as early as possible so that research participants and patients referred for clinical evaluation can be correctly classified and diagnosed for needed services, including recruitment into possible clinical trials and future disease-modifying interventions.

The goal of identifying early neurodegenerative alterations is consistent with the Boston Process Approach (BPA) of neuropsychological assessment, which argues that the errors and strategies an individual uses to complete the task at hand is often more informative in elucidating underlying brain-behavior relationships compared to a simple, single summary test score [3, 4]. For example, a person with subtle brain changes may achieve a summary score that is statistically within the normal or in the acceptable range, but only with considerable effort, along with many self-corrected errors. Traditional, quantitative scoring would classify such performance as unimpaired. However, the ability to quantify the process or how the task was completed could reveal subtle neurocognitive inefficiencies. The BPA explicitly emphasizes the multifactorial nature of neuropsychological tests in that most tests rely on multiple skills for successful completion. Therefore, impaired performance may be due to a multitude of reasons. As such, the BPA promotes meticulous recording of a person’s performance as a means to identify underlying affected cognitive processes [4]. Digital assessment technology facilitates the application of the BPA and thus, has the potential to be a more sensitive and specific method of detecting subtle, emergent cognitive impairment, particularly among those presumed to be asymptomatic based on traditional measures.

The Digit Symbol Substitution Test [5] or the Digit Symbol Test (DSST) is one of the original subtests used by Wechsler to construct his well-known intelligence scales. It is a timed test assessing graphomotor information processing speed where participants are asked to fill in boxes with a symbol from a number-symbol key at the top of the page as quickly as possible. The traditional test score is calculated on the basis of the total number of correctly transcribed symbols in 90 or 120 s. This test is well-known to be one of the most sensitive tests for detecting cognitive dysfunction [6, 7] because successful performance requires the coordination of multiple domains, including cognitive (e.g., attention, processing speed, episodic memory, and perception) and non-cognitive factors (e.g., motor speed and dexterity, visual scanning, and motivation). Deficits in any of these abilities could contribute to overall DSST test difficulty [6, 8]. However, the multifactorial nature of the DSST test also contributes to the test’s lack of specificity as to the source of impairment [6].

In line with implementing a process approach that might disambiguate the source of difficulty on this test, additional DSST subtests and testing procedures have been utilized [9]. For example, a symbol copy task asks participants to simply copy, rather than transcribe, a series of symbols to isolate the contribution of motor deficits. An incidental learning task is available where individuals write the digit symbol pairs that were learned after completing the original coding task to shed light on whether impaired associative learning may have hindered performance. Finally, to identify change in test performance, recording performance in 30-s intervals can capture deficits in establishing set (i.e., understanding and engaging in the task) and fatigue over the course of the test. Adding these subtests, however, increases testing burden and adds more time-consuming post-scoring procedures.

Interfacing standard DSST performance with digital pen technology obviates many of these problems (see [10] for review). The integration of recording output using a digital pen during neuropsychological test performance allows the examiner to capture data that is rich in process information that would otherwise be labor-intensive or impossible to collect. Digital pens can be used in place of an ordinary ballpoint pen to record time-stamped data throughout the task. To date, the majority of published work reporting use of the digital pen with traditional neuropsychological tests has been with the Clock Drawing Test (CDT). Software parses the total time to completion into time spent actually drawing (called ‘ink time’) versus time not spent drawing (called ‘think time’ [11]). Studies show that digitally captured ‘think time’ data can differentiate cognitively healthy adults from participants with mild cognitive impairment (MCI) or early dementia [12], cerebral small vessel disease [13], and depression [14]. Metrics extracted from the digital pen data are also able to capture additional facets of graphomotor production (e.g., number of pen strokes) and processing speed (e.g., decision-making latencies prior to individual components that are drawn) that differ by age [15] and clinical population [16, 17].

Here, we present an analysis of performance on the DSST captured using a digital pen among participants in a study of longevity. The DSST involves real-time learning of novel information whereas previous analyses of digital pen data have focused on the CDT which draws upon well-learned information (i.e., semantic knowledge of a clock). We hypothesized that digital DSST data would allow us to identify participants who get faster or slower during the test. Different thinking and writing time patterns of behavior may be associated with cognitive and physical functions that underlie test performance. Therefore, the aims of this study were: 1) to determine whether there are distinct patterns of performance for ‘thinking time’ and ‘writing time’ during the test; and 2) to assess whether and/or how these patterns correlate with measures assessing neurocognitive abilities versus health and functional outcome measures.

MATERIALS AND METHODS

Participants

Participants were from the Long Life Family Study (LLFS), a family study of healthy aging and longevity. Details of participant selection and study measures have been previously described (see [18, 19]). Briefly, participants were selected based on evidence of familial longevity as determined by a family longevity selection score which rated the survival exceptionality of all members of the proband generation [19]. Families selected to participate had at least two siblings in the proband generation and an individual in the offspring generation, who all had capacity to provide consent and were willing to participate in an in-home assessment. All interested family members from the proband and offspring generations were recruited. Informed consent was obtained from all study participants. To date, 4,953 participants from 539 families have completed up to two in-person assessments approximately eight years apart: Visit 1 (performed 2006–2009) and Visit 2 (performed 2014–2017). At Visit 1 participants in the proband generation were 90 years of age on average (range 55 to 110 years) and those in the offspring generation were 61 years of age (range 25 to 88 years). All participants provided written informed consent prior to participation. The Institutional Review Boards at all of the Field Centers (Boston University, Columbia University, and University of Pittsburgh) and the Data Management and Coordinating Center (Washington University, St. Louis) in the United States reviewed and approved this project and the regional Institutional Review Board in Denmark reviewed and approved this project.

Health and function data

At each visit, participants completed questionnaires to provide sociodemographic data, self-reported diagnoses of medical conditions, and ability to perform activities of daily living. The questionnaires also included the Pittsburgh Fatigability Scale [20, 21] (at Visit 2 and annually thereafter), a measure of perceived physical and mental fatigability in older adults with higher scores (0–50 for both subscales) reflecting greater fatigability. Health and function assessments included the Short Physical Performance Battery (comprising balance assessment, gait speed, and chair stands), grip strength, anthropometrics (e.g., height, weight, and abdominal circumference), blood pressure, spirometry, a carotid ultrasound (at Visit 2 only), a cognitive assessment, and the 10 item Center for Epidemiologic Studies Depression Scale (CES-D). Spirometry was used to measure lung function (i.e., forced expiratory volume during the first second of exhalation, FEV1). The carotid ultrasound examination was used to measure intima-media thickness (IMT) of the common carotid artery, a marker of subclinical vascular disease. Examiners also recorded the participant’s current medications. A blood draw was performed for genetic and laboratory analyses and biobanking at the central laboratory of the LLFS at the University of Minnesota. Participants were contacted every one to three years to collect information on vital status, changes in health, medications, and daily activities and to administer a brief test of cognitive function, a modified version of the Telephone Interview for Cognitive Status (TICS) [22, 23].

Cognitive assessment

At Visit 2 the cognitive assessment included the Mini-Mental Status Examination (MMSE), Hopkins Verbal Learning Test –Revised (HVLT), the Clock Drawing Test (CDT) command and copy conditions, Trail Making Tests Parts A and B, Logical Memory from the Wechsler Memory Scale –Revised, Number Span Test from the National Alzheimer’s Coordinating Center Uniform Data Set, Digit Symbol Substitution Test (DSST) from the Wechsler Adult Intelligence Scale –Revised, semantic fluency (animals), and phonemic fluency (i.e., the Controlled Oral Word Association Test; letters F,A,S) and the assignment of a Clinical Dementia Rating score. Participants used a digital pen for all written responses to test items (e.g., MMSE pentagons and sentence, CDT, Trails A&B, DSST). The digital pen looks and writes like a wide-body ballpoint pen but has a small digital camera aimed at the tip of the pen that is able to capture movements across specially formatted paper. The camera records x and y coordinates of the pen’s position on the paper at 13-ms intervals allowing for playback of the participant’s process of drawing or writing. Digital pen data for the DSST were analyzed in this study. DSSTs with a score of zero, or noted to be affected by hearing, vision, inability to write, participant refusal, experimenter error, environmental distractions, or physical limitations (n = 62) were omitted from the analyses.

Extraction of digital metrics

An in-person Visit 2 assessment with digital pen data collection of the DSST was completed by 2,634 participants. A loss of 303 digital pen files occurred due to improper storage and therefore, a total of 2,331 unique DSST digital pen files were converted to.txt files that included separate records for the coordinates and time for each pen stroke, where a new stroke started each time the pen was placed on the paper. We prioritized the extraction of thinking time, which based on evidence from other digitally-administered tests, may be a valuable metric for detecting differences in cognition in aging [15] and early Alzheimer’s disease [12]. In line with the BPA, we aimed to capture behaviors that are elicited specifically by the DSST; and, therefore focused on the analysis of change in performance over a repetitive task which may inform about set engagement, associative learning, and fatigue [3]. To assign each stroke to one of the boxes of the DSST exam grid, the exam grid was first traced using the digital pen to generate the x and y coordinates of the corners of each of the 93 boxes and then these x and y coordinates were widened by 5%to account for pen strokes written slightly outside a box (Fig. 1). Starting from the first box, each pen stroke was then assigned to a box if at least 75%of the graphic output for a stroke (i.e., the x and y coordinates sampled at 13-ms intervals) was in the box. Participants who had strokes that could not be assigned to a box or filled in boxes out of order, were excluded from the analysis. There were 23 participants who were excluded because they did not complete any segments (i.e., completed less than 8 boxes). In total, 1,594 participant files were included in the analysis. As shown in Supplementary Table 1, participants whose pen files were excluded were older (mean age 74.7±12.3 versus 70.2±10.0 years, p < 0.0001), and more likely to be female (58.2%versus 53.4%, p = 0.03). Excluded participants also differed in terms of educational attainment (p = 0.02) and field center (p < 0.0001). Excluded participants did not differ significantly on any of the other health or functional characteristics examined in the following analyses.

Fig. 1

Assignment of pen strokes to boxes.

Next, the ‘thinking’ or non-writing time prior to the transcription of the response into each box and ‘writing’ time for each box were determined. The ‘thinking’ time prior to each box was defined as the time from the lifting of the pen in the prior box to the placement of the pen in the next box. Thinking time was not assigned to the first box. The ‘writing’ time for a box was defined as the sum of the time writing all strokes assigned to the box.

Statistical analysis

Bayesian model-based clustering was used to group participants based on 1) median ‘thinking’ time over non-overlapping segments of eight consecutive boxes, and 2) median ‘writing’ time over the same segments of eight consecutive boxes. The number of completed segments varied across participants (range 1–10) (distribution shown in Supplementary Table 2). The advantage of Bayesian model-based clustering is that it allows for analyzing trajectories of differing lengths (e.g., number of completed segments of consecutive boxes), it provides parametric forms of the cluster-specific trajectories, and inference on the parameters is based on their marginal distribution rather than their conditional distribution that depends on nuisance parameters [24]. The outcome in this analysis was based on the median ‘thinking’ time or ‘writing’ time over all segments of multiple consecutive boxes to smooth the data. Segment lengths of four and eight boxes were considered, and eight box segments were chosen for analysis as they sufficiently smoothed the data. Outcome variables were normalized by subtracting the median time for the first segment from the median time of each subsequent segment to assess patterns of change independent of the initial time.

Linear regression was used to model the outcome as a function of the segment number, assessing both linear and quadratic models. No intercept term was included as the first value of the outcome was zero for all participants by design. Accordingly, the initial zero values were omitted from the regressions. In model-based clustering, the coefficients for each term in the model are allowed to differ by cluster and uninformative priors were assumed for all parameters. For a fixed number of clusters, the model was fit via Markov Chain Monte Carlo using 5,000 samples after discarding the first 1,500 samples. Model fitting started with both linear and quadratic models assuming two clusters, and then the number of clusters was increased until the deviance information criterion (DIC) began to increase. The best fitting linear model was compared with the best fitting quadratic model, and the model with the lower DIC was selected as the final model. DIC is ideally suited for selection of mixture models with hidden variables because of the correction by the effective number of parameters [25]. Summary distributions of the cluster membership suggest that cluster allocation was very accurate, see Supplementary Table 3.

Cluster membership was assessed for associations with sociodemographic characteristics, health and physical function data, APOE genotype, and neuropsychological test scores. All comparisons were adjusted for age, sex, education, and field center (Boston, Denmark, New York, or Pittsburgh). Continuous variables were compared by fitting linear models, and categorical variables by fitting logistic regression models. To account for family relatedness, these models were fit using generalized estimating equations with an exchangeable working correlation matrix. Time to death was compared by fitting a Cox Proportional Hazards model, adjusting for age, sex, education, and field center, and using a robust sandwich estimate of the covariance matrix. Change in TICS score (as scored in [26] which includes tasks of counting backwards, subtracting sevens, and immediate and delayed recall of a word list, score range 0–27) over follow-up after the Visit 2 assessment was modeled with a linear mixed effects model, adjusting for age, sex, education, field center, follow-up time, cluster, and an interaction between cluster and follow-up time. A random intercept and slope for follow-up time were included to account for the repeated measures. Finally, agreement between thinking time and writing time cluster membership was assessed using a weighted kappa statistic with linear weights. Analyses were conducted in RStudio, using the package rjags, and SAS v9.4.

RESULTS

Table 1 shows demographic characteristics, DSST scores, and digital metrics for the 1,594 participants included in the analysis. Participants were 70 years of age on average, ranging from 45 to 103 years, 52%female, predominantly non-Hispanic, white (99%), highly educated (61%with a college degree or advanced degree), and generally cognitively healthy (median MMSE score = 29.0, IQR 29.0-30.0). Average ‘thinking’ time and ‘writing’ time were strongly associated with DSST raw score (Pearson correlation coefficient –0.79 and –0.64, respectively, both p < 0.0001). Supplementary Table 3 shows that DSST raw score, average ‘thinking’ time, and average ‘writing’ time were generally associated with all of the health and functional characteristics, with the exception of grip strength and gait speed.

Table 1

Participant characteristics of the Long Life Family Study sample, n = 1,594

	Mean±SD or n (%)
Age, y	70.2±10.0
Sex, female	833 (52.3)
Race, white	1579 (99.0)
Ethnicity, Hispanic^*	4 (0.3)
Highest level of education
Up to a high school degree	616 (38.6)
College degree	608 (38.1)
Advanced degree	370 (23.2)
MMSE score^∧, 0–30	29.0 (29.0-30.0)
DSST score, 0–93	46.3±12.6
DSST writing time, s	34.4±9.6
DSST thinking time, s	56.1±10.2
Follow-up time^∧, y	2.7 (1.9–3.2)
Deceased over follow-up	82 (5.1)

^*64 participants were missing ethnicity data. ^∧Reported as Median (Q1-Q3). MMSE, Mini-Mental State Examination, maximum score is 30; DSST, Digit Symbol Substitution Test.

Thinking time trajectories

Based on DIC, a model with four clusters and a linear term for segment number was the best fit to the ‘thinking’ time data (Fig. 2). The four patterns of ‘thinking’ time performance trajectories across the 90 s of the DSST are shown in Fig. 3. Most participants had relatively stable trajectories with only minor within-task changes in ‘thinking’ time and fell into one of two clusters; individuals in cluster 2 (n = 476) had a slight increase in ‘thinking’ time and individuals in cluster 3 (n = 857) had a slight decrease in ‘thinking’ time (Table 2). In contrast clusters 1 and 4 showed greater within-task changes in ‘thinking’ time. Cluster 1 (n = 168) comprised “slowing thinkers”, individuals who performed at an average rate at the beginning of the test and increased their ‘thinking’ time across subsequent intervals. Cluster 4 (n = 93), the “quickening thinkers”, showed the opposite pattern with a significantly longer ‘thinking’ time in the first segment (1.9 versus 1.1 s, p < 0.0001), and a decrease in ‘thinking’ time across the intervals.

Fig. 2

Deviance Information Criterion (DIC) values for number of model clusters.

Fig. 3

Thinking time trajectory clusters. Clusters of participants with similar patterns of change in thinking time per segment during the Digit Symbol Substitution Test (color/black lines) and their fitted trajectories (black lines) are shown overlaid on the data for all participants (gray lines). The intercepts for the fitted trajectories were chosen so that the lines pass through the center of the data for each cluster.

Table 2

Regression coefficients and 95%credible intervals from model-based clustering of thinking and writing time, n = 1,594

Outcome	Cluster 1 Slowing	Cluster 2 Stable-slowing	Cluster 3 Stable-quickening	Cluster 4 Quickening
Thinking time	0.131 (0.116; 0.146)	0.006 (0.003; 0.010)	–0.044 (–0.051; –0.038)	–0.137 (–0.164; –0.118)
Writing time	0.129 (0.057; 0.144)	0.012 (–0.009; 0.018)	–0.021 (–0.035; –0.019)	–0.068 (–0.109; –0.061)

Regression coefficients from models of thinking time or writing time as a linear function of segment number.

Descriptive characteristics of the ‘thinking’ time trajectory clusters (Table 3) revealed similar health and functional parameters between the two stable clusters (clusters 2 and 3) as well as between the clusters of slowing and quickening thinkers (clusters 1 and 4). The clusters of slowing and quickening thinkers (clusters 1 and 4, respectively) were older (p < 0.0001), had a lower DSST raw score (p < 0.0001) and HVLT Total Recall score (p = 0.04), took longer to complete the CDT (command condition p = 0.01; copy condition p < 0.001, results not shown), and trended toward lower scores on HVLT Delayed Recall (p = 0.05). These clusters did not differ on measures of physical function. Over a median of 2.6 years of follow up, the cluster of quickening thinkers (cluster 4) showed a significant decline in TICS of about half a point per year (–0.64, SE 0.29, p = 0.03) with the slowing thinkers (cluster 1) showing a similar trend (–0.45, SE 0.24, p = 0.06). In reference to cluster 3 (i.e., the stable-quickeners), the quickening thinkers (cluster 4) had a significantly lower mortality rate (adjusted hazard ratio [aHR] 0.33, 95%confidence interval [CI] 0.12–0.94, p = 0.04) whereas the other clusters had similar mortality to cluster 3.

Table 3

Health and functional characteristics of thinking time trajectory clusters

					p
	Cluster 1 Slowing	Cluster 2 Stable-slowing	Cluster 3 Stable-quickening	Cluster 4 Quickening
Cluster size	168 (10.5)	476 (29.9)	857 (53.8)	93 (5.8)
Avg. time 1st segment, s	1.1 ± 0.5	1.3 ± 0.4	1.1 ± 0.6	1.9 ± 0.7	< 0.0001
Demographics
Age, y	74.1 ± 8.9	70.3 ± 9.6	68.8 ± 9.8	75.9 ± 10.7	< 0.0001
Sex, female	73 (44.0)	260 (55.8)	455 (54.4)	45 (48.9)	0.15
Highest level of education					0.28
Up to a high school degree	79 (47.0)	170 (35.7)	312 (36.4)	55 (59.1)
College degree	55 (32.7)	182 (38.2)	348 (40.6)	23 (24.7)
Advanced degree	34 (20.2)	124 (26.1)	197 (23.0)	15 (16.1)
Field center (row %)					0.007
Boston	36 (8.1)	124 (27.8)	261 (58.5)	25 (5.6)
Denmark	87 (15.0)	168 (29.0)	293 (50.5)	32 (5.5)
New York	19 (10.3)	57 (30.8)	98 (53.0)	11 (6.0)
Pittsburgh	26 (6.8)	127 (33.2)	205 (53.5)	25 (6.5)
Long-lived family	118 (71.1)	617 (73.8)	347 (74.5)	74 (80.4)	0.57
Health/comorbidities
Stroke	3 (1.8)	12 (1.4)	7 (1.5)	2 (2.2)	1.00
Diabetes	10 (6.0)	31 (3.7)	19 (4.1)	7 (7.7)	0.46
Depression	22 (13.6)	151 (18.7)	75 (16.6)	16 (17.8)	0.79
Hypertension	57 (34.6)	233 (28.0)	135 (29.2)	40 (43.5)	0.55
Atrial fibrillation	8 (4.9)	26 (3.2)	13 (2.9)	2 (2.2)	0.71
Body mass index, kg/m²	27.2±4.6	27.3±4.3	27.2±4.5	27.1±4.8	0.96
Carotid IMT, mm	0.88±0.15	0.83±0.16	0.84±0.15	0.89±0.16	0.70
CES-D score≥14	5 (3.0)	6 (1.3)	15 (1.8)	3 (3.2)	0.15
Physical function
Grip strength, kg	29.5±10.4	29.3±11.1	28.7±10.2	26.3±11.4	0.53
Gait speed, m/s	1.00±0.26	1.04±0.24	1.04±0.25	0.94±0.24	0.13
PFS mental score, 0–50	8.1±8.5	7.0±7.7	6.7±7.1	10.2±8.8	0.16
PFS physical score, 0–50	13.7±9.3	12.3±8.2	12.3±8.5	15.2±8.8	0.69
FEV1, liters	2.4±0.7	2.5±0.8	2.5±0.8	2.4±0.7	0.23
Genetic risk factors
APOE E2 allele carrier	30 (18.6)	145 (18.1)	77 (17.3)	19 (21.6)	0.86
APOE E4 allele carrier	31 (19.3)	174 (21.7)	101 (22.8)	17 (19.3)	0.73
Cognitive function
MMSE<24	4 (2.5)	11 (1.3)	4 (0.9)	0 (0.0)	0.77
DSST score	37.1 ± 8.7	47.9 ± 11.1	48.2 ± 13.1	37.4 ± 10.1	< 0.0001
Logical Memory Immediate	13.5±4.6	14.1±4.2	13.9±3.9	13.1±4.3	0.84
Logical Memory Delayed	11.6±5.2	12.7±4.6	12.4±4.3	11.4±5.1	0.71
HVLT Total Recall	21.9 ± 6.1	24.1 ± 5.6	24.7 ± 5.9	21.8 ± 5.8	0.04
HVLT Delayed Recall	6.3 ± 3.7	7.7 ± 3.5	8.0 ± 3.4	6.7 ± 3.5	0.05
Semantic fluency	20.4±6.2	22.3±6.3	21.8±5.9	19.4±6.7	0.19
Phonemic fluency	34.0±13.3	39.0±13.4	38.4±13.8	33.5±14.8	0.13
CDT Command thinking time^∧	22.1 (17.8–30.8)	18.0 (14.1–23.3)	18.0 (13.8–23.9)	21.8 (16.4–30.0)	0.02
CDT Command writing time^∧	14.8 (12.2–18.2)	12.4 (10.3–15.9)	13.0 (10.4–15.7)	14.1 (11.7–16.3)	0.02
CDT Command total time^∧	37.2 (31.0–48.0)	30.9 (25.6–39.2)	30.9 (25.3–39.8)	35.7 (29.7–46.8)	0.01
Clinical Dementia Rating score (> 0)	21 (12.7)	52 (6.4)	27 (5.9)	13 (14.3)	0.10
Longitudinal outcomes
Mortality^* (HR)	1.02 (CI 0.55, 1.92); p = 0.94	0.69 (CI 0.40, 1.19); p = 0.19	Ref	0.33 (CI 0.12, 0.94); p = 0.04	0.13
TICS^** (slope for follow up time)	–0.45 (SE 0.24; p = 0.06)	0.08 (SE 0.14; p = 0.59)	0.07 (SE 0.11; p = 0.51)	–0.64 (SE 0.29; p = 0.03)	0.03^†

Mean±SD for continuous variables; counts (%) for categorical variables. p-values for comparisons adjusted for age, sex, education level, and field center. ^∧Median (Q1-Q3). p-value based on log-transformed variable. ^*Modeling time to death, adjusted for age, education level, and field center and stratified on sex. ^**Modeling TICS score as a function of age, sex, education level, field center, follow-up time, cluster, and an interaction of cluster and follow-up time. The p-value in each column is for a test that the slope is equal to zero. N = 1,014, as the Denmark field center did not perform the TICS exam. ^†p-value for cluster ^*follow-up time interaction. PFS, Pittsburgh Fatigability Scale; FEV1, Forced expiratory volume; IMT, intima-media thickness; MMSE, Mini-Mental State Examination; DSST, Digit Symbol Substitution Test; HVLT, Hopkins Verbal Learning Test-Revised; CDT, Clock Drawing Test; CESD, Center for Epidemiologic Studies Depression Scale; TICS, Telephone Interview for Cognitive Status.

Writing time trajectories

Similar to ‘thinking’ time, a model with four clusters and a linear term for segment number was the best fit to the ‘writing’ time data based on DIC (Fig. 2). Most participants demonstrated relatively stable ‘writing’ time across the test represented by clusters 2 (n = 181) and 3 (n = 1142) as shown in Fig. 4. A small number of individuals (n = 26, cluster 1) demonstrated slowing ‘writing’ speed across the test, spending more time writing in the last segment of 8 completed boxes than during the first segment. Cluster 4 (n = 245) showed a quickening trajectory in which individuals spent more time writing symbols in the first segment of 8 boxes in comparison with individuals in the other three clusters and then spent less and less time writing symbols as the test progressed.

Fig. 4

Writing time trajectory clusters. Clusters of participants with similar patterns of change in writing time per segment during the Digit Symbol Substitution Test (color/black lines) and their fitted trajectories (black lines) are shown overlaid on the data for all participants (gray lines). The intercepts for the fitted trajectories were chosen so that the lines pass through the center of the data for each cluster.

Health and functional characteristics were generally similar between the two stable ‘writing’ time trajectory clusters (cluster 2 and 3) except that cluster 2 contained a greater proportion of females than cluster 3 (67%and 52%respectively, Table 4). Both cluster 1, the slowing writers, and cluster 4, the quickening writers were older and had poorer DSST raw scores (both p < 0.0001) yet there were no differences between clusters on the other neuropsychological tests performed at Visit 2. Perceived physical fatigability was the only health and functional characteristic that differed between all clusters (p = 0.03). Analysis of health outcomes found that cluster 1 had a greater annualized decline in general cognitive ability (i.e., TICS score) over follow up (–1.83, SE 0.74, p = 0.02), and cluster 4, the quickening writers, had a significantly higher risk of mortality over follow up (aHR 1.70, CI 1.03–2.81, p = 0.04). As physical function values were notably lower for cluster 1 but may have been obscured by the small sample size of this cluster in the primary analysis, secondary analyses were performed to compare all health and functional characteristics for cluster 1 to clusters 2–4. Cluster 1 had the poorer performance in several domains including grip strength (p = 0.02), gait speed (p = 0.02), and perceived physical fatigability (p = 0.008).

Table 4

Health and functional characteristics of writing time trajectory clusters

					p
	Cluster 1 Slowing	Cluster 2 Stable-slowing	Cluster 3 Stable-quickening	Cluster 4 Quickening
Cluster size	26 (1.6)	181 (11.4)	1142 (71.6)	245 (15.4)
Avg. time 1st segment, s	1.1 ± 0.4	0.8 ± 0.4	0.9 ± 0.3	1.3 ± 0.4	< 0.0001
Demographics
Age, y	74.5 ± 11.9	69.1 ± 10.2	69.4 ± 9.6	74.7 ± 10.2	< 0.0001
Sex, female	15 (60.0)	117 (67.2)	586 (52.2)	115 (48.1)	0.007
Highest level of education					0.12
Up to a high school degree	11 (42.3)	61 (33.7)	436 (38.2)	108 (44.1)
College degree	12 (46.2)	64 (35.4)	437 (38.3)	95 (38.8)
Advanced degree	3 (11.5)	56 (30.9)	269 (23.6)	42 (17.1)
Field center (row %)					0.001
Boston	8 (1.8)	49 (11.0)	332 (74.4)	57 (12.8)
Denmark	7 (1.2)	40 (6.9)	426 (73.5)	107 (18.5)
New York	5 (2.7)	28 (15.1)	128 (69.2)	24 (13.0)
Pittsburgh	6 (1.6)	64 (16.7)	256 (66.8)	57 (14.9)
Long-lived family	17 (68.0)	138 (79.3)	828 (73.8)	173 (72.4)	0.66
Health/comorbidities
Stroke	1 (4.0)	3 (1.7)	16 (1.4)	4 (1.7)	0.81
Diabetes	2 (8.0)	9 (5.2)	43 (3.9)	13 (5.4)	0.90
Depression	7 (29.2)	31 (18.2)	192 (17.7)	34 (14.6)	0.47
Hypertension	13 (52.0)	48 (27.6)	320 (28.7)	84 (35.4)	0.32
Atrial fibrillation	2 (8.3)	3 (1.8)	34 (3.1)	10 (4.3)	0.66
Body mass index, kg/m²	26.6±5.1	27.5±4.5	27.2±4.4	27.2±4.3	0.69
Carotid IMT, mm	0.88±0.18	0.82±0.14	0.84±0.15	0.88±0.16	0.94
CES-D score ≥ 14	0 (0.0)	2 (1.1)	20 (1.8)	7 (3.0)	0.22
Physical function
Grip strength, kg	23.3±7.9	27.0±10.2	29.5±10.7	28.3±11.6	0.12
Gait speed, m/s	0.88±0.23	1.00±0.24	1.05±0.24	0.99±0.27	0.10
PFS mental, 0–50	11.7±9.7	7.8±8.4	6.8±7.2	8.5±9.0	0.15
PFS physical, 0–50	20.4 ± 10.7	13.1 ± 8.8	12.1 ± 8.2	14.4 ± 9.4	0.03
FEV1, liters	2.2±0.7)	2.4±0.7	2.6±0.8	2.3±0.9	0.54
Genetic risk factors
APOE E2 allele carrier	1 (4.2)	34 (21.0)	189 (17.6)	47 (20.3)	0.17
APOE E4 allele carrier	6 (25.0)	36 (22.2)	236 (21.9)	45 (19.4)	0.71
Cognitive function
MMSE < 24	2 (8.3)	1 (0.6)	14 (1.3)	2 (0.9)	0.16
DSST score	36.6 ± 12.6	48.8 ± 12.9	47.5 ± 12.5	39.9 ± 10.4	<0.0001
Logical Memory Immediate	12.7±2.8	14.1±3.8	14.0±4.2	13.4±4.5	0.86
Logical Memory Delayed	10.9±4.4	12.3±4.2	12.6±4.6	11.7±5.0	0.71
HVLT Total Recall	21.2±5.6	24.7±5.3	24.4±5.9	22.3±6.2	0.12
HVLT Delayed Recall	6.5±3.8	7.6±3.7	7.9±3.5	6.9±3.6	0.05
Semantic fluency	19.6±6.1	21.6±6.8	22.2±6.2	20.1±5.9	0.19
Phonemic fluency	32.2±14.7	39.9±15.2	38.3±13.6	34.7±13.2	0.19
CDT Command thinking time^∧	21.8 (15.6–29.3)	17.1 (13.2–23.8)	18.4 (14.4–24.2)	20.5 (16.2–28.1)	0.50
CDT Command writing time^∧	14.4 (10.7–17.2)	12.7 (10.5–17.2)	12.8 (10.5–15.6)	14.4 (12.1–18.2)	0.38
CDT Command total time^∧	34.7 (28.1–49.7)	30.9 (24.4–40.9)	31.4 (25.9–39.7)	35.5 (28.8–45.5)	0.67
Clinical Dementia Rating score (> 0)	5 (20.0)	18 (10.5)	68 (6.2)	22 (9.4)	0.08
Longitudinal outcomes
Mortality^* (HR)	2.24 (CI 0.69, 7.28);	1.35, CI 0.66, 2.74); p = 0.41	Ref	1.70 (CI 1.03, 2.81); p = 0.04	0.14
TICS^** (slope for follow up time)	–1.83 (SE 0.74; = 0.02)	0.29 (SE 0.21; p = 0.18)	–0.07 (SE 0.10; p = 0.48)	–0.07 (SE 0.19; p = 0.73)	0.04^†

Mean (SD) for continuous variables; counts (%) for categorical variables. p-values for comparisons adjusted for age, sex, education level, and field center. ^∧Median (Q1-Q3). p-value based on log-transformed variable. ^*Modeling time to death, adjusted for age, education level, and field center and stratified on sex. ^**Modeling TICS score as a function of age, sex, education level, field center, follow-up time, cluster, and an interaction of cluster and follow-up time. The p-value in each column is for a test that the slope is equal to zero. N = 1,014, as the Denmark field center did not perform the TICS exam. ^†p-value for cluster ^*follow-up time interaction. PFS, Pittsburgh Fatigability Scale; FEV1, Forced expiratory volume; IMT, intima-media thickness; MMSE, Mini-Mental State Examination; DSST, Digit Symbol Substitution Test; HVLT, Hopkins Verbal Learning Test-Revised; CDT, Clock Drawing Test; CESD, Center for Epidemiologic Studies Depression Scale; TICS, Telephone Interview for Cognitive Status.

Cluster concordance

Figure 5 shows concordance between ‘thinking’ time and ‘writing’ time clusters. Overall, 47%of individuals had concordant ‘thinking’ and ‘writing’ time trajectories (weighted kappa 0.02, CI –0.01–0.05). Concordance ranged from 0.1%for individuals classified as both slowing thinkers and slowing writers (cluster 1 for both trajectories) to 41.5%for individuals who were classified as stable-quickening for both ‘thinking’ and ‘writing’ time trajectories (cluster 3). Out of the 261 individuals in the clusters that had the lowest DSST raw scores and the greatest amount of change in ‘thinking’ time (clusters 1 and 4), only 78 (30%) also fell into clusters of ‘writing’ time that showed significant increases or decreases during the test. Notably, discordant change patterns were common; 33%of slowing thinkers (cluster 1) were quickening writers (cluster 4) and 46%of slowing writers (cluster 1) were quickening thinkers (cluster 4).

Fig. 5

Participant overlap between thinking time and writing time clusters.

DISCUSSION

Novel digital metrics revealed differences in performance patterns (i.e., changes in ‘thinking’ time and ‘writing’ time) during a brief neuropsychological test that have distinct cognitive and physical function correlates. We identified subsets of individuals who showed significant changes in ‘thinking’ speed who also had lower episodic memory test scores but did not differ in physical function. In contrast individuals who had decreasing ‘writing’ speed did not differ in cognitive scores but instead had poorer perceived physical but not mental fatigability as well as poorer gait speed and grip strength upon secondary analyses. Taken together, coupling digital technology with traditional neuropsychological test administration suggests that metrics measuring ‘thinking’ time versus ‘writing’ time confer information about the processes underlying overall summary test scores that provides clinically meaningful information beyond traditional test scores.

The low concordance of ‘thinking’ time and ‘writing’ time is consistent with the tenets underlying the BPA and underscores the added value of digital DSST metrics. Poorer DSST scores were associated with clusters that demonstrated significant within-task changes in ‘thinking’ time or ‘writing’ time, yet, only 30%of individuals who had an unstable ‘thinking’ trajectory also had an unstable ‘writing’ trajectory and many had opposing trajectories (e.g., slowing thinking and quickening writing). This suggests that by parsing out ‘thinking’ time from ‘writing’ time we were able to uncover differential processes contributing to lower DSST scores. Some individuals have poorer scores due to differences in the information processing component of the test, whereas other individuals have poorer scores due to graphomotor components. This further strengthens the notion that overall scores for test performance on the DSST confound the source of process-specific deficits and omit information that may have clinical utility.

Poor DSST performance as measured by traditional scoring has been associated with a higher risk for future cognitive impairment and dementia [27, 28]. Yet, in this study we found that only certain ‘thinking’ time and ‘writing’ time trajectories were associated with incident cognitive decline of more than half a point per year on the TICS. Interestingly it was the quickening thinkers (cluster 4), but also the slowing writers (cluster 1) who showed significant cognitive declines over follow up with slowing thinkers (cluster 1) showing a similar trend. In comparison, a ten-point lower raw score on the DSST was associated with a decline of 0.2 points per year on the TICS (Supplementary Table 4). This suggests that impairments in engaging in cognitive tasks and maintaining motor speed may be early, subtle neurocognitive biomarkers suggesting impending cognitive decline.

These ‘thinking’ time and ‘writing’ time trajectories were identified among healthy aging research study participants but may have utility for individuals presenting to a memory clinic. The development of a classifier to predict an individual’s ‘thinking’ time and ‘writing’ time cluster based on their digital DSST data could indicate their risk of related cognitive and physical deficits as well as incident cognitive decline. This information could signal to clinicians that in-depth testing is warranted and inform whether neuropsychological assessments or physical/neurological assessments may be of greater value. This increased specificity would also be valuable in identifying whether therapeutic interventions for each individual should be targeted toward cognitive or physical enhancement.

Neuropsychological correlates of performance trajectories

Digital DSST ‘thinking’ time clusters associated with cognitive abilities that underlie performance on this test including episodic memory and visuospatial function (CDT). Although prior research has concluded that episodic memory has only a negligible contribution to overall DSST performance estimated at 5–7%[29], our results suggest that for a subset of individuals who show change in ‘thinking’ time over the course of the test, poorer episodic memory function may be related to poorer DSST performance. Lower performance on the learning trials of the HVLT may indicate deficits in attention or engaging in the DSST task, whereas performance on delayed recall may reflect difficulty remembering the task instructions or deficits in associative memory (i.e., learning the symbol-digit pairs). Interestingly, the association of cluster membership with differences in episodic memory was only evident on the list learning task and was not seen for paragraph recall. A notable difference between the two tests is that the HVLT word list is repeated three times allowing for additional opportunities to learn the stimuli and implement strategies to more effectively remember word lists which may relate more directly to an individual’s ability to learn the symbol code on the DSST.

DSST ‘thinking’ time but not ‘writing’ time clusters were associated with digitally obtained measures from the CDT including total time, thinking time, and writing time. As the CDT is also a multifactorial test drawing upon cognitive functions including visuospatial function, working memory, conceptualization, and organization, it is difficult to ascertain which specific underlying constructs relate to slower performance on both tests. Additionally, one would expect slow graphomotor speed to affect writing and drawing tests similarly, yet, the DSST ‘writing’ time trajectories were not associated with CDT writing time. One of the fundamental differences between the CDT and DSST is that the CDT accesses information already stored in semantic memory (i.e., shape of the outer circle, placement of the numbers, and time setting) and is completed without time restrictions. In contrast, the DSST is a timed test that requires establishing and maintaining mental set on a non-automatized test with greater dependence on visual scanning. Additionally, participants often self-correct or redraw components on the CDT to conform to their semantically-stored image, which results in longer writing times and thereby contaminates writing time with thinking. This disassociation may also reflect differences in the constructs of writing speed as compared to change in writing speed. Additional studies are needed to deconstruct the multiple cognitive skills captured in writing time and how these constructs vary across tests.

The association of ‘thinking’ time clusters with neurocognitive, but not physical function correlates, with the exception of CDT writing time, suggests that the isolation of ‘thinking’ time (i.e., non-writing time) was able to reduce the contribution of motor function associated with DSST performance [29] and better indicate the contribution of cognitive processing. However, it may still be confounded by motor demands related to visual scanning [30], and incorporation of measures of eye tracking [31] could help refine the ‘thinking’ time metric as a measure of information processing.

Writing time trajectories may also provide better metrics of graphomotor speed than traditional DSST scores. Comparisons of DSST scores with performance on a simpler symbol copy task in other studies have shown that approximately 50%of the variance in DSST scores is related to psychomotor speed and that this correlation increases with age [29]. The dissociation of ‘writing’ time from ‘thinking’ time in the current analysis, rather than gross performance on a symbol copy test, might be a more direct measure of graphomotor speed as the symbol copy task is confounded by perceptual speed [29].

Strengths and limitations

The strengths of this study lie in the wide age range of participants (45 to 103 years), the availability of in-depth phenotyping across health domains, including physical function tests, measures of vascular health, and a multi-domain neuropsychological protocol, as well as the fact that this is a healthy aging cohort which may be particularly valuable for identifying early, subtle markers of cognitive impairment. One limitation of the study is that the LLFS cohort is a selected sample of individuals with familial longevity and thus a predisposition for healthy aging. This may have biased our results toward negative findings for associations of ‘thinking’ and ‘writing’ time patterns with stroke, hypertension, diabetes, atrial fibrillation, or subclinical measures of vascular health (i.e., carotid IMT) which have been previously associated with DSST performance [32, 33]. Additionally, participants are highly educated and predominantly non-Hispanic white and therefore analyses of DSST performance trajectories in other demographic and clinical cohorts is warranted. Secondly, the cognitive protocol does not include tests of visual scanning or sustained attention which may have revealed differential associations with ‘thinking’ time clusters. Thirdly, although performance on the DSST and similar coding tests have been noted to be better predictors of death among older adults than other cognitive tests [34 –36], and chronological age [37], the differences in mortality risk for some clusters needs to be examined with caution as the period of follow-up was relatively short (median 2.6 years), the number of deaths was low (n = 82), and other confounders of mortality risk should be considered. Finally, these analyses did not include adjustment for multiple comparisons and thus should be replicated in larger study populations.

CONCLUSIONS

Integration of digital technologies with standard neuropsychological testing protocols allows for the capture of high-precision process data that would not otherwise be able to be captured by the examiner. This is particularly valuable for the administration of brief, highly sensitive, multifactorial tests as the digital data may allow for differentiation of the contributions of dysfunction in specific underlying cognitive and motor processes, data that are confounded in overall test scores. Thus, digital metrics, such as the ‘thinking’ and ‘writing’ trajectories identified in this study, may specifically inform the selection and timing of in-depth neuropsychological assessments or could be a marker for other subclinical risk factors associated with the disablement pathway (e.g., fatigability) and may help target appropriate interventions.

Footnotes

ACKNOWLEDGMENTS

We would like to thank Ben Wasserman and the Framingham Heart Study for sharing PowerShell scripts that aided in the conversion of the digital pen files. We would also like to thank the extraordinary participants of the Long Life Family Study who make this work possible.

This work was supported by the National Institute on Aging (K01AG057798 to S.L.A., U01AG023749 to S.C., U01AG023755 to T.T.P., U01AG023712, U01AG023744, U01AG023746, U19AG063893); the National Institute of General Medical Sciences Interdisciplinary Training Grant for Biostatisticians (T32 GM74905) to B.S.; the Boston University School of Medicine Department of Medicine Career Investment Award to S.L.A.; and the Marty and Paulette Samowitz Foundation to T.T.P. Additionally, the Claude D. Pepper Older Americans Independence Center, Research Registry and Developmental Pilot Grant (NIH P30 AG024827), and the Intramural Research Program, National Institute on Aging supported N.W.G. to develop the Pittsburgh Fatigability Scale.

Authors’ disclosures available online ().

The supplementary material is available in the electronic version of this article: .

References

Villemagne

, Burnham

, Bourgeat

, Brown

, Ellis

, Salvado

, Szoeke

, Macaulay

, Martins

, Maruff

, Ames

, Rowe

, Masters

, Australian Imaging Biomarkers and Lifestyle (AIBL) Research Group (2013) Amyloid beta deposition, neurodegeneration, and cognitive decline in sporadic Alzheimer’s disease: A prospective cohort study. Lancet Neurol 12, 357–367.

Jansen

, Ossenkoppele

, Knol

, Tijms

, Scheltens

, Verhey

, Visser

, Amyloid Biomarker Study G, Aalten

, Aarsland

, Alcolea

, Alexander

, Almdahl

, Arnold

, Baldeiras

, Barthel

, van Berckel

, Bibeau

, Blennow

, Brooks

, van Buchem

, Camus

, Cavedo

, Chen

, Chetelat

, Cohen

, Drzezga

, Engelborghs

, Fagan

, Fladby

, Fleisher

, van der Flier

, Ford

, Forster

, Fortea

, Foskett

, Frederiksen

, Freund-Levi

, Frisoni

, Froelich

, Gabryelewicz

, Gill

, Gkatzima

, Gomez-Tortosa

, Gordon

, Grimmer

, Hampel

, Hausner

, Hellwig

, Herukka

, Hildebrandt

, Ishihara

, Ivanoiu

, Jagust

, Johannsen

, Kandimalla

, Kapaki

, Klimkowicz-Mrowiec

, Klunk

, Kohler

, Koglin

, Kornhuber

, Kramberger

, Van Laere

, Landau

, Lee

, de Leon

, Lisetti

, Lleo

, Madsen

, Maier

, Marcusson

, Mattsson

, de Mendonca

, Meulenbroek

, Meyer

, Mintun

, Mok

, Molinuevo

, Mollergard

, Morris

, Mroczko

, Van der Mussele

, Na

, Newberg

, Nordberg

, Nordlund

, Novak

, Paraskevas

, Parnetti

, Perera

, Peters

, Popp

, Prabhakar

, Rabinovici

, Ramakers

, Rami

, Resende de Oliveira

, Rinne

, Rodrigue

, Rodriguez-Rodriguez

, Roe

, Rot

, Rowe

, Ruther

, Sabri

, Sanchez-Juan

, Santana

, Sarazin

, Schroder

, Schutte

, Seo

, Soetewey

, Soininen

, Spiru

, Struyfs

, Teunissen

, Tsolaki

, Vandenberghe

, Verbeek

, Villemagne

, Vos

, van Waalwijk van Doorn

, Waldemar

, Wallin

, Wiltfang

, Wolk

, Zboch

, Zetterberg

(2015) Prevalence of cerebral amyloid pathology in persons without dementia: A meta-analysis. JAMA 313, 1924–1938.

Ashendorf

, Swenson

, Libon

(2013) The Boston process approach to neuropsychological assessment: A practitioner’s guide, Oxford University Press, Oxford.

Kaplan

(1988) A process approach to neuropsychological assessment In Clinical neuropsychology and brain function: Research, measurement, and practice, Boll T, Bryant BK, eds. American Psychological Association, Washington, DC, pp. 127–167.

Wechsler

(1981) Wechsler Adult Intelligence Scale-Revised (WAIS-R), The Psychological Corporation, New York.

Jaeger

(2018) Digit Symbol Substitution Test: The case for sensitivity over specificity in neuropsychological testing. J Clin Psychopharmacol 38, 513–519.

Proust-Lima

, Amieva

, Dartigues

, Jacqmin-Gadda

(2007) Sensitivity of four psychometric tests to measure cognitive changes in brain aging-population-based studies. Am J Epidemiol 165, 344–350.

Lezak

(2012) Neuropsychological assessment. Oxford University Press, Oxford; New York.

Kaplan

, Fein

, Morris

, Delis

(1991) WAIS-R as a Neuropsychological Instrument. The Psychological Corporation, San Antonio, TX.

10.

Diaz-Orueta

, Blanco-Campal

, Lamar

, Libon

, Burke

(2020) Marrying past and present neuropsychology: Is the future of the process-based approach technology-based? Front Psychol 11, 361.

11.

Davis

, Libon

, Au

, Pitman

, Penney

(2015) THink: Inferring cognitive status from subtle behaviors. AI Mag 36, 49–60.

12.

Muller

, Preische

, Heymann

, Elbing

, Laske

(2017) Increased diagnostic accuracy of digital vs. conventional clock drawing test for discrimination of patients in the early course of Alzheimer’s disease from cognitively healthy individuals. Front Aging Neurosci 9, 101.

13.

Zhao

, Wei

, Do

, Huang

(2019) Assessing performance on digital clock drawing test in aged patients with cerebral small vessel disease. Front Neurol 10, 1259.

14.

Cohen

, Penney

, Davis

, Libon

, Swenson

, Ajilore

, Kumar

, Lamar

(2014) Digital Clock Drawing: Differentiating “thinking” versus “doing” in younger and older adults with depression. J Int Neuropsychol Soc 20, 920–928.

15.

Piers

, Devlin

, Ning

, Liu

, Wasserman

, Massaro

, Lamar

, Price

, Swenson

, Davis

, Penney

, Au

, Libon

(2017) Age and graphomotor decision making assessed with the digital clock drawing test: The Framingham Heart Study. J Alzheimers Dis 60, 1611–1620.

16.

Libon

, Penney

, Davis

, Tabby

, Eppig

, Nieves

, Bloch

, Donohue

, Brennan

, Rife

, Wicas

, Lamar

, Price

, Au

, Swenson

, Garrett

(2014) Deficits in processing speed and decision making in relapsing-remitting multiple sclerosis: The digit clock drawing test (dCDT). J Mult Scler 1, 1–8.

17.

Hizel

, Warner

, Wiggins

, Tanner

, Parvataneni

, Davis

, Penney

, Libon

, Tighe

, Garvan

, Price

(2019) Clock drawing performance slows for older adults after total knee replacement surgery. Anesth Analg 129, 212–219.

18.

Newman

, Glynn

, Taylor

, Sebastiani

, Perls

, Mayeux

, Christensen

, Zmuda

, Barral

, Lee

, Simonsick

, Walston

, Yashin

, Hadley

(2011) Health and function of participants in the Long Life Family Study: A comparison with other cohorts. Aging (Albany NY) 3, 63–76.

19.

Sebastiani

, Hadley

, Province

, Christensen

, Rossi

, Perls

, Ash

(2009) A family longevity selection score: Ranking sibships by their longevity, size, and availability for study. Am J Epidemiol 170, 1555–1562.

20.

Glynn

, Santanasto

, Simonsick

, Boudreau

, Beach

, Schulz

, Newman

(2015) The Pittsburgh Fatigability scale for older adults: Development and validation. J Am Geriatr Soc 63, 130–135.

21.

LaSorda

, Gmelin

, Kuipers

, Boudreau

, Santanasto

, Christensen

, Renner

, Wojczynski

, Andersen

, Cosentino

, Glynn

(2020) Epidemiology of perceived physical fatigability in older adults: The Long Life Family Study. J Gerontol A Biol Sci Med Sci 75, e81–e88.

22.

Brandt

, Spencer

, Folstein

(1988) The Telephone Interview for Cognitive Status. Neuropsychiatry Neuropsychol Behav Neurol 1, 111–117.

23.

Welsh

, Breitner

, Magruder-Habib

(1993) Detection of dementia in the elderly using telephone screening of cognitive status. Neuropsychiatry Neuropsychol Behav Neurol 6, 103–110.

24.

Sebastiani

, Andersen

, Sweigart

, Du

, Cosentino

, Thyagarajan

, Christensen

, Schupf

, Perls

(2020) Patterns of multi-domain cognitive aging in participants of the Long Life Family Study. Geroscience 42, 1335–1350.

25.

Spiegelhalter

, Best

, Carlin

, van der Linde

(2002) Bayesian measures of model complexity and fit. J R Stat Soc Ser B 64, 583–616.

26.

Andersen

, Sweigart

, Sebastiani

, Drury

, Sidlowski

, Perls

(2019) Reduced prevalence and incidence of cognitive impairment among centenarian offspring. J Gerontol A Biol Sci Med Sci 74, 108–113.

27.

Rosano

, Perera

, Inzitari

, Newman

, Longstreth

, Studenski

(2016) Digit Symbol Substitution test and future clinical and subclinical disorders of cognition, mobility and mood in older adults. Age Ageing 45, 688–695.

28.

Backman

, Jones

, Berger

, Laukka

, Small

(2005) Cognitive impairment in preclinical Alzheimer’s disease: A meta-analysis. Neuropsychology 19, 520–531.

29.

Joy

, Kaplan

, Fein

(2004) Speed and memory in the WAIS-III Digit Symbol–Coding subtest across the adult lifespan. Arch Clin Neuropsychol 19, 759–767.

30.

Joy

, Fein

, Kaplan

(2003) Decoding digit symbol: Speed, memory, and visual scanning. Assessment 10, 56–65.

31.

Stephens

(2006) Age-related decline in Digit-Symbol performance: Eye-movement and video analysis. Arch Clin Neuropsychol 21, 101–107.

32.

Sherzai

, Sherzai

(2017) Cardiovascular risk factors and cognitive function in the elderly ages 60 and older: An analysis of the National Health and Nutrition Evaluation Survey. Alzheimers Dement 13, P1224–P1225.

33.

Zeki Al Hazzouri

, Vittinghoff

, Sidney

, Reis

, Jacobs

Jr. , Yaffe

(2015) Intima-media thickness and cognitive function in stroke-free middle-aged adults: Findings from the Coronary Artery Risk Development in Young Adults Study. Stroke 46, 2190–2196.

34.

Duff

, Mold

, Gidron

(2009) Cognitive functioning predicts survival in the elderly. J Clin Exp Neuropsychol 31, 90–95.

35.

Pavlik

, de Moraes

, Szklo

, Knopman

, Mosley

Jr. , Hyman

(2003) Relation between cognitive function and mortality in middle-aged adults: The atherosclerosis risk in communities study. Am J Epidemiol 157, 327–334.

36.

Aichele

, Rabbitt

, Ghisletta

(2016) Think fast, feel fine, live long: A 29-year study of cognition, health, and survival in middle-aged and older adults. Psychol Sci 27, 518–529.

37.

Wettstein

, Spuling

, Cengia

(2020) Trajectories of functional health and its associations with information processing speed and subjective well-being: The role of age versus time to death. Psychol Aging 35, 190–203.