Within-Person Test Score Distributions: How Typical Is “Normal”?

Abstract

We evaluated within-person variability across a cognitive test battery by analyzing the shape of the distribution of each individual’s scores within a battery of tests. We hypothesized that most healthy adults would produce test scores that are normally distributed around their own personal battery-wide, within-person (wp) mean. Using cross-sectional data from 327 neurologically healthy adults, we computed each person’s mean, standard deviation, skew, and kurtosis for 30 neuropsychological measures. Raw scores were converted to T-scores using three degrees of calibration: (a) none, (b) age, and (c) age, sex, race, education, and estimated premorbid IQ. Regardless of calibration, no participant showed abnormal within-person skew (^wpskew) and only 10 (3.1%) to 16 (4.9%) showed ^wpkurtosis greater than 2. If replicated in other samples and measures, these findings could illuminate how healthy individuals are endowed with different cognitive abilities and provide the foundation for a new method of inference in clinical neuropsychology.

Keywords

normal distribution neuropsychological cognitive test intra-individual variability skew kurtosis

Introduction

Many investigators have examined the range of intra-individual variability (IIV) in cognitive test performance, beginning with the concept of test score “scatter” (Wechsler, 1939). Kaufman (1976) proposed the consideration of inter-subtest scatter as measured by the difference between a person’s highest and lowest Wechsler subtest scores, also referred to as “maximum discrepancy” (Schretlen et al., 2003). Others advocate computing an intra-individual standard deviation (ISD) based on a person’s z-transformed test scores (Lindenberger & Baltes, 1997). Still others have defined IIV in terms of the deviation of a person’s test scores from his or her own mean test score (McLean et al., 1989) or IQ (Rabbitt, 1993). Importantly, each of these approaches ignore skewness and kurtosis, parameters required to assess the shape of within-person test score distributions.

Cognitive IIV has been demonstrated in healthy aging as well as numerous clinical populations, including traumatic brain injury, HIV, and neurodegenerative diseases (Bangen et al., 2019; Dykiert et al., 2012; Vance et al., 2021). The literature has shown that increased cognitive IIV predicts cognitive decline, abnormal neuroimaging findings, and even mortality (Halliday et al., 2019; Hilborn et al., 2009; Holtzer et al., 2008; Vance et al., 2021). In a healthy aging sample, increased cognitive IIV was associated with reduced white matter volume and increased frontal lobe white matter hyperintensities (Bunce et al., 2007), as well as increased frontal lobe activation during cognitive task performance (Bellgrove et al., 2004). However, Matarazzo (1990) and Kline et al. (1993) provided convincing evidence that such cognitive IIV does not measure what we think it does. For example, Matarazzo found that greater variability among cognitive tests was associated with higher, not lower, IQ.

Despite the above-mentioned advancements, we still do not have established population norms for cognitive IIV. In clinical practice, neuropsychologists compare a person’s performance on any given cognitive test with the score distribution of a normative sample as opposed to an examinee’s own within-person distribution of test scores. This raises at least two questions: First, do most healthy people produce scores that are normally distributed on a battery of cognitive tests? Second, does the shape of a person’s test score distribution depend on his or her own overall mean performance? If healthy people produce normal within-person test score distributions regardless of their overall ability, then departures from normality might help diagnose cognitive dysfunction.

Consider the hypothetical scenario in which two people, X and Y, both earn T-scores of 52 on a verbal fluency test (test A). This places both their performances in the middle of a normative distribution (Figure 1A). However, Figure 1A sheds no light on how their verbal fluency performance compares with their other test scores. Figure 1B and 1C plots the same T-scores relative to each person’s own test score distribution, which we created for this purpose. Person X’s verbal fluency T-score of 52 represents a personal strength in a hypothetical within-person distribution that is positively skewed, more variable than usual, and has a low average mean. Person Y’s verbal fluency T-score of 52 represents a personal weakness in a minimally negatively skewed distribution with typical variability, and an above average mean. This shows that identical population-based T-scores can have different clinical meanings in the context of a person’s own test score distribution. What remains unknown at present is whether the cognitive test scores of most healthy persons are normally distributed and whether parameters of within-person test score distributions vary as a function of a person’s overall ability, demographic characteristics, or some other factor(s). If most healthy adults produce normally distributed scores across a test battery, then measuring departures from within-person normality might prove useful for diagnosis and offer a novel approach to clinical inference.

Figure 1.

(A) Normative Distribution. Scores on Test A for Two Hypothetical Participants (Persons X and Y) Are Starred. Note That Persons X and Y Earned the Same Score on Test A. (B, C) Score Distributions for Persons X and Y, With the Score for Test A Starred in Each. Note That Despite Having Identical Scores on Test A, the Participants’ Overall Distributions of Scores on the Entire Test Battery Differ From One Another. For Example, Test A Is a Relative Strength for Person X But a Relative Weakness for Person Y. Despite These Differences, Neither Distribution Is Markedly Abnormal.

In this study, we extend existing IIV analyses by measuring the overall shape of within-person (wp) test score distributions. We hypothesized that most healthy adults would produce test score distributions that are normal (i.e., with minimal within-person skew [^wpskew] and kurtosis [^wpkurtosis]) on a battery of cognitive tests (Schretlen & Sullivan, 2013). We previously found that healthy older adults differed from patients with cognitive impairment not only in terms of within-person mean (^wpmean) scores and standard deviations (^wpSD), but also in terms of ^wpskew and ^wpkurtosis (Reckess et al., 2014). In that cross-sectional study, the within-person test score distributions of patients shifted increasingly away from normality with the severity of their cognitive impairment (Reckess et al., 2014). In the current study, we examine the range of four parameters (^wpmean, ^wpSD, ^wpskew, and ^wpkurtosis) of the within-person score distributions produced by healthy adults.

Finally, the current study also aimed to determine whether within-person distributions across the adult life span vary as a function of participant characteristics or calibration for them. Applying demographic calibrations can increase the precision of expected performance, but its impact on the properties of within-person distributions is unknown.

Method

Participants

The study sample consists of 327 neurologically normal participants in the Johns Hopkins Aging, Brain Imaging, and Cognition (ABC) study, which was used to derive normative data for the Calibrated Neuropsychological Normative System (CNNS) software program (Schretlen et al., 2010). The ABC study was approved by our institution’s Institutional Review Board, and all participants gave written informed consent. Data collection procedures and sample characteristics are described in detail elsewhere (Schretlen et al., 2008). Participants ranged in age from 18 to 90 (M = 54.8, SD = 18.8). As shown in Table 1, the sample included non-Latino White (80.2%), Black (18.0%), and Asian/Latino/Other (1.8%) participants, and slightly more women (56.6%) than men. Participants completed from 3 to 20 years (M = 14.2, SD = 3.0) of schooling. Exclusionary criteria included presence of neurological disease (e.g., Parkinson’s disease, Alzheimer’s disease, history of stroke or brain injury), severe mental illness (e.g., bipolar disorder, schizophrenia), or current substance abuse/dependence.

Table 1.

Demographic Makeup of the Sample.

Characteristic	Sample
N	327
Age in years (M±SD)	54.8 ± 18.8
Education (M±SD)	14.2 ± 3.0
% Women	56.6
% Race (non-Latino White/Black/Asian/Latino/other	80.2/18/1.8

Neuropsychological Measures

The current analyses were conducted using 30 measures (Table 2) derived from 19 tests included in the CNNS battery. These measures capture major cognitive domains while maximizing the number of participants with complete data. Where possible, summary scores (e.g., total letter-cued fluency) were used rather than scores for individual trials. Raw scores were converted to T-scores using CNNS software. For each measure, three T-scores were derived: (a) uncalibrated (i.e., direct conversion from scaled scores to T-scores for the entire sample), (b) age-calibrated, and (c) fully calibrated (i.e., adjusted for age, sex, race, years of education, and estimated premorbid IQ based on performance on the Hopkins Adult Reading Test (HART; Schretlen et al., 2009). The methods used to derive the regression-based norms on which these T-scores were based are described elsewhere (Testa et al., 2009).

Table 2.

Neuropsychological Measures.

Test	Measure(s)	Function
Mini-Mental State Examination (MMSE)^a	Total score	Global cognition
Right–left Orientation^b	Sum	Discrimination and recognition of right from left
Grooved Pegboard Test (GPT)^c	Dominant hand	Manual speed and dexterity
Grooved Pegboard Test (GPT)^c	Non-dominant hand	Manual speed and dexterity
Perceptual Comparison Test (PCT)^d	Total score	Psychomotor processing speed
Trail Making Test (TMT)^e	Part A	Psychomotor processing speed
Trail Making Test (TMT)^e	Part B	Psychomotor processing speed and set-shifting
Digit Span (WAIS-R; WAIS-III)^f	Total score	Auditory attention and working memory
Brief Test of Attention (BTA)^g	Total score	Divided auditory attention
Cognitive Estimation Task^h	Total score	Ability to provide appropriate cognitive estimates
Modified Wisconsin Card Sorting Test (mWCST)ⁱ	Number of categories	Concept formation
Modified Wisconsin Card Sorting Test (mWCST)ⁱ	Number of errors	Attentional shifting
Calibrated Ideational Fluency Assessment (CIFA)^j	Letter fluency (S, P)	Letter-guided/phonemic verbal fluency
	Animals/supermarket items	Category-guided/semantic verbal fluency
	Design fluency (novel items)	Ability to generate novel designs
Boston Naming Test (BNT) (30-Item)^k	Total score	Visual confrontation naming
Benton Facial Recognition Task (BFRT)^l	Total score	Facial recognition
Rey Complex Figure Test (RCFT)^m	Copy (score)	Complex visuoconstruction
Visual Reproductions (WMS-R)ⁿ	Immediate recall	Visuospatial learning
	Delayed recall	Visuospatial memory
Brief Visuospatial Memory Test—Revised (BVMT-R)^o	Learning (Trials 1–3)	Visuospatial learning
	Delayed recall	Visuospatial recall
	Discrimination	Visuospatial recognition
Hopkins Verbal Learning Test—Revised (HVLT-R)^p	Learning (Trials 1–3)	Auditory word list learning
	Delayed recall	Auditory word list recall
	Discrimination	Auditory word list recognition
Logical Memory (WMS-R)^q	Immediate recall	Auditory story learning
	Delayed recall	Auditory story recall
Prospective Memory Test	Total score	Ability to recall an intention to do something
Recency Discrimination Test^r	Total score	Memory for temporal order

Folstein et al. (1975). ^bBenton et al. (1994). ^cKlove (1963), Reitan and Davison (1974). ^dSalthouse and Babcock (1991). ^eArmy Individual Test Battery (1944). ^fWechsler (1981, 1997). ^gSchretlen (1997). ^hAxelrod and Millis (1994), Shallice and Evans (1978). ⁱSchretlen (2010). ^jSchretlen and Vannorsdall (2010). ^kSelnes et al. (1988). ^lBenton et al. (1994). ^mCorwin and Bylsma (1993), Osterrieth (1944), Rey (1941). ⁿWechsler (1987). ^oBenedict (1997). ^pBrandt and Benedict (2001). ^qWechsler (1987). ^rManning et al. (2007). WAIS = Wechsler Adult Intelligence Scale; WMS = Wechsler Memory Scale; R = Revised; III = Third Edition.

Analyses

For the 327 study participants, we first computed each person’s ^wpmean, ^wpSD, ^wpskew, and ^wpkurtosis based on his or her T-scores using the three levels of calibration described above. Using Pearson’s (r) correlations, we examined associations among the resulting 12 parameters (i.e., four parameters at each level of calibration). We also examined association between each parameter and five participant characteristics, using Pearson’s r for age, education, and estimated IQ or point biserial (r_pb) correlations for sex and race.

We next examined the 327 within-person test score distributions for departures from normality in two ways. We first conducted Kolmogorov–Smirnoff (KS) to test the normality of ^wpmean, ^wpSD, ^wpskew, and ^wpkurtosis distributions at each calibration level for normality. We then tallied the number of participants who produced ^wpskew and ^wpkurtosis coefficients that exceeded an absolute value of ±2, as recommended by Hair (2021) to establish non-normality. To test the effects of score calibration on ^wpmean, ^wpSD, ^wpskew, and ^wpkurtosis, we also conducted three paired-samples t-tests (uncalibrated vs. age-calibrated, uncalibrated vs. fully calibrated, and age-calibrated vs. fully calibrated) for each parameter.

Finally, we sought to determine whether specific cognitive measures contributed to high levels of within-person variability, as this could indicate within-person distributional properties depend on the specific tests used. To examine this, we tallied the percentage of participants for whom each measure appeared as the highest or lowest of each person’s score distribution. If a person obtained identically high or low scores on two measures, both were counted. All statistics were conducted using IBM SPSS Statistics version 28.

Results

Table 3 shows mean, minimum, and maximum values for the ^wpmean, ^wpSD, ^wpskew, and ^wpkurtosis coefficients by calibration level, along with the results of KS tests for the normality of each distribution parameter. Regardless of calibration level, ^wpmean, ^wpSD, and ^wpskew were normal. While the KS test for age-calibrated ^wpskew was nominally significant (p = .047), it did not survive Bonferroni adjustment for multiple comparisons (p < .013). In contrast, KS tests showed that distributions of ^wpkurtosis departed from normality at every level of test score calibration (all ps < .001).

Table 3.

Within-Person Test Score Distribution Parameter Values for Three Levels of Test Score Calibration and Kolmogorov–Smirnov Tests of Normality for Each Parameter’s Distribution.

Distribution parameter	Calibration level	M	Minimum	Maximum	Kolmogorov–Smirnov test
Distribution parameter	Calibration level	M	Minimum	Maximum	Statistic	df	p
^wpMean	None	49.82	32.63	63.47	.02	327	>.199
^wpMean	Age	50.00	29.63	62.26	.04	327	>.199
^wpMean	Full	50.08	36.60	59.23	.04	327	>.199
^wp SD	None	7.58	4.18	11.13	.03	327	>.199
^wp SD	Age	8.61	4.56	12.95	.02	327	>.199
^wp SD	Full	9.16	5.13	13.10	.03	327	>.199
^wpSkew	None	0.00	−1.53	1.60	.04	327	>.199
^wpSkew	Age	−0.11	−1.57	1.25	.05	327	.047
^wpSkew	Full	−0.11	−1.58	1.27	.05	327	.054
^wpKurtosis	None	0.13	−1.28	5.10	.11	327	<.001
^wpKurtosis	Age	0.05	−1.23	4.48	.12	327	<.001
^wpKurtosis	Full	0.06	−1.29	5.32	.12	327	<.001

Note. ^wp = within-person. SD = standard deviation. For calibration, none = no calibration; age = age-calibrated; full = calibrated age, sex, education, race, and estimated premorbid IQ.

Table 4 presents the frequency distributions of ^wpmean, ^wpSD, ^wpskew, and ^wpkurtosis by calibration level. As shown, 95% of the 327 participants produced a ^wpmean between 38.9 and 58.4 for age-calibrated scores or 41.1 and 57.9 for fully adjusted scores, and 95% produced a ^wpSD between 5.6 and 11.3 for age-calibrated scores or 6.2 and 12.2 for fully calibrated scores. Notably, 95% of participants produced ^wpskew coefficients that ranged from –1.1 to .82 for age-calibrated or .78 for fully calibrated scores. No participant produced a ^wpskew coefficient that exceeded the absolute value of 2, the threshold conventionally accepted to signify departure from normality (Hair, 2013). In fact, just three persons produced a ^wpskew > 1 for age-calibrated scores and just one produced ^wpskew > 1 with fully calibrated scores. More participants showed negative than positive ^wpskew with age-calibrated (61.2%) and fully calibrated (57.8%) scores, but not with uncalibrated (48.3%) scores. Consistent with the KS results, a few participants produced ^wpkurtosis coefficients that exceeded 2. All of these were positive (leptokurtic), and the number of participants with ^wpkurtosis coefficients >2 decreased with level of test score calibration (uncalibrated, 4.9%; age-calibrated, 3.7%; and fully calibrated, 3.1%). In short, while KS testing revealed non-normal distributions of ^wpkurtosis, fewer than 4% of participants showed markedly leptokurtic within-person test score distributions with age-calibrated or fully calibrated scores.

Table 4.

Frequency Distributions of Within-Person Parameters by Score Calibration Level.

Distribution parameter	Calibration level	Parameter value to percentile rank conversion
Distribution parameter	Calibration level	2.5	5.0	10	25	50	75	90	95	97.5
^wpMean	None	36.7	39.4	42.5	45.9	50.3	54.0	57.1	58.2	58.6
^wpMean	Age	38.9	40.9	43.0	47.0	50.2	53.7	56.1	57.1	58.4
^wpMean	Full	41.1	43.3	45.0	47.7	50.2	52.7	55.2	56.5	57.9
^wp SD	None	5.1	5.5	6.0	6.7	7.5	8.5	9.2	9.6	10.3
^wp SD	Age	5.6	6.3	6.8	7.7	8.6	9.6	10.3	10.9	11.3
^wp SD	Full	6.2	6.7	7.2	8.1	9.2	10.2	11.0	11.6	12.2
^wpSkew	None	−1.1	−.87	−.69	−.30	.01	.33	.63	.76	.89
^wpSkew	Age	−1.1	−1.0	−.76	−.37	−.10	.21	.46	.60	.82
^wpSkew	Full	−1.1	−.91	−.73	−.43	−.08	.22	.49	.68	.78
^wpKurtosis	None	−1.1	−.96	−.84	−.48	−.10	.51	1.3	2.0	3.0
^wpKurtosis	Age	−1.1	−1.0	−.93	−.60	−.14	.43	1.2	1.8	2.5
^wpKurtosis	Full	−1.1	−.95	−.86	−.58	−.07	.46	1.2	1.7	2.2

Note. ^wp = within-person. SD = standard deviation. For calibration, none = no calibration; age = age-calibrated; full = fully calibrated (scores calibrated for age, sex, education, race, and estimated premorbid ability).

We next examined associations among within-person distribution parameters. These analyses involved 18 Pearson’s r correlations (i.e., uncalibrated ^wpmean with uncalibrated ^wpSD, ^wpskew, and ^wpkurtosis, followed by uncalibrated ^wpSD with ^wpskew and ^wpkurtosis, followed by ^wpskew with ^wpkurtosis, and then repeating these for age-calibrated and fully calibrated scores). To maintain an α of <.05, we applied a Bonferroni correction (p < .003) to reject the null hypothesis of no association. Most correlations were small and non-significant, but three age-calibrated parameters survived Bonferroni correction: ^wpmean with ^wpSD: r = −.28; p < .001, ^wpmean with ^wpskew: r = .22; p < .001, and ^wpskew with ^wpkurtosis: r = −.35; p < .001). Thus, when using age-calibrated test scores, participants with higher ^wpmeans showed within-person distributions that were slightly less variable, more negatively skewed (i.e., with more scores clustering above and a few scores further below their ^wpmeans), and slightly more leptokurtic than those with lower ^wpmeans. Notably, while statistically significant, these three correlations were relatively weak. In sum, test battery ^wpmeans were nearly identical across levels of calibration, ^wpSDs increased with degree of calibration (from none to fully calibrated), and ^wpskew and ^wpkurtosis averaged close to zero across study participants.

Next, correlations between within-person distribution parameters and participant characteristics are shown in Table 5. Using a Bonferroni correction of p < .001 to maintain an overall α < .05 for rejecting the null hypothesis, uncalibrated ^wpmean scores correlated inversely with age (r = −.62, p < .001) and positively with education (r = .40, p < .001) and estimated IQ (r = .54, p < .001). Black participants produced lower ^wpmean T-scores than other participants (r_pb = −.28, p < .001), but ^wpmean T-scores did not vary by sex. Uncalibrated ^wpSD, ^wpskew, and ^wpkurtosis did not correlate significantly with any participant characteristic. When test scores were calibrated for age, ^wpmean did not correlate with age or sex but did correlate with years of education (r = .46, p < .001), race (r_pb = −.42, p < .001), and estimated IQ (r = .23, p < .001). As expected, fully calibrating test scores eliminated significant correlations between ^wpmean and every participant characteristic shown in Table 5. Age-calibrated ^wpSD correlated with race (r_pb = .20, p < .001), as Black participants showed slightly greater variability across tests than other participants. However, this was true only for age-calibrated scores. No other calibrated within-person parameter correlated with any other participant characteristic.

Table 5.

Correlations Between Distributional Properties and Participant Characteristics.

Calibration	Parameter	Age	Education	Sex^a	Race^a	IQ
Uncalibrated	^wpMean T-score	−0.62*	0.40*	0.04	−0.28*	0.23*
	^wpStandard deviation	0.05	−0.05	−0.03	0.10	−0.08
	^wpSkew	0.14	0.03	−0.05	0.00	0.03
	^wpKurtosis	−0.08	<0.01	−0.04	−0.02	<0.01
Age calibrated	^wpMean T-score	0.0	0.46*	<−0.01	−0.42*	0.59*
	^wpStandard deviation	0.13	−0.14	−0.03	0.20*	−0.14
	^wpSkew	0.01	−0.09	0.00	0.06	−0.14
	^wpKurtosis	−0.05	0.05	0.01	−0.07	0.09
Fully calibrated	^wpMean T-score	−0.01	−0.00	0.03	−0.01	−0.02
	^wpStandard deviation	0.13	−0.11	0.02	−0.16	−0.11
	^wpSkew	−0.02	−0.00	−0.00	−0.00	−0.02
	^wpKurtosis	−0.06	−0.02	−0.00	−0.04	−0.01

Point-biserial correlations (Men [Sex] and White [Race] coded as the smaller value).

p < .001 (two-tailed); based on Bonferroni correction for multiple correlations to maintain α < .05.

To test the effects of test score calibration on ^wpmean, ^wpSD, ^wpskew, and ^wpkurtosis, we conducted three paired-samples t-tests (uncalibrated vs. age-calibrated, uncalibrated vs. fully calibrated, and age-calibrated vs. fully calibrated) for each parameter. The first three t-tests revealed no differences in ^wpmeans as a result of calibration level. However, the next three revealed substantial effects of calibration on ^wpSD (uncalibrated vs. age-calibrated t₃₂₆ = −29.88, p < .001; uncalibrated vs. fully calibrated t₃₂₆ = −38.46, p < .001; age-calibrated vs. fully calibrated t₃₂₆ = −20.93, p < .001). These findings show that increasing the level of calibration tends to increase within-person performance variability across the tests administered. The third set of paired-samples t-tests showed that compared with uncalibrated scores, both age-calibrated (t₃₂₆ = 6.81, p < .001) and fully calibrated (t₃₂₆ = 5.82, p < .001) ^wpskew coefficients were larger. Age-calibrated and fully calibrated ^wpskew coefficients did not differ (t₃₂₆ = −.59, p = .557), and no level of test score calibration significantly altered ^wpkurtosis.

Finally, we ranked all 30 test scores by the frequency with which each appeared as an individual’s lowest or highest score across the 327 participants to determine whether any individual measure(s) was especially likely to be high or low. There was large variability across individuals in terms of which test score was highest or lowest, with no specific test score being the lowest or highest for more than 12% of participants. This suggests that the patterns observed here are not unique to the specific tests administered.

Discussion

The primary aim of this study was to determine whether most healthy adults produce normally distributed scores regardless of their age, sex, race, years of education, IQ, and overall mean score on a battery of neuropsychological tests. To our knowledge, this and one other study of older adults with differing degrees of cognitive impairment (Reckess et al., 2014) are the first to examine IIV using this approach. As shown in Table 3, KS tests found that the within-person test score distributions of ^wpmean, ^wpSD, and ^wpskew were all normal, and this finding held regardless of how test scores were calibrated. The average ^wpmeans (49.8–50.1) and ^wpSDs (7.6–9.2) were close to their expected values, and the mean levels of ^wpskew and ^wpkurtosis were miniscule. No participant produced a ^wpskew coefficient that exceeded |2|, and 95% produced ^wpskew values between –1.1 and .9. Contrary to our hypothesis, KS tests showed that ^wpkurtosis distributions were not normal. However, just 10 to 16 of the 327 participants produced ^wpkurtosis coefficients that exceeded 2, and 95% produced ^wpkurtosis between –1.1 and 2.2 to 3.0, depending on test score calibration. The descriptive values shown in Tables 3 and 4 provide provisional markers of what to expect from reasonably healthy adults who complete a test battery with 27 to 30 cognitive measures. In short, 100% of healthy adults showed normal levels of ^wpskew and over 95% showed normal ^wpkurtosis. Thus, regardless of how their cognitive performance was calibrated, the overwhelming majority of study participants produced normal within-person test score distributions across the measures included in this test battery.

We found higher rates of slightly negative than positive ^wpskew, which averaged –.11 for both age-calibrated and fully calibrated test scores. This is consistent with prior research (Binder et al., 2009) and suggests that it is more common for healthy adults to produce a few very low scores than a few very high scores. Conversely, in a previous study, patients referred for dementia workups showed higher rates of positive than negative ^wpskew (Reckess et al., 2014). In that study, healthy older adults produced a mean ^wpskew of –.1 (which is nearly identical to the ^wpskew seen here), while patients referred for dementia workups produced ^wpskew coefficients that averaged +.1 to 1.2, and increased with dementia severity. Taken together, these findings suggest that healthy adults tend to produce mildly negative and lower ^wpskew coefficients than persons with dementia, whose ^wpskew coefficients turn increasingly positive as their illness progresses and their cognitive test performance worsens.

In the present sample, abnormal ^wpkurtosis was always positive, and no participant showed ^wpkurtosis below –1.3. Negative kurtosis denotes a flat (platykurtic) distribution. This occurs when most scores are represented an equal number of times. For example, if one plots the number of times a six-sided die lands on each side after 100 throws, the resulting distribution will be platykurtic. Such a pattern is hard to imagine for a cognitive test battery. None of the participants in this study showed marked platykurtosis, which suggests that doing so would be highly unusual. In the 3.1% to 4.9% of participants who showed abnormal ^wpkurtosis, it was always leptokurtic. This means that an unusually large number of their scores were close to their mean while a few were farther than expected from it.

Another goal of this study was to examine how within-person distribution parameters correlate with one another. These analyses revealed that most correlations among within-person parameters were weak and statistically insignificant. The three exceptions all involved age-calibrated scores and showed that ^wpmeans correlated inversely with ^wpSD and positively with ^wpskew, while ^wpskew correlated inversely with ^wpkurtosis. These finding suggest that participants who produced relatively higher ^wpmeans showed slightly less variability and slightly more negatively skewed test score distributions, and that negatively skewed distributions were more leptokurtic, although these correlations were weak and likely of little clinical significance.

Of greater clinical significance is the question of whether within-person distribution parameters vary by participant characteristics. These analyses showed that uncalibrated ^wpmean scores varied with age, race, years of education, and estimated IQ in the expected directions (Table 5). Fully calibrating test performance uncoupled the association of ^wpmean scores with every participant characteristic. Calibrating test performance for age alone strengthened the association between ^wpmean T-score and IQ, while fully calibrating test performance weakened the same association. This is consistent with expectation because IQ scores are calibrated only for age. White participants produced higher ^wpmean scores than Black participants using uncalibrated and age-calibrated scores, but the difference disappeared with full calibration. Importantly, ^wpSD, ^wpskew, and ^wpkurtosis showed no association with any participant characteristic, implying that these aspects of cognitive performance are reasonably independent of a person’s age, sex, race, educational background, and estimated premorbid ability. Thus, the within-person test score parameters that define distribution normality—^wpskew and ^wpkurtosis—did not vary by any participant characteristic examined here.

We also assessed the impact of test score calibration on within-person distribution parameters with a series of paired-samples t-tests, which showed that degree of calibration had no effect on ^wpmean or ^wpkurtosis. Increasing the degree of calibration from none to age only to full calibration tended to broaden the within-person dispersion of test scores, as reflected by increasing ^wpSD from each level of calibration to the next. Also, both age and full calibration significantly lowered ^wpskew coefficients (from 0 to –.11) and increased the proportion of participants who showed negative ^wpskew compared with findings based on uncalibrated test scores. The impact of test score calibration on ^wpSD is interesting, but it does not affect within-person distribution normality, while the impact of test score calibration on ^wpskew does. However, Reckess et al. (2014) found that clinical patients tended to show higher rates of positive ^wpskew than healthy controls, that ^wpskew initially became increasingly positive as dementia severity worsened, and that average ^wpskew reached a zenith of 1.2 in the most severely demented patients tested in that study. Thus, using either age-calibrated or fully calibrated scores may serve to increase the sensitivity or specificity of using ^wpskew as a disease marker to the extent such calibration tends to decrease the ^wpskew coefficients of healthy adults.

In sum, the present findings indicate that a healthy adult’s performance across cognitive tests should approximate a Gaussian distribution in terms of within-person variance, skewness, and kurtosis, regardless of the person’s own test score mean, demographic background, and estimated premorbid IQ. If these results withstand cross-validation in other healthy samples and measures, then evaluating within-person test score distributions for departures from “normality” could offer a novel approach to clinical inference in neuropsychology. That is, computing an examinee’s test battery ^wpmean, ^wpSD, ^wpskew, and ^wpkurtosis, could enable the clinician or researcher to easily determine whether the distribution of a person’s cognitive performance is normal or abnormal. Prior research has found IIV to be a sensitive marker of underlying pathology in dementia (Holtzer et al., 2008; Reckess et al., 2014), Alzheimer’s disease (Duchek et al., 2009; Tractenberg & Pietrzak, 2011), Parkinson’s disease (de Frias et al., 2007), HIV (Arce Rentería et al., 2019;Vance et al., 2021), schizophrenia (Cole et al., 2011), and other conditions. Looking at ^wpskew and ^wpkurtosis in these clinical populations would complement the prior IIV research, and extend our understanding of their associated cognitive profiles.

One goal of the original ABC study (from which the current data were taken) was to establish norms for healthy adults across the adult age range. Volunteers were carefully screened for eligibility. Screening included physical and neurological examinations, psychiatric interviews, laboratory blood tests, and brain magnetic resonance imaging (MRI) scans. Thus, we are fairly confident in the composition of the sample. Of course, some participants could have had an undiagnosed disease or condition that affects cognitive function, possibly inflating the observed rates of abnormal ^wpskew or ^wpkurtosis. However, the finding that KS testing showed that no participant produced a ^wpskew greater than|1.6| argues against this for within-person skew. The presence of unrecognized disease could have contributed to the abnormal leptokurtosis shown by a few participants, but the fact that fewer than 4% of participants showed this pattern suggests that any such impact was likely minimal. Neither can we exclude the possibility that some aspect of our subject selection process depressed rates of abnormal within-person distributions, although we used a rigorous community sampling framework. Both possibilities highlight the need to replicate these findings in other healthy samples and measures.

A second potential limitation of this study is that we used a single battery of standard co-normed neuropsychological tests. It is possible that our findings are unique to this specific test battery. We tried to evaluate this possibility by examining the frequency with which each measure appeared as a person’s highest or lowest score. The resulting frequencies ranged from 1% to 12%. This range seems relatively small, especially since a person could have more than one “highest” or “lowest” score (e.g., two or more test scores tied for either position). Indeed, the total number of highest and lowest scores exceeds 100% of cases for precisely this reason. Thus, we found no single measure that tended to pull test score distributions in one direction or another. For this reason, while a different test battery could yield within-person test score distributions that are less Gaussian in shape, if the tests are scored using the same norms, and the sample consists of healthy adults, then it is reasonable to expect the present findings to withstand replication. Future research will be needed to determine the minimum number of cognitive tests or domains that need to be administered to obtain reliably normal within-person test score distributions.

Third, we did not formally assess performance validity, and some participants may have put forth suboptimal effort. However, study participants agreed to a full day of assessments that also included blood testing, brain imaging, neurological examination, and structured psychiatric interview, and most brought a family member or friend as a knowledgeable informant. These procedures likely served to discourage who were not quite motivated to join the study, and this belief was supported by the comments of many participants. In addition, we found very low rates of failing embedded performance validity indices for the tests administered. Just one participant (0.3%) scored below 6 on the Brief Test of Attention (BTA) (Busse & Whiteside, 2012) and two (0.6%) scored below 6 on the Hopkins Verbal Learning Test—Revised (HVLT-R) delayed Recognition Discrimination index (Bailey et al., 2018; Sawyer et al., 2017). Slightly more (4.3%) scored below demographically calibrated T-scores (using CNNS) of 34 for the Trail-Making Test Parts A and B (4.3% and 3.4%, respectively) or 30 for the Grooved Pegboard Test with their dominant and non-dominant hands (1.7% and 1.4%, respectively) (Abeare et al., 2019; Erdodi et al., 2018; Erdodi & Lichtenstein, 2021; Jinkerson et al., 2023; Link et al., 2022). These rates of performance validity test (PVT) failure are all lower than expected based on the specificity rates for all six of these embedded performance validity measures. For these reasons, we doubt that rates of invalid cognitive performance contributed significantly to the findings reported here.

Finally, this study does not clarify whether the few participants who produced elevated ^wpkurtosis did so as a result of more trait-like or state-like factors. If a small percentage of healthy persons reliably produce leptokurtic within-person distributions on cognitive testing, this might denote a pattern of “splinter” strengths or weaknesses that is atypical but not abnormal. On the other hand, if a small percentage of neurologically healthy persons produce leptokurtic within-person test score distributions, but the individuals comprising this subgroup differ over time, this could indicate that doing so reflects clinically meaningless “noise” in testing. This again underscores the need for replication, ideally using a design in which participants are tested twice to examine the stability of any observed departures from normality.

Conclusions and Future Directions

To our knowledge, this is the first demonstration that most healthy adults produce normal within-person distributions on a standardized neuropsychological test battery. If this proves to be a replicable and generalizable phenomenon, examining parameters of within-person test score distributions could offer a new, complementary approach to the diagnosis of cognitive disorders, above and beyond traditional deficit measurement. Just as researchers must inspect the normality of data before proceeding with statistical analyses, the present findings suggest that it may be important for clinicians to look for departures from normality of the within-person test score distributions of patients referred for assessment before interpreting a few low scores. In the research setting, these findings raise the intriguing possibility that an older adult diagnosed with mild cognitive impairment (MCI) based on a few low scores in an otherwise normal distribution might be at much lower risk of converting to dementia than a person with the same low scores in a clearly abnormal within-person distribution. For example, if the hypothetical person depicted in Figure 1B produced a few T-scores between 25 and 35, this might represent a higher risk of conversion to dementia than the same low scores produced by the hypothetical person depicted in Figure 1C.

Finally, in addition to its potential as a novel method of inference in neuropsychology, the fact that most healthy adults produce Gaussian within-person test score distributions is fascinating in its own right. At its core, this finding speaks to the fundamental way in which human beings are endowed with mental abilities. Just as diverse mental abilities are normally distributed in the population at large, it appears that diverse mental abilities are distributed normally within each member of that population.

Footnotes

Acknowledgements

The authors gratefully acknowledge Dr. Jason Brandt for creating the archival database from which data for the clinical samples were drawn.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Psychological Assessment Resources, Inc., D.J.S. is entitled to a share of royalties on sales of a test and software used in the study described in this article. The terms of this arrangement are being managed by the Johns Hopkins University in accordance with its conflict of interest policies.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the Therapeutic Cognitive Neuroscience Professorship (B.G.); the Therapeutic Cognitive Neuroscience Fund (B.G.); the Benjamin and Adith Miller Family Endowment on Aging, Alzheimer's, and Autism (B.G.); the William and Mary Ann Wockenfuss Research Fund Endowment (B.G.); and United States Department of Health and Human Services, National Institutes of Health, National Institute of Mental Health Grant MH60504 (D.J.S.).

ORCID iD

Alison S. Buchholz

References

Abeare

Sabelli

Taylor

Holcomb

Dumitrescu

Kirsch

Erdodi

(2019). The importance of demographically adjusted cutoffs: Age and education bias in raw score cutoffs within the Trail Making Test. Psychological Injury and Law, 12, 170–182.

Arce Rentería

Byrd

Coulehan

Miranda

Fuentes

Rosario

A. K.

Morris

E. P.

Rivera Mindt

. (2019). Neurocognitive intra-individual variability within HIV+ adults with and without current substance use. Neuropsychology, 34(3), 321–330.

Army Individual Test Battery. (1944). Manual of directions and scoring. War Department, Adjutant General’s Office.

Axelrod

B. N.

Millis

S. R.

(1994). Preliminary standardization of the Cognitive Estimation Test. Assessment, 1(3), 269–274.

Bailey

K. C.

Soble

J. R.

Bain

K. M.

Fullen

(2018). Embedded performance validity tests in the Hopkins verbal learning test—Revised and the brief visuospatial memory test—Revised: A replication study. Archives of Clinical Neuropsychology, 33(7), 895–900.

Bangen

K. J.

Weigand

A. J.

Thomas

K. R.

Delano-Wood

Clark

L. R.

Eppig

Werhane

M. L.

Edmonds

E. C.

Bondi

M. W.

(2019). Cognitive dispersion is a sensitive marker for early neurodegenerative changes and functional decline in nondemented older adults. Neuropsychology, 33(5), 599–608. https://doi.org/10.1037/neu0000532

Bellgrove

M. A.

Hester

Garavan

(2004). The functional neuroanatomical correlates of response variability: Evidence from a response inhibition task. Neuropsychologia, 42(14), 1910–1916.

Benedict

H. R. B.

(1997). Brief Visuospatial Memory Test-Revised professional manual. Psychological Assessment Resources.

Benton

A. L.

Sivan

A. B.

deS Hamsher

Varney

N. R.

Spreen

(1994). Contributions to neuropsychological assessment: A clinical manual (2nd ed.). Oxford University Press.

10.

Binder

L. M.

Iverson

G. L.

Brooks

B. L.

(2009). To err is human: “Abnormal” neuropsychological scores and variability are common in healthy adults. Archives of Clinical Neuropsychology, 24(1), 31–46.

11.

Brandt

Benedict

H. R. B.

(2001). Hopkins Verbal Learning Test-Revised professional manual. Psychological Assessment Resources.

12.

Bunce

Anstey

K. J.

Christensen

Dear

Wen

Sachdev

(2007). White matter hyperintensities and within-person variability in community-dwelling adults aged 60–64 years. Neuropsychologia, 45(9), 2009–2015.

13.

Busse

Whiteside

(2012). Detecting suboptimal cognitive effort: Classification accuracy of the Conner’s continuous performance test-II, brief test of attention, and trail making test. The Clinical Neuropsychologist, 26(4), 675–687.

14.

Cole

V. T.

Weinberger

D. R.

Dickinson

(2011). Intra-individual variability across neuropsychological tasks in schizophrenia: A comparison of patients, their siblings, and healthy controls. Schizophrenia Research, 129(1), 91–93.

15.

Corwin

Bylsma

F. W.

(1993). Psychological examination of traumatic encephalopathy. The Clinical Neuropsychologist, 7, 3–21.

16.

de Frias

C. M.

Dixon

R. A.

Fisher

Camicioli

. (2007). Intraindividual variability in neurocognitive speed: A comparison of Parkinson’s disease and normal older adults. Neuropsychologia, 45(11), 2499–2507.

17.

Duchek

J. M.

Balota

D. A.

Tse

C. S.

Holtzman

D. M.

Fagan

A. M.

Goate

A. M.

(2009). The utility of intraindividual variability in selective attention tasks as an early marker for Alzheimer’s disease. Neuropsychology, 23(6), 746–758.

18.

Dykiert

Der

Starr

J. M.

Deary

I. J.

(2012). Age differences in intra-individual variability in simple and choice reaction time: Systematic review and meta-analysis. PLOS ONE, 7(10), e45759. https://doi.org/10.1371/journal.pone.0045759

19.

Erdodi

L. A.

Kirsch

N. L.

Sabelli

A. G.

Abeare

C. A.

(2018). The Grooved Pegboard Test as a validity indicator—A study on psychogenic interference as a confound in performance validity research. Psychological Injury and Law, 11, 307–324.

20.

Erdodi

L. A.

Lichtenstein

J. D.

(2021). Information processing speed tests as PVTs. In Boone

K. B.

(Ed.), Assessment of feigned cognitive impairment. A neuropsychological perspective (pp. 218–247). The Guilford Press.

21.

Folstein

M. F.

Folstein

S. E.

McHugh

P. R.

(1975). “Mini-mental state..” A practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research, 12(3), 189–198.

22.

Hair

J. F.

Jr. Hult

G. T. M.

Ringle

C. M.

Sarstedt

(2021). A primer on partial least squares structural equation modeling (PLS-SEM). Sage.

23.

Hair

J. F.

Jr. Ringle

C. M.

Sarstedt

(2013). Partial least squares structural equation modeling: Rigorous applications, better results and higher acceptance. Long Range Planning, 46(1–2), 1–12.

24.

Halliday

D. W.

Gawryluk

J. R.

Garcia-Barrera

M. A.

MacDonald

S. W.

(2019). White matter integrity is associated with intraindividual variability in neuropsychological test performance in healthy older adults. Frontiers in Human Neuroscience, 13, 352.

25.

Hilborn

J. V.

Strauss

Hultsch

D. F.

Hunter

M. A.

(2009). Intraindividual variability across cognitive domains: Investigation of dispersion levels and performance profiles in older adults. Journal of Clinical and Experimental Neuropsychology, 31(4), 412–424.

26.

Holtzer

Verghese

Wang

Hall

C. B.

Lipton

R. B.

(2008). Within-person across-neuropsychological test variability and incident dementia. JAMA, 300(7), 823–830.

27.

Jinkerson

J. D.

L. H.

Kennedy

Armistead-Jehle

Nelson

J. T.

Seegmiller

R. A.

(2023). Grooved Pegboard adds incremental value over memory-apparent performance validity tests in predicting psychiatric symptom report. Applied Neuropsychology: Adult, 24, 1–9

28.

Kaufman

A. S.

(1976). Verbal-performance IQ discrepancies on the WISC-R. Journal of Consulting and Clinical Psychology, 44(5), 739–744.

29.

Kline

R. B.

Snyder

Guilmette

Castellanos

(1993). External validity of the profile variability index for the K-ABC, Stanford-Binet, and WISC-R: Another cul-de-sac. Journal of Learning Disabilities, 26(8), 557–567.

30.

Klove

(1963). Clinical neuropsychology. In Forster

F.M.

(Ed.), Medical clinics of North America (pp. 1647–1658). Saunders.

31.

Lindenberger

Baltes

P. B.

(1997). Intellectual functioning in old and very old age: Cross-sectional results from the Berlin Aging Study. Psychology and Aging, 12(3), 410–432.

32.

Link

J. S.

L. H.

Armistead-Jehle

Seegmiller

R. A.

(2022). Validation of Grooved Pegboard cutoffs as an additional embedded measure of performance validity. The Clinical Neuropsychologist, 36(8), 2331–2341. https://doi.org/10.1080/13854046.2021.1942556

33.

Manning

K. J.

Gordon

Pearlson

G. D.

Schretlen

D. J.

(2007). The relationship of recency discrimination to explicit memory and executive functioning. Journal of the International Neuropsychological Society, 13(4), 710–715.

34.

Matarazzo

J. D.

(1990). Psychological assessment versus psychological testing: Validation from Binet to the school, clinic, and courtroom. American Psychologist, 45(9), 999–1017.

35.

McLean

J. E.

Kaufman

A. S.

Reynolds

C. R.

(1989). Base rates of WAIS-R subtest scatter as a guide for clinical and neuropsychological assessment. Journal of Clinical Psychology, 45(6), 919–926.

36.

Osterrieth

P. A.

(1944). Le test de copie d'une figure complexe; contribution à l'étude de la perception et de la mémoire [Test of copying a complex figure; contribution to the study of perception and memory]. Archives de Psychologie, 30, 206–356.

37.

Rabbitt

(1993). Does it all go together when it goes? The Nineteenth Bartlett Memorial Lecture. The Quarterly Journal of Experimental Psychology A: Human Experimental Psychology, 46A(3), 385–434.

38.

Reckess

G. Z.

Varvaris

Gordon

Schretlen

D. J.

(2014). Within-person distributions of neuropsychological test scores as a function of dementia severity. Neuropsychology, 28(2), 254–260.

39.

Reitan

R. M.

Davison

L. A.

(1974). Clinical neuropsychology: Current status and applications. V. H. Winston & Sons.

40.

Rey

(1941). L'examen psychologique dans les cas d'encephalopathie traumatique. Les problems [The psychological examination in cases of traumatic encepholopathy. Problems]. Archives de Psychologie, 28, 215–285.

41.

Salthouse

T. A.

Babcock

R. L.

(1991). Decomposing adult age differences in working memory. Developmental Psychology, 27, 763–776.

42.

Sawyer

R. J.

Testa

S. M.

Dux

(2017). Embedded performance validity tests within the Hopkins verbal learning test–Revised and the brief visuospatial memory test–Revised. The Clinical Neuropsychologist, 31(1), 207–218.

43.

Schretlen

D. J.

(1997). Brief Test of Attention professional manual. Psychological Assessment Resources.

44.

Schretlen

D. J.

(2010). Modified Wisconsin Card Sorting Test (M-WCST) professional manual. Psychological Assessment Resources.

45.

Schretlen

D. J.

Munro

C. A.

Anthony

J. C.

Pearlson

G. D.

(2003). Examining the range of normal intraindividual variability in neuropsychological test performance. Journal of the International Neuropsychological Society, 9(6), 864–870.

46.

Schretlen

D. J.

Sullivan

A. C.

(2013). Intra-individual variability in cognitive test performance. In Koffler

Morgan

Baron

I. S.

Greiffenstein

(Eds.), Neuropsychology science and practice (Vol. 1, pp. 39–60). Oxford University Press.

47.

Schretlen

D. J.

Testa

S. M.

Pearlson

G. D.

Staff

(2010). Calibrated Neuropsychological Normative System Software Portfolio (CNNS-SP). Psychological Assessment Resources.

48.

Schretlen

D. J.

Testa

S. M.

Winicki

J. M.

Pearlson

G. D.

Gordon

(2008). Frequency and bases of abnormal performance by healthy adults on neuropsychological testing. Journal of the International Neuropsychological Society, 14(3), 436–445.

49.

Schretlen

D. J.

Vannorsdall

T. D.

(2010). Calibrated Ideational Fluency Assessment (CIFA) prefessional manual. Psychological Assessment Resources.

50.

Schretlen

D. J.

Winicki

J. M.

Meyer

S. M.

Testa

S. M.

Pearlson

G. D.

Gordon

(2009). Development, psychometric properties, and validity of the Hopkins Adult Reading Test (HART). Clinical Neuropsychologist, 23(6), 926–943.

51.

Selnes

O. A.

Carson

Rovner

Gordon

(1988). Language dysfunction in early- and late-onset possible Alzheimer’s disease. Neurology, 38(7), 1053–1056.

52.

Shallice

Evans

M. E.

(1978). The involvement of the frontal lobes in cognitive estimation. Cortex, 14(2), 294–303.

53.

Testa

S. M.

Winicki

J. M.

Pearlson

G. D.

Gordon

Schretlen

D. J.

(2009). Accounting for estimated IQ in neuropsychological test performance with regression-based techniques. Journal of the International Neuropsychological Society, 15(6), 1012–1022.

54.

Tractenberg

R. E.

Pietrzak

R. H.

(2011). Intra-individual variability in Alzheimer’s disease and cognitive aging: Definitions, context, and effect sizes. PLOS ONE, 6(4), e16973.

55.

Vance

D. E.

Del Bene

V. A.

Frank

J. S.

Billings

Triebel

Buchholz

Rubin

L. H.

Woods

S. P.

Fazeli

P. L.

(2021). Cognitive intra-individual variability in HIV: An integrative review. Neuropsychology Review, 32(4), 855–876.

56.

Wechsler

(1939). The measurement of adult intelligence. Williams & Wilkins Co.

57.

Wechsler

(1981). Wechsler Adult Intelligence Scale-Revised. Psychological Corporation.

58.

Wechsler

(1987). Wechsler Memory Scale-Revised. The Psychological Corporation.

59.

Wechsler

(1997). Wechsler Memory Scale—Third edition. The Psychological Corporation.