Cross-Validating the Executive Errors Scale of the Repeatable Battery for the Assessment of Neuropsychological Status

Abstract

The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) was developed to briefly measure a broad range of cognitive abilities, but it initially lacked a scale to evaluate executive functioning. Robert Spencer and colleagues recently created an Executive Errors scale (RBANS-EE) that quantified executive functioning (EF) errors committed during four RBANS subtests: List Learning, Semantic Fluency, Coding, and List Recall. In the present paper we cross-validated the RBANS-EE with a sample of 234 U.S. military veterans (M _AGE = 67.2, SD = 11.5 years; M _EDUCATION = 13.3, SD = 2.4 years) who completed the RBANS and various EF criterion measures as part of neuropsychological assessments they underwent during their clinical care. We found the RBANS-EE to be significantly correlated with most of the criterion EF measures. The RBANS-EE scale demonstrated modest ability to classify EF impairment at mild and severe levels; and, similarly, the RBANS-EE was modestly capable of accurately classifying those veteran respondents who were determined to have or to not have a neurocognitive disorder. Overall, the RBANS-EE can be quickly calculated, adds no administration time to an RBANS assessment, and yields useful scores to screen for EF dysfunction without replacing standalone EF tests.

Keywords

executive functioning repeatable battery for the assessment of neuropsychological status assessment veterans

Introduction

Randolph (1998) developed the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) as a brief, objective measure of cognitive abilities. More recently, the RBANS has been rated as the seventh most frequently used neuropsychological assessment instrument, and it was rated the third most frequently used mental status or global cognition instrument (Rabin et al., 2016). The RBANS has demonstrated adequate reliability (Duff et al., 2003; Randolph, 2012) and satisfactory validity for detecting cognitive impairment (Duff et al., 2008). In terms of comprehensiveness, the RBANS falls between a cognitive screening test and a full-fledged neuropsychological test battery (Block, Johnson-Greene, et al., 2017). The RBANS is ideal when stand-alone screening tests do not provide sufficient coverage or depth and full evaluations are infeasible or justifiable (Groth-Marnat & Wright, 2016). In clinical practice, the RBANS is often used as a stand-alone instrument in evaluations for dementia or as part of a larger test battery with individuals who have subtler forms of cognitive dysfunction (Randolph, 2012). Among its limitations, however, it does not include a specific scale for measuring executive function (EF; Shaughnessy et al., 2019).

Executive function is a multi-faceted construct involving many aspects of cognition, including organizing, planning, and problem-solving. Executive dysfunction is central, or involved to varying degrees, in many neurologic disorders, including Huntington’s Disease, frontotemporal dementia, cortico-frontal cerebrovascular accidents and neoplasms, and traumatic brain injury (Caeyenberghs et al., 2014; Fiorention et al., 2013; You et al., 2014); and it characterizes various forms of psychopathology, such as Attention-deficit/Hyperactivity Disorder (ADHD; Willcut et al., 2005). For these reasons, the assessment of EF is critical to an accurate diagnosis in many clinical situations (Rabinovici et al., 2015), and many stand-alone tests have been developed to assess a broad range of separate abilities that fall beneath the EF umbrella (see Luria, 1980; Lezak et al., 2012; Strauss et al., 2006).

To address the RBANS weakness for measuring executive dysfunction, attempts have been made to reapportion standard metrics from within the RBANS with tasks that might relate to EF or be mediated by brain regions associated with EF. For example, Randolph et al. (1998) proposed examining a cortical/subcortical ratio score that can be derived from the RBANS to compare an examinee’s performances on tests believed to be responsive to those anatomical brain regions. Duff et al. (2009) further proposed examining scores on visual versus verbal subtests, noting that factor analyses have generally uncovered visual and verbal factor scores. These methods, however useful they may be for some clinical purposes, have not explicitly addressed executive dysfunction, nor do they examine qualitative aspects of an examinee’s performance.

Within its various subtests, the RBANS offers many opportunities to sample lapses in EF that are not explicitly quantified in existing indices or subtest scores. In research with other tests, investigators have shown that errors of intrusions and repetitions observed during fluency tasks and on tests involving learning and recalling word lists can represent not only specific memory lapses, but also failures to keep track of prior responses and/or the objectives of the task(s) at hand that increase with advancing age (Hankee, 2013; Rasmusson et al., 1998) and are evident among individuals with executive dysfunction (Holden et al., 2020; Kiselica & Benge, 2019). Although only modestly reliable (Woods et al., 2005) when used in isolation (e.g., Delis et al., 2017), scores based on these errors have been shown to correlate with formal measures of EF (Hill et al., 2012; Smith et al., 2020) and frontal lobe brain integrity (Baldo et al., 2002). Although we are not aware of aggregated EF scores from separate tests, such combinations may confer psychometric advantages in that aggregated evidence from varied sources may provide a particularly robust single measure of EF.

To this end, Spencer et al. (2018) developed the RBANS Executive Errors scale (RBANS-EE) in a post-hoc fashion, using errors on items and subtests throughout the RBANS to quantify several different aspects of EF into a single measure. The RBANS-EE score includes the total number of repetition and intrusion errors occurring during the RBANS subtests of List Learning, List Recall, and Semantic Fluency, as well as errors made during the Coding subtest. The RBANS-EE produces an index or percentage score by dividing the sum of these various errors by the total number of responses on those subtests. Higher RBANS-EE scores indicate a more problematic performance. In their initial study, Spencer et al. (2018) found that individuals with RBANS-EE scores below 5% did not typically perform poorly on other tests of EF within a broad neuropsychological test battery, whereas individuals with scores higher than 15.0% frequently exhibited parallel difficulties on other tests of EF. In a subsequent study, Ringdahl et al. (2019) observed that individuals classified as having no neurocognitive disorder, a mild neurocognitive disorder, and a major neurocognitive disorder obtained average RBANS-EE scores of 1.4%, 6.0%, and 9.5%, respectively. Additionally, examinees’ RBANS-EE scores have been moderately to strongly positively correlated with other tests of executive dysfunction, and these scores successfully distinguished people with and without neurocognitive disorders (area under the curve [AUC] = .88). More recently, Bolton et al. (2020) found moderate correlations between RBANS-EE scores and other measures of EF, with an adequate ability to identify impairment (defined as scores that were 1.5 standard deviations below the mean for respondents in the normative sample) on the Trail Making Test Part B (TMT B) in a sample of older adults.

Despite encouraging data supporting the use of RBANS-EE as a screening instrument for executive dysfunction with clear construct validity, the scale requires both cross-validation in other clinical samples and evidence that it provides either incremental utility within a broad neuropsychological test battery that includes other measures of EF or the capacity to substitute more briefly for these other EF measures. Accordingly, we had four broad aims in the current project. First, we sought to re-address the RBANS-EE construct and replicate its concurrent validity, hypothesizing a pattern of positive correlations between the RBANS-EE and other measures of EF in another participant sample. Second, we sought to cross validate Spencer et al.’s (2018) earlier neuropsychological impairment screening statistics, hypothesizing that the RBANS-EE scale would successfully detect executive dysfunction defined categorically at 1.0 SD and 2.0 SD below the mean score for the normative population on the other measures of EF. Third, we planned to analyze the RBANS-EE’s predictive validity for classifying examinees with and without a neurocognitive disorder and hypothesized that it would be comparable in its predictive validity for this purpose with other EF criterion measures. Finally, we sought to examine the incremental validity of the RBANS-EE scale, predicting that it would predict executive dysfunction at levels beyond the other criterion tests of EF.

Method

Participants

Participants in this study were 313 U.S. military veterans who completed the RBANS as part of a comprehensive neuropsychological evaluation within a Midwestern Veterans Affairs facility. We used retrospective clinical data for this purpose, with approval by the Institutional Review Board of this facility. As these data were collected from clinical evaluations that did not use a fixed battery, clinicians decided which measures to administer, based on the specifics of each patient’s referral question. They tended to use the RBANS for older veterans and tended to give formal measures of executive functioning to all but the most profoundly impaired. We excluded 29 veteran participants from analyses because of invalid test data, as determined by clinical consensus among the authors after reviewing behavioral and psychometric data. We excluded data from another 50 veteran participants who were not administered other EF criterion measures. Thus, the primary remaining participant sample for further data analyses was composed of 234 veterans who received neuropsychological assessments as part of their medical care (93.2% male; M _AGE = 67.2 years, SD = 11.5; M _EDUCATION = 13.3 years, SD = 2.4). The sample was predominantly White (91.9%) and Black (7.3%) with other races combined comprising less than 1% of the sample. Chi-square tests and analyses of variance (ANOVA) that compared differences between (a) those with problematic performance validity, (b) those not taking enough EF tests, and (c) the final sample did not differ with respect to sex, years of education, race, or Effort Index scores. However, participants in the final, selected sample tended to be older than those in the two excluded groups. The primary diagnoses, made at the conclusion of the neuropsychological evaluation and not incorporating the RBANS EE performance, included (a) mild neurocognitive disorder (n = 70, 29.9%); (b) major neurocognitive disorder (n = 49, 20.9%); (c) psychiatric disorder (n = 44, 18.8%); (d) sleep disorder (n = 4, 1.7%); (e) substance use disorder (n = 2, .9%); (f) no diagnosis (n = 59, 25.2%)l and (g) other (n = 5, 2.1%). These diagnoses were dichotomized into those with a neurocognitive disorder (n = 123, 52.6%) and those without a neurocognitive disorder (n = 111, 47.4%).

Measures

As noted earlier, the RBANS-EE scale is an EF composite index that and is a summation of errors committed on the List Learning, Semantic Fluency, Coding, and List Recall subtests, divided by the total responses (including errors) on those subtests (Spencer et al., 2018). The RBANS-EE score is multiplied by 100 to generate a percentage score. We used percentages throughout our analyses because no age stratified normative data pertaining to RBANS-EE exist.

Criterion measures included the Trail Making Test Part B (TMT B; Reitan, 1955), letter-based oral word fluency (FAS; Benton et al., 1976), semantically-based oral word fluency (Animals; Goodglass et al., 1972), and Wisconsin Card Sorting Test perseverative errors (WCST-PE; Heaton et al., 1993). Raw scores for the criterion measures were transformed into normative data, based on test manual norms for WCST-PE performances and Mitrushina et al. (2005) norms for TMT B, FAS, and Animals performances. EF criterion scores were averaged together to create composite scores, and levels of EF impairment were defined as 1.0 (mild) and 2.0 (severe) standard deviations below the normative mean on the composite score.

Design and Procedure

We calculated descriptive statistics for all measures, and we examined the relationship between the RBANS-EE scale score and its components with the criterion measures using Spearman correlations due to significant positive skew (1.33–.16) and kurtosis (2.00–.32). We used corrected item-total correlations to report on the internal consistency of the RBANS-EE, but we acknowledge difficulty attaining a respectable value due to characteristics of the measure (e.g., floor effects, non-normal distribution, small number of scale items). Normative scores for the EF criterion measures were averaged together to create EF composite scores. We examined the predictive validity of the RBANS-EE scale in classifying EF impairment at two levels, mild and severe, as defined as EF composite scores at −1.0 and −2.0 standard deviations below the normative mean, respectively. Predictive validity was considered adequate if the receiver operating characteristic area under the curve (AUC) was .70 or greater (Hosmer et al., 2000). We compared the AUC of RBANS-EE and the criterion measures to detect whether participants were or were not diagnosed with a neurocognitive disorder as their primary diagnosis. We assigned neurocognitive disorder diagnoses in accordance with the Diagnostic and Statistical Manual of Mental Disorders—Fifth Edition (DSM-5; American Psychiatric Association, 2013). To test whether the RBANS-EE added predictive value in assessing EF, we used hierarchical regressions with the criterion measures with significant correlations with RBANS-EE. In each case, one criterion measure served as the dependent variable and the other(s) were entered in step one of the regressions to test whether RBANS-EE at step two added incremental prediction, as assessed by a change in variance accounted for (R²).

Results

The participant sample averaged an RBANS-EE score of 5.9% (SD = 5.0) with a range of .0–26.5%, and these scores significantly correlated with age, r = .26 (p < .001). Sex and years of education were not significantly related to RBANS-EE scores. Among the components of the RBANS-EE scale, the only significant inter-correlation observed was between the List Learning and List Recall (r = .20, p = .002).

Table 1 displays the mean scores and inter-correlations between the RBANS-EE, its components and the four criterion measures. RBANS-EE total and component scores were compared with each of the four criterion measures. In all instances, the total RBANS-EE score, relative to its components, shared a stronger correlation with the criterion measures. All correlations, except for List Recall Errors, were in the expected direction. Notably, List Recall errors had the most variability among the RBANS-EE components. Increased RBANS-EE scores were significantly correlated with slower time on TMT B (r = −.42, p < .001), more perseverative errors on the WCST-PE (r = .33, p < .001), and lower scores for semantic fluency (r = −.20, p < .01), but no significant correlation was observed with letter-based oral word fluency (r = −.12), although the association was in the expected direction. TMT B was significantly correlated with most of the component scores for RBANS-EE, letter-based oral word fluency was not significantly related to any RBANS-EE component, and semantically-based oral word fluency and WCST-PE perseverative errors were significantly correlated with some RBANS-EE component scores. Among the RBANS-EE components, Coding Errors were the most consistently correlated with the criterion measures. Regarding divergent validity, RBANS-EE demonstrated small to medium negative correlations with RBANS index scores (Immediate Memory Index [r = −.32, p < .001]; Visuospatial/Construction Index [r = −.32, p < .001]; Language Index [r = −.19, p = .003]; Attention Index [r = −.29, p < .001]; Delayed Memory Index [r = −.35, p < .001]). Importantly, these correlations may be slightly inflated, as aspects of the subtests that comprise the index scores contribute to the denominators of RBANS-EE scores. Overall, these associations were weaker than those of RBANS indices with Trails B, and they were comparable to WCST-PE. The internal consistency, measured by corrected item-total correlations, ranged from .06 (Coding errors) to .16 (List Learning errors). This is potentially a weakness of the scale, as it indicates that each item is relatively independent of the others. Nevertheless, each of these somewhat orthogonal predictors were significantly correlated with at least one traditional EF test. The total RBANS-EE scale correlated with each component, List Learning (r = .71), Semantic Fluency (r = .37), Coding (r = .47), and List Recall (r = .31).

Table 1.

Descriptive Statistics and Spearman Correlation Matrix for the RBANS EE and Criterion Measures.

	M (SD)	TMT B	FAS	Animals	WCST-PEa
	M (SD)	166.7 (86.8)	29.6 (11.5)	15.3 (5.3)	26.6 (19.0)
RBANS-EE	5.9 (5.0)	.42***	−.12	−.20**	.33***
LL errors	8.8 (9.3)	.15*	−.00	−.05	.16
SF errors	5.2 (7.4)	.16*	−.03	−.5	.01
CD errors	3.8 (7.4)	.30***	−.11	−.13*	.20*
LR errors	6.6 (17.3)	−.05	.03	.15*	−.04

Notes. N = 234.

^an = 149; Error type is represented by the percentage of errors per the total number of responses.

EE = Executive Errors; LL = List Learning; SF = Semantic Fluency; CD = Coding; LR = List Recall; TMT = Trail Making Test; WCST-PE = Wisconsin Card Sorting Test Perseverative Errors.

***p < .001 **p < .01 *p < .05.

We examined the rate of mild and severe EF impairment, according to the four criterion measures, stratified by RBANS-EE score. From Table 2, in a sample where EF was mildly impaired (more than one standard deviation below the age-referenced mean) in 38.9% of participants and severely impaired (more than two standard deviations below the age-referenced mean) in 8.5%, RBANS-EE scores below 5% were associated with reduced risk of impairment and scores above 5% were associated with increased risk. Using >5% as a cutoff, RBANS-EE had 64% sensitivity and specificity to detect mild executive impairments and 70% sensitivity and 55% specificity to detect severe impairments. Viewed more continuously, RBANS-EE scores of 1% or less obtained mildly and severely impaired executive functioning scores in 19.2% and 3.8% of the cases. Conversely, among those with RBANS-EE scores above 15%, three-quarters of these participants experienced at least mild executive functioning impairments and one quarter had severe impairments.

Table 2.

Executive Errors Percent Impaired on Executive Functioning Composite.

RBANS-EE	N	Mild EF Impairment, %	Severe EF Impairment, %
.0–1.0%	26	19.2	3.8
1.1–5.0%	98	28.6	5.1
5.1–10.0%	66	45.5	10.6
10.1–15.0%	32	59.4	12.5
>15.0%	12	75.0	25.0
Total	234	38.9	8.5

Notes. EF = Executive functioning composite; Mild and severe EF was defined as 1.0 and 2.0 standard deviations below the normative mean, respectively.

Receiver operating characteristic areas under the curve (AUC) were used to determine the degree to which the RBANS-EE predicted mild and severe EF impairments as defined by the four criterion measures, individually and in composite scores. Displayed in Table 3, the RBANS-EE detected mild impairments with letter-based oral word fluency and severe impairments with letter-based oral word fluency, semantically-based oral word fluency, and WCST-PE at chance levels. The RBANS-EE was associated with mild and severe impairment on TMT B and the Executive Composite score at levels just shy of clinically adequate (.70) and reached adequate levels for detecting mild (.71) and severe (.78) impairments on a composite measure including TMT B and WCST perseverative errors. To offer perspective on the relative value of each test for detecting cognitive impairment, broadly defined, we used AUCs for RBANS-EE and the four criterion measures. As displayed in Table 4, the RBANS-EE obtained a value of .68 (.59–.76), which was marginally superior to one of the criterion measures and inferior to the other three.

Table 3.

RBANS-EE Scale Classification Analyses of Executive Dysfunction.

Test	Mild Impairment			Severe Impairment
Test	BR%	AUC	(CI 95%)	BR%	AUC	(CI 95%)
EF composite	38.9%	.68	(.61–.75)	8.5%	.69	(.55–.82)
TMT & WCST composite^a	35.6%	.71	(.62–.80)	10.7%	.78	(.67–.89)
TMT B	46.2%	.68	(.61–.74)	32.5	.67	(.60–.74)
FAS	46.6%	.51	(.43–.58)	12.0%	.49	(.37–.60)
Animals	40.1%	.57	(.50–.65)	6.4%	.53	(.36–.71)
WCST-PE^a	28.2%	.60	(.50–.70)	6.0%	.60	(.38–.81)

Notes. N = 234.

^an = 149.

EF = Executive functioning.

Table 4.

RBANS-EE Scale and Criterion Measures Classification of Neurocognitive Disorder.

Test	AUC	(CI 95%)
Executive errors	.68	(.59–.76)
TMT B	.83	(.77–.90)
FAS	.67	(.58–.76)
Animals	.79	(.71–.86)
WCST-PE^a	.79	(.72–.87)

Notes. N = 234.

^an = 149.

Base rate of neurocognitive disorder = 52.6%.

The TMT B and WCST-PE were most strongly related to the RBANS-EE, through both correlational and AUC analyses. Thus, in supplemental analyses, we used hierarchical regression analyses to test whether the RBANS-EE added unique predictive information in assessing these two criterion measures. For each analysis, one criterion measure served as the dependent variable, and the other criterion measure was entered as a predictor in step one of the models with RBANS-EE added in step two. In predicting perseverative errors, TMT B scores were significant at step one (R² = .258, p < .001) and the RBANS-EE added 1.6% (p = .078). In predicting TMT B scores, perseverative errors were significant at step one (R² = .258, p < .001) and the RBANS-EE added 7.6% (p < .001).

Discussion

The RBANS-EE, although related to diagnoses of neurocognitive disorders and generally correlated with stand-alone measures of EF, was not sufficiently precise to replace the use of standard measures of EF. The RBANS-EE still provided relevant data and merits additional research, but it has yet to demonstrate stand-alone validity as an EF measure. These conclusions differ from the results of other recent efforts to validate the RBANS-EE (Bolton et al., 2020; Ringdahl et al., 2019; Spencer et al., 2018).

Although parsing apart cognitive constructs necessarily involves semantics, it is possible to articulate how the RBANS-EE functions phenomenologically as a test of EF. Manifestly, the RBANS-EE measures lapses in attention (Coding errors) as well as poor self-monitoring (repetition and intrusion errors). Statistically, our results are generally consistent with prior studies (Bolton et al., 2020; Ringdahl et al., 2019; Spencer et al., 2018) that reported strong correlations between the RBANS-EE and both TMT B, which measures processing speed, working memory, and set shifting, and WCST-PE, which measures mental flexibility and other aspects of metacognition. Unexpectedly, the RBANS-EE was not significantly related to either of the two oral word fluency tasks or to List Recall errors. It is possible that fluency tasks differed significantly from the component scores within the RBANS-EE, although these scores have been moderately correlated in other samples (Ringdahl et al., 2019; Spencer et al., 2018). List Recall errors had the most variability, the largest positive skew, and relatively few occurrences in our sample, which may account for these discrepant results with Ringdahl et al. (2019) who found List Recall errors to have diagnostic value. Additional research is needed to determine which components account for the most variance in the overall scale and clarify which facets of executive functioning have sufficient coverage.

Regarding clinical utility, the RBANS-EE was nearly as predictive of clinical diagnosis as were the other criterion measures. Numerically, the RBANS-EE was marginally more predictive than letter-based oral word fluency, and was less predictive than semantically-based oral word fluency, WCST-PE, and TMT B. However, because data for the criterion tests were available to the individuals determining clinical diagnoses while the RBANS-EE was not used for this purpose, the actual values on these tests may be more equivalent. Additional research is needed to examine this issue with criterion measures based on independent sources of data, such as brain imaging, functional status, biomarkers, or prognoses.

Regarding incremental utility, the RBANS-EE did not predict WSCT-PE performance beyond TMT B; however, RBANS-EE did predict TMT B beyond WCST-PE. These results indicate that RBANS-EE in its current form has marginal incremental validity within a battery that includes other tests of EF, and therefore its scores serve more of a screening or triage function than that of an independent measure of executive functioning within a battery.

With modifications, RBANS-EE, or at least the concept underlying RBANS-EE can be applied to other settings and with other cognitive tasks. One concept apparent across studies examining the RBANS-EE is that each individual component measure has relatively weak validity when compared to the collective measure. Therefore, the value of the RBANS-EE is in the aggregation of its component parts; attending to any one aspect of the test would invite excessive randomness to the evaluation. Perhaps additional research could take a similar approach as the RBANS-EE by adding additional dysexecutive observations to a composite scale. Although typically evaluated in isolation from other qualitative indices, other qualitive errors have demonstrated validity in detecting executive dysfunction. Tolle et al. (2019) observed that tabulating the violations from the instructions of the examiner when engaging in the Grooved Pegboard was associated with executive dysfunction generally, but especially with impulsivity. Possin et al. (2012) found that repetitions during design fluency was a distinctive characteristic of individuals with behavioral variant frontotemporal dementia and those with lesions to the right frontal lobe. Similarly, the number of errors committed during TMT B is strongly predictive of a behavioral variant of frontotemporal dementia and frontal lobe lesions (Kiselica & Benge, 2019) and may be more predictive than time to completion (Kopp et al., 2015). Perhaps adding qualitative observations from more tests could lead to a more stable, battery-wide, aggregate collection of otherwise-discarded measures into a useful metric.

Limitations and Directions for Further Research

This study and RBANS-EE have significant limitations. Foremost among them is that the concept of EF is broad, heterogenous, and complex. While executive functioning is often discussed as a single entity, it is varied in its elements. Thus, it is unrealistic for any summary measure to encompass the entire construct. No single measure can capture all its qualities, and abnormal summary scores should only be used as signal flags for to delving into a deeper assessment, perhaps by altering neuropsychological test batteries to more fully explore component abilities subsumed under the executive functioning umbrella. In this study, we followed the direction of Spencer et al. (2018) who considered errors on coding as either impulsive or the consequence of haste. We regarded repetitions and intrusions as examples poor self-monitoring and failures to distinguish between presented and novel information, respectively. We ultimately do not believe that evaluators should rely solely on the RBANS EE as a robust measure of EF; rather they should acknowledge that it quantifies observable behavior that helps mark some aspects of EF and has adequate psychometric justification as an acceptable screening tool. Separate tests of component skills falling under the umbrella of EF tend to correlate only moderately (e.g., Delis-Kaplan Executive Function System; Delis et al., 2001), and, like these tools, the RBANS-EE does not represent a single construct. This point is illustrated by the scale’s poor internal consistency but adequate correlations with criterion measures. If the items are correlated with outcomes and have cumulative predictive utility, internal consistency becomes less relevant. Scales with uncorrelated items are not necessarily undesirable, provided the measures correlate with their intended criteria. With these caveats, high RBANS-EE values indicate potential carelessness and/or deficient self-monitoring, and these problems should be further investigated with additional complementary methodologies.

For this retrospective study, we employed a demographically narrow participant sample consisting of predominantly male veterans, and data regarding race and ethnicity were not recorded. As a result, our findings may not be representative of populations with broader demographics, likely limiting their generalizability. The RBANS-EE should be evaluated in more diverse samples. Further research should also use other aspects of EF as criterion measures. For example, Spencer et al. (2018) included seven executive measures that, included tests of social judgment, planning, and response inhibition. Until future researchers explore the relationship between RBANS-EE and a wider array of EF skills, we urge caution in interpreting RBANS-EE performances.

While there are advantages to constructing EF tests from theory (e.g., Miyake et al., 2000), this approach involves constructing new stimuli; the RBANS-EE repurposes qualitative data from existing stimuli, creating a limitation underlies some of the shortcomings of the RBANS-EE, such as its modest classification statistics. An ideal test of the value of RBANS EE within an overall battery might be to demonstrate its incremental validity in relationship to standard neuropsychological practices for measuring EF. Although our data set was not well-suited for this purpose, we recommend this approach for subsequent research and, without evidence of such incremental utility, we view the RBANS EE as best suited for EF screening.

Conclusion

The RBANS-EE has now been researched in four independent samples by three groups of researchers, though no normative data or indices of test-retest reliability are available. The rationale for an aggregate measure like the RBANS-EE would be further strengthened by demonstrating its association with neuropathology, especially conditions associated with frontal pathology. Intrusion errors on other list learning tests have been associated with Huntington’s disease (Holden et al., 2020) and frontal lesions (Baldo et al., 2002), and repetition errors are common with ADHD (Pollak et al., 2008) and prolonged methamphetamine use (Woods et al., 2005). Similar research with the RBANS-EE is needed. The RBANS-EE scale takes no additional time to administer, making the information it yields essentially “cost free.” Our findings clearly demonstrate a screening value for this scale, even if it does not eliminate the need for further testing of EF.

Footnotes

Author Contributions

This work was authored as part of the contributor’s official duties as an Employee of the United States Government and is therefore a work of the United States Government. In accordance with 17 U. S. C 105, no copyright protection is available for such works under U. S. Law.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Brian D. Gradwohl

Author Biographies

Brian D. Gradwohl earned his Ph.D. at Fuller Theological Seminary and is the Director of the Neuropsychology section at Trinity Health in West Michigan. Dr. Gradwohl's research interests include psychometrics, aging and dementia, and validity.

Andrew C. Hale earned his Ph.D. at Western Michigan University and is a neuropsychologist at the VA Ann Arbor Healthcare System. He is also a Clinical Assistant Professor at Michigan Medicine, Department of Psychiatry. Dr. Hale's clinical and research interest include traumatic brain injury, statistical methods, and measures of performance and symptom validity.

Robert J. Spencer earned his Ph.D. at the University of Maryland, Baltimore County and is the Chief of the Neuropsychology service at the VA Ann Arbor Healthcare System. He is also Clinical Assistant Professor at Michigan Medicine. Dr. Spencer’s research interests include issues of measurement.

References

American Psychiatric Association . (2013). Diagnostic and statistical manual of mental disorders (5th ed). American Psychiatric Association.

Baldo

J. V.

Delis

Kramer

Shimamura

A. P.

(2002). Memory performance on the California verbal learning test-II: Findings from patients with focal frontal lesions. Journal of the International Neuropsychological Society: JINS, 8(4), 539–546. https://doi.org/10.1017/s135561770281428x

Benton

A. L.

Hamsher

Sivan

A. B.

(1976). Multilingual aphasia examination. AJA.

Block

C. K.

Johnson-Greene

Pliskin

Boake

(2017). Discriminating cognitive screening and cognitive testing from neuropsychological assessment: Implications for professional practice. The Clinical Neuropsychologist, 31(3), 487–500. https://doi.org/10.1080/13854046.2016.1267803

Bolton

Powell

Singh

Musil

(2020). Utility of the repeatable battery for the assessment of neuropsychological status (RBANS) executive errors scale in a clinical sample of older adults. Denver, CO: Poster presented at the meeting of International Neuropsychological Society Conference.

Caeyenberghs

Leemans

Leunissen

Gooijers

Michiels

Sunaert

Swinnen

S. P.

(2014). Altered structural networks and executive deficits in traumatic brain injury patients. Brain Structure & Function, 219(1), 193–209. https://doi.org/10.1007/s00429-012-0494-2

Delis

D. C.

Kaplan

Kramer

J. H.

(2001). Delis-Kaplan executive function system (D-KEFS). The Psychological Corporation.

Duff

Humphreys Clark

J. D.

O’Bryant

S. E.

Mold

J. W.

Schiffer

R. B.

Sutker

P. B.

(2008). Utility of the RBANS in detecting cognitive impairment associated with Alzheimer’s disease: Sensitivity, specificity, and positive and negative predictive powers. Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists, 23(5), 603–612. https://doi.org/10.1016/j.acn.2008.06.004

Duff

Langbehn

D. R.

Schoenberg

M. R.

Moser

D. J.

Baade

L. E.

Mold

J. W.

Scott

J. G.

Adams

R. L.

Adams

R. L.

(2009). Normative data on and psychometric properties of verbal and visual indexes of the RBANS in older adults. The Clinical Neuropsychologist, 23(1), 39–50. https://doi.org/10.1080/13854040701861391

10.

Duff

Patton

Schoenberg

M. R.

Mold

Scott

J. G.

Adams

R. L.

(2003). Age- and education-corrected independent normative data for the RBANS in a community dwelling elderly sample. The Clinical Neuropsychologist, 17(3), 351–366. https://doi.org/10.1076/clin.17.3.351.18082

11.

Fiorentino

Gleichgerrcht

Roca

Cetkovich

Manes

Torralva

(2013). The INECO frontal screening tool differentiates behavioral variant—frontotemporal dementia (bv-FTD) from major depression. Dementia & Neuropsychologia, 7(1), 33–39. https://doi.org/10.1590/S1980-57642013DN70100006

12.

Goodglass

Kaplan

(1972). Assessment of aphasia and related disorders. Lea & Febiger.

13.

Groth-Marnat

Wright

A. J.

(2016). Handbook of psychological assessment (6th ed). John Wiley & Sons.

14.

Hankee

L. D.

Preis

S. R.

Beiser

A. S.

Devine

S. A.

Liu

Seshadri

Wolf

P. A.

(2013). Qualitative neuropsychological measures: Normative data on executive functioning tests from the framingham offspring study. Experimental Aging Research, 39(5), 515–535. https://doi.org/10.1080/0361073X.2013.839029

15.

Heaton

R. K.

Chelune

G. J.

Talley

J. L.

Kay

G. G.

Curtiss

(1993). Wisconsin card sorting test manual. Psychological Assessment Resources, Inc.

16.

Hill

B. D.

Alosco

Bauer

Tremont

(2012). The relation of executive functioning to CVLT-II learning, memory, and process indexes. Applied Neuropsychology. Adult, 19(3), 198–206. https://doi.org/10.1080/09084282.2011.643960

17.

Holden

H. M.

Tierney

S. M.

Graves

L. V.

Beltran-Najera

Woods

S. P.

Snell

C. M.

Delis

D. C.

Corey-Bloom

Gilbert

P. E.

Gilbert

P. E.

(2020). Identification of subtle verbal memory deficits in premanifest Huntington disease using the California verbal learning test. Cognitive and Behavioral Neurology: Official Journal of the Society for Behavioral and Cognitive Neurology, 33(1), 16–22. https://doi.org/10.1097/WNN.0000000000000219

18.

Hosmer

D. W.

Lemeshow

(2000). Applied logistic regression (2nd ed.) Wiley.

19.

Kiselica

A. M.

Benge

J. F.

(2019). Quantitative and qualitative features of executive dysfunction in frontotemporal and Alzheimer's dementia. Applied Neuropsychology: Adult, 28(4), 449–463. https://doi.org/10.1080/23279095.2019.1652175

20.

Kopp

Rösser

Tabeling

Stürenburg

H. J.

Haan

Karnath

Wassel

(2015). Errors on Trail Making Test are associated with right hemispheric frontal lobe damage in stroke patients. Behaivoral Neurology, 1–10. https://doi.org/10.1155/2015/309235

21.

Lezak

M. D.

Howieson

D. B.

Loring

D. W.

Tranel

(2012). Neuropsychological assessment (5th ed.) Oxford University Press.

22.

Luria

A. R.

(1980). Higher cortical functions in man (2nd ed.) Basic.

23.

Mitrushina

Boone

K. B.

Razani

D’Elia

L. F.

(2005). Handbook of normative data for neuropsychological assessment (2nd ed.) Oxford University Press.

24.

Miyake

Friedman

N. P.

Emerson

M. J.

Witzki

A. H.

Howerter

(2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41(1), 49–100. https://doi.org/10.1006/cogp.1999.0734

25.

Pollak

Kahana-Vax

Hoofien

(2008). Retrieval processes in adults with ADHD: A ravlt study. Developmental Neuropsychology, 33(1), 62–73. https://doi.org/10.1080/87565640701729789

26.

Possin

K. L.

Chester

S. K.

Laluz

Bostrom

Rosen

H. J.

Miller

B. L.

Kramer

J. H.

(2012). The frontal-anatomic specificity of design fluency repetitions and their diagnostic relevance for behavioral variant frontotemporal dementia. The Journal of International Neuropsychological Society, 18(5), 834–844. https://doi.org/10.1017/S1355617712000604

27.

Rabin

L. A.

Paolillo

Barr

W. B.

Barr

W. B.

(2016). Stability in test-usage practices of clinical neuropsychologists in the United States and Canada over a 10-year period: A follow-up survey of INS and nan members. Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists, 31(3), 206–230. https://doi.org/10.1093/arclin/acw007

28.

Rabinovici

G. D.

Stephens

M. L.

Possin

K. L.

(2015). Executive dysfunction. Behavioral Neurology and Neuropsychiatry, 21(3), 646–659. https://doi.org/10.1212/01.CON.0000466658.05156.54

29.

Randolph

(1998). Repeatable battery for the assessment of neuropsychological status manual. The Psychological Corporation.

30.

Randolph

(2012). Repeatable battery for the assessment of neuropsychological status update. The Psychological Corporation.

31.

Randolph

Tierney

M. C.

Mohr

Chase

T. N.

(1998). The repeatable battery for the assessment of neuropsychological status (RBANS): Preliminary clinical validity. Journal of Clinical and Experimental Neuropsychology, 20(3), 310–319. https://doi.org/10.1076/jcen.20.3.310.823

32.

Rasmusson

X. D.

Zonderman

A. B.

Kawas

Resnick

S. M.

(1998). Effects of age and dementia on the trail making test. The Clinical Neuropsychologist, 12(2), 169–178. https://doi.org/10.1076/clin.12.2.169.2005

33.

Reitan

R. M.

(1955). The relation of the trail making test to organic brain damage. Journal of Consulting Psychology, 19(5), 393–394. https://doi.org/10.1037/h0044509

34.

Ringdahl

E. N.

Stephens

C. N.

Aden

J. K.

Seegmiller

R. A.

(2019). Clinical application of the repeatable battery for the assessment of neuropsychological status (RBANS) executive errors scale. Chicago, IL: Poster presented at the meeting of Annual Academy of Clinical Neuropsychology Conference.

35.

Shaughnessy

M. F.

Rucker

Sanchez

A. A.

(2019). A reflective review of the RBANS. ARC Journal of Psychiatry, 4(1), 3–9.

36.

Smith

E. E.

Smith

J. A.

Juengst

S. B.

(2020). Cognitive process scores associated with self-reported behavioral dysfunction on the frontal systems behavior scale (FrSBe) in chronic traumatic brain injury. Journal of Clinical and Experimental Neuropsychology, 42(1), 90–100. https://doi.org/10.1080/13803395.2019.1676882

37.

Spencer

R. J.

Kitchen Andren

K. A.

Tolle

K. A.

(2018). Development of a scale of executive functioning for the RBANS. Applied Neuropsychology. Adult, 25(3), 231–236. https://doi.org/10.1080/23279095.2017.1284664

38.

Strauss

Sherman

E. M. S.

Spreen

(2006). A compendium of neuropsychological tests: Administration, norms, and commentary (3rd ed.) Oxford University Press.

39.

Tolle

K. A.

Rahman-Filipiak

A. M.

Hale

A. C.

Kitchen Andren

K. A.

Spencer

R. J.

(2019). Grooved Pegboard Test as a measure of executive functioning. Applied Neuropsychology: Adult, 27(5), 414–420. https://doi.org/10.1080/23279095.2018.1559165

40.

Willcutt

E. G.

Doyle

A. E.

Nigg

J. T.

Faraone

S. V.

Pennington

B. F.

(2005). Validity of the executive function theory of attention-deficit/hyperactivity disorder: A meta-analytic review. Biological Psychiatry, 57(11), 1336–1346. https://doi.org/10.1016/j.biopsych.2005.02.006

41.

Woods

S. P.

Scott

J. C.

Conover

Marcotte

T. D.

Heaton

R. K.

Grant

HIV Neurobehavioral Research Center Group (2005). Test-retest reliability of component process variables within the hopkins verbal learning test-revised. Assessment, 12(1), 96–100. https://doi.org/10.1177/1073191104270342

42.

You

S. C.

Geschwind

M. D.

Sha

S. J.

Apple

Satris

Wood

K. A.

Johnson

E. T.

Gooblar

Feuerstein

J. S.

Finkbeiner

Kang

G. A.

Miller

B. L.

Hess

C. P.

Kramer

J. H.

Possin

K. L.

Possin

K. L.

(2014). Executive functions in premanifest Huntington’s disease. Movement Disorders: Official Journal of the Movement Disorder Society, 29(3), 405–409. https://doi.org/10.1002/mds.25762