Working Memory,Processing Speed,and Memory Functioning Are Minimally Predictive of Victoria Symptom Validity Test Performance

Abstract

A sound performance validity test is accurate for detecting invalid neuropsychological test performance and relatively insensitive to actual cognitive ability or impairment. This study explored the relationship of several cognitive abilities to several performance indices on the Victoria Symptom Validity Test (VSVT), including accuracy and response latency. This cross-sectional study examined data from a mixed clinical sample of 88 adults identified as having valid neurocognitive test profiles via independent validity measures, and who completed the VSVT along with objective measures of working memory, processing speed, and verbal memory during their clinical neuropsychological evaluation. Results of linear regression analyses indicated that cognitive test performance accounted for 5% to 14% of total variance for VSVT performance across indices. Working memory was the only cognitive ability to predict significant, albeit minimal, variance on the VSVT response accuracy indices. Results show that VSVT performance is minimally predicted by working memory, processing speed, or delayed verbal memory recall.

Keywords

performance validity test effort cognition neuropsychology assessment

Since the Slick criteria for determining malingered neurocognitive dysfunction were first published in 1999 (Slick et al., 1999), there has been a surge of research on performance validity tests (PVTs) accompanied by a growing recognition for routine assessment of performance validity during neuropsychological evaluation. Although neuropsychological tests have been shown to be very sensitive to detecting brain dysfunction due to neurological, neurodevelopmental, medical, and psychiatric conditions (Braun et al., 2011), invalid performance can mask the true relationship between the severity of an individual’s neurocognitive dysfunction and observed neuropsychological test performance. PVTs serve to discriminate between individuals providing insufficient effort or engagement on cognitive tests from those with bona fide cognitive impairment based on objective, empirically validated cut-scores. Because obtaining scores below these cutoffs has been shown to be atypical even for individuals with known neurocognitive disorders when normally motivated, failures on these measures due to cognitive impairment is unlikely, particularly when multiple PVTs are failed (Larrabee, 2014b). For example, Critchfield et al. (2019) recently compared performance on a neuropsychological battery, including one embedded and three freestanding PVTs, among patients with valid performance and no cognitive impairment, patients with valid performance and cognitive impairment due to known medical/neurological conditions, and patients with invalid performance in a VA population. While patients with impairment and valid performance and those with invalid performance showed minimal performance differences on cognitive tests, the former averaged <1 PVT failure, whereas the latter averaged 2.40 PVT failures. Thus, even individuals with known cognitive impairment due to organic brain dysfunction are generally able to pass PVTs. Additionally, while a false-positive error based on a single PVT failure may occur at various rates depending on the specificity (SP) and sensitivity (SN) of the measure, the likelihood of false-positive errors decreases as multiple PVTs are used together versus independently (Davis & Millis, 2014; Larrabee, 2014a; Loring et al., 2016; Webber et al., 2018), which is why practice standards recommend the administration of multiple PVTs throughout all neuropsychological test batteries (American Academy of Clinical Neuropsychology, 2007).

The Victoria Symptom Validity Test (VSVT; Slick et al., 1997) is a well-validated PVT for detecting suboptimal engagement and malingering of cognitive impairment (Gervais et al., 2004). The VSVT is a computer-administered measure, designed as a forced-choice recognition test to assess cognitive test performance validity. Using both response accuracy and latency for easy and difficult items, the VSVT is able to improve its SN in detecting response bias by increasing perceived difficulty on test items, without changing the actual difficulty of the test. As each trial block is administered and retention interval times are increased, patients are told that the difficulty of the test may increase to enhance the perceived difficulty. However, previous research has shown that the increase of retention interval time does not result in significantly more errors, even among patients with documented memory impairment (Slick et al., 1997). The VSVT also offers suggested cutoff scores for response latency (i.e., the amount of time taken by the examinee to respond), though these have been found to be less sensitive than the different response accuracy indices (Silk-Eglit et al., 2016).

Despite the original recommendation made by the VSVT manual (Slick et al., 1997) and interpretative report, which includes classification ranges based on binomial probability theory such that a score is not flagged as invalid unless it is below statistical chance, other cut-scores with robust sensitivities and specificities have been empirically derived using a diversity of clinical and medicolegal populations. These validation studies have taken three types of designs: (a) simulated malingering designs that compare examinees instructed to feign impairment with those instructed to perform optimally; (b) differential prevalence designs that evaluate base rates of performance validity in specific clinical samples; and (c) known groups designs that evaluate the classification accuracies of the PVT for detecting valid from invalid performance. Several simulation studies found a cut-score of <16 VSVT difficult items correct resulted in high SN (.75-.88) and SP (.94-1.00; Slick et al., 1996; Strauss et al., 1999; Strauss et al., 2002; Tan et al., 2002). Similarly, a more liberal cut-score of <21 had .80 to .93 SN/.94 SP for detection of simulated attention-deficit/hyperactivity disorder and reading disorder (Frazier et al., 2008). Among a sample of acute severe traumatic brain injury (TBI) inpatients, healthy participants instructed to simulate neuropsychological impairment, and healthy controls, a cutoff of <44 total items correct yielded excellent SN (.90) and SP (.96-1.00; Macciocchi et al., 2006). They also found 36% of the variance in VSVT total items correct performance could be explained by the Benton Visual Form Discrimination and Controlled Oral Word Association, whereas Trails A and Symbol-Digit Modalities Test Oral scores contributed 45% of the shared variance in a VSVT difficult item response latency prediction model. Therefore, they concluded that people with documented severe TBI and concomitant severe visual and initiation impairments may be at risk for false-positive classification or greater difficulty responding accurately and quickly to stimuli.

Several differential prevalence design studies have also been conducted to validate VSVT cut-scores (i.e., Doss et al., 1999; Grote et al., 2000; Keary et al., 2013; Loring, Lee, & Meador, 2005; Loring, Larrabee, Lee, & Meador, 2007). Grote et al. (2000) evaluated VSVT performance in a group of non-compensation-seeking patients with intractable epilepsy and compensation-seeking patients presenting with memory complaints, primarily due to mild traumatic brain injury (mTBI). They found that 100% of the non-compensation-seeking patients obtained difficult item correct scores of ≥18, whereas only 50.9% of compensation-seeking patients performed at or above this cut-score. Easy and difficult item response latencies were also evaluated as measures of performance invalidity, with easy item response latency (≥3 seconds; SN = .28; SP = .97) being substantially less sensitive at detecting invalid performance among the compensation-seeking examinees compared with difficult item response latency (≥4 seconds; SN = .56; SP = .90). Subsequent studies examining VSVT difficult items correct cutoffs of <18 and <21 in a large sample of patients with epilepsy who were evaluated presurgically (Loring et al., 2005) and a heterogeneous, clinically referred sample of other neurological conditions (e.g., dementia, cerebrovascular event, multiple sclerosis, TBI; Loring et al., 2007) concluded that these cutoffs may result in high failure rates in clinical samples. In particular, Loring et al. (2005) noted that patients with epilepsy being evaluated for surgical candidacy who had low intellectual functioning were at particular risk for scoring below cutoffs on VSVT difficult items. Nonetheless, neither of these studies definitively screened out examinees with an external incentive and, therefore, may have inadvertently retained compensation-seeking examinees in their clinical sample (Grote, 2007).

More recently, Keary et al. (2013) examined the relationships between VSVT performance, full-scale intelligence quotient (FSIQ), and working memory in a large sample of patients with intractable epilepsy evaluated presurgically. They noted that 5% of the sample failed to achieve scores of ≥18 difficult items correct and 13% failed to achieve scores of ≥21 difficult items correct. The study also found that working memory was an important mediator of the relationship between VSVT difficult item performance and FSIQ among epilepsy surgery candidates, leading the authors to conclude that low difficult item scores may not always indicate frank malingering or insufficient effort, but may reflect significant working memory deficits in this population.

Known groups designs have only been used to investigate optimal VSVT cut-scores and classification accuracies in two previous studies (Jones, 2013; Silk-Eglit et al., 2016). In the Jones (2013) study, response accuracy indices (i.e., item correct scores) accurately discriminated “nonmalingerers” from “probable-to-definite malingerers” with relatively high sensitivities and specificities at optimal cut-scores: <23 easy items (SN = .52; SP = .95), <20 difficult items (SN = .91; SP = .93), and <44 total items (SN = .91; SP = .93). Finally, Silk-Eglit et al. (2016) explored the best approach for using the recommended VSVT cutoff scores and its classification accuracy in a medico-legal sample of examinees with mTBI and recommended more conservative cut-scores to serve as a clinical practice guide, specifically within the mTBI population. As such, they recommended the use of the following cut-scores as these had the strongest classification accuracies: <23 easy items correct (SN = .32; SP = .95), <18 difficult items correct (SN = .68; SP = .90), and <41 total items correct (SN = .68; SP = .90).

Despite the VSVT’s demonstrated ability to identify invalid performance among patients with mTBI, the role of cognitive functioning on VSVT performance in general clinical samples remains incompletely explored outside of circumscribed clinical populations. Notably, several studies have examined VSVT performance in epilepsy samples, and more recently, Keary et al. (2013) found that individuals with low FSIQ and working memory deficits may be prone to false-positive classification. Macciocchi et al. (2006) also suggested that tests involving visuoperception, processing speed, and verbal fluency could significantly predict VSVT performance across different response accuracy and latency indices among patients with severe TBI. As a caveat to both studies, results may be unintentionally biased as independent criterion PVTs were not included to ensure valid neuropsychological test performance on their predictor variables; rather both studies excluded participants in active litigation, which is at best only a proxy of performance invalidity. With these limitations in the existing literature in mind, the purpose of this study was to further investigate the role of several higher order cognitive abilities (i.e., working memory, processing speed, and verbal memory) on performance across four common VSVT indices (i.e., total items correct, difficult items correct, the difference score [easy items correct − difficult items correct], and difficult items response latency). These particular cognitive abilities were identified as variables of interest for two reasons. First, as the VSVT task structure requires processing and maintenance of verbal/numerical information in active stores to generate a response, working memory and processing speed should be associated with task performance. Given the previous findings, it was hypothesized that working memory would play a more significant role in performance across the various response accuracy indices, whereas processing speed would be a stronger predictor of difficult items response latency. Second, PVT performance can, at times, be negatively affected by genuine memory impairment (e.g., Green et al., 2011; Greve et al., 2008; Merten et al., 2007); therefore, memory function was also investigated as a possible contributor.

Method

Participants

This cross-sectional study examined data from a mixed clinical sample of 123 adults who completed the VSVT, Wechsler Adult Intelligence Scale–Fourth edition (WAIS-IV; Wechsler, 2008) working memory index (WMI) and processing speed index (PSI), and California Verbal Learning Test–Second edition (CVLT-II) during comprehensive neuropsychological evaluation. In addition to the above cognitive measures, all patients completed multiple independent freestanding and embedded PVTs administered throughout their evaluations, which were examined to ensure that only valid data were included in analyses (see Table 1 for all possible PVTs and associated cut-scores). Patients completed a mean of 3.41 criterion PVTs (SD = 1.10; range: 2-7), and those who failed ≥1 PVT were excluded from analyses, which is more conservative than the standard practice of using ≥2 PVT failures to indicate probable invalidity (Boone, 2009; Larrabee, 2014a). Of the 123 initial patients, 88 had 0 PVT failures and were retained for analysis, whereas 35 had ≥1 PVT failure (M = 1.46, SD = 0.74) and were excluded. This approach allowed for objective verification of valid performance and confidence that obtained cognitive test scores likely reflect examinees’ optimal working memory, speed, and memory abilities, which is critical as invalid performance has been repeatedly shown to result in artificially low neuropsychological test scores (Critchfield et al., 2019; Proto et al., 2014). Inclusion of invalid neuropsychological scores would introduce artifacts into the regression analyses as low scores would no longer necessarily reflect impaired performance and any genuine relationships between the cognitive abilities of interest and VSVT performance would be obscured.

Table 1.

Freestanding and Embedded Performance Validity Tests Used to Identify Invalid Performance.

PVT	Cutoff criteria	Failure rate	Reference
TOMM (n = 33)	Trial 2 <45	10/33 (30%)	Tombaugh (1996)
WMT (n = 9)	IR, DR, or CNS ≤82.5	6/9 (67%)	Green (2003)
DCT (n = 9)	E-score ≥14 Depression, PTSD, anxiety ≥15 Learning disorder ≥19 Mild cognitive disorder	2/9 (22%)	Boone et al. (2002b)
b test (n = 3)	E-score ≥90	0/3 (0%)	Boone et al. (2002a)
HVLT-R (n = 4)	Recognition discrimination ≤5	0/4 (0%)	Bailey, Soble, Bain, et al. (2018)
BVMT-R (n = 109)	Recognition discrimination ≤4	16/109 (15%)	Bailey, Soble, Bain, et al. (2018)
CVLT-II (n = 124)	Forced-choice recognition ≤14	9/124 (7%)	Schwartz et al. (2016)
RBANS (n = 14)	EI >3	2/14 (14%)	Silverberg et al. (2017)
WMS-IV logical memory (n = 17)	Recognition Total correct ≤18	2/17 (12%)	Soble et al. (2019)
WCST (n = 106)	Failure to maintain set >3	4/106 (4%)	Greve et al. (2009)

Note. PVT = performance validity test; TOMM = Test of Memory Malingering; WMT = Word Memory Test; IR = immediate recognition; DR = delayed recognition; CNS = consistency; DCT = Dot Counting Testing; PTSD = posttraumatic stress disorder; HVLT-R = Hopkins Verbal Learning Test–Revised; BVMT-R = Brief Visuospatial Memory Test–Revised; CVLT-II = California Verbal Learning Test–Second edition; RBANS = Repeatable Battery for the Assessment of Neuropsychological Status; WMS-IV = Wechsler Memory Scale–Fourth edition; WCST = Wisconsin Card Sorting Test.

The final sample was 54% male (n = 48) and 46% female (n = 40) with a mean age of 31.7 years (SD = 10.2; range: 18-58) and mean education of 15.4 years (SD = 2.3; range: 11-21). This sample was also diverse regarding ethnic/racial composition, with the following breakdown: 48% Caucasian (n = 42), 20% Hispanic (n = 18), 17% African American (n = 15), 13% Asian (n = 11), and 2% Middle Eastern (n = 2). All patients had a high level of English proficiency and identified English as their primary language. Patient diagnostic characteristics are presented in Table 2.

Table 2.

Patient Diagnostic Characteristics.

Diagnosis	n (%)
No diagnosis	2 (2)
ADHD	34 (39)
Learning disorder	4 (5)
Mild neurocognitive disorder	18 (21)
Mild cognitive impairment-amnestic	1
Stroke/cerebrovascular disease	2
Multiple sclerosis	2
Substance-induced	1
Traumatic brain injury	1
Multiple comorbid etiologies	8
Electrical injury	3
Depressive disorder	8 (9)
Anxiety disorder	10 (11)
PTSD	2 (2)
Other psychiatric disorder	3 (3)
Chronic pain	3 (4)
Sleep disorder	2 (2)
History of subdural hematoma	1 (1)
History of anti-NMDA receptor encephalitis	1 (1)

Note. ADHD = attention-deficit/hyperactivity disorder; PTSD = posttraumatic stress disorder; NMDA = N-methyl D-aspartate.

Measures

Victoria Symptom Validity Test (Slick et al., 1997)

The VSVT is a PVT that utilizes a forced-choice paradigm. The test consists of 48 items presented across three blocks of 16 trials. In each trial, a single five-digit number sequence is presented for 5 seconds. Following the presentation of the number sequence, there is a delay interval in which no stimuli is presented and the length of this delay interval differs across each block of 16 trials (5-, 10-, and 15-second delays). Once the delay interval is complete, the original five-digit number sequence is presented alongside a foil five-digit number sequence. The examinee is asked to indicate which sequence was previously displayed. On 24 of the 48 trials, the foil does not share any common digits with the stimuli; therefore, they are considered “easy items.” In the other 24 trials, the foil is similar to the stimuli, but has two of the middle digits transposed; these items are considered “difficult items.” The test also records response latencies representing the amount of time it takes the examinee to choose one of the two number sequences. The variables of interest in the current study have been previously identified as different indices for detecting performance invalidity and are reported as follows with their respective sensitivities and specificities at optimal cut-scores: total items correct (SN = .68-.91; SP = .90), difficult items correct (SN = .68-.91; SP = .90-.93), the difference score (i.e., easy items correct − difficult items correct; SN = .64-.80; SP = .90-.96), and difficult items response latency (SN = .40-.57; SP = .93-.95; Jones, 2013; Silk-Eglit et al., 2016).

Wechsler Adult Intelligence Scale–Fourth Edition Working Memory Index (Wechsler, 2008)

The WMI is composed of the Digit Span subtest, which requires examinees to repeat back strings of digits of increasing length in forward, backward, and sequencing order, and the Arithmetic subtest, which requires examinees to mentally solve a series of arithmetic problems. The variable of interest for this study was the WMI age-corrected standard score. The WMI score was selected over its component subtests (i.e., Digit Span/Arithmetic), given it has been demonstrated index scores are more psychometrically stable and clinically useful than individual subtest scores on IQ measures (Livingston et al., 2003).

Wechsler Adult Intelligence Scale–Fourth Edition Processing Speed Index (Wechsler, 2008)

The PSI is composed of the Coding subtest, which requires examinees to copy symbols that are paired with numbers as quickly as possible, and the Symbol Search subtest, which requires examinees to quickly scan a group of symbols and determine whether one of the symbols in the group matches a target symbol. Both tasks have a 120-second time limit. The variable of interest for this study was the PSI age-corrected standard score. The PSI score was selected over its component subtests (i.e., Coding/Symbol Search) based on the same rationale described above for the WMI.

California Verbal Learning Test–Second Edition (Delis et al., 2000)

The CVLT-II is a test of verbal learning and memory in which examinees are presented with a list of 16 words over five learning trials, followed by short and long delay free and cued recall trials, as well as a recognition trial. The variable of interest for this study was the CVLT Long Delay Free Recall (LDFR) total correct age/gender-corrected z-score, given that this score has been identified as the best indicator of actual delayed memory functioning on this test (Donders, 2008).

Data Analyses

Descriptive statistics were calculated for the WMI, PSI, CVLT-II LDFR, and VSVT performance indices (i.e., total items correct, difficult items correct, difference score, and difficult items response latency). As one might expect with PVT performance among valid-performing participants, statistical testing of normality revealed that the four VSVT performance indices were significantly skewed and kurtotic (total items correct: D(87) = 0.25, p < .001, M skewness = −1.88, M kurtosis = 2.91; difficult items correct: D(87) = 0.24, p < .001, M skewness = −1.80, M kurtosis = 2.66; difference score: D(87) = 0.24, p < .001, M skewness = 1.97, M kurtosis = 3.68; difficult items response latency: D(87) = 0.21, p < .001, M skewness = 4.51, M kurtosis = 28.49). As such, the data for each of these variables were transformed in order to run the planned statistical analyses (e.g., correlations, linear regression analyses), which assume normality. Given that total items correct and difficult items correct were negatively skewed, the variables were reflected before logarithmic transformations were applied. For the difference score and difficult items response latency, which were positively skewed, logarithmic transformations were applied. One participant had a negative value for VSVT difference score and was therefore excluded from subsequent analyses that included this variable given that the transformation could not be applied.

Pearson’s correlational analyses then examined the relationship between the VSVT variables, WMI, PSI, and CVLT-II LDFR. Next, four linear regression analyses were conducted with WMI, PSI, and CVLT-II LDFR scores entered as the predictor variables, and with the VSVT performance indices (i.e., total items correct, difficult items correct, difference score, and difficult items response latency) entered as the dependent variable in each respective model to determine the effects of working memory, processing speed, and verbal memory on each VSVT variable. Tolerance and variance inflation factor (VIF) diagnostics assessed for problematic multicollinearity between the predictor variables (i.e., WMI, PSI, and CVLT-II LDFR) in the regression models.

Results

Means, standard deviations, and correlation coefficients for the primary clinical variables and the VSVT indices are presented in Table 3. As expected, there were robust correlations between VSVT variables, including total items correct, difficult items correct, and easy items correct − difficult items correct, although these VSVT response accuracy variables had only modest correlations with VSVT difficult items response latency. All VSVT validity indicators (i.e., total items correct, difficult items correct, and easy items correct − difficult items correct) had small to medium significant correlations with WMI and 3 of 4 (i.e., total items correct, difficult items correct, and difference score; Table 3) with PSI. None of the VSVT variables were significantly correlated with CVLT-II LDFR.

Table 3.

Pearson Correlations Between Specific Cognitive Abilities and the VSVT Performance Indices.

	M (SD)	PSI	CVLT-II LDFR	VSVT difficult items correct	VSVT total items correct	VSVT difference score	VSVT difficult items response latency
WMI	99.91 (13.7)	.50**	.11	.32**	.32**	−.34**	−.21*
PSI	100.50 (14.1)	—	.13	.29**	.27**	−.29**	−.12
CVLT-II LDFR	−0.11 (1.5)		—	.02	−.05	−.01	−.03
VSVT difficult items correct	21.25 (3.9)			—	.99**	.98**	.39**
VSVT total items correct	44.90 (4.5)				—	.96**	.40**
VSVT difference score	2.36 (3.39)					—	.37**
VSVT difficult items response latency	2.98 (1.84)						—

Note. VSVT = Victoria Symptom Validity Test; WMI = working memory index; PSI = processing speed index; CVLT-II LDFR = California Verbal Learning Test–Second edition (CVLT-II) Long Delay Free Recall. WMI and PSI are presented as standard scores (M = 100; SD = 15), CVLT-II LDFR is presented as a z-score (M = 0; SD = 1), and VSVT scores are presented as raw scores. n = 88, except for VSVT difference score (n = 87).

p < .05. **p < .01.

Collinearity diagnostics were within acceptable limits and did not reveal evidence of problematic multicollinearity between the predictor variables: WMI, PSI, and CVLT-II LDFR (Tolerance = .74-.98; VIF = 1.02-1.35). As seen in Table 4, results of the linear regression analyses revealed that the both overall models for the Items correct indices (i.e., total items correct, difficult items correct) significantly predicted VSVT difficult items correct (p = .01), with WMI, PSI, and CVLT-II LDFR explaining 13% of the total variance in the model. WMI was the only significant predictor in both models (p ≤ .05). Results for the overall regression model examining VSVT difference score was also significant (p = .01) with a similar amount of variance (i.e., 14%) explained by the cognitive abilities, with WMI again emerging as the only significant predictor in the model (p = .04). In contrast, the overall regression model examining WMI, PSI, and CVLT-II LDFR as predictors of VSVT difficult item response latency was nonsignificant (p = .27).

Table 4.

Regression Models.

Model	R ²	Model F	p	Β	SE Β	β	t	p
VSVT difficult items correct
Overall model	.13	4.08	.01
WMI				.02	.01	.24	2.00	.05
PSI				.01	.01	.18	1.50	.14
CVLT-II LDFR				−.04	.06	−.07	−0.70	.49
VSVT total items correct
Overall model	.13	3.99	.01
WMI				.02	.01	.25	2.09	.04
PSI				.01	.01	.16	1.34	.19
CVLT-II LDFR				−.06	.07	−.09	−0.90	.37
VSVT difference score
Overall model	.14	4.30	.01
WMI				−.02	.01	−.25	−2.14	.04
PSI				−.01	.01	−.17	−1.42	.16
CVLT-II LDFR				.02	.06	.04	0.37	.72
VSVT difficult items response latency
Overall model	.05	1.34	.27
WMI				−.01	.004	−.20	−1.65	.10
PSI				−.001	.004	−.02	−0.15	.88
CVLT-II LDFR				<−.001	.03	−.001	−0.001	.99

Note. SE = standard error; VSVT = Victoria Symptom Validity Test; WMI = working memory index; PSI = processing speed index; CVLT-II LDFR = California Verbal Learning Test–Second edition (CVLT-II) Long Delay Free Recall. n = 88 for all models, except for VSVT difference score (n = 87).

Discussion

The VSVT previously has demonstrated replicable accuracy for detecting noncredible cognitive test performance as a freestanding PVT in neuropsychological evaluations, though limited research has been conducted examining the role of actual cognitive functioning on VSVT performance. Consideration of this relationship is important, as the ideal PVT will have robust SN/SP for detecting poor engagement but will be minimally affected by actual cognitive impairment. Current results showed that these cognitive abilities explained relatively little variance in VSVT response accuracy performance (i.e., total items correct, difficult items correct, and difference score), with overall models accounting for approximately 13% to 14% of the total variance. Further examination of the three significant regression models revealed that working memory explained a significant amount of variance of the VSVT response accuracy indices, whereas processing speed and verbal memory were largely noncontributory.

A clinically useful and psychometrically robust PVT is one that is accurate for detecting invalid performance while remaining relatively insensitive to actual cognitive ability(ies) and neurocognitive impairment. Although it has been shown that cognitive impairment does not result in invalid overall performance (i.e., failure on multiple PVTs administered throughout a neuropsychological evaluation; Critchfield et al., 2019), performance on some individual PVTs have been found, to varying degrees, to be adversely affected by bona fide cognitive impairment or more strongly correlated with actual cognitive abilities, such as memory and processing speed (e.g., Alverson et al., 2019; Bailey, Soble, & O’Rourke, 2018; Bain et al., 2019; Bain & Soble, 2021; Soble et al., 2018; Webber & Soble, 2018), which can limit the utility of these PVTs or necessitate use of different cut-scores in certain clinical populations. While the VSVT’s accuracy for detecting invalid performance has been established and replicated (e.g., Grote et al., 2000; Silk-Eglit et al., 2016), current results add to this literature by providing further evidence that actual cognitive abilities have nonsignificant to small effects on VSVT performance across two commonly used indices. Thus, this study replicated prior results that similarly reported working memory exerted an effect on VSVT performance in a circumscribed epilepsy sample (Keary et al., 2013), and extended these findings by examining additional cognitive abilities of interest (i.e., processing speed, delayed verbal memory) in a more diverse, mixed clinical neuropsychiatric sample. This is particularly relevant given that these cognitive abilities have been found to be especially vulnerable to neurological dysfunction (Hillary et al., 2010), and are affected in many conditions, including neurodegenerative disorders (e.g., Alzheimer’s disease, Parkinson’s disease; Bondi et al., 2009; Gunzler et al., 2011), moderate–severe TBI (Dikmen et al., 2009), neurological conditions (e.g., left temporal lobe epilepsy, multiple sclerosis; Chiaravalloti & DeLuca, 2008; Loring et al., 2008), and neurodevelopmental disorders (e.g., attention-deficit/hyperactivity disorder; Willcutt, 2010). Specific VSVT characteristics, including ample stimulus presentation time (i.e., 5 seconds) and the fact that a response is solicited immediately after stimulus presentation, thus only requiring examinees to briefly hold information in active attention stores, may explain why cognitive tests of processing speed and actual memory function explained minimal variance in task performance, whereas working memory explained slightly more variance. A majority of the remaining variance is likely reflective of the basic attentional engagement or effort (i.e., performance validity) needed to perform the task.

While this study had several major strengths, including the use of a diverse, mixed clinical sample and multiple independent (i.e., non-VSVT) PVTs to ensure valid neuropsychological test performance, as well as broad investigation of diverse cognitive abilities (i.e., working memory, processing speed, and verbal memory), there were several limitations. First, the relatively modest sample size prevented further subanalyses for specific clinical diagnoses. Follow-up studies that utilize a prospective design with larger sample sizes and different diagnostic compositions may allow for replication of current results and further examination of the relationships of these cognitive abilities to VSVT performance within different clinical populations. Additionally, among those with cognitive impairment, the overwhelming majority were cases of mild cognitive impairment; therefore, findings may not generalize to more severe-/dementia-level cognitive impairment. Given that our sample was relatively young (M = 31.7 years) and well educated (M = 15.4 years), future replication studies are warranted to ensure more broad generalizability in the general population. Moreover, there has been a paucity of research on VSVT performance in minority populations and, as such, we acknowledge that the existing cut-scores, which have been validated in predominately Caucasian samples in the United States, may not be as psychometrically robust in these populations.

In summary, the current investigation demonstrated that working memory, processing speed, and verbal memory minimally predict VSVT performance in a heterogeneous clinical sample, though working memory was a significant predictor of the response accuracy indices. Thus, actual cognitive abilities appear to have a limited role in VSVT performance and poor performance is more likely representative of suboptimal test engagement. That being said, reduced performance on VSVT response accuracy indices may be associated with working memory deficits and, in such cases, clinicians should use their clinical judgment and look for convergent findings across other PVTs and incorporate other sources of information, such as clinical history and behavioral observations, and structured criteria (i.e., Slick criteria).

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Zachary J. Resch

Jason R. Soble

References

Alverson

W. A.

O’Rourke

J. J. F.

Soble

J. R.

(2019). The Word Memory Test genuine memory impairment profile discriminates genuine memory impairment from invalid performance in a mixed clinical sample with cognitive impairment. The Clinical Neuropsychologist, 33(8), 1420-1435. https://doi.org/10.1080/13854046.2019.1599071

American Academy of Clinical Neuropsychology. (2007). American Academy of Clinical Neuropsychology (AACN) practice guidelines for neuropsychological assessment and consultation. The Clinical Neuropsychologist, 21(2), 209-231. https://doi.org/10.1080/13825580601025932

Bailey

K. C.

Soble

J. R.

Bain

K. M.

Fullen

(2018). Embedded performance validity tests in the Hopkins Verbal Learning Test–Revised and the Brief Visuospatial Memory Test–Revised: A replication study. Archives of Clinical Neuropsychology, 33(7), 895-900. https://doi.org/10.1093/arclin/acx111

Bailey

K. C.

Soble

J. R.

O’Rourke

J. J. F.

(2018). Clinical utility of the Rey 15-Item Test, recognition trial, and error scores for detecting noncredible neuropsychological performance validity in a mixed-clinical sample of veterans. The Clinical Neuropsychologist, 32(1), 119-131.

Bain

K. M.

Soble

J. R.

(2019). Validation of the Advanced Clinical Solutions Word Choice Test (WCT) in a mixed clinical sample: Establishing classification accuracy, sensitivity/specificity, and cutoff scores. Assessment, 26(7), 1320-1328. https://doi.org/10.1177/1073191117725172

Bain

K. M.

Soble

J. R.

Webber

T. A.

Messerly

J. M.

Bailey

K. C.

Kirton

J. W.

McCoy

K. J. M.

(2021). Cross-validation of three Advanced Clinical Solutions performance validity tests: Examining combinations of measures to maximize classification of invalid performance. Applied Neuropsychology: Adult, 28(1), 24-34. https://doi.org/10.1080/23279095.2019.1585352

Bondi

M. W.

Salmon

D. P.

Kaszniak

A. W.

(2009). The neuropsychology of dementia. In Grant

Adams

K. M.

(Eds.), Neuropsychological assessment of neuropsychiatric and neuromedical disorders (3rd ed., pp. 159-198). Oxford University Press.

Boone

K. B.

(2009). The need for continuous and comprehensive sampling of effort/response bias during neuropsychological examination. The Clinical Neuropsychologist, 23(4), 729-741. https://doi.org/10.1080/13854040802427803

Boone

K. B.

Herzberg

D. S.

(2002a). The b Test™ manual. Western Psychological Services.

10.

Boone

K. B.

Herzberg

D. S.

(2002b). The Dot Counting Test manual. Western Psychological Services.

11.

Braun

Tupper

Kaufman

McCrea

Postal

Westerveld

Wills

Deer

(2011). Neuropsychological assessment: A valuable tool in the diagnosis and management of neurological, neurodevelopmental, medical, and psychiatric disorders. Cognitive and Behavioral Neurology, 24(3), 107-114. https://doi.org/10.1097/WNN.0b013e3182351289

12.

Chiaravalloti

N. D.

DeLuca

(2008). Cognitive impairment in multiple sclerosis. Lancet: Neurology, 7(12), 1139-1151. https://doi.org/10.1016/S1474-4422(08)70259-X

13.

Critchfield

E. A.

Soble

J. R.

Marceaux

J. C.

Bain

K. M.

Bailey

K. C.

Webber

T. A.

Alverson

W. A.

Messerly

González

D. A.

O’Rourke

J. J. F.

(2019). Cognitive impairment does not cause performance validity failure: Analyzing performance patterns among unimpaired, impaired, and noncredible participants across six tests. The Clinical Neuropsychologist, 33(6), 1083-1101. https://doi.org/10.1080/13854046.2018.1508615

14.

Davis

J. J.

Millis

S. R.

(2014). Examination of performance validity test failure in relation to number of tests administered. The Clinical Neuropsychologist, 28(2), 199-214. https://doi.org/10.1080/13854046.2014.884633

15.

Delis

D. C.

Kramer

J. H.

Kaplan

Ober

(2000). The California Verbal Learning Test–II (2nd ed.). The Psychological Corporation.

16.

Dikmen

S. S.

Corrigan

J. D.

Levin

H. S.

MacHamer

Stiers

Weisskopf

M. G.

(2009). Cognitive outcome following traumatic brain injury. Journal of Head Trauma Rehabilitation, 24(6), 430-438. https://doi.org/10.1097/HTR.0b013e3181c133e9

17.

Donders

(2008). A confirmatory factor analysis of the California Verbal Learning Test–Second Edition (CVLT-II) in the standardization sample. Assessment, 15(2), 123-131. https://doi.org/10.1177/1073191107310926

18.

Doss

R. C.

Chelune

G. J.

Naugle

R. I.

(1999). Victoria Symptom Validity Test: Compensation-seeking vs. non-compensation-seeking patients in a general clinical setting. Journal of Forensic Neuropsychology, 1(4), 5-20. https://doi.org/10.1300/J151v01n04_02

19.

Frazier

T. W.

Frazier

A. R.

Busch

R. M.

Kerwood

M. A.

Demaree

H. A.

(2008). Detection of simulated ADHD and reading disorder using symptom validity measures. Archives of Clinical Neuropsychology, 23(5), 501-509. https://doi.org/10.1016/j.acn.2008.04.001

20.

Gervais

R. O.

Rohling

M. L.

Green

Ford

(2004). A comparison of WMT, CARB, and TOMM failure rates in non-head injury disability claimants. Archives of Clinical Neuropsychology, 19(4), 475-487. https://doi.org/10.1016/j.acn.2003.05.001

21.

Green

(2003). Green‘s Word Memory Test for Windows: User‘s manual. Green’s Publishing.

22.

Green

Montijo

Brockhaus

(2011). High specificity of the Word Memory Test and Medical Symptom Validity Test in groups with severe verbal memory impairment. Applied Neuropsychology, 18(2), 86-94. https://doi.org/10.1080/09084282.2010.523389

23.

Greve

K. W.

Heinly

M. T.

Bianchini

K. J.

Love

J. M.

(2009). Malingering detection with the Wisconsin Card Sorting Test in mild traumatic brain injury. The Clinical Neuropsychologist, 23(2), 343-362. https://doi.org/10.1080/13854040802054169

24.

Greve

K. W.

Ord

Curtis

K. L.

Bianchini

K. J.

Brennan

(2008). Detecting malingering in traumatic brain injury and chronic pain: A comparison of three forced-choice symptom validity tests. The Clinical Neuropsychologist, 22(5), 896-918. https://doi.org/10.1080/13854040701565208

25.

Grote

C. L.

(2007). Forced-choice recognition tests of malingering. In Larabee

G. J.

(Ed.), Assessment of malingered neurocognitive deficits (pp. 44–79). New York, NY: Oxford University Press.

26.

Grote

C. L.

Kooker

E. K.

Garron

D. C.

Nyenhuis

D. L.

Smith

C. A.

Mattingly

M. L.

(2000). Performance of compensation seeking and non-compensation seeking samples on the Victoria Symptom Validity Test: Cross-validation and extension of a standardization study. Journal of Clinical and Experimental Neuropsychology, 22(6), 709-719. https://doi.org/10.1076/jcen.22.6.709.958

27.

Gunzler

S. A.

Schoenberg

M. R.

Riley

D. R.

Walter

Maciunas

R. J.

(2011). Parkinson’s disease and other movement disorders. In Schoenberg

M. R.

Scott

J. G.

(Eds.), The little black book of neuropsychology: A syndrome-based approach (pp. 567-646). Springer.

28.

Hillary

F. G.

Genova

H. M.

Medaglia

J. D.

Fitzpatrick

N. M.

Chiou

K. S.

Wardecker

B. M.

Franklin

R. G.

Jr. Wang

DeLuca

(2010). The nature of processing speed deficits in traumatic brain injury: Is less brain more? Brain Imaging and Behavior, 4(2), 141-154. https://doi.org/10.1007/s11682-010-9094-z

29.

Jones

(2013). Victoria Symptom Validity Test: Cutoff score for psychometrically defined malingering groups in a military sample. The Clinical Neuropsychologist, 27(8), 1373-1394. https://doi.org/10.1080/13854046.2013.851740

30.

Keary

T. A.

Frazier

T. W.

Belzile

C. J.

Chapin

J. S.

Naugle

R. I.

Najm

I. M.

Busch

R. M.

(2013). Working memory and intelligence are associated with Victoria Symptom Validity Test hard item performance in patients with intractable epilepsy. Journal of the International Neuropsychological Society, 19(3), 314-323. https://doi.org/10.1017/S1355617712001397

31.

Larrabee

G. J.

(2014a). False-positive rates associated with the use of multiple performance and symptom validity tests. Archives of Clinical Neuropsychology, 29(4), 364-373. https://doi.org/10.1093/arclin/acu019

32.

Larrabee

G. J.

(2014b). Test validity and performance validity: Considerations in providing a framework for development of an ability-focused neuropsychological test battery. Archives of Clinical Neuropsychology, 29(7), 695-714. https://doi.org/10.1093/arclin/acu049

33.

Livingston

R. B.

Jennings

Reynolds

C. R.

Gray

R. M.

(2003). Multivariate analyses of the profile stability of intelligence tests: High for IQs, low to very low for subtest analyses. Archives of Clinical Neuropsychology, 18(5), 487-507. https://doi.org/10.1016/S0887-6177(02)00147-6

34.

Loring

D. W.

Goldstein

F. C.

Chen

Drane

D. L.

Lah

J. J.

Zhao

Larrabee

G. J.

(2016). False-positive error rates for Reliable Digit Span and Auditory Verbal Learning Test performance validity measures in amnestic mild cognitive impairment and early Alzheimer disease. Archives of Clinical Neuropsychology, 31(4), 313-331. https://doi.org/10.1093/arclin/acw014

35.

Loring

D. W.

Larrabee

G. J.

Lee

G. P.

Meador

K. J.

(2007). Victoria Symptom Validity Test performance in a heterogeneous clinical sample. The Clinical Neuropsychologist, 21(3), 522-531. https://doi.org/10.1080/13854040600611384

36.

Loring

D. W.

Lee

G. P.

Meador

K. J.

(2005). Victoria Symptom Validity Test performance in non-litigating epilepsy surgery candidates. Journal of Clinical and Experimental Neuropsychology, 27(5), 610-617. https://doi.org/10.1080/13803390490918471

37.

Loring

D. W.

Strauss

Hermann

B. P.

Barr

W. B.

Perrine

Trenerry

M. R.

Chelune

Westerveld

Lee

G. P.

Meador

K. J.

Bowden

S. C.

(2008). Differential neuropsychological test sensitivity to left temporal lobe epilepsy. Journal of the International Neuropsychological Society, 14(3), 394-400. https://doi.org/10.1017/S1355617708080582

38.

Macciocchi

S. N.

Seel

R. T.

Alderson

Godsall

(2006). Victoria Symptom Validity Test performance in acute severe traumatic brain injury: Implications for test interpretation. Archives of Clinical Neuropsychology, 21(5), 395-404. https://doi.org/10.1016/j.acn.2006.06.003

39.

Merten

Bossink

Schmand

(2007). On the limits of effort testing: Symptom validity tests and severity of neurocognitive symptoms in nonlitigant patients. Journal of Clinical and Experimental Neuropsychology, 29(3), 308-318. https://doi.org/10.1080/13803390600693607

40.

Proto

D. A.

Pastorek

N. J.

Miller

B. I.

Romesser

J. M.

Sim

A. H.

Linck

J. F.

(2014). The dangers of failing one or more performance validity tests in individuals claiming mild traumatic brain injury-related postconcussive symptoms. Archives of Clinical Neuropsychology, 29(7), 614-624. https://doi.org/10.1093/arclin/acu044

41.

Schwartz

E. S.

Erdodi

Rodriguez

Ghosh

J. J.

Curtain

J. R.

Flashman

L. A.

Roth

R. M.

(2016). CVLT-II Forced Choice Recognition trial as an embedded validity indicator: A systematic review of the evidence. Journal of the International Neuropsychological Society, 22(8), 851-858. https://doi.org/10.1017/S1355617716000746

42.

Silk-Eglit

G. M.

Lynch

J. K.

McCaffrey

R. J.

(2016). Validation of Victoria Symptom Validity Test cutoff scores among mild traumatic brain injury litigants using a known-groups design. Archives of Clinical Neuropsychology, 31(3), 231-245. https://doi.org/10.1093/arclin/acv108

43.

Silverberg

N. D.

Wertheimer

J. C.

Fichtenberg

N. L.

(2007). An Effort Index for the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS). The Clinical Neuropsychologist, 21(5), 841-854. https://doi.org/10.1080/13854040600850958

44.

Slick

D. J.

Hopp

Strauss

Spellacy

F. J.

(1996). Victoria Symptom Validity Test: Efficiency for detecting feigned memory impairment and relationship to neuropsychological tests and MMPI-2 validity scales. Journal of Clinical and Experimental Neuropsychology, 18(6), 911-922. https://doi.org/10.1080/01688639608408313

45.

Slick

D. J.

Hopp

Strauss

Thompson

G. B.

(1997). Victoria Symptom Validity Test: Professional manual. Psycho-logical Assessment Resources.

46.

Slick

D. J.

Sherman

E. M. S.

Iverson

G. L.

(1999). Diagnostic criteria for malingered neurocognitive dysfunction: Proposed standards for clinical practice and research. The Clinical Neuropsychologist, 13(4), 545-561. https://doi.org/10.1076/1385-4046(199911)13:04;1-Y;FT545

47.

Soble

J. R.

Bain

K. M.

Bailey

K. C.

Kirton

J. W.

Marceaux

J. M.

Critchfield

E. A.

McCoy

K. J. M.

O’Rourke

J. J. F.

(2019). Evaluating the accuracy of the Wechsler Memory Scale-Fourth Edition (WMS-IV) logical memory embedded validity index for detecting invalid test performance. Applied Neuropsychology: Adult, 26(4), 311-318. https://doi.org/10.1080/23279095.2017.1418744

48.

Soble

J. R.

Santos

O. A.

Bain

K. M.

Kirton

J. W.

Bailey

K. C.

Critchfield

E. A.

O’Rourke

J. J. F.

Highsmith

J. M.

Gonzalez

D. A.

(2018). The Dot Counting Test adds up: Validation and response pattern analysis in a mixed clinical veteran sample. Journal of Clinical and Experimental Neuropsychology, 40(4), 317-325. https://doi.org/10.1080/13803395.2017.1342773

49.

Strauss

Hultsch

D. F.

Hunter

Slick

D. J.

Patry

Levy-Bencheton

(1999). Using intraindividual variability to detect malingering in cognitive performance. The Clinical Neuropsychologist, 13(4), 420-432. https://doi.org/10.1076/1385-4046(199911)13:04;1-Y;FT420

50.

Strauss

Slick

D. J.

Levy-Bencheton

Hunter

MacDonald

S. W. S.

Hultsch

D. F.

(2002). Intraindividual variability as an indicator of malingering in head injury. Archives of Clinical Neuropsychology, 17(5), 423-444. https://doi.org/10.1093/arclin/17.5.423

51.

Tan

J. E.

Slick

D. J.

Strauss

Hultsch

D. F.

(2002). How’d they do it? Malingering strategies on symptom validity tests. The Clinical Neuropsychologist, 16(4), 495-505. https://doi.org/10.1076/clin.16.4.495.13909

52.

Tombaugh

T. N.

(1996). Test of Memory Malingering (TOMM). Multi Health Systems.

53.

Webber

T. A.

Critchfield

E. A.

Soble

J. R.

(2020). Convergent, discriminant, and concurrent validity of nonmemory-based Performance Validity Tests. Assessment, 27(7), 1399-1415. https://doi.org/10.1177/1073191118804874

54.

Webber

T. A.

Soble

J. R.

(2018). Utility of various WAIS-IV Digit Span indices for identifying noncredible performance validity among cognitively impaired and unimpaired examinees. The Clinical Neuropsychologist, 32(4), 657-670. https://doi.org/10.1080/13854046.2017.1415374

55.

Willcutt

E. G.

(2010). Attention-deficit/hyperactivity disorder. In Yeates

K. O.

Ris

M. D.

Taylor

H. G.

Pennington

B. F.

(Eds.), Pediatric neuropsychology: Research, theory, and practice (2nd ed., pp. 393-417). Guildford Press.