Standalone Performance Validity Tests May Be Differentially Related to Measures of Working Memory,Processing Speed,and Verbal Memory in Patients With Multiple Sclerosis

Abstract

Cognitive functioning may account for minimal levels (i.e., 5%–14%) of variance of performance validity test (PVT) scores in clinical examinees. The present study extended this research twofold: (a) by determining the variance cognitive functioning explains within three distinct PVTs (b) in a sample of patients with multiple sclerosis (pwMS). Seventy-five pwMS (M_age = 48.50, 70.6% female, 80.9% White) completed the Victoria Symptom Validity Test (VSVT), Word Choice Test (WCT), Dot Counting Test (DCT), and three objective measures of working memory, processing speed, and verbal memory as part of clinical neuropsychological assessment. Regression analyses in credible groups (ns ranged from 54 to 63) indicated that cognitive functioning explained 24% to 38% of the variance in logarithmically transformed PVT variables. Variance from cognitive testing differed across PVTs: verbal memory significantly influenced both VSVT and WCT scores; working memory influenced VSVT and DCT scores; and processing speed influenced DCT scores. The WCT appeared least related to cognitive functioning of the included PVTs. Alternative plausible explanations, including the apparent domain/modality specificity hypothesis of PVTs versus the potential sensitivity of these PVTs to neurocognitive dysfunction in pwMS were discussed. Continued psychometric investigations into factors affecting performance validity, especially in multiple sclerosis, are warranted.

Keywords

performance validity neuropsychological assessment domain/modality specificity multiple sclerosis

Neuropsychologists are tasked with determining the validity of cognitive data obtained from examinees (Larrabee, 2014; Sweet et al., 2021). Performance validity tests (PVTs), which may be standalone or embedded within existing measures (Wodushek & Domen, 2020), function to identify suboptimal test engagement or otherwise noncredible performance, as these factors meaningfully jeopardize the accuracy of interpretations of test findings (An et al., 2017). As such, PVTs are ideally designed to be as insensitive as possible to actual brain-related impairment (Green & Flaro, 2021), and factor analytic work has suggested that PVTs and bona fide measures of cognition tend to be represented by distinct latent dimensions (Ord et al., 2021; Van Dyke et al., 2013). Recommendations for continuous sampling of validity throughout neuropsychological assessment are provided by experts (e.g., Boone, 2009; Erdodi, 2021).

Notably, some (but not all; e.g., Critchfield et al., 2019) prior research has indicated that the presence of notable cognitive dysfunction may result in higher than expected rates of abnormal scores on PVTs, and ultimately increase the risk of false positives (i.e., incorrectly concluding noncredible performance; Laurent et al., 2019; Miller & Axelrod, 2018). In particular, this is commonly reported among examinees with very low IQ (Dean et al., 2008) and those with dementia syndromes (Davis, 2018; Dean et al., 2009). The degree to which the use of traditional PVT cutoff scores in those with milder degrees of cognitive impairment is mixed, however (Loring et al., 2016; Maiman et al., 2019; Quattlebaum et al., 2019; Teichner & Wagner, 2004). As such, understanding the degree to which tests of cognitive abilities psychometrically overlap with PVT scores is essential in neuropsychological research and practice.

Recently, Resch, Soble, et al., (2021) took a unique approach and determined the extent to which cognitive functioning explained variance within a standalone PVT in credibly performing examinees. Specifically, they explored overlap between the Victoria Symptom Validity Test (VSVT; Slick et al., 1997), a well-validated, memory-based PVT (see Resch, Webber, et al., 2021), and objective measures of auditory working memory (Wechsler Adult Intelligence Scale, Fourth Edition [WAIS-IV] Working Memory Index [WMI]; Wechsler, 2008), processing speed (WAIS-IV Processing Speed Index [PSI]; Wechsler, 2008), and verbal memory (California Verbal Learning Test, Second Edition Long Delay Free Recall [CVLT-II]; Delis et al., 2000). Their mixed clinical sample included 88 adults (M age = 31.7), most of whom had primary diagnoses of attention-deficit/hyperactivity disorder (ADHD; 39%), a psychiatric condition (28%), or a mild neurocognitive disorder (mND; 21%). Participants completed an average of 3.41 PVTs, and only those who passed all administered PVTs were operationally defined as valid and included to conservatively ensure that obtained test scores likely reflected optimal performance without excessive statistical artifact.

Resch, Soble, et al.’s (2021) results showed significant, yet modest, correlations between working memory/processing speed and logarithmically transformed VSVT variables. Verbal memory did not correlate with VSVT variables. Regression models showed that combined cognitive performance explained between 5% and 14% of the variance across several VSVT variables; working memory, processing speed, and verbal memory accounted for only 13% of the variance in VSVT Total correct. Notably, WMI was the only cognitive variable that tended to account for unique variance in VSVT performance, though at a marginally significant level (ps ranged from .04 to .10). In sum, Resch, Soble, et al.’s (2022) findings suggested that performance on the VSVT appeared to be minimally impacted by working memory, processing speed, and verbal memory.

Although meaningful in its contribution, Resch, Soble, et al.’s (2021) work poses three limitations. First, Resch, Soble, et al.’s (2021) operationalization of validity may have unnecessarily yielded data that were not adequately representative of a mixed clinical sample. As above, participants completed an average of 3.41 PVTs (apart from the VSVT) and conservatively were excluded if they performed outside expected limits on even one; this definition of validity may have been overly cautious (e.g., Schroeder et al., 2019), as recent work has indicated that examinees failing 0-1 PVT may be justifiably classified as valid, and those failing ≥ 2 classified as invalid (Rhoads, Neale, et al., 2021), though there is great debate in the literature regarding number of PVT failures needed to define invalidity (Odland et al., 2015; Proto et al., 2014). Of note, one-sample t tests by the present authors using Resch, Soble, et al.’s (2021) descriptive data indicated that neither WMI (M = 99.91, p = .951), PSI (M = 100.5, p = .740), nor CVLT-II (M = −.11, p = .493) were significantly different from normative, standardization population means.¹ If the included participants were representative of a sample composed of clinically referred examinees with diverse neurodevelopmental, psychiatric, and neurocognitive disorders, it is reasonable to expect that at least subtle-to-mild decreases from normative performance may be detectable in aspects of working memory, processing speed, and/or verbal memory. Prior literature has described such differences from healthy controls in these domains in samples of participants with ADHD (Fuermaier et al., 2015; Poysophon & Rao, 2018; Theiling & Petermann, 2016), psychiatric conditions (Hermens et al., 2011; Moghadasin & Dibajnia, 2021), and mND (Economou et al., 2007; Greenaway et al., 2006; Lumpkin & Sheerin, 2019). As such, it is likely that the participants included by Resch, Soble, et al. (2021) were more closely akin to a sample of cognitively intact/healthy participants (i.e., a normative sample) rather than a mixed clinical sample commensurate with typical neuropsychological practice, limiting the extent to which these results are generalizable to other clinically referred examinees.

Second, Resch, Soble, et al.’s (2021) focus was solely on the VSVT, limiting its applicability to other PVTs, particularly those using different test paradigms. Of note, recent work has suggested that PVTs may be domain/modality specific (see Erdodi, 2019). In the context of developing novel PVTs, the domain/modality specificity hypothesis posits that “the extent to which . . . variables are similar in relevant psychometric features (test paradigm, cognitive domain, sensory modality, etc.)” (Erdodi et al., 2018, p. 108) can affect their utility, classification accuracy, and psychometric robustness. By extension, PVTs may be most sensitive to noncredible or otherwise reduced performance within the cognitive domain(s) they appear to assess (Victor et al., 2013), and memory-based PVTs may fail to detect the signal of all manifestations of invalidity (Dandachi-FitzGerald & Merckelbach, 2013). Consider the VSVT, which appears to assess both functions of working memory and recognition memory. Examinees (consciously or otherwise) demonstrating invalid performance on those domains may be more likely to perform poorly on the VSVT and bona fide cognitive tests of those abilities. Other measures, including the Test of Memory Malingering (Tombaugh, 1996) and Word Choice Test (WCT; Pearson, 2009), more distinctly appear to assess visual and verbal recognition memory, respectively. Furthermore, measures such as the Dot Counting Test (Boone et al., 2002) appear to measure attention and processing speed without any appearance of assessing memory, such that they may be more appropriate to detect noncredible performance in speed than in memory. Continued work exploring the relationships between cognitive functioning across domains and PVTs of various paradigms/modalities is needed.

Third, although Resch, Soble, et al.’s (2021) utilization of a mixed clinical sample may provide benefits for generalizability, it included relatively high proportions of patients with ADHD diagnoses who were relatively young (M age = 31.7); moreover, conclusions drawn from such samples may be limited for clinical groups with more specific presenting concerns and neurological conditions. For example, there has been a burgeoning interest in performance validity in the context of specific disease states, including multiple sclerosis (Domen et al., 2020; Galioto et al., 2020; Sanborn et al., 2021), a chronic, inflammatory, autoimmune disease that presents with various clusters of physical, cognitive, and psychological symptoms (Macaron & Ontaneda, 2019). Base rates of invalidity in such patients may be upward of 20% (e.g., Galioto et al., 2020; Nauta et al., 2021), consistent with rates in other clinical populations (McWhirter et al., 2020).

The Present Study

In light of these limitations, the present study sought to contribute to the literature by extending Resch, Soble, et al.’s (2021) prior work by (a) determining the extent to which cognitive functioning in the domains of auditory working memory, processing speed, and verbal memory accounts for variance within three separate, standalone PVTs (b) in a sample of patients with multiple sclerosis (pwMS). It was hypothesized that, in line with prior work, cognitive functioning would account for relatively minimal (i.e., ≤15%) variance in PVT scores.

Methods

Patient Characteristics

Participants were identified from an archival clinical data set of pwMS referred for clinical neuropsychological assessment within a large, Midwestern, academic medical center. Seventy-five (75) participants with complete data for variables of interest were included (M age = 48.51, SD = 12.04), and they had an average of 13.87 years of education (SD = 2.37) and disease duration of 14.60 years (n = 73; SD = 12.48). Most participants were women (70.7%), identified as White (82.7%), and were diagnosed with relapsing-remitting subtype (80.0%). Of the 74 participants with known disability seeking status, just over one quarter (26.7%) reported applying for or considering applying for disability benefits. Demographic information, which appeared well-representative of population-level characteristics of pwMS and of the entire archival data set from which participants were selected, and other descriptive data are provided in Table 1.

Table 1.

Demographic and Descriptive Results

Groups	Total sample (N = 75)	Group 1 (n = 63)	Group 2 (n = 54)	Group 3 (n = 56)
Variable	M (SD) or n (%)
Age, years	48.51 (12.04)	48.70 (12.78)	49.65 (12.72)	49.50 (12.57)
Education, years	13.87 (2.37)	14.24 (2.28)	14.41 (2.19)	14.29 (2.32)
Gender
Men	22 (29)	20 (32)	19 (35)	19 (34)
Women	53 (71)	43 (68)	35 (65)	37 (66)
Racial/ethnic identity
Caucasian/White	62 (83)	55 (87)	46 (85)	46 (82)
African American/Black	10 (13)	5 (8)	5 (9)	7 (13)
Other	3 (4)	3 (5)	3 (6)	3 (5)
MS subtype
RRMS	60 (80)	49 (78)	43 (80)	45 (80)
PPMS	9 (12)	8 (13)	8 (15)	8 (14)
SPMS	5 (7)	5 (8)	3 (6)	3 (5)
Monophasic	1 (1)	1 (2)	0 (0)	0 (0)
Disease duration, years	n = 73; 14.60 (12.48)	n = 61; 16.02 (13.03)	n = 53; 17.26 (13.34)	n =55; 17.07 (13.18)
Pursuing disability benefits	n = 74; 20 (26.7)	n = 62; 14 (22.2)	n = 53; 9 (16.7)	n = 55; 10 (17.9)
VSVT Total Score	43.05 (6.34)	44.63 (4.67)	46.39 (1.58)	46.20 (1.82)
VSVT Fail	18 (24)	10 (15.9)	—	—
WCT	46.15 (7.58)	48.52 (1.74)	48.48 (2.37)	48.73 (1.62)
WCT Fail	9 (12)	—	1 (1.9)	—
DCT-E	13.74 (6.23)	12.22 (3.91)	11.73 (3.90)	12.67 (6.08)
DCT-E Fail	7 (7.3)	—	—	3 (5.4)
DS ACSS	8.19 (2.43)	8.68 (2.23)	8.96 (2.22)	8.86 (2.26)
SDMT T	39.86 (13.15)	41.59 (13.26)	42.22 (12.98)	41.32 (13.48)
CVLT-II LDFR z	−.99 (1.37)	−.78 (1.22)	−.68 (1.15)	−.63 (1.11)

Note. Total N/n is equal to that denoted in the top row of each column unless otherwise noted. Group 1 (n = 63) represents participants who passed both the WCT and DCT-E. Group 2 (n = 54) represents participants who passed both VSVT and DCT-E. Group 3 (n = 56) represents participants who passed both the VSVT and WCT. MS = multiple sclerosis; RRMS = relapsing-remitting MS; PPMS = primary progressive MS; SPMS = secondary progressive MS; Disease Duration, Years = Years since reported symptom onset, not date of diagnosis; VSVT = Victoria Symptom Validity Test; WCT = Word Choice Test; DCT-E = Dot Counting Test E-score; DS = Wechsler Adult Intelligence Scale, Fourth Edition Digit Span subtest; ACSS = age-corrected scaled score, with M = 10 and SD = 3; SDMT = Symbol Digit Modalities Test, oral trial presented as a T score, with M = 50 and SD = 10; CVLT-II LDFR = California Verbal Learning Test, Second Edition, Long Delay Free Recall presented as a z score, with M = 0 and SD = 1.

Measures

Performance Validity Tests

Detailed descriptions of each PVT’s paradigm are not provided herein to protect test security. The Victoria Symptom Validity Test (VSVT; Slick et al., 1997) is a standalone, memory-based PVT. A recent analysis indicated the best performing variable on the VSVT was Total score ≤ 40 to indicate noncredible performance (Resch, Webber, et al., 2021), and only this score was of interest to the current study. Response latency–based variables were not considered given that they tend to contribute no unique benefit for classification accuracy above and beyond accuracy scores (Cerny et al., 2021).

The WCT (Pearson, 2009) is a standalone PVT with good construct validity demonstrated in experimental and clinical work (Erdodi et al., 2014; Lace, Grant, et al., 2021). A recent meta-analysis suggested an optimal cutoff score of ≤42 to denote noncredible performance (Bernstein et al., 2021).

The Dot Counting Test (DCT; Boone et al., 2002) is a standalone PVT with good construct validity in diverse clinical samples (Rhoads, Resch, et al., 2021; Soble et al., 2018). The E-score (DCT) was the variable of interest. The authors chose a conservative cutoff for noncredible performance of ≥22 to remain with similar work (Lace, Merz, & Galioto, 2021) and because this score may be appropriate in other populations with significant neurological involvement (e.g., cortical stroke, mild dementia; Boone et al., 2002).

Of note, the administration of three PVTs was deemed appropriate for routine clinical care given the relatively high base rate of invalidity seen in pwMS (Nauta et al., 2021) and review of previous literature that has administered four to five standalone PVTs to clinical neuropsychological examinees (Pliskin et al., 2021; Soble et al., 2020; Soble et al., 2021).

Cognitive Measures

The Digit Span subtest from the WAIS-IV (Wechsler, 2008) was used as a measure of auditory working memory. It is one of the two subtests that comprises the WMI composite—which was used by Resch, Soble, et al. (2021)—within the WAIS-IV. The age-corrected scaled score (ACSS; M = 10, SD = 3) for Digit Span was of interest.

The Symbol Digit Modalities Test (SDMT; Smith, 1973) was used as a measure of processing speed. It is a substitution task that is similar in appearance to and strongly correlated with Coding (previously Digit Symbol Coding) within various iterations of the WAIS (Bowler et al., 1992; Morgan & Wheelock, 1992). The oral trial was preferred over the written trial in the present study as it is widely used as the processing speed measure of choice in neuropsychological assessment of pwMS (Arnett & Strober, 2014; Benedict et al., 2006; Langdon et al., 2012) and minimizes psychomotor burden. Of interest was the age- and education-corrected T scores (M = 50, SD = 10) for each participant calculated according to the test manual (Smith, 1973).

The Long Delay Free Recall (CVLT-II) trial of the California Verbal Learning Test, Second Edition (CVLT-II; Delis et al., 2000) was used as a measure of verbal memory. Resch, Soble, et al. (2021) used this variable to represent verbal memory, as well. The CVLT-II is widely used as a verbal learning/memory test of choice with pwMS (Benedict et al., 2006; Langdon et al., 2012; Merz et al., 2018). The age- and gender-corrected z score (M = 0, SD = 1) for CVLT-II was of interest, with prior work supporting this score as a good representation of this construct (Donders, 2008).

Procedures

Participants were identified from an institutional review board–approved registry. All participants were referred for comprehensive neuropsychological assessment between 2020 and 2021 as part of medical care and were evaluated by a board certified clinical neuropsychologist or clinical neuropsychologist trained under Houston Conference Guidelines. All neuropsychological tests were administered by a Certified Specialist in Psychometry or competently trained psychometrist, clinical psychology doctoral student, or postdoctoral neuropsychology fellow familiar with standardized test administration and scoring procedures. Only participants with complete data for the PVT and cognitive variables of interest were included.

Statistical Analyses

Except for those performed in Microsoft Excel where described, analyses were performed with SPSS 26.0. Consistent with both Resch, Soble, et al. (2021) and logical expectation, PVT variables were not normally distributed, VSVT Total D(75) = .26, p < .001; WCT D(75) = .33, p < .001; DCT D(75) = .15, p < .001. VSVT Total (skewness = −1.66, kurtosis = 2.25) and WCT (skewness = −3.21, kurtosis = 11.10) demonstrated a negative skew, and DCT (skewness = 2.08, kurtosis = 6.79) demonstrated a positive skew. As such, VSVT Total and WCT were reflected (subtracting each participant’s score from the maximum score for either variable plus one) and logarithmically transformed. DCT was also logarithmically transformed. These approaches were consistent with Resch, Soble, et al.’s (2021) methods. No PVT scores were identified as univariate outliers (i.e., ≥|3.29| SD from the mean; Field, 2013) after transformations, and no scores for cognitive variables were identified as univariate outliers either.

Bivariate Pearson’s correlations between each pair of variables for the total sample were calculated. A series of linear regressions were conducted with Digit Span, SDMT, and CVLT-II scores entered as predictor variables as one set, and (logarithmically transformed) PVT variables entered as the outcome. Tolerance (acceptable values are ≥ .10) and variance inflation factor (VIF) diagnostics (acceptable values are ≤ 10; Hair et al., 2010) were examined as indicators of possible multicollinearity for each regression model. To control for Type 1 error in the context of three separate regression analyses, conservative critical α = .017 (i.e., .05/3) was chosen a priori for evaluating models and individual predictors.

In an attempt to borrow from Resch, Soble, et al.’s (2021) methodology of utilizing only participants operationally defined as “credible,” pwMS were included in a regression analysis for a PVT only if they passed both other PVTs. For example, for the VSVT regression (Group 1), pwMS passing both the WCT and DCT were included (n = 63). For the WCT regression (Group 2), pwMS passing both the VSVT and DCT were included (n = 54). For the DCT regression (Group 3), pwMS passing both the VSVT and WCT were included (n = 56). Participants were retained even if they performed outside normal limits on the PVT of interest for the specific regression. This approach was deemed appropriate, as the mean VSVT Total scores for pwMS passing both the WCT and DCT (M = 44.98, SD = 4.18) was not significantly different from the mean VSVT Total scores reported by Resch, Soble, et al. (2021; M = 44.90, SD = 4.50, Welch’s t [130] = 0.36, p = .723),² indicating that the operational definitions of credibility likely did not produce systematic differences in PVT scores between samples and studies.

Results

As shown in Table 1, the mean scores for each PVT were within normal limits for the total sample and for each group. Mean scores for Digit Span tended to fall in the average range, albeit at the lower end. Mean scores for SDMT and CVLT-II tended to fall in the low average range, with mild variability among groups (Guilmette et al., 2020). One-sample t tests indicated that mean scores for Digit Span, SDMT, and CVLT-II across all three groups were significantly lower than the population mean for each (all ps ≤ .001).

Bivariate correlations between each pair of variables for the total sample (N = 75) are shown in Table 2. Correlations between each pair of variables for each Group are presented in supplemental material (Supplemental Table 1). In the total sample, correlations between pairs of transformed PVTs were low to moderate (rs ranged from .26 to .52). Correlations between transformed PVTs and Digit Span, SDMT, and CVLT-II ranged from low to moderate (rs ranged from −.58 to −.16). Correlations between Digit Span, SDMT, and CVLT-II were generally low (rs ranged from .21 to .26) in magnitude and not suggestive of meaningful multicollinearity or psychometric redundancy. Notably, the strength of some correlations differed markedly across groups. For example, VSVT-CVLT Total sample r = −.46 (suggesting approximately 21% shared variance), while VSVT-CVLT Group 3 r = −.13 (suggesting approximately 1%–2% shared variance). Nearly all correlations across groups (except those falling close to zero) shared the same directionality (see Supplemental Table 1).

Table 2.

Bivariate Correlations in the Total Sample

Variable	1. ^a	2. ^a	3. ^c	4.	5.	6.
VSVT ^a	—	.52**^b	.47**	−.58**	−.21	−.46**
WCT ^a		—	.26*	−.36**	−.26*	−.56**
DCT ^c			—	−.53**	−.47**	−.16
DS				—	.26*	.25*
SDMT					—	.21
CVLT-II						—

Note. Total group N = 75. Pearson’s (r) correlation coefficients are displayed. VSVT = Victoria Symptom Validity Test Total Score; WCT = Word Choice Test; DCT = Dot Counting Test E-score; DS = WAIS-IV Digit Span Age-Corrected Scaled Score. CVLT-II = California Verbal Learning Test, Second Edition, Long Delay Free Recall z score.

For correlations, variable was reflected and logarithmically transformed. As such, the directionality of Pearson’s (r) correlation coefficients should be reversed when interpreting these results from a practical/clinical perspective, except for the Pearson’s correlation between logarithmically transformed VSVT Total Score and WCT (^b), which can be interpreted as is. ^c For Pearson’s (r) correlation coefficients, variable was logarithmically transformed.

p < .05. **p < .01.

Table 3 displays results from regression analyses. For the VSVT Total score regression (n = 63), multicollinearity diagnostics were acceptable (Tolerance ranged from .96 to .99; VIF ranged from 1.01 to 1.04). Results indicated that the overall model was significant, F(3, 59) = 9.456, p < .001, and accounted for 32.5% of the variance in VSVT Total scores. Digit Span (p < .001) and CVLT-II (p = .006) emerged as significant individual predictors. Squared semipartial correlations indicated that Digit Span accounted for more than two and one-third times as much unique variance in VSVT Total scores (22.3%) than did CVLT-II (9.4%), and SDMT accounted for essentially no unique variance (0.1%). In light of the reflected and logarithmically transformed VSVT Total score, decreases in Digit Span and CVLT-II were associated with decreased VSVT Total scores.

Table 3.

Regression Models

Model	R ²	B (SE)	β	t (p)	sr ²
VSVT Total ^a (n = 63)
Overall model	.325	—	—	—	—
Constant	—	1.047 (.203)	—	5.161 (<.001)	—
DS	—	−.081 (.018)	−.475	−4.413 (<.001)	.223
SDMT	—	.001 (.003)	.036	.329 (.743)	.001
CVLT-II LDFR	—	−.097 (.034)	−.311	−2.874 (.006)	.094
WCT ^a (n = 54)
Overall model	.244	—	—	—	—
Constant	—	.581 (.205)	-	2.828 (.007)	—
DS	—	−.030 (.017)	−.219	−1.779 (.081)	.048
SDMT	—	−.002 (.003)	−.102	−.797 (.429)	.010
CVLT-II LDFR	—	−.105 (.034)	−.397	−3.098 (.003)	.145
DCT-E ^b (n = 56)
Overall model	.384	—	—	—	—
Constant	—	1.565 (.091)	—	17.282 (<.001)	—
DS	—	−.028 (.008)	−.372	−3.367 (.001)	.134
SDMT	—	−.006 (.001)	−.451	−3.963 (<.001)	.186
CVLT-II LDFR	—	.024 (.017)	.154	1.364 (.178)	.022

Note. VSVT = Victoria Symptom Validity Test; WCT = Word Choice Test; DCT-E = Dot Counting Test E-score; DS = Wechsler Adult Intelligence Scale, Fourth Edition Digit Span subtest; SDMT = Symbol Digit Modalities Test, oral trial; CVLT-II LDFR = California Verbal Learning Test, Second Edition, Long Delay Free Recall; sr² = semipartial correlation squared.

Variable was reflected and logarithmically transformed. As such, the directionality of regression coefficients should be reversed when considering these results from a practical/clinical perspective. ^b Variable was logarithmically transformed.

For the WCT regression (n = 54), multicollinearity diagnostics were also acceptable (Tolerance ranged from .92 to 1.00; VIF ranged from 1.00 to 1.08). The overall model was significant, F(3, 50) = 5.373, p = .003, and accounted for 24.4% of the variance in WCT scores. Only CVLT-II (p = .003) emerged as a significant individual predictor. Squared semipartial correlations showed that CVLT-II accounted for just over three times as much unique variance in WCT scores (14.5%) than did Digit Span (4.8%), and SDMT accounted for minimal unique variance (1%). In light of the reflected and logarithmically transformed WCT variable, decreases in CVLT-II were associated with decreased WCT scores.

For the DCT regression (n = 56), multicollinearity diagnostics were acceptable (Tolerance ranged from .92 to .97; VIF ranged from 1.03 to 1.09). The overall model was significant, F(3, 52) = 10.806, p < .001, and accounted for 38.4% of the variance in DCT scores. SDMT emerged as a significant individual predictor (p < .001), as did Digit Span (p = .001). Squared semipartial correlations indicated that SDMT accounted for approximately one and one-third times as much unique variance in DCT scores (18.6%) than did Digit Span (13.4%), and CVLT-II accounted for minimal (2.2%) unique variance. As DCT was only logarithmically transformed and not reflected, decreases in Digit Span and SDMT were associated with increased (i.e., trending toward invalidity) DCT scores.

Discussion

Neuropsychologists are tasked to determine the validity of obtained cognitive data, and some prior work has indicated that bona fide neuropsychological dysfunction may negatively impact examinees’ performance on measures of validity. The present study sought to determine the extent to which cognitive functioning in the domains of working memory, processing speed, and verbal memory accounted for variance within three separate, standalone PVTs in a sample of pwMS. Several findings deserve further discussion.

First, regarding demographics, the present sample appeared to well represent the population-level characteristics of pwMS, such that the modal pwMS tends to be a White, middle-aged woman with relapsing-remitting subtype (Hadgkiss et al., 2013; Minden et al., 2006). Relatedly, regarding cognitive test scores, although still within broad normal limits (i.e., low average to average range), the mean performances on cognitive tests in the total sample and its groups of operationally defined credible performers were significantly lower than normative data and appeared more consistent with scores derived from a clinical sample than those reported by Resch, Soble, et al. (2021). This pattern is consistent with other work indicating that samples of pwMS demonstrate low average to average performance on aspects of working memory, processing speed, and verbal memory (Archibald & Fisk, 2000; Beatty, 2004; Lafosse et al., 2013; Lindau et al., 2022; Merz et al., 2018).

Second, the pattern of obtained findings were contrary to both hypothesized results and to Resch, Soble, et al.’s (2021) conclusions. Resch, Soble, et al. (2021) found that working memory, speed, and verbal memory significantly but minimally predicted performance on VSVT, accounting for 13% of variance in total scores when all three neuropsychological domains were included in the regression model; a similar pattern was expected herein. In contrast, despite including the same or otherwise very similar neuropsychological measures (i.e., Digit Span, SDMT, and CVLT-II) and similar methodological and statistical approaches, the present study found that performance on neuropsychological tasks accounted for approximately between 24% and 38% of variance in PVT scores, with 32.5% on VSVT Total scores. These patterns are exceptionally higher than those reported by Resch, Soble, et al. (2021) and suggest that in pwMS—and potentially other clinical populations with even subtle yet meaningful cognitive difficulties—the psychometric overlap between PVTs and cognitive domains of working memory, speed, and verbal memory appear more strongly related than previously thought. Importantly, it appeared that the WCT was the least affected (relatively speaking) by cognitive functioning out of the three PVTs of interest and may be a promising tool for the assessment of validity in this population; this finding aligns with a recent poster presentation indicating that the WCT shared the less variance with objective neuroradiological disease burden in pwMS than did DCT and VSVT (Lace et al., 2022). Notably, the difference in magnitude between some correlations across groups may be due to idiosyncrasies of PVT paradigms (including the domain/modality specificity hypothesis discussed below) or that continuous PVT variables may remain somewhat non-normally distributed and affected by outliers even after applying appropriate statistical corrections (such as logarithmic transformation as was done herein and in Resch, Soble et al., 2021).

Of particular relevance was the pattern of relationships between cognitive tests and PVTs. Specifically, verbal memory significantly influenced both VSVT and WCT scores, working memory influenced VSVT and DCT scores, and processing speed influenced DCT scores. At least two possible explanations for these findings deserve consideration. First, it may be that the PVTs explored herein are especially (even overly) sensitive to bona fide neurocognitive dysfunction, particularly in this sample of MS patients. That is, while a “robust PVT is . . . relatively insensitive to actual cognitive ability(ies)” (Resch, Soble, et al., 2021, p. 1620), the degree to which these particular PVTs (i.e., VSVT, WCT, and DCT) remain insensitive to neurocognitive problems, at least with pwMS, may be inadequate. Consider the relatively high proportion of unique variance that Digit Span (13.4%) and SDMT (18.6%) accounted for in DCT scores. In addition to its purpose as a PVT, it is possible that the DCT actually is (at least in part) a legitimate measure of attention/working memory and processing speed. Similarly, it may be that both the VSVT and WCT (again, at least in part) assess verbal memory to a degree above and beyond what would be expected if they were only measures of performance validity. If this explanation is true, it may be the likelihood of incorrectly identifying invalidity may be unnecessarily elevated with increasing degrees of neurocognitive dysfunction. Of note, a recent systematic review and cross-validation of the VSVT specifically stated that “caution is recommended among patients with . . . working memory deficits due to concerns for increased risk of false positives” (Resch, Webber, et al., 2021, p. 331). Thus, it may be that the three PVTs utilized herein are too easily contaminated by neurocognitive dysfunction and of limited appropriateness for clinical use, at least with pwMS.

Alternatively, and a more favored explanation in the authors’ opinion, these findings may provide adjunctive support for the domain/modality specificity hypothesis of PVTs (Erdodi, 2019; Rai & Erdodi, 2021) rather than argue for their conservative or limited use. The construct of invalidity does not always manifest similarly across examinees nor is it equally detectable by all types of PVTs (e.g., Dandachi-FitzGerald & Merckelbach, 2013), as “patients may fail one type of [performance] validity test more than another because of the relevance of the test material to their presenting complaints” (Gervais et al., 2004, p. 476). The domain/modality specificity hypothesis is particularly relevant in developing new PVTs, as the match between criterion and predictor PVTs across test-specific characteristics can meaningfully influence their psychometric properties (Rai & Erdodi, 2021). By extension, PVTs that appear to measure certain cognitive domains (e.g., memory, processing speed) are likely to better detect noncredible performance demonstrated by examinees on actual tests of those abilities. Thus, experts recommend continuous, multi-domain sampling of validity throughout neuropsychological assessment (Boone, 2009).

Considering these findings in light of the domain/modality specificity hypothesis, it is reasonable to expect stronger psychometric relationships between variables that tap the same “cognitive construct” when performance on tests to that end trends toward invalidity. It may be that pwMS who performed invalidly with respect to attention/processing speed but not necessarily with memory performed worse more readily on tests that appeared to assess the former—which, by extension, include SDMT, Digit Span, and DCT. Relatedly, it may be that pwMS who performed invalidly with regard to memory but not necessarily with attention/processing speed more often performed worse than their actual abilities on tests appearing to measure memory—which, by extension, include VSVT, WCT, CVLT-II, and Digit Span. The relationship between PVTs and performance on “true” cognitive tests may be a function of the role of effort (or validity broadly speaking) within those neuropsychological factors. As such, it may be that the relatively high degree of psychometric overlap between constructually similar cognitive tests and PVTs speaks to their ability to detect the “signal” of noncredible performance within their specific domain/modality, rather than an oversensitivity to overt neurocognitive dysfunction or neurobiological insult. Consider the high proportion of unique variance accounted for by Digit Span in VSVT Total scores (22.3%). This finding may certainly reflect the paradigmatic resemblance of each test (i.e., recalling/recognizing strings of digits), and also relate to prior research supporting the use of Digit Span as a PVT (e.g., Babikian et al., 2006; Hurtubise et al., 2020; Shura et al., 2020; Webber & Soble, 2018), though exploration of this latter claim extends beyond the scope of this article. Both possibilities relate to the domain/modality specificity hypothesis and may, at least to some extent, account for the psychometric overlap between Digit Span and VSVT herein. Similar arguments may be made for both symbol-digit/digit-symbol coding tests and CVLT-II, which have been identified as possible embedded PVTs (e.g., Erdodi & Abeare, 2020; Erdodi et al., 2017; Persinger et al., 2018; Wolfe et al., 2010), albeit with relatively less support than the body of work on Digit Span.

Importantly, the authors firmly acknowledge that firm support for or refutations against either of the above explanations cannot be adequately reached by these findings alone given the cross-sectional nature employed herein. Future studies may seek to employ experimental methods to parse apart reasons behind and mechanisms of invalidity.

The present study, of course, is not without its own limitations. First, the current sample size was relatively small and thus increased the possibility of Type II error. However, this factor was not fatal as the number of participants in each regression met widely accepted statistical standards (Harris, 1985; Wilson VanVoorhis & Morgan, 2007). Second, though the present study aimed to replicate and extend findings from Resch, Soble, et al. (2021), some comparisons were not exact due to differences in operationalization of select measures and differing statistical analyses. Specifically, the current study did not include additional VSVT variables (e.g., latency), though the rationale for VSVT variables included (i.e., Total Score was the best performing variable in a recent meta-analysis; response latency is not additively helpful in identifying invalidity; Cerny et al., 2021; Resch, Webber, et al., 2021) was deemed appropriate by the authors. Finally, the use of logarithmic transformations is a common practice when variables demonstrate remarkable skew (Tabachnick & Fidell, 2013) and was consistent with methodology outlined by Resch, Soble, et al. (2021). However, such transformations beget some degree of interpretive difficulty and should be understood in the context of transformation (Feng et al., 2014; Tabachnick & Fidell, 2013). Future work may seek to utilize statistical techniques, such as categorical regression techniques (CATREG) in SPSS (see van der Kooij, 2007), which may minimize the need for data transformation(s).

Conclusion

In sum, the current study provided evidence that cognitive abilities, particularly working memory, processing speed, and verbal memory, may account for a greater portion of PVT performance than previously suggested. Alternative explanations were proposed, including the possible oversensitivity of these PVTs to bona fide cognitive dysfunction and providing support for the domain/modality specific hypothesis (Erdodi, 2019). Taken at face value, the authors are likely proponents of the latter explanation, such that PVTs may be more sensitive to invalid performance demonstrated on tests within narrow domains and that psychometric overlap (even between PVTs and cognitive ability tests) increases with greater paradigmatic similarity between measures. Nonetheless, clinicians should use prudence and careful judgment when interpreting PVT failures, especially in pwMS and similar populations, and utilize all clinical information in reaching conclusions. Encouragingly, cognitive functioning appeared least meaningfully related to WCT scores (24.4% variance) compared with VSVT (32.5% variance) and DCT (38.5% variance) scores. Continued work exploring invalidity and cognitive functioning in pwMS and other clinical populations is warranted.

Supplemental Material

sj-docx-1-asm-10.1177_10731911231178289 – Supplemental material for Standalone Performance Validity Tests May Be Differentially Related to Measures of Working Memory, Processing Speed, and Verbal Memory in Patients With Multiple Sclerosis

Supplemental material, sj-docx-1-asm-10.1177_10731911231178289 for Standalone Performance Validity Tests May Be Differentially Related to Measures of Working Memory, Processing Speed, and Verbal Memory in Patients With Multiple Sclerosis by John W. Lace, Victoria Sanborn and Rachel Galioto in Assessment

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

John W. Lace

Supplemental Material

Supplemental material for this article is available online.

Notes

References

K. Y.

Kaploun

Erdodi

L. A.

Abeare

C. A.

(2017). Performance validity in undergraduate research participants: A comparison of failure rates across tests and cutoffs. The Clinical Neuropsychologist, 31(1), 193–206. https://doi.org/10.1080/13854046.2016.1217046

Archibald

C. J.

Fisk

J. D.

(2000). Information processing efficiency in patients with multiple sclerosis. Journal of Clinical and Experimental Neuropsychology, 22(5), 686–701. https://doi.org/10.1076/1380-3395(200010)22:5;1-9; FT686

Arnett

P. A.

Strober

L. B.

(2014). Multiple sclerosis and demyelinating disorders. In Parsons

M. W.

Hammeke

T. A.

Snyder

P. J.

(Eds.), Clinical neuropsychology: A pocket handbook for assessment (3rd ed.). (pp. 287–312). American Psychological Association.

Babikian

Boone

K. B.

Arnold

(2006). Sensitivity and specificity of various digit span scores in the detection of suspect effort. The Clinical Neuropsychologist, 20(1), 145–159. https://doi.org/10.1080/13854040590947362

Beatty

(2004). RBANS analysis of verbal memory in multiple sclerosis. Archives of Clinical Neuropsychology, 19(6), 825–834. https://doi.org/10.1016/j.acn.2003.12.001

Benedict

R. H. B.

Cookfair

Gavett

Gunther

Munschauer

Garg

Weinstock-Guttman

(2006). Validity of the minimal assessment of cognitive function in multiple sclerosis (MACFIMS). Journal of the International Neuropsychological Society, 12(4), 549–558. https://doi.org/10.1017/S1355617706060723

Bernstein

M. T.

Resch

Z. J.

Ovsiew

G. P.

Soble

J. R.

(2021). A systematic review and meta-analysis of the diagnostic accuracy of the Advanced Clinical Solutions Word Choice Test as a performance validity test. Neuropsychology Review, 31, 349–359. https://doi.org/10.1007/s11065-020-09468-y

Boone

K. B.

(2009). The need for continuous and comprehensive sampling of effort/response bias during neuropsychological examinations. The Clinical Neuropsychologist, 23(4), 729–741. https://doi.org/10.1080/13854040802427803

Boone

K. B.

Herzberg

(2002). The dot counting test. Western Psychological Services.

10.

Bowler

Sudia

Mergler

Harrison

Cone

(1992). Comparison of digit symbol and symbol digit modalities tests for assessing neurotoxic exposure. Clinical Neuropsychologist, 6(1), 103–104. https://doi.org/10.1080/13854049208404123

11.

Cerny

B. M.

Rhoads

Leib

S. I.

Jennette

K. J.

Basurto

K. S.

Durkin

N. M.

Ovsiew

G. P.

Resch

Z. J.

Soble

J. R.

(2021). Mean response latency indices on the Victoria Symptom Validity Test do not contribute meaningful predictive value over accuracy scores for detecting invalid performance. Applied Neuropsychology: Adult, 29, 1304–1311. https://doi.org/10.1080/23279095.2021.1872575

12.

Critchfield

Soble

J. R.

Marceaux

J. C.

Bain

K. M.

Chase Bailey

Webber

T. A.

Alex Alverson

Messerly

Andrés González

O’Rourke

J. J. F.

(2019). Cognitive impairment does not cause invalid performance: Analyzing performance patterns among cognitively unimpaired, impaired, and noncredible participants across six performance validity tests. The Clinical Neuropsychologist, 33(6), 1083–1101. https://doi.org/10.1080/13854046.2018.1508615

13.

Dandachi-FitzGerald

Merckelbach

(2013). Feigning ≠ feigning a memory deficit: The medical symptom validity test as an example. Journal of Experimental Psychopathology, 4(1), 46–63. https://doi.org/10.5127/jep.025511

14.

Davis

J. J.

(2018). Performance validity in older adults: Observed versus predicted false positive rates in relation to number of tests administered. Journal of Clinical and Experimental Neuropsychology, 40(10), 1013–1021. https://doi.org/10.1080/13803395.2018.1472221

15.

Dean

A. C.

Victor

T. L.

Boone

K. B.

Arnold

(2008). The relationship of IQ to effort test performance. The Clinical Neuropsychologist, 22(4), 705–722. https://doi.org/10.1080/13854040701440493

16.

Dean

A. C.

Victor

T. L.

Boone

K. B.

Philpott

L. M.

Hess

R. A.

(2009). Dementia and effort test performance. The Clinical Neuropsychologist, 23(1), 133–152. https://doi.org/10.1080/13854040701819050

17.

Delis

D. C.

Kramer

J. H.

Kaplan

Ober

B. A.

(2000). California Verbal Learning Test (2nd ed.). Psychological Corporation.

18.

Domen

C. H.

Greher

M. R.

Hosokawa

P. W.

Barnes

S. L.

Hoyt

B. D.

Wodushek

T. R.

(2020). Are established embedded performance validity test cut-offs generalizable to patients with multiple sclerosis? Archives of Clinical Neuropsychology, 35(5), 511–516. https://doi.org/10.1093/arclin/acaa016

19.

Donders

(2008). A confirmatory factor analysis of the California Verbal Learning Test—Second edition (CVLT-II) in the standardization sample. Assessment, 15(2), 123–131. https://doi.org/10.1177/1073191107310926

20.

Economou

Papageorgiou

S. G.

Karageorgiou

Vassilopoulos

(2007). Nonepisodic memory deficits in amnestic MCI. Cognitive and Behavioral Neurology, 20(2), 99–106. https://doi.org/10.1097/WNN.0b013e31804c6fe7

21.

Erdodi

L. A.

(2019). Aggregating validity indicators: The salience of domain specificity and the indeterminate range in multivariate models of performance validity assessment. Applied Neuropsychology: Adult, 26(2), 155–172. https://doi.org/10.1080/23279095.2017.1384925

22.

Erdodi

L. A.

(2021). Five shades of gray: Conceptual and methodological issues around multivariate models of performance validity. Neurorehabilitation, 49(2), 179–213. https://doi.org/10.3233/NRE-218020

23.

Erdodi

L. A.

Abeare

C. A.

(2020). Stronger together: The Wechsler Adult Intelligence Scale—Fourth edition as a multivariate performance validity test in patients with traumatic brain injury. Archives of Clinical Neuropsychology, 35(2), 188–204. https://doi.org/10.1093/arclin/acz032

24.

Erdodi

L. A.

Abeare

C. A.

Lichtenstein

J. D.

Tyson

B. T.

Kucharski

Zuccato

B. G.

Roth

R. M.

(2017). Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) processing speed scores as measures of noncredible responding: The third generation of embedded performance validity indicators. Psychological Assessment, 29(2), 148–157. https://doi.org/10.1037/pas0000319

25.

Erdodi

L. A.

Hurtubise

J. L.

Charron

Dunn

Enache

McDermott

Hirst

R. B.

(2018). The D-KEFS Trails as performance validity tests. Psychological Assessment, 30(8), 1082–1095. https://doi.org/10.1037/pas0000561

26.

Erdodi

L. A.

Kirsch

N. L.

Lajiness-O’Neill

Vingilis

Medoff

(2014). Comparing the recognition memory test and the word choice test in a mixed clinical sample: Are they equivalent? Psychological Injury and Law, 7(3), 255–263. https://doi.org/10.1007/s12207-014-9197-8

27.

Feng

Wang

Chen

X. M.

(2014). Log-transformation and its implications for data analysis. Shanghai Archives of Psychiatry, 26(2), 105–109. https://doi.org/10.3969/j.issn.1002-0829.2014.02.009

28.

Field

(2013). Discovering statistics using IBM SPSS statistics. SAGE.

29.

Fuermaier

A. B. M.

Tucha

Koerts

Aschenbrenner

Kaunzinger

Hauser

Weisbrod

Lange

K. W.

Tucha

(2015). Cognitive impairment in adult ADHD—Perspective matters! Neuropsychology, 29(1), 45–58. https://doi.org/10.1037/neu0000108

30.

Galioto

Dhima

Berenholz

Busch

(2020). Performance validity testing in multiple sclerosis. Journal of the International Neuropsychological Society, 26(10), 1028–1035. https://doi.org/10.1017/S1355617720000466

31.

Gervais

Rohling

Green

Ford

(2004). A comparison of WMT, CARB, and TOMM failure rates in non-head injury disability claimants. Archives of Clinical Neuropsychology, 19(4), 475–487. https://doi.org/10.1016/j.acn.2003.05.001

32.

Green

Flaro

(2021). Performance validity test failure predicts suppression of neuropsychological test results in developmentally disabled children. Applied Neuropsychology: Child, 10(1), 65–81. https://doi.org/10.1080/21622965.2019.1604342

33.

Greenaway

M. C.

Lacritz

L. H.

Binegar

Weiner

M. F.

Lipton

Munro Cullum

(2006). Patterns of verbal memory performance in mild cognitive impairment, Alzheimer disease, and normal aging. Cognitive and Behavioral Neurology, 19(2), 79–84. https://doi.org/10.1097/01.wnn.0000208290.57370.a3

34.

Guilmette

T. J.

Sweet

J. J.

Hebben

Koltai

Mahone

E. M.

Spiegler

B. J.

Stucky

Westerveld

, & Conference Participants. (2020). American Academy of Clinical Neuropsychology consensus conference statement on uniform labeling of performance test scores. The Clinical Neuropsychologist, 34(3), 437–453. https://doi.org/10.1080/13854046.2020.1722244

35.

Hadgkiss

E. J.

Jelinek

G. A.

Weiland

T. J.

Pereira

N. G.

Marck

C. H.

van der Meer

D. M.

(2013). Methodology of an international study of people with multiple sclerosis recruited through web 2. 0 platforms: Demographics, lifestyle, and disease characteristics. Neurology Research International, 2013, 1–12. https://doi.org/10.1155/2013/580596

36.

Hair

J. F.

Black

Babin

B. J.

Anderson

R. E.

(2010). Multivariate data analysis: Global edition (7th ed.). Academic Press.

37.

Harris

R. J.

(1985). A primer of multivariate statistics (2nd ed.). Academic Press.

38.

Hermens

D. F.

Redoblado Hodge

M. A.

Naismith

S. L.

Kaur

Scott

Hickie

I. B.

(2011). Neuropsychological clustering highlights cognitive differences in young people presenting with depressive symptoms. Journal of the International Neuropsychological Society, 17(2), 267–276. https://doi.org/10.1017/S1355617710001566

39.

Hurtubise

Baher

Messa

Cutler

Shahein

Hastings

Carignan-Querqui

Erdodi

L. A.

(2020). Verbal fluency and digit span variables as performance validity indicators in experimentally induced malingering and real world patients with TBI. Applied Neuropsychology: Child, 9(4), 337–354. https://doi.org/10.1080/21622965.2020.1719409

40.

Lace

J. W.

Grant

A. F.

Ruppert

Kaufman

D. A. S.

Teague

C. L.

Lowell

Gfeller

J. D.

(2021). Detecting noncredible performance with the neuropsychological assessment battery, screening module: A simulation study. The Clinical Neuropsychologist, 35(3), 572–596. https://doi.org/10.1080/13854046.2019.1694703

41.

Lace

J. W.

Merz

Z. C.

Galioto

(2021). Exploring the structured inventory of malingered symptomatology in patients with multiple sclerosis. Psychological Injury and Law, 14(4), 291–303. https://doi.org/10.1007/s12207-021-09424-y

42.

Lace

J. W.

Nakamura

Husak

Ontaneda

Galioto

(2022, February 4). Standalone performance validity test scores and neuroradiological disease burden in patients with multiple sclerosis [Poster Presentation]. Presented virtually at the 2022 annual meeting of the International Neuropsychological Society, New Orleans, LA, United States. https://perma.cc/47N5-C28G.

43.

Lafosse

J. M.

Mitchell

S. M.

Corboy

J. R.

Filley

C. M.

(2013). The nature of verbal memory impairment in multiple sclerosis: A list-learning and meta-analytic study. Journal of the International Neuropsychological Society, 19(9), 995–1008. https://doi.org/10.1017/S1355617713000957

44.

Langdon

Amato

Boringa

Brochet

Foley

Fredrikson

Hämäläinen

Hartung

H.-P.

Krupp

Penner

Reder

Benedict

(2012). Recommendations for a brief international cognitive assessment for multiple sclerosis (BICAMS). Multiple Sclerosis Journal, 18(6), 891–898. https://doi.org/10.1177/1352458511431076

45.

Larrabee

G. J.

(2014). Test validity and performance validity: Considerations in providing a framework for development of an ability-focused neuropsychological test battery. Archives of Clinical Neuropsychology, 29(7), 695–714. https://doi.org/10.1093/arclin/acu049

46.

Laurent

R. L.

Whiteside

D. M.

Basso

M. R.

(2019). Performance validity testing in an older adult population: Considerations for clinical practice. In Ravdin

L. D.

Katzen

H. L.

(Eds.), Handbook on the neuropsychology of aging and dementia (pp. 77–88). Springer.

47.

Lindau

Lundberg

Najström

(2022). WAIS-IV short form applied to a mixed neurological Swedish clinical sample. Nordic Psychology, 74(2), 114–124. https://doi.org/10.1080/19012276.2021.1918225

48.

Loring

D. W.

Goldstein

F. C.

Chen

Drane

D. L.

Lah

J. J.

Zhao

Larrabee

G. J.

(2016). False-positive error rates for reliable digit span and auditory verbal learning test performance validity measures in amnestic mild cognitive impairment and early Alzheimer disease. Archives of Clinical Neuropsychology, 31(4), 313–331. https://doi.org/10.1093/arclin/acw014

49.

Lumpkin

J. C. M.

Sheerin

C. M.

(2019). Digit span sequencing as a neurocognitive screening tool in an aging, veteran population. Psychology & Neuroscience, 12(2), 180–190. https://doi.org/10.1037/pne0000140

50.

Macaron

Ontaneda

(2019). Diagnosis and management of progressive multiple sclerosis. Biomedicines, 7(3), 56. https://doi.org/10.3390/biomedicines7030056

51.

Maiman

Del Bene

V. A.

MacAllister

W. S.

Sheldon

Farrell

Arce Rentería

Slugh

Nadkarni

S. S.

Barr

W. B.

(2019). Reliable digit span: Does it adequately measure suboptimal effort in an adult epilepsy population? Archives of Clinical Neuropsychology, 34(2), 259–267. https://doi.org/10.1093/arclin/acy027

52.

McWhirter

Ritchie

C. W.

Stone

Carson

(2020). Performance validity test failure in clinical populations—A systematic review. Journal of Neurology, Neurosurgery & Psychiatry, 91(9), 945–952. https://doi.org/10.1136/jnnp-2020-323776

53.

Merz

Z. C.

Wright

J. D.

Vander Wal

J. S.

Gfeller

J. D.

(2018). A factor analytic investigation of the Mercy Evaluation of Multiple Sclerosis. The Clinical Neuropsychologist, 32(8), 1431–1453. https://doi.org/10.1080/13854046.2018.1426786

54.

Miller

J. B.

Axelrod

B. N.

(2018). Performance validity assessment: Disentangling dementia from the disinterested and disingenuous. In Bush

S. S.

Heck

A. L.

(Eds.), Forensic geropsychology: Practice essentials (pp. 25–47). American Psychological Association.

55.

Minden

S. L.

Frankel

Hadden

Perloff

Srinath

K. P.

Hoaglin

D. C.

(2006). The Sonya Slifka longitudinal multiple sclerosis study: Methods and sample characteristics. Multiple Sclerosis Journal, 12(1), 24–38. https://doi.org/10.1191/135248506ms1262oa

56.

Moghadasin

Dibajnia

(2021). Verbal and practical intelligence in general anxiety, obsessive compulsive and major depression disorders. European Review of Applied Psychology, 71(1), 100630. https://doi.org/10.1016/j.erap.2021.100630

57.

Morgan

S. F.

Wheelock

(1992). Digit symbol and symbol digit modalities tests: Are they directly interchangeable? Neuropsychology, 6(4), 327–330. https://doi.org/10.1037/0894-4105.6.4.327

58.

Nauta

Bertens

van Dam

Huiskamp

Driessen

Geurts

Uitdehaag

Fasotti

Hulst

de Jong

Klein

(2021). Performance validity in outpatients with multiple sclerosis and cognitive complaints. Multiple Sclerosis Journal, 28, 642–653.

59.

Odland

A. P.

Lammy

A. B.

Martin

P. K.

Grote

C. L.

Mittenberg

(2015). Advanced administration and interpretation of multiple validity tests. Psychological Injury and Law, 8(1), 46–63. https://doi.org/10.1007/s12207-015-9216-4

60.

Ord

A. S.

Shura

R. D.

Sansone

A. R.

Martindale

S. L.

Taber

K. H.

Rowland

J. A.

(2021). Performance validity and symptom validity tests: Are they measuring different constructs? Neuropsychology, 35(3), 241–251. https://doi.org/10.1037/neu0000722

61.

Pearson. (2009). Advanced clinical solutions word choice test.

62.

Persinger

V. C.

Whiteside

D. M.

Bobova

Saigal

S. D.

Vannucci

M. J.

Basso

M. R.

(2018). Using the California Verbal Learning Test, Second Edition as an embedded performance validity measure among individuals with TBI and individuals with psychiatric disorders. The Clinical Neuropsychologist, 32(6), 1039–1053. https://doi.org/10.1080/13854046.2017.1419507

63.

Pliskin

J. I.

DeDios Stern

Resch

Z. J.

Saladino

K. F.

Ovsiew

G. P.

Carter

D. A.

Soble

J. R.

(2021). Comparing the psychometric properties of eight embedded performance validity tests in the Rey auditory verbal learning test, Wechsler memory scale logical memory, and brief visuospatial memory test–revised recognition trials for detecting invalid neuropsychological test performance. Assessment, 28(8), 1871–1881. https://doi.org/10.1177/1073191120929093

64.

Poysophon

Rao

A. L.

(2018). Neurocognitive deficits associated with ADHD in athletes: A systematic review. Sports Health: A Multidisciplinary Approach, 10(4), 317–326. https://doi.org/10.1177/1941738117751387

65.

Proto

D. A.

Pastorek

N. J.

Miller

B. I.

Romesser

J. M.

Sim

A. H.

Linck

J. F.

(2014). The dangers of failing one or more performance validity tests in individuals claiming mild traumatic brain injury-related postconcussive symptoms. Archives of Clinical Neuropsychology, 29(7), 614–624. https://doi.org/10.1093/arclin/acu044

66.

Quattlebaum

Martin

Moltisanti

Clark

Schroeder

(2019). A-76 specificity of the Repeatable Battery for the Assessment of Neuropsychological Status digit span as a validity indicator in patients with mild cognitive impairment and dementia. Archives of Clinical Neuropsychology, 34(6), 936–936. https://doi.org/10.1093/arclin/acz034.76

67.

Rai

J. K.

Erdodi

L. A.

(2021). Impact of criterion measures on the classification accuracy of TOMM-1. Applied Neuropsychology: Adult, 28(2), 185–196. https://doi.org/10.1080/23279095.2019.1613994

68.

Resch

Z. J.

Soble

J. R.

Ovsiew

G. P.

Castillo

L. R.

Saladino

K. F.

DeDios-Stern

Schulze

E. T.

Song

Pliskin

N. H.

(2021). Working memory, processing speed, and memory functioning are minimally predictive of Victoria Symptom Validity Test performance. Assessment, 28(6), 1614–1623. https://doi.org/10.1177/1073191120911102

69.

Resch

Z. J.

Webber

T. A.

Bernstein

M. T.

Rhoads

Ovsiew

G. P.

Soble

J. R.

(2021). Victoria Symptom Validity Test: A systematic review and cross-validation study. Neuropsychology Review, 31(2), 331–348. https://doi.org/10.1007/s11065-021-09477-5

70.

Rhoads

Neale

A. C.

Resch

Z. J.

Cohen

C. D.

Keezer

R. D.

Cerny

B. M.

Jennette

K. J.

Ovsiew

G. P.

Soble

J. R.

(2021). Psychometric implications of failure on one performance validity test: A cross-validation study to inform criterion group definition. Journal of Clinical and Experimental Neuropsychology, 43(5), 437–448. https://doi.org/10.1080/13803395.2021.1945540

71.

Rhoads

Resch

Z. J.

Ovsiew

G. P.

White

D. J.

Abramson

D. A.

Soble

J. R.

(2021). Every second counts: A comparison of four Dot Counting Test scoring procedures for detecting invalid neuropsychological test performance. Psychological Assessment, 33(2), 133–141. https://doi.org/10.1037/pas0000970

72.

Sanborn

Lace

Gunstad

Galioto

(2021). Considerations regarding noncredible performance in the neuropsychological assessment of patients with multiple sclerosis: A case series. Applied Neuropsychology: Adult, 30, 458–467. https://doi.org/10.1080/23279095.2021.1971229

73.

Schroeder

R. W.

Martin

P. K.

Heinrichs

R. J.

Baade

L. E.

(2019). Research methods in performance validity testing studies: Criterion grouping approach impacts study outcomes. The Clinical Neuropsychologist, 33(3), 466–477. https://doi.org/10.1080/13854046.2018.1484517

74.

Shura

R. D.

Martindale

S. L.

Taber

K. H.

Higgins

A. M.

Rowland

J. A.

(2020). Digit Span embedded validity indicators in neurologically-intact veterans. The Clinical Neuropsychologist, 34(5), 1025–1037. https://doi.org/10.1080/13854046.2019.1635209

75.

Slick

D. J.

Hopp

Strauss

Thompson

G. B.

(1997). Manual for the Victoria Symptom Validity Test. Psychological Assessment Resources.

76.

Smith

(1973). Symbol digit modalities test [Manual]. Western Psychological Services.

77.

Soble

J. R.

Alverson

W. A.

Phillips

J. I.

Critchfield

E. A.

Fullen

O’Rourke

J. J. F.

Messerly

Highsmith

J. M.

Bailey

K. C.

Webber

T. A.

Marceaux

J. C.

(2020). Strength in numbers or quality over quantity? Examining the importance of criterion measure selection to define validity groups in performance validity test (PVT) Research. Psychological Injury and Law, 13(1), 44–56. https://doi.org/10.1007/s12207-019-09370-w

78.

Soble

J. R.

Santos

O. A.

Bain

K. M.

Kirton

J. W.

Bailey

K. C.

Critchfield

E. A.

O’Rourke

J. J. F.

Highsmith

J. M.

González

D. A.

(2018). The Dot Counting Test adds up: Validation and response pattern analysis in a mixed clinical veteran sample. Journal of Clinical and Experimental Neuropsychology, 40(4), 317–325. https://doi.org/10.1080/13803395.2017.1342773

79.

Soble

J. R.

Sharp

D. W.

Carter

D. A.

Jennette

K. J.

Resch

Z. J.

Ovsiew

G. P.

Critchfield

E. A.

(2021). Cross-validation of a forced-choice validity indicator to enhance the clinical utility of the Rey Auditory Verbal Learning Test. Psychological Assessment, 33(6), 568–573. https://doi.org/10.1037/pas0001018

80.

Sweet

J. J.

Heilbronner

R. L.

Morgan

J. E.

Larrabee

G. J.

Rohling

M. L.

Boone

K. B.

Kirkwood

M. W.

Schroeder

R. W.

Suhr

J. A.

, & Conference Participants. (2021). American Academy of Clinical Neuropsychology (AACN) 2021 consensus statement on validity assessment: Update of the 2009 AACN consensus conference statement on neuropsychological assessment of effort, response bias, and malingering. The Clinical Neuropsychologist, 35(6), 1053–1106. https://doi.org/10.1080/13854046.2021.1896036

81.

Tabachnick

B. G.

Fidell

L. S.

(2013). Using multivariate statistics (6th ed.). Pearson.

82.

Teichner

Wanger

M. T.

(2004). The Test of Memory Malingering (TOMM): Normative data from cognitively intact, cognitively impaired, and elderly patients with dementia. Archives of Clinical Neuropsychology, 19(3), 455–464. https://doi.org/10.1016/S0887-6177(03)00078-7

83.

Theiling

Petermann

(2016). Neuropsychological profiles on the WAIS-IV of adults with ADHD. Journal of Attention Disorders, 20(11), 913–924. https://doi.org/10.1177/1087054713518241

84.

Tombaugh

T. N.

(1996). Test of Memory Malingering. MHS.

85.

van der Kooij

A. J

. (2007). Prediction accuracy and stability of regression with optimal scaling transformations [Doctoral Thesis]. Universiteit Leiden.

86.

Van Dyke

S. A.

Millis

S. R.

Axelrod

B. N.

Hanks

R. A

. (2013). Assessing effort: Differentiating performance and symptom validity. The Clinical Neuropsychologist, 27(8), 1234–1246. https://doi.org/10.1080/13854046.2013.835447

87.

Victor

T. L.

Boone

K. B.

Kulick

A. D.

(2013). Assessing noncredible sensory, motor, and executive function, and test battery performance in mild traumatic brain injury cases. In Carone

D. A.

Bush

S. S.

(Eds.), Mild traumatic brain injury: Symptom validity assessment and malingering (pp. 269–301). Springer.

88.

Webber

T. A.

Soble

J. R.

(2018). Utility of various WAIS-IV Digit Span indices for identifying noncredible performance validity among cognitively impaired and unimpaired examinees. The Clinical Neuropsychologist, 32(4), 657–670. https://doi.org/10.1080/13854046.2017.1415374

89.

Wechsler

(2008). Wechsler adult intelligence scale (4th ed.). Pearson.

90.

Wilson VanVoorhis

C. R.

Morgan

B. L

. (2007). Understanding power and rules of thumb for determining sample sizes. Tutorials in Quantitative Methods for Psychology, 3(2), 43–50.

91.

Wodushek

T. R.

Domen

C. H.

(2020). Comparing two models of performance validity assessment in patients with Parkinson’s disease who are candidates for deep brain stimulation surgery. Applied Neuropsychology: Adult, 27(1), 9–21. https://doi.org/10.1080/23279095.2018.1473251

92.

Wolfe

P. L.

Millis

S. R.

Hanks

Fichtenberg

Larrabee

G. J.

Sweet

J. J.

(2010). Effort indicators within the California Verbal Learning Test-II (CVLT-II). The Clinical Neuropsychologist, 24(1), 153–168. https://doi.org/10.1080/13854040903107791

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB