Comparing the Psychometric Properties of Eight Embedded Performance Validity Tests in the Rey Auditory Verbal Learning Test,Wechsler Memory Scale Logical Memory,and Brief Visuospatial Memory Test–Revised Recognition Trials for Detecting Invalid Neuropsychological Test Performance

Abstract

This cross-sectional study evaluated eight embedded performance validity tests (PVTs) previously derived from the Rey Auditory Verbal Learning Test (RAVLT), Wechsler Memory Scale–IV–Logical Memory (LM), and Brief Visuospatial Memory Test–Revised (BVMT-R) recognition trials among a single mixed clinical sample of 108 neuropsychiatric patients (83 valid/25 invalid) with (n = 54) and without (n = 29) mild neurocognitive disorder. Among the overall sample, all eight recognition PVTs significantly differentiated valid from invalid performance (areas under the curve [AUCs] = .64-.81) with 26% to 44% sensitivity (≥89% specificity) at optimal cut-scores depending on the specific PVT. After subdividing the sample by cognitive impairment status, all eight PVTs continued to reliably identify invalid performance (AUC = .68-.91) with markedly increased sensitivities of 56% to 80% (≥89% specificity) in the unimpaired group. In contrast, among those with mild neurocognitive disorder, RAVLT False Positives and LM became nonsignificant, whereas the other six PVTs remained significant (AUC = .64-.77), albeit with reduced sensitivities of 32% to 44% (≥89% specificity) at optimal cut-scores. Taken together, results cross-validated BVMT-R and most RAVLT recognition indices as effective embedded PVTs for identifying invalid neuropsychological test performance with diverse populations including examinees with and without suspected mild neurocognitive disorder, whereas LM had more limited utility as an embedded PVT, particularly when mild neurocognitive disorder was present.

Keywords

performance validity neuropsychology assessment memory psychometrics

Irrespective of a clinical or research setting, accurate (and useful) assessment of memory is contingent on memory test scores that are a valid and credible reflection of the examinee’s true cognitive abilities. Several factors may affect whether a test score is an accurate reflection of the examinee’s true memory functioning, most notably inadequate task engagement. Misidentification of memory deficits can have serious repercussions, including increased health care utilization and excessive spending of resources (Horner et al., 2014; Lippa, 2018), introduction of iatrogenic effects secondary to unnecessary medications, and possible inaccuracies in the inclusion of participants in research samples. Performance validity tests (PVTs) are measures designed to objectively assess credibility of test performance due to their ability to identify potential response bias or low effort (Boone, 2018). Thus, within the past decade, multiple position papers published by national neuropsychological organizations, such as the National Academy of Neuropsychology and the American Academy of Clinical Neuropsychology (Bush et al., 2005; Heilbronner et al., 2009), have advocated for routine inclusion of PVTs in neuropsychological evaluations to accurately interpret low test results as indicative of underlying brain dysfunction, inform diagnostic conclusions, and render appropriate treatment recommendations.

Given that it has been posited that memory impairment probably is the variety of cognitive dysfunction most commonly feigned in personal injury claims (Williams, 1998), it is particularly important to include PVTs when assessing memory functioning. While several freestanding PVTs are highly sensitive for detecting invalid performance among clinical and medicolegal samples (e.g., Denning, 2012) and have tended to outperform embedded PVTs (e.g., Bain et al., 2019), they simultaneously bear the associated costs of additional evaluation time and financial expenditures, as well as potential burden to the examinee (e.g., increased fatigue, stress). More recently, embedded PVTs have increased in popularity in clinical and research settings as a means of assessing validity while minimizing these drawbacks. Embedded indicators are derived from existing neuropsychological tests, and thereby serve the dual purpose of providing information about the examinee’s cognitive function and the validity of his or her performance. Thus, without additional tests, embedded PVTs can save the clinician valuable evaluation time and cost as well as minimize patient burden while allowing for continuous sampling of validity throughout the duration of an evaluation, consistent with practice standards (Boone, 2009). These PVTs may also be more resistant to coaching as their embedded nature may make them more difficult for an examinee to identify (Schutte & Axelrod, 2013), which helps preserve test integrity and security. Furthermore, embedded PVTs generally maintain robust sensitivity for detection of invalid performance, especially when multiple tests are aggregated (Erdodi et al., 2014; Larrabee, 2008; Meyers et al., 2011; Schutte et al., 2011). Given the benefits of embedded PVTs, there has been a rapid proliferation in empirical evidence for these measures (see Rickards et al., 2018, for review), with many being derived from recognition paradigms on standard memory tests.

While several PVTs embedded in memory measures have been developed, recognition PVTs from three commonly administered memory measures are the focus of this study: Rey Auditory Verbal Learning Test (RAVLT; Rey, 1941), Weschler Memory Scale–IV (WMS-IV; Wechsler, 2009) Logical Memory (LM), and Brief Visuospatial Memory Test–Revised (BVMT-R; Benedict, 1997). Prior RAVLT findings have shown that an Effort Score (ES) equation derived from the recognition trial (i.e., Hits − False Positives + Primacy Recognition [i.e., number of the first five words on List A recognized]) effectively discriminated valid from noncredible medicolegal examinees with an optimal cut-score of ≤12 (Boone et al., 2005). More recently, a notably lower optimal ES cutoff of ≤4 was found in a mixed clinical veteran sample, whereas a Recognition Hits cutoff of ≤9 had comparable specificity (i.e., ≥90%), but better sensitivity (i.e., 48% vs. 37%; Whitney & Davis, 2015). The WMS-IV LM Recognition trial has also been used as an embedded PVT within the Advanced Clinical Solutions program (ACS; Pearson, 2009), though studies have yielded mixed results regarding its utility. In a study comparing experimental malingerers, mixed-etiology patients, and healthy controls, performance on LM Recognition discriminated between groups, with malingerers performing worse than the other groups (Bouman et al., 2016). In contrast, more recent research with mixed clinical and traumatic brain injury (TBI) samples have raised questions regarding the psychometric soundness of LM Recognition, and suggested that, at best, it may have limited utility for detecting invalid performance among those with suspected cognitive impairment (Bain et al., 2019; Miller et al., 2011; Soble et al., 2019). Finally, BVMT-R Recognition Discrimination (RD; recognition hits minus False Positives) and Recognition Hits have been investigated as embedded PVTs. Initially, Denning (2012) found Recognition Hits ≤4 had 45% sensitivity/89% specificity for identifying invalid performance. Sawyer et al. (2017) reported neither BVMT-R RD nor Hits had adequate classification accuracy in another mixed clinical veteran sample. However, in a subsequent replication study, RD ≤4 evidenced robust sensitivity (50%) and specificity (93%) in a mixed clinical veteran sample, albeit with the caveat that ≤3 may be indicated for patients with amnestic disorders (Bailey, Soble, Bain, et al., 2018). More recently, RD ≤4 (54% sensitivity/93% specificity) and Hits ≤4; (41% sensitivity/95% specificity), adequately differentiated validity groups in a large sample of consecutive outpatients (Olsen et al., 2019). RD ≤4 findings also were replicated among a diverse mixed clinical neuropsychiatric sample with 40% sensitivity and ≥90% specificity when examined separately among cognitively impaired and unimpaired patients (Resch et al., 2020).

While many of these previously identified RAVLT, LM, and BVMT-R recognition embedded PVTs appear promising, several key limitations are present in the existing literature. Notably, few studies examined/compared more than 1 to 2 validity indices within individual memory measures. Likewise, with some exception (e.g., Bailey, Soble, Bain, et al., 2018; Sawyer et al., 2017), few studies have examined embedded PVTs from more than one memory measure within the same clinical sample. Consequently, there is a paucity of research that allows for direct comparison of these embedded recognition PVTs across memory measures to determine which indices have the most robust psychometric properties and maximum classification accuracy for identifying invalid performance. Given the numerous benefits of embedded PVTs, particularly when assessing memory, research investigating the effectiveness of these measures independently and in combination is vital for establishing a more accurate, objective way to assess performance validity of individuals undergoing memory testing, whether via clinical evaluation or for research purposes (Boone, 2018; Schroeder et al., 2019). Therefore, the objective of this study was to investigate the relative accuracy of all previously identified embedded PVTs derived from the RAVLT, WMS-IV LM, and BVMT-R recognition trials for detecting invalid neurocognitive test performance among a single clinical sample.

Method

Participants

Data for this cross-sectional study were obtained from a mixed clinical sample of 118 patients referred for neuropsychological evaluation at an academic medical center from 2018 to 2019 due to patient complaints of and/or provider concern for possible cognitive impairment, or for purposes of baseline evaluation prior to neurosurgical intervention (e.g., cerebral bypass, temporal lobectomy). All patients completed the RAVLT, WMS-IV LM, and BVMT-R as well as four independent criterion PVTs, including the Medical Symptom Validity Test (MSVT; Green, 2004), Test of Memory Malingering (TOMM; Tombaugh, 1996) Trial 1, ACS Word Choice Test (WCT; Pearson, 2009), and Dot Counting Test (DCT; Boone et al., 2002a) during their clinical evaluation. All patients consented to contributing their data to an institutional review board-approved database study. Of the 118 initial participants, a small subset (n = 10) met Diagnostic and Statistical Manual of Mental Disorders–Fifth edition (DSM-5; APA, 2013) diagnostic criteria for major neurocognitive disorder (i.e., dementia) and were excluded from analyses, which is supported by prior literature showing a substantially increased false positive rate of PVT failures among those with dementia (Dean et al., 2009). The final sample of 108 was 44% male (n = 47)/56% female (n = 61), and diverse in terms of age (M = 44.3 years; SD = 16.3; range = 18-78), educational attainment (M = 14.2 years; SD = 2.5; range = 8-20), and racial composition: 40% Caucasian (n = 43), 35% African American (n = 38), 17% Hispanic (n = 19), 5% Asian (n = 5), and 3% Middle Eastern (n = 3).

Validity group status was determined based on performance on four independent criterion PVTs (see Table 1). Specifically, consistent with current practice standards in clinical neuropsychology, which call for failure on at least two well-validated PVTs to identify performance invalidity (Larrabee, 2008, 2014; Meyers et al., 2014), those with ≥2 criterion PVT failures were classified as invalid (n = 25), and those with 0 (n = 68) or 1 (n = 15) failure(s) were classified as valid (n = 83). Indeed, 16% of the sample (n = 17) was compensation-seeking (i.e., active disability claim or litigation) at the time of evaluation. Of these 17 patients, 7 failed ≥2/4 criterion PVTs and were classified as invalid. The remaining 10 patients who were compensation-seeking all failed 0/4 criterion PVTs and were thus retained in the valid group. Of the 83 in the valid group, 29 had normal cognitive performance on objective testing and did not evidence cognitive impairment or meet DSM-5 criteria for a mild neurocognitive disorder as defined below (35%; “valid-cognitively unimpaired group”), whereas 54 (65%) met formal DSM-5 criteria for a mild neurocognitive disorder based on their neuropsychological evaluation results (“valid-cognitively impaired group”), which specifically requires a modest decline in cognitive function from baseline (preferably as verified by objective neuropsychological testing); however, the deficits do not interfere with independent daily functioning. For this study, a modest decline in cognitive function from premorbid baseline (as required by DSM-5) was operationalized as at least two impaired neuropsychological test scores (i.e., ≥1 standard deviation based on normative data and Test of Premorbid Function [Pearson, 2009] estimated premorbid IQ) within at least one cognitive domain without evidence of impaired activities of daily living. Areas of objective cognitive impairment among the valid-cognitively impaired group were not limited to memory but were heterogeneous with a number of contributing etiologies (see Table 2).

Table 1.

Criterion Performance Validity Tests (PVTs) Used to Establish Validity Groups.

Study PVT	Cutoff(s) for test failure	Reference(s)
MSVT	≤85% Correct on IR, DR, and/or CNS without the Genuine Memory Impairment Profile for those with clinical history of significant cognitive impairment	Green (2004)
TOMM T1	≤40	Denning (2012)
WCT	≤42	Bain and Soble (2019) Barhon et al. (2015)
DCT	≥14 Depression; PTSD; anxiety; mild TBI ≥15 Learning disorder ≥19 Mild neurocognitive disorder ≥20 Schizophrenia; TBI with positive imaging ≥21 Mild dementia	Boone et al. (2002b)

Note. MSVT = Medical Symptom Validity Test; IR = Immediate Recognition; DR = Delayed Recognition; CNS = Consistency; TOMM T1 = Test of Memory Malingering-Trail 1; WCT = Advanced Clinical Solutions Word Choice Test; DCT = Dot Counting Test; PTSD = posttraumatic stress disorder; TBI = traumatic brain injury.

Table 2.

Diagnostic Composition by Validity Group for the Overall Sample (N = 108).

	n (%)
Valid-unimpaired group	29 (27)
No diagnosis	4 (14)
Depressive disorder	4 (14)
Anxiety disorder	3 (10)
Posttraumatic stress disorder (PTSD)	1 (3)
Attention deficit hyperactivity disorder (ADHD)	5 (17)
Substance use disorder	2 (7)
Somatic symptom/conversion disorder	1 (3)
Chronic pain	4 (14)
Nonacute traumatic brain injury (TBI)/concussion	4 (14)
Other medical condition	1 (3)
Valid-impaired group	54 (50)
Stroke/cerebrovascular disease	12 (22)
Epilepsy	8 (15)
ADHD/learning disorder	7 (13)
Mild cognitive impairment-amnestic subtype	2 (4)
Substance-induced	3 (6)
TBI	2 (4)
Tumor	2 (4)
Parkinson disease/Lewy body	2 (4)
Human immunodeficiency virus (HIV)	2 (4)
Sleep apnea	1 (2)
Multiple sclerosis	1 (2)
Psychotic disorder	3 (6)
Multiple comorbid medical etiologies	7 (13)
Unspecified/unclear etiology	2 (4)
Invalid group	25 (23)
Primary psychiatric disorder with medical comorbidity	10 (40)
Depression and/or anxiety	8
PTSD	1
Somatic symptom	1
ADHD	5 (20)
Nonacute TBI/concussion	2 (8)
Chronic pain	1 (4)
Other medical condition	3 (12)
Unclear etiology	4 (16)

Measures

Rey Auditory Verbal Learning Test (Rey, 1941; Schmidt, 1996)

The RAVLT is one of the oldest and most commonly used verbal learning and memory measures. A list of unrelated words (List A) is presented orally across multiple learning trials, followed by a distractor list (List B), short and long delayed free recall trials, and a recognition trial. The recognition trial is presented either as a paragraph containing all words from List A that the examinee must identify, or, more commonly, as a list in which the List A words to be identified are intermixed with List B words and distractors (Strauss et al., 2006). Based on prior studies (e.g., Boone et al., 2005; Whitney & Davis, 2015), the following embedded PVTs from the RAVLT Recognition were examined in this study: (1) Total Hits (True Positives), (2) Total False Positives, (3) Recognition Discrimination (i.e., Total Hits − False Positives), and (4) ES (i.e., Hits − False Positives + Primacy Recognition [i.e., number of the first five words on List A recognized]). Of note, some previous studies used the paragraph recognition format to calculate the ES (e.g., Boone et al., 2005), whereas others used the list recognition format (e.g., Whitney & Davis, 2015). In this study, the list recognition format was used.

Wechsler Memory Scale–Fourth Edition (WMS-IV; Wechsler, 2009) Logical Memory

WMS-IV LM is a commonly administered measure of verbal memory. In the standard Adult Battery (ages 16-69 years), examinees are presented with two stories and instructed to freely recall details of each story both immediately and after a delay. A recognition task consisting of yes/no questions is then presented, and the total score is calculated using the number of correct responses. WMS-IV also has an Older Adult version of LM for those ≥70 years old that involves a shorter first story that is repeated twice, as well as a different highest possible recognition score. For this study, the only embedded PVT derived from the standard Adult LM Recognition was the Total Correct score. Given 14/108 patients were administered the Older Adult version of LM, which has a different possible total Recognition score, these 14 participants were excluded from all LM analyses.

Brief Visuospatial Memory Test–Revised (Benedict, 1997)

The BVMT-R is a visuospatial learning and memory task in which examinees are shown and asked to reproduce a visual array consisting of figures across several learning trials followed by a delayed recall trial and a recognition trial in which the target figures are intermixed with foils. All patients in this study completed BVMT-R Form 1. Based on prior studies (e.g., Bailey, Soble, Bain, et al., 2018; Denning, 2012), the following embedded PVTs derived from the BVMT-R Recognition Trial were examined in this study: (1) Total Hits (True Positives), (2) Total False Positives, and (3) Recognition Discrimination (i.e., Total Hits − False Positives).

Statistical Analyses

Descriptive statistics were calculated for the valid and invalid groups for all eight recognition embedded PVTs, including the RAVLT indices: (1) Total Hits, (2) Total False Positives (FP), (3) Recognition Discrimination (RD), (4) ES; LM: (5) Recognition Total Correct; and the BVMT-R indices: (6) Total Hits, (7) Total False Positives (FP), and (8) Recognition Discrimination (RD). Correlation analyses were then performed to examine relationships between the eight recognition PVTs among the valid groups. Two sets of analyses of variance (ANOVAs) with validity (valid/invalid) and validity-impairment (valid-unimpaired, valid-impaired, invalid) groups entered as the fixed factors and the recognition PVTs as the dependent variables were conducted with follow-up Games-Howell post hoc tests to assess for significant differences between validity groups. The false discovery rate (FDR) procedure with a 0.05 maximum FDR was implemented to control the family wise error rate related to multiple comparisons (Benjamini & Hochberg, 1995; Glickman et al., 2014). Next, receiver operating characteristic (ROC) curve analyses were performed to assess the accuracy of each embedded PVT for identifying invalid performance. Areas under the curve (AUCs) >.90 indicated high accuracy, .70 to .90 moderate accuracy, and .50 to .69 low accuracy (Swets, 1988). For PVTs with a significant AUC, optimal cut-scores that maximized sensitivity while maintaining acceptable specificity (≥90%), per practice standards (Boone, 2012), were identified. All ROC analyses were repeated after subdividing patients by cognitive impairment status to determine the effect(s) of mild neurocognitive disorder on each recognition memory PVT.

Results

Descriptive statistics for all eight recognition PVTs subdivided by validity and validity-impairment status are included in Table 4. See Table 3 for correlations between the recognition PVTs. Strong correlations were found for recognition indices within each of the memory measures, which is not surprising given several indices are derived from the same scores (e.g., the BVMT RD score is derived from the Total Hits and FP scores; therefore, it will correlate highly with each of these scores). However, between measures, nonsignificant correlations emerged between RAVLT and BVMT indices, whereas small to medium correlations were found between LM and all RAVLT and BVMT indices except for Total Hits.

Table 3.

Correlations Between Embedded Performance Validity Indices for the Valid Groups.

	RH	RFP	RD	RE	LM	BH	BFP	BD
RH	—	−.14	.48***	.53***	.20	−.07	.04	−.07
RFP		—	−.94***	−.91***	−.43***	−.06	.21	−.18
RD			—	.99***	.45***	.03	−.17	.14
RE				—	.44***	.03	−.16	.13
LM					—	.06	−.29**	.25*
BH						—	−.09	.72***
BFP							—	−.76***
BD								—

Note. n = 83 for main analyses except Logical Memory (n = 81). RH = Rey Auditory Verbal Learning Test (RAVLT) Hits; RFP = RAVLT False Positives; RD = RAVLT Recognition Discrimination; RE = RAVLT Effort Score; LM = Logical Memory; BH = Brief Visuospatial Memory Test–Revised (BVMT-R) Hits; BFP = BVMT-R False Positives; BD = BVMT-R Recognition Discrimination.

p < .05. **p < .01. **p < .001.

As seen in Table 4, ANOVAs with validity group (i.e., valid/invalid) as the fixed factor revealed significant overall effects and medium to large effect sizes for all RAVLT and BVMT PVTs, whereas LM evidenced a small effect size. Similarly, ANOVAs with validity-impairment group (i.e., valid-unimpaired, valid-impaired, invalid) as the fixed factor and the eight recognition PVTs as dependent variables revealed significant overall effects and large effect sizes for all RAVLT and BVMT PVTs, except for LM, which was nonsignificant. Follow-up Games-Howell post hoc tests for the RAVLT Total Hits and False Positives found that the valid-unimpaired group performed significantly better than valid-impaired and invalid groups, but nonsignificant differences emerged between the latter groups. For RAVLT RD and ES, the valid-unimpaired group scored significantly higher than the valid-impaired group, but both valid groups performed better than the invalid group. For BVMT, valid-unimpaired and valid-impaired groups generally performed comparably, and both groups performed significantly better than the invalid group.

Table 4.

Performance on Recognition Memory Embedded Performance Validity Tests (PVTs) by Validity Group.

	Valid (n = 83)	Invalid (n = 25)	F	η_p²
	M (SD)	M (SD)	F	η_p²
RAVLT Hits	12.81 (2.01)	10.52 (3.55)	16.83***	.14
RAVLT False Positives	4.27 (4.99)	8.48 (7.01)	11.23**	.10
RAVLT Discrimination	8.54 (5.64)	2.04 (7.76)	21.24***	.17
RAVLT Effort Score	13.11 (5.97)	5.80 (8.39)	23.61***	.18
LM Recognition	23.56 (2.55)	21.87 (5.08)	4.49*	.05
BVMT-R Hits	5.75 (0.60)	4.68 (1.31)	32.60***	.24
BVMT-R False Positives	0.25 (0.64)	1.08 (1.26)	19.47***	.16
BVMT-R Discrimination	5.49 (0.92)	3.60 (1.98)	44.89***	.30
	Valid-unimpaired (n = 29)	Valid- impaired (n = 54)	Invalid (n = 25)	F	η_p²	Post hoc
	M (SD)	M (SD)	M (SD)	F	η_p²	Post hoc
RAVLT Hits	13.76 (1.48)	12.30 (2.08)	10.52 (3.55)	12.47***	.19	^a,b
RAVLT False Positives	1.72 (2.02)	5.63 (5.56)	8.48 (7.01)	11.26***	.18	^a,b
RAVLT Discrimination	12.03 (2.96)	6.67 (5.86)	2.04 (7.76)	20.28***	.28	^a,b,c
RAVLT Effort Score	16.93 (3.15)	11.06 (6.13)	5.80 (8.39)	22.26***	.30	^a,b,c
LM Recognition	24.00 (2.24)	23.30 (2.71)	21.87 (5.08)	2.61	.05	—
BVMT-R Hits	5.93 (0.26)	5.65 (0.71)	4.68 (1.31)	17.63***	.25	^a,b,c
BVMT-R False Positives	0.14 (0.74)	0.31 (0.58)	1.08 (1.26)	10.16***	.16	^a,c
BVMT-R Discrimination	5.79 (0.77)	5.33 (0.95)	3.60 (1.98)	24.11***	.32	^a,c

Note. N = 108 for all main analyses except for Logical Memory (n = 94). RAVLT = Rey Auditory Verbal Learning Test; LM = Logical Memory; BVMT-R = Brief Visuospatial Memory Test–Revised.

Significant difference between valid-unimpaired and invalid groups. ^bSignificant difference between valid-unimpaired and valid-impaired groups. ^cSignificant difference between valid-impaired and invalid groups per Games-Howell post hoc tests.

p < .05. **p < .01. ***p < .001. All p values are false discovery rate-corrected.

For the overall sample, all ROC curve analyses were significant with AUCs of .64 to .81. Optimal cut-scores for each of the significant recognition PVTs as well as associated sensitivities/specificities and positive/negative predictive values are included in Tables 5 and 6. In general, measures evidenced acceptable specificity (i.e., ≥90%; Boone, 2012) with sensitivities of 26% to 44%. Overall, BVMT RD had the best sensitivity for detecting invalid performance. When ROC curve analyses were completed separately by cognitive impairment status, a different pattern emerged. Namely, among the valid-unimpaired group, AUCs for all eight recognition PVTs were significant and ranged from .68 to .91. Moreover, sensitivities ranged from 56% to 80% while maintaining ≥89% specificity. Conversely, among the valid-impaired group, AUCs decreased (i.e., .61-.77) and were nonsignificant for RAVLT FP and LM. All other RAVLT and BVMT indices remained significant with sensitivities of 32% to 44% and ≥89% specificity. Again, BVMT RD most accurately identified invalid cases with 44% sensitivity and 94% specificity among patients with mild neurocognitive disorder.

Table 5.

Receiver Operating Characteristic Curve Analyses for the Overall Sample (N = 108).

Recognition PVT	AUC	Invalid cut-score	SN (%)	SP (%)	Base rate (.40) PPV NPV		Base rate (.20) PPV NPV
RAVLT Hits	.70**	≤8	24	96	0.80	0.65	0.60	0.83
		≤9	36	93	0.77	0.69	0.56	0.85
		≤10	40	84	0.63	0.68	0.38	0.85
RAVLT False Positives	.68**	≥10	40	86	0.66	0.68	0.42	0.85
		≥12	36	89	0.69	0.68	0.45	0.85
		≥13	28	90	0.65	0.65	0.41	0.83
RAVLT Discrimination	.75***	≤ −2	28	94	0.76	0.66	0.54	0.84
		≤ −1	32	94	0.78	0.67	0.57	0.85
		≤0	32	93	0.75	0.67	0.53	0.85
RAVLT Effort Score	.76***	≤2	28	95	0.79	0.66	0.58	0.84
		≤3	32	94	0.78	0.67	0.57	0.85
		≤4	32	93	0.75	0.67	0.53	0.85
LM Recognition (n = 94)	.64*	≤18	22	96	0.79	0.65	0.58	0.83
		≤19	26	92	0.68	0.65	0.45	0.83
		≤20	30	85	0.57	0.65	0.33	0.83
BVMT-R Hits	.75***	≤3	20	99	0.93	0.65	0.83	0.83
		≤4	40	94	0.82	0.70	0.63	0.86
		≤5	64	82	0.70	0.77	0.47	0.90
BVMT-R False Positives	.71**	≥1	56	82	0.67	0.74	0.44	0.88
		≥2	32	95	0.81	0.68	0.62	0.85
		≥3	12	99	0.89	0.63	0.75	0.82
BVMT-R Discrimination	.81***	≤2	32	98	0.91	0.68	0.80	0.85
		≤3	44	95	0.85	0.72	0.69	0.87
		≤4	60	88	0.77	0.77	0.56	0.92

Note. Bolded rows indicate optimal cut-scores. PVT = Performance Validity Test; AUC = area under the curve; SN = Sensitivity; SP = Specificity; PPV = Positive Predictive Value; NPV = Negative Predictive Value; RAVLT = Rey Auditory Verbal Learning Test; LM = Logical Memory; BVMT-R = Brief Visuospatial Memory Test–Revised.

p < .05. **p < .01. ***p < .001.

Table 6.

Receiver Operating Characteristic Curve Analyses for Patients With and Without Mild Neurocognitive Disorder.

Recognition PVT	AUC	Invalid cut-score	SN, %	SP, %	Base rate (.40) PPV NPV		Base rate (.20) PPV NPV
Cognitively unimpaired vs. invalid (n = 54)
RAVLT Hits	.80***	≤10	40	97	0.90	0.71	0.77	0.87
		≤11	60	90	0.80	0.77	0.60	0.90
		≤12	64	79	0.67	0.77	0.43	0.90
RAVLT False Positives	.80***	≥4	68	83	0.73	0.80	0.50	0.91
		≥5	60	90	0.80	0.77	0.60	0.90
		≥6	56	90	0.79	0.75	0.58	0.89
RAVLT Discrimination	.90***	≤6	72	93	0.87	0.83	0.72	0.93
		≤8	80	90	0.84	0.87	0.67	0.95
		≤9	80	86	0.79	0.87	0.59	0.95
RAVLT Effort Score	.91***	≤11	76	90	0.84	0.85	0.66	0.94
		≤13	80	90	0.84	0.87	0.67	0.95
		≤14	88	83	0.78	0.91	0.56	0.97
LM Recognition (n = 50)	.68*	≤20	30	83	0.54	0.64	0.31	0.83
		≤21	56	89	0.77	0.75	0.56	0.89
		≤22	61	85	0.73	0.77	0.50	0.90
BVMT-R Hits	.80***	≤3	20	100	1.0	0.65	1.0	0.83
		≤4	40	100	1.0	0.71	1.0	0.87
		≤5	64	93	0.86	0.79	0.70	0.91
BVMT-R False Positives	.75**	≥1	56	97	0.93	0.77	0.82	0.90
		≥2	32	97	0.88	0.68	0.73	0.85
		≥3	12	97	0.73	0.62	0.50	0.82
BVMT-R Discrimination	.86***	≤3	44	97	0.91	0.72	0.79	0.87
		≤4	60	97	0.93	0.78	0.83	0.91
		≤5	80	90	0.84	0.87	0.67	0.95
Mild neurocognitive disorder vs. invalid (n = 79)
RAVLT Hits	.64*	≤8	24	94	0.73	0.65	0.50	0.83
		≤9	36	89	0.69	0.68	0.45	0.85
		≤10	40	78	0.55	0.66	0.31	0.84
RAVLT False Positives	.61	—	—	—	—	—	—	—
RAVLT Discrimination	.67*	≤ −2	28	91	0.67	0.65	0.44	0.83
		≤ −1	32	91	0.70	0.67	0.47	0.84
		≤0	32	89	0.66	0.66	0.42	0.84
RAVLT Effort Score	.68*	≤2	28	93	0.73	0.66	0.50	0.84
		≤3	32	91	0.70	0.67	0.47	0.84
		≤4	32	89	0.66	0.66	0.42	0.84
LM Recognition (n = 67)	.62	—	—	—	—	—	—	—
BVMT-R Hits	.72**	≤3	20	98	0.87	0.65	0.71	0.83
		≤4	40	91	0.75	0.69	0.53	0.86
		≤5	64	76	0.64	0.76	0.40	0.89
BVMT-R False Positives	.68*	≥1	56	74	0.59	0.72	0.35	0.87
		≥2	32	94	0.78	0.67	0.57	0.85
		≥3	12	100	1.0	0.63	1.0	0.82
BVMT-R Discrimination	.77***	≤2	32	98	0.91	0.68	0.80	0.85
		≤3	44	94	0.83	0.72	0.65	0.87
		≤4	60	83	0.70	0.76	0.47	0.89

p < .05. **p < .01. ***p < .001.

Discussion

This study examined eight previously identified embedded PVTs derived from the recognition trials of three common memory tests, RAVLT, WMS-IV LM, and BVMT-R, among a single, demographically diverse, mixed neuropsychiatric sample with and without mild neurocognitive disorder. Among the total sample, results revealed that all RAVLT and BVMT indices, as well as LM, significantly differentiated valid from invalid performance. For the RAVLT and BVMT, RD (and RAVLT ES), which accounts for both Total Hits and FP, was more accurate than either Hits or FP alone. Comparatively, BVMT RD had the most robust psychometric properties across all indices at its optimal cut-score. In contrast, LM Recognition had a relatively weak effect size for accurately distinguishing valid from invalid groups. When the sample was further subdivided by cognitive impairment status, a more divergent pattern emerged. Among the valid-cognitively unimpaired group, all eight recognition scores remained able to significantly differentiate valid from invalid performance with robust sensitives while maintaining acceptable specificity (i.e., ≥90%; Boone, 2012). However, among the valid-cognitively impaired group, RAVLT FP and LM were nonsignificant, whereas all other RAVLT and BVMT indices remained significant. Taken together, findings suggest the embedded PVTs from the RAVLT (excluding FP) and BVMT recognition trials can be effectively used in broad clinical populations, encompassing both those with normal cognitive functioning and those with known or suspected mild neurocognitive disorder. By contrast, WMS-IV LM recognition generally was ineffective at identifying invalid performance in the context of mild neurocognitive disorder and therefore warrants caution when used as an embedded PVT.

Findings from this study add further evidence to a growing literature base demonstrating that the BMVT-R RD is a psychometrically-robust embedded PVT (e.g., Bailey, Soble, Bain, et al., 2018; Olsen et al., 2019). The fact that RD held constant across analyses for patients with and without mild neurocognitive disorder provides further support that it can accurately differentiate invalid from valid performance even in the context of cognitive impairment. However, current results suggested that an optimal cut-score of ≤3 may be more broadly applicable for patients with more diverse areas of cognitive impairment than originally noted in the Bailey, Soble, Bain, et al. (2018) study, which identified this lower cutoff as specific to amnestic cognitive impairment. In contrast, among examinees with normal cognitive performance, results showed that increasing the BVMT RD cutoff to ≤5 achieved a substantial increase in sensitivity (i.e., 80%) while maintaining 90% specificity. Regarding the RAVLT, results similarly demonstrated that the RD and ES performed the best across all RAVLT indices and had comparable psychometric properties. However, unlike the BVMT RD, discrepant optimal cut-scores and sensitivities/specificities emerged for these two RAVLT indices based on whether analyses were performed with the overall sample versus subdivided by the unimpaired or impaired groups, which is consistent with more recent literature indicating that some PVTs are more adversely affected by cognitive impairment and may require different cut-scores among cases of cognitive impairment or with specific clinical groups (e.g., Alverson et al., 2019; Bailey, Soble, & O’Rourke, 2018; Boone et al., 2002a; Boone et al., 2002b; Webber & Soble, 2018). Similar RAVLT ES discrepancies also are present in the existing literature among studies utilizing different clinical populations. For example, Boone et al. (2005) used a sample of controls and a clinical group composed of nearly 40% primary mood or anxiety disorders and found a significantly higher optimal ES cutoff ≤12 using the paragraph recognition paradigm, whereas Whitney and Davis (2015) used a mixed clinical sample with a higher rate of cognitive impairment and reported a lower optimal ES score cutoff (i.e., ≤4) using the list recognition paradigm. Current findings approximate these studies and support the use of these RAVLT embedded PVTs among clinical populations that are likely to present with normal neurocognitive function (e.g., healthy controls, patients with nonacute mild TBI; Soble et al., 2017) as well as examinees with mild neurocognitive disorder albeit with the caveat that different cut-scores are indicated (and are accompanied by decreased sensitivity). Finally, current LM results mirrored prior findings (Bain et al., 2019; Soble et al., 2019), which similarly concluded that LM recognition has relatively poor utility as an embedded PVT, particularly in cases of mild neurocognitive disorder, in which it was unable to accurately differentiate valid from invalid performance.

While this study had several notable strengths, including use of a demographically diverse neuropsychiatric sample, adherence to recommended guidelines of using ≥2 PVT failures across multiple, independent criterion PVTs to identify invalid examinees (Larrabee, 2008, 2014; Meyers et al., 2014; Schroeder et al., 2019), and direct comparison of multiple embedded recognition PVTs across three common memory tests, some limitations are indicated. First, the diversity of this mixed clinical sample precluded further subgroup analyses by specific clinical diagnoses (e.g., amnestic mild neurocognitive disorder) to examine potential discrepant performance patterns based on etiology of impairment. There were an insufficiently small number of patients with dementia to perform separate analyses comparing them with those with mild neurocognitive disorder. While some research has found that those with significant cognitive impairment have a higher rate of PVT failure (Dean et al., 2009), others have reported that the overall false positive decreases when multiple PVT failures are used as the criterion for invalid performance (e.g., Critchfield et al., 2019; Loring et al., 2016; Webber et al., 2018), though differences between methodologies hinders direct comparisons between studies. In any event, current findings may not generalize to examinees with more severe cognitive impairment/dementia diagnoses; thus, future studies should examine whether these results replicate among this specific clinical population for purposes of appropriate cross-validation. Finally, although the cognitively unimpaired group displayed objectively normal neurocognitive performance on testing, it was not a true nonclinical/healthy control group given that these patients presented for clinical neuropsychological evaluation.

In sum, results contribute to a growing literature base demonstrating the effectiveness of embedded PVTs as useful tools for empirically assessing test performance credibility while minimizing evaluation expenditures and time demands. Although LM had questionable utility as a PVT among individuals with mild neurocognitive disorder in this diverse neuropsychiatric sample, multiple embedded PVTs within two common memory measures of material-specific verbal and nonverbal memory, the RAVLT and BVMT-R, respectively, evidenced good sensitivity for identifying noncredible test performance across patients with and without mild neurocognitive disorder, while maintaining acceptable specificity.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Zachary J. Resch

Jason R. Soble

References

Alverson

W. A.

O’Rourke

J. J. F.

Soble

J. R.

(2019). The Word Memory Test Genuine Memory Impairment Profile discriminates genuine memory impairment from invalid performance in a mixed clinical sample with cognitive impairment. The Clinical Neuropsychologist, 33(8), 1420-1435. https://doi.org/10.1080/13854046.2019.1599071

American Psychiatric Association (2013). Diagnostic and statistical manual of mental disorders – fifth edition (DSM-5). Arlington, VA: American Psychiatric Publishing.

Bailey

K. C.

Soble

J. R.

Bain

K. M.

Fullen

(2018). Embedded performance validity tests in the Hopkins Verbal Learning Test–Revised and the Brief Visuospatial Memory Test–Revised: A replication study. Archives of Clinical Neuropsychology, 33(7), 895-900. https://doi.org/10.1093/arclin/acx111

Bailey

K. C.

Soble

J. R.

O’Rourke

J. J. F.

(2018). Clinical utility of the Rey 15-Item Test, recognition trial, and error scores for detecting noncredible neuropsychological performance validity in a mixed-clinical sample of veterans. The Clinical Neuropsychologist, 32(1), 119-131. https://doi.org/10.1080/13854046.2017.1333151

Bain

K. M.

Soble

J. R.

(2019). Validation of the Advanced Clinical Solutions Word Choice Test (WCT) in a mixed clinical sample: Establishing classification accuracy, sensitivity/specificity, and cutoff scores. Assessment, 26(7), 1320-1328. https://doi.org/10.1177/1073191117725172

Bain

K. M.

Soble

J. R.

Webber

T. A.

Messerly

J. M.

Bailey

K. C.

Kirton

J. W.

McCoy

K. J. M.

(2019). Cross-validation of three Advanced Clinical Solutions performance validity tests: Examining combinations of measures to maximize classification of invalid performance. Applied Neuropsychology: Adult. Advance online publication. https://doi.org/10.1080/23279095.2019.1585352

Barhon

L. I.

Batchelor

Meares

Chekaluk

Shores

E. A.

(2015). A comparison of the degree of effort involved in the TOMM and the ACS Word Choice Test using dual task paradigm. Applied Neuropsychology Adult, 22(2), 114-123. https://doi.org/10.1080/23279095.2013.863775

Benedict

R. H. B.

(1997). Brief Visuospatial Memory Test–Revised. Psychological Assessment Resources.

Benjamini

Hochberg

(1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289-300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x

10.

Boone

K. B.

(2009). The need for continuous and comprehensive sampling of effort/response bias during neuropsychological examinations. The Clinical Neuropsychologist, 23(4), 729-741. https://doi.org/10.1080/13854040802427803

11.

Boone

K. B.

(2012). Clinical practice of forensic neuropsychology: An evidence-based approach. Guilford Press.

12.

Boone

K. B.

(2018). Assessment of neurocognitive performance validity. In Morgan

J. E.

Ricker

J.H.

(Eds.), Textbook of clinical neuropsychology (2nd ed., pp. 39-50). Routledge. https://doi.org/10.4324/9781315271743-4

13.

Boone

K. B.

Herzberg

D. S.

(2002a). The b Test manual. Western Psychological Services.

14.

Boone

K. B.

Herzberg

D. S.

(2002b). The Dot Counting Test manual. Western Psychological Services.

15.

Boone

K. B.

Wen

(2005). Comparison of various RAVLT scores in the detection of noncredible memory performance. Archives of Clinical Neuropsychology, 20(3), 301-319. https://doi.org/10.1016/j.acn.2004.08.001

16.

Bouman

Hendriks

M. P. H.

Schmand

B. A.

Kessels

R. P. C.

Aldenkamp

A. P.

(2016). Indicators of suboptimal performance embedded in the Wechsler Memory Scale–Fourth Edition (WMS-IV). Journal of Clinical and Experimental Neuropsychology, 38(4), 455-466. https://doi.org/10.1080/13803395.2015.1123226

17.

Bush

S. S.

Ruff

R. M.

Troster

A. I.

Barth

J. T.

Koffler

S. P.

Pliskin

N. H.

Reynolds

C. R.

Silver

C. H.

(2005). Symptom validity assessment: Practice issues and medical necessity. Archives of Clinical Neuropsychology, 20(4), 419-426. https://doi.org/10.1016/j.acn.2005.02.002

18.

Critchfield

E. A.

Soble

J. R.

Marceaux

J. C.

Bain

K. M.

Bailey

K. C.

Webber

T. A.

Alverson

W. A.

Messerly

González

D. A.

O’Rourke

J. J. F.

(2019). Cognitive impairment does not cause performance validity failure: Analyzing performance patterns among unimpaired, impaired, and noncredible participants across six tests. The Clinical Neuropsychologist, 33(6), 1083-1101. https://doi.org/10.1080/13854046.2018.1508615

19.

Dean

A. C.

Victor

T. L.

Boone

K. B.

Philpott

L. M.

Hess

R. A.

(2009). Dementia and effort test performance. The Clinical Neuropsychologist, 23(1), 133-152. https://doi.org/10.1080/13854040701819050

20.

Denning

J. H.

(2012). The efficiency and accuracy of the Test of Memory Malingering Trial 1, errors on the first 10 items of the Test of Memory Malingering, and five embedded measures in predicting invalid test performance. Archives of Clinical Neuropsychology, 27(4), 417-432. https://doi.org/10.1093/arclin/acs044

21.

Erdodi

L. A.

Roth

R. M.

Kirsch

N. L.

Lajiness-O’Neill

Medoff

(2014). Aggregating validity indicators embedded in Conners’ CPT-II outperforms individual cutoffs at separating valid from invalid performance in adults with traumatic brain injury. Archives of Clinical Neuropsychology, 29(5), 456-466. https://doi.org/10.1093/arclin/acu026

22.

Glickman

M. E.

Rao

S. R.

Schultz

M. R.

(2014). False discovery rate control is a recommended alternative to Bonferroni-type adjustments in health studies. Journal of Clinical Epidemiology, 67(8), 850-857. https://doi.org/10.1016/j.jclinepi.2014.03.012

23.

Green

(2004). Green’s Medical Symptom Validity Test (MSVT) for Microsoft Windows: User’s manual. Green’s Publishing.

24.

Heilbronner

R. L.

Sweet

J. J.

Morgan

J. E.

Larrabee

Millis

S. R.

(2009). American Academy of Clinical Neuropsychology consensus conference statement on the neuropsychological assessment of effort, response bias, and malingering. The Clinical Neuropsychologist, 23(7), 1093-1129. https://doi.org/10.1080/13854040903155063

25.

Horner

M. D.

VanKirk

K. K.

Dismuke

C. E.

Turner

T. H.

Muzzy

(2014). Inadequate effort on neuropsychological evaluation is associated with increased healthcare utilization. The Clinical Neuropsychologist, 28(5), 703-713. https://doi.org/10.1080/13854046.2014.925143

26.

Larrabee

G. J.

(2008). Aggregation across multiple indicators improves the detection of malingering: Relationship to likelihood ratios. The Clinical Neuropsychologist, 22(4), 666-679. https://doi.org/10.1080/13854040701494987

27.

Larrabee

G. J.

(2014). False-positive rates associated with the use of multiple performance and symptom validity tests, Archives of Clinical Neuropsychology, 29(4), 364-373. https://doi.org/10.1093/arclin/acu019

28.

Lippa

S. M.

(2018). Performance validity testing in neuropsychology: A clinical guide, critical review, and update on a rapidly evolving literature. The Clinical Neuropsychologist, 32(3), 391-421. https://doi.org/10.1080/13854046.2017.1406146

29.

Loring

D. W.

Goldstein

F. C.

Chen

Drane

D. L.

Lah

J. J.

Zhao

Larrabee

G. J.

(2016). False-positive error rates for Reliable Digit Span and Auditory Verbal Learning Test performance validity measures in amnestic mild cognitive impairment and early Alzheimer disease. Archives of Clinical Neuropsychology, 31(4), 313-331. https://doi.org/10.1093/arclin/acw014

30.

Meyers

J. E.

Miller

R. M.

Thompson

L. M.

Scalese

A. M.

Allred

B. C.

Rupp

Z. W.

Dupaix

Z. P.

Junghyun Lee

(2014). Using likelihood ratios to detect invalid performance with performance validity measures. Archives of Clinical Neuropsychology, 29(3), 224-235. https://doi.org/10.1093/arclin/acu001

31.

Meyers

J. E.

Volbrecht

Axelrod

B. N.

Reinsch-Boothby

(2011). Embedded symptom validity tests and overall neuropsychological test performance. Archives of Clinical Neuropsychology, 26(1), 8-15.

32.

Miller

J. B.

Millis

S. R.

Rapport

L. J.

Bashem

J. R.

Hanks

R. A.

Axelrod

B. N.

(2011). Detection of insufficient effort using the Advanced Clinical Solutions for the Wechsler Memory Scale, Fourth Edition. The Clinical Neuropsychologist, 25(1), 160-172. https://doi.org/10.1080/13854046.2010.533197

33.

Olsen

D. H.

Schroeder

R. W.

Heinrichs

R. J.

Martin

P. K.

(2019). Examination of optimal embedded PVTs within the BVMT-R in an outpatient clinical sample. The Clinical Neuropsychologist, 33(4), 732-742. https://doi.org/10.1080/13854046.2018.1501096

34.

Pearson

(2009). Advanced Clinical Solutions for WAIS-IV and WMS-IV: Administration and scoring manual. Psychological Corporation.

35.

Resch

Z. J.

Pham

A. T.

Abramson

D. A.

White

D. J.

DeDios-Stern

Ovsiew

G. P.

Castillo

L. R.

Soble

J. R.

(2020). Examining independent and combined accuracy of embedded performance validity tests in the California Verbal Learning Test-II and Brief Visuospatial Memory Test-Revised for detecting invalid performance. Applied Neuropsychology: Adult. Advanced online publication. https://doi.org/10.1080/23279095.2020.1742718

36.

Rey

(1941). L’examen psychologique dans les cas d’ence’phalopathie traumatique: Les problems [The psychological examination in cases of traumatic encephalopathy: Problems]. Archives de Psychologie, 28, 215-285.

37.

Rickards

T. A.

Cranston

C. C.

Tourajdi

Bechtold

K. T.

(2018). Embedded performance validity testing in neuropsychological assessment: Potential clinical tools. Applied Neuropsychology: Adult, 25(3), 219-230. https://doi.org/10.1080/23279095.2017.1278602

38.

Sawyer

R. J.

Testa

S. M.

Dux

(2017). Embedded performance validity tests within the Hopkins Verbal Learning Test–Revised and the Brief Visuospatial Memory Test–Revised. The Clinical Neuropsychologist, 31(1), 207-218. https://doi.org/10.1080/13854046.2016.1245787

39.

Schmidt

(1996). Rey Auditory Verbal Learning Test: A handbook. Western Psychological Services.

40.

Schroeder

R. W.

Martin

P. K.

Heinrichs

R. J.

Baade

L. E.

(2019). Research methods in performance validity testing studies: Criterion grouping approach impacts study outcomes. The Clinical Neuropsychologist, 33(3), 466-477. https://doi.org/10.1080/13854046.2018.1484517

41.

Schutte

Axelrod

B. N.

(2013). Use of embedded cognitive symptom validity measures in mild traumatic brain injury cases. In Carone

D. A.

Bush

S. S.

(Eds.), Mild traumatic brain injury: Symptom validity assessment and malingering (pp. 159-181). Springer.

42.

Schutte

Millis

Axelrod

VanDyke

(2011). Derivation of a composite measure of embedded symptom validity indices. The Clinical Neuropsychologist, 25(3), 454-462. https://doi.org/10.1080/13854046.2010.550635

43.

Soble

J. R.

Bain

K. M.

Bailey

K. C.

Kirton

J. W.

Marceaux

J. M.

Critchfield

E. A.

McCoy

K. J. M.

O’Rourke

J. J. F.

(2019). Evaluating the accuracy of the Wechsler Memory Scale–Fourth edition (WMS-IV) Logical Memory Embedded Validity Index for detecting invalid test performance. Applied Neuropsychology: Adult, 26(4), 311-318. https://doi.org/10.1080/23279095.2017.1418744

44.

Soble

J. R.

Critchfield

E. C.

O’Rourke

J. J. F.

(2017). Neuropsychological evaluation in TBI [Special issue]. Physical Medicine and Rehabilitation Clinics of North America, 28(2), 339-350. https://doi.org/10.1016/j.pmr.2016.12.009

45.

Strauss

Sherman

E. M. S.

Spreen

(2006). A compendium of neuropsychological tests: Administration, norms, and commentary (3rd ed.). Oxford University Press.

46.

Swets

J. A.

(1988). Measuring the accuracy of diagnostic systems. Science, 240(4857), 1285-1293. https://doi.org/10.1126/science.3287615

47.

Tombaugh

T. N.

(1996). Test of Memory Malingering (TOMM). North Tonawanda, NY: Multi Health Systems.

48.

Webber

T. A.

Critchfield

E. A.

Soble

J. R.

(2018). Convergent, discriminant, and concurrent validity of nonmemory-based performance validity tests. Assessment. Advance online publication. https://doi.org/10.1177/1073191118804874

49.

Webber

T. A.

Soble

J. R.

(2018). Utility of various WAIS-IV Digit Span indices for identifying noncredible performance validity among cognitively impaired and unimpaired examinees. The Clinical Neuropsychologist, 32(4), 657-670. https://doi.org/10.1080/13854046.2017.1415374

50.

Wechsler

(2009). Administration and scoring manual for the Wechsler Memory Scale–fourth edition. Pearson.

51.

Whitney

K. A.

Davis

J. J.

(2015). The non-credible score of the Rey Auditory Verbal Learning Test: Is it better at predicting non-credible neuropsychological test performance than the RAVLT recognition score? Archives of Clinical Neuropsychology, 30(2), 130-138. https://doi.org/10.1093/arclin/acu094

52.

Williams

J. M.

(1998). The malingering of memory disorder. In Reynolds

C. R.

(Ed.), Detection of malingering during head injury litigation (pp. 105-132). Plenum Press. https://doi.org/10.1007/978-1-4419-7469-3_4