Reliable Digit Span

Abstract

Reliable Digit Span (RDS) is a heavily researched symptom validity test with a recent literature review yielding more than 20 studies ranging in dates from 1994 to 2011. Unfortunately, limitations within some of the research minimize clinical generalizability. This systematic review and cross-validation study was conducted to address these limitations, thus increasing the measure’s clinical utility. Sensitivity and specificity rates were calculated for the ≤6 and ≤7 cutoffs when data were globally combined and divided by clinical groups. The cross-validation of specific diagnostic groups was consistent with the data reported in the literature. Overall, caution should be used when utilizing the ≤7 cutoff in all clinical groups and when utilizing the ≤6 cutoff in the following groups: cerebrovascular accident, severe memory disorders, mental retardation, borderline intellectual functioning, and English as a second language. Additional limitations and cautions are provided.

Keywords

Reliable Digit Span (RDS)review analysis symptom validity malingering

During the past 10 to 15 years, research on the topic of neurocognitive symptom validity testing (SVT) has vastly expanded (Larrabee, 2003; Whitney, Davis, Shepard, Bertram, & Adams, 2009). This expansion is likely due, in part, to neuropsychologists’ growing awareness of three findings. First, neuropsychologists are unable to reliably discriminate between individuals who are providing suspect effort and individuals who are providing adequate effort based solely on general neuropsychological test data (Heaton, Smith, Lehman, & Vogt, 1978), self-reported symptoms (Lees-Haley & Brown, 1993), or clinical judgment (Millis & Putnam, 1996). Second, patient effort on testing accounts for a substantial amount of variance in neuropsychological test performance, with some studies indicating that it might account for as much as 50% of the variance in forensic settings (Constantinou, Bauer, Ashendorf, Fisher, & McCaffrey, 2005; Green, Rohling, Lees-Haley, & Allen, 2001). Third, base rates of malingering during neuropsychological evaluations are substantially higher than previously believed, even in nonforensic settings (Boone, 2007).

Initially, SVTs were used primarily in medicolegal settings, as base rates of malingering are particularly high in these settings (Larrabee, 2003). Additionally, in these setting, it is crucial that the validity of the neuropsychological test results be empirically determined, as the test results often help determine severity of cognitive impairments, which could have a large impact on whether patients receive substantial external incentives. More recently, clinicians and professional organizations have urged the use of SVTs in nonforensically based evaluations as well (American Academy of Clinical Neuropsychology, 2007; Bush et al., 2005). In fact, the American Academy of Clinical Neuropsychology recently sponsored a consensus conference on neuropsychological SVT and the expert neuropsychologists opined that, “The assessment of effort and genuine reporting of symptoms is important in all evaluations” to help ensure that the obtained neuropsychological test results are a valid representation of the patient’s current cognitive abilities (Heilbronner et al., 2009, p. 1121).

With the push to include SVTs in all neuropsychological evaluations, clinicians must be aware of the major findings and trends in SVT research. Research published by Larrabee (2008) and by Victor, Boone, Serpa, Buehler, and Ziegler (2009) has provided a model for SVT usage that has resulted in impressive hit rates. Both studies employed multiple SVTs throughout the evaluations and both studies found that failure of any two SVTs resulted in a sensitivity rate above 80% and a specificity rate above 90%. It is important to note that clinicians should choose SVTs that have sensitivity rates as high as possible while ensuring that specificity rates be maintained at or above 90% to retain the high hit rates published in the literature (Boone, 2007).

Of the multiple SVTs that clinicians might choose from, one of the oldest and most heavily researched is Reliable Digit Span (RDS; Boone, 2007). Greiffenstein, Baker, and Gola (1994) originally derived RDS from the Digit Span subtest of the Wechsler Adult Intelligence Scale–Revised (Wechsler, 1981). They calculated the measure by “summing the longest string of digits repeated without error over two trials under both forward and backward conditions” (Greiffenstein et al., 1994, pp. 219-220).

A recent review of the literature yielded more than 20 studies on RDS, with one study being a meta-analytic review of the measure (Jasinski, Berry, Shandera, & Clark, 2011). The meta-analysis indicates that there are strong effect sizes across Wechsler Adult Intelligence Scale (WAIS) test versions (e.g., WAIS-R vs. WAIS-III). Additionally, the meta-analysis indicates that RDS effectively discriminates between individuals providing credible effort and individuals providing suspect effort (average weighted effect size of 1.34). This information is crucial, as it indicates that the measure is valid and effective despite updated WAIS test versions. The meta-analysis provides useful information and it has many strengths; however, it is not without limitations.

The first limitation of the meta-analysis is that it does not report sensitivity and specificity rates for multiple cutoff scores; instead, it shows sensitivity and specificity rates for a cutoff score of 7.1. Second, the meta-analysis does not report sensitivity and specificity rates for different clinical groups; rather, it reports global sensitivity and specificity rates based on nine published studies. Third, the meta-analysis does not report information on many clinical strengths and weaknesses of the measure (e.g., its use with patients who have low IQs or patients who speak English as a second language), as it would be challenging to quantify these factors in a meta-analysis given the available published data. Consequently, we must look to other sources to provide this clinically applicable information. It was with these limitations in mind that this systematic review and cross-validation study was structured. This article will report (a) data on individual studies, (b) sensitivity and specificity rates for various cutoff scores with multiple clinical groups, and (c) limitations that clinicians are likely to face when using RDS. This systematic review will add to the body of published knowledge by synthesizing and reporting practical data on the clinical utility of RDS.

Method

Study Selection and Inclusion Criteria

Different strategies may be used to select studies for inclusion in a review article (Demakis, 2006). Some researchers adopt a stance whereby all relevant studies are included whether they are published or not. This approach is generally used to avoid the documented phenomenon termed publication bias, in which research with significant findings is more likely to be published than research with nonsignificant findings (Begg, 1994). However, it is noted that unpublished data may be of lower quality than published data, and some efforts are needed to ensure that only high-quality data are used (Martin, Pérez, Sacristán, & Álvarez, 2005). To remedy this conundrum, this systematic review uses refereed published professional articles and only relevant data from additional sources that are believed to be of high quality (i.e., scholarly dissertations and data published in books written or edited by leaders in the field of neuropsychological symptom validity testing).

A variety of techniques were used to gather data for this study. First, online databases were searched using EBSCOhost search engines. These engines searched PsycARTICLES, PsychEXTRA, PsychINFO, Psychology and Behavioral Sciences Collection, Medline, and other electronic databases. The keywords entered in the search engines included “Reliable Digit Span,” “Reliable Digits,” “RDS,” “Digit Span and malingering,” “Digit Span and effort,” “Digit Span and response bias,” “WAIS and malingering,” “WAIS and effort,” and “WAIS and response bias.” After all the articles were collected, the reference sections of the articles were searched for additional studies that may have been missed in the original search.

Each study identified as potentially containing data relevant to this systematic review was examined. When a study failed to provide enough data for inclusion in this article, the authors attempted to contact the primary author of that study. Primary authors were also contacted for studies that included data for one RDS cutoff but not other RDS cutoffs. The authors were asked for additional data related to their studies, and some of the authors were also asked if they had access to other RDS data, such as scholarly dissertations, that they would be willing to share for this systematic review.

After all potential RDS studies were obtained (n = 28), the inclusion criteria were narrowed to determine which studies would be included in this systematic review. The first criterion was that the studies had to use RDS cutoffs of ≤7 and/or ≤ 6. This criterion was established because nearly all studies used one or both of these cutoff scores, whereas other cutoff scores were rarely published. Additionally, the available literature suggests that these two cutoff scores resulted in sensitivity and/or specificity rates comparable with other embedded SVTs. The second criterion was that the studies had to provide data allowing for the calculation of sensitivity and/or specificity rates for the RDS cutoffs. The necessary data includes (a) the total number of subjects and the sensitivity and/or specificity rates or (b) the total number of subjects, the number of subjects correctly identified as providing adequate effort via RDS, and/or the number of subjects correctly identified as providing suspect effort via RDS. The third criterion for inclusion was that the studies had to classify participants using (a) participants who had no incentive to feign deficits (for specificity rates only), (b) simulated malingerers (for sensitivity rates only), or (c) neurocognitive validity criteria based on the method proposed by Slick, Sherman, and Iverson (1999; for sensitivity and/or specificity rates). In general, the Slick et al. criteria require the presence of an external incentive and failure of an SVT at worse than chance rates or failure of two or more SVTs not at worse than chance rates (see Slick et al., 1999 for more detailed information). This third criterion was to ensure that the data included in this systematic review met current standards in identifying suspect effort. The fourth criterion was that the studies could not reuse previously reported data. This final criterion was included to eliminate the possibility of redundant data in the systematic review. When redundant data were identified in studies, the redundant data were excluded, whereas the nonredundant data were included. When this strategy was not possible (e.g., studies that mixed both previously published and newly published data into one group), the most complete data set was included in this systematic review.

The literature that met the aforementioned inclusion criteria included 21 refereed professional articles, 1 book, and 1 scholarly dissertation. Five refereed professional articles were excluded because they did not meet the inclusion criteria. A listing of these articles can be found in the appendix of this analysis.

Data Analysis From the Literature Review

After the appropriate literature was identified, data for calculation of sensitivity and specificity rates were pooled for all RDS studies. The pooled data was also divided into patient diagnostic groups. Most of the diagnostic groups yielded specificity rates of 90% or higher for at least one of the two frequently cited cutoff scores. The few diagnostic groups that consistently produced specificity rates below 90% for both cutoff scores were identified as “special groups” (i.e., cerebrovascular accident, severe memory disorders, mental retardation, borderline IQ and below, English as a second language, and patients of Hispanic background). These “special groups” were observed to be outliers in the current analysis and previous research has noted that most of these groups perform poorly on Digit Span or similar span-based tests despite the belief that adequate effort was provided (Boone, 2007; Lezak, Howieson, & Loring, 2004; Schroeder & Marshall, 2010). Consequently, these groups were analyzed separately from the remaining data. The remaining data were divided into the following diagnostic classes: nonclinical participants such as controls, volunteers, and simulators; mixed clinical non–traumatic brain injury (non-TBI) patients; and traumatic brain injury (TBI) patients. Finally, the TBI group was further divided into the following groups: (a) postconcussive and mild TBI and (b) moderate to severe TBI. This final division was performed because TBI patients are the most likely patients to be seen in forensic neuropsychological evaluations, and it is important to determine if severity of TBI will impact sensitivity and/or specificity rates on RDS.

The sensitivity and specificity rates were calculated for the pooled data using the empirical Bayesian analysis methods. The nonlinear mixed (NLMIXED) procedure of SAS System for Windows (Version 9.2) was used in the analysis. The Bayesian analysis method was used because it accounts for the inverse association between sensitivity and specificity rates, which can potentially underestimate the true values of the rates when data are pooled in studies such as this (Irwig, Macaskill, Glasziou, & Fahey, 1995). Additionally, the Bayesian method controls for extraneous variables such as between-study correlations and between-study heterogeneity (Macaskill, 2004), which is likely to occur in large pooled data sets. Unfortunately, the Bayesian method has a notable limitation as well. The drawback is that the number of studies included in the calculation is limited because of inclusion criteria that are inherent to the method.

In addition to the Bayesian calculations, weighted mean sensitivity and specificity rates were conducted for the pooled data. The weighted mean sensitivity and specificity rates have the advantage of including all studies into the calculations; however, they have disadvantages in that they do not control for extraneous variables and they are likely to result in an underestimate of true sensitivity and specificity rates. Because of these limitations, 95% confidence intervals were added to the calculations.

Data Analysis From a Personal Database

In addition to the data reported in the literature, a personal database was also examined for this study. The personal database provided additional and comparative information about the “special groups” because minimal information regarding these groups was available in the literature. The data from the personal database is not reported with the data from the literature, as the current results from the personal database have not been previously published. Instead, the personal database was used as a cross-validation data set for the previously identified “special groups.”

The personal database included 364 inpatients and 443 outpatients referred for neuropsychological evaluations between 2007 and 2009. All patients were referred to a board-certified neuropsychologist practicing in the Department of Psychiatry at Hennepin County Medical Center. An approval from the institutional review board was obtained from Hennepin County Medical Center to use the data in this study.

All patients in the database completed comprehensive neuropsychological evaluations. During the evaluations, multiple SVTs were interspersed throughout the test batteries. The SVTs included in most test batteries were: RDS (Greiffenstein et al., 1994), Sentence Repetition raw score (Schroeder & Marshall, 2010), CVLT-II Forced Choice Recognition Test errors (Root, Robbins, Chang, & van Gorp, 2006), Logical Memory Rarely Missed Item Index (Killgore & DellaPietra, 2000), Rey Complex Figure recognition scores (Lu, Boone, Cozolino, & Mitchell, 2003), the average dominant hand Finger Tapping score (Arnold et al., 2005), and the Dot Counting Test (Boone, Lu, & Herzberg, 2002). Only patients who were identified as providing a credible effort—not failing Slick et al. (1999) criteria—were included in this study, as the purpose of the database was to cross-validate specificity rates in the “special groups.”

Results

Table 1 shows the specificity rates for each study included in this systematic review. Table 2 provides the pooled specificity rates when using the Bayesian method and the weighted average method. Specificity rates calculated by the Bayesian method were included only for the “all subjects combined—excluding special groups” group. When this group was divided into subgroups, the subgroups contained smaller sample sizes and nonconvergence of the Newton–Raphson maximization algorithm rendered the Bayesian calculations unreliable. Consequently, the specificity rates for these subgroups were calculated using only weighted averages. Because no limitations are placed on which studies can be included when using weighted averages, opposed to when using the Bayesian method, this calculation allowed the sample sizes to be maximized in each subgroup.

Table 1.

Review of Specificity Rates for Reliable Digit Span Cutoff Scores of ≤7 and ≤6

Study	Subjects	Ratio of true negatives using ≤7	Specificity (%)	Ratio of true negatives using ≤6	Specificity (%)
Greiffenstein et al. (1995)	Moderate to severe TBI	39/68	57	—	—
Greiffenstein et al. (1995)	Persistent postconcussive	46/68	68	—	—
Meyers and Volbrecht (1998)	Mild TBI nonlitigating	47/49	96	48/49	98
Strauss et al. (2000)	College control	15/21	72	—	—
Inman and Berry (2002)	College controls and TBI controls	48/48	100	48/48	100
Duncan and Ausborn (2002)	Nonneurological prison inmate	96/134	72	121/134	90
Mathias et al. (2002)	TBI mild, moderate, and severe	28/30	93	29/30	97
Strauss et al. (2002)	Controls and TBI mixed severity	—	—	38/40	95
Larrabee (2003)	Moderate to severe TBI	27/29	94	27/27	100
Etherton et al. (2005)	College control	20/20	100	20/20	100
Etherton et al. (2005)	College cold pain	20/20	100	20/20	100
Etherton et al. (2005)	Moderate to severe TBI	63/69	91	68/69	99
Heinly et al. (2005)	Mild TBI	64/77	83	72/77	93
Heinly et al. (2005)	Moderate to severe TBI	63/69	91	68/69	99
Heinly et al. (2005)	Psychiatric	92/128	72	113/128	88
Heinly et al. (2005) ^a	Cerebrovascular accident^a	290/517	56	362/517	70
Heinly et al. (2005) ^a	Severe memory disorders^a	109/228	48	155/228	68
Babikian et al. (2006)	Mixed clinical patients & controls	68/88	77	82/88	93
Schwarz et al. (2006)	Undergrad controls	—	—	22/22	100
Axelrod et al. (2006)	TBI mixed severities	22/29	76	27/29	93
Greve et al. (2007)	Toxic exposure	34/38	89	37/38	97
Marshall & Happe (2007) ^a	Mental retardation^a	8/71	11	22/71	31
Graue et al. (2007) ^a	Mental retardation^a	—	—	4/26	15
Dean et al. (2008)	Mixed clinical IQs 80 to ≥120	45/56	80	56/56	100
Dean et al. (2008) ^a	Mixed clinical IQs 50-79^a	22/47	47	31/47	66
Whitney et al. (2009)	Mixed clinical	22/26	85	24/26	92
Greve et al. (2010)	Chronic pain	150/176	85	174/176	99
Marshall et al. (2010)	Adult attention deficit disorder	62/66	94	64/66	97
Schroeder (2010)	Adult attention deficit disorder	75/88	85	83/88	94
Schroeder (2010)	Outpatient psychological disorders	49/49	100	49/49	100
Schroeder & Marshall (2011)	Psychotic disorders	85/103	83	99/103	96
Schroeder & Marshall (2011)	Non-psychotic disorders	138/177	78	170/177	96
Salazar et al. (2007) ^a	ESL outpatients^a	13/25	52	21/25	84
Salazar et al. (2007)	Caucasian outpatients	63/76	83	76/76	100
Salazar et al. (2007)	African American outpatients	24/30	80	28/30	93
Salazar et al. (2007) ^a	Hispanic outpatients^a	12/27	44	22/27	81
Salazar et al. (2007)	Asian outpatients	14/16	88	15/16	94

Note. ESL = English as a second language, TBI = traumatic brain injury.

indicates a special group.

Table 2.

Weighted Mean and Bayesian Calculated Specificity Rates for Reliable Digit Span Cutoff Scores of ≤7 and ≤6

Studies	Subjects	Total N ≤ 7	Specificity in % [95% CI]	Total N ≤ 6	Specificity in % [95% CI]
Multiple studies (wm)	All except special groups	1,848	82 [78%, 86%]	1,751	96 [95%, 97%]
Multiple studies (bm)	All except special groups	1,012	85 [79%, 91%]	953	97 [95%, 99%]
Multiple studies (wm)	Controls only	61	90 [50%, 100%]	62	100 [100%]
Multiple studies (wm)	Mixed clinical only	1,163	82 [78%, 86%]	1,163	95 [93%, 97%]
Multiple studies (wm)	All TBI patients	488	82 [72%, 92%]	350	97 [94%, 100%]
Multiple studies (wm)	PC and Mild TBI	194	81 [47%, 100%]	126	95 [64%, 100%]
Multiple studies (wm)	Mod/Sev TBI	235	82 [55%, 100%]	165	99 [98%, 100%]
Heinly et al. (2005)	Cerebrovascular accident	517	56	517	70
Heinly et al. (2005)	Memory disorders	228	48	228	68
Multiple studies	Mental retardation	71	11	97	27
Dean et al. (2008)	IQ 50-59	—	—	3	33
Dean et al. (2008)	IQ 60-69	—	—	12	33
Dean et al. (2008)	IQ 70-79	—	—	32	81
Salazar et al. (2007)	ESL outpatients	25	52	25	84
Salazar et al. (2007)	Hispanic outpatients	27	44	27	81

Note. CI = confidence interval; TBI = traumatic brain injury. Confidence limits were not reported when the pooled data were composed of two or fewer studies: wm = weighted means; bm = Bayesian method; PC = postconcussive; Mod/Sev = Moderate to Severe; ESL = English as a second language.

As can be seen in Table 2, the RDS cutoff score of ≤7 resulted in global specificity rates of 82% and 85% when calculated using weighted averages and the Bayesian method, respectively. The RDS cutoff score of ≤6 resulted in global specificity rates of 96% and 97% when calculated using weighted averages and the Bayesian method, respectively. When the subject pool was divided further, specificity rates (calculated via weighted averages) for the ≤7 cutoff score were below 90% for all subgroups except the control subjects group, whereas the ≤6 cutoff score yielded specificity rates above 90% for all the groups except the “special groups.”

Table 3 shows the sensitivity rates for each study included in this systematic review. Table 4 shows the sensitivity rates when the data are pooled and divided by diagnostic groups. Global sensitivity rates for the ≤7 cutoff equaled 48% and 58% when using weighted averages and the Bayesian method, respectfully. Global sensitivity rates for the ≤6 cutoff equaled 30% and 35% when using weighted averages and the Bayesian method, respectfully. In the clinical subgroups, the ≤7 cutoff resulted in sensitivity rates ranging from 42% to 72%, whereas the ≤6 cutoff resulted in rates ranging from 26% to 38%.

Table 3.

Sensitivity Rates for Reliable Digit Span Cutoff Scores of ≤ 7 and ≤ 6

Study	Subjects	Ratio of true positives using ≤7	Sensitivity (%)	Ratio of true positives using ≤6	Sensitivity (%)
Greiffenstein et al. (1995)	Postconcussive PM	47/53	89	—	—
Strauss et al. (2000)	College simulator malingerers	14/20	71	—	—
Inman and Berry (2002)	College simulator malingerers	12/44	27	7/44	16
Duncan and Ausborn (2002)	Prison malingerers	37/54	68	31/54	57
Mathias et al. (2002)	Mixed TBI PM	16/24	67	9/24	38
Strauss et al. (2002)	Simulated malingerers	—	—	16/34	47
Larrabee (2003)	Mixed TBI DMND	13/24	50	6/24	23
Etherton et al. (2005)	College simulator malingerers	13/20	65	8/20	40
Etherton et al. (2005)	Definite MND pain patient	21/35	60	13/35	37
Heinly et al. (2005)	Mild TBI malingerers	34/48	71	22/48	46
Heinly et al. (2005)	Mod/Sev TBI PM	12/23	52	6/23	26
Babikian et al. (2006)	Mixed clinical malingerers	41/66	62	30/66	45
Schwarz et al. (2006)	Symptom coached simulators	—	—	9/21	43
Schwarz et al. (2006)	Test-coached simulators	—	—	9/21	43
Axelrod et al. (2006)	Mild TBI malingerers	23/36	64	14/36	39
Greve et al. (2007)	Toxic clinical malingerers	25/46	54	21/46	46
Graue et al. (2007)	Simulated mental retardation	—	—	14/25	56
Whitney et al. (2009)	Mixed PM	8/20	40	5/20	25
Ylioja et al. (2009)	Postconcussive PM	16/33	48	9/33	27
Greve et al. (2010)	Chronic pain malingerers	90/185	49	45/185	24
Marshall et al. (2010)	Adult ADHD evaluation PM	59/267	22	37/267	14

Note. TBI = traumatic brain injury; MND = malingered neurocognitive dysfunction; ADHD = attention deficit/hyperactivity disorder; PM = Probable Malingerers; DMND = Definite Malingering of Neuropsychological Dysfunction; Mod/Sev = Moderate to Severe.

Table 4.

Weighted Mean Sensitivity Rates for All Studies With Reliable Digit Span Cutoff Scores of ≤7 and ≤6

Studies	Subjects	Total N for ≤7	Mean sensitivity in % [95% CI]	Total N for ≤6	Mean sensitivity in % [95% CI]
All studies (weighted)	All studies	998	48 [39%, 57%]	1026	30 [24%, 36%]
All studies (Bayesian method)	All studies	965	58 [48%, 68%]	1026	35 [28%, 42%]
Multiple studies (weighted)	Simulators	84	46 [0%, 100%]	165	38 [24%, 52%]
Multiple studies (weighted)	Mixed clinical (except TBI)	673	42 [27%, 57%]	673	27 [16%, 38%]
Multiple studies (weighted)	All TBI patient malingerers	241	67 [54%, 80%]	188	35 [25%, 45%]
Multiple studies (weighted)	PC and mild TBI PM	134	72 [18%, 100%]	81	38 [0%, 100%]
Multiple studies (weighted)	Mod/Sev TBI PM	23	52 [—]	23	26 [—]

Note. CI = confidence interval; TBI = traumatic brain injury. No confidence intervals could be calculated for the moderate/severe TBI group; weighted = weighted means; PC = postconcussive; PM = probable malingerers; Mod/Sev = Moderate to Severe.

Table 5 presents the specificity rates for varying clinical groups from the personal database. As can be seen, the patients with severe memory disorders, patients who speak English as a second language, patients of Native American and Hispanic backgrounds, and patients with IQ scores in the 70s and below do not produce specificity rates above 90% when using either RDS cutoff score. The <12 years, <11 years, and <10 years of education groups all produced specificity rates near 90% when using the ≤6 cutoff score.

Table 5.

Specificity Rates for Reliable Digit Span Cutoff Scores of ≤7 and ≤6 From Personal Database

Study	Subjects	Ratio of true negatives using ≤7	Specificity (%)	Ratio of true negatives using ≤6	Specificity (%)
HCMC database^a	Severe memory disorder^a	7/19	37	11/19	58
HCMC database^a	ESL patients^a	7/22	32	16/22	73
HCMC database	Caucasian psychiatric patients	334/434	77	401/434	92
HCMC database	AA psychiatric patients	102/149	68	139/149	92
HCMC database	Asian psychiatric patients	11/13	85	13/13	100
HCMC database^a	NA psychiatric patients^a	13/20	65	17/20	85
HCMC database^a	Hispanic psychiatric patients^a	9/16	56	11/16	69
HCMC database	< 12 years education (M = 10.0)	102/156	65	15/156	90
HCMC database	< 11 years education (M = 8.9)	41/74	55	65/74	88
HCMC database	< 10 years education (M = 7.7)	21/35	60	31/35	89
HCMC database	FSIQ 80+ (M = 98.9)	354/424	83	410/424	97
HCMC database	FSIQ 80s (M = 84.9)	83/116	72	110/116	95
HCMC database^a	FSIQ 70s (M = 74.9)^a	57/95	60	81/95	85
HCMC database^a	FSIQ 60s (M = 65.2)^a	15/39	38	28/39	72

Note. ^aindicates a special group, HCMC = Hennepin County Medical Center; FSIQ = full scale intelligence quotient, ESL = English as a second language, AA = African American, NA = Native American.

Discussion

RDS is one of the most heavily researched SVTs available to neuropsychologists. In fact, a recent literature review yielded more than 20 studies, ranging in dates from 1994 to 2011, which evaluated the clinical utility of RDS. Unfortunately, issues within some of the studies limit their clinical generalizability. As a result, this systematic review and cross-validation study was conducted to address the limitations and improve clinical utility.

The results of this systematic review and cross-validation study indicate that the RDS cutoff score of ≤7 achieved a global sensitivity rate of 48% when using weighted averages and 58% when using the Bayesian method; however, this cutoff score also produced inadequate specificity rates (i.e., <90%) for the pooled data (using both statistical methods) and for all clinical subgroups (using weighted averages). It is of interest to note that the technical manual for the Advanced Clinical Solutions for the WAIS-IV and the Wechsler Memory Scale–4th edition (WMS-IV; Wechsler, 2009) indicates that an RDS cutoff score of ≤7 produced specificity rates below 90% for all clinical groups analyzed during their standardization process (their raw data was not added to the systematic review, as the data was not made available to the authors). Therefore, this systematic review and the Advanced Clinical Solutions technical manual indicate that an RDS cutoff score of ≤7 produces inadequate specificity rates in many clinical groups.

As expected, this systematic review indicates that a cutoff score of ≤6 resulted in lower sensitivity rates than the ≤7 cutoff; however, the ≤6 cutoff score achieved more appropriate specificity rates. The ≤6 cutoff score resulted in global sensitivity rates of 30% and 35% when calculated with weighted averages and the Bayesian method, respectively. Global specificity rates resulted in 96% and 97% when calculated with weighted averages and the Bayesian method, respectively. This cutoff also resulted in specificity rates above 90% for all clinical subgroups except the “special groups” (using weighted averages). Interestingly, the Advanced Clinical Solutions technical manual (Wechsler, 2009) also indicates that an RDS cutoff score of ≤6 produces specificity rates of 90% or greater in many clinical groups. These groups include the temporal lobectomy, Asperger’s disorder, attention deficit/hyperactivity disorder, major depressive disorder, and anxiety disorder groups. The Advance Clinical Solutions technical manual further indicates that an RDS cutoff score of ≤6 results in specificity rates near, but slightly lower than, 90% for individuals with autism, reading disorders, mathematics disorders, and TBIs. Although the Advanced Clinical Solutions technical manual indicates that the TBI group achieves a specificity rate slightly below 90% when using the ≤6 cutoff score, this systematic review, which includes a much larger sample size of patients with TBIs, indicates that both the postconcussive/mild TBI and moderate/severe TBI groups achieve a specificity rate greater than 90% when using this cutoff score.

In addition to the aforementioned results, this systematic review and cross-validation study also identified clinical samples in which RDS failed to produce adequate specificity rates. These samples include patients with CVAs, severe memory disorders (e.g., dementia), mental retardation, IQ scores in the borderline intellectual functioning range and below, and those who speak English as a second language. Individuals of Hispanic and Native American origin also produced specificity rates below 90% for both RDS cutoffs. It was noted, however, that individuals in these two groups had lower educational levels and IQ scores than individuals of other racial backgrounds who were included in this study. Thus, it is possible that the lower IQ scores resulted in the lower specificity rates for these two groups. Consequently, more research is needed on RDS performances by individuals of Hispanic and Native American origin. Of note, patients who were of Hispanic background in the Salazar, Lu, Wen, and Boone (2007) study were proficient in English and were tested in English. Patients who were of Hispanic background in the personal database were native Spanish speakers and were tested either directly in Spanish or through the use of an interpreter.

One might wonder if adjusted RDS cutoff scores could be used with the aforementioned “special groups” or if other SVTs should be used. Based on the current data and the available literature, adjusted RDS cutoff scores would likely be of little benefit. Many of the special groups would require substantially lower cutoff scores to maintain specificity rates of 90%. As can be seen in this study, lowering the cutoff score from ≤7 to ≤6 resulted in a large sensitivity rate decrease. Lowering the cutoff score even further would be expected to have an even more adverse effect on sensitivity rates. This is demonstrated in a study by Meyers and Volbrecht (1998), which indicated that only 10.6% of litigating patients received RDS scores of ≤5 and only 2.1% received scores of ≤4. Consequently, it is recommended that clinicians use other SVTs that maintain adequate sensitivity and specificity rates when evaluating these “special group” patients.

Although the results of this systematic review and cross-validation study indicate that an RDS cutoff score of ≤6 can be used effectively in many clinical samples, it is important to remember that no single SVT is perfect at detecting suspect effort. Obtaining a failing score on one SVT might be suggestive of the possibility of suspect effort, but it does not conclusively indicate that the patient is providing suspect effort. Conversely, a passing score on an SVT is not proof that a patient is providing credible effort. As Boone (2009) has noted, patients feign deficits in a variety of ways and some patients are more sophisticated than others. Thus, it is emphasized that multiple SVTs be interspersed throughout the evaluation to maximize the probability of making correct statements regarding the validity of the neuropsychological evaluation.

Finally, a formal diagnosis of malingering should not be based solely on SVT failures. As Slick et al. (1999) have indicated, there are many factors that go into making a formal diagnosis of malingering. First, by definition, a diagnosis of malingering requires the presence of a substantial external incentive. Second, SVT failures should not be “fully accounted for by psychiatric, neurological, or developmental factors” (Slick et al., 1999, p. 552). Third, factors other than malingering (fatigue, a desire to end testing, etc.) can cause a patient to fail two or more SVTs (Marshall et al., 2010; Schroeder & Marshall, 2010). Fourth, evidence from other sources, such as self-reports and behavioral observations, can all be suggestive of suspect effort regardless of SVT failure (Slick et al., 1999). Fifth, inconsistencies between neuropsychological test data and known patterns of brain functioning, observed behavior, reliable collateral reports, or documented background history may also suggest suspect effort. Consequently, RDS and other SVTs can aid in making a diagnosis of probable neurocognitive malingering; however, these measures are neither sufficient nor required for the diagnosis.

Footnotes

Appendix

The following articles included Reliable Digit Span but were excluded from the data analysis portion of this study because the inclusion criteria were not met or the data was used in another study.

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The authors received no financial support for the research, authorship, and/or publication of this article.

References

American Academy of Clinical Neuropsychology. (2007). American Academy of Clinical Neuropsychology (AACN) practice guidelines for neuropsychological assessment and consultation. The Clinical Neuropsychologist, 21, 209-231.

Arnold

Boone

K. B.

Dean

Wen

Nitch

McPherson

(2005). Sensitivity and specificity of finger tapping scores for the detection of suspect effort. The Clinical Neuropsychologist, 19, 105-120.

Axelrod

B. N.

Fichtenberg

N. L.

Millis

S. R.

Wertheimer

J. C.

(2006). Detecting incomplete effort with digit span from the Wechsler Adult Intelligence Scale—Third Edition. The Clinical Neuropsychologist, 20, 513-523.

Babikian

Boone

K. B.

Arnold

(2006). Sensitivity and specificity of various digit span scores in the detection of suspect effort. The Clinical Neuropsychologist, 20, 145-159.

Begg

C. B.

(1994). Publication bias. In Cooper

Hedges

L. V.

(Eds.), The handbook of research synthesis (pp. 399-409). New York, NY: Russell Sage Foundation.

Boone

K. B.

(2007). Assessment of feigned cognitive impairment: A neuropsychological perspective. New York, NY: Guilford Press.

Boone

K. B.

(2009). The need for continuous and comprehensive sampling of effort/response bias during neuropsychological examinations. The Clinical Neuropsychologist, 23, 729-741.

Boone

K. B.

Herzberg

(2002). The Dot Counting Test. Los Angeles, CA: Western Psychological Services.

Bush

Ruff

Troster

Barth

Koffler

Pliskin

. . . Silver

C. H.

(2005). Symptom validity assessment: Practice issues and medical necessity—NAN Policy and Planning Committee. Archives of Clinical Neuropsychology, 20, 419-426.

10.

Constantinou

Bauer

Ashendorf

Fisher

McCaffrey

R. J.

(2005). Is poor performance on recognition memory effort measures indicative of generalized poor performance on neuropsychological tasks? Archives of Clinical Neuropsychology, 20, 191-198.

11.

Dean

A. C.

Victor

T. L.

Boone

K. B.

Arnold

(2008). The relationship if IQ to effort test performance. The Clinical Neuropsychologist, 22, 705-722.

12.

Demakis

G. J.

(2006). Meta-analysis in neuropsychology: Basic approaches, findings, and applications. The Clinical Neuropsychologist, 20, 10-26.

13.

Duncan

S. A.

Ausborn

D. L.

(2002). The use of Reliable Digits to detect malingering in a criminal forensic pretrial population. Assessment, 9, 56-61.

14.

Etherton

J. L.

Bianchini

K. J.

Ciota

M. A.

Greve

K. W.

(2005). Reliable digit span is unaffected by laboratory-induced pain: Implications for clinical use, Assessment, 12, 101-106.

15.

Etherton

J. L.

Bianchini

K. J.

Greve

K. W.

Heinly

M. T.

(2005). Sensitivity and specificity of Reliable Digit Span in malingered pain-related disability. Assessment, 12, 130-136.

16.

Graue

L. O.

Berry

D. T. R.

Clark

J. A.

Sollman

M. J.

Cardi

Hopkins

Werline

(2007). Identification of feigned mental retardation using the new generation of malingering detection instruments: Preliminary findings. The Clinical Neuropsychologist, 21, 929-942.

17.

Green

Rohling

M. L.

Lees-Haley

P. R.

Allen

L. A.

(2001). Effort has a greater effect on test scores than severe brain injury in compensation claimants. Brain Injury, 15, 1045-1060.

18.

Greiffenstein

M. F.

Baker

W. J.

Gola

(1994). Validation of malingered amnesia measures with a large clinical sample. Psychological Assessment, 6, 218-224.

19.

Greiffenstein

M. F.

Gola

Baker

W. J.

(1995). MMPI-2 validity scales versus domain specific measures in detection of factitious traumatic brain injury. The Clinical Neuropsychologist, 9, 230-240.

20.

Greve

K. W.

Bianchini

K. J.

Etherton

J. L.

Meyers

J. E.

Curtis

K. L.

Ord

J. S.

(2010). The Reliable Digit Span Test in chronic pain: Classification Accuracy in detecting malingered pain-related disability, The Clinical Neuropsychologist, 24, 137-152.

21.

Greve

K. W.

Springer

Bianchini

K. J.

Black

F. W.

Heinly

M. T.

Love

J. M.

. . . Ciota

M. A.

(2007). Malingering in toxic exposure: Classification accuracy of reliable digit span and WAIS-III Digit Span Scaled Scores. Assessment, 14, 12-21.

22.

Heaton

R. K.

Smith

H. H.

Jr. Lehman

R. A. W.

Vogt

A. T.

(1978). Prospects for faking believable deficits on neuropsychological testing. Journal of Consulting and Clinical Psychology, 46, 892-900.

23.

Heilbronner

R. L.

Sweet

J. J.

Morgan

J. E.

Larrabee

G. J.

Millis

S. R.

, & Conference Participants. (2009). American Academy of Clinical Neuropsychology consensus conference statement on the neuropsychological assessment of effort, response bias, and malingering. The Clinical Neuropsychologist, 23, 1093-1129.

24.

Heinly

M. T.

Greve

K. W.

Bianchini

K. J.

Love

J. M.

Brennan

(2005). WAIS Digit Span-based indicators of malingered neurocognitive dysfunction: Classification accuracy in traumatic brain injury. Assessment, 12, 429-444.

25.

Inman

T. H.

Berry

D. T. R.

(2002). Cross-validation of indicators of malingering: A comparison of nine neuropsychological tests, four tests of malingering, and behavioral observations. Archives of Clinical Neuropsychology, 17, 1-23.

26.

Irwig

Macaskill

Glasziou

Fahey

(1995). Meta-analytic methods for diagnostic test accuracy. Journal of Clinical Epidemiology, 48, 119-130.

27.

Jasinski

L. J.

Berry

D. T. R.

Shandera

A. L.

Clark

J. A.

(2011). Use of the Wechsler Adult Intelligence Scale Digit Span subtest for malingering detection: A meta-analytic review. Journal of Clinical and Experimental Neuropsychology, 33, 300-314.

28.

Killgore

W. D.

DellaPietra

(2000). Using the WMS-III to detect malingering: Empirical validation of the Rarely Missed Index (RMI). Journal of Clinical and Experimental Neuropsychology, 22, 761-771.

29.

Larrabee

G. J.

(2003). Detection of malingering using atypical performance patterns on standard neuropsychological tests. The Clinical Neuropsychologist, 17, 410-425.

30.

Larrabee

G. J.

(2008). Aggregation across multiple indicators improves the detection of malingering: Relationship to likelihood ratios. The Clinical Neuropsychologist, 22, 666-679.

31.

Lees-Haley

P. R.

Brown

R. S.

(1993). Neuropsychological complaint base rates of 170 personal injury claimants. Archives of Clinical Neuropsychology, 8, 203-209.

32.

Lezak

M. D.

Howieson

D. B.

Loring

D. W.

(2004). Neuropsychological assessment (4th ed.). New York, NY: Oxford University Press.

33.

Boone

K. B.

Cozolino

Mitchell

(2003). Effectiveness of the Rey-Ostrerrieth Complex Figure Test and the Meyers and Meyers Recognition Trial in the detection of suspect effort. The Clinical Neuropsychologist, 17, 426-440.

34.

Macaskill

(2004). Empirical Bayes estimates generated in a hierarchical summary ROC analysis agreed closely with those of a full Bayesian analysis. Journal of Clinical Epidemiology, 57, 925-932.

35.

Marshall

Happe

(2007). The performance of individuals with mental retardation on cognitive tests assessing effort and motivation. The Clinical Neuropsychologist, 21, 826-840.

36.

Marshall

Schroeder

O’Brien

Fischer

Ries

Blesi

Barker

(2010). Effectiveness of symptom validity measures in identifying cognitive and behavioral symptom exaggeration in adult attention deficit hyperactivity disorder. The Clinical Neuropsychologist, 24, 1204-1237.

37.

Martin

J. L. R.

Pérez

Sacristán

Álvarez

(2005). Is grey literature essential for a better control of publication bias in psychiatry? An example from three meta-analyses of schizophrenia. European Psychiatry, 20, 550-553.

38.

Mathias

C. W.

Greve

K. W.

Bianchini

K. J.

Houston

R. J.

Crouch

J. A.

(2002). Detecting malingered neurocognitive dysfunction using the Reliable Digit Span in traumatic brain injury. Assessment, 9, 301-308.

39.

Meyers

J. E.

Volbrecht

(1998). Validation of reliable digits for detection of malingering. Assessment, 5, 303-307.

40.

Millis

S. R.

Putnam

S. H.

(1996). Detection of malingering in postconcussive syndrome. In Rizzo

Tranel

(Eds.), Head injury and postconcussive syndrome (pp. 481-498). New York, NY: Churchill Livingstone.

41.

Root

J. C.

Robbins

R. N.

Chang

van Gorp

W. G.

(2006). Detection of inadequate effort on the California Verbal Learning Test (2nd ed.): Forced choice recognition and critical item analysis. Journal of the International Neuropsychological Society, 12, 688-696.

42.

Salazar

X. F.

P. H.

Wen

Boone

K. B.

(2007). The use of effort tests in ethnic minorities and in non-English-speaking and English as a second language populations. In Boone

K. B.

(Ed.). Assessment of feigned cognitive impairment: A neuropsychological perspective (pp. 366-383). New York, NY: The Guilford Press.

43.

Schroeder

R. W.

(2010). Validation of Digit Span-based effort indices with an adult attention deficit hyperactivity disorder population and a population of individuals who report attention difficulties that are due to other psychological disorders (Unpublished doctoral dissertation). Argosy University, Eagan, MN.

44.

Schroeder

R. W.

Marshall

P. S.

(2010). Validation of the Sentence Repetition Test as a measure of suspect effort. The Clinical Neuropsychologist, 24, 326-343.

45.

Schroeder

R. W.

Marshall

P. S.

(2011). Evaluation of the appropriateness of multiple symptom validity indices in psychotic and non-psychotic psychiatric populations. The Clinical Neuropsychologist, 25, 437-453.

46.

Schwarz

L. R.

Gfeller

J. D.

Oliveri

M. V.

(2006). Detecting feigned impairment with the digit span and vocabulary subtests of the Wechsler Adult Intelligence Scale—Third Edition. The Clinical Neuropsychologist, 20, 741-753.

47.

Slick

D. J.

Sherman

E. M. S.

Iverson

G. L.

(1999). Diagnostic criteria for malingered neurocognitive dysfunction: Proposed standards for clinical practice and research. The Clinical Neuropsychologist, 4, 545-561.

48.

Strauss

Hultsch

D. F.

Hunter

Slick

D. J.

Patry

Levy-Bencheton

(2000). Using intraindividual variability to detect malingering in cognitive performance. The Clinical Neuropsychologist, 14, 420-432.

49.

Strauss

Slick

D. J.

Levy-Bencheton

Hunter

MacDonald

S. W. S.

Hultsch

D. F.

(2002). Intraindividual variability as an indicator of malingering in head injury. Archives of Clinical Neuropsychology, 17, 423-444.

50.

Victor

T. L.

Boone

K. B.

Serpa

J. G.

Buehler

M. A.

Ziegler

E. A.

(2009). Interpreting the meaning of multiple symptom validity test failure. The Clinical Neuropsychologist, 23, 297-313.

51.

Wechsler

(1981). Wechsler Adult Intelligence Scale—Revised. San Antonio, TX: The Psychological Corporation.

52.

Wechsler

(2009). Advanced Clinical Solutions for the WAIS-IV and WMS-IV. San Antonio, TX: Pearson.

53.

Whitney

K. A.

Davis

J. J.

Shepard

P. H.

Bertram

D. M.

Adams

K. M.

(2009). Digit span age scaled score in middle-aged military veterans: Is it more closely associated with TOMM failures than reliable digit span? Archives of Clinical Neuropsychology, 24, 263-272.

54.

Ylioja

S. G.

Baird

A. D.

Podell

(2009). Developing a spatial analog of the Reliable Digit Span. Archives of Clinical Neuropsychology, 24, 729-739.