Reliabilities of WMS–IV Discrepancy Score Comparisons for Subtests,Indexes,and Selected WAIS–IV Composites

Method

Procedure

Discrepancy score reliabilities were derived by the formula of Thorndike and Hagen (1969). This formula is applicable to two tests with equal SDs, for which internal consistency reliability coefficients are obtainable

r = { [ ( r 1 + r 2 ) / 2 ] − r 12 } / ( 1 − r 12 )

where r is the discrepancy score reliability, r_1, and r₂ are the split-half internal consistency reliability coefficients for tests 1 and 2, and r₁₂ is the correlation between these test scores.

To utilize the formula, internal consistency reliabilities and intercorrelational data for the composite and subtest scores were obtained from the WMS–IV Technical and Interpretive Manual (Wechsler, 2009b). Technical Manual Table 3.1 (p. 46) provides reliability coefficients for the five indexes across nine age ranges (i.e., 16–17, 18–19, 20–24, 25–29, 30–34, 35–44, 45–54, 55–64, and 65–69 years) and the average of all age ranges in the Adult standardization sample. Also in the Technical Manual Table are the reliabilities for five indexes across five age ranges (i.e., 65–69, 70–74, 75–79, 80–84, and 85–90 years) and the average of all age ranges in the Older Adult standardization group. Average reliabilities of 14 subtests in the 16–69 age range and eight subtests in the 69–90 age range were similarly obtained from Technical Manual Table 3.1. The intercorrelations of the WMS–IV subtests and indexes were taken from Technical Manual Table 4.1 (p. 57) for the total Adult standardization sample while Technical Manual Table 4.2 (p. 58) reports intercorrelations of index and subtest scores from the Older Adult normative group. Technical Manual Tables A.1 through A.14 (pp. 184–197) provided the necessary subtest and index score intercorrelations separately for each of the 14 age groups. To calculate the ability–memory discrepancy score reliabilities, the intercorrelations of the selected WAIS–IV composites (i.e., VCI, PRI, WMI, and GAI) with WMS–IV index scores were obtained from Technical Manual Table 4.12 (p. 75) for both the Adult and Older Adult batteries. The WAIS–IV Technical and Interpretive Manual (Wechsler, 2008b) provided the average reliabilities of the VCI, PRI, and WMI (Technical Manual Table 4.1, p. 42), while the internal consistency of the GAI was taken from Lichtenberger and Kaufman (2013, Table 5.1, p 163).

All calculated discrepancy score reliabilities in the present study pertain only to the Simple-Difference method (Wechsler, 2009b). Additional discrepancy score reliabilities could not be calculated because internal consistency coefficients and intercorrelations for the Predicted-Difference and Contrast Scores methods are not available. Likewise, discrepancy score reliability coefficients for the WMS–IV process subtest scores were not derived in the present analyses because internal consistency data are not accessible for these tasks.

Results

Discrepancy score reliabilities for the 14 subtests of the Adult Battery are presented in Table 1 and range from .00 to .93 (mdn =.77). Of the 91 coefficients, 35 (38.5%) were ≥ .80 and three (3.3%) were ≥ .90. Table 2 provides the discrepancy score reliabilities of the seven Older Adult Battery subtests, ranging from .25 to .94 (mdn = .79). Twelve (50%) were ≥ .80 while three (12.5%) are ≥ .90. The 10 discrepancy score reliabilities for the five primary indexes of the Adult Battery for nine age groups and the average of all groups are presented in Table 3. The reliability coefficients ranged from .46 to .94 (mdn = .76), with 49 coefficients ≥ .80 and 10 ≥ .90. Also displayed in Table 3 are the six discrepancy score reliabilities for the four primary indexes of the Older Adult Battery across five age groups and the average of all groups. The coefficients ranged from .19 to .93 (mdn = .57), with 15 (41.6%) of the 36 coefficients ≥ .80 and 10 (27.8%) ≥ .90. Table 4 shows the ability–memory discrepancy score reliabilities for the Simple-Difference method. For the Adult Battery, the coefficients range from .82 to .93 (mdn = .88), with 8 (40%) ≥ .80 and 12 (60%) ≥ .90. The Older Adult Battery ability–memory comparisons ranged from to .85 to .94 (mdn = .89), with 10 (62.5%) ≥ .80 and 6 (37.5%) ≥ .90.

OPEN IN VIEWER

Table 1.

WMS–IV Subtest Discrepancy Score Reliabilities for the Adult Battery.

Subtest	1	2	3	4	5	6	7	8	9	10	11	12	13	14
LM I
LM II	.27
VPA I	.79	.82
VPA II	.72	.74	.34
DE I	.77	.80	.84	.77
DE I Content	.73	.76	.80	.74	.12
DE I Spatial	.73	.77	.79	.73	.19	.56
DE II	.78	.80	.84	.77	.42	.44	.46
DE II Content	.74	.76	.81	.74	.42	.15	.60	.00
DE II Spatial	.72	.75	.80	.74	.61	.60	.47	.15	.58
VR I	.81	.84	.90	.83	.89	.75	.75	.82	.77	.77
VR II	.85	.87	.93	.86	.85	.80	.79	.85	.81	.79	.87
SA	.81	.85	.89	.82	.79	.75	.74	.81	.75	.76	.85	.91
SSP	.76	.80	.85	.77	.73	.69	.70	.77	.72	.73	.81	.87	.82

OPEN IN VIEWER

Table 2.

WMS–IV Subtest Discrepancy Score Reliabilities for the Older Adult Battery.

Subtest	1	2	3	4	5	6	7	8
LM I
LM II	.25
VPA I	.80	.78
VPA II	.64	.58	.53
VR I	.84	.85	.92	.87	.75
VR II	.88	.87	.94	.90	.77	.84
SSP	.75	.77	.87	.79	.63	.76	.83

Note. LM = Logical Memory; VPA = Verbal Paired Associates; VR = Visual Reproduction; SSP = Symbol Span; WMS–IV = Wechsler Memory Scale–Fourth Edition.

OPEN IN VIEWER

Table 3.

Discrepancy Score Reliabilities for WMS–IV Index Scores in the Adult and Older Adult Batteries.

Age group	AMI–VMI	AMI–VWMI	AMI–IMI	AMI–DMI	VMI–VWMI	VMI–IMI
Adult battery
16 to 17	.92	.84	.69	.70	.80	.72
18 to 19	.94	.87	.76	.77	.85	.80
20 to 24	.86	.89	.50	.64	.84	.85
25 to 29	.90	.91	.65	.65	.85	.69
30 to 34	.92	.89	.71	.61	.87	.75
35 to 44	.93	.89	.73	.72	.84	.76
45 to 54	.89	.88	.68	.61	.86	.58
55 to 64	.93	.90	.74	.72	.83	.69
65 to 69	.92	.87	.69	.65	.82	.73
Average	.91	.88	.69	.68	.85	.73
Older adult battery
65 to 69	.93		.45	.35		.85
70 to 74	.91		.39	.25		.85
75 to 79	.93		.54	.36		.69
80 to 84	.92		.45	.40		.82
85 to 90	.90		.44	.19		.83
Average	.93		.50	.35		.86

Age Group	VMI–DMI	VWMI–IMI	VWMI–DMI	IMI–DMI
Adult battery
16 to 17	.72	.76	.74	.56
18 to 19	.89	.82	.87	.61
20 to 24	.64	.85	.86	.54
25 to 29	.69	.86	.86	.50
30 to 34	.71	.83	.84	.46
35 to 44	.74	.83	.84	.50
45 to 54	.69	.78	.85	.60
55 to 64	.76	.80	.85	.70
65 to 69	.72	.76	.80	.56
Average	.71	.82	.84	.58
Older adult battery
65 to 69	.83			.46
70 to 74	.71			.42
75 to 79	.82			.56
80 to 84	.80			.54
85 to 90	.74			.57
Average	.80			.54

Note. AMI = Auditory Memory Index; VMI = Visual Memory Index; VWMI = Visual Working Memory Index; IMI = Immediate Memory Index; DMI = Delayed Memory Index; WMS–IV = Wechsler Memory Scale–Fourth Edition.

OPEN IN VIEWER

Table 4.

WAIS–IV and WMS–IV Index Discrepancy Score Reliabilities for the Adult and Older Adult Batteries.

Composite	WMS–IV composite
	AMI	VMI	VWMI	IMI	DMI
Adult battery
GAI	.91	.92	.85	.88	.89
VCI	.90	.93	.88	.90	.90
PRI	.91	.88	.82	.87	.88
WMI	.89	.91	.83	.87	.88
Older adult battery
GAI	.91	.93	–	.88	.87
VCI	.90	.94	–	.89	.88
PRI	.91	.89	–	.87	.85
WMI	.89	.91	–	.87	.85

Note. AMI = Auditory Memory Index; DMI = Delayed Memory Index; GAI = General Ability Index; IMI = Immediate Memory Index; VCI = Verbal Comprehension Index; PRI = Perceptual Reasoning Index; WMI = Working Memory Index; VMI = Visual Memory Index; VWMI =Visual Working Memory Index; WMS–IV = Wechsler Memory Scale–Fourth Edition.

Discussion

When conducting profile analyses with the WMS–IV, certain standards of discrepancy score reliability are necessary. For clinical decision making, internal consistency of at least ≥ .90 (e.g., Guilford & Fruchter, 1978; Ponterotto & Ruckdeschel, 2007) is reasonable, whereas if hypothesis development is the objective, a lesser standard (i.e., ≥ .80) is acceptable (Sattler, 2008). Reliance on discrepancy scores with reliabilities of ≤ .74 has been questioned since their clinical utility is uncertain (Drozdick et al., 2011). Therefore, serious thought is recommended before practitioners base their hypotheses or clinical decisions about individual examinees on discrepancy scores that lack a high degree of internal consistency (Charter, 2002).

There is substantial variation among the subtest discrepancy score reliability coefficients in the WMS–IV Adult Battery, with only three of 91 (3.3%) ≥ .90 and 35 of 91(38.5%) ≥ .80. The WMS–IV Technical and Interpretive Manual (Wechsler, 2009b) points out that when interpreting the Scale at the subtest level, clinicians should avoid overinterpretation of subtest differences. The reasons given for this guidance are (a) the presence of considerable normal variation in the cognitive functioning of normal individuals and (b) the fact that low scores and relatively large discrepancy scores occur frequently in the general population (Carrasco et al., 2015). The present study suggests a third reason for caution when interpreting subtest score differences, namely, that many discrepancy scores are themselves notoriously unreliable. In the present study, 56 of 91 (61.5%) subtest difference scores had reliabilities < .80. Nevertheless, Table F.4 (p. 156) in the WMS–IV Administration and Scoring Manual (Wechsler, 2009a) provides the critical values necessary for determining statistically significant differences for all possible pairs of primary subtest scores. Proper use of this resource suggests a need for practitioners to have specific interpretive hypotheses in mind prior to consulting WMS–IV Administration and Scoring Manual Table F.4 in order to reduce the probability of erroneous conclusions. For the Older Adult Battery, the number of discrepancy scores with acceptably high reliabilities is somewhat better since 12.5% (3/24) are ≥ .90 and 50% (12/24) are ≥ .80. Nevertheless, it would still be inappropriate to use Administration and Scoring Manual Table F.4 in a fishing expedition in search of statistically significant subtest differences. In general, the subtest discrepancy score reliabilities of the WMS–IV are improved over those of the WMS–III (Charter, 2002). For example, on the combined Adult and Older Batteries, 33% (38/115) of the coefficients were ≤ .74, whereas on the WMS–III, 65.4% (36/55) of the coefficients were at or below this level. Conversely, 40.9% (47/115) of WMS–IV coefficients were ≥ .80, but only 10.9% (6/56) of the WMS–III reliabilities were at or above this magnitude.

In the Adult Battery, discrepancy scores for the AMI–VMI, AMI–VWMI, and VMI–VWMI comparisons have reliabilities ≥ .80 across the nine age groups in the standardization sample. Most of these index discrepancy scores are interpretable and are likely to be useful for decision making and hypothesis development. For the most part, the difference score reliabilities for the VWMI–IMI and VWMI–DMI are ≥ .80 and suitable for generating hypotheses worthy of further investigation. The remaining index comparisons yielded scores that, with a few exceptions, are too low for serious interpretation (i.e., IMI–DMI, AMI–IMI, VMI–DMI, and AMI–DMI). Strict rules for interpreting WMS–IV discrepancy scores reliabilities do not exist, but a practical recommendation for clinical practice is to calculate reliabilities for each examinee comparable to his/her age peers. For example, the reliability of the VMI–DMI difference score is .89 for an examinee 18–19 years old, whereas the reliability is .64 if the examinee is 21 years of age. In the former instance, the discrepancy score may be used for hypothesis development and possible interpretation, whereas in the latter case, the coefficient is probably too low for either. For the Older Adult Battery, only the AMI–VMI discrepancy scores were ≥ .90 across the five age ranges and large enough to justify clinical decision making. The VMI–IMI and VMI–DMI comparisons were less impressive but nevertheless in many instances are high enough to allow for hypothesis generation. The discrepancy score reliabilities for the AMI–IMI, AMI–DMI, and IMI–DMI comparisons are so low that they should probably not be used clinically. It is typically a good idea to evaluate discrepancy score reliabilities according to the examinee’s age group, regardless of which WMS–IV battery is being utilized. Relative to the WMS–III (Charter, 2002), the index discrepancy score reliabilities on the WMS–IV have improved, as the former produced 18.7% (21/112) with coefficients ≥ .80 while the latter reported 47% (64/136) of this magnitude or better. A reduction in the number of poor reliabilities is also observed as 64.3% (72/112) of WMS–III coefficients were ≤ .74. Only 46.3% (63/136) of the WMS–IV reliabilities were at comparable levels.

For the Adult Battery, the ability–memory discrepancy score reliabilities were satisfactory, with all coefficients ≥ .82 and 40% ≥ .90. Reliabilities for the Older Adult Battery were similarly strong, with all values ≥ .85 and 37.5% (6/16) ≥ .90. Both batteries produced ability–memory discrepancy scores with sufficient reliability to justify their clinical application in terms of hypothesis development (36 instances) and decision making (14 instances). When predicting memory performance, the GAI and VCI yielded comparable reliabilities. Moreover, similarly reliable difference scores emerged regardless of which cognitive ability measures were used to predict the AMI and VMI scores. With respect to the WMS–III, ability–memory comparisons have been reported (Charter, 2002) for the eight memory indexes and the three WAIS–III IQs (i.e., Verbal, Performance, and FS). Discrepancy score reliabilities were highly similar regardless of which ability measure was used to predict memory performance. The range of WMS–III discrepancy score reliabilities across the 24 ability–memory comparisons was .71–.89 (mdn = .84), which is lower than that for the WMS–IV. On the latter Scale, the range of reliabilities across 36 ability–memory differences was .82–.94 (mdn = .89), with 38.9% (14/36) of the discrepancy score reliabilities ≥ .90. Unlike the WMS–III, all of the ability–memory comparisons provided by the WMS–IV allow for hypothesis formulation (22/36; 61.1%) or both hypothesis development and clinical decision making (38.9%).

The findings of the present study will allow practitioners to determine whether individual discrepancy scores have sufficient reliability for clinical decision making. The logical next step in the process involves computing confidence intervals (CIs) for each score and then comparing the results to an established discrepancy score standard. The recommended formulas for determining the lower and upper limits are taken from Charter (1999) and Charter and Feldt (2009)

Lower = ( X 1 − X 2 ) − Z [ SD ( 2 − r 1 − r 2 ) 1 / 2 ]

Upper = ( X 1 − X 2 ) + Z [ SD ( 2 − r 1 − r 2 ) 1 / 2 ]

Lower = ( 105 − 92 ) − 1.96 [ 15 ( 2 − .97 − .94 ) 1 / 2 ] = 4.18

Upper = ( 105 − 92 ) + 1.96 [ 15 ( 2 − .97 − .94 ) 1 / 2 ] = 17.50

The interval does not include zero and is therefore significant. However, the 13-point discrepancy is within the interval (i.e., 4.18–17.50) but is also below the selected 15-point difference needed to justify a clinical decision. Although clinical information may influence the final interpretation of the discrepancy, from a statistical perspective, additional evidence would be needed to defend clinical action as the true score difference between indexes may be as low as 4 points.

Consider a case involving the comparison of two indexes from the WMS–IV with a discrepancy score reliability of .54 (based on the average internal consistency for the Older Adult Battery). The assessment practitioner is interested in detecting possible problems with the retention of information over time. Because of the modest discrepancy score reliability and the specific hypothesis under investigation, an IMI > DMI discrepancy that occurs in < 5% of the standardization sample is selected as the minimum value required to justify clinical action. According to Table F.4 in the Administration and Scoring Manual (Wechsler, 2009a), this required difference is 14 points, which is both rare and statistically significant (Table F.3 in the Administration and Scoring Manual). The WMS–IV examination yields an IMI of 100 and a DMI of 80, the internal consistencies are .95 and .92 for the IMI and DMI, and the Z = 1.96 for the two-tailed confidence limits at the p = .05 level of significance

Lower = ( 100 − 80 ) − 1.96 [ 15 ( 2 − .95 − .92 ) 1 / 2 ] = 9.42

Upper = ( 105 − 92 ) + 1.96 [ 15 ( 2 − .97 − .94 ) 1 / 2 ] = 30.58

This is a statistically significant result because the interval is entirely above zero, including the minimum required 14-point discrepancy score, and the obtained score of 20 points exceeds the preselected required difference. In this situation, the examiner may decide that action should be taken; however, caution is warranted as the true difference may be as low as nine points. When the lower bound of the CI is above the previously chosen discrepancy score, the clinician can confidently recommend action that is statistically supported.

The present investigation demonstrated that the WMS–IV, in terms of discrepancy score reliabilities, provides a marked improvement over the WMS–III. However, the picture is incomplete because it was based exclusively on the WMS–IV Technical and Interpretive Manual (Wechsler, 2009b). Therefore, discrepancy score reliabilities were assessed only for the Simple-Difference method of comparing subtest scaled scores and selected WAIS–IV and WMS–IV indexes. Discrepancy score reliabilities for the Process and Contrast scaled scores as well as the Predicted-Difference method for evaluating ability–memory comparisons and the numerous additional scores were not examined. Because reliability coefficients for the Process, Contrast, and Predictive-Difference scores were not provided in the WMS–IV Manual, discrepancy score reliabilities could not be determined. Perhaps the publisher will correct these omissions in future revisions of the memory scale. It is also the case that the present study did not address discrepancy score reliabilities for the supplementary scores provided by the Advanced Clinical Solutions for the WAIS–IV and WMS–IV computer program (Wechsler, 2009c).