Administration and Scoring Errors of Graduate Students Learning the WISC-IV

Method

A total of 21 students were enrolled in a full-year psychological testing course at an urban university in Western Canada. However, two students did not complete all six WISC assessments as they opted to administer another cognitive test (the Stanford Binet, 5th ed.) and were excluded from the study. Of the remaining 19 students, one student was a first-year PhD student and the remaining 18 students were second-year students enrolled in a master’s-level program preparing them for professional practice in psychology. None of the students reported previous coursework or experience administering individually administered, Level C tests of intelligence. Each student completed a minimum of 6 WISC-IV assessments for a total of 114 assessments. Only the first six assessments were used in this study.

The data were collected during the course of the 2009-2010 academic year. Students attended class a total of 26 times, with each class lasting 2 hours and 50 minutes. The WISC-IV was taught over the course of three lectures and was the first intelligence test taught to students. Each student was provided with a WISC-IV kit for use during the academic year. All students were paired with a licensed psychologist in the local community who worked with the student for the entire year (primarily to assist with report writing). All supervising psychologists had at least 5 years of experience working as a registered psychologist in Alberta. Uniform instruction of the WISC was provided by the primary author to all students who met weekly for 3-hour classes. Whereas an entire year was devoted to the class, teaching of the WISC included four classes, including a brief history of the Wechsler scales, followed by detailed teaching of administration, scoring, and interpretation of the WISC-IV. Students completed two practice tests and were required to submit a sample video with an administration of the WISC-IV prior to working with clients. The graduate students’ participants were children referred by their parents/guardians to the University of Alberta Education clinic, which is an outpatient community clinic. All referral participants and parents/guardians were aware that clinicians were graduate psychology students in training and provided written informed consent.

All students were required to submit WISC-IV protocols to one of two teaching assistants (TAs), both of whom were enrolled in a PhD program in psychology. Both TAs had completed the equivalent of three courses in psychological assessment. TAs were provided with a detailed rubric that contained every possible scoring error from the WISC-IV. This included a checklist of items from the front page of the WISC-IV (for instance, ensuring the client’s age was calculated correctly, proper use of appropriate norm tables, addition of subscale scores, etc.) and then all the administration and scoring rules for each of the 10 core subtests as per the WISC-IV manual. Every test item on the Vocabulary, Similarities, and Comprehension subtests was evaluated for proper use of queries, prompts, and accuracy of assigned score (i.e., a score of 0, 1, or 2). Overall, there were 271 potential item errors reviewed for each WISC-IV protocol. Each student was given a grade for the accuracy of scoring, with each scoring error constituting a 0.5% deduction in their total score (a total score out of 100) for each report written by the student. Students were required to correct errors and resubmit them to the TAs for a second evaluation. Two practicing PhD-level psychologists (including the course instructor) separately reviewed 10% of the protocols to ensure interrater reliability. There was an agreement of 93% on the total errors noted.

Results

Results from this study indicated that graduate students committed scoring errors on the majority of test protocols (94%). This figure is similar to other recent studies (Loe et al., 2007) that found 98% of all protocols to have errors. There were a total of 511 errors with 107 of 114 protocols having errors, resulting in a mean of 4.48 errors per protocol (see Table 1). For the current study, there was a wide range of error totals per protocol, ranging from no errors to 28 errors. Not surprisingly, subtests comprising the Verbal Comprehension Index had many more errors (406) than subtests comprising the Perceptual Reasoning Index (56). The Working Memory Index subtests had more errors (43) compared with the Processing Speed Index (6).

OPEN IN VIEWER

Table 1.

Frequency of Administration and Recording Errors by Subtest on the Wechsler Intelligence Scale for Children (4th Ed.)

	Protocols with errors			Errors per protocol
	N	%	Total errors	Mean	Range
Block design	33	27.5	40	0.33	0-2
Similarities (total errors)	57	48.0	115	0.96	0-9
Query errors only	32	26.7	52	0.43	0-6
Digit span	31	27.0	42	0.34	0-6
Picture concepts	4	3.0	4	0.03	0-1
Coding	1	1.0	1	0.01	0-1
Vocabulary (total errors)	63	59.0	144	1.20	0-6
Query errors only	43	36.0	71	0.59	0-6
Letter-number sequencing	9	8.0	9	0.08	0-1
Matrix reasoning	12	10.0	13	0.11	0-2
Comprehension (total errors)	73	61.0	153	1.28	0-14
Query errors only	47	39.0	82	0.68	0-9
Symbol search	5	4.0	5	0.04	0-1
Total errors	112	93.0	754	6.28	0-28

Note: N of protocols = 114.

As anticipated, graduate students made the greatest number of errors on the Similarities, Vocabulary, and Comprehension subtests, which accounted for 80.2% of all errors (see Table 2). For the current study, any response where the query was missed, a query used incorrectly, or used when it shouldn’t were identified as an error. Based on this procedure, it was found that problems with querying represented the most common source of student error on these three subtests (37.3% of all errors). Students had the most difficulty with querying on the Comprehension subtest. This is likely due to the wider range of possible responses given by test participants on this subtest. Identification of query errors led to changes in subtest scaled scores for a large number of the WISC-IV protocols. In contrast to query errors, graduate students made very few errors (5.9% of all protocols) due to inattention/human error (e.g., failure to add scores correctly, starting at the wrong item, etc.). The effects of errors on subtest and index scores were not calculated for each subject. However, based on the range of errors identified for a child of mean age from this study, the corrected VCI would range from 0 to 8 scale score points (mean = 3 points) higher, and the corrected FSIQ would range from 0 to 6 points (mean = 2 points) higher.

OPEN IN VIEWER

Table 2.

Frequency of Errors for Similarities, Vocabulary, and Comprehension Subtests per Protocol

	Similarities		Vocabulary		Comprehension
	N	%	N	%	N	%
Incorrect use of a query	31	26.5	30	34.2	71	60.7
Wrong starting point	1	0.9	0	0.0	0	0.0
Failure to use reversal rule for basal	5	4.3	4	3.4	3	2.6
Discontinued too early	1	0.9	0	0.0	0	0.0
Discontinued too late	5	4.3	2	1.7	1	0.9
Change in score	46	39.3	66	56.4	56	47.8
Failed to ask for second response	—	—	—	—	8	6.8
Failure to add items correctly	2	1.7	5	4.3	4	3.4

Note: N of protocols = 114

The front page of the WISC-IV provides the summary data for subtests and index scores and requires the use of appropriate norm tables located at the back of the manual. Completing this page requires careful attention to the age of the participant, adding subtest and index scores correctly, and using the appropriate normative tables. Results identified only a small proportion of students who made errors on the front page (4.3% of all errors) resulting in incorrect subtest and index score calculations (see Table 3). In several cases, the same student made errors on multiple occasions, indicating that some students were more inattentive and prone to making more human errors. As noted above, because students were penalized for any error, it is likely that students took careful precautions to ensure they didn’t lose marks, which resulted in making very few errors due to inattention/carelessness.

OPEN IN VIEWER

Table 3.

Frequency of Front-Page Errors on the Wechsler Intelligence Scale for Children (4th Ed.)

	Number of students making errors (total = 19)
	N	Percent of all errors
Demographic information recorded incorrectly	1	6.0
Miscalculation of chronological age	1	6.0
Incorrectly transferred raw scores to front page	2	12.0
Failed to use correct norm tables	0	0
Failed to record raw scores for each subtests	4	24.0
Used wrong norm tables	4	24.0
Added Index/IQ scores incorrectly	2	12.0
Failed to fill in subtest profile	4	3.4
Failed to fill in composite score profile	4	3.4
Total errors	22

Finally, a series of one-way repeated-measures ANOVAs (analyses of variance) was conducted to determine whether the number of errors for the entire administration, the Verbal Comprehension Index subtests, and then Vocabulary, Similarities, and Comprehension, changed over the course of six practice administrations. Table 4 describes mean error rates for each practice administration for the total administration, the Verbal Comprehension Index subtests, and then for Vocabulary, Similarities, and Comprehension. Over the six administrations, the repeated-measures ANOVA results were not significant for the total test, F(5, 90) = 1.609, p = .166, eta² = .082; the Verbal Comprehension Index, F(5, 90) = 1.692, p = .145, eta² = .086; Vocabulary, F(5, 90) = 0.874, p = .502, eta² = .046; Similarities, F(5, 90) = 0.996, p = .425, eta² = .052; and Comprehension, F(5, 90) = 1.794, p = .122, eta² = .091. Although the repeated-measures ANOVA was not significant for any of the analyses conducted, it is noteworthy that follow-up, pairwise comparisons between administrations revealed several significant effects for the total (1 with 3, p < .01; 2 with 3, p < .05; 3 with 6, p < .05) and the verbal index (1 with 3, p < .01; 2 with 3, p < .0001; 3 with 6, p < .05). A similar result was found for the comprehension subtest (1 with 3, p < .001; 3 with 4, p < .05; 3 with 6, p < .001) but not for the Vocabulary and Similarities subtests.

OPEN IN VIEWER

Table 4.

Errors by Administration for the Wechsler Intelligence Scale for Children (4th Ed.)

	Administration number
	1	2	3	4	5	6
Block design	8	4	8	8	7	7
Similarities (total errors)	27	17	15	18	9	24
Digit span	6	5	7	4	11	6
Picture concepts	2	0	0	0	0	2
Coding	0	0	0	0	0	1
Vocabulary (total errors)	31	33	15	22	16	23
Letter-number sequencing	2	2	1	1	0	1
Matrix reasoning	2	2	1	3	2	3
Comprehension (total errors)	30	29	14	37	16	23
Symbol search	1	1	0	0	2	2
Total errors	109	93	61	93	63	92 a

Note: N of protocols = 114

a

Repeated-measures ANOVA (analysis of variance), F(5, 90) = 1.609, p = .16.

Discussion

Results from the current study continue to show that graduate students are prone to making multiple errors when administering and scoring the WISC-IV. Research on errors with previous versions of the WISC suggests a wide range of mean errors, from 7.8 errors per protocol to 45.2 per protocol. Our results were lower than most studies with a mean of 4.48 errors per protocol. However, findings were reasonably consistent with Loe et al.’s study who found a mean of 5.6 errors per protocol with the WISC-IV. Similar to previous studies, we found that errors were discovered on a majority of protocols (for instance, Loe et al., 2007, found 98% and Belk et. al found 100% had errors). Yet the decrease in mean errors per protocol identified compared with previous editions of the WISC is likely multifactorial. Primarily, the newest version of the WISC (WISC-IV) has improved scoring criterion, provided more defined outcomes for scoring on specific subtests, and new subtests altogether. For instance, Object Assembly (WISC-III) was a core subtest on earlier versions of the WISC, and there were more opportunities for making mistakes when scoring this subtest, compared with Picture Concepts (WISC-IV), which involves defined outcomes for scoring.

As identified by previous research, problems with querying responses for Similarities, Vocabulary, and Comprehension represent the most frequent and significant sources of error. In our study, errors with querying/prompting among subtests in the VCI represented 80% of all errors. Loe et al. (2007) also found a significant number of query errors but found more errors per protocol than our study. The evidence suggests that students clearly struggle with the concept of querying/prompting a response. This is likely a result of uncertainty of when to correctly query (whether missing a query or querying too much) and problems with conceptualizing what constitutes a 0, 1, or 2 point response. In spite of students being provided with corrective feedback in the form of being penalized for incorrect use of a query, our results did not show a significant decrease in the number of query errors by the sixth assessment. Findings have important implications for professors teaching the WISC and strongly suggest that considerable time should be spent on this topic. This is consistent with the findings of Conner and Woodall (1983) who found that administration errors of graduate students significantly decreased, but only after receiving feedback about the type and number of errors committed. Given that the Similarities, Vocabulary, and Comprehension subtests represent core subtests and contribute to a child’s General Ability Index (GAI), the importance of reducing errors on these subtests cannot be overstated. Errors could have significant implications for children referred for special education coding, especially with the recent emphasis of using the GAI in identifying learning disabilities (Bremner et al., 2011).

Compared to other studies, our results suggest that as a whole, only a small proportion of students were prone to making clerical/careless errors. Our results were much lower than previous studies (Belk et al., 2002; Loe et al., 2007; Slate et al., 1991). The importance of using correct normative tables and adding subtest and index scores was emphasized during the teaching of the WISC-IV. Perhaps a combination of making this a point of emphasis during classroom instruction and penalizing students for “human error” was enough to encourage students to double-check their work. In spite of this reasonably positive outcome, results suggest that graduate students are still prone to these types of careless errors, even after multiple administrations, and should be carefully monitored and checked for errors.

One of the most important outcomes of this study concerns the methodology of repeated practice to ensure improved student competence. Since the use of “practice makes perfect” mentality is intuitive, it continues to lack empirical support. In spite of the admonishment that students need at least five or six repetitions (Alfonso et al., 1998), the existing research fails to support this supposition. Previous studies using 5 administrations (Belk et al., 2002), 10 administrations (Slate et al., 1991; Slate & Jones, 1990), and even 15 administrations (Conner & Woodall) have not found improvements with repetition. The recent study by Loe et al. (2007) again found no significant improvements across three practice events. We sought to evaluate trends across six administrations in keeping with the recommendations by Alfonso et al., who suggested five or six administrations might be required. Our results suggest a mild trend for improvement across repeated trials. Of interest, in our study there was a statistically significant decrease in the number of errors from the first to third administration. Yet there appeared to be a “rebound effect” wherein student errors rebounded by the sixth administration. The explanation of this trend is speculative but may involve what we conceptualize as “examiner drift.” It is possible that students were initially diligent to correct errors but became less attentive to learn from their mistakes with repeated trials. In our cohort, students were required to submit two reports by the end of the first term. This meant that there was a much heavier load of work during the second semester and we can only speculate that the combination of increased stress and work coupled with examiner drift contributed to these findings. In spite of using a form of negative punishment (i.e., point deductions), students did not show the improvements that were expected with repeated administrations.

The problem of “examiner drift,” however, is not unique to graduate students, as noted in studies with experienced professionals (Bradley et al., 1980; Sherrets et al., 1979). Thus, even seasoned examiners are prone to making similar errors to graduate students specifically on the verbal subtests of the WISC that require querying and subjective scoring. This finding underscores the importance of providing rigorous training methods to curtail examiner errors.

The findings from our study were consistent with past research and highlight the need for alternative training models. Outcomes from available research suggest that more passive approaches (expecting students to learn from their mistakes) have not helped reduce student errors. Although graduate students are believed to be highly motivated, this alone does not translate into decreasing errors with repeated administrations. Even punitive approaches (penalizing mistakes with reduced grades) did not yield significant reductions in administration and scoring errors. However, results from the current study suggest that the approach of penalizing students may have helped to reduce the number of clerical/careless errors.

To ensure professional competence in the administration of all psychological test measures, alternative delivery models should include more “hands on” approaches to training. This includes ensuring supervised instruction of graduate students provides a methodology that will establish professional competence. Consistent with the arguments of several studies headed by Slate (Slate & Jones, 1990; Slate et al., 1993) we recommend several important components, including the following: (a) reviewing the most common sources of administration and scoring errors during instruction, (b) providing specific feedback about the type and number of errors committed, (c) having students review each other’s administrations and scoring, and (d) requiring more than one video administration of the WISC to be reviewed by the supervisor/professor.

Other forms of feedback may also be helpful. For instance, Thompson and Hodgins (1994) developed an algorithm to enhance student scoring. This feedback tool was given to graduate students in addition to standard instructions about administration and scoring of the WAIS-R. Results were promising. To our knowledge, no other studies have been published developing other algorithms or procedures. This is likely due to the evolving computer scoring programs that have become available and the reliance on one’s previous learning. Yet, consistent with the current body of research, such an approach is needed and could be potentially helpful. Future research could evaluate the effectiveness of the use of such an algorithm and the different methods recommended above.

Another possibility is enhancing the ease of scoring test items from the Wechsler scales. Changes to the WISC-IV and WAIS-IV included adding new subtests with more defined scoring outcomes (Picture Concepts) and removing other subtests outside the core battery. Not surprisingly, this change resulted in fewer errors. Yet the primary source of errors continues to be among Vocabulary, Similarities, and Comprehension subtests. The possibility of including items with more defined outcomes (several word responses) could improve scoring accuracy. Other intelligence scales use this approach. However, this would need to be balanced with ensuring the validity of the constructs is accurately measured.

As with any study, there were weaknesses and limitations. The participants within this study came from several program streams (counseling psychology and school psychology) and, therefore, came with a diverse range of exposure to psychological testing. Since most students were in a master’s program, it is likely they had a wide range of exposure and experience with psychological testing from their undergraduate programs. In addition, students did not have a prerequisite course to ensure their knowledge level was on a reasonably equal level at the start of the course. A second limitation was varying workloads during the course of the academic year. Some students had greater demands with practicum course work, and it is likely that time constraints were different. Whereas it is difficult to control a students’ experience outside the classroom, the range of course demands could also have played a role in a student’s stress levels and time availability for this course. Consistent with the Loe et al. study (2007), this study was restricted to the 10 core subtests and the student errors on other subtests could not be examined.

In summary, the findings from this study echo results from prior research that graduate students are prone to making a significant number of administration and scoring errors on the WISC-IV that does not appear to improve with practice. Alternative delivery models that address this concern need to be considered and empirically tested.