Countrywide norms declared obsolete: best practice alert for IQ testing in a multicultural context

The International Test Commission (ITC) (2015) has set the standards for best practice in the psychometric test arena, and it is the responsibility of both test producers and the practitioners making use of those tests to be critically aware of situations that may fail to meet such standards. One such instance relates to the South African standardization of the US-based Wechsler Adult Intelligence Scale–Fourth Edition (WAIS-IV^SA) that has recently been launched for clinical use in this country (Wechsler, 2014). Commensurate with the traditional mode of norming of the US-based Wechsler intelligence tests, the South African adaptation was conducted on the basis of population (i.e., countrywide) norming. There is a problem, however, in that the value of population-based norms is increasingly being called into question on scientific grounds. Many clinicians will assume that because an intelligence test standardization is “South African,” it will be a clinically relevant instrument and preferable to the US or UK version of the test for use in the country. However, this is an erroneous assumption that has been the topic of a watershed series of commentaries published recently in an international forum (Shuttleworth-Edwards, 2016; Suchy, 2016; Sunderaraman, Zahodne, & Manly, 2016; Taylor, 2016). It is important for South African clinicians to be fully briefed on the concerns raised that are now part of the scientific literature.

Practitioners who do not understand the issues are likely to misuse the intelligence test, and this will undermine the clinical judgments and conclusions they make for diagnostic and placement purposes. The central problem, highlighted in the title of the article by Shuttleworth-Edwards (2016), is that a sample that is “generally representative is representative of none.” This means that the normative data do not constitute a sample against which any member of the population can be meaningfully benchmarked. In her editorial titled “Population-based norms in crisis,” Suchy (2016) states as follows: “The series of articles starts with an impassioned commentary by Ann Shuttleworth-Edwards, in which she attempts to raise awareness of the challenges faced by South African practitioners.” Suchy points out that traditionally, populations have been defined by country borders. She submits that this has been a method of convenience rather than reflecting a globally applicable theory for the collection of clinically valid norms. The problem she highlights is as follows:

The method is only appropriate if the countries are characterized by a single language and a unified educational content . . . not only is this not completely true of any country in the world, but it is also becoming progressively less true, due to exponential increases in globalization, migration, and population diversity around the world. (p. 975)

In South Africa, with its demographic diversity, the notion of population-based norms must be deemed obsolete. Instead, Suchy (2016) calls for new approaches to be considered, and the delineation of relevant reference groups for norming that make biopsychosocial sense. The ITC (2015) defines a test as obsolete “when its underlying theory, item content, norms, or technical adequacy do not meet the needs for its intended purpose, professional standards, or when its continued use would lead to inappropriate or inaccurate decisions or diagnoses.” Importantly, for the purposes of this commentary, the term “obsolete” is being used with respect to the population-based norming aspect of intelligence tests in multicultural contexts (rather than with respect to any other aspects such as item content or technical parameters). Having regard to the ITC definition, it is a worrying revelation to discover that the population-based standardization of the WAIS^SA is indeed obsolete despite its fairly recent release. The use of a test such as this, where practitioners are at risk of returning incorrect decisions or erroneous diagnoses, constitutes a problem demanding some immediate remedies, in addition to more long-term solutions.

Shuttleworth-Edwards (2016) details her reasons for questioning the utility of the South African intelligence test norms derived for the WAIS-IV^SA (Wechsler, 2014), and those of its predecessor the South African (SA) Wechsler Adult Intelligence Scale–Third Edition (WAIS-III) (Claassen, Krynouw, Patersen, & Mathe, 2001, cited in Shuttleworth-Edwards, 2016). She points out that the “mixed bag” norms supplied for these two standardizations do not have a refined level of utility for any of the ethnic/race groups within South Africa. Demographically focused research has shown that English and African first language individuals with advantaged educational backgrounds do not differ substantively from the US intelligence test standard, whereas African first language individuals with disadvantaged educational backgrounds score between 20 and 25 IQ points below the US standard (see review in Shuttleworth-Edwards, 2016). The WAIS-IV^SA standardization is based on approximately 80% African first language individuals, whose educational background is not specifically known, although the majority are likely to be educationally disadvantaged.

Consequently, if the same raw scores are used to calculate an IQ based on the US standardization and then to calculate an IQ based on the WAIS^SA standardization, the WAIS^SA IQ scores are higher in the order of 23 1Q points (Shuttleworth-Edwards, 2016). Moreover, the extent of inflation ranges from little to substantial increases depending on the functional modality that is most depressed due to the effects of educational disadvantage, with Working Memory being the least and Processing Speed being the most. For an advantaged South African, the scores are substantially inflated, leading to false interpretations. For a disadvantaged South African, the calculated IQ may well be a better reflection of the individual’s potential (or true IQ). However, it is also likely to be misleading because it could easily be taken to imply that the individual has developed skills and capabilities that he or she has not. That is to say, a WAIS^SA IQ will not have the predictive capacity usually associated with an IQ score. Also, the attempt to diagnose reduced cognitive function due to a brain injury will be obscured, especially in light of the marked inflation on the Processing Speed Index that is a prime indicator of brain dysfunction relative to other functions.

Sunderaraman et al. (2016) present a commentary from a group well known for their extensive research on minority groups in the United States. These authors strongly endorse the stance taken by Shuttleworth-Edwards as outlined above. They criticize for-profit test companies that develop products that are out of touch with the needs of practitioners, and call upon practitioners to hold test companies accountable to strict, ethical standards in the production of accurate measurement of cognitive function. Nicola Taylor, currently a director of JvR Psychometrics, the company that developed the South African adaptation of the WAIS-IV under license by Pearson, has acknowledged the need for improvements in the test (Taylor, 2016). This includes the need for more reference norms for subsets in the population, as well as the need for more clinically relevant explanations in the test manual. However, practical solutions for meaningful practice are needed in the here and now. Accordingly, an increasing number of clinicians in the country have been following a more appropriate demographically focused approach than is possible using the population-based norms of the SA WAIS-III and WAIS-IV ^SA, as follows. The stance (as delineated under Points 1–3 below) is supported by case-based research in support of its clinical and ecological validity (e.g., Shuttleworth-Edwards, 2012).