Comparing Differences in Accuracy across Conditions or Individuals: An Argument for the Use of Log Odds

Abstract

In experimental psychology, the degree of difference between the proportions of correctly solved items on two related tests (such as word lists) has been calculated by different methods, for example, a simple difference (e.g. as used in within-subjects ANOVAs), difference relative to potential gain, quotient, difference between standardized z-scores, or by Signal Detection Theory's d’, every one of which may yield different results. The present article discusses the choice of methods with an example from reading research concerned with contextual facilitation in readers with different abilities. Assuming that the total number of correctly solved items captures all relevant variance in subjects’ abilities (i.e. it is a sufficient measure), it is demonstrated that the logarithm of the quotient between odds for the frequencies of correct responses (log odds) is the most suitable method of calculation. For example, calculations based on log odds provide an appropriate ranking of the subjects, from relevant for repeated-measures ANOVAs. The aims of the present paper are to draw attention to the problem of comparing differences, to evaluate current methods of calculation, and to present a consistent solution to the problem. We illustrate the problem by applying several current methods of expressing differences in proportion correct to the same set of data: they yield radically different results. The choice of an appropriate method depends on the assumptions made about the underlying metric. We argue for a solution-the log odds measure-on the basis of Rasch's (1968a, 1968b) measurement model. This relies on a demonstration that, given a set of test items given to a set of subjects, the proportion of correct is, in technical terms, a sufficient statistic -that is, it captures all relevant variation in a subject's ability (or an item's difficulty). The example for the presentation of the problem is selected from recent reading research using comparison of accuracy in two related reading tasks. Reading accuracy with two tasks is a suitable example because it is a simple, much-used design. Researchers

Get full access to this article

View all access options for this article.

References

Allerup

(1987). Raschmodeller—principper og anvendelse [Rasch models—principles and applications] Copenhagen: The Danish Institute for Educational Research.

Allerup

(1994). Theory of Rasch measurement. International Encyclopedia of Education. 2nd ed. Oxford: Pergamon Press.

Allerup

Elbro

(1997). The use of log odds transformation on score values in the dichotomous Rasch model Manuscript in preparation.

Baron

Strawson

(1976). Use of orthographic and word-specific knowledge in reading words aloud. Journal of Experimental Psychology: Human Perception and Performance 2, 386–393.

Boder

(1971). Developmental dyslexia: A diagnostic screening procedure based on three characteristic patterns of reading and spelling. Bateman

Learning disorders, Vol 4: Reading. Seattle, WA: Special Child Publications 297–347.

Brady

Fowler

Stone

Winbury

(1994). Training phonological awareness: A study with inner-city kindergarten children. Annals of Dyslexia 44, 26–59.

Bryant

Impey

(1986). The similarities between normal readers and developmental and acquired dyslexics. Cognition 24, 121–137.

Coltheart

(1982). The psycholinguistic analysis of acquired dyslexias: Some illustrations. Philosophical Transactions of the Royal Society B298, 151–164.

Elbro

(1990). Differences in dyslexia. A study of reading strategies and deficits in a linguistic perspective Copenhagen: Munksgaard.

10.

Elbro

Rasmussen

Spelling

(1996). Teaching reading to disabled readers with language disorders: A controlled evaluation of synthetic speech feedback. Scandinavian Journal of Psychology 37, 140–155.

11.

Elliott

P. B.

(1964). Appendix 1—Tables of d’. Swets

J. A.

Signal detection and recognition by human observers. Contemporary readings. New York: John Wiley 651–684.

12.

McCarthy

Warrington

E. K.

(1986). Phonological reading: Phenomena and paradoxes. Cortex 22, 359–380.

13.

Mitterer

J. O.

(1982). There are at least two kinds of poor readers: Whole-word poor readers and recoding poor readers. Canadian Journal of Psychology 36, 445–461.

14.

Olson

R. K.

Kliegl

Davidson

B. J.

Foltz

(1985). Individual and developmental differences in reading disability. MacKinnon

G. E.

Waller

T. G.

Reading research: Advances in theory and practice. New York: Academic Press Vol 4.

15.

Rasch

A mathematical theory of objectivity and its consequences for model construction

European Meeting on Statistics, Econometrics, and Management. Amsterdam 1968.

16.

Rasch

(1968). Probabilistic models for some intelligence and attainment tests Chicago, IL: Chicago University Press.

17.

Snowling

M. J.

(1980). The development of grapheme-phoneme correspondence in normal and dyslexic readers. Journal of Experimental Child Psychology 29, 294–305.

18.

Swets

J. A.

Signal detection and recognition by human observers: Contemporary readings. New York: John Wiley 1964.

19.

Tunmer

W. E.

Chapman

J. W.

(1997). Language prediction skill, phonological recoding ability, and beginning reading. Hulme

Joshi

R. M.

Reading and spelling: Development and disorder. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc 33–67.