Effectiveness of Various Automated Readability Measures for the Competitive Evaluation of User Documentation

Abstract

I examined samples from a number of companys' user publications using several automated reading measures and a graphics/text ratio. The goal was to answer two questions: Were there reliable differences in writing style among the competitors? If so, were these differences related to their rank position in published surveys of user satisfaction with documentation? Of the measures included in the study, only the Cloudiness Count had any significant relationship to rank position in the surveys. A second evaluation, focused on the components of the Cloudiness Count, indicated that both of its components (passive voice and ‘empty’ words - a type of infrequent word) contributed equally to its effectiveness. This is consistent with psycholinguistic research that indicates that it is harder for people to extract the meaning from a passive sentence relative to its active counterpart, and that word frequency is the variable with the most influence on the speed of lexical access.

Get full access to this article

View all access options for this article.

References

Bailey

R. W.

(1989). Human performance engineering: Using human factors/ergonomics to achieve computer system usability. Englewood Cliffs, NJ: Prentice-Hall.

Broadbent

D. E.

(1977). Language and ergonomics. Applied Ergonomics, 8, 15–18.

Cohen

(2004). The need for a writing style checker. Paper presented at the 51st Annual Conference of the Society for Technical Communications, Baltimore, MD, May 9–12, 2004.

Collins-Thompson

Callan

(2005). Predicting reading difficulty with statistical language models. Journal of the American Society for Information Science and Technology, 56, 1448–1462.

Dataquest, Inc. (1991a). Score report: Customer satisfaction – personal computers. Third Quarter 1990 through First Quarter 1991. San Jose, CA: Author.

Dataquest, Inc. (1991b). Score report: Customer satisfaction – personal computers. Fourth Quarter 1990 through Third Quarter 1991. San Jose, CA: Author.

Forster

K. I.

(1990). Lexical processing. In Osherson

D. N.

Lasnik

(Eds.), Language (pp. 95–131). Cambridge, MA: MIT Press.

Fry

E. B.

(1989). Reading formulas – maligned but valid. Journal of Reading, 32, 292–297.

Garrett

M. F.

(1990). Sentence processing. In Osherson

D. N.

Lasnik

(Eds.), Language (pp. 133–175). Cambridge, MA: MIT Press.

10.

Klare

G. R.

(1984). Readability. In Pearson

P. D.

(Ed.), Handbook of Reading Research. New York, NY: Longmans.

11.

Miller

G. A.

(1962). Some psychological studies of grammar. American Psychologist, 17, 748–762.

12.

Whaley

C. P.

(1978). Word-nonword classification time. Journal of Verbal Learning and Verbal Behavior, 17, 143–154.

13.

Zakaluk

B. L.

Samuels

S. J.

(1988). Readability: Its past, present, and future. Newark, DE: International Reading Association.