Complexity of Illustrations in PISA-2009 Science Items and Its Relationship to the Performance of Students from Shanghai-China,the United States,and Mexico

Abstract

Background

While illustrations are widely used in international test comparisons, very scant research has been conducted on their design and on their influence on student performance. It is not clear how the features of illustration act in combination supporting students’ access to the content of items or increasing their interpretation demands.

Purpose

The purpose of this study was to examine how the characteristics of illustrations used in science items are related to the performance of students in PISA-2009.

Population

We used PISA-2009 data on the responses of 1571 to 1582, 1601 to 1617, and 11,662 to 11,746 students, respectively from Shanghai-China, the U.S., and Mexico.

Research Design

This was a secondary analysis study that compared the three PISA-2009 jurisdictions as to the magnitudes and directions of the correlations between the complexity of illustrations used in science items and the difficulty of those items. We used the information on item difficulty provided by the Organisation for Economic Co-operation and Development (OECD, 2010). The features of the illustrations were coded with a system that coded the presence or absence of about 100 types of illustration features. The complexity of each illustration was computed as the number of different types of features observed.

Findings

For the U.S. and Mexico, items with higher illustration complexities were more difficult than items with lower illustration complexities. For Shanghai-China, the correlation of number of illustration features and item difficulty was close to zero. This pattern of correlations is consistent with the three jurisdictions’ ranking in PISA-2009.

Conclusions

While this was an exploratory study that examined the performance of only three of the 65 PISA-2009 jurisdictions, the results speak to the importance of examining illustration complexity as a factor that shapes student performance in science tests and which should be addressed systematically in the design of science items.

Get full access to this article

View all access options for this article.

References

Ametller

, & Pinto

(2002). Students' reading of innovative images of energy at secondary school level. International Journal of Science Education, 24(3), 285–312.

Bloomer

R. H.

(1960). Children's preferences and responses as related to styles and themes of illustration. The Elementary School Journal, 60(6), 334–340.

Boling

, Eccarius

, Smith

, & Frick

(2004). Instructional illustrations: Intended meanings and learner interpretations. Journal of Visual Literacy, 24(2), 185–204.

Carrick

C. E.

(1978). Design of instructional illustrations in medicine. Journal of Visual Communication in Medicine, 1(4), 161–173.

Colin

, Chauvet

, & Viennot

(2002). Reading images in optics: students' difficulties and teachers' views. International Journal of Science Education, 24(3), 313–332.

Dimopoulos

, Koulaidis

, & Sklaveniti

(2003). Towards an analysis of visual images in school science textbooks and press articles about science and technology. Research in Science Education, 33(2), 189–216.

Evans

M. A.

, Watson

, & Willows

D. M.

(1987). A naturalistic inquiry into illustrations in instructional textbooks. In Willows

& Houghton

(Eds.), The psychology of illustration: Volume 1, basic research (pp. 86–115). New York: Springer-Verlag.

Fichtner

, Peitzman

, & Sasser

(1994). What's fair? Assessing subject matter knowledge of LEP students in sheltered classrooms. In Peitzman

& Gadda

(Eds.), With different eyes: Insights into teaching language minority students across the disciplines (pp. 143–159). Los Angeles: UCLA Publishing Center for Academic Interinstitutional Programs.

Filippatou

, & Pumfrey

P. D.

(1996). Pictures, titles, reading accuracy and reading comprehension: A research review (1973-95). Educational Researcher, 38(3), 259–291.

10.

Fleming

M. L.

(1966). Instructional illustrations: A survey of types occurring in print materials for four subject areas (Final Report, November, 1966, Project 1381, NDEA Title VII A). Washington, DC: U.S. Office of Education.

11.

Goldsmith

(1987). The analysis of illustration in theory and practice. In Willows

& Houghton

(Eds.), The psychology of illustration: Volume 2, The psychology of illustration (pp. 53–85). New York: Springer-Verlag.

12.

Halliday

M. A. K.

(1978). Language as social semiotic: The social interpretation of language and meaning. Hodder Arnold.

13.

Harp

S. F.

, & Mayer

R. E.

(1998). how seductive details do their damage: A theory of cognitive interest in science learning. Journal of Educational Psychology, 90, 414–434.

14.

Hunter

, Crismore

, & Pearson

P. D.

(1987). visual displays in basal readers and social studies textbooks. In Willows

D. M.

& Houghton

H. A.

(Eds.), The psychology of illustration: Volume 2, the psychology of illustration (pp. 116–135). New York: Springer-Verlag.

15.

Knight

, Gunawardena

C. N.

, & Aydin

C. H.

(2009). Cultural interpretations of the visual meaning of icons and images used in North American web design. Educational Media International, 46(1), 17–35.

16.

Kopriva

R. J.

(2008). Improving testing for English language learners. New York: Routledge.

17.

Kosslyn

S. M.

(2006). Graph design for the eye and mind. Oxford, England: Oxford University Press.

18.

Kress

, & van Leeuwen

(2001). Multimodal discourse. The modes and media of contemporary communication. London: Arnold.

19.

Lemke

J. L.

(1998). Multiplying meaning: visual and verbal semiotics in scientific text. In Martin

J. R.

& Veel

(Eds.), Reading science: Critical and functional perspectives on discourses of science (pp. 87–113). New York: Routledge.

20.

Levie

W. H.

, & Lentz

(1982). Effects of text illustrations: A review of research. Educational Communications and Technology Journal, 30(4), 195–232.

21.

Levin

J. R.

, Anglin

G. J.

, & Carney

R. N.

(1987). On empirically validating functions of pictures in prose. In Willows

D. M.

& Houghton

H. A.

(Eds.), The psychology of illustration, volume 1: Basic research (pp. 51–85). New York. Springer-Verlag.

22.

Martiniello

(2009). Linguistic complexity, schematic representations, and differential item functioning for English language learners in math tests. Educational Assessment, 14, 160–179.

23.

Mayer

R. E.

, Heiser

, & Lonn

(2001). Cognitive constraints on multimedia learning: When presenting more material results in less understanding. Journal of Educational Psychology, 93(1), 187–198.

24.

Miller

W. A.

(1938). Reading with and without pictures. The Elementary School Journal, 38(9), 676–682.

25.

National Assessment of Educational Progress. (2005). NAEP 2005 science assessment public released items for grade 8. Retrieved from http://nces.ed.gov/nationsreportcard/itmrlsx/search.aspx?subject=science

26.

Nisbett

R. E.

, & Miyamoto

(2005). The influence of culture: holistic versus analytic perception. Trends in Cognitive Sciences, 9(10), 467–473.

27.

Organisation for Economic Co-operation and Development (OECD). (2009). PISA 2009 assessment framework: Key competencies in reading, mathematics and science. Retrieved from http://www.oecd.org/dataoecd/11/40/44455820.pdf

28.

Organisation for Economic Co-operation and Development (OECD). (2010). PISA 2009 database: Compendium for the cognitive item responses. Retrieved from http://pisa2009.acer.edu.au/downloads/Comp_Cogn09_Dec10.zip

29.

Organisation for Economic Co-operation and Development (OECD). (2012). PISA 2009 technical report. Paris: OECD Publishing. Retrieved from https://dx-doi-org.web.bisu.edu.cn/10.1787/9789264167872-en

30.

O'Malley

J. M.

, & Valdez-Pierce

(1996). Authentic assessment for English language learners: Practical approaches for teachers. Reading, MA: Addison Wesley Publishing.

31.

Pauwels

(2008). An integrated model for conceptualising visual competence in scientific research and communication. Visual Studies, 23(2), 147–161.

32.

Schiffman

H. F.

(1996). Linguistic culture and language policy. London, England: Routledge.

33.

Shanahan

T. B.

(2006). A comparison of graphic and visual test accommodations in measuring urban fifth-grade English language learners' science content acquisition (Doctoral dissertation). University of Southern California, Los Angeles, CA.

34.

Solano-Flores

(2011). Development of illustrations as image supports for English language learners in large-scale testing: A report on the procedure for designing vignette illustrations. Paper presented at the CADRE ELL-STEM Roundtable session, Advancing English Language Learners in Science and Math: Realizing the Promise, at the annual meeting of the American Educational Research Association, April 7-11.

35.

Solano-Flores

, Barnett-Clarke

, & Kachchaf

R. R.

(in press). Semiotic structure and meaning making: The performance of English language learners on mathematics tests. Educational Evaluation.

36.

Solano-Flores

, Nguyen-Le

, & Wang

(2012). Linking generalizability and effectiveness in the evaluation of testing accommodations for English language learners. Paper presented at the symposium, Limits of Generalizing in Psychological Research, organized by Kadriye Ercikan, International Congress of Psychology in Cape Town, South Africa, July 22-27, 2012.

37.

Solano-Flores

, & Wang

(2011). Conceptual framework for analyzing and designing illustrations in science assessment: Development and use in the testing of linguistically and culturally diverse populations. Paper presented the Annual Conference of the National Council on Measurement in Education, New Orleans, LA, April 7-11.

38.

Wang

(2012). The use of illustrations in large-scale science assessment: A comparative study (Unpublished doctoral dissertation). University of Colorado, Boulder, CO.

39.

Wang

, Chia

, Kachchaf

, & Solano-Flores

(2012). Item illustration complexity and the performance of English language learners in a science test. Paper presented at the Annual Conference of the American Educational Research Association, Vancouver, Canada, April 13-17, 2012.

40.

Wang

, & Solano-Flores

(2011a). Illustrations with graphic devices in large-scale science assessments: An exploratory cross-cultural study on students' perception and interpretation. Paper presented at the annual meeting of the National Council on Measurement in Education, April 7-11, New Orleans, LA.

41.

Wang

, & Solano-Flores

(2011b). The use of illustrations in large-scale science assessment: A comparative study. Paper presented at the annual meeting of the American Educational Research Association, April 7-11, New Orleans, LA.

42.

Washington

W. N.

, & Godfrey

R. R.

(1974). The effectiveness of illustrated items. Journal of Educational Measurement, 11(2), 121–124.

43.

Weidenmann

(1994). Codes of instructional pictures. In Schnotz

& Kulhavy

(Eds.), Comprehension of graphics (pp. 29–42). Amsterdam, Netherlands: North-Holland.

44.

Wellington

, & Osborne

(2001). Language and literacy in science education. Buckingham, England: Open University Press.

45.

Winn

(1987). Charts, graphs, and diagrams in educational materials. In Willows

& Houghton

(Eds.), The psychology of illustration: Volume 1, basic research (pp. 152–198). New York: Springer-Verlag.

46.

Zimet

S. F.

(1966). Children's interest and story preferences: A critical review of the literature. The Elementary School Journal, 67(3), 122–130.