Helping Poor Readers Demonstrate Their Science Competence

The increasing relevance of standardized assessments reinforces the importance of high-quality instruments for research on student competencies (e.g., Martin, Mullis, Foy, & Stanco, 2012; Organisation for Economic Co-Operation and Development [OECD], 2007). Because variations of item characteristics such as item format or construct-irrelevant student characteristics might systematically influence students’ responses to test items, item development should build more systematically on cognitive theories and models (e.g., Embretson & Gorin, 2001; Gorin, 2006; Mislevy, 2006). Test items constructed to assess science competence typically employ so-called item stimuli, which introduce real-life phenomena or everyday contexts; the associated items relate to these stimuli (e.g., OECD, 2009). To comprehend and solve the problem presented in a test item, it is necessary to construct an adequate internal mental representation on the basis of the external information provided by the item stimulus (Leighton & Sternberg, 2004). Research suggests that when test items present pictures in addition to textual information, it supports mental model construction of the given task, thus facilitating the subsequent problem-solving process (Brünken, Steinbacher, Schnotz, & Leutner, 2001; Schnotz & Kürschner, 2008). However, the supporting effect depends on the informational relation between text and picture (Saß, Wittwer, Senkbeil, & Köller, 2012). If, on one hand, text and picture contain equivalent information, either source by itself presents all information necessary to construct an adequate mental model to solve the item. Equivalent pictures may facilitate identifying relevant information; they seem particularly suitable when complex information needs to be conveyed. Provided students are able to utilize the information conveyed by the pictures, items containing equivalent pictures should be easier to solve than the same items without pictorial information. If, on the other hand, the text and picture in an item stimulus serve complementary functions, both representations contain unique information; consequently, both representations need to be considered to construct an adequate mental model (Ainsworth, 2006). In this case, relevant information in both representations has to be identified and integrated with each other to solve the item. These additional cognitive demands likely contribute to item difficulty.

Item characteristics may facilitate but also impair those integration processes and thus the construction of an adequate mental model; in the worst case, they lead to misapprehensions. Cognitive theories such as cognitive load theory (Sweller, 2005) or the cognitive theory of multimedia learning (Mayer, 2005) provide principles of how (learning) material has to be designed to enhance the construction of coherent mental models. A familiar principle is the spatial contiguity effect (e.g., Mayer, 1989; Moreno & Mayer, 1999). Physically placing all relevant information in close proximity reduces unnecessary visual searching and decreases cognitive demands when processing multiple representations. Specific instructional aids that enhance coherence formation may act as a further facilitator when building a mental model from information provided in multiple formats: Students benefit from highlighting and referencing corresponding elements in multiple representations in terms of learning outcomes (e.g., Florax & Plötzner, 2010; Seufert, 2003). Multimedia research showed that students frequently focus on textual information rather than pictorial information when processing material such as science textbook passages (Hannus & Hyönä, 1999; Mason, Tornatora, & Pluchino, 2013). Therefore supporting text–picture integration is commonly suggested in constructing multimedia learning material. Moreover, Reinking, Hayes, and McEneaney (1988) investigated the effect of explicit cues in learning from science texts with accompanying pictures and found that cues increased poor readers’ attention to the picture, thereby improving learning progress. In the assessment context, however, it is still an open question how students process multiple representations in item stimuli. Item design principles may (differentially) increase or decrease item difficulty. Equivalent pictures, for instance, may reduce reading demands by serving as an alternative informational source. Pictorial information might thus be especially helpful for students with difficulties understanding and retrieving information from written texts. Integration aids intended to facilitate integration of multiple representations by directing test-takers’ attention to particular pictorial elements may likewise particularly benefit students with low reading comprehension. Integration aids might thus level out differences in students’ science performance caused by differences in reading comprehension.

We present an experiment investigating the effect of text–picture integration aids on students’ performance in a science test. We were particularly interested in whether the effect of integration aids on test performance is moderated by the informational relation between text and picture as either complementary or equivalent. Moreover, we investigated reading comprehension as a potential moderator of the effect of integration aids on test performance.

Results and Discussion

The 2 × 2 mixed-model ANOVA showed a non-significant main effect of integration aids, F(1, 89) = 2.61, p = .11, η_p² = .03; yet the interaction effect was indeed statistically significant, F(1, 89) = 15.42, p < .001, η_p² = .15 (see Figure 1). Presenting items in the integrated format (M = 0.72, SD = 0.17) did not generally increase students’ science performance as compared with presentation in the non-integrated format (M = 0.66, SD = 0.19). Simple comparisons, however, demonstrated a statistically significant difference in performance as a function of informational relation when items were presented in an integrated format: Performance was better when text and picture were complementary (M = 0.78, SD = 0.18) instead of equivalent (M = 0.66, SD = 0.20), t(44) = 4.29, p < .001, r = .54. When items were presented without integration aids, no significant difference occurred as a function of informational relation, t(45) = −1.31, p = .20, r = .19, that is, students performed equally well on complementary (M = 0.64, SD = 0.24) and equivalent items (M = 0.68, SD = 0.18).

OPEN IN VIEWER

Figure 1.

Science performance as a function of experimental conditions.

To answer the question whether the effect of integration aids depends on students’ reading comprehension, we computed moderated regression analyses (Aiken & West, 1991). Two science performance scores were calculated for each student: one for the items with equivalent and one for the items with complementary information. These two scores were separately regressed on the integration format condition (effect coded), on reading comprehension (z-standardized), and on a product term of both first-order terms. As indicated by a non-significant interaction term for information-equivalent items (β = .04, p = .68), the effect of integration format condition on test performance did not differ as a function of reading comprehension when textual and pictorial information were equivalent. For information-complementary items, however, the interaction effect was statistically significant, β = −.24, p = .016. Simple slope regression analysis revealed a significant simple slope for the non-integrated condition, β = .51, p < .001, and a non-significant simple slope for the integrated condition, β = .04, p = .80 (see Figure 2). Reading comprehension had a meaningful influence on performance on the science test when—and only when—the complementary information carried by text and picture was presented in a non-integrated format.

OPEN IN VIEWER

Figure 2.

Science performance on complementary items as a function of integration format and reading comprehension (1 SD above and 1 SD below the mean).

Successfully handling multiple representations is a crucial ability in modern societies. Diagrams and graphics in combination with text is a well-established way to represent scientific information in science, science education, and—more generally—to communicate science to the public (Krajcik & Sutherland, 2010). Students should therefore acquire the ability to comprehend and draw inferences from multiple representations in science education. For this reason, multiple external representations are commonly used in standardized science assessments (e.g., OECD, 2009). Our study was the first to demonstrate that pictures in the item stimulus might differentially affect good and poor readers’ science test performance: Presenting complementary information in an integrated format annihilated the relationship of reading comprehension and performance in the science test. Poor readers who were thus supported in comprehending an item stimulus performed as well as good readers in the science test. A possible explanation of this effect is that text–picture integration aids reduced the amount of construct-irrelevant variance.

Previous research indicated that some children mostly rely on the text when processing multimedia representations (Hannus & Hyönä, 1999; Mason et al., 2013). When textual and pictorial information are complementary, test-takers ignoring the pictures will be unable to comprehend the item stimulus and the task it presents. Our study showed that when text and picture contained complementary information, guiding students’ attention to the picture leads to a better science performance than the non-integrated mode of presentation. When supported in text–picture integration, test-takers may have been better able to identify and consider relevant information and thereby build more adequate mental models.

Currently, the majority of science tests are not likely to employ an integrated item format. Presenting complementary information in a non-integrated format, however, seems to counteract the purpose of multiple external representations; at least students who only recently started secondary school seem incapable of taking full advantage of them. Instructional aids facilitating comprehension of the item stimulus enable test-takers to demonstrate levels of science competence, which would have gone unrecognized otherwise. Consequently, science tests should be constructed according to design principles avoiding split attention.