Abstract
Features of eighth-grade history textbooks were examined through replication of a 20-year-old study that investigated “considerateness” of textbooks. Considerate texts provide clear, coherent information and include features that promote students’ comprehension, such as explicit use of organizational structures, a range of question types dispersed within and at the end of chapters, and highlighted new vocabulary. Conversely, inconsiderate texts can impede student learning because comprehension is influenced by coherence and clarity between and among new vocabulary, sentences, paragraphs, and passages at macro and micro levels throughout texts’ chapters. Results of this study indicate areas where today’s texts are more clear and coherent than those 20 years ago, and areas where improvements within textbooks are still needed. Implications for practice are discussed.
Schools in the United States have undergone reform initiatives resulting in an increasing emphasis on high achievement for all students and accountability for each school. For example, the federal legislation No Child Left Behind Act (2001) currently mandates that all students, including students with disabilities, participate in high-stakes assessments and demonstrate grade-level mastery of curriculum. At the same time that standards have increased and curricula have been revised, American school rooms now consist of a more diverse demographic of learners than at any other time in history (Jitendra et al., 2001; Vaughn et al., 2009). Throughout these changing times, the textbook remains a steady companion to teachers in content area classrooms and frequently becomes the curriculum teachers use to instruct toward their states’ standards (Davila & Talanquer, 2010; Hall, 2004; Issitt, 2004; Twyman & Tindal, 2006; Vidal-Abarca, Martinez, & Gilabert, 2000).
Moreover, because teachers and students rely so heavily on textbooks to meet curriculum requirements (Crawford & Carnine, 2000; Lavere, 2008; Repoussi & Tutiaux-Guillon, 2010), it is important for textbooks to address the needs of students, including those who struggle with reading (Harniss, Dickson, Kinder, & Hollenbeck, 2001; Roseman, Stern, & Koppal, 2010). Textbook reading is often assigned as homework with corresponding comprehension tasks (Zigmond, 2006). Such tasks require students to successfully navigate textbooks independently. However, secondary teachers often assume that students who can read words from text correctly can also comprehend well (Edmunds et al., 2009), and as a result, they do not explicitly teach students how to acquire new information from texts. For many students, particularly struggling readers, the textbook may be their most difficult academic challenge (Dull & Van Garderen, 2005; Hall, 2004; Lovitt & Horton, 1994). Therefore, the way that textbooks are written must be clear and coherent to ensure that they are communicative to all students. It is not surprising then that researchers have begun to investigate the content of these textbooks.
Researchers have analyzed textbooks from multiple disciplines with a focus on topics such as inclusiveness and interactive style (e.g., Green & Naidoo, 2008), representation of multiple points of view (Marino, 2011), multiculturalism (e.g., Hsu & Chepyator-Thomson, 2010), use of visual representations (e.g., Lee, 2010), and alignment with national standards (e.g., Jitendra, Deatline-Buchman, & Sczesniak, 2005). Analyses of textbooks fall within two major categories. First, content is analyzed related to each discipline’s standards. The standards are typically written by a team of experts from a professional organization, then published as guides for professionals’ use. Second, textbook content is analyzed related to more discrete factors related to students’ use, such as coherent and well-connected content, difficulty level of questions, and clarity of text structures. Our research falls within the latter category, but next we provide an overview of how textbooks are examined related to the discipline’s standards because some methodology (e.g., how textbooks were selected) from these studies is pertinent to this study.
Content Standards and Textbooks
Investigations of alignment of a discipline’s standards with content in textbooks range from single topics to analysis of how a single event is portrayed across texts. For example, Chiappetta and Fillman (2007) analyzed five high school biology chapters, or sections of chapters, to determine how specific topics were described. The researchers selected texts based on those used in Texas because Texas is one of the top states purchasing the largest quantity of texts. Indeed, three of the texts were the most widely used in the United States at the time. Their examination focused on specific content per text (i.e., not the entire text) to determine how themes, as emphasized in the 1996 National Science Education Standards, were dealt with in each text. Their findings indicated a satisfactory balance of the themes across texts, which provided some evidence that publishers were including information aligned to science standards when developing new or updated textbooks.
Other researchers have analyzed mathematics texts to compare content with the standards put forth by the National Council of Teachers of Mathematics (NCTM) in 2000 (Jitendra, Griffin, et al., 2005; Jones & Tarr, 2007; Sorto, 2011; Stylianides, 2009). Although the NCTM note that “practically all states base their state mathematics frameworks and benchmarks on the NCTM Standards” (NCTM FAQ, n.d.; http://www.nctm.org/standards/faq.aspx), even current texts are not entirely aligned with the detail provided by the mathematics standards (Jitendra, Griffin, & Xin, 2010). Broemmel and Lucas (2010) suggested two reasons for misalignment. First, textbook publishers are not able to keep pace with the variance among states’ mathematics standards, even when most states align with the National Council’s standards. Second, there is a lengthy period of time for texts’ adoptions, typically five or more years. Moreover, this lag time is exacerbated given states’ budgetary constraints. Lorsbach (2008), in describing science textbook adoption processes, indicated that educators anticipated using a curriculum for 10 years, so they limited the copyright of texts examined to recent publication dates (i.e., within 2 years of decision making). Consequently, some texts used could be published between 5 and 12 years ago before a more recently published text is adopted.
Moreover, the specificity of some disciplines’ standards can either help or hinder researchers’ examination of textbooks’ content when the goal is to determine alignment between standards and textbooks. The 2000 NCTM Standards, for example, are detailed and specific as to what students should know and be able to do at each grade level. However, the recently published 2010 Social Studies Standards are more general versus specific statements (National Council for the Social Studies, 2010). For example, one statement is “reinterpret events by relating knowledge from multiple disciplines” (p. 164). Clearly, there is a need to operationalize content from the Social Studies Standards before texts’ analyses could occur. In addition, decisions for how to incorporate states’ standards with the broad statements in the National Council’s standards need to be made. In short, some discipline’s standards are more detailed and specific, whereas others are more general and broad.
Features in Textbooks
In addition to examining textbooks to determine how well content aligns with the discipline’s standards, there has been much interest in looking more carefully at how content is presented in texts. In fact, one may find an excellent alignment to a discipline’s standards, but how clearly the text is written may fall short: unclear text structures, use of new vocabulary with no glossary or definition, or fragmented connections of information within and across chapters. Armbruster and colleagues use the term “considerate” to describe texts’ features that are well developed to promote students’ understanding of the content (Armbruster, 1986; Armbruster & Anderson, 1988; Kantor, Anderson, & Armbruster, 1983).
In addition, textbook studies have focused on promoting students’ learning of higher level thinking skills. To that end, Jitendra et al. (2001) focused on four middle school geography textbooks. The primary purposes of this analysis were to determine how fact based the texts were and to determine the extent to which they promoted complex thinking and problem solving; however, aspects related to text considerateness were also captured in items of the analysis. Results showed that only one textbook provided an overview or statement of purpose to prepare the reader for the chapter. In addition, these textbooks contained primarily reiteration or summarization questions (60.2%); higher level questions were less frequently used, with 38.9% of the questions requiring skills such as predicting, applying, or evaluating. Researchers also found that, on average, the textbooks were written 2 to 4 years above the grade level of the intended audience.
In a similar examination of chemistry textbooks, Pappa and Tsaparlis (2011) drew from a convenience sampling of texts to examine specific topics in chapters (or parts of chapters) for what types of questions were asked. They found that fewer procedural knowledge questions were used across texts and that most of the declarative knowledge questions were at a basic versus advanced level. The researchers emphasized that for students to acquire higher level knowledge and skills, teachers need to create those types of questions and ask more open-ended questions. Teachers who rely on the texts’ questions alone may not be sufficiently challenging students who need to acquire more complex aspects of the content.
Dole and Shield (2008) examined two middle school mathematics texts to determine how well proportional reasoning was presented. They identified four outcomes for ratio and proportion, and corresponding indicators to use as the criteria from which texts’ content related to these topics was portrayed. The researchers found low alignment with six mathematics principles, which are critical for students’ coherent and connected understanding of bigger ideas in mathematics. Texts’ content was fragmented within a topic, and switches to a new topic omitted explicit connections between or among topics.
Underlying these results from textbook studies is the way written language promotes or hinders students’ understanding from reading (or listening to) texts’ content. At a macro level, the overall structure for individual chapters needs to be coherent and contain connections to other chapters as well as content within that chapter. At a micro level, the language used to explicitly indicate when connections are being made and when text structures, such as sequencing or cause–effect relationships, are used must be clear; wording (such as signal words) should be accurately used; questions’ placement should be interspersed across the chapter; and new vocabulary should be highlighted.
Roseman et al. (2010) contended that “to be considered high-quality, textbooks must themselves be coherent and help students make the connections necessary to organize their new knowledge into a coherent and meaningful whole” (p. 48). Similarly, Ainsworth and Burcham (2007) noted that “texts on the same topic manipulated in seemingly subtle ways can have dramatic impacts upon learning” (p. 286). In their study, learners who read information from a coherent text answered more implicit, or inferential, questions. The researchers adapted a selection from the original textbook to develop the more coherent text. Across passages, they replaced pronouns with nouns, connected new concepts with familiar concepts, and added words that signaled relationships within the content (e.g., signal words, such as because or however). Moreover, the researchers incorporated headings and summaries across the larger sections (i.e., major headings within the chapter, the chapter), and developed connections within paragraphs to the more global topic. The former changes are considered local coherence, whereas those at the latter level are global coherence.
In examining the research-to-date related to these micro-level elements, the research by Kinder, Bursuck, and Epstein (1992) stood out as a foundational study that magnified micro-level elements of social studies textbooks. Kinder et al.’s study was unique in consolidating discrete aspects of content in social studies textbooks that affect the meaning students derive from reading texts. Their study occurred after the term “considerate” was used by Armbruster and colleagues when referring to the design and delivery of content in students’ textbooks (Armbruster & Anderson, 1988; Kantor et al., 1983). Although research on textbooks’ content preceded use of the term “considerate,” given the preponderance of evidence indicating how much educators relied on textbooks as the curriculum and to guide their instruction, more researchers began to investigate the extent to which discipline-specific textbooks were “considerate” for the learners who used them (Roseman et al., 2010). To that end, Kinder et al.’s analysis provided a prototype from which subsequent studies and supplemental strategies were developed (Crawford & Carnine, 2000; de Oliveira, 2010; Dickson, 1999; Harniss et al., 2001; Jitendra, Cole, Hoppes, & Wilson, 1998; Jitendra et al., 2005; Mastropieri, Scruggs, & Graetz, 2003).
“Considerate” Texts
In 1992, Kinder et al. conducted an analysis of eighth-grade history textbooks and determined that they were not “considerate” for readers. Textbooks adopted in the two largest states with state textbook adoption policies (California and Texas) were evaluated for elements that make texts easier to understand: global coherence, local coherence, types of embedded questions, vocabulary, and readability.
For global coherence, findings showed that although 80% of the chapters examined had introductions for that chapter, only 60% actually overviewed content in that chapter, and only 30% had reviews of previous chapter’s content. Consequently, readers were often left on their own to determine how content connected within and among chapters.
One aspect of local coherence (i.e., clarity at a paragraph level) is how well explicit signals prompted readers about what type of information was being presented. The most frequently used text structure in history texts found by Kinder et al. (1992) were temporal sequences (34%), cause/effect and problem/solution (29%), descriptive listings (26.7%), and compare/contrast (5%). These structures were often signaled by specific words or phrases. For example, a sentence that began “There are four components of the . . . ” would signal the reader that a descriptive listing followed. For local coherence at the sentence level, there were clear pronoun referents in 82% of the text sections evaluated. As to local coherence, the texts examined by Kinder et al. seemed clearer for readers.
Question types and placement, as well as how new vocabulary is introduced, are also features that assist readers’ understanding of the text. Kinder et al.’s (1992) analysis of question placement within chapters revealed that most questions were placed within (range of 17 to 95) or at the end of chapters (range of 23 to 69), but that very few questions were placed at the beginning of the chapter (range of 0 to 5). A majority of these questions were detail oriented (range of 22 to 97), and the remaining questions were more general (range of 16 to 55). Although all textbooks highlighted new vocabulary (approximately 1.25 new words per page) and definitions could be deduced by savvy readers based on usage within the context of sentences, no texts provided guides for new terms’ pronunciations or more explicit definitions. As such, readers were largely reliant on themselves or their teachers to discern how to pronounce new vocabulary and what the new vocabulary meant. Moreover, readers who needed to know the vocabulary’s definitions to comprehend main ideas and details, not to mention making inferences or using analytical skills, were clearly at a disadvantage when texts provided no guidance for new vocabulary.
Finally, whether content was written for the grade level of students using the text was determined using the Fry Readability Scale to analyze five 100-word passages per text. Kinder et al. (1992) found that the average readability across eighth-grade social studies textbooks was at the 10.9 grade level with a range from the 9th grade to the 3rd year of college. Typical 8th-grade students, then, likely struggled in reading these texts.
These results from Kinder et al.’s (1992) study two decades ago spur the question, “To what extent have texts improved since then?” Logically, one might assume that publishers have had time to improve texts and attend to those aspects, simply for the purpose of ensuring that students as readers would be better able to understand the content as written. However, findings from recent studies indicate that this may not be the case. Roseman et al. (2010) noted that none of the textbooks they analyzed provided explicit connections, which could promote students’ understanding, between and among topics.
Just as Chiappetta and Fillman (2007) analyzed biology texts to determine improvements after 15 years, we also recognized the value in comparing current day results to those from texts’ analyses conducted previously. Moreover, we chose to analyze at the discrete level (as Kinder et al., 1992, did) because the extent to which a discipline’s content standards are included in that text is irrelevant if the textbook does not promote students’ learning from the text (in other words, if the text is not “considerate”). Consequently, this study examined elements that Kinder et al. (1992) analyzed, and adhered to the same methodology so that improvements could be discerned. The elements examined by Kinder et al. included the following:
Global coherence
Local coherence
Questions
New vocabulary
Readability
In the next section, each of these elements is described along with results from texts’ analyses since Kinder et al.’s study.
Text Features That Affect Comprehension
More than 30 years ago, Kintsch and van Dijk (1978) described how students’ comprehension is influenced by coherence and clarity between and among new vocabulary, sentences, paragraphs, and passages at macro and micro levels throughout texts’ chapters. Students’ comprehension increases when they recognize explicit and clear schemas, or organizational structures. Equally important for students’ accurate comprehension is their background knowledge and capacity to recognize and organize the content’s schema (Armbruster, 1986). That is, readers interact with the text’s schema, which is the pattern or organization of how content is portrayed in a text (Twyman & Tindal, 2006). Students’ comprehension problems often occur because they have not yet acquired the background knowledge to recognize the content in the text (e.g., using the periodic table in science), or the text’s content is insufficient in providing clues that guide readers to organizational structures. McNamara, Kintsch, Songer, and Kintsch (1996) found that middle school students who read expository content that included clear global and local coherence recalled significantly more than peers who read content without global and local coherence. When text is presented coherently and logically, readers are able to read more quickly and remember more information (Meyer, 2003).
Global Coherence
The integration of ideas among and between chapters is referred to as global coherence (Harniss et al., 2001). Global coherence requires students to connect content previously processed in the chapter and taps into students’ long-term memory (Albrecht & O’Brien, 1993). Explicit use of segues, reviews, transitions, and advance organizers helps students determine the authors’ logic in transmitting information on a global level. Conversely, lack of overviews can lead to students wondering about the purpose of their reading.
In their analysis of coherency in four high school biology texts, Roseman et al. (2010) described coherency as explicit identification of key ideas and connections among those ideas to communicate clearly stated information. These researchers noted how students’ background knowledge influenced coherency because previous experiences provide a foundation on which to build new knowledge and emphasized the explicit nature of the ideas connecting to each other. Roseman et al. note that coherency can be further compounded because texts are less likely to make connections between key ideas at all, and, when connections are made, the connections are not in close proximity to each other within the text or chapter. They comment that their expert reviewers had difficulty finding and identifying connections, which can be indicative of problems students would ultimately experience deciphering coherency. Relatedly, McCrudden, Schraw, Hartley, and Kiewra (2004) experimented with texts’ organization by presenting college students with content that was integrated (concept defined followed by an example) versus segregated (all concepts defined in one area with examples provided later in the text). Students rated integrated organization as easier to comprehend than segregated organization. There was also a positive correlation between students’ ease of comprehension ratings and students’ learning of facts and concepts. Researchers noted that the integrated organization exemplifies well-organized text content and reduces students’ cognitive load so they can focus on connecting concepts to examples versus searching and remembering. In sum, how well a text is organized and presents information influences students’ learning.
Another type of global coherence is text structure. Students may struggle to comprehend content in textbooks if they have limited understanding of the various ways textbooks are structured (Bryant, Ugel, Thompson, & Hamff, 1999). Although terminology varies, text structures for expository text are generally described as (a) listing or description, (b) temporal or sequence, (c) cause/effect, (d) compare/contrast, and (e) problem/solution (Meyer & Poon, 2001). Lehman and Schraw (2002) found that relevance could be a mediating factor when texts contained breaks in coherence, indicating that educators’ insertion of how content is meaningful to learners can help them develop connections that the text does not have. However, the researchers also concurred that enhancing texts’ coherence increases students’ understanding. Alternatively, students’ comprehension and reading flow are disrupted with only intermittent use of coherent structures.
Local Coherence
Local coherence refers to how text relates at the sentence level and taps into students’ short-term memory (Barzelay & Lapata, 2008; Long & Chong, 2001). Sudden switches from one topic to the next can impede comprehension. Similarly, pronouns with ambiguous referents can confuse readers. At a semantic level, local coherence consists of identifying pronoun referents clearly as well as matching synonyms explicitly instead of relying on students to infer such connections (McNamara et al., 1996; Venable, 2003). Local coherence is particularly necessary when students answer questions about the content they read. For example, if “they” could refer to three different people or groups, is the pronoun referent, as written in the text, clear to the students? Texts that contain ambiguous referents can lead to problems with understanding.
Questions
In addition, questions can help activate learners’ capacities to engage in lower and higher order thinking. Placement of questions at the beginning, within, and at the end of chapters is a way to provide both explicit and frequent cues to actively engage students in the reading process, which in turn can increase comprehension. For example, Jo and Bednarz (2011) noted that questions placed throughout a chapter are used to help students monitor their comprehension as they read. Moreover, they described the importance of having different levels of questions to prompt students’ higher level thinking skills. These researchers used four geography texts adopted for use in Texas and analyzed the first two units in the texts. Among their findings were that end-of-chapter questions did not consistently encompass the critical knowledge, concepts, and skills from the chapter. Consequently, if questions at the beginning and throughout the chapter are ignored or perceived as options for comprehension, students may be missing out on critical content as well as comprehension prompts. Similarly, Davila and Talanquer (2010) examined end-of-chapter questions in three widely used undergraduate chemistry texts, and they found major differences in the type of questions per text. Nonetheless, all texts’ questions indicated the need for more high-level questions, such as synthesis and evaluation, which the researchers found seriously lacking in the texts.
Vocabulary
Vocabulary knowledge plays an important role in the comprehension of text (S. W. Fry & Gosky, 2007; Wood, Vintinner, Hill-Miller, Harmon, & Hedrick, 2009). Textbooks that include supports for identifying, pronouncing, and defining new vocabulary facilitate students’ learning of terms, whereas lack of supports can impede vocabulary learning. Furthermore, some readers struggle with new vocabulary words being introduced with new content information (Carnine & Carnine, 2004), which is problematic because content area textbooks inherently contain large numbers of new vocabulary (Mastropieri et al., 2003). Stahl (2003) further claimed that the relationship between vocabulary and students’ comprehension of the text is strong; learners who cannot pronounce nor define terms may find further understanding of the text prohibitive. Moreover, Stahl noted that dictionary definitions can be circuitous when definitions contain more unfamiliar terms. Consequently, how new vocabulary is introduced, defined, and used within expository texts can promote or inhibit students’ further understanding of the content.
Brown (2011), in his study of how vocabulary is introduced in nine textbooks, found that, by far, emphasis was on terms and definitions. He noted that stronger associations could be made for students if synonyms were identified and concepts were described in terms of what they are and are not. Wood et al. (2009) found that new vocabulary infrequently received multiple exposures in content texts despite research indicating the need for students to have repeated opportunities to learn and use new vocabulary.
Readability
Readability is determined through formulas that aim to gauge the difficulty of text (e.g., Flesch–Kincaid and Fry Readability Scale). Readability of textbooks has historically been at higher grade levels than the grade levels of students for whom they are intended (e.g., Jitendra et al., 2001; Kinder et al., 1992). Textbook publishers often respond to findings such as these by shortening sentences and/or using simpler vocabulary (Harniss et al., 2001). However, the use of short disconnected sentences generally causes text to be more difficult to comprehend (Chambliss, 1994). This is because other factors influence students’ comprehension as well, including students’ background knowledge as well as the texts’ coherence and clarity of writing (Armbruster & Anderson, 1988; Gunning, 2003). Similarly, McTigue and Slough (2010) cautioned against relying on readability analysis as a sole predictor of a text’s match to student reading ability. Although readability alone does not dictate comprehension (O’Toole & King, 2010), readability remains one of several important factors to consider (e.g., Davison, 1984; E. Fry, 2002).
The Current Investigation
Two decades ago, seminal work conducted by Kinder et al. (1992) investigated whether eighth-grade history textbooks contained clear, coherent information and included features that promote students’ comprehension—in other words, to determine if the textbooks were “considerate.” They found that texts were actually “inconsiderate” to learners. Considering the current emphasis on curriculum and the continued expectation that students will learn from independent reading, textbooks need to be as “considerate” as possible to facilitate teachers’ instruction and students’ learning. The current study seeks to determine whether textbooks used today are indeed more “considerate” for readers by looking at the same five elements that Kinder et al. investigated nearly 20 years ago: global coherence, local coherence, questions, vocabulary, and readability. Specifically, the following research question was investigated:
Method
Text Selection
Our initial intention was to select the most current edition of textbooks used in the original 1992 Kinder et al. study. However, due to considerable changes in the textbook publishing industry, including large numbers of mergers and acquisitions of publishers, the original textbooks examined are now out of print. Instead, we chose to replicate textbook selection procedures from Kinder et al. (1992). In the original study, 10 of the eighth-grade textbooks on the California and Texas book adoption lists were selected. Today, California and Texas have the largest state populations (2010 U.S. Census), and they are both textbook adoption states, meaning that all schools in the state can only adopt textbooks on the state-approved list. In addition, these two states are recognized as purchasing the largest share of texts among publishers (Stein, Stuen, Carnine, & Long, 2001). Wiley and Barr (2007) identified California as the largest, followed closely by Texas. Textbooks on the 2010 statewide adoption lists for California and Texas were selected for this analysis (n = 6): The American Journey: Early Years (2009); The American Nation: Beginnings Through 1877 (Davidson, 2003); Creating America: A History of the United States: Beginnings Through Reconstruction (Garcia, Ogle, Risinger, Stevos, & Jordan, 2002); The American Republic (Ritchie & The National Geographic Society, 2003); Call to Freedom: Beginnings to 1877 (Stucky & Sulrucci, 2001); and Why We Remember: United States History Through Reconstruction (Viola, 1998). It should be noted that the editions currently in use were examined. In addition, our analyses included three more texts than the quantity noted by the American Textbook Council’s 2011 listing of widely adopted history texts. We did not examine supplemental activities, whether technological or otherwise, because these are typically optional for teachers to use (e.g. Jitendra et al., 2005), and Kinder et al. (1992) confined their analyses to the student versions of the text.
Because chapters within a single textbook have consistent text formatting and organization (e.g., how headings are formatted, inclusion of chapter introductions, bold or highlighted vocabulary, etc.), one chapter from each textbook was randomly selected for analysis. However, consistent with Kinder et al., a pre–civil war chapter was selected to help control for variation in text due to content. Guidance for the quantity of text that was analyzed in the current study was informed by previous investigations. For example, Green and Naidoo (2008) examined common content across three texts, then focused more deeply on 20% of the common content. Jitendra et al. (2005) focused on specific mathematics skill lessons across five texts, with the texts selected based on consultations with mathematics educators, teachers, and school administrators who noted that the texts were representative of mathematical texts adopted in the United States. For a study on geography texts, Jitendra et al. (2001) analyzed one lesson from a chapter in each of four textbooks that was about five countries. Roseman et al. (2010) analyzed a series of pages that contained the concepts they were analyzing in each of four biology texts. In sum, researchers identify portions of texts for analyses, such as a subset of one chapter, one chapter, or specific topics within chapters across texts, as representative of how texts’ authors present information.
Coding Procedures
The first and second authors developed systematic procedures for coding important elements of each textbook, and then trained two doctoral students on coding procedures. The training included modeling, guided practice, and independent practice. Following training, two doctoral students independently coded each selected chapter. Areas coded were global and local coherence, types of questions, new vocabulary words, and readability.
Global coherence
Global coherence of text was evaluated through the identification of textbook format, paragraph text structures, and text structure signal words. Global coherence among chapters of the textbook was evaluated by identifying the presence and quality of chapter introductions and summaries. Global coherence among chapter sections was determined by evaluating how well chapter headings/subheadings described the text within the section. In addition, 20% of the total number of paragraphs for each chapter were coded to determine the text structure (i.e., listing/descriptive, temporal, compare/contrast, cause/effect, problem/solution, or an unclear text structure) and whether “signal words” were present to help readers identify the text structure (e.g., including or such as might signal a descriptive text structure, while first and next might signal a temporal text structure).
These paragraphs were selected using a systematic process to obtain a selection of paragraphs that were representative of the chapter. First, all of the paragraphs in the chapter were counted, and this number was multiplied by 0.20 to determine the number of paragraphs that represented 20% of the chapter. Next, an online random number generator was used to randomly select this number of headings in the chapter. Finally, an online random number generator was used to select one paragraph from each of these sections.
Local coherence
An online random numbers generator was used to select the first of five consecutive paragraphs, which were then examined to judge local coherence. Specifically, the number and clarity of pronouns were examined (e.g., Is the pronoun referent for “he” clear?).
Questions
All questions present throughout each chapter were coded by their placement (beginning, middle, or end of the chapter) and type (main idea, detail, or non-content-related questions). Detail questions were those for which the correct response simply involved restating or copying a sentence from within the chapter’s content. Main idea questions were deemed more global in nature, and as such, responses could not be generated by simply restating an existing sentence in the text. Non-content-related questions contained no information relevant to the social studies topic being studied (e.g., “Do you volunteer?”).
Vocabulary
All vocabulary presented in either bold or italics within a chapter were tallied and classified as either a term or a person. In addition, coders determined if words were defined in the text, margin, or glossary.
Readability
Five 100-word passages were selected at 10%, 25%, 50%, 75%, and 90% placements within the chapter, per sampling procedures described by Backhaus, Mell, and Sabel (2007), and analyzed to derive the Flesch–Kincaid grade level and reading ease scores. This readability formula was used rather than the Fry formula originally used by Kinder et al. (1992) because it is now an easily accessible formula embedded in word processing programs, such as Microsoft Word, that are widely available within schools.
Protocols
For every area coded, corresponding protocols with detailed directions were developed for the scorers to record information (see coding procedures above). These data were then entered into SPSS statistical software for analysis.
Reliability Procedures
All selected textbook chapters were independently coded by two trained researchers. Interrater agreement for initial coding was calculated by dividing agreements by agreements and disagreements. The mean interrater agreement between the primary coders was 80.09% with a range from 66.13% to 90.55%. This range was due mainly to variability in length of chapters (and therefore possible agreements), formatting of textbooks (in some books it was difficult to identify whether questions were part of the vignettes or the main text of the chapter), and inconsistency in the text structures (as reported in the results, 22% of paragraphs coded had an unclear text structure). Because agreement was below 80% for two of the textbooks, additional steps were taken to ensure accuracy of coding. For any coding discrepancy, a third researcher independently coded any item where a discrepancy occurred and reconciled the item to reach 100% agreement.
Results
Descriptive statistics were used to analyze coded information from each textbook related to global and local coherence, content questions, vocabulary, and readability. Findings for each textbook are presented in Table 1.
Findings Across Textbooks.
Note. Reading ease interpretation (Flesch, 1948): 0–30 (very difficult), 30–50 (difficult), 50–60 (fairly difficult), 60–70 (standard), 70–80 (easy), 90–100 (very easy).
Global Coherence
Chapters in five of the six textbooks included a chapter introduction with an overview of the current chapter’s content; however, none of the texts provided a summary of the previous chapter’s content. Fewer textbooks (n = 3) included a chapter summary with a review of the current chapter’s content, and none of these summaries provided a preview of the next chapter.
The average number of headings across texts’ chapters was 34 (range of headings = 23–56). For the purposes of data analysis, 20% of chapter headings were evaluated for clarity. On average, 95% of headings clearly represented the content of that section (range = 80%–100%).
Text structure
Across textbook chapters, descriptive text structures occurred most frequently (51%), followed by cause/effect (16%), temporal (9%), compare/contrast (2%), and problem solution (1%). Notably, the mean number of unclear text structures was relatively large (22%). Across textbooks, more than half of the paragraphs included signal words intended to cue the reader to the text structure (61%). However, signal words in 43% of these paragraphs did not accurately match the actual text structure. For example, words that would typically signal a temporal structure were found in a descriptive paragraph, or the same paragraph contained words that would typically signal cause/effect and compare/contrast text structures. In both of these cases, the signal words did not give a clear indication of the correct text structure.
Local Coherence
Pronoun referents were used to determine local coherence. The numbers of pronouns in selected chapters of each textbook ranged between four and nine. Across textbooks, only one pronoun within sampled text was not clear as to which noun it referenced.
Questions
Questions were evaluated by question placement (beginning, middle, or end of chapter) and type (main idea, detail, or non-content-related questions). The total number of questions in coded chapters ranged from 31 to 63. As Table 1 shows, the proportion of questions found at the beginning, middle, and end of the chapters varied widely across textbooks. On average, 60% of questions were located at the end of the chapters. Although all textbooks had more questions at the end of the chapter, all but one textbook had at least some questions placed at the beginning (prior to reading) or middle (during reading).
The proportion of question types also varied widely across textbooks. On average, 70% of questions were detail oriented, and these types of questions were the most frequently posed in all but one of the textbooks, Call to Freedom (Stucky & Sulrucci, 2001). However, main idea questions were posed in all but one of the textbooks. In addition, at least one non-content-related question was posed in each of the books. For example, one non-content-related question, found in The American Journey (Appleby et al., 2009), served to activate students’ prior knowledge and interest in the topic: “Do you volunteer to help people, such as the elderly, with their daily chores?” (p. 49).
Vocabulary
On average, 31 new vocabulary words were introduced per chapter. Of these, 71% were new terms, and 29% were new people. All terms were defined within text and a glossary at the back of the textbook; however, only one of the six textbooks defined vocabulary in text margins.
Readability
The mean Flesch–Kincaid reading ease score across texts was 52.54 (range = 49.18–55.46), which corresponds to “fairly difficult” reading according to Flesch (1948). However, for most books, the variability in reading ease among selected 100-word passages within a textbook chapter was large (see Figure 1). For example, reading ease scores for Chapter 6 of The American Republic (Ritchie & The National Geographic Society, 2003) varied from 37.1 (difficult) to 62.2 (standard). The mean grade-level reading score across texts was 9.25 (range = 8.60–9.92). Similar to the reading ease scores, the range of grade levels within a single text indicated variability for five of the six texts.
FK RE average and range per text (FK RE maximum and FK RE minimum).
Discussion
Shand (2009) stated that many educators use textbooks as the central resource for making curriculum decisions. If the textbook is written in a manner that makes it difficult for students to access information, significant instructional adjustments must occur. Therefore, educators need to know how to analyze texts and access other sources for areas where the textbook is lacking or information is not presented in a way that supports student learning. In addition, McCrudden et al. (2004) noted that texts’ organization can either ease students’ cognitive load for learning and remembering concepts, or present information in ways that tax students’ cognition, subsequently making content more complex to comprehend. Findings from this study illustrate areas where textbooks can be analyzed by teachers to determine if they are “considerate” to readers, thereby minimizing obstacles to student learning.
Findings from the current study show that introductions to chapters continue to be a consistent feature in textbooks; however, summaries of key information at the end of the chapter were less frequently provided (50%) than an earlier analysis (90%) conducted by Kinder et al. (1992). Furthermore, when summaries were provided, none included a preview of the next chapter. A summary plays an important role in helping students monitor whether they have identified and understood key information from chapters that are often quite dense. In addition, summaries provide coherency between chapter topics; in other words, they link one chapter’s main ideas to the next chapter’s main ideas (i.e., How do they connect? What is the flow from this to the next chapter?). Teachers need to be aware when textbooks do not include this important feature and plan to supplement the text through instruction. Moreover, Roseman et al. (2010) noted that texts’ provision of connections is not insurance that students will realize the importance of the connections. The researchers discuss the importance of teachers realizing the impact of such connections, and emphasizing connections with students. Other instructional methods can be used to increase clear and critical connections. For example, a teacher might use a graphic organizer to visually display key points from within sections of a chapter. Similarly, when global coherence (connections within and across chapters) is low, teachers’ pedagogy needs to include strategies to help make explicit connections for students, who otherwise may not activate prior knowledge of how previous chapters are connected to the current chapter.
The current study, like Kinder et al. (1992), found that all or nearly all subheadings clearly represented the content found in that section of the text. This is an indicator that headings are a reliable feature in text that can be used by teachers when teaching students strategies for understanding their textbooks. Conversely, more variance in text structures of paragraphs occurred 20 years ago than in the textbooks that were analyzed in the current study, which were primarily descriptive (50.04%). While consistent text structures are certainly more “considerate” to readers and mediate difficulty level (readability) of text, it might also be an indication that students are required to use higher level analytical skills less often. Gunning (2003) noted the importance of students’ skill levels extending beyond factual or summary proficiency and into higher order areas such as compare/contract and cause/effect content. According to the current study’s results, educators cannot rely solely on textbooks, but will need to plan supplemental lessons to ensure that students acquire these higher order thinking skills.
Perhaps more concerning is the fact that there was such a large number of unclear text structures (22.57%). Unclear text can be misleading to students as they work to comprehend the information provided. In addition, teaching students signal words that help them identify text structure is a common teaching practice. However, findings from the current study suggest that caution should be used with this type of instruction. Although signal words are often found in textbooks, findings from our study showed that the signal words were not helpful for identifying text structure 43% of the time. It is not simply the presence of signal words, but the clear use of signal words, that facilitates students’ use of text structures for learning. This finding is consistent with research conducted by Twyman and Tindal (2006) that found that typical texts’ schema, in general, are not explicit in cueing students about the text structure. Our finding is further disconcerting because even when explicit cueing may be present, it may not be accurate or clear more than two fifths of the time.
Another important finding from the current analysis is that many textbooks are beginning to disburse questions throughout the chapter, which can prompt students intermittently to set a purpose for reading as well as to check their comprehension as they are reading. Kinder et al. (1992) found very few, if any, questions at the beginning of the chapter, with half in the middle of the chapter and about two fifths at the end of the chapter. Furthermore, Kinder et al. reported fewer main ideas and almost twice as many detail questions across the total questions in the chapter, with no data collected on non-content-related questions. In comparison, the current study found that textbooks contained questions that are still predominantly detail oriented. This is consistent with other research findings that a disproportionate quantity of content in textbooks focuses on more factual versus inferential and higher order skills desired by educators who use those texts (Davila & Talanquer, 2010; Jitendra et al., 2001; Jo & Bednarz, 2011). As a result, educators are responsible for ensuring that students have access to sufficiently challenging questions, which may mean educators need to construct the questions themselves if they are not present in the text (Pappa & Tsaparlis, 2011).
Both the current study and Kinder et al. (1992) found that the average grade level for textbooks was above the grade level the texts were designed for (i.e., eighth grade) and difficulty ranged beyond high school grades. For the current study, the range within each text was quite variable for all except one text. McTigue and Slough (2010) found that a similar range in the texts’ passages that they analyzed using the Flesch–Kincaid formula, which may indicate a range in readability, as measured by a formula, is not unusual to find across a single text. Although readability grade levels should not be used as the only item contributing to a student’s comprehension of a text (Gunning, 2003), the variance within a single text can be indicative that not all passages are written with even approximately the same grade level in mind. In addition, O’Toole and King (2010) noted that quantitative measures of readability may not correlate to comprehensibility, and any measures need to be calibrated by educators’ knowledge of who the readers are.
Future Research
Consistent with previous textbook analysis (Jitendra et al., 2005; Kinder et al., 1992), supplemental resources, whether technological or hard copy, were not analyzed because they are options for teachers, whereas the reading of the textbook chapter itself is likely required of all students. However, although not part of the formal analysis of the current study, we observed common features of textbooks that were not reported in the Kinder et al. study in 1992. Quality varied, but all textbooks had some sort of an introduction that includes a timeline of events of the chapter, a narrative story that introduces each section of the chapter, and vignettes and other features related to the topic (including illustrations, photographs, original artifacts/primary sources, and maps). Although titled differently, there were also components in many of the books that served the same function. For example, some consistent textbook sections were intended to help students make connections to what they already know. For example, in Call to Freedom, this feature was called “Build on What You Know,” but in Creating America, it was called “What Do You Know?/What Do You Want to Know?” Textbooks also included features that were intended to assist students with reading comprehension such as reading checks, reading strategies, tips for analyzing primary sources, and guidance in how to take notes while reading. In addition, many texts are including learning aids to help students with comprehension of text, including reading checks embedded throughout the chapter, labels on questions to help students identify their purpose (e.g., define, identify, summarize, find the main idea), prompts to promote critical thinking, and prompts to promote connections to writing. Similar to how we found inaccurately labeled or unclear text structures in our study, Lavere (2008) also found questions labeled as “Critical Thinking” that were actually factual questions. Consequently, the questions themselves need to be analyzed to ascertain their accurate function, particularly related to the intent for students to comprehend at higher levels. Future research needs to investigate the presence of these features related to findings in the research literature related to reading comprehension instruction.
All books published after 2001 also included embedded reference to technology resources for students such as videos and CD-ROMs that accompanied the textbook, companion websites to the textbook, or external websites (e.g., CNNfyi.com). More generally, digital texts are beginning to replace and/or supplement traditional textbooks. However, Repoussi and Tutiaux-Guillon (2010) noted that even with new technologies, social studies textbooks prevail as the primary source for educators and students. Content, whether in traditional or technological formats, still needs to attain features of considerateness for student readers. Moreover, although technology may seem seductive as a tool, firm research documenting that technological tools result in increased student learning needs to occur. For example, Fisher, Lapp, and Wood (2011), in their examination of students’ comprehension of an online or a print-based science article, found that simply putting content online did not promote students’ learning. Conversely, Clay, Zorfass, Brann, Kotula, and Smolkowski (2009) compared eighth-grade students’ use of a virtual thesaurus to an online dictionary tool for social studies vocabulary learning. Researchers found that students using each tool made significant gains on posttests, and, interestingly, students indicated a preference for the online dictionary tool, although they made comparable gains using the visual thesaurus. In short, although an intuitive appeal exists when content is technologically based and there is research to support the use of technology (S. W. Fry & Gosky, 2007), educators should be cautious regarding widespread acceptance and use absent corresponding research detailing effective implementation. Just as the “considerate” features of texts were examined in this study with some features left wanting, those same “considerate” features apply whether the text is print based via a traditional hard copy of a text or transposed into a digital format. Although it was not the purpose of the current study, future research needs to investigate how issues related to coherence and clear presentation of content translates to digital and online texts as well as other technological formats.
Although it appears that some school systems and states may be using more digital texts, all students are not yet accessing even the current editions of the texts they have. For example, one text we reviewed was published in 1998. It may be that this year or next year the students have a more recent text, but the texts students were using were the ones included in this study. As noted by several researchers, there is often a lag period between the time of text adoption and text use (Broemmel & Lucas, 2010; Lorsbach, 2008). Consequently, it is important for researchers to ensure that they are examining texts being used so that their analysis has more relevance.
As noted earlier, there are two categories of research on textbooks; the first is related to the discipline’s standards and the second is related to more discrete content within the text. As for the former, the recently published 2010 Social Studies Standards (National Council for the Social Studies, 2010) provide more of a framework for educators versus detailed descriptions of what students should know and be able to do at each grade level. Consequently, future researchers who examine content related to the national Social Studies organization need to first determine how to transform general statements into content-specific criteria. Two sources can be of guidance. The 1996 National Standards for History (National Center for History in the Schools, 1996) provide more detail per grade level, so guidance may be discerned from this resource. In addition, devising a way to coalesce the standards across states, perhaps in selecting those that are more specific and noted across a preponderance of states, may lead to a plausible process.
Implications for Practice
Educators continue to rely on textbooks for pedagogical guidance for themselves and as a support for student learning. If texts’ content is fragmented, unclear, and disconnected, educators need to be aware of how to analyze content to determine how well these critical features are presented. In addition, when texts’ content is not conducive to facilitating students’ comprehension (i.e., a “considerate” text), educators need to know how to design their instruction to include effective features that texts lack and students need, such as enhanced organization (Twyman & Tindal, 2006), development of background knowledge (Roseman et al., 2010), scaffolding, interspersed questioning (Jo & Bednarz, 2011), reviews of information, deeper examination of vocabulary (Brown, 2011), and integration of reading and writing (Harniss et al., 2001). Although our work focuses on middle school history textbooks, educators from all grade levels for all content areas will benefit from this research because any educators who primarily use textbooks to guide their instruction need to first be aware of the quality of those texts. As noted by Zabihi and Pordel (2011), ultimately the educators are tasked as the mediators to bridge any divide between content in texts and the pedagogical needs of their students.
Repeatedly, throughout the texts’ analyses that occurred after the Kinder et al. 1992 study, researchers emphasized that absent the content in texts providing clear, accurate, coherent, and communicative information, it was the responsibility of educators to do so (Broemmel & Lucas, 2010; Chiappetta & Fillman, 2007; Dole & Shield, 2008; Jones & Tarr, 2007). In addition, pedagogical techniques that facilitate student learning when aspects of texts are inconsiderate for learners, such as graphic organizers to clarify connections, questioning techniques throughout a unit, preteaching vocabulary, and similar methods, have been necessary to mitigate any disconnects between considerate and inconsiderate text content (Bryant et al., 1999; Hall, 2004; Harniss et al., 2001; Mastropieri et al., 2003; Stahl, 2003). Text publishers should be emboldened to require text writers and graphic designers to design with not just content standards in mind, but also standards of clarity and coherence that promote students’ comprehension. Moreover, caution should be used when digitalizing content that may not yet be well developed for students. There is still a need to firmly design and develop texts’ content so information is more comprehensible for students. Until texts are entirely considerate for more learners, educators need to know how to analyze their texts and make pedagogical decisions that mediate differences when inconsiderate features are evident.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.