Lexical Coverage and Readability of Science Textbooks for English-Medium Instruction Secondary Schools in Hong Kong

Abstract

Textbooks are an important source of knowledge input on which the transmission of academic knowledge often relies, especially in the early stages of academic learning. Adopting a corpus-based approach, this study evaluates the text difficulty of science textbooks used in secondary English-medium instruction schools in Hong Kong, with a focus on their lexical coverage and readability. It compares the English language used in English-medium science textbooks with that in English as a foreign language textbooks. The analysis reveals that the text difficulty of the English-medium science textbooks is inappropriate in terms of the coverage of academic words and the readability level. The results also show that the coverage of words from the Academic Word List (AWL) and the General Service List (GSL) as well as the text readability levels, vary across scientific topics. These findings will inform textbook design and the development of pedagogical strategies to facilitate students’ learning of subject content in the medium of English.

Keywords

English-medium instruction corpus-based approach text difficulty teaching materials

Introduction

Reading in a foreign language presents learners with multiple challenges, such as unknown syntactic features and unfamiliar meanings of particular vocabulary. When encountering a new register of written text, such as academic texts, the challenges become even greater, as different written registers have been found to have significant differences in terms of their lexico-grammatical features (Biber et al., 1998). However, an increasing number of young students in non-Anglophone countries are now expected to read academic texts in English due to the rise of English as medium of instruction in primary and secondary schools (Dearden, 2014). Concerning the possible challenges these students may have, a number of studies have been conducted to examine their academic performance, and these have reported that academic achievements may be undermined by the reading challenge (Lo & Lo, 2014; Roussel et al., 2017). In response to these challenges, attempts have been made to improve pedagogical practice in English-medium instruction (EMI) contexts (e.g., Hu & Gao, 2021; Lo & Macaro, 2012), and to promote self-regulated learning strategies to facilitate students’ learning of subject content through the medium of English (Hu & Gao, 2018b, 2020; Jaekel, 2020; Mahboob, 2014). In addition, research has been undertaken to evaluate the quality of textbooks so that a deeper understanding of students’ challenges can be achieved (e.g., Maxwell-Reid & Lau, 2016). Since textbooks are considered an important source of knowledge input (Harwood, 2005), and the transmission of academic content knowledge relies on the language used (Lorenzo, 2017), it is necessary to explore whether textbooks can function as appropriate pedagogical resources in supporting the learning of EMI academic subjects.

Description of the language used in textbooks is thought to be a fair way to evaluate the quality of language textbooks or academic textbooks written in foreign languages, as it shows what the language used in the textbooks actually looks like (Harwood, 2005). Over the past three decades, a growing number of studies have examined the language used in textbooks to evaluate their quality (see a review in Barbieri & Eckhardt, 2007). On the one hand, some researchers investigate the text difficulty that the academic textbooks pose for learners, highlighting the lexico-grammatical load of the language for the academic register (Fang, 2005; Hsu, 2014). On the other hand, some studies compare the language used in textbooks with the target language, examining whether the textbooks provide proper language input (Miller, 2011; Moreno, 2003). Some studies also compare the language in textbooks with that used in teachers’ lessons, to see how they complement each other for academic language development (e.g., García et al., 2018). However, very few studies have compared the text difficulty, such as lexical coverage and readability level, of the textbooks used for English as a foreign language classes (EFL textbooks) with that of textbooks for content subject classes in EMI contexts (English-medium academic textbooks) (e.g., Miller, 2011; Wood & Appel, 2014). Such a comparison is important, as it helps teachers to understand the gaps between these textbooks in terms of their text difficulty, and serves as an initial step for language and subject content teachers to work together and build a shared community of scholarship in supporting learners’ acquisition of academic subject content through the medium of English.

To fill this research gap, this study examines the English language used in English-medium science textbooks and EFL textbooks for secondary 1 students in Hong Kong, with a focus on their lexical coverage and readability levels. It aims to help determine whether students are adequately prepared by EFL textbooks to overcome the language challenges they face when learning subject content in English, and to inform the design of future textbooks and the collaboration between EFL teachers and content subject teachers.

Literature Review

Evaluation of EFL/ESL/EAP Textbooks

Many studies have evaluated the quality of textbooks for English courses, such as EFL courses, English as a second language (ESL) courses, and English for academic purposes (EAP) courses, comparing the language features covered in the textbooks with the target language to examine whether the textbooks have appropriate text difficulty or whether they provided sufficient input in the target language. For example, Chen (2016) assessed the text difficulty of EFL textbooks commonly used in Taiwan, focusing on their lexical coverage and readability level. The words used in the textbooks were compared with British National Corpus (BNC) corpus-based frequency lists. The results revealed inappropriate progression of text difficulty in the textbooks. Crossley et al. (2007) compared language features such as the ratio of causal verbs and causal particles in two text corpora compiled from seven beginning ESL textbooks. The texts in one corpus were simplified, while those in the other were authentic. Their analysis showed that these texts differed significantly in the focal language aspects. Biber and Reppen (2002) compared the use of adjectives, participial adjectives, and nouns as nominal premodifiers in six popular ESL-EFL grammar textbooks with the Longman Grammar of Spoken and Written English (LGSWE) by Biber et al. (1999), which was assumed to represent the grammatical features of actual language use. They found that the textbooks covered only a few grammatical features included in LGSWE, that the grammatical topics were introduced in an inappropriate order in the textbooks, and that the word selection in the textbooks were problematic. Barbieri and Eckhardt (2007) focused on direct and indirect reported speech in ESL grammar and writing textbooks, and compared them with the TOEFL 2000 Spoken and Written Academic Language Corpus and the Longman Spoken and Written English Corpus. Like Biber and Reppen’s (2002) study, this study also showed insufficient discussion of the language features in the textbooks.

Language in English-Medium Academic Textbooks

Analysis of the language in English-medium academic textbooks mainly concerns the language demands imposed on the students. Previous studies have analyzed the genre of the texts, the use of rhetorical figure of speech and visual images, and coverage and frequency of difficult words. For example, Maxwell-Reid and Lau (2016) examined the language support EMI science textbooks gave to facilitate students’ learning of scientific knowledge. Three series of science textbooks that were commonly used in Hong Kong secondary EMI schools were analyzed and compared. The results showed that the three series of science textbooks were different in terms of the difficulty of the language they used. This different text difficulty was reflected in different genres used to deliver the same topic, different roles played by the analogies in the construction of technicality, and differing extents of the use of images to convey meaning. Hu and Gao (2018a) also discussed the text difficulty of EMI science textbooks used in secondary EMI schools in Hong Kong. Through the lens of systemic functional linguistics, their analysis focused on the use of abstract nouns, complex lengthy nouns, and lengthy nouns without verbs, aiming to examine the abstraction and information density of the language used in the textbooks. The results revealed that the language used in the textbooks was problematic, because difficult language features were commonly used and were not introduced progressively. Despite this, techniques were suggested for used in the textbooks to simplify their language, including noun replacement, de-nominalisation, information unpacking, and information reduction. Hsu (2011) examined the vocabulary size of 48 English-medium business course books, and the vocabulary demands of different business subjects. The study also compared the vocabulary thresholds of the textbooks with those of business research articles. The results showed that 95% of the words in business textbooks come from the most frequent 3,500 word families and proper nouns, and the vocabulary demand of a business textbook is less than that of a research article. However, the study also revealed that the vocabulary demand varied across subjects.

English-Medium Academic Textbooks versus EFL/ESL/EAP Textbooks

As reviewed in sections 2.1 and 2.2, previous studies of textbook evaluation have focused on either EFL/ESL/EAP textbooks or English-medium academic textbooks. Only a few studies have compared these two types of textbooks in terms of their text difficulty. These studies contribute to a better understanding of whether students are well prepared to learn academic subjects with English as the medium of instruction.

For example, Wood and Appel (2014) examined the gaps between the textbooks used for first-year business and engineering courses in a Canadian university and the EAP textbooks used in the same university, with a focus on their use of multiword constructions. They found that the presentation of multiword constructions in the EAP materials was not strong enough to prepare the students to read business or engineering course materials. Similarly, Miller (2011) compared the textbooks used in several disciplines (i.e., the humanities, the social sciences, and the natural sciences) with university ESL textbooks, focusing on word coverage, word and sentence length, nominal modification features, and text readability. The study found that the percentage of academic vocabulary and the use of nominal modification were significantly different between the two textbooks corpora, but there were no differences in terms of their readability.

To our knowledge, no studies have compared the language in ESL/EFL and subject textbooks at lower educational levels, although an increasing number of schools use English as the medium of instruction in contexts such as Hong Kong, Malaysia, Indonesia, and Pakistan. To address this gap, this study examines the text difficulty of EFL textbooks and English-medium science textbooks used at Secondary 1 (or Grade 7) in Hong Kong. Informed by the analysis of the studies reviewed above, our analysis of text difficulty focused on two aspects: the coverage of words, and text readability. In addition, we were also informed by the findings of previous studies indicating that text difficulty may vary across disciplines (e.g., Hsu, 2011), and attempted to find out whether the text difficulty varied across scientific topics to further inform textbook design and pedagogical scheme development. The study answers the following research questions:

What is the coverage of academic and general words used in English-medium science textbooks and EFL textbooks at Hong Kong Secondary 1?

What is the coverage of academic and general words used in texts on different scientific topics?

Are the readability levels of the texts in English-medium science textbooks suitable for Secondary 1 students in Hong Kong?

Are the readability levels of the texts on different scientific topics suitable for Secondary 1 students in Hong Kong?

The Study

Method

The examination of the coverage of academic and general words in the focal sets of textbooks in this study was based on the calculation of frequencies of academic words and general words, and the evaluation of the readability levels of the texts involved a count of letters, syllables, and words as the initial step of calculation. Therefore, the study adopted a corpus-based approach to examine the text difficulty. Corpus-based approach is the most appropriate method for this study, as compared with conventional approaches, such as discourse analysis, it “is concerned not only with what is said or written, where, when and by whom, but how often particular forms are used” (Kennedy, 1998, p. 9). As Wood and Appel (2014) note, corpus-driven research has a tendency to examine the language features of specific registers, with the analysis of academic articles, classroom lectures, textbooks, and so on. Such studies are viewed as more consistent and reliable than conventional textbook analysis (Biber et al., 1998), and “have potential implications for materials design, selection and supplementation as well as instructional focus for reading classes” (Miller, 2011, p. 44).

The Corpora

In many secondary schools in Hong Kong, science-related subjects are taught with English as the medium of instruction, and English-medium science textbooks are used in these schools. The science textbook corpus in this study was composed of three series of English-medium textbooks that have dominated the Hong Kong Secondary 1 science textbook market (Maxwell-Reid & Lau, 2016). These included Mastering Science 1A and 1B published by Oxford, Understanding Integrated Science for the 21st Century Units 1 to 6 published by Aristo (hereafter referred to as “Understanding Integrated Science”), and Interactive Science 1A and 1B published by Longman and Pearson. A total of 123 texts were included in this corpus. The texts covered various genres, such as reading passages, instructions, exercises, and so on. To examine the language features as related to the topics, six sub-corpora were compiled according to the text topics (i.e., introducing science, living things, cells and human reproduction, energy, water, and matter as particles).

EFL textbooks for Secondary 1 published by the same publishers were compiled to form the English textbook corpus. The EFL textbooks were Oxford English 1A and 1B published by Oxford, Aristo Success 1A and 1B published by Aristo, and Longman Elect JS 1A and 1B and Longman Elect 1A and 1B, both published by Pearson. This corpus contained 165 texts on topics related to things happening in students’ everyday lives (e.g., Hong Kong people who stand out in their fields, geographical information about Hong Kong, popular foods, sports, etc.). The details of the textbooks and the texts are provided in Tables 1 and 2.

Table 1.

Details of Textbook Corpora.

Subject	Number of texts	Total running words
EFL	165	155,770
Science	123	195,659

Note. EFL = English as a foreign language.

Table 2.

Details of Sub-Corpora (Topics) in the Science Textbook Corpus.

Topic	Number of texts	Total running words
Introducing science	18	33,846
Living things	19	33,144
Cells and human reproduction	21	36,692
Energy	19	30,442
Water	25	40,050
Matter as particles	21	38,445

Data Analysis

The analysis of the textbook corpora focused on the coverage and complexity of words and sentences. Two word lists were used for the comparison: Coxhead’s (2000) Academic Word List (AWL) and the General Service List (GSL). The AWL contains 570 of the most frequently used word families in academic texts, while the GSL contains 2,000 words that represent the most frequent words in written English. The analysis first examined the coverage of the academic words in the two textbook corpora and the sub-corpora. As has been noted, academic words posed significant challenges to readers of academic texts as they “occur with lower frequency than general-service vocabulary does” (Coxhead, 2000). Calculating the coverage of the academic words in both texts in the EFL and science textbooks, the study examined whether the English-written science textbooks used a high proportion of words that students did not learn in English classes, as well as whether the academic word demands in the texts of science textbooks varied by topics. On the other hand, the science texts are not composed of academic words alone, but also contain general words. Using GSL, the study also examined the extent to which the general words for reading the texts in science textbooks were covered in EFL textbooks, and whether the coverage varied by science text topics. The coverage of the words in the AWL and GSL was computed with Vocabprofile, VP English v.2.9 (Cobb, 2002) (see Tables 4 and 5).

It is assumed that “longer words may demand more decoding of inflections, and longer sentences provide space for more clauses and intricate clausal and phrasal embedding” (Miller, 2011, p. 36), and thus they are ‘more difficult to deal with from an English language learner’s perspective’ (Chen, 2016, p. 68). A number of formulas have been developed to measure text readability from different perspectives, taking into account average word length at phonological level (number of characters/number of words; number of syllables/number of words) and/or average sentence length at grammatical levels (number of words/number of sentences). To examine the readability levels of English-medium science textbooks, the study employed multiple formulas that are widely used to score complexity or readability from various perspectives (Chen, 2016; Miller, 2011). The formulas include the Automated Readability Index (ARI) (average number of characters/letters per word and average number of words per sentence considered), Flesch Reading Ease (average number of syllables per words and average number of words per sentence considered), Gunning Fog (number of words of three or more syllables excluding proper nouns, combinations of easy words or hyphenated words, and two-syllable verbs made into three with -es and -ed endings), and SMOG (number of polysyllables in three samples of ten sentences considered). Readability Calculator (https://www.online-utility.org/english/readability_test_and_improve.jsp) was used to calculate the readability. The computed results (see Tables 6 and 7) produce approximate representations of the grade level that U.S. students would need to comprehend the text. We compared the results with the scores for Grade 7, bearing in mind that the texts comprehensible to Grade 7 students in the United States will probably be beyond the level of Grade 7 students in Hong Kong, because many students in the United States are English native speakers while Hong Kong students using the English-medium science textbooks are basically EFL learners. Higher scores indicate lower readability levels in the ARI, Gunning Fog, and SMOG, while the opposite is true for Flesch Reading Ease. The details of the readability formulas are listed in Table 3.

Table 3.

Details of the Readability Formulas.

Readability formula	Language features taken into consideration	Scores for US 7th grade students
ARI	Average number of characters/letters per words and average number of words per sentence	8
Flesch Reading Ease	Average number of syllables per words and average number of words per sentence	70.0–80.0
Gunning Fog	Average number of words of three or more syllables	7
SMOG	Number of polysyllables in 3 samples of 10 sentences	7

Independent sample t-tests were used to identify the differences between texts in EFL textbooks and those in science textbooks in terms of the target language features (i.e., academic word coverage, word and sentence lengths, and readability levels). A one-way analysis of variance (ANOVA) test was run to examine the differences across topics. All the data sets have been square rooted to ensure normality (Qiu, in press). Outliers have been excluded and assumptions for the one-way ANOVA test (e.g., homogeneity of variance) have been met.

Findings and Discussion

The following sections discuss the results of the analysis. These suggest that word coverage and readability levels were different between EFL and English-medium science textbooks and among different scientific topics. The coverage of academic and general words in the EFL and English-medium science texts is presented first, followed by the readability computation results. Qualitative data are provided in each section to illustrate and explain the statistical analysis.

Coverage of AWL and GSL Words: Comparison Between EFL and English-Medium Science Textbook Corpora

The coverage of AWL and GSL words in EFL texts and English-medium science texts was computed. The results show that 4.46% of the total running words in the science textbook corpus were from the AWL and 82.85% from the GSL, while the texts in the EFL textbook corpus contained 2.23% of their total running words from the AWL and 86.04% from the GSL. This indicates that the percentages of AWL words were low in both the science texts and the EFL texts, and those of GSL words were the opposite.

The low percentages of AWL words can be explained by the topics covered in the two textbook corpora. As described in Section 3.1, the texts in the science textbooks covered six science-related topics only, and thus the words that were not relevant to the six topics remained unused in the textbooks (e.g., academy, accommodate, bias, community, comprehensive, edit, ethic, income, fee, norm, publication, reject, etc.). The texts in the EFL textbooks were all about people, things and events that happen in everyday life. While they contained very few academic words, some words in AWL were used (e.g., link, drama, context, focus, feature, remove, physical, topic, task, etc.). The low coverage of AWL in both the EFL and science textbooks indicates that AWL may not be a suitable vocabulary learning material for beginning EMI academic learners. This supports calls for subject-related wordlists to be developed in order to facilitate students’ learning of subject content in the medium of English (e.g., Hsu, 2014).

The coverage of AWL and GSL words in the two textbook corpora were also compared. As shown in Table 4, the science texts used in the first year of secondary schools used significantly more AWL words than the EFL texts did, in common with the findings for textbooks used in universities (Miller, 2011). As for the GSL words, their coverage was significantly lower in the science texts compared to the EFL texts. These differences indicate that the English-medium science textbooks used a large number of academic words that the students had not been exposed before, but the general words used in the textbooks would probably have been learnt by them.

Table 4.

Means (SD), Independent-Sample t-Tests, and Effect Sizes for Coverage of AWL and GSL Words in EFL and Science Textbooks.

Word (%)	EFL texts	Science texts	t	p	Effect size (Cohen’s d)
AWL	1.42 (0.47)	2.04 (0.55)	10.164	.000**	1.21
GSL	9.27 (0.22)	9.10 (0.25)	−5.988	.000**	0.72

Note. Square root transformation has been conducted for all data sets. EFL = English as a foreign language; AWL = Academic Word List; GSL = General Service List.

means p < .01.

Coverage of AWL and GSL Words: Comparison Across Text Topics

To answer Research Question 2, the study compared the coverage of AWL and GSL in texts on different scientific topics. The results showed statistically significant differences among the topics in terms of both AWL and GSL; the texts on “energy” contained the most AWL words (9.56%) but the fewest GSL ones (76.89%), while the texts on “matter as particles” contained the fewest AWL words (3.08%) but the most GSL ones (86.73%). The means, ANOVA test results, and effect sizes are presented in Table 5.

Table 5.

Means (SD), ANOVA Test Results, and Effect Sizes for Coverage of AWL and GSL Words in Science Texts by Topics.

Word (%)	Introducing science	Living things	Cells and human reproduction	Energy	Water	Matter as particles	F	p	Effect size (partial η²)
AWL	1.99 (0.50)	1.89 (0.40)	2.34 (0.40)	3.04 (0.59)	1.86 (0.35)	1.71 (0.40)	19.565	.000**	.59
GSL	9.27 (0.12)	9.19 (0.18)	8.91 (0.26)	8.77 (0.22)	9.03 (0.20)	9.31 (0.19)	21.980	.000**	.50

Note. Square root transformation has been conducted for all data sets. ANOVA = analysis of variance; AWL = Academic Word List; GSL = General Service List

means p < .01.

A post hoc analysis of the topics in terms of their use of the words from AWL and GSL gives a clearer picture of this variation. The results indicate that the topics “energy” and “cells and human reproduction” in the textbooks were significantly more difficult than the rest of the topics, given the significantly larger coverage of AWL words and the smaller coverage of GSL words.

The results indicated different text difficulty in the English-medium science textbooks across topics. To explain this difference, we scrutinized the academic words used in the texts about “energy” that cover the most AWL words, as well as the “water” and “matter as particles” texts that cover the fewest. Extracts 1 to 4 are short paragraphs that were randomly selected from the science texts (two on “energy” and two on “matter as particles”) to illustrate which AWL words were included (the AWL words are underlined).

Energy

Extract 1

Have you heard the sound given out by the vibrating strings of a violin or the sound given out by the vibrating wings of a mosquito? When an object vibrates, it gives out sound. Sound is a form of energy. We call it sound energy. The stronger the object vibrates, the more sound energy it gives out.

Extract 2

When an electric toaster is switched on, heat is given out. In this process, electrical energy is converted to heat energy. This example shows us that energy can be changed or converted from one form to another. The process is called energy change or energy conversion. Let’s look at more examples of energy conversions below.

Matter as particles

Extract 3

A substance can exist in different states. It can also change from one state to another state. For example, when water (liquid state) is heated to 100 °C, it changes to steam (gas state); when the steam is cooled, it changes back to water. The processes involved in the changes of states of water are shown below.

Extract 4

Scientists found that matter is made up of lots of very tiny particles. Particles are too tiny to be seen with our naked eyes or even under a light microscope. Nowadays, scientists can obtain images of particles using an extremely powerful kind of electron microscope.

As shown in the above extracts, the topic terms (e.g., energy, matter, particle) were needed to describe the topics, and for this reason they were likely to be frequently used. Thus, the texts about the topics with AWL words as the topic words were likely to have higher AWL coverage than other texts. The fact that the topic term in the texts about “energy” (i.e., energy) was a word from AWL while those in the texts about “water” and “matter as particles” (i.e., matter, particle) were not partly explains the high AWL coverage in the former and the opposite in the latter. Despite the few AWL head words in Extracts 1 and 2 (i.e., energy, process, convert), the heavy repetition of the topic term (i.e., energy) led to the comparatively larger number of AWL running words in the two extracts (15), compared to the texts of “matter as particles,” which had only four AWL head words.

Readability Level: Comparison Between EFL and Science Textbook Corpora

Research Question 3 concerns the readability levels of the texts in science textbooks, which are indicated by the scores generated from readability formulas that concern word/sentence length (Miller, 2011). This section reports the results concerning the readability of texts in the EFL and science textbooks, in terms of their formula scores. Contrary to previous studies that showed no readability differences between university English textbooks and English-medium academic textbooks (e.g., Miller, 2011), the results in this study, as illustrated in Table 6, showed that the difference between EFL texts and science texts was statistically significant in terms of their readability level, according to all the calculation methods: ARI (EFL: 4.35; science: 5.16), Flesch Reading Ease (EFL: 70.67; science: 62.93), Gunning Fog (EFL: 7.86; science: 8.57), and SMOG (EFL: 8.71; science: 9.50). This suggests that language differences between registers may be greater at lower educational levels.

Table 6.

Means (SD), Independent-Samples t-Test Results, and Effect Sizes for Readability Levels of Texts in EFL and Science Textbooks.

Readability features	EFL texts	Science texts	T	p	Effect size (Cohen’s d)
ARI	2.07 (0.42)	2.25 (0.28)	4.156	.000**	0.50
Flesch reading ease	8.44 (0.45)	7.94 (0.36)	−9.981	.000**	1.23
Gunning fog	2.78 (0.26)	2.90 (0.21)	4.146	.000**	0.51
SMOG	2.95 (0.19)	3.07 (0.13)	5.931	.000**	0.74

Note. Square root transformation has been conducted for all data sets. EFL = English as a foreign language.

means p < .01.

The results of the calculation indicate that the readability level of the science texts was significantly lower than that of the EFL texts, taking into account different combinations of language aspects (i.e., number of characters/letters per word, number of syllables per word, number of words per sentence, number of trisyllables and polysyllables). Moreover, the readability level of the science texts was shown to be lower than the scores for Grade 7 students in the United States, calculated by Flesch Reading Ease (Hong Kong: 62.93 vs. US: 70–80), Gunning Fog (Hong Kong: 8.57 vs. US: 7), and SMOG (Hong Kong: 9.50 vs. US: 7), when the numbers of syllables in the words or sentences were considered. This indicates that our students, who were EFL learners, had to read texts that were more difficult than those given to the students in the United States, many of whom are English native speakers; this further shows the inappropriateness of the text difficulty in the Hong Kong textbooks. Examples of the most frequently used words with three or more syllables in the texts include “energy,” “particle experiment,” “temperature activity,” “apparatus microscope,” “laboratory solution,” “material density,” “electricity impurity,” “conversion characteristic,” “variable chemical,” “organism pollution,” “thermometer,” and “reproduction.”

The low readability levels of the science texts suggest that both English and academic teachers need to be aware of the language challenges for EMI students, work collaboratively to tailor learning materials to their students’ needs, and provide extra facilitation in and outside English reading classes. Meanwhile, bridging courses focusing on academic English features and literacy may also help. On the other hand, the designers of English academic textbooks should pay more attention to the language use and simplify it in certain ways (Hu & Gao, 2018a).

Readability Level: Comparison Across Text Topics

To answer Research Question 4, the study compared the readability levels of the texts on different scientific topics. The results show a significant difference among the topics in terms of readability levels according to Flesch Reading Ease, Gunning Fog, and SMOG; the texts on “energy” had the lowest readability level (see Table 7), indicated by the mean scores of Flesch Reading Ease (58.32), Gunning Fog (10.56), and SMOG (10.56), while the texts on “matter as particles” had the highest readability level, with mean scores of Flesch Reading Ease (67.39), Gunning Fog (7.42), and SMOG (9.12). Nevertheless, despite the highest readability level of “matter as particles” among the selected science texts, its readability was still lower than that of the texts aimed at Grade 7 in the United States.

Table 7.

Means (SD), ANOVA Test Results, and Effect Sizes for the Readability in Science Texts by Topics.

Readability features	Introducing science	Living things	Cells and human reproduction	Energy	Water	Matter as particles	F	p	Effect size (partial η²)
ARI	2.23 (0.31)	2.25 (0.28)	2.24 (0.23)	2.21 (0.34)	2.34 (0.23)	2.20 (0.24)	.889	.491	.03
Flesch Reading Ease	7.99 (0.30)	8.03 (0.34)	7.90 (0.43)	7.67 (0.42)	7.79 (0.28)	8.21 (0.23)	6.257	.000**	.23
Gunning Fog	2.92 (0.16)	2.86 (0.16)	2.99 (0.17)	3.10 (0.16)	2.87 (0.13)	2.72 (0.16)	11.991	.000**	.43
SMOG	3.05 (0.12)	3.03 (0.12)	3.09 (0.11)	3.17 (0.10)	3.05 (0.10)	3.02 (0.13)	4.075	.002**	.27

Note. Square root transformation has been conducted for all data sets. ANOVA = analysis of variance.

means p < .01.

The complexity of the topic words greatly contributes to the readability levels of the texts. An in-depth analysis of the texts about “energy” shows that trisyllables and polysyllables (e.g., “energy,” ‘conversion “generate,” ‘electricity’) were used in the chapter title “Energy” and its sub-section titles such as “Energy in our daily life,” ‘Energy conversions “Generating electricity,” ‘Energy sources and us “Some forms of energy,” ‘Some common energy changes “How electricity is generated,” etc. As the topic words, these words were repeatedly used throughout the chapter. For example, when the types of energy were introduced, the word “energy” was contained in all its hyponyms such as “heat energy,” ‘light energy “sound energy,” ‘kinetic energy “electrical energy,” ‘chemical energy,’ and “potential energy.” In the same vein, the section about “generating electricity” involved a discussion of electricity supply, electricity companies, how electricity is generated in power stations, generators in power stations, driving a generator etc., where the topic words “generating” and “electricity” and their variations such as “generate,” ‘generator’ and “electrical” were repeatedly used. The calculation of word frequency in the texts indicates that the word families of “energy” and “electricity” were the most frequently used content words (1,491 times for “energy” and 462 times for “electricity”). The analysis of the texts of “matter as particles” shows similar results. The topic word “particle” was the second most frequently used content word (485 times). Unlike “energy,” the texts about “matter as particles” did not involve hyponyms containing “particle,” which explains why “particle” appeared much less frequently than “energy.”

A post hoc analysis of the topics in terms of their Flesch Reading Ease scores shows that the score for “energy” was significantly lower than most of the other topics (i.e., “introducing science,” ‘living things “cells and human reproduction,” and “matter as particles”), which means that it was more difficult to read than other topics according to a consideration of word length (average number of syllables per word) and sentence length (average number of words per sentence). The Gunning Fog scores indicated that the texts about “matter as particles” were significantly less difficult than the other topics when the average number of words with three or more syllables was taken into consideration; the texts on “energy” were still significantly more difficult than all the other topics except “cells and human reproduction.” The SMOG scores again show the difficulty of the topic “energy,” since its SMOG score was significantly higher than all the other topics. This indicates that the texts on “energy” contained many more polysyllable words in 3 samples of 10 sentences than other texts did.

Conclusion

Corpus-driven research has a tendency to examine the language features of specific registers, with the analysis of academic articles, classroom lectures, textbooks, and so on. (Wood & Appel, 2014). Such studies, as Miller (2011) notes, “have potential implications for materials design, selection and supplementation as well as instructional focus for reading classes” (p. 44). Learning materials for learners for whom English is a foreign language and who are in the early stage of academic learning attract particular attention, since it is assumed that textbooks are extremely important learning resources at the initial stage of academic learning (Wood & Appel, 2014). The rise of English as the medium of instruction in non-Anglophone areas should encourage the examination of EMI learning materials.

Previous studies have contributed to our understanding of the nature of the academic registers used in university textbooks (e.g., Miller, 2011; Parodi, 2010; Wood & Appel, 2014). Our study shifted the focus from the language used in university textbooks to that used in secondary school textbooks, to further enrich our understanding of the text difficulty posed by English-medium textbooks used in secondary EMI schools. We aimed to gauge the similarities and differences in terms of the difficulty between different types of texts, and discuss the implications of these differences for textbook design and pedagogical support. The study has concluded that an inappropriate level of AWL words was used in the science textbooks, and that the readability levels of the science texts were unsuitable for the EFL learners. In addition, the coverage of AWL and GSL words and the text readability levels varied across topics. Although our analysis focused on the textbooks used in a particular context, Hong Kong, we also suggest possible problems regarding subject content textbook design in wider EMI contexts. The study recommends that English-medium academic textbook designers should pay more attention to the language use in textbooks, and collaborative work between English teachers and academic subject teachers should be conducted to develop school-based learning materials that are suitable for EMI learners in terms of both content and language. Additional support for academic English learning may also help.

We acknowledge that the study has limitations. First, although AWL has often been used as a match list when textbook corpora were examined (e.g., Miller, 2011), the low coverage of AWL in both EFL and science textbooks in our study indicates that it may not be an ideal tool for the evaluation of science textbooks or a suitable learning material for beginning EMI science learners. In addition, the percentages shown in our study provide only a surface-level idea of the words used in the textbooks. We therefore call for subject-related wordlists as tools for the evaluation of textbooks used in EMI contexts, and further analysis of vocabulary demands in future studies (e.g., type/token ratio). Second, the corpora in our study were compiled from textbooks used in Hong Kong only. Given the small corpus size, we suggest that duplication studies should be conducted to examine textbooks used in other EMI contexts. Third, our study only focused on the text difficulty of the textbooks designed for EMI students. Although the analysis of the language used in these textbooks helps us to understand students’ EMI learning challenges, more actions are needed to achieve effective EMI learning. Future studies may further discuss the pedagogical practices that facilitate students to read difficult subject-related language more easily.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work has been generously supported by Standing Committee on Language Education and Research (SCOLAR) (Hong Kong) (EDB(LE)/P&R/EL/164/19), the Guangdong Provincial Social Sciences Research Fund (Grant No.: GD20ZD27), and Tongshan Social Sciences Research Project funded by Sun Yat-sen University (Grant No.: Tongshan 201901).

ORCID iDs

Jingjing Hu

Xuesong(Andy) Gao

Xuyan Qiu

References

Barbieri

Eckhardt

S. E. B.

(2007). Applying corpus-based findings to form-focused instruction: The case of reported speech. Language Teaching Research, 11(3), 319–346. http://doi.org/10.1177/1362168807077563

Biber

Conrad

Reppen

(1998). Corpus linguistics: Investigating language structure and use. Cambridge University Press.

Biber

Johansson

Leech

Conrad

Finegan

(1999). Longman grammar of spoken and written English. Longman.

Biber

Reppen

(2002). What does frequency have to do with grammar teaching? Studies in Second Language Acquisition, 24, 199–208. http://doi.org/10.1017/S0272263102002048

Chen

A. C.

(2016). A critical evaluation of text difficulty development in ELT textbook series: A corpus-based approach using variability neighbor clustering. System, 58, 64–81. http://doi.org/10.1016/j.system.2016.03.011

Cobb

(2002). Web vocabprofile. http://www.lextutor.ca/vp/, an adaptation of Heatley, Nation & Coxhead’s (2002) Range.

Coxhead

(2000). A new academic word list. TESOL Quarterly, 34(2), 213–238.

Crossley

S. A.

Louwerse

M. M.

McCarthy

P. M.

McNamara

D. S.

(2007). A linguistic analysis of simpliﬁed and authentic texts. The Modern Language Journal, 91(1), 15–30. http://doi.org/10.1111/j.1540-4781.2007.00507.x

Dearden

(2014). English as a medium of instruction – A growing global phenomenon. University of Oxford. http://doi.org/10.1080/0013191610140107

10.

Fang

(2005). Scientific literacy: A systemic functional linguistics perspective. Science Education, 89, 335–347. http://doi.org/10.1002/sce.20050

11.

García

J. R.

Montanero

Lucero

Cañedo

Sánchez

(2018). Comparing rhetorical devices in history textbooks and teachers’ lessons: Implications for the development of academic language skills. Linguistics and Education, 47, 16–26. http://doi.org/10.1016/j.linged.2018.07.004

12.

Harwood

(2005). What do we want EAP teaching materials for? Journal of English for Academic Purposes, 4, 149–161. http://doi.org/10.1016/j.jeap.2004.07.008

13.

Hsu

(2011). The vocabulary thresholds of business textbooks and business research articles for EFL learners. English for Specific Purposes, 30, 247–257. http://doi.org/10.1016/j.esp.2011.04.005

14.

Hsu

(2014). Measuring the vocabulary load of engineering textbooks for EFL undergraduates. English for Specific Purposes, 33, 54–65. http://doi.org/10.1016/j.esp.2013.07.001

15.

Gao

X. A.

(2018a). Linguistic demands in English-language science textbooks in Hong Kong. The Asian Journal of Applied Linguistics, 5(1), 170–180.

16.

Gao

X. A.

(2018b). Self-regulated strategic writing for academic studies in an English-medium-instruction context. Language and Education, 32(1), 1–20. http://doi.org/10.1080/09500782.2017.1373804

17.

Gao

X. A.

(2020). Appropriation of resources by bilingual students for self-regulated learning of science. International Journal of Bilingual Education and Bilingualism, 23(5), 567–583. http://doi.org/10.1080/13670050.2017.1386615

18.

Gao

X. A.

(2021). Understanding subject teachers’ language-related pedagogical practices in content and language integrated learning classrooms. Language Awareness, 30(1), 42–61. https://doi.org/10.1080/09658416.2020.1768265

19.

Jaekel

(2020). Language learning strategy use in context: the effects of self-efficacy and CLIL on language proficiency. International Review of Applied Linguistics in Language Teaching, 58(2), 195–220. https://doi.org/10.1515/iral-2016-0102

20.

Kennedy

(1998). An introduction to corpus linguistics. Pearson Education.

21.

Y. Y.

E. S. C.

(2014). A meta-analysis of the effectiveness of English-medium education in Hong Kong. Review of Educational Research, 84(1), 47–73. http://doi.org/10.3102/0034654313499615

22.

Y. Y.

Macaro

(2012). The medium of instruction and classroom interaction: Evidence from Hong Kong secondary schools. International Journal of Bilingual Education and Bilingualism, 15(1), 29–52. https://doi.org/10.1080/13670050.2011.588307

23.

Lorenzo

(2017). Historical literacy in bilingual settings: Cognitive academic language in CLIL history narratives. Linguistics and Education, 37, 32–41. http://doi.org/10.1016/j.linged.2016.11.002

24.

Mahboob

(2014). Meeting the challenges of English-medium higher education in Hong Kong. IRAL—International Review of Applied Linguistics in Language Teaching, 52(2), 183–203. http://doi.org/10.1515/iral-2014-0008

25.

Maxwell-Reid

Lau

(2016). Genre and technicality in analogical explanations: Hong Kong’s English language textbooks for junior secondary science. Journal of English for Academic Purposes, 23, 31–46. https://doi.org/10.1016/j.jeap.2016.05.005

26.

Miller

(2011). ESL reading textbooks vs. University textbooks: Are we giving our students the input they may need? Journal of English for Academic Purposes, 10(1), 32–46. http://doi.org/10.1016/j.jeap.2010.12.002

27.

Moreno

A. I.

(2003). Matching theoretical descriptions of discourse and practical applications to teaching: The case of causal metatext. English for Specific Purposes, 22, 265–295. https://doi.org/10.1016/S0889-4906(02)00021-2

28.

Parodi

(2010). The rhetorical organization of the textbook genre across disciplines: A “colony-in-loops”? Discourse Studies, 12(2), 195–222. http://doi.org/10.1177/1461445609356500

29.

Qiu

(in press). Picture or non-picture? The influence of narrative task types on lower- and higher-proficiency EFL learners’ oral production. International Review of Applied Linguistics in Language Teaching. https://doi.org/10.1515/iral-2017-0094

30.

Roussel

Joulia

Tricot

Sweller

(2017). Learning subject content through a foreign language should not ignore human cognitive architecture: A cognitive load theory approach. Learning and Instruction, 52, 69–79. http://doi.org/10.1016/j.learninstruc.2017.04.007

31.

Wood

D. C.

Appel

(2014). Multiword constructions in first year business and engineering university textbooks and EAP textbooks. Journal of English for Academic Purposes, 15, 1–13. http://doi.org/10.1016/j.jeap.2014.03.002