Are you SLiM? Developing an instrument for civic scientific literacy measurement (SLiM) based on media coverage

Abstract

The purpose of this study is to develop an instrument to assess civic scientific literacy measurement (SLiM), based on media coverage. A total of 50 multiple-choice items were developed based on the most common scientific terms appearing in media within Taiwan. These questions covered the subjects of biology (45.26%, 22 items), earth science (37.90%, 19 items), physics (11.58%, 6 items) and chemistry (5.26%, 3 items). A total of 1034 students from three distinct groups (7th graders, 10th graders, and undergraduates) were invited to participate in this study. The reliability of this instrument was 0.86 (KR 20). The average difficulty of the SLiM ranged from 0.19 to 0.91, and the discrimination power was 0.1 to 0.59. According to participants’ performances on SLiM, it was revealed that 10th graders (Mean = 37.34±0.23) performed better than both undergraduates (Mean = 33.00±0.33) and 7th graders (Mean = 26.73±0.45) with significant differences in their SLiM.

Keywords

assessment media scientific literacy term extraction textbooks

1. Introduction

There have been long-standing debates about how to create a meaningful science education platform for all citizens. Douglas Roberts (2007) discussed the two “visions” of science education. Vision one focuses only on the education of future scientists (i.e. those who will pursue careers in science or engineering). This vision pays little attention to what the majority of students need to know about science and whether or not those needs are being met. The second vision, according to Roberts, is one of a science education designed for all citizens and not merely for those who will pursue science-based careers. As in the first vision just mentioned above, the resulting science curricula in most countries consist of watered-down versions of curricula originally designed to prepare only a minor part of the age group for future academic studies (Millar, 2008). The second vision presented better reflects the goal of creating a science education that is meaningful for all individuals. The question that follows is: how do we begin to create a science education that promotes civic scientific literacy?

People in modern society come into contact with knowledge about science and technology through various media. DeBoer (2000) states that a scientifically literate citizen ought to have the ability to critically read and discuss scientific reports presented in the media. The importance of the media must be taken into account as it relates to the goal of scientific literacy. Accordingly, when evaluating civic scientific literacy, the scientific concepts taught in school and discussed in the media must both be emphasized. Hence, in this study, scientific terms conveyed in both science textbooks and news media serve as the foundation to develop a civic scientific literacy measurement (SLiM) based on media coverage.

In this section, aspects of civic scientific literacy and existing instruments to measure civic scientific literacy are introduced. In the following sections, the development of SLiM is presented and the differences between SLiM and other instruments are discussed.

Aspects of civic scientific literacy

There are a plethora of definitions of scientific literacy (e.g. DeBoer, 2000; Laugksch, 2000; Murcia, 2009). Some authors question whether or not the concept of scientific literacy has any real meaning and some think of the term as merely an empty slogan (e.g. Bybee, 1997); the term has been used in different ways since its introduction in the USA in the late 1950s. Today, “Scientific Literacy” is a term used to describe what the ordinary individual may need to know in order to be an active member in modern society.

One of the most influential authors in the field of scientific literacy is Jon Miller, who regards scientific literacy as a multidimensional construct. Miller (1983) divides scientific literacy into three main aspects: (1) an understanding of the norms and methods of science (i.e. the nature of science), (2) an understanding of key scientific terms and concepts, and (3) an awareness and understanding of the impact of science and technology on society. Most research on scientific literacy conducted up until now has focused on the second aspect, that is to say the study of individuals’ understanding of scientific terms and concepts. Some research has been done on the first aspect of the nature of science, but virtually no studies have been compiled on the third aspect. Following the same trend, Shamos (1995) has suggested three forms of scientific literacy with a connection to society. The first form, “cultural scientific literacy,” relates to possessing a “lexicon” of scientific terms, and being able to read and understand science-related news that appears in the media. The second form, “functional scientific literacy,” is more active than the former one and requires the individual to be able to converse, read and write coherently in a non-technical but meaningful context – that is, the individual must be able to communicate the essence of science-related news to others. The third form, “true scientific literacy,” is according to Shamos more difficult to attain. It requires an individual to be aware of the major theories of science, how these were arrived at, and why they are widely recognized. Furthermore, the truly scientifically literate individual should be able to understand something about the nature of science and the ways scientists pose questions, conduct inquiries, and arrive at conclusions (Shamos, 1995). The third form is similar to the first aspect mentioned by Miller (1983). According to the above-mentioned definitions of scientific literacy, we find that a major aspect of civic scientific literacy is people’s understanding of scientific terms and related concepts in the context of society, especially the scientific terms/concepts presented in news media. This is a guiding principle for the development of SLiM in this study.

How to measure civic scientific literacy?

Miller (1998) points out that the development of a test instrument designed to cover a conceptual core understanding of scientific literacy would serve as a valid means of measuring civic scientific literacy. Miller takes his starting point as identifying a set of basic constructs, such as DNA and atomic structure, which are intellectual foundations for reading and understanding contemporary issues. From this set of basic constructs, which have been used also for cross-national studies of scientific literacy, test items are constructed. The general aim of Miller’s approach to study scientific literacy is to give a measurement of people’s understanding of a basic core of conceptual knowledge about science. The results from the past investigations have been documented for decades (Science and Engineering Indicators, 2008).

Miller (1998) points out the potential danger of testing issues that are widely discussed currently, but which might be of less importance as a part of a general scientific literacy in the future. He gives the example of the test items provided and constructed by the National Association of Science Writers in the USA, in the first attempt to provide a measure of scientific literacy in 1957. In the test, the major items relating to issues that were of great concern at that time, such as the polio vaccination and fluoridation of water, soon proved to be outdated.

A study of scientific literacy in biology by Lord and Rauscher (1991) takes a similar approach to Miller’s; the report takes a conceptual core of scientific understanding as the starting point for their measurement. Interestingly, they developed their test instrument, measuring this conceptual core, from primary and middle school science textbooks, and test a population of students at different stages of their college education. However, the detailed selection process of scientific terms was not clearly described in their study.

Measuring scientific literacy from media

A different way to measure scientific literacy has been presented by Brossard and Shanahan (2006). They highlight the problem that past studies of scientific literacy inevitably have taken their starting point from what experts’ think the scientifically literate person should know. Measures that rely on, what Brossard and Shanahan call “ideal” knowledge are prone to have a problem of subjectivity. As experts from different fields of science may have diverging views on what terms should be included in a scientific literacy test, it is, according to Brossard and Shanahan, possible that “the biases and prejudices of the scientific community can influence the overall definition of literacy” (2006: 50). Instead of using experts’ selection of content to test, Brossard and Shanahan use the 31 most frequently occurring scientific terms (from a list of randomly selected terms from a scientific dictionary, the Oxford Dictionary of Science) from media to construct a media-based scientific literacy test.

The idea of using the media as a source providing ideas for developing science education and testing scientific literacy is, however, not new. Long before the era of computers, Hopkins (1925) analyzed 2770 articles, and concluded that biology content was more prevalent in newspapers and magazines than content from the other science subjects. The same tendency was found by Martin (1945), who also suggested that these results might be taken into account when reorganizing the science curriculum. Interestingly, the editor of the journal in which Martin’s article appeared cautioned against using content analyses of media to make educational decisions. According to the editor’s note to Martin’s article, the subject itself should determine the content of the curriculum, even if analyses of the science content in newspapers might point to some desirable additions to the curriculum (Martin, 1945). In the 1960s, Koelsche (1965) made an attempt to extract scientific terms from the media that citizens should comprehend; if they understood these terms they could be qualified as being “scientifically literate.”

In summary, civic scientific literacy is a multidimensional concept, which could and should be approached and measured from several different viewpoints. The viewpoint of defining and measuring an understanding of a conceptual core of science is one model. Another important aspect is from the opinion of experts in different scientific fields as well as that of science educators. In this study, we focus on the viewpoint of how the occurrences of scientific terms in the media can aid us in constructing instruments to measure civic scientific literacy; this view corresponds to Miller’s second aspect of scientific literacy (1983) about understanding key scientific terms and concepts. Through the use of modern computer software, we are able to scan the media to find the most frequently used scientific terms during different time periods as well as to make continuous updates on current terms. Through frequently updated computer scans, we can avoid the problem highlighted by Miller (1998), that topics used to test scientific literacy might fall out of public interest and become outdated.

When using science in the media, it can be valuable to reflect upon the fact that the intentions behind publishing news related to science in a newspaper might be very different from the intentions in the minds of textbook writers and curriculum designers. However, mapping these sources together may provide us with a fresh view of what scientific terms are considered prevalent in present-day society.

The current study – SLiM (civic scientific literacy measurement based on media coverage)

In this article we present a way to measure the public’s SLiM (Civic Scientific Literacy Measurement Based on Media Coverage). SLiM presents a measurement by detailing which scientific topics are most often discussed in the media. It is of value to note that this is a system which can and will constantly update itself.

An approach based on the most frequently occurring terms in the media, similar to that presented by Brossard and Shanahan (2006), was adopted to construct our instrument to test civic scientific literacy. Brossard and Shanahan pointed out their instrument was “dry” and it would be better to select the terms outside of the conventionally accepted dictionary definitions. Furthermore, a difference between the way science textbooks were used in the study by Lord and Rauscher (1991) and the SLiM approach presented here is that the terms selected to be used in the SLiM instrument were screened from the most common scientific terms appearing in the media.

Accordingly, we have revised the methodology by defining the scientific terms as terms present in the indexes of Taiwanese junior high science textbooks and mapping them with the terms most frequently appearing in newspaper articles (see appendix,¹ SEM 1). In addition, we constructed the test items by giving a context with co-occurring scientific terms in the same news articles. This system also employed daily life contexts to develop test items in this study.

Aim and research questions

The purpose of this study is to develop and validate an instrument for investigating civic scientific literacy based upon scientific terms occurring in the media. SLiM instruments can be constructed in different countries with different science curricula, and it has the advantage of being an updatable system. Thus SLiM can form a complement to existing ways of evaluating scientific literacy. As mentioned previously, scientific literacy can be evaluated according to different aspects. Here, we focus on evaluating individuals’ familiarity with scientific terms and concepts presented in the media, and their ability to apply these terms correctly in daily life contexts which appear consistently in newspaper articles.

Based upon the above, three research questions are asked, which are: (1) What are the most common scientific terms presented in Taiwanese news media? (2) What are the degrees of item difficulty and what is the discrimination power of SLiM? (3) What are the differences in performance on SLiM between the different groups participating in this study?

2. Methods

Construction of the instrument

In the following section, the way we extracted scientific terms to develop SLiM and the format of SLiM are described.

Term extraction

The basic idea for developing the instrument to investigate the public’s SLiM is to find scientific terms from both textbooks and news media. The scientific terms taken from textbooks were based upon the complete indexes of four major Taiwanese junior high textbooks (2001 version). The books were from four different publishers and the indexes were provided by the National Institute for Compilation and Translation in Taiwan. The topics included nature, life, and technology. A total of 3657 index terms (or 2037 terms if only distinct ones are counted) were listed from the textbooks.

The media terms were based on the complete news reports (covering all subjects) from a collection of four Taiwanese newspapers including United Daily News, United Express, Ming Hseng News, and Economic Daily News, which are all from the United Daily News Group in Taiwan during the years 2001 to 2002 (i.e. corresponding to the time of the printing of the textbooks). This newspaper collection has been widely used by many research groups in Taiwan, due to its comprehensiveness (two years of daily news stories, more than 1000 each day on average). Only key terms that were repeated at least twice in a news document were extracted based on an automatic keyword extraction algorithm (Tseng, 1998). These include dictionary terms (covering over 120,000 terms) and out-of-vocabulary terms (those not in the dictionary). In total, 901,446 documents starting from 1 January 2000 to 31 December 2001 were scanned, and 1,082,937 terms (including scientific and non-scientific terms) were extracted. To better understand each keyword’s context, we also extracted what we call “related terms” for each keyword. They were identified based on whether they co-occurred often with their keyword in the same sentences (Tseng, 2002). After filtering those without related terms (appendix, SEM 2), 323,918 keywords remained; these keywords were normally topic-relevant or context-positive terms. In contrast, those removed terms were normally rare terms or outliers. Finally, the keywords and their related terms were matched with the 2037 terms from the textbook indexes. This resulted in a list of 876 keywords. The detailed process is presented in the appendix, SEM 3.

One assumption of this study was that the higher the frequency of scientific terms presented in news media, the more frequently people may have discussed related issues during that period of time. Accordingly, those term-related issues were hot topics and important in Taiwan. To obtain the final terms for SLiM instrument development, the most frequent 100 keywords (each occurred in at least 2950 news articles) and their related terms were examined by four experts. After excluding non-scientific terms (e.g., e-mail, computer, and so on), 95 terms (including keywords and their related terms) were selected. To present these terms and their relations in a compact form, for use by experts to develop the SLiM instrument, they were visualized by a visualization tool (de Nooy, Mrvar and Batagelj, 2005) in a term network (concept map) as shown in Figure 1. Note that a link in the network denotes that the two terms at the ends of the link co-occur often in the same sentences and their exact relation represented by this link can be explained by the co-occurring sentences retrieved from the news articles through a search engine which indexed the whole news corpus.

Figure 1.

A term network (concept map) of SLiM.

The format of SLiM

Different formats and assessments have been developed for the purpose of evaluating scientific literacy worldwide. To develop the SLiM instrument, we decided to skip fill-in the blank items, as this would make data processing slower and also possibly give rise to problems of comparability in future cross-national studies. We wanted to prevent our participants from simply guessing true- or false-items. Thus, the multiple-choice format was chosen as the most suitable test design to explore civic SLiM.

For each of the terms we developed a test item, consisting of a multiple-choice question with four alternatives, the last alternative being “I don’t know” for each question. The test persons were asked to click the alternative “I don’t know” rather than simply guess the answer of a question if they had no idea about which alternative was the correct choice. The rationale for this was to avoid students clicking the correct alternative by pure chance.

As the terms were identified from occurrence in the media, the test items were developed with the intention to provide a context of daily life-related questions. We also questioned whether the reader could understand the meaning and use of a term in a daily life context. The most commonly co-occurring terms were used to provide multiple-choice alternatives. However, if some terms were too difficult to develop as a context-based item, we developed the items as conceptual items instead. Hence, two groups of items were developed, which included conceptual and contextual items. Examples of conceptual and contextual items are as follows.

Example: Conceptual Item

Fill in the correct order from smaller to bigger:

□ Molecules-Cells-Atoms

□ Molecules-Atoms-Cells

□ Atoms-Molecules-Cells

□ I don’t know

In this item, the keyword “molecule” was to be combined with the co-occurring terms “atom” and “cell.” These terms refer to basic concepts in science, which may be of importance in daily life. The terms are seen as a background knowledge that can help to structure information, rather than having any direct application. The question about scale relationship could therefore be viewed as a testing of basic conceptual knowledge rather than a testing of the ability to apply this knowledge in a daily life-related context.

Example: Contextual Item

Why do we recommend consumers to use vegetable oils, like olive oil, instead of oil from animals, such as butter?

□ Because it is natural

□ Because it contains unsaturated fatty acids

□ Because it contains saturated fatty acids

□ I don’t know

In this item, the keyword “structure” was to be combined with the co-occurring term “fatty acid.” In this example, there was a discussion in the media about how we should choose different food products relating to their content of saturated and unsaturated fatty acids; thus, we created a contextually based test item with connection to this discussion. We also did inter-rater reliability for coding the conceptual and contextual items by four professors and one master program student. The Kappa value was 0.639–0.701. All test items of the SLiM instrument are presented in the appendix, SEM 7.

The limitations of the SLiM instrument

Our intention was to develop the SLiM instrument using the most commonly occurring scientific terms in both media and textbooks. However, this strategy gave unequal numbers of scientific terms for the different subject areas of science. This unequal distribution must be kept in mind while evaluating the results from the different subject areas. More items could be added in future studies from areas that are not well-represented in media coverage (i.e. chemistry and physics).

The format of multiple-choice items makes larger sample sizes possible. At the same time, it gives rise to limitations. For instance, it is hard to test important aspects of scientific literacy such as reflective thinking and the ability to actively use scientific concepts in reasoning and argumentation with simple multiple-choice alternatives. However, we hope to overcome these limitations in future studies, by including interviews and other methods that might probe deeper into these crucial aspects of scientific literacy.

Validation of the instrument

The test items were sent out for validation by nine experts in different areas of science: three from life science, one from physics and chemistry each, and four from earth science. The experts we invited in this study were asked to revise the test items of SLiM to make the items more understandable and to clarify terms when possible. The experts also checked the correctness of science concepts, added/deleted/revised options of each item and its alignment with the key terms and co-occurring terms. The items were revised according to the experts’ comments before this pilot study.

Pilot test of the instrument

Participants

The basic idea of the SLiM study is to test Taiwanese citizens’ scientific literacy, defined as their familiarity with the scientific content presented in daily media. In this pilot study, we adopted a convenience sampling approach instead of a random sampling and invited groups of 7th graders, 10th graders and undergraduates to represent the citizens who had received an elementary degree, a junior high degree and a senior high degree, respectively. These three groups were utilized to examine the reliability of the SLiM test. Among these participants, the undergraduates were mainly non-science majors at university. All the participants were from northern Taiwan.

A total of 1034 participants answered the SLiM questionnaires, and 954 (92.3%) people were a valid sample. Among these 954 participants, there were 335 seventh graders, 352 tenth graders and 267 undergraduates. The detailed distribution of participants is presented in the appendix, SEM 4.

Data collection and analysis

After inviting participants from different groups and schools, we sent SLiM questionnaires to the teachers of each class. On average, it took students 40 minutes to finish responding to the SLiM items.

To answer the research questions developed in this study, we used KR 20 and Kappa methods (to show the reliabilities of the SLiM items). Furthermore, description statistics, ANOVA and Pearson correlation coefficients were conducted to analyze participants’ performance regarding SLiM. To test the correlation and dependence of the test items from different subject areas among different groups of students in this study, the Pearson correlation coefficient was used to check a linear relationship between two variables. All analyses were conducted using Statistical Package for Social Sciences (SPSS) version 12.0.

3. Results

The most common scientific terms presented in Taiwanese news media

In total, 95 scientific terms matched from textbooks and the media were chosen to develop SLiM. According to the distribution of terms, the subject of biology was represented by the highest number of terms (45.26%), earth science was the second (37.90%), and the subject areas of physics (11.58%) and chemistry (5.26%) had the lowest representations. In total, 50 items were generated. According to the distribution of terms of each subject field (appendix, SEM 6), the number of items relating to each subject area (biology 22, earth science 19, physics 6 and chemistry 3) mirrored the occurrence of scientific terms from the respective subject area in the media.

It was found that biological terms were very often related to health (e.g. diabetes) and the human body (e.g. heart). In earth science, the terms were often related to local phenomena, such as typhoons and earthquakes, which occur in Taiwan from time to time. Also, new scientific discoveries in biotechnology/genetic engineering were often presented. Environmental issues such as global warming, and celestial phenomena like comets and asteroids were regularly obtained in the area of earth science. In physics and chemistry, the terms were related to technical applications, such as diesel engines, batteries and thermal conductors.

Reliability, difficulty, discrimination power and correlations among the items

Regarding the reliability of SLiM, as shown in this pilot study, the value of reliability of SLiM was 0.86 (KR 20) from the whole 954 participants. Concerning the reliabilities of each group, these showed 0.87 for 7th graders (n = 335), 0.62 for 10th graders (n = 352), and 0.74 for undergraduates (n = 267). The average difficulty of the whole 50 items of SLiM ranged from 0.19 to 0.91, and the discrimination powers were 0.1 to 0.59.

According to the Pearson product-moment correlation coefficients, it was found that the items from the different subjects (biology, earth science, chemistry and physics) were all correlated with significance (r = 0.391 to 0.687, p < .01). The correlation among each subject to the total SLiM questionnaire was high as well (r = 0.567 to 0.912, p < .01). The detailed results are presented in Table 1.

Table 1.

Correlations of students’ performance in each subject (total participants n =954).

Subject	Group
	Biology	Physics	Chemistry	Earth science
Biology	1	–	–	–
Physics	0.523**	1	–	–
Chemistry	0.420**	0.391**	1	–
Earth science	0.687**	0.615**	0.448**	1
Total score (T)	0.892**	0.737**	0.567**	0.912**

Note: ** p < .01.

Moreover, Table 2 presents the Pearson correlation coefficients among three different groups of participants. They all show significant correlations among the different subjects, apart from the 10th graders’ performance in chemistry and earth science (p > .05). In the group of 7th graders, the value of correlation was from 0.358 to 0.911 (p < .01). The Pearson correlations were also significantly high among 10th graders from 0.193 to 0.827 (p < .01), and the undergraduate group, r = 0.254 to 0.846 (p < .01).

Table 2.

Correlations of students’ performance among different subjects and groups of participants.

Subject	Group
	7th graders (n = 335)					10th graders (n = 352)					Undergraduates (n = 267)
	B	P	C	E	T	B	P	C	E	T	B	P	C	E	T
Biology (B)	1	–	–	–	–	1	–	–	–	–	1	–	–	–	–
Physics (P)	0.526**	1	–	–	–	0.313**	1	–	–	–	0.265**	1	–	–	–
Chemistry (C)	0.409**	0.358**	1	–	–	0.246**	0.193**	1	–	–	0.284**	0.254**	1	–	–
Earth science (E)	0.699**	0.606**	0.448**	1	–	0.340**	0.312**	0.099	1	–	0.529**	0.356**	0.360**	1	–
Total score (T)	0.904**	0.722**	0.553**	0.911**	1	0.827**	0.614**	0.398**	0.717**	1	0.846**	0.546**	0.505**	0.845**	1

Note: ** p < .01.

The participants’ performances on SLiM

The results of average scores on the SLiM test revealed that 10th graders performed best (Mean = 37.34±0.23), undergraduates were the second highest group (Mean = 33.00±0.33), and 7th graders were the last group (Mean = 26.73±0.45).

To understand whether the same trend could be seen also in the different subjects of biology, chemistry, physics and earth science, we analyzed the performances in the different groups of 7th graders, 10th graders and undergraduates further. Figure 2 shows that 10th graders performed best, undergraduates were second, and then 7th graders performed least well among all the four different subjects in SLiM. Hence, the similar trend of the performances among these three groups of participants was also found in all four subjects. The detailed scores are presented in Table 3.

Figure 2.

The average scores of the performances of the three groups of participants in each subject area and the percentages of correct answers (average score divided by number of items) for the four subject areas for the different groups of participants.

Table 3.

The average scores of all participants’ performances regarding SLiM.

Subject	Group
	7th graders (n = 335)					10th graders (n = 352)					Undergraduates (n = 267)					Scheffe test
	Mean	SD	Std error	95% confidence interval for Mean		Mean	SD	Std error	95% confidence interval for Mean		Mean	SD	Std error	95% confidence interval for Mean
				Lower bound	Upper bound				Lower bound	Upper bound				Lower bound	Upper bound
Biology	12.21	3.9	0.21	11.80	12.63	15.74	2.4	0.13	15.49	15.99	14.00	2.8	0.17	13.66	14.34	10th graders > 7th graders *
																10th graders > Undergraduates *
																Undergraduates > 7th graders*
Physics	3.08	1.5	0.08	2.92	3.24	4.70	1.2	0.06	4.58	4.83	4.41	1.2	0.07	4.27	4.55	10th graders > 7th graders *
																10th graders > Undergraduates *
																Undergraduates > 7th graders*
Chemistry	1.28	0.9	0.05	1.18	1.37	1.89	0.7	0.04	1.82	1.97	1.75	0.8	0.05	1.65	1.84	10th graders > 7th graders *
																10th graders > Undergraduates
																Undergraduates > 7th graders*
Earth science	10.16	3.6	0.19	9.77	10.54	15.01	1.8	0.10	14.82	15.20	12.84	2.4	0.15	12.54	13.13	10th graders > 7th graders *
																10th graders > Undergraduates *
																Undergraduates > 7th graders*
Total score	26.73	8.3	0.45	25.84	27.62	37.34	4.2	0.23	36.90	37.79	33.00	5.5	0.33	32.34	33.65	10th graders > 7th graders *
																10th graders > Undergraduates *
																Undergraduates > 7th graders*

Note: * p < .05; SD, standard deviation.

According to the ANOVA computing results, there were significant differences in the performances on SLiM among these three groups of participants (p < .05), except in the subject of chemistry. In chemistry, 10th graders performed better than 7th graders and undergraduates, but there was no significant difference between 10th graders and undergraduates (p > .05). Table 3 also shows the details of the Scheffe test.

After considering the different numbers of items in each subject area developed in SLiM, we could conclude that chemistry was the most difficult subject for all three groups. This conclusion was reached after dividing average scores by number of items in each subject (Figure 2). However, since there were only three items related to chemistry in the SLiM instrument, this might be of minor significance. The 10th graders performed best in earth science, and then physics. In contrast, undergraduates performed best in physics, and then earth science. The 7th graders performed best in biology, then in earth science. Overall, percentages of correct answers among all participants were highest in physics, followed by earth science, biology, and finally in chemistry. Detailed percentages of the performances in different subject areas in each group of participants are shown in the appendix, SEM 5.

4. Discussion and implications

Scientific literacy has been a central goal of science education for many years. The idea of developing scientific literacy tests based on scientific terms occurring in the media is not new, but has been discussed for several decades (e.g. Hopkins, 1925; Martin, 1945; Koelsche, 1965; Brossard and Shanahan, 2006). However, the use of contemporary computer software and computational power breathes new life into the idea of screening media (to extract what scientific topics are currently in the media). It is our firm belief that updated screenings of scientific concepts presented in the media can give valuable and important information to educators. This type of information may be used to provide an empirical base for discussions about scientific literacy and the development of a “science for all citizens.” The SLiM approach presented and tested here could offer an updatable instrument to provide this information for science educators, such as for future textbook writers and in the long term, for future curriculum reforms.

The methods of the two most up-to-date studies of scientific literacy based on the media, Brossard and Shanahan (2006) and the present SLiM study, differ in several aspects. Compared to Brossard and Shanahan (2006), SLiM has the advantage of developing the items from textbooks instead of from dictionaries. The terms in SLiM cover a broader area than the sample of dictionary terms used by Brossard and Shanahan (2006), since SLiM covers the complete indexes of junior high textbooks. Furthermore, SLiM provides a daily life context (as in the original newspaper articles).

The results from the pilot test of the SLiM questionnaire showed that the terms relating to the subject of biology appeared most frequently in the media (45.26%). This result is in line with the results of Hopkins (1925) and Martin (1945). Concerning the second most frequent subject area, earth science (37.90%), this result may be related to the special natural phenomena of Taiwan, i.e. earthquakes, typhoons and landslides. The frequent occurrence of earth science terms also shows current public concerns in Taiwan.

The results of the reliability, discrimination power and Pearson correlation tested in this study all ensure that SLiM can be a good instrument to evaluate civic scientific literacy. Moreover, the ANOVA computing results also show that SLiM has the ability to differentiate the different levels of participants, i.e. 10th graders performed generally better than undergraduates, and 7th graders were the least well performing group. Regarding these performance results, it is not surprising that the 7th graders performed least well, since the 7th graders had not learnt about those concepts entirely. The reason that 10th graders performed best is likely to be that the 10th graders had just prepared and passed the national entering exam of senior high schools. The results of the undergraduates can be connected to the fact that those undergraduates taking part in the study were more or less non-science majors, and had possibly forgotten what they have learnt in junior high schools after three years of senior high school learning. However, since it was a pilot study to test the feasibility of the SLiM instrument, we did not consider conducting a random sampling procedure. Through SLiM instruments developed in the future, it will be possible, with a random sampling technique, to make claims about the differential scientific literacy of groups with different educational levels. Moreover, regarding the topics related to chemistry, the performances from the groups of 10th graders and undergraduates showed no significant differences. This result might have been caused by the number of items being limited, and by the questions being more related to conceptual items. The students were all less familiar with those concepts.

The results of the percentages of correct answers showed that the subject of physics had the highest percentage of correct answers, followed by earth science, biology and chemistry. This result was similar to the data presented in the Science and Engineering Indicators (2008), which showed that students performed better in the area of physical science. Again, items regarding physics and earth science in SLiM were more daily life-related. However, the reason why biology items were more difficult for the participants than physics and earth science needs further investigation (since previous research results show biology to be more interesting than the other subjects in school for students). Also, adopting diverse assessment practices like open-ended and interview approaches may bring us more understanding of students’ ideas (Chang, Yeh and Barufaldi, 2010).

Since undergraduate performance was below that of the 10th graders in this pilot study, we worry that the adult public will perform even lower than the undergraduates. It is necessary to invite more participants from different educational backgrounds and age groups to participate in the testing of SLiM. Meanwhile, it is an important task to think about how to maintain civic scientific literacy and prevent individuals from forgetting those concepts after graduation from school life. In the modern age, we have a large number of different media types; the media is the main resource for the public to receive scientific information and knowledge after their school life has ended. SLiM was developed based upon one of these media types: namely the newspaper. It is worthwhile to investigate whether the public can really understand those scientific concepts appearing in the media. Otherwise, without this common base of scientific literacy, it is hard for science educators to convey new scientific discoveries or technological inventions to the public.

Footnotes

Acknowledgements

The authors wish to thank the National Science Council (NSC) of Taiwan for financing this study under contracts NSC 99-2631-S-003-022 and NSC 97-2511-S-003-043-MY3.

Notes

Author Biographies

Carl-Johan Rundgren is at the National Taiwan Normal University, Taiwan and the Swedish National Graduate School in Science, Technology and Mathematics Education Research (FontD), Linköping University, Sweden.

Shu-Nu Chang Rundgren is at the Swedish National Graduate School in Science, Technology and Mathematics Education Research (FontD), Linköping University, Sweden.

Yuen-Hsien Tseng, Pei-Ling Lin and Chun-Yen Chang are at the National Taiwan Normal University, Taiwan.

References

Brossard

Shanahan

(2006) Do they know what they read? Building a scientific literacy measurement instrument based on science media coverage. Science Communication 28(1): 47–63.

Bybee

(1997) Toward an understanding of scientific literacy. In: Graeber

Bolte

(eds) Scientific Literacy. Kiel: Leibniz Institut für die Pädagogik der Naturwissenschaften (IPN), 37–68.

Chang

Yeh

Barufaldi

(2010) The positive and negative effects of science concept tests on student conceptual understanding. International Journal of Science Education 32(2): 265–282.

de Nooy

Mrvar

Batagelj

(2005) Exploratory Social Network Analysis with Pajek. Cambridge: Cambridge University Press.

DeBoer

(2000) Scientific literacy: Another look at its historical and contemporary meanings and its relationship to science education reform. Journal of Research in Science Teaching 37(6): 582–601.

Hopkins

(1925) A study of magazine and newspaper science articles with relation to courses in sciences for high schools. School Science and Mathematics 25: 793–800.

Koelsche

(1965) Scientific literacy as related to the media of mass communication. School Science and Mathematics 65: 719–725.

Laugksch

(2000) Scientific literacy: A conceptual overview. Science Education 84(1): 71–94.

Lord

Rauscher

(1991) A sampling of basic life science literacy in a college population. The American Biology Teacher 53(7): 419–424.

10.

Martin

(1945) A chronological survey of published research studies relating to biological materials in newspapers and magazines. School Science and Mathematics 45(6): 543–550.

11.

Millar

(2008) Taking scientific literacy seriously as a curriculum aim. Asia-Pacific Forum on Science Learning and Teaching 9(2), URL: http://www.ied.edu.hk/apfslt/

12.

Miller

(1983) Scientific literacy: A conceptual and empirical overview. Daedalus 112(2): 29–48.

13.

Miller

(1998) The measurement of civic scientific literacy. Public Understanding of Science 7: 203–223.

14.

Murcia

(2009) Re-thinking the development of scientific literacy through a rope metaphor. Research in Science Education 39: 215–229.

15.

Roberts

(2007) Scientific literacy/science literacy. In: Abell

Ledermann

(eds) Handbook of Research on Science Education. Mahwah, NJ: Lawrence Erlbaum Associates, 729–780.

16.

Science and Engineering Indicators (2008) National Science Foundation, Volume 1. Arlington, VA: National Science Board.

17.

Shamos

(1995) The Myth of Scientific Literacy. New Brunswick, NJ: Rutgers University Press.

18.

Tseng

Y-H

(1998) Multilingual keyword extraction for term suggestion. Paper presented at the 21st International ACM SIGIR Conference on Research and Development in Information Retrieval – SIGIR ’98, Australia.

19.

Tseng

Y-H

(2002) Automatic thesaurus generation for Chinese documents. Journal of the American Society for Information Science and Technology 53(13): 1130–1138.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.59 MB