Are you SLiM? Developing an instrument for civic scientific literacy measurement (SLiM) based on media coverage

Aspects of civic scientific literacy

There are a plethora of definitions of scientific literacy (e.g. DeBoer, 2000; Laugksch, 2000; Murcia, 2009). Some authors question whether or not the concept of scientific literacy has any real meaning and some think of the term as merely an empty slogan (e.g. Bybee, 1997); the term has been used in different ways since its introduction in the USA in the late 1950s. Today, “Scientific Literacy” is a term used to describe what the ordinary individual may need to know in order to be an active member in modern society.

One of the most influential authors in the field of scientific literacy is Jon Miller, who regards scientific literacy as a multidimensional construct. Miller (1983) divides scientific literacy into three main aspects: (1) an understanding of the norms and methods of science (i.e. the nature of science), (2) an understanding of key scientific terms and concepts, and (3) an awareness and understanding of the impact of science and technology on society. Most research on scientific literacy conducted up until now has focused on the second aspect, that is to say the study of individuals’ understanding of scientific terms and concepts. Some research has been done on the first aspect of the nature of science, but virtually no studies have been compiled on the third aspect. Following the same trend, Shamos (1995) has suggested three forms of scientific literacy with a connection to society. The first form, “cultural scientific literacy,” relates to possessing a “lexicon” of scientific terms, and being able to read and understand science-related news that appears in the media. The second form, “functional scientific literacy,” is more active than the former one and requires the individual to be able to converse, read and write coherently in a non-technical but meaningful context – that is, the individual must be able to communicate the essence of science-related news to others. The third form, “true scientific literacy,” is according to Shamos more difficult to attain. It requires an individual to be aware of the major theories of science, how these were arrived at, and why they are widely recognized. Furthermore, the truly scientifically literate individual should be able to understand something about the nature of science and the ways scientists pose questions, conduct inquiries, and arrive at conclusions (Shamos, 1995). The third form is similar to the first aspect mentioned by Miller (1983). According to the above-mentioned definitions of scientific literacy, we find that a major aspect of civic scientific literacy is people’s understanding of scientific terms and related concepts in the context of society, especially the scientific terms/concepts presented in news media. This is a guiding principle for the development of SLiM in this study.

How to measure civic scientific literacy?

Miller (1998) points out that the development of a test instrument designed to cover a conceptual core understanding of scientific literacy would serve as a valid means of measuring civic scientific literacy. Miller takes his starting point as identifying a set of basic constructs, such as DNA and atomic structure, which are intellectual foundations for reading and understanding contemporary issues. From this set of basic constructs, which have been used also for cross-national studies of scientific literacy, test items are constructed. The general aim of Miller’s approach to study scientific literacy is to give a measurement of people’s understanding of a basic core of conceptual knowledge about science. The results from the past investigations have been documented for decades (Science and Engineering Indicators, 2008).

Miller (1998) points out the potential danger of testing issues that are widely discussed currently, but which might be of less importance as a part of a general scientific literacy in the future. He gives the example of the test items provided and constructed by the National Association of Science Writers in the USA, in the first attempt to provide a measure of scientific literacy in 1957. In the test, the major items relating to issues that were of great concern at that time, such as the polio vaccination and fluoridation of water, soon proved to be outdated.

A study of scientific literacy in biology by Lord and Rauscher (1991) takes a similar approach to Miller’s; the report takes a conceptual core of scientific understanding as the starting point for their measurement. Interestingly, they developed their test instrument, measuring this conceptual core, from primary and middle school science textbooks, and test a population of students at different stages of their college education. However, the detailed selection process of scientific terms was not clearly described in their study.

Measuring scientific literacy from media

A different way to measure scientific literacy has been presented by Brossard and Shanahan (2006). They highlight the problem that past studies of scientific literacy inevitably have taken their starting point from what experts’ think the scientifically literate person should know. Measures that rely on, what Brossard and Shanahan call “ideal” knowledge are prone to have a problem of subjectivity. As experts from different fields of science may have diverging views on what terms should be included in a scientific literacy test, it is, according to Brossard and Shanahan, possible that “the biases and prejudices of the scientific community can influence the overall definition of literacy” (2006: 50). Instead of using experts’ selection of content to test, Brossard and Shanahan use the 31 most frequently occurring scientific terms (from a list of randomly selected terms from a scientific dictionary, the Oxford Dictionary of Science) from media to construct a media-based scientific literacy test.

The idea of using the media as a source providing ideas for developing science education and testing scientific literacy is, however, not new. Long before the era of computers, Hopkins (1925) analyzed 2770 articles, and concluded that biology content was more prevalent in newspapers and magazines than content from the other science subjects. The same tendency was found by Martin (1945), who also suggested that these results might be taken into account when reorganizing the science curriculum. Interestingly, the editor of the journal in which Martin’s article appeared cautioned against using content analyses of media to make educational decisions. According to the editor’s note to Martin’s article, the subject itself should determine the content of the curriculum, even if analyses of the science content in newspapers might point to some desirable additions to the curriculum (Martin, 1945). In the 1960s, Koelsche (1965) made an attempt to extract scientific terms from the media that citizens should comprehend; if they understood these terms they could be qualified as being “scientifically literate.”

In summary, civic scientific literacy is a multidimensional concept, which could and should be approached and measured from several different viewpoints. The viewpoint of defining and measuring an understanding of a conceptual core of science is one model. Another important aspect is from the opinion of experts in different scientific fields as well as that of science educators. In this study, we focus on the viewpoint of how the occurrences of scientific terms in the media can aid us in constructing instruments to measure civic scientific literacy; this view corresponds to Miller’s second aspect of scientific literacy (1983) about understanding key scientific terms and concepts. Through the use of modern computer software, we are able to scan the media to find the most frequently used scientific terms during different time periods as well as to make continuous updates on current terms. Through frequently updated computer scans, we can avoid the problem highlighted by Miller (1998), that topics used to test scientific literacy might fall out of public interest and become outdated.

When using science in the media, it can be valuable to reflect upon the fact that the intentions behind publishing news related to science in a newspaper might be very different from the intentions in the minds of textbook writers and curriculum designers. However, mapping these sources together may provide us with a fresh view of what scientific terms are considered prevalent in present-day society.

The current study – SLiM (civic scientific literacy measurement based on media coverage)

In this article we present a way to measure the public’s SLiM (Civic Scientific Literacy Measurement Based on Media Coverage). SLiM presents a measurement by detailing which scientific topics are most often discussed in the media. It is of value to note that this is a system which can and will constantly update itself.

An approach based on the most frequently occurring terms in the media, similar to that presented by Brossard and Shanahan (2006), was adopted to construct our instrument to test civic scientific literacy. Brossard and Shanahan pointed out their instrument was “dry” and it would be better to select the terms outside of the conventionally accepted dictionary definitions. Furthermore, a difference between the way science textbooks were used in the study by Lord and Rauscher (1991) and the SLiM approach presented here is that the terms selected to be used in the SLiM instrument were screened from the most common scientific terms appearing in the media.

Accordingly, we have revised the methodology by defining the scientific terms as terms present in the indexes of Taiwanese junior high science textbooks and mapping them with the terms most frequently appearing in newspaper articles (see appendix, ¹ SEM 1). In addition, we constructed the test items by giving a context with co-occurring scientific terms in the same news articles. This system also employed daily life contexts to develop test items in this study.

Aim and research questions

The purpose of this study is to develop and validate an instrument for investigating civic scientific literacy based upon scientific terms occurring in the media. SLiM instruments can be constructed in different countries with different science curricula, and it has the advantage of being an updatable system. Thus SLiM can form a complement to existing ways of evaluating scientific literacy. As mentioned previously, scientific literacy can be evaluated according to different aspects. Here, we focus on evaluating individuals’ familiarity with scientific terms and concepts presented in the media, and their ability to apply these terms correctly in daily life contexts which appear consistently in newspaper articles.

Based upon the above, three research questions are asked, which are: (1) What are the most common scientific terms presented in Taiwanese news media? (2) What are the degrees of item difficulty and what is the discrimination power of SLiM? (3) What are the differences in performance on SLiM between the different groups participating in this study?

Construction of the instrument

In the following section, the way we extracted scientific terms to develop SLiM and the format of SLiM are described.

Term extraction

The basic idea for developing the instrument to investigate the public’s SLiM is to find scientific terms from both textbooks and news media. The scientific terms taken from textbooks were based upon the complete indexes of four major Taiwanese junior high textbooks (2001 version). The books were from four different publishers and the indexes were provided by the National Institute for Compilation and Translation in Taiwan. The topics included nature, life, and technology. A total of 3657 index terms (or 2037 terms if only distinct ones are counted) were listed from the textbooks.

The media terms were based on the complete news reports (covering all subjects) from a collection of four Taiwanese newspapers including United Daily News, United Express, Ming Hseng News, and Economic Daily News, which are all from the United Daily News Group in Taiwan during the years 2001 to 2002 (i.e. corresponding to the time of the printing of the textbooks). This newspaper collection has been widely used by many research groups in Taiwan, due to its comprehensiveness (two years of daily news stories, more than 1000 each day on average). Only key terms that were repeated at least twice in a news document were extracted based on an automatic keyword extraction algorithm (Tseng, 1998). These include dictionary terms (covering over 120,000 terms) and out-of-vocabulary terms (those not in the dictionary). In total, 901,446 documents starting from 1 January 2000 to 31 December 2001 were scanned, and 1,082,937 terms (including scientific and non-scientific terms) were extracted. To better understand each keyword’s context, we also extracted what we call “related terms” for each keyword. They were identified based on whether they co-occurred often with their keyword in the same sentences (Tseng, 2002). After filtering those without related terms (appendix, SEM 2), 323,918 keywords remained; these keywords were normally topic-relevant or context-positive terms. In contrast, those removed terms were normally rare terms or outliers. Finally, the keywords and their related terms were matched with the 2037 terms from the textbook indexes. This resulted in a list of 876 keywords. The detailed process is presented in the appendix, SEM 3.

One assumption of this study was that the higher the frequency of scientific terms presented in news media, the more frequently people may have discussed related issues during that period of time. Accordingly, those term-related issues were hot topics and important in Taiwan. To obtain the final terms for SLiM instrument development, the most frequent 100 keywords (each occurred in at least 2950 news articles) and their related terms were examined by four experts. After excluding non-scientific terms (e.g., e-mail, computer, and so on), 95 terms (including keywords and their related terms) were selected. To present these terms and their relations in a compact form, for use by experts to develop the SLiM instrument, they were visualized by a visualization tool (de Nooy, Mrvar and Batagelj, 2005) in a term network (concept map) as shown in Figure 1. Note that a link in the network denotes that the two terms at the ends of the link co-occur often in the same sentences and their exact relation represented by this link can be explained by the co-occurring sentences retrieved from the news articles through a search engine which indexed the whole news corpus.

OPEN IN VIEWER

Figure 1.

A term network (concept map) of SLiM.

The format of SLiM

Different formats and assessments have been developed for the purpose of evaluating scientific literacy worldwide. To develop the SLiM instrument, we decided to skip fill-in the blank items, as this would make data processing slower and also possibly give rise to problems of comparability in future cross-national studies. We wanted to prevent our participants from simply guessing true- or false-items. Thus, the multiple-choice format was chosen as the most suitable test design to explore civic SLiM.

For each of the terms we developed a test item, consisting of a multiple-choice question with four alternatives, the last alternative being “I don’t know” for each question. The test persons were asked to click the alternative “I don’t know” rather than simply guess the answer of a question if they had no idea about which alternative was the correct choice. The rationale for this was to avoid students clicking the correct alternative by pure chance.

As the terms were identified from occurrence in the media, the test items were developed with the intention to provide a context of daily life-related questions. We also questioned whether the reader could understand the meaning and use of a term in a daily life context. The most commonly co-occurring terms were used to provide multiple-choice alternatives. However, if some terms were too difficult to develop as a context-based item, we developed the items as conceptual items instead. Hence, two groups of items were developed, which included conceptual and contextual items. Examples of conceptual and contextual items are as follows.

OPEN IN VIEWER

Example: Conceptual Item

Fill in the correct order from smaller to bigger:

□ Molecules-Cells-Atoms

□ Molecules-Atoms-Cells

□ Atoms-Molecules-Cells

□ I don’t know

In this item, the keyword “molecule” was to be combined with the co-occurring terms “atom” and “cell.” These terms refer to basic concepts in science, which may be of importance in daily life. The terms are seen as a background knowledge that can help to structure information, rather than having any direct application. The question about scale relationship could therefore be viewed as a testing of basic conceptual knowledge rather than a testing of the ability to apply this knowledge in a daily life-related context.

OPEN IN VIEWER

Example: Contextual Item

Why do we recommend consumers to use vegetable oils, like olive oil, instead of oil from animals, such as butter?

□ Because it is natural

□ Because it contains unsaturated fatty acids

□ Because it contains saturated fatty acids

□ I don’t know

In this item, the keyword “structure” was to be combined with the co-occurring term “fatty acid.” In this example, there was a discussion in the media about how we should choose different food products relating to their content of saturated and unsaturated fatty acids; thus, we created a contextually based test item with connection to this discussion. We also did inter-rater reliability for coding the conceptual and contextual items by four professors and one master program student. The Kappa value was 0.639–0.701. All test items of the SLiM instrument are presented in the appendix, SEM 7.

Validation of the instrument

The test items were sent out for validation by nine experts in different areas of science: three from life science, one from physics and chemistry each, and four from earth science. The experts we invited in this study were asked to revise the test items of SLiM to make the items more understandable and to clarify terms when possible. The experts also checked the correctness of science concepts, added/deleted/revised options of each item and its alignment with the key terms and co-occurring terms. The items were revised according to the experts’ comments before this pilot study.

Pilot test of the instrument

The most common scientific terms presented in Taiwanese news media

In total, 95 scientific terms matched from textbooks and the media were chosen to develop SLiM. According to the distribution of terms, the subject of biology was represented by the highest number of terms (45.26%), earth science was the second (37.90%), and the subject areas of physics (11.58%) and chemistry (5.26%) had the lowest representations. In total, 50 items were generated. According to the distribution of terms of each subject field (appendix, SEM 6), the number of items relating to each subject area (biology 22, earth science 19, physics 6 and chemistry 3) mirrored the occurrence of scientific terms from the respective subject area in the media.

It was found that biological terms were very often related to health (e.g. diabetes) and the human body (e.g. heart). In earth science, the terms were often related to local phenomena, such as typhoons and earthquakes, which occur in Taiwan from time to time. Also, new scientific discoveries in biotechnology/genetic engineering were often presented. Environmental issues such as global warming, and celestial phenomena like comets and asteroids were regularly obtained in the area of earth science. In physics and chemistry, the terms were related to technical applications, such as diesel engines, batteries and thermal conductors.

Reliability, difficulty, discrimination power and correlations among the items

Regarding the reliability of SLiM, as shown in this pilot study, the value of reliability of SLiM was 0.86 (KR 20) from the whole 954 participants. Concerning the reliabilities of each group, these showed 0.87 for 7th graders (n = 335), 0.62 for 10th graders (n = 352), and 0.74 for undergraduates (n = 267). The average difficulty of the whole 50 items of SLiM ranged from 0.19 to 0.91, and the discrimination powers were 0.1 to 0.59.

According to the Pearson product-moment correlation coefficients, it was found that the items from the different subjects (biology, earth science, chemistry and physics) were all correlated with significance (r = 0.391 to 0.687, p < .01). The correlation among each subject to the total SLiM questionnaire was high as well (r = 0.567 to 0.912, p < .01). The detailed results are presented in Table 1.

OPEN IN VIEWER

Table 1.

Correlations of students’ performance in each subject (total participants n =954).

Subject	Group
	Biology	Physics	Chemistry	Earth science
Biology	1	–	–	–
Physics	0.523**	1	–	–
Chemistry	0.420**	0.391**	1	–
Earth science	0.687**	0.615**	0.448**	1
Total score (T)	0.892**	0.737**	0.567**	0.912**

Note: ** p < .01.

Moreover, Table 2 presents the Pearson correlation coefficients among three different groups of participants. They all show significant correlations among the different subjects, apart from the 10th graders’ performance in chemistry and earth science (p > .05). In the group of 7th graders, the value of correlation was from 0.358 to 0.911 (p < .01). The Pearson correlations were also significantly high among 10th graders from 0.193 to 0.827 (p < .01), and the undergraduate group, r = 0.254 to 0.846 (p < .01).

OPEN IN VIEWER

Table 2.

Correlations of students’ performance among different subjects and groups of participants.

Subject	Group
	7th graders (n = 335)					10th graders (n = 352)					Undergraduates (n = 267)
	B	P	C	E	T	B	P	C	E	T	B	P	C	E	T
Biology (B)	1	–	–	–	–	1	–	–	–	–	1	–	–	–	–
Physics (P)	0.526**	1	–	–	–	0.313**	1	–	–	–	0.265**	1	–	–	–
Chemistry (C)	0.409**	0.358**	1	–	–	0.246**	0.193**	1	–	–	0.284**	0.254**	1	–	–
Earth science (E)	0.699**	0.606**	0.448**	1	–	0.340**	0.312**	0.099	1	–	0.529**	0.356**	0.360**	1	–
Total score (T)	0.904**	0.722**	0.553**	0.911**	1	0.827**	0.614**	0.398**	0.717**	1	0.846**	0.546**	0.505**	0.845**	1

Note: ** p < .01.

The participants’ performances on SLiM

The results of average scores on the SLiM test revealed that 10th graders performed best (Mean = 37.34±0.23), undergraduates were the second highest group (Mean = 33.00±0.33), and 7th graders were the last group (Mean = 26.73±0.45).

To understand whether the same trend could be seen also in the different subjects of biology, chemistry, physics and earth science, we analyzed the performances in the different groups of 7th graders, 10th graders and undergraduates further. Figure 2 shows that 10th graders performed best, undergraduates were second, and then 7th graders performed least well among all the four different subjects in SLiM. Hence, the similar trend of the performances among these three groups of participants was also found in all four subjects. The detailed scores are presented in Table 3.

OPEN IN VIEWER

Figure 2.

The average scores of the performances of the three groups of participants in each subject area and the percentages of correct answers (average score divided by number of items) for the four subject areas for the different groups of participants.

OPEN IN VIEWER

Table 3.

The average scores of all participants’ performances regarding SLiM.

Subject	Group
	7th graders (n = 335)					10th graders (n = 352)					Undergraduates (n = 267)					Scheffe test
	Mean	SD	Std error	95% confidence interval for Mean		Mean	SD	Std error	95% confidence interval for Mean		Mean	SD	Std error	95% confidence interval for Mean
				Lower bound	Upper bound				Lower bound	Upper bound				Lower bound	Upper bound
Biology	12.21	3.9	0.21	11.80	12.63	15.74	2.4	0.13	15.49	15.99	14.00	2.8	0.17	13.66	14.34	10th graders > 7th graders *
																10th graders > Undergraduates *
																Undergraduates > 7th graders*
Physics	3.08	1.5	0.08	2.92	3.24	4.70	1.2	0.06	4.58	4.83	4.41	1.2	0.07	4.27	4.55	10th graders > 7th graders *
																10th graders > Undergraduates *
																Undergraduates > 7th graders*
Chemistry	1.28	0.9	0.05	1.18	1.37	1.89	0.7	0.04	1.82	1.97	1.75	0.8	0.05	1.65	1.84	10th graders > 7th graders *
																10th graders > Undergraduates
																Undergraduates > 7th graders*
Earth science	10.16	3.6	0.19	9.77	10.54	15.01	1.8	0.10	14.82	15.20	12.84	2.4	0.15	12.54	13.13	10th graders > 7th graders *
																10th graders > Undergraduates *
																Undergraduates > 7th graders*
Total score	26.73	8.3	0.45	25.84	27.62	37.34	4.2	0.23	36.90	37.79	33.00	5.5	0.33	32.34	33.65	10th graders > 7th graders *
																10th graders > Undergraduates *
																Undergraduates > 7th graders*

Note: * p < .05; SD, standard deviation.

According to the ANOVA computing results, there were significant differences in the performances on SLiM among these three groups of participants (p < .05), except in the subject of chemistry. In chemistry, 10th graders performed better than 7th graders and undergraduates, but there was no significant difference between 10th graders and undergraduates (p > .05). Table 3 also shows the details of the Scheffe test.

After considering the different numbers of items in each subject area developed in SLiM, we could conclude that chemistry was the most difficult subject for all three groups. This conclusion was reached after dividing average scores by number of items in each subject (Figure 2). However, since there were only three items related to chemistry in the SLiM instrument, this might be of minor significance. The 10th graders performed best in earth science, and then physics. In contrast, undergraduates performed best in physics, and then earth science. The 7th graders performed best in biology, then in earth science. Overall, percentages of correct answers among all participants were highest in physics, followed by earth science, biology, and finally in chemistry. Detailed percentages of the performances in different subject areas in each group of participants are shown in the appendix, SEM 5.