Establishing an everyday scientific reasoning scale to learn how non-scientists reason with science

Abstract

Scientific concepts and core ideas are fundamental for scientific inquiry and research. However, they are not always understood by non-scientists who encounter science in the media, conversations with friends, and other daily contexts. To assess how non-scientists reason with science in daily life, we extend the work described by Drummond and Fischhoff by developing an everyday scientific reasoning scale and demonstrating its ability to predict the use and application of daily scientific information. This article features three studies describing the development, validation, and use of the everyday scientific reasoning scale. Findings demonstrate an association between respondents’ scores on the everyday science reasoning scale and their level of education and suggest that using daily scenarios for framing science facilitates the process of understanding scientific concepts. These results have important implications for communicating science in society and engaging diverse populations with science.

Keywords

public understanding of science science in daily life science literacy science reasoning

1. Introduction

Scientists take pride in basing their work on a long tradition of scientific research which has defined and refined itself over hundreds of years to best describe and explain natural phenomena. The “scientific method” is said to be rigorous and robust as it utilizes systematic observations, measurements, and experiments which are used to formalize explanations and prove hypotheses about how the world works.

However, the scientific method is in no way one coherent method; rather, it is comprised of diverse practices and concepts that collectively aim to minimize error and provide reliability and replicability to a study (Bauer, 1994). The value of this practice is in developing a culture or mindset of thinking both analytically and critically about complex situations. These practices provide the ability to make logically deductive arguments from simple premises and identify salient variables, patterns in data, and numerical fluency (Claussen and Osborne, 2013). Furthermore, these scientific practices take into consideration a number of core ideas and concepts, which are fundamental for scientific inquiry and research, as well as application of science in everyday life. Such concepts include (but are not limited to) replication, causality, randomness and biases which are largely used to describe research methods and procedures and to assess the quality and validity of scientific research (National Research Council, 2012).

While these concepts are widely accepted by scientists, they are much harder to convey to non-scientists and thus are not always understood by lay audiences. These publics encounter “science” in many daily situations and settings and are expected to make decisions on issues such as health, nutrition, vaccination and many more based on their knowledge and understanding. Adding complexity to how people make decisions, researchers have argued that “people are motivated reasoners and are influenced by their hopes and emotions in addition to their prior beliefs. . .” (Shah et al., 2017: 255) rather than basing their decisions solely on scientific evidence. It is, therefore, important to think about public encounters with science from the perspective of the public and consider their abilities to reason with science.

For this purpose, the term “science literacy” was coined in the 1950s “as a means of expressing the disposition and knowledge needed to engage with science—both in an individual’s personal life and in the context of civic issues” (National Academies of Sciences, Engineering, and Medicine, 2016: 27). Since then and despite largescale adoption of the term, researchers and practitioners have not been able to formalize one accepted definition of what science literacy means and what is incorporated in it (National Academies of Sciences, Engineering, and Medicine, 2016). Current measures included in science literacy surveys for adults predominantly assess scientific opinions and factual knowledge rather than examining understanding and use of scientific information. The most widely used tool to date is the NSF Survey on Science Literacy, which primarily examines scientific knowledge (National Science Board, 2006; Roos, 2014) using 10 to 15 true/false questions on variety of scientific topics. For example, “Electrons are smaller than atoms. True or False?” While these questions have been used as measures for defining public’s scientific literacy for over 35 years, it is not clear that knowing such scientific facts can assist people in making informed decisions. Some critics of such knowledge-based questionnaires claim that the “. . . surveys de-contextualize, and hence reify science as an entity whose definitional identity exists independently of any specific experience and context within which ‘the,’ equally reified, public encounters it” (Kallerud and Ramberg, 2002). In other words, examining scientific knowledge alone separates science from its social and cultural contexts, where public engagement and learning of science often occur (Tal and Dierking, 2014). In addition, procedural knowledge which includes an understanding of the procedures and constructs that science uses to establish its claims is just as important when examining public ability to reason with science (Kind and Osborne, 2017). Ultimately, as discussed by the National Academies of Sciences, Engineering, and Medicine (2016) report on science literacy, current measures of science literacy “may not capture as much of the depth and breadth of the knowledge nor the full diversity of scientific reasoning required for science literacy” (p. 107).

This article aims to address these concerns and provide a salient tool for exploring science reasoning, complementing existing science literacy measures. Acknowledged as an integral part of science literacy, scientific reasoning is often conceptualized as the “intentional knowledge-seeking behaviors and the coordination of theory and evidence” (Woolley et al., 2018: 102). Furthermore, Drummond and Fischhoff (2017) conclude that if we expect people to evaluate scientific research (and science in daily contexts in general), people need to be able to reason with evidence in varying contexts.

However, Sharon and Baram-Tsabari (2020) point to the challenges of understanding lay audiences’ ability to transfer scientific reasoning skills. In fact, most of the evidence on successful transfer “comes from studies conducted on ‘toy tasks, simple tasks, and simulated tasks’ which provide ‘friendly constraints and scaffolds that facilitate successful transfer’” (Chinn & Golan Duncan, 2018: 91, as quoted in Sharon and Baram-Tsabari, 2020: 883). As lay audiences encounter science in daily life, often in “unfriendly” environments (e.g. medical treatment), there is a need to examine their reasoning abilities in more authentic environments.

We describe the development and validation of an innovative scale measuring people’s ability to reason with science embedded in daily life. In the development of the scale, we aimed to include scenarios, situations, and dilemmas people may encounter in their daily life, in the media, in conversations with friends, and in other daily encounters. Using daily context to examine the different aspects of scientific reasoning will enable us to assess the pitfalls and challenges in laypersons’ understanding of science and consequently their use and application of scientific evidence in everyday life (National Academies of Sciences, Engineering, and Medicine, 2017).

The goal of this article is three-fold: first, to share the development of a tool for measuring scientific reasoning using day-to-day scenarios (namely, the everyday scientific reasoning scale); second, to validate the tool and demonstrate its ability to predict the use of scientific information in other contexts; and third, to measure and compare scientific reasoning skills within society. This article features three consecutive studies which collectively describe the formulation and establishment of a new, validated empirical scale to assess scientific reasoning in daily scenarios.

2. Theoretical framework

Before discussing the development of our scientific reasoning scale, we would like to explicate two terms used sometime interchangeably yet developed independently by two schools of thought—science literacy and public understanding of science. Traditionally, science literacy is used in the context of science education research, while public understanding of science in the context of science communication research. Yet, definitions of both terms have evolved and transformed greatly over the past years. Conceptualized as early as the nineteenth century, science literacy has often been referred to in specific school contexts addressing the content science learning (Ryder, 2001). Yet, recent perspectives build on a much broader definition of science literacy described by Roberts and Bybee (2014) as vision II of science literacy. This vision includes acquaintance with how the scientific method works as well as the ability to make sense of relevant scientific information, a rather “sophisticated understanding of science” (Pasek, 2018).

Conversely, public understanding of science has originated from a gap identified in the public’s knowledge and attitudes toward science and, as a field, aspired to enhance their understanding of science through increased efforts in science education, national debates, science festivals, and so on (Bauer, 2009; Brossard and Lewenstein, 2009). Bauer et al. (2007) trace the origins of the concept of public understanding of science to a paradigmatic shift in the mid-1980s from the use of science literacy which predominantly discussed knowledge deficits and education to the use of public understanding of science which included public attitudes and the need for contextualization. These early conceptions of public understanding have been criticized as “oversimplified” and as misconceiving of what publics actually know and what they are capable of grasping (Jasanoff, 2014), leading to a broadening of definitions. Recent conceptualizations include addressing how people come in contact and respond to various aspects of science, representations of science in the public sphere, and interactions between science and society (Burns and Medvecky, 2018; Jasanoff, 2014).

Following these contemporary definitions, we find some components of scientific literacy and public understanding of science to be interchangeable as they describe similar ideas and aspirations. Both relate to laypersons’ ability to apply scientific knowledge and use it in various contexts related to one’s life (Huxster et al., 2018). Nevertheless, we acknowledge that many science communication researchers to date are uncomfortable using the term science literacy which implies an inherent deficit approach.

In our work here, we emphasize one aspect of science literacy and public understanding of science, namely, scientific reasoning. Klahr and Dunbar (1988) conceived of scientific reasoning as problem solving that is characterized as a guided search and information-gathering task (Zimmerman, 2005). Morris et al. (2012) expand this idea arguing that “scientific reasoning differs from other types of information seeking in that it requires additional cognitive resources as well as an integration of cultural tools” (p. 61). Brought together, scientific reasoning is defined as the ability to rationalize between competing pieces of evidence and to aggregate content, procedural, and epistemic knowledge to draw reasonable inferences (Čavojová et al., 2020). Such a practice requires both deductive and inductive skills and systematic attendance to information (Morris et al., 2012).

One way to conceive scientific reasoning is “thinking like a scientist,” implying people attain skills to evaluate scientific research, reason with evidence, and make decisions (Drummond and Fischhoff, 2017; Kuhn, 2002). In an attempt to measure individual differences in scientific reasoning, Drummond and Fischhoff (2017) constructed a scale measuring a set of skills needed to evaluate scientific findings and factors that determine their quality. Based on “lab-like” scenarios, Drummond and Fischhoff’s (2017) scale demonstrated good reliability and was shown to predict (a) whether individuals held beliefs consistent with the scientific consensus and (b) the ability to use scientific information in other contexts. However, their scale assumed some level of scientific knowledge as they used scenarios based on scientific lab experiences for the items examined. This may introduce bias toward individuals with greater scientific knowledge, mistakenly interpreting knowledge as reasoning skills. Furthermore, while science reasoning is an important skill for those working in the science arena, it is equally important for lay audiences who encounter science on a daily basis (National Academies of Sciences, Engineering, and Medicine, 2017; Zeidler et al., 2009). Since “transferability of scientific reasoning depends on the specific task a person is transferring to” (Sharon and Baram-Tsabari, 2020: 883), it is not clear if Drummond and Fischhoff’s (2017) scale can accurately determine peoples’ ability to reason with scientific information in daily contexts.

Since science is prevalent in our daily lives and is often at the heart of much daily decision-making, there is a growing need for examining peoples’ everyday scientific reasoning. We define everyday scientific reasoning as the ability to reason with scientific information people encounter on a daily basis. Our work on the development of an everyday scientific reasoning scale extends the work described by Drummond and Fischhoff (2017) as it measures adult scientific reasoning with a demonstrated ability to predict the use and application of scientific information in daily context.

3. Study design and results

This study is built on three consecutive studies which jointly describe the formulation and establishment of a new method for measuring scientific reasoning in daily contexts.

Study 1: Development of a scientific reasoning scale using everyday scenarios.

Study 2: Validation of the scale, investigating the relationship between the newly developed everyday science reasoning scale and the previous lab-oriented reasoning scale, and its ability to predict respondents use and application of scientific information.

Study 3: Examination of individual scientific reasoning capabilities and individual differences therein using the everyday scientific reasoning scale.

In the procedure of developing and testing the scale, we built on the work of Diamantopoulos and Winklhofer (2001) who detail the process for constructing and testing a scale with formative indicators. The first two studies outlined hereafter are equivalent to Diamantopoulous and Winklhofer’s content and indicator specification phase (study 1) and the external validity phase (study 2). In study 3, we applied the validated scale to examine individual differences. In the following, we will describe for each of the studies respective aims, methods, and results.

Study 1: Developing the scientific reasoning scale

The aim of this study was to construct a new scientific reasoning scale based on day-to-day decision-making scenarios and formulated around information laypeople often encounter daily. We chose to frame the scale in the context of nutrition, since it is a popular topic in society, affecting personal decision-making and with plenty of related information published and shared in traditional and social media.

Study 1 design and methodology

Since our critique of Drummond and Fischhoff’s (2017) scale (referred to from here on as the lab-oriented science reasoning scale) was aimed at the operationalization of the test items, not the scientific concepts chosen, we followed the scientific concepts described in their article, recontextualizing them for our purposes. Therefore, 11 scientific concepts were tested in the new reasoning scale (referred to from here as the everyday science reasoning scale)—blind/double blind, causality, confounding variables, consensus, construct validity, control group, ecological validity, history, random assignment to condition, reliability, and response bias. We did, however, change one of the indicator concepts that was included in the lab-oriented scale. After peer debriefing, we decided not to include the concept of “Maturation” which was complex to transfer to a daily context. In its stead, we added a more general concept—scientific consensus—based on recent public media debates, especially surrounding climate change (National Academies of Sciences, Engineering, and Medicine, 2017). Each of the 11 concepts was transformed into a statement and then requested a true or false response.

The initial version of the everyday reasoning scale was developed by the authors and reviewed by a group of 10 experts in the fields of science communication and education research, as well as an external expert in nutrition, to ensure content accuracy and validity. Four of the research experts further assessed the construct validity of the scale, confirming that the scientific concepts were adequately expressed and used in each question.

In addition, cognitive validity was independently assessed by six representatives from our target population (general public with various scientific and education backgrounds). These representatives answered the scale orally and expressed their opinions on the questions and questionnaire structure and phrasing in the presence of one of the authors.

Study 1 results

Following the validity assessments, the statements of the everyday scale were refined in preparation for study 2. For example, in the item regarding causality, we used carrots as an example, but respondents noted that carrots had a lot of sugars when compared to other vegetables, which prompted us to use cucumbers in the item instead. Another example was the phrasing of several items (history, reliability, and response bias) which proved to be unclear and wordy and were thus edited and rephrased.

The final reasoning scale starts with the description of an overweight man (Amir) who wants to find a research-proven method to lose weight. It then refers respondents to the 11 items, each describing different aspects of the man’s search for an effective diet. Finally, respondents are asked to evaluate the different statements by responding to a related true or false question.

Study 2: Validating the everyday scientific reasoning scale

Study 2 design and methodology

Study 2 was dedicated to validating the everyday scientific reasoning scale as developed in study 1. To do so, we aimed to formulate a validity argument by comparing the everyday scientific reasoning scale with three other validated constructs, namely,

the lab-oriented scientific reasoning scale developed by Drummond and Fischhoff (2017);

an analysis task assessing reasoning of science-related news in other daily contexts (Golumbic et al., 2016);

a scientific interest questionnaire based on an annual national survey conducted in Israel (Ministry of Science, 2018).

Implementation of study 2 utilized an online questionnaire comprised of five consecutive sections, as illustrated in Figure 1. To reduce bias, sections 1 and 3 included either the everyday science reasoning scale or the lab-oriented reasoning scale, distributed randomly but equally among participants and followed by an open-ended question, asking respondents to comment on the questionnaire. Section 2 comprised of socio-demographic questions asking about gender, age, education level, science education level, current occupation, and socioeconomic status of the respondents. Section 4 included an analysis task assessing science-related news clippings on the topic of air pollution (see Golumbic et al., 2016) and was used to determine the ability of respondents to use and apply scientific information. Section 5 included an interest questionnaire assessing participants’ interest in a variety of scientific-based topics such as biology, astronomy, health, and technology (Ministry of Science, 2018). Institutional review board (IRB) approval was obtained from the Technion institutional committee for both study 2 and study 3 (approval: 2018-056). All participants expressed their full consent for the academic use of the data.

Figure 1.

Five consecutive sections of study 2 validating the everyday scientific reasoning scale.

The research population for the validation study were young adults with high school–level education, who were undertaking tertiary education as undergraduates or graduate students in various scientific and non-scientific fields. The rational for choosing this population was the relatively small diversity within this group which could better serve for validating the research tool with a small sample. Participants were recruited through student social media from multiple academic institutions, and respondents were offered a modest gift card (30 NIS) in return for fully completing the questionnaire.

Our sample consisted of 64 participants. The average age of respondents was 26.8, and there were 64.1% female respondents. More than a half of respondents (55%) were students from Social Science and Humanities, and 45% were Science, Technology, Engineering, and Mathematics (STEM) majors.

In order to check for external validity of the newly developed everyday scientific reasoning scale, we ran a multiple indicator multiple causes (MIMIC) model (suggested, for instance, by Diamantopoulos and Winklhofer, 2001) using the software Mplus 8.2 (Muthén and Muthén, 1998–2017). Specifically, we chose the three external constructs (lab-oriented scientific reasoning, analysis task, and scientific interest) as reflective indicators for a latent variable with the 11 items of the everyday scientific reasoning as formative indicators (for model specification, see also Figure 2). We expected that everyday scientific reasoning would be closely related to the lab-oriented scientific reasoning (large effect, r ≈ .50 or larger) and moderately related to analysis task and scientific interest (moderate effect, r ≈ .30). In addition to the χ² test, model fit was evaluated using the incremental fit index CFI (comparative fit index) and the absolute fit index RMSEA (root mean square error of approximation) and respective cut-off criteria (CFI > .95; RMSEA < .06) as suggested by Hu and Bentler (1999).

Figure 2.

Results for the MIMIC model in the pilot study (N = 64).

Study 2 results

Results for the MIMIC model can be found in Figure 2. Results indicate that everyday scientific reasoning is differentially predicted by its 11 formative indicators. Specifically, we found that one indicator was moderately related to the overall latent construct (confounding variables, β = .37, p < .01); most indicators showed small correlations, some of which were not statistically significant likely due to the sample size (blind/double blind, β = .28, p < .05; causality, β = .17, p = .22; consensus, β = .29, p < .05; control group, β = .12, p = .44; ecological validity, β = .26, p < .05; history, β = .12, p = .40; random assignment to condition, β = .25, p = .07; response bias, β = .22, p = .10); and two indicators appeared not to be related to the latent variable (construct validity, β = –.01, p = .96; reliability, β = .07, p = .57). Furthermore, some of the items appeared to be quite difficult to solve for the individuals in the pilot sample as indicated by low probabilities (<50%) of answering the item correctly; specifically, these items were blind/double blind (34% gave the correct answer), consensus (8%), construct validity (30%), and response bias (47%).

The latent variable everyday scientific reasoning, in turn, predicted the external constructs as hypothesized. Everyday scientific reasoning predicted the original lab-oriented scientific reasoning (β = .70, p < .001) and explained 49% of its variance which constitutes a large effect. The other two external criteria were moderately related to everyday scientific reasoning with 9% of variance explained in the analysis task (β = .31, p < .05) and 10% of variance explained in scientific interest (β = .32, p < .05).

Overall, the model fit the data very well, χ² = 23.294, df = 23, p = .44, RMSEA = 0.014, 90% confidence interval (CI) = [0.000, 0.104], and CFI = .99 and can be seen as indicative of a sufficient external validity of the new everyday scientific reasoning scale.

We therefore calculated a summative score for the 11 everyday scientific reasoning items (M = 6.36, standard deviation (SD) = 1.99, Min = 1, Max = 10). Using this score, we found no differences for individuals who first answered the items on lab-oriented and then on everyday scientific reasoning (M = 6.25, SD = 1.94) and individuals who were given the reverse order of instruments (M = 6.44, SD = 2.06), t(62) = –0.38, p = .70. Furthermore, we found no differences between female (M = 6.22, SD = 2.12) and male individuals (M = 6.61, SD = 1.77), t(62) = –0.75, p = .46.

Following this assessment, and to improve the validity of the scale, all items which received 50% or less of correct answers (i.e. construct validity, response bias, and consensus) were rephrased and subsequently re-examined by experts to determine cognitive and content validity (as described above). This process was aided by the open-ended question which served to highlight possible misunderstandings and improve cognitive validity. For example, the most substantial modification made was to the item measuring scientific consensus which received just 8% of correct answers, indicating its low comprehensibility. We thus overhauled the framing of this item, changing it from “Amir can conclude that this diet is safe” to “Amir can conclude that this diet is dangerous.” The final version of the everyday scale can be found in Supplemental Appendix A compared to the lab-oriented scale.

Study 3: Examining individual differences with the everyday scientific reasoning scale

Building on the outcome of studies 1 and 2, a full dissemination of the scale was conducted within the general Israeli population.

The aims of study 3 were to investigate the extent of scientific reasoning skills within the Israeli general population and to specifically examine individual differences using the newly developed and validated everyday scientific reasoning scale. Thereby and based on prior investigations (Drummond and Fischhoff, 2017; Ministry of Science, 2018), we did not expect any gender or age differences; rather, we hypothesized that individuals with higher education levels have higher scores in scientific reasoning than those with lower education levels and that individuals with a higher level of science education (e.g. a degree in a STEM field) should have higher scientific reasoning abilities than those with other educational backgrounds (e.g. a degree in arts or humanities).

Study 3 methodology

The research population was defined as the general Hebrew speaking adult population in Israel, who have a child in the school system (ages 6–18). This population was chosen since it most reflects the portion of population that has a higher probability of encountering or seeking out information regarding dietary habits for their own use in addition to their families’ use.

The full dissemination of the scale was done using an online survey company (ipanel.co.il) and using a convenience sampling strategy, resulting in a total of N = 513 respondents. Sample demographics are provided in Table 1; the composition of the sample was similar to that of the overall Israeli population regarding gender and age distribution but not identical, so that conclusions from findings within this sample cannot, strictly speaking, be transferred to the overall population.

Table 1.

Demographics of respondents in study 3.

Demographic parameter	Percentages and N (in parentheses) of the research sample
Gender	Male	Female
Gender	58.5%(299)	41.5%(212)
Age	34 and below	35–44	45–54	55 and above	Average age = 41.4Min = 27Max = 70
Age	16%(82)	50%(255)	30%(153)	4%(20)	Average age = 41.4Min = 27Max = 70
Education	K–12	Professional Certification	BA	MA/PhD
Education	28%(141)	11%(57)	43%(215)	18%(90)
Highest level of science education	Mandatory	High school–intensive	Professional course	BA	MA/PhD
Highest level of science education	29%(144)	14.5%(71)	19.5%(96)	29%(143)	8%(38)

Data analysis

Data analysis was conducted with SPSS software. One-way analyses of variance (ANOVAs) were used to compare group scores, followed by Bonferroni post hoc tests when significant values were found. Chi-square tests were used with categorical variables. Correlations were also examined between variables to strengthen the results presented, using Pearson’s r.

With regards to the everyday scientific reasoning scale and opposed to study 2, in this study, respondents had the option of answering each item as either true or false, skipping the “do not know” option.

Study 3 results

Within the sample, individuals on average had 6.8 correct answers out of 11 (60.2%), with a SD of 1.9 (18%). The scores appear normally distributed (Supplemental Appendix B) with only slight skewness (–0.21, standard error (SE) = 0.11) and kurtosis (–0.17, SE = 0.22).

No differences were found between participants as a function of gender, F(1,511) = 0.705, p = .401, nor was age correlated to scientific reasoning ability (r = –.034, p = .448). However, scores were found to differ depending on individuals’ general education level, F(3,509) = 14.411, p < .001, and scientific education level, F(4,488) = 5.695, p < .001.

Specifically, regarding general education levels and according to the post hoc tests, respondents with tertiary degrees were found to achieve higher scores (BA: M = 63.22%, SD = 16.75%; MA/PhD: M = 66.36%, SD = 18.26%) relative to respondents who had a K–12 education (M = 53.21%, SD = 17.53%). Furthermore, respondents with a post-graduate degree, that is, MA or PhD, had higher scores relative to respondents who acquired a professional certification (M = 56.82%, SD = 18.11%). There were no differences between BA and MA/PhD or between BA and professional certification. Group differences are illustrated in Figure 3a.

Figure 3.

Average scores among respondents with different levels of (a) general and (b) scientific education.

For science education levels, respondents with a scientific tertiary degree were found to receive higher scores (BA: M = 64.09%, SD = 17.29%; MA/PhD: M = 68.53%, SD = 19.32%) relative to respondents who last studied science as part of a professional course (M = 56%, SD = 19.28%). Furthermore, respondents with a post-graduate degree, MA or PhD, but not BAs were found to receive higher scores relative to respondents who last studied science in middle or high school as part of mandatory studies (M = 58.07%, SD = 16.70%). No further differences between groups were found (see Figure 3b).

When looking at each of the scientific concepts separately, the results reveal a variation of correct answers across items, with averages ranging from 80% to 37% of correct answers. Random assignment to condition was the concept most widely understood with a mean of 80.7%. Double blind was the concept that was most difficult to understand and made use of in daily context, as only 37.8% answered this item correctly (Supplemental Appendix C). In addition, the items causality and history both displayed a relatively low average of around 50%.

Analysis of each of the scientific concepts based on the respondents’ level of education revealed differences in the ways these groups understand specific concepts, specifically the constructs random assignment to conditions, reliability, consensus, confounding variables, construct validity, and history (see Supplemental Appendix D). Effect sizes, however, were small except for the construct random assignment to conditions where the effect was moderate ( $η_{p}^{2}$ = .07). Post hoc comparisons revealed that differences in these constructs overall came from K–12 differing compared to BA and MA/PhD levels (see Figure 4); however, sample sizes in the education groups and, therefore, power of the post hoc tests were not always sufficient to clarify where the overall differences stemmed from.

Figure 4.

Percentage of correct answers for statistically significant scientific concepts of the scientific scale, based on respondents’ general level of education.

Individual differences albeit with regards to fewer concepts were found between respondents with different level of scientific education, namely, with regards to consensus, confounding variables, and blind/double blind with small effect sizes (see Supplemental Appendix E). However, and as revealed by the post hoc tests, the only notable differences between the science education levels were found for respondents with a BA who scored higher on confounding variables compared to those with a mandatory science education level and who scored higher on blind/double blind than those who had completed a professional course. As such, the differences in each of the scientific concepts that comprise scientific reasoning according to science education level were overall negligible and unsystematic.

3. Discussion

This series of studies aimed to develop, validate, and use a novel scientific reasoning scale to examine the public’s ability to reason with science in everyday contexts. Over the three studies presented here, we describe and demonstrate the ability of this scale to disclose science-based decisions and to predict the use of scientific information in other contexts. This is significant as scientific information is all around us, in the mainstream media, in social media, in school WhatsApp groups, and more. Thus, “people increasingly face the need to integrate information from science with their personal values and other considerations as they make important life decisions, such as those about medical care, the safety of foods, and a changing climate” (National Academies of Sciences, Engineering, and Medicine, 2017: 11); our scale contributes to expanding our understanding of these processes.

In study 1, we constructed a new everyday scientific reasoning scale, maintaining the scientific concepts chosen by Drummond and Fischhoff (2017) but reintroducing them in daily context. The resulting items were validated by expert ratings and cognitive validity ratings and were revised based on these ratings when necessary. Using these revised items, we tested the external validity of the scientific reasoning scale in study 2 by comparing respondents’ (N = 64) scores on the everyday reasoning scale with those of the lab-oriented reasoning scale, an analysis task, and scientific interest. This was accomplished by running a MIMIC model which showed excellent fit to the data and evidenced the hypothesized relations of everyday scientific reasoning to the external criteria. Together as a whole, studies 1 and 2 were able to confirm validity of the newly developed everyday scientific reasoning scale. Our approach to the investigation of scientific reasoning is that it is important to examine the use of it in its organic environment, where people encounter science and are required to make daily decisions. Hence, the everyday reasoning scale developed in this study more closely aligns to the theoretical framework discussed above in determining peoples’ ability to reason with scientific information in daily contexts. Further studies are still required to understand possible relationships between scale scores and different social attitudes. Also, future studies should focus on rigorous statistical testing of the final scale as different social cultures and world events may influence reasoning strategies.

After validating the scale, it was distributed to a large group of participants in study 3 to examine average capabilities of scientific reasoning and determine individual differences. Findings from study 3 demonstrate a relatively high level of scientific reasoning and, further, an association between everyday scientific reasoning and respondents’ level of education—as education level progresses from K–12 to tertiary education, scientific reasoning also increases. This finding is consistent with previous studies which report that individuals with higher levels of education exhibit greater scientific knowledge, interest in science, and an increased likelihood of taking part in scientific activities (Funk et al., 2017; National Science Board, 2018). However, while previous studies have investigated this association within a scientific context, our results illustrate a similar trend using day-to-day scenarios. Our findings highlight the social divide between those who have tertiary education and those who do not, even in their ability to reason with daily science such as nutrition. Our findings also demonstrate that respondents with tertiary education performed similarly irrespective of their major and whether their degree was scientific or not, suggesting formal science education may not be the main factor in determining people’s ability to reason with science in daily contexts. In other words, science reasoning seems to be developed through exposure to academia, rather than within school environments.

The fact that formal science education did not play a larger role in people’s ability to reason with science in our daily context can be explained in two ways. The first relates to the nature of science education both in schools and in higher education. Much of the learning focuses on content knowledge and skill development, and does not emphasize the nature of science, its complexity, and/or its uncertainties (Capps and Crawford, 2013). Furthermore, science in the classroom often does not relate to students’ experiences and interests outside of the classroom nor discuss ways in which science can be relevant and useful in everyday life (Leden et al., 2017). This makes the process of applying knowledge (both content and procedural) to daily contexts more complex and often unachievable. The second reason relates to education more generally and to the capabilities of non-scientists to reason with information we provided. It is possible (as we hope is the case) that basic scientific concepts, such as the ones used for building our scale, can be logically inferred by informed audiences and/or taught within general education courses. It may also be the case that some of the principles used in this scale are also relevant and used in other disciplines that are accessible to a more diverse population.

According to Bray et al. (2012), non-scientists (those who do not have tertiary science education and/or do not work in science) can understand complex scientific issues given they are made accessible and presented in a clear simple fashion. This notion was further established by Golumbic et al. (2020) who found that frequent users of a scientific data presentation platform, were able to understand information presented at a similar level, regardless of their formal science education. As such, our findings may indicate that framing science in daily scenarios facilitates the process of understanding scientific concepts and hence suggests that this is an appropriate framing for diverse populations to engage with science.

This idea has immense significance to the field of science communication as it addresses public processes of understanding and engaging with science and examines interactions between science and society (Bucchi and Trench, 2014). To improve these interactions, science communication training programs emphasize skills that can help scientists make their science more accessible to the public (Baram-Tsabari and Lewenstein, 2013; Barel-Ben David et al., 2020; Brownell et al., 2013; Mulder et al., 2008). One way to accomplish this is by presenting the scientific information in a known context and connecting the message and information to the audiences’ daily lives. By doing so, we translate the unfamiliar to familiar. Our findings reinforce this message, seeing that the everyday context contributes to the understanding of the scientific concept, despite differences found in general education levels.

In addition, our findings revealed that while some scientific reasoning concepts were more familiar to our respondents, others were less recognized and more complex. For example, random assignment to condition was the most widely understood concept. This is not surprising as this is a concept relevant in many social science disciplines as well as in the natural sciences. Perhaps this is a concept that is also more intuitive for people to comprehend. Conversely, three concepts seemed to be the trickiest for respondents to answer correctly: double blind was the most difficult concept for respondents to make use of in daily context, as were the concepts of causality and history. Double blind procedure is widely used in clinical trials for drugs but much less for other types of scientific studies, which may explain the low level of familiarity found here. Interpreting causality in data relations is a concept that is often misunderstood and mispresented in the media with correlation, leading to numerous humorous websites such as “spurious correlations.”¹ Causal inferences is oftentimes difficult for scientists as well; thus, it is not surprising to find that it is a difficult concept for laypeople to understand. Finally, history is a scientific reasoning concept that refers to events that may influence scientific finding and introduce bias that needs to be considered when reasoning with evidence. Since this concept is rather obscure and not well defined, it is possible that many people (as indicated by our finding) do not fully recognize the influence of history in scientific research.

Our reasoning scale is in no way an exhaustive scale for examining public reasoning capabilities across the scientific domain. It serves as one of many indicators that collectively investigate how non-scientists understand, use, and engage with science in their daily lives, grounding previous work and suggesting an empirical tool for its examination. This tool was used to measure scientific reasoning across the population in Israel, with the sample composition in study 3 similar albeit not identical to that of the adult Israeli population. While diverse in its nature, the Israeli population remains a small portion of global society, and thus, this work may not be reflective of worldwide trends. Future work should investigate the use of this tool more broadly, measuring scientific reasoning among different nationalities and cultures. For this purpose, a version of the reasoning scale has been translated, revalidated, and offered for the use of researchers in English (see Supplemental Appendix A), with the intention of doing so in additional languages. Although healthy lifestyle is a common concept in some cultural settings, it might differ across nationalities and cultures. Indeed, different sociocultural norms should be an integral part of broadening the use of this tool to other nationalities and locations (Bencze et al., 2020). This might demand further modifications to adjust the everyday context or theme of the scale, while measuring the same scientific concepts.

With the development of this scale, we wish to practically contribute to the idea proposed by Bauer et al. (2007), who advocate for broadening the scope and range of the types of science literacy data collected. They describe an ambitious research agenda for public understanding of science:

Contextualizing survey results through a reframing of the knowledge–attitude problem and within a framework of science indicators, analyzing data in search for cultural indicators, the global integration and analysis of longitudinal databases, and the mobilization of additional, preferably qualitative data streams with a long-term perspective. (p. 90)

Building on this agenda of incorporating multiple tools and methods, we propose the use of the everyday scientific reasoning scale to learn how non-scientists reason with science as part of a wider research agenda.

Supplemental Material

sj-docx-1-pus-10.1177_09636625221098539 – Supplemental material for Establishing an everyday scientific reasoning scale to learn how non-scientists reason with science

Supplemental material, sj-docx-1-pus-10.1177_09636625221098539 for Establishing an everyday scientific reasoning scale to learn how non-scientists reason with science by Yaela N. Golumbic, Keren Dalyot, Yael Barel-Ben David and Melanie Keller in Public Understanding of Science

Footnotes

Acknowledgements

The authors would like to thank the Applied Science Communication Research Group at the Technion for their help with the scale validation. Special thanks to Prof. Ayelet Baram-Tsabari for her support and mentoring throughout the process.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was partially funded by EIT FOOD, under project #FoodScienceClass (KD). Additionally, the distribution and development of the survey in study three was supported by the Israeli Ministry of Science and Technology under Grant Number 3-13697.

ORCID iDs

Yaela N. Golumbic

Keren Dalyot

Yael Barel-Ben David

Melanie Keller

Supplemental material

Supplemental material for this article is available online.

Notes

Author biographies

Yaela N. Golumbic is a science communication researcher, emphasizing on public engagement with science. Her work lies in the intersection of research and practice and aims to make science accessible and relevant to peoples’ day-to-day lives.

Keren Dalyot is a senior research associate at the Faculty of Education in Science and Technology at the Technion–Israel Institute of Technology. Her research focuses on how publics engage with socio-scientific issues in different platforms and how we can cultivate in youth interest in socio-scientific issues.

Yael Barel-Ben David is the director of the Citizen Lab—a hub for citizen science and science communication aiming to bring science and communities closer together. Her research interests include science, risk, and health communication.

Melanie Keller is a postdoctoral researcher at IPN—Leibniz-Institute for Science and Mathematics Education in Kiel, Germany. Her research focuses on emotional processes in science education and science communication.

References

Baram-Tsabari

Lewenstein

(2013) An instrument for assessing scientists’ written skills in public communication of science. Science Communication 35: 56–85.

Barel-Ben David

Garty

Baram-Tsabari

(2020) Can scientists fill the science journalism void? Online public engagement with science stories authored by scientists. PLoS ONE 15(1): e0222250.

Bauer

(1994) Scientific Literacy and the Myth of the Scientific Method. Champaign, IL: University of Illinois Press.

Bauer

(2009) The evolution of public understanding of Science-discourse and comparative evidence. Science, Technology and Society 14(2): 221–240.

Bauer

Allum

Miller

(2007) What can we learn from 25 years of PUS survey research? Liberating and expanding the agenda. Public Understanding of Science 16(1): 79–95.

Bencze

Pouliot

Pedretti

Simonneaux

Zeidler

(2020) SAQ, SSI and STSE education: Defending and extending “science-in-context.” Cultural Studies of Science Education 15(3): 825–851.

Bray

France

Gilbert

(2012) Identifying the essential elements of effective science communication: What do the experts say? International Journal of Science Education, Part B: Communication and Public Engagement 2(1): 23–41.

Brossard

Lewenstein

(2009) A critical appraisal of models of public understanding of science. In: Kahlor

Stout

(eds) Communicating Science: New Agendas in Communication. London: Routledge, pp. 11–39.

Brownell

Price

Steinman

(2013) Science communication to the general public: Why we need to teach undergraduate and graduate students this skill as part of their formal scientific training. Journal of Undergraduate Neuroscience Education 12(1): E6–E10.

10.

Bucchi

Trench

(2014) Routledge Handbook of Public Communication of Science and Technology, 2nd edn. London: Routledge.

11.

Burns

Medvecky

(2018) The disengaged in science communication: How not to count audiences and publics. Public Understanding of Science 27(2): 118–130.

12.

Capps

Crawford

(2013) Inquiry-based instruction and teaching about nature of science: Are they happening? Journal of Science Teacher Education 24(3): 497–526.

13.

Čavojová

Šrol

Ballová Mikušková

(2020) How scientific reasoning correlates with health-related beliefs and behaviors during the COVID-19 pandemic? Journal of Health Psychology. https://doi.org/10.1177/1359105320962266

14.

Claussen

Osborne

(2013) Bourdieu’s notion of cultural capital and its implications for the science curriculum. Science Education 97(1): 58–79.

15.

Diamantopoulos

Winklhofer

(2001) Index construction with formative indicators: An alternative to scale development. Journal of Marketing Research 38(2): 269–277.

16.

Drummond

Fischhoff

(2017) Development and validation of the Scientific Reasoning Scale. Journal of Behavioral Decision Making 30(1): 26–38.

17.

Funk

Gottfried

Mitchell

(2017) Science News and Information Today. Pew Research Center.

18.

Golumbic

Baram-Tsabari

Fishbain

(2016) Increased knowledge and scientific thinking following participation of school students in air-quality research. In: Proceedings of Indoor Air 2016. Available at: https://www.isiaq.org/docs/Papers/Paper832.pdf

19.

Golumbic

Fishbain

Baram-Tsabari

(2020) Science literacy in action: Understanding scientific data presented in a citizen science platform by non-expert adults. International Journal of Science Education, Part B: Communication and Public Engagement 10(3): 232–247.

20.

Bentler

(1999) Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal 6(1): 1–55.

21.

Huxster

Slater

Leddington

LoPiccolo

Bergman

Jones

, et al. (2018) Understanding “understanding” in Public Understanding of Science. Public Understanding of Science 27: 756–771.

22.

Jasanoff

(2014) A mirror for science. Public Understanding of Science 23(1): 21–26.

23.

Kallerud

Ramberg

(2002) The order of discourse in surveys of public understanding of science. Public Understanding of Science 11(3): 213–224.

24.

Kind

Osborne

(2017) Styles of scientific reasoning: A cultural rationale for science education? Science Education 101(1): 8–31. https://doi.org/10.1002/sce.21251

25.

Klahr

Dunbar

(1988) Dual space search during scientific reasoning. Cognitive Science 12(1): 1–48.

26.

Kuhn

(2002) What is scientific thinking and how does it develop? In: Goswami

(ed.) Blackwell Handbook of Childhood Cognitive Development. Hoboken, NJ: Blackwell, pp. 371–393.

27.

Leden

Hansson

Redfors

(2017) From black and white to shades of grey. Science & Education 26(5): 483–511.

28.

Ministry of Science (2018) Perceptions and attitudes of the Israeli public on science, technology and space. Tel Aviv: GeoCatography.

29.

Morris

Croker

Masnick

Zimmerman

(2012) The emergence of scientific reasoning. In: Kloos

Morris

Amaral

(eds) Current Topics in Children’s Learning and Cognition. London: InTech, pp. 61–82. https://doi.org/10.5772/53885

30.

Mulder

HAJ

Longnecker

Davis

(2008) The state of science communication programs at universities around the world. Science Communication 30(2): 277–287.

31.

Muthén

(1998-2017) Mplus User’s Guide (8th ed.). Los Angeles, CA: Muthén & Muthén.

32.

National Academies of Sciences, Engineering, and Medicine (2016) Science Literacy: Concepts, Contexts, and Consequences. Washington, DC: National Academies of Sciences, Engineering, and Medicine. https://doi.org/10.17226/23595

33.

National Academies of Sciences, Engineering, and Medicine (2017) Communicating Science Effectively: A Research Agenda. Washington, DC: National Academies of Sciences, Engineering, and Medicine. https://doi.org/10.17226/23674

34.

National Research Council (2012) A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. Washington, DC: The National Academies Press.

35.

National Science Board (2006) Science and Engineering Indicators. Alexandria, VA: National Science Foundation.

36.

National Science Board (2018) Science and Engineering Indicators 2018. Alexandria, VA: National Science Foundation.

37.

Pasek

(2018) It’s not my consensus: Motivated reasoning and the sources of scientific illiteracy. Public Understanding of Science 27(7): 787–806.

38.

Roberts

Bybee

(2014) Scientific literacy, science literacy, and science education. In: Lederman

Abell

(eds) Handbook of Research on Science Education, Vol. II. London: Routledge, pp. 559–572.

39.

Roos

(2014) Measuring science or religion? A measurement analysis of the National Science Foundation sponsored science literacy scale 2006–2010. Public Understanding of Science 23(7): 797–813.

40.

Ryder

(2001) Identifying science understanding for functional scientific literacy. Studies in Science Education 36(1): 1–44.

41.

Shah

Michal

Ibrahim

Rhodes

Rodriguez

(2017) What makes everyday scientific reasoning so challenging? In: Ross

(ed.) Psychology of Learning and Motivation—Advances in Research and Theory, Vol. 66. Amsterdam: Elsevier, pp. 251–299.

42.

Sharon

Baram-Tsabari

(2020) “Can science literacy help individuals identify misinformation in everyday life?” Science Education 104(5): 873–94.

43.

Tal

Dierking

(2014) Editorial learning science in everyday life. Journal of Research in Science Teaching 51(3): 251–259.

44.

Woolley

Deal

Green

Hathenbruck

Kurtz

Park

TKH

, et al. (2018) Undergraduate students demonstrate common false scientific reasoning strategies. Thinking Skills and Creativity 27: 101–113.

45.

Zeidler

Sadler

Applebaum

Callahan

(2009) Advancing reflective judgment through socioscientific issues. Journal of Research in Science Teaching 46(1): 74–101.

46.

Zimmerman

(2005) The development of scientific reasoning skills: what psychologists contribute to an understanding of elementary science learning. Final report to the National Research Concil.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.08 MB