The Role of Internet-Specific Justification Beliefs in Source Evaluation and Corroboration During Web Search on an Unsettled Socio-Scientific Issue

Abstract

This study explored relationships between Internet-specific justification beliefs and source evaluation and corroboration during Web search. Fifty university students completed the Internet-Specific Epistemic Justification Inventory (ISEJ), which targeted beliefs concerning the justification of Internet-based knowledge claims about natural science issues. Two to five days later, they conducted a Web search in order to communicate a justified position regarding an unsettled and unfamiliar socio-scientific issue. Using think-aloud and trace methodologies, participants’ source evaluation and corroboration behaviors were examined. Furthermore, the extent and relevance of their post-search written justifications for their recommendation about the controversial issue were analyzed. Results showed that beliefs in justification by authority positively predicted comments regarding source evaluation, the percentage of visited websites that were listed beyond the first three Google search results, and the likelihood of opening multiple browser tabs. Beliefs in personal justification negatively predicted comments regarding corroboration of information across websites and the number of relevant aspects included in the written justifications. Finally, participants with stronger beliefs in justification by multiple sources gave more extensive justifications for their recommendation and included more relevant aspects in those justifications.

Keywords

internet-specific epistemic beliefs justification for knowing web search think-aloud trace data socio-scientific issues

Introduction

Should we use sunscreen containing nanoparticles? What are the advantages of sunscreen containing nanoparticles and how safe is it to use? Might it have any adverse effects on our health? Individuals increasingly turn to the Internet to find information and learn about such unsettled socio-scientific issues, for which they typically possess low prior knowledge and, thus, can be considered laypeople (Brossard, 2013; Su et al., 2015). They typically use general search engines, especially Google (e.g., Purcell et al., 2012), which provides quick and easy access to a large amount and a great variety of information sources on all kinds of topics. However, because anyone can publish freely on the Internet, usually without editorial control, information on controversial scientific issues encountered online can vary considerably in terms of accuracy and completeness, with websites listed at the top of the first search engine results page (SERP) not necessarily being the most reliable or balanced ones (Lewandowski, 2011). Moreover, different websites more often than not present conflicting accounts on the issues in question. Thus, Internet users themselves need to critically justify claims about socio-scienctific issues they encounter online before they can accept them as valid knowledge.

However, individuals can hold different beliefs regarding how to justify science-related knowledge claims, such as by drawing on their own prior knowledge and reasoning, by judging the authoritativeness of the source, or by cross-checking and comparing multiple information sources for consistency (e.g., Bråten, Brandmo, et al., 2019; Bråten & Ferguson, 2014; Greene et al., 2008). The main goal of the present study was to examine if and how university students’ beliefs concerning the justification of knowledge claims about natural science issues encountered on the Internet would predict how they actually searched for and evaluated information in a naturalistic Web search context. We asked them to learn about an unsettled and unfamiliar socio-scientific issue: nanoparticles in sunscreen. To gain insights into what they did and thought during their Web search, we examined their query and navigation logs (i.e., trace data) and think-aloud comments (i.e., verbal protocol data). Before we further specify the research questions and hypotheses for our study, we discuss relevant theoretical frameworks and prior empirical work.

Theoretical and Empirical Background

Information Problem Solving on the Internet

The process of learning about complex issues by searching the Internet can be conceptualized as a problem-solving process driven by an information problem (e.g., Brand-Gruwel et al., 2009). Brand-Gruwel et al. (2009) described five component skills for successfully solving information problems, that is, for making informed decisions about scientific issues by searching the Internet: (1) defining the information problem at hand; (2) formulating corresponding search queries that are submitted to the search engine; (3) evaluating search results provided on the search engine results page(s) (SERP) to determine which information sources to access; (4) processing and evaluating information provided by websites in the light of source parameters, own prior knowledge, and information from other sources; and (5) synthesizing and integrating information across multiple websites to reach a solution to the information problem, that is, to build a comprehensive mental representation of the issue and communicate the solution (for a similar model, see Kiili et al., 2018).

Previous research, however, has shown that when solving information problems on the Internet, laypersons focus predominantly on the first few information sources presented by the search engine (e.g., the top three search results; Kim et al., 2015; Lorigo et al., 2008; Van Deursen, 2012; Wineburg & McGrew, 2019). Also, without explicit prompts or instructions, laypersons seldom engage in source evaluation on SERPs and on websites, that is, they rarely consider or check the authors’ or information providers’ expertise and trustworthiness or the up-to-dateness and scientific evidence of the information (for a recent review, see Bråten, Stadtler, et al., 2018). Further, they rarely seem to corroborate evidence across websites as a basis for their trustworthiness judgments (List & Alexander, 2018; Wiley et al., 2009). In contrast, domain experts as well as professional fact checkers display more and better source evaluation and consider more search results beyond the first few results provided by the search engine (Brand-Gruwel et al., 2017; Rouet et al., 1997; von der Mühlen et al., 2016; Wineburg, 1991; Wineburg & McGrew, 2019). Moreover, professional fact checkers typically open multiple tabs simultaneously in the browser to compare and navigate between websites and to read “laterally” (e.g., Wineburg & McGrew, 2019).

Yet, laypersons differ both with respect to how important or necessary they consider it to justify science-related knowledge claims and with respect to how such claims should be justified, such as through personal justification, justification by authority, or justification by multiple sources (e.g., Bråten, Brandmo, et al., 2019; Bråten & Ferguson, 2014; Ferguson et al., 2012, 2013; Greene et al., 2008). Such stances or beliefs about the justification of knowledge claims belong to the construct of personal epistemology, which refers to individuals’ conceptions about the nature of knowledge and the process of knowing (e.g., Bråten, 2011; Hofer & Bendixen, 2012; Hofer & Pintrich, 1997).

Beliefs Concerning the Justification of Knowledge Claims

Personal epistemology may refer not only to beliefs about the justification of knowledge claims, but also to beliefs about the certainty, simplicity, and source of knowledge (e.g., Bråten, 2011; Hofer & Bendixen, 2012; Hofer & Pintrich, 1997). It can be argued, however, that beliefs concerning the justification of knowledge claims are at the core of personal epistemology (see also, Sandoval et al., 2016). Further, Greene and colleagues have argued that beliefs about the justification of knowledge cannot be captured by a single dimension but, rather, should be differentiated into several dimensions addressing different means of justifying knowledge claims (Greene et al., 2008, 2010).

In accordance with this view, Bråten and colleagues have distinguished between the justification of knowledge claims by three different means, that is, through personal justification, justification by authority, and justification by multiple sources (e.g., Bråten, Brandmo, et al., 2019; Bråten & Ferguson, 2014). In this categorization, personal justification refers to the validation of knowledge claims by relying on one’s own judgments of plausibility, which can be based on one’s own opinion about the issue or, alternatively, on one’s own knowledge and reasoning. Directly evaluating the validity of a knowledge claim based on one’s own personal judgment is similar to a first-hand approach to evaluation, as described by Bromme and colleagues (e.g., Bromme et al., 2010; Stadtler & Bromme, 2014). Such an approach cannot be used reliably by laypersons, however, due to their lack of prior domain knowledge (Bromme & Goldman, 2014; Scharrer et al., 2012).

In contrast, justification by authority refers to an indirect validation of knowledge claims by evaluating the competency or expertise of the source that provides them. This approach is similar to what Bromme and colleagues described as a second-hand approach to evaluation (Bromme et al., 2010; Stadtler & Bromme, 2014). In particular, asking oneself “whom to believe” (Stadtler & Bromme, 2014) may be an adaptive strategy when laypersons read conflicting documents on complex and unfamiliar scientific issues (Barzilai et al., 2015). That said, it should be noted that domain experts are also more able than laypersons to evaluate the expertise and trustworthiness of information sources (e.g., Brand-Gruwel et al., 2017; Rouet et al., 1997; von der Mühlen et al., 2016; Wineburg, 1991).

Finally, justification by multiple sources concerns validating knowledge claims by cross-checking and comparing several external information sources. When multiple information sources provide consistent accounts regarding a particular knowledge claim, this approach, in and of itself, might result in that claim being considered valid by readers. However, when contradictory information is encountered across multiple information sources, readers may also have to draw upon personal justification or justification by authority in order to decide which of the conflicting claims may be valid (Braasch & Bråten, 2017; Stadtler & Bromme, 2014).

In the area of multiple document comprehension, a number of recent empirical studies have been conducted within the trichotomous framework of justification beliefs involving personal justification, justification by authority, and justification by multiple sources. As will be summarized in the following, they have indicated that these types of beliefs regarding the justification of knowledge differentially predict individuals’ learning and comprehension performance when reading multiple documents about controversial socio-scientific issues (for reviews, see Bråten & Strømsø, 2020; Bråten et al., 2016).

The Role of Justification Beliefs in Multiple Document Comprehension

Beliefs in justification of knowledge claims through corroboration across multiple sources have been shown to be positively related to multiple document comprehension among both secondary-school and university students. For example, students’ reports of such beliefs have been found to be related to more integrated argumentation in essays written after the reading of multiple documents on a controversial socio-scientific issue (e.g., Bråten et al., 2013; Bråten, Ferguson, et al. 2014; Ferguson & Bråten, 2013). Further, Bråten, Anmarkrud, et al. (2014) found that beliefs in the importance or necessity of justifying knowledge claims in natural science by checking multiple sources for consistency positively predicted secondary-school students’ engagement in understanding a set of science texts. Using verbal protocol analysis, Greene and colleagues (Greene et al., 2014, 2018) found that the extent to which university students justified knowledge claims based on their consistency with other claims while conducting a Web search on a scientific topic was positively related to their learning or comprehension outcomes (although not statistically significantly so).

In contrast, beliefs in personal opinion as a means of justifying knowledge claims have been found to be negatively related to multiple document comprehension (e.g., Bråten et al., 2013; Ferguson & Bråten, 2013). Further, Barzilai et al. (2015) found that university students who relied more on their own opinion or knowledge (as indicated by think-aloud protocols) while reading a set of partly conflicting blog-posts on an unfamiliar socio-scientific issue engaged less in source evaluation. Moreover, comments referring to participants’ own opinion or knowledge were also negatively related to the richness of students’ written argumentation in that study.

Beliefs in justification by authority have been found to be unrelated to multiple document comprehension in most studies (e.g., Bråten et al., 2013; Bråten, Ferguson, et al., 2014). Notably, in one study, such beliefs were positively related to the multiple document comprehension of ethnic-minority high-school students, whereas no such relation was found for ethnic-majority high-school students (Strømsø et al., 2016). With regard to Web-based learning, Mason and colleagues (Mason et al., 2010a, 2010b, 2011) found that students who reflected more on the scientific evidence addressed by the websites during their Web search (as indicated by think-aloud protocols), also had better learning outcomes.

Internet-Specific Epistemic Beliefs

Another line of research on epistemic beliefs has focused on the Internet as a particular type of knowledge resource, that is, on Internet-specific epistemic beliefs. In other words, these beliefs concern what knowledge and knowing are like on the Internet (Bråten et al., 2005). The 36-item Internet-Specific Epistemological Questionnaire (ISEQ) developed by Bråten et al. (2005) addressed beliefs concerning the certainty, simplicity, and source of Internet-based knowledge, as well as beliefs concerning the justification of knowledge claims encountered on the Internet. With respect to beliefs concerning the justification of knowledge claims encountered on the Internet through the use of multiple sources, reasoning, and prior knowledge activation (as measured by a single dimension comprising 4 items), several studies have indicated that stronger beliefs on this dimension positively predict students’ integrated and balanced argumentation, critical evaluation of websites, and adaptive self-regulatory or online help-seeking strategies during Web searches (Chiu et al., 2013; Kammerer et al., 2013; Knight et al., 2017; Lee et al., 2014; Strømsø & Bråten, 2010; for an extensive review of studies using the ISEQ, see Bråten, Brandmo, et al., 2019). Interestingly, Knight et al. (2017), who examined pairs of undergraduate students jointly searching the Internet for information about an unfamiliar and controversial scientific issue, did not find any significant relationships between Internet-specific justification beliefs as measured by the ISEQ and behavioral measures obtained during Web search, specifically the number of unique queries formulated or the number of unique websites visited. Still, these authors found that the more students believed that Internet-based knowledge claims need to be justified, the less trustworthy they rated the least trustworthy websites they encountered during their Web search. In addition, a tendency for a reversed pattern was found for the most trustworthy websites encountered, indicating that students with higher justification beliefs were better at discriminating between trustworthy and untrustworthy information sources.

Kammerer et al. (2015) distinguished between Internet-specific beliefs concerning justification by multiple sources and beliefs concerning personal justification in students’ course work, using the original ISEQ justification items to capture these two dimensions (including two items each). Results showed that the more individuals believed that Internet-based knowledge claims need to be compared and corroborated across multiple sources, the more time they spent on reliable websites during Web search on a controversial medical issue, and the more likely they were to make an informed decision in accordance with the reliable websites. In contrast, the more individuals believed that knowledge claims should be evaluated in relation to their own knowledge and reasoning, the more they focused on less reliable, commercial websites.

Finally, building on the trichotomous framework of justification beliefs (Ferguson et al., 2012, 2013; Greene et al., 2008, 2010), Bråten, Brandmo, et al. (2019) recently developed and validated a new instrument: The Internet-Specific Epistemic Justification Inventory (ISEJ). This instrument addresses justification beliefs when using the Internet as a knowledge resource in terms of the three dimensions of justification by multiple sources, personal justification, and justification by authority, with four items included in each dimension. Importantly, whereas research on Internet-specific epistemic beliefs using the ISEQ has targeted beliefs about Internet-based knowledge and knowing in relation to students’ course work or in general (see also, Strømsø & Kammerer, 2016), the ISEJ, similar to Ferguson et al.’s (2013) Justification for Knowing Questionnaire (JFK-Q), is designed to assess beliefs about justification for knowing on the Internet in relation to a specific content domain (such as natural science, educational science, or history). Thus, the domain-specific nature of epistemic beliefs is taken into consideration (Bråten et al., 2009; Muis et al., 2006; Sandoval et al., 2016).

In their validation study of the ISEJ, including pre-service teachers, Bråten, Brandmo, et al. (2019) found that individuals strongly believing that information about educational topics encountered on the Internet needs to be corroborated by comparing multiple sources perceived themselves to be more competent and reportedly put more time, effort, and persistence into reading about such topics on the Internet. Conversely, individuals who believed in evaluating information about educational topics on the Internet by means of their own prior knowledge and reasoning perceived what they read about such topics on the Internet to be less difficult and less confusing. This could be due to higher prior knowledge, but, alternatively, it also could indicate a lack of awareness of the vast amounts of heterogeneous information that are available on the Internet (Bråten, Brandmo, et al., 2019). Finally, pre-service teachers who believed that information about educational topics encountered on the Internet needs to be justified on the basis of the competency or expertise of the source judged it less likely that they could obtain key information and solutions to academic issues on the Internet (perhaps because of lower perceived own competency). Further validation of the ISEJ recently has been provided by Cheng et al. (2020), who investigated structural relationships among upper-secondary school students’ beliefs about Internet-specific justification in the domain of science and more general beliefs about knowledge and justification for knowing in science, using a Chinese version of the ISEJ. However, it should be noted that this preliminary validation research was limited to self-report data, which might deviate from individuals' actual search and evaluation behaviors during Web search (cf. Knight et al., 2017), as also indicated by previous studies comparing individuals’ reported and actual search and evaluation behavior (e.g., Flanagin & Metzger, 2007; List & Alexander, 2018).

For example, List and Alexander (2018) examined the relationship between undergraduate students’ reported practices with regard to source evaluation and corroboration across multiple sources and their actual behavior when reading and navigating across a set of six digital documents about a controversial issue. However, no relationships between reported and actual behavior were found. Specifically, students’ reported degree of checking the up-to-dateness, author, or about-us information of a website was unrelated to their actual behaviors (i.e., the likelihood of accessing such information) during the digital reading task. Likewise, students’ reported degree of corroborating information and checking its completeness by seeking out additional websites was unrelated to the number of unique websites visited or revisited during task processing, which were considered indicators of corroborating information across multiple websites or validating information in light of content presented in other websites, respectively.

The Present Study

Given this theoretical and empirical background analysis, we set out to explore university students’ beliefs concerning the justification of knowledge claims about natural science topics encountered on the Internet, as measured with the ISEJ questionnaire (Bråten, Brandmo, et al., 2019). In particular, we aimed at investigating the differential relationships between the dimensions of justification by multiple sources, personal justification, and justification by authority on the one hand, and individuals’ actual source evaluation and corroboration behaviors during Web search on the unsettled and unfamiliar issue of nanoparticles in sunscreen on the other (for similar approaches, see Knight et al., 2017; Ulyshen et al., 2015). To achieve this aim, we analyzed individuals' think-aloud protocols and query and navigation logs, as well as the extent and relevance of their written justifications for their recommendations regarding the controversial issue that they produced after the Web search.

Given previous findings regarding the role of domain expertise or domain knowledge in individuals’ query and navigation behavior during Web search, we expected individuals’ prior domain knowledge to positively predict the number of websites they would visit as well as the number of queries they would submit to the search engine during the course of their Web search (Monchaux et al., 2015; Sanchiz et al., 2017; White et al., 2009; Zhang et al., 2015). Likewise, we expected prior domain knowledge to positively predict comments indicating source evaluation and corroboration (Brand-Gruwel et al., 2017; Rouet et al., 1997; von der Mühlen et al., 2016; Wineburg, 1991) and the extent and relevance of the post-reading written justifications (Bråten et al., 2013; Bråten & Strømsø, 2010; Gil et al., 2010). Therefore, we included individuals’ prior domain knowledge as an additional predictor variable in this study.

With respect to the role of justification beliefs after controlling for prior domain knowledge, we expected beliefs in justification by multiple sources to positively predict the extent and relevance of individuals’ written justifications (cf. Bråten et al., 2011). Further, based on the assumption that beliefs in justification by multiple sources would be related to increased efforts to evaluate and corroborate sources and explore a wider range of information (Bråten, Brandmo, et al., 2019; Knight et al., 2017), we expected that such beliefs would positively predict individuals’ extent of query and navigation activites as well as their think-aloud comments concerning source evaluation and corroboration.

In contrast, because we asked students to research a topic that we presumed was quite unfamiliar to them, we expected beliefs in personal justification to negatively predict the extent and relevance of students’ written justifications regarding the socio-scientific issue (Bråten et al., 2008), as well as their behaviors and comments indicating source evaluation (Barzilai et al., 2015) and corroboration (Bråten et al., 2011). Still, we considered it likely that beliefs in personal justification, that is, justification based on own prior knowledge and reasoning, would be positively correlated with individuals’ prior domain knowledge regarding the scientific issue in question.

Finally, we expected beliefs in justification by authority to positively predict indications of source evaluation because such beliefs likely increase readers’ attention to the competency and expertise of the source, including its presentation of scientific evidence (Bråten et al., 2008). However, we did not expect beliefs in justification by authority to be uniquely related to the extent or relevance of their written justifications (Bråten et al., 2013), nor did we assume that justification by authority would be uniquely related to individuals’ query or navigation activities, with such activities more likely to be related to justification by multiple sources than to justification by authority.

Method

Participants

Participants were 54 university students from different majors at a large German university, who were compensated with 10 € for their participation. However, three participants did not think aloud at all, and for one other participant, technical problems occurred. Thus, these four participants were excluded from all analyses, resulting in a final sample of 50 students (40 female) with an overall mean age of 24.52 years (SD = 3.44). Thirty-six students were majoring in social sciences and humanities, and 14 in natural sciences. The vast majority (92%) reported to have very low or rather low prior knowledge of issues concerning nanotechnology on a 5-point scale ranging from 1 (very low) to 5 (very high). The study was approved in advance by the local ethics committee, and participants gave their written consent at the beginning of both the online questionnaire and the lab session.

Task and Materials

Web Search Task

To examine participants’ search and evaluation behaviors during Web search about an unsettled and controversial natural science issue, they were presented with a request from a fictitious friend who was said to work as a landscape gardener. As he purportedly spent a lot of time outdoors, the use of sunscreen was extremely important for him. In the task scenario, the friend reported that he considered using sunscreen containing mineral nanoparticles (i.e., particles of zinc oxide and titanium oxide) in the future. He further reported that advantages of these particles are (1) that they effectively reflect ultraviolet (UV) light, thus eliminating the need for chemical UV filters that convert UV radiation on the skin into heat and that can cause allergies or unwanted hormonal side effects; (2) that such side effects are not known to occur with UV filters containing mineral nanoparticles; and (3) that particularly high sun protection factors can be achieved through the use of nanoparticles. However, the fictitious friend was said to wonder whether sunscreen containing nanoparticles may also pose certain health risks. Therefore, he was unsure whether he should use a sunscreen containing nanoparticles in the future.

Based on this information, participants were asked to conduct a Web search to learn about possible health risks associated with using sunscreen containing nanoparticles, with the goal of giving the friend a justified recommendation as to whether he should or should not use sunscreen containing nanoparticles. For their Web search, participants were provided open access to the Internet via the Mozilla Firefox browser, with Google presented as the starting page. They were free to use any kind of information sources they could find. Time was limited to 20 minutes.

The Internet-Specific Epistemic Justification Inventory (ISEJ)

To assess participants’ epistemic beliefs concerning justification of knowledge claims on the Internet, we used the Internet-Specific Epistemic Justification Inventory (ISEJ; Bråten, Brandmo, et al., 2019), which is based on a multidimensional conceptualization of justification for knowing (Ferguson et al., 2012, 2013; Greene et al., 2008, 2010). The 12-item-questionnaire targets justification beliefs when using the Internet as a knowledge resource and captures the three dimensions of justification by multiple sources, personal justification, and justification by authority. In this study, the items were formulated to address natural science as the content domain. All items of the ISEJ are displayed in the Appendix. The items were presented in German to the participants. Participants rated each item on a 10-point anchored scale ranging from 1 (strongly disagree) to 10 (strongly agree).

The four items that assessed beliefs concerning justification by multiple sources focused on the evaluation of information found on the Internet by checking several information sources and comparing information across websites (sample item: “To evaluate whether information I find on the Internet about a natural science topic is reliable, I check whether it is consistent with information on other websites.”) The reliability estimate (Cronbach’s α) for this dimension was .91 in the present study.

The four items that assessed beliefs about personal justification concerned the evaluation of information found on the Internet on the basis of prior knowledge and reasoning (sample item: “When I find information about a natural science topic on the Internet, I evaluate whether this information is consistent with my own understanding of the topic.”) The reliability estimate for this dimension was α = .89 in the present study.

Finally, the four items that assessed beliefs concerning justification by authority focused on the evaluation of information located on the Internet in light of the competency and expertise of the source (sample item: “When I read something about a natural science topic on the Internet, I evaluate whether this information is written by an expert.”) The reliability estimate for this dimension was α = .94 in this study.

Prior Domain Knowledge

To measure students’ prior knowledge regarding nanotechnology, we used an adapted version of the Public Knowledge of Nanotechnology (PKNT) test by Lin et al. (2013; also see Gottschling et al., 2019). On this knowledge test, participants answered eight multiple-choice questions with four response alternatives about different concepts in nanotechnology, such as size and scale, structure of matter, or current applications of nanomaterials. The internal consistency reliability (Cronbach’s α) of participants’ scores was .58 in the present study, and test-retest reliability (Pearson’s r) in an independent sample of university students (n = 37) was .74.

For further characterization of the sample, additional individual differences measures were administered (see Supplemental Material for an overview of these measures as well as for descriptive information and intercorrelations involving these measures).

Procedure

The study consisted of two separate sessions: an online session and a lab session. First, the Internet-Specific Epistemic Justification Inventory (ISEJ; Bråten, Brandmo, et al., 2019) was administered as an online questionnaire two to five days before the lab session. In the individual lab session, which lasted approximately one hour, participants used a laptop to self-report demographic information and prior knowledge, interest, and attittudes about nanotechnology, before they completed the prior knowledge multiple-choice test as well as a computer-based version of a German reading comprehension test (Schneider et al., 2007).

When these measures were completed, participants were informed that they would conduct a Web search and that they would be asked to think aloud during their task processing. Specifically, they received the following written think-aloud instruction consistent with the standard proposed by Ericsson and Simon (1993):

In the following you will conduct a Web search. We want to know what you think during your Web search. For this purpose, please ‘think aloud’ during the whole Web search task. By ‘thinking aloud’ we mean that you continuously say everything you think about. Don’t try to plan what you say or change your actions as a result. Act as if you were alone, with no one listening, and just keep talking. It is very important that you verbalize everything that comes to mind. It is not important that your thoughts are thought through or formulated in a linguistically correct way. If you’re silent for a while, the experimenter will remind you to keep talking. As long as you only read, you don't need to speak; that is, you should not read aloud. To practice the ‘thinking aloud’, you will first be presented with a website on the history of your university, which you should read through and verbalize everything that comes to your mind while reading. In this way you can get used to ‘thinking aloud’.

Participants were presented with the website about the university’s history through the Mozilla Firefox browser on a 15.6-inch laptop (1920 x 1080 pixels resolution). They practiced thinking aloud for about five minutes before they received another written instruction for the Web search task including the task scenario as described in the ‘Web Search Task’ section. Participants were also reminded to verbalize all thoughts that came to mind during the Web search. Then, the Mozilla Firefox browser was opened again and the Google search engine was presented so that they could start their Web search. The browser cache was cleared for each participant.

Participants could navigate freely on the Internet for 20 minutes, entering any query and accessing any website and service they wanted. Three minutes before the time was up, the experimenter informed them about the remaining time. Whenever they stopped verbalizing their thoughts for approximately 30 seconds, the experimenter prompted them to think aloud. During the whole Web search session, participants had a paper copy of the task scenario and instructions available. Audio and screen recordings as well as trace-data logs of participants’ query and navigation activities were recorded with the SMI (SensoMotoric Instruments) ExperimentCenter 3.7 software. This software can also be used for eye-tracking recordings, which was not taken advantage of in the present study.

After the Web search, participants were asked to give their recommendation regarding the use of sunscreen containing nanoparticles (yes or no) and provide a written justification for their recommendation.

Data Analysis

Verbalizations Concerning Source Evaluation and Corroboration

Participants’ think-aloud protocols were transcribed using audio and screen recordings and transcripts were segmented into separate comments. In segmenting the protocols, a comment was defined as a continuous statement referring to a particular idea that could vary from only a few words to several sentences (cf. Chi et al., 1989). The transcripts, on average, contained 637.88 words (SD = 354.49; min = 91, max = 1963) and 36.98 segments (SD = 17.88; min = 10, max = 90). Each segment was coded in terms of whether or not it referred to source evaluation and corroboration of information across websites. Further, for each comment referring to source evaluation or corroboration, we coded whether it occurred (a) while participants were inspecting SERPs (i.e., while using Google or Google scholar, respectively) or (b) while they were processing a website.

Source evaluation was coded if participants referred to the competency, expertise, trustworthiness, objectivity, potential biases, scientificness, or up-to-dateness of a source (i.e., author, publisher, information provider) or a document (i.e., an article, a website), to the presentation of scientific evidence or inclusion of scientific references, or to a lack of any of these source features (cf. List & Alexander, 2018; Strømsø & Bråten, 2014). Segments were also coded in this category if participants expressed the need to find competent, scientific, scholarly, academic, unbiased, trustworthy, or up-to-date information. Example comments are: “OK, the article mentions scientific studies and, thus, could be trusted.”; “They are definitely selling something.”; “I don’t know whether the source is reliable or not.”; and “Maybe I can find some pages from renowned magazines or from universities that have researched this.”

Corroboration of information across websites was coded if participants commented on the consistency or inconsistency of particular information with information from other websites (cf. List & Alexander, 2018). Segments were also coded in this category if participants commented on the need to check the consistency of information across websites or to find divergent perspectives. Example comments are “There they say again that the skin can actually repel these nanoparticles, but I’m not so sure whether this is true, because I have read on the other sites that it isn’t like that, or that they don’t know yet.”; “OK, here it sounds pretty dangerous in contrast to the other article, which was rather half and half.”; “Well, there they actually write the same as on the other pages.”; and “Let’s see what another page has to say about that.”

Two raters independently coded the think-aloud protocols of 11 participants (22%), resulting in an inter-rater reliability (Cohen’s κ) of .93 for source evaluation and .77 for corroboration. All disagreements (4 for source evaluation and 10 for corroboration) were resolved through discussion. Subsequently, one rater coded the rest of the protocols with respect to source evaluation and corroboration. Finally, the second rater was provided with a list containing only those segments coded as indicating source evaluation or corroboration by the first rater (in a blinded version and without knowing that the list contained only coded segments) and was asked to independently code each of these segments as “source evaluation” and/or “corroboration” or as “neither/nor.” The inter-rater reliability (Cohen’s κ) in this final stage was .89 for source evaluation and .96 for corroboration, with all disagreements (14 for source evaluation and 4 for corroboration comments) resolved through discussion.

Query and Navigation Behaviors

In accordance with previous research (e.g., Kim et al., 2015; Knight et al., 2017; List & Alexander, 2018; Van Deursen, 2012; Wineburg & McGrew, 2019), we analyzed the trace-data logs and screen recordings to obtain the following five indicators of participants’ query and navigation behaviors: (a) the number of unique search queries made during Web search as an indicator of an aim to explore a wider range of information (cf. Knight et al., 2017; Shah et al., 2016), with unique search queries defined as differing in at least one word or in the order of words from the other queries a participant submitted during the Web search (corrections because of typing errors were not considered as new queries); (b) the number of unique websites visited during Web search as an indicator of consideration and corroboration of information across different sources (cf. Knight et al., 2017; List & Alexander, 2018); (c) the number of websites revisited at least once as an indicator of re-conceptualization or re-evaluation of information in a website in light of information presented in other websites (cf. List & Alexander, 2018); (d) the percentage of search results selected from SERP ranks 1–3 (i.e., the number of selected search results from ranks 1–3 divided by the number of all selected search results), as an indicator of participants' degree of bias toward the rank order (Kim et al., 2015; Van Deursen, 2012); and (e) whether or not participants opened multiple tabs simultaneously in the browser as an indication of “lateral reading” (cf. Wineburg & McGrew, 2019).

Written Justifications

Following previous research in the area of multiple document comprehension (Bråten, Brante, et al., 2018; De La Paz et al., 2017; Latini et al., 2019), we analyzed the extent of the the written justifications that participants provided for their recommendations to the fictitious friend after the Web search. Specifically, we counted the number of words of the written justifications, with this metric indicating the comprehensiveness of the task solution as well as task engagement (Bråten, Brante, et al., 2018; De La Paz et al., 2017; Latini et al., 2019) and serving as a proxy for the quality of the written product (e.g., Guo et al., 2013).

Additional qualitative analysis involved the coding of the content of the written justifications in terms of whether or not they mentioned that (a) risks are low or no risks are known and/or that advantages predominate, (b) a coating procedure could reduce the potential risks associated with nanoparticles, (c) there is (only) risk when the skin is damaged, (d) sprays are risky because nanoparticles could be inhaled, (e) risks are high or uncertain, (f) the advantage is only cosmetic, and (g) mineral sunscreen without nanoparticles could be used instead. The coding scheme emerged from both a consultation of nanotechnology experts (from the Leibniz Institute for New Materials, Saarbrücken, Germany) on relevant aspects of the topic and a bottom-up analysis of the written justifications. We used the number of relevant aspects each participant mentioned in their recommendation as an additional dependent measure. Further, it was coded whether any sources (e.g., particular websites, institutions, scientists, or studies) were mentioned in the justifications. Two raters independently coded all justifications, achieving an overall inter-rater agreement of 95%. The disagreements were resolved through discussion.

Results

Factor Analysis and Intercorrelations of the ISEJ Dimensions

Because the ISEJ items were translated to German and formulated to address beliefs about the domain of natural science, and because the sample size was quite small, we considered it most appropriate to investigate the dimensionality of the ISEJ scores through exploratory factor analysis. A maximum likelihood exploratory factor analysis with oblique rotation on the 12 items indicated three factors with eigenvalues greater than 1 that explained 80.52% of the total variance. Justification by authority, personal justification, and justification by multiple sources had eigenvalues of 6.05, 2.47, and 1.15, respectively. All 12 items loaded on their designated factor and all factors had high loadings (>.70) and low overlap (<.22) for any item. Thus, mean scores for the four scale items of each ISEJ dimension were calculated and used in subsequent statistical analyses. The three ISEJ dimensions were positively correlated (see Table 1), with justification by multiple sources being positively correlated with both justification by authority (r = .61, p < .001) and personal justification (r = .46, p = .001). The correlation between justification by authority and personal justification was .24, which was not statistically significant with this sample size (p = .095).

Table 1.

Summary of Intercorrelations and Descriptive Statistics for All Reported Variables.

Variable	1	2	3	4	5	6	7	8	9	10	11	12	13
1. Prior domain knowledge	–	−.03	−.10	.30*	−.08	.30*	.07	.10	.41**	.27†	.38**	.00	.37**
2. ISEJ Multiple sources		–	.61***	.46**	.00	.18	.29*	.23	.08	.12	.04	−.18	.22
3. ISEJ Authority			–	.24†	.15	.42**	.18	.04	.06	.24†	.11	−.35*	.35*
4. ISEJ Personal				–	−.28*	.13	.01	−.09	.29*	.11	.19	−.00	.32*
5. No. of corroboration comments					–	.35*	−.10	−.18	−.14	.08	−.17	−.23	−.05
6. No. of source evaluation comments						–	−.13	−.24†	.24†	.10	.15	−.18	.18
7. No. of words in written justification							–	.53***	−.02	−.03	−.03	.06	.12
8. No. of relevant aspects included in written justification								–	−.21	−.05	−.16	−.05	−.02
9. No. of unique queries									–	.54**	.63***	.06	.41**
10. No. of unique websites visited										–	.42**	−.45**	.29*
11. No. of websites revisited											–	.03	.65***
12. Percentage of search results from SERP ranks 1–3 (in %)												–	−.04
13. Opening of multiple tabs (yes/no)													–
M (SD)	3.82 (1.95)	7.38 (2.01)	7.12 (2.42)	7.60 (1.77)	1.90 (2.10)	2.66 (2.68)	60.68 (39.82)	1.60 (1.07)	4.68 (3.18)	7.70 (3.13)	2.40 (2.33)	57.37% (24.08)	52% (yes)
Skewness	0.21	−0.99	−1.12	−0.86	0.85	1.03	2.71	1.61	1.21	0.63	1.29	−0.08
Kurtosis	−0.76	0.92	0.43	0.94	−0.62	0.08	12.10	3.18	2.10	0.92	2.31	−0.24

Note. ***p < .001, **p < .01 *p < .05, †p < .10, two−tailed.

Descriptives for Query and Navigation Behaviors

Analyses of the query and navigation log files revealed that participants, on average, submitted 4.68 (SD = 3.18) different queries and visited 7.70 (SD = 3.13) different websites (see Table 1). Typical queries were, for instance, “nanoparticles in sunscreen,” “nanoparticles sunscreen health risks,” or “nanoparticles sunscreen advantages and disadvantages.” Concerning the visited websites, similar to results reported by Knight et al. (2017), participants predominantly visited an alternative-medicine website, the website of a sunscreen producer, online newspapers, consumer magazines, and consumer service sites.

Table 2 gives an overview of the most visited websites, that is, those that were visited by at least 5 participants (10%). Please note that the alternative medicine website and the website of the sunscreen producer, which were visited by 84% and 62% of the participants, respectively, were often the first two search results presented by Google (see also Figure 1 for an example screenshot of a Google SERP for the query “nanoparticles in sunscreen”).

Table 2.

Overview of the Most Visited Websites (By at Least 10% of the Participants).

URL	No. ofparticipants visiting	Type of website
www.zentrum-der-gesundheit.de	42	Alternative-medicine site (with shop)
www.beyer-soehne.de	31	Sunscreen producer
www.welt.de	29	Newspaper
www.sueddeutsche.de	21	Newspaper
www.utopia.de	20	Consumer magazine
www.oekotest.de	19	Consumer magazine
www.de.wikipedia.org	15	Encyclopedia
www.nanopartikel.info	12	Scientific site
www.femininundfit.de	10	Consumer magazine
www.nanotruck.de	8	Governmental site
www.pharmazeutische-zeitung.de	8	Medical magazine
www.test.de	7	Consumer service site
www.sonnencreme-test.eu	6	Consumer service site
www.bfr.bund.de	6	Governmental site
www.verbraucherservice-bayern.de	6	Consumer service site
www.zeit.de	6	Newspaper
www.ncbi.nlm.nih.gov	5	Scientific database
www.augsburger-allgemeine.de	5	Newspaper
www.spiegel.de	5	Newspaper

Figure 1.

Example Screenshot of a SERP for the Query “Nanoparticles in Sunscreen” [Black & White Version in Print].

All participants used Google’s general search engine for their searches and four participants (8%) in addition used Google scholar. On average, 57.37% (SD = 24.08) of the search results were selected from SERP ranks 1–3, and 10% of the participants did not at all go beyond the first three search results on a SERP during the entire search session. 52% of the participants opened multiple (i.e., at least two) tabs in their browser, whereas 48% used only one browser tab (i.e., did not open multiple websites in parallel) during their Web search.

Descriptives for Source Evaluation and Corroboration Comments

Analyses of the think-aloud protocols showed that participants, on average, produced 2.66 (SD = 2.68) comments indicating source evaluation and 1.90 (SD = 2.10) comments indicating corroboration (see Table 1). Almost all corroboration comments were expressed on websites (M = 1.84, SD = 2.10), with only three participants expressing one corroboration comment each while using Google. Source evaluation comments were expressed both while using Google (M = 1.00, SD = 1.43) and while reading websites (M = 1.66, SD = 1.79).

Descriptives for the Written Justifications

The majority (64%) of the participants recommended their fictitious friend not to use sunscreen containing nanoparticles. The average length of participants’ written justifications was 60.68 words (SD = 39.82), without any statistically significant difference between those who recommened the use of sunscreen containing nanoparticles (M = 52.17, SD = 28.73) and those who did not (M = 65.47, SD = 44.59), t(48) = −1.14, p = .261.

A qualitative analysis of the written justifications indicated that all participants who recommended not to use sunscreen containing nanoparticles justified this recommendation with the fact that risks are high or uncertain. Ten out of these 32 participants mentioned that risks are particularly high when the skin is damaged (i.e., by cuts or wounds) and two participants mentioned that sun sprays might be particularly harmful because nanoparticles could be inhaled. Further, eight participants mentioned that the advantage of sunscreen containing nanoparticles would only be cosmetic (because it does not leave a white film on the skin), and four participants mentioned that mineral sunscreen without nanoparticles could be used instead.¹ Two participants acknowledged that a coating procedure could reduce the potential risks associated with nanoparticles.

Of the 18 participants who recommended the use of sunscreen containing nanoparticles, all but three justified their recommendation with the fact that risks are low or that no risks are known, and/or that the advantages predominate. The three remaining participants had no clear argumentation. Four of the 15 participants who found risks to be low or undocumented or advantages to predominate acknowledged that sun sprays might be harmful because nanoparticles could be inhaled, one of them acknowledged that there might be a risk when the skin is damaged (i.e., by cuts or wounds), and another participant acknowledged that potential side effects have not been clarified yet.

Overall, participants included an average of 1.60 (SD = 1.07) relevant aspects in their written justification. Further, only seven participants (four giving a negative recommendation and three giving a positive recommendation) referred to sources, specifically to “the pharmaceutical newspaper,” “a newspaper article,” “a reliable study,” “one study,” “independent research of the European Union,” “a producer of sunscreen,” and “an environmental organization.”

Regression Analyses

Because our dependent variables were count data, we performed quasi-Poisson regression analyses with these variables, using the ISEJ dimensions (i.e., the mean scores for the four scale items) of justification by authority, justification by multiple sources, and personal justification as predictors in each analysis. In addition, prior domain knowledge was included as a predictor in each analysis. Zero-order correlations between all variables included in the regression analyses are displayed in Table 1.

Source Evaluation and Corroboration Comments

Table 3 shows the results of the quasi-Poisson regression analyses with number of source evaluation and corroboration comments, respectively, as the dependent variables.

Table 3.

Results of Regression Analyses for Source Evaluation and Corroboration Comments.

Dependent variable	Predictor	B	SE B	t	p
No. of source evaluation comments	ISEJ Multiple sources	−0.17	0.10	−1.64	.109
	ISEJ Authority	0.40	0.11	3.76	<.001***
	ISEJ Personal	−0.06	0.08	−0.69	.493
	Prior knowledge	0.20	0.07	3.05	.004**
No. of corroboration comments	ISEJ Multiple sources	−0.03	0.12	−0.22	.828
	ISEJ Authority	0.14	0.11	1.29	.204
	ISEJ Personal	−0.21	0.10	−2.17	.035*
	Prior knowledge	0.04	0.09	0.44	.666

Note. ***p < .001, **p < .01 *p < .05, two-tailed.

For source evaluation comments, the overall regression model was statistically significant, χ²(4, 45) = 50.44, p < .001. Both justification by authority and prior domain knowledge were statistically significant positive predictors. That is, the more participants believed that information about natural science topics encountered on the Internet needs to be justified by referring to the competency or expertise of the source, the more source evaluation they displayed during their Web search. Additional exploratory analyses showed that this was true for source evaluation comments expressed both while scrutinizing search results on SERPs, B = 0.49, SE = 0.17, t(45) = 2.98, p = .005, and while processing websites, B = 0.35, SE = 0.12, t(45) = 2.92, p = .005. Likewise, participants with higher domain knowledge displayed more source evaluation during their Web search. Additional exploratory analyses indicated that this was particularly the case for comments expressed on SERPs, B = 0.29, SE = 0.10, t(45) = 2.95, p = .005, but not significantly so for source evaluation comments expressed on websites, B = 0.15, SE = 0.08, t(45) = 1.93, p = .060.

For corroboration comments, the overall regression model was also significant, χ²(4, 45) = 15.12, p = .004. Personal justification was a unique, statistically significant negative predictor. This indicated that participants who believed in evaluating information about natural science topics on the Internet by means of their own prior knowledge and reasoning, reflected less on the corroboration of information across websites.

Query and Navigation Behaviors

Table 4 shows the results of the regression analyses for the query and navigation variables (i.e., quasi-Poisson regression for count data, linear regression for percentage score, and logistic regression for dichotomous data).

Table 4.

Results of Regression Analyses for Query and Navigation Behaviors.

Dependent variable	Predictor	B	SE B	t	p
No. of unique queries	ISEJ Multiple sources	−0.01	0.06	−0.15	.882
	ISEJ Authority	0.01	0.05	0.33	.744
	ISEJ Personal	0.08	0.06	1.26	.213
	Prior knowledge	0.12	0.05	2.55	.014*
No. of unique websites visited	ISEJ Multiple sources	−0.01	0.04	−0.22	.827
	ISEJ Authority	0.05	0.03	1.70	.097†
	ISEJ Personal	−0.01	0.05	−0.30	.770
	Prior knowledge	0.06	0.03	2.11	.041*
No. of revisited websites	ISEJ Multiple sources	−0.05	0.10	−0.05	.631
	ISEJ Authority	0.07	0.08	0.93	.358
	ISEJ Personal	0.05	0.09	0.49	.626
	Prior knowledge	0.18	0.07	2.44	.019*
Percentage of search results from SERP ranks 1–3 (in %)	ISEJ Multiple sources	0.05	2.31	0.02	.984
	ISEJ Authority	−3.85	1.75	−2.20	.033*
	ISEJ Personal	1.47	2.27	0.65	.521
	Prior knowledge	−0.83	1.85	−0.45	.657
Opening of multiple tabs (yes/no)	ISEJ Multiple sources	−0.10	0.26	Waldsχ² = 0.15	.702
	ISEJ Authority	0.48	0.23	Waldsχ² = 4.22	.040*
	ISEJ Personal	0.24	0.25	Waldsχ² = 0.96	.328
	Prior knowledge	0.50	0.21	Waldsχ² = 5.79	.016*

Note. *p < .05, ^†p < .10, two-tailed.

With the number of unique search queries as the dependent variable, the overall regression model was statistically significant, χ²(4, 45) = 21.41, p < .001, with prior domain knowledge being the only statistically significant predictor. That is, the higher participants’ domain knowledge, the more queries they entered during the course of their Web search.

When including the number of visited websites as the dependent variable, the overall regression model did not quite reach statistical significance, χ²(4, 45) = 9.27, p = .055. Still, prior domain knowledge was a statistically significant positive predictor. That is, participants with higher domain knowledge visited more websites during the course of their Web search. In addition, beliefs concerning justification by authority were also positively related to the number of websites that participants visited during their Web search, although not statistically significantly so (p = .097).

With the number of revisited websites as the dependent variable, the overall regression model was statistically significant, χ²(4, 45) = 18.28, p = .001. Prior domain knowledge was the only statistically significant (positive) predictor.

When including the percentage of search results selected from SERP ranks 1–3 as the dependent variable, the overall regression model was not statistically significant, R² = .13, F(4, 45) = 1.74, p = .157. Still, justification by authority was a statistically significant negative predictor. That is, participants who believed that information about natural science topics encountered on the Internet needs to be justified by referring to the competency or expertise of the source, selected a higher percentage of websites further down the SERPs that were returned from their searches, possibly because they were looking for more authoritative sources.

Finally, when including the opening of multiple browser tabs as dependent variable, the overall regression model was significant, Nagelkerke’s R² = .39, χ² (4, 45) = 17.38, p = .002. Both justification by authority and prior domain knowledge were statistically significant positive predictors. That is, participants who believed that information about natural science topics encountered on the Internet needs to be justified by referring to the competency or expertise of the source, were more likely to open multiple tabs simultaneously in the browser during their Web search. Likewise, participants with higher domain knowledge were more likely to open several browser tabs.

Written Justifications

The results of the two quasi-Poisson regression analyses involving individuals’ written justifications are displayed in Table 5. With regard to the extent (i.e., length, in number of words) of participants’ written justifications, the overall regression model was statistically significant, χ²(4, 45) = 157.45, p < .001. Beliefs in justification by multiple sources were a unique, statistically significant positive predictor, suggesting that participants who realized the importance of justifying knowledge claims by considering multiple information sources, also were more likely to be engaged in constructing a comprehensive task product.

Table 5.

Results of Regression Analyses for Length of and Relevant Aspects in Written Justification.

Dependent variable	Predictor	B	SE B	t	p
No. of words in written justification	ISEJ Multiple sources	0.13	0.06	2.12	.040*
	ISEJ Authority	0.01	0.04	0.15	.881
	ISEJ Personal	−0.08	0.06	−1.36	.182
	Prior knowledge	0.05	0.05	1.04	.302
No. of relevant aspects included in written justification	ISEJ Multiple sources	0.16	0.06	2.63	.012*
	ISEJ Authority	−0.04	0.04	−0.91	.369
	ISEJ Personal	−0.13	0.06	−2.14	.038*
	Prior knowledge	0.07	0.05	1.44	.156

Note. *p < .05, two-tailed.

Furthermore, with regard to the number of relevant aspects included in the written justifications, the overall regression model was not significant, χ²(4, 45) = 5.47, p = .243. Still, as for the length of the written justifications, justification by multiple sources was a statistically significant positive predictor. Moreover, personal justification was a statistically significant negative predictor. This indicated that participants who believed in evaluating information about natural science topics on the Internet by means of their own prior knowledge and reasoning included fewer relevant aspects in their written justification.

Discussion

In the age of digitization, the Internet provides easy access to vast amounts of information about all kinds of topics. Therefore, individuals increasingly turn to the Internet to solve information problems, such as to make decisions about unfamiliar and unsettled socio-scientific issues. However, given the relaxed parameters for publishing and the lack of editorial control on the Internet, users themselves need to critically justify claims about socio-scientific issues they encounter online before they can accept them as valid knowledge. Yet, individuals differ both in how important or necessary they consider it to justify science-related knowledge claims encountered on the Internet and in how they think such claims should be justified, such as by drawing on their own prior knowledge and reasoning (i.e., personal justification), by judging the authoritativeness of the source (i.e., justification by authority), or by cross-checking and comparing multiple information sources for consistency (i.e., justification by multiple sources; Bråten, Brandmo, et al., 2019). In the present study, we assessed beliefs concerning these three forms of justification in a sample of German university students by means of the Internet-Specific Epistemic Justification Inventory (ISEJ; Bråten, Brandmo, et al., 2019), with items specified to address the domain of natural science. Results of the exploratory factor analysis supported the three-factor structure of justification by authority, personal justification, and justification by multiple sources in the present sample. Notably, this is one of the very first studies that have used the ISEJ to examine Internet-specific justification beliefs about the domain of natural science.

Moreover, this study is the ﬁrst to investigate whether and how individuals’ beliefs concerning these three forms of Internet-specific justification predict how they actually search for and evaluate information on their Internet to learn about an unsettled and unfamiliar socio-scientific issue, as well as their written justifications that they produce for their recommendations after the search.

The Role of Justification by Multiple Sources

Our assumptions regarding justiﬁcation by multiple sources were only partly confirmed in the present study. As expected, this dimension positively predicted the extent and relevance of individuals’ post-search written justifications for their recommendation (in terms of the length of the written justification and the number of relevant aspects mentioned, respectively). This ﬁnding is consistent with both the theoretical notion that beliefs in justification by multiple sources play a role in constructing a broader, more extensive representation of an issue discussed in various documents (Bråten et al., 2011) and prior research on multiple document comprehension (e.g., Bråten et al., 2013; Bråten, Ferguson, et al., 2014; Ferguson & Bråten, 2013; Kammerer et al., 2013). However, unlike prior research that used a controlled and limited set of preselected documents, the present study explored a more naturalistic Web search context offering free access to a large variety of information sources. This approach enabled us to analyze individuals’ naturalistic query and navigation behaviors in addition to their concurrent think-alouds. Beliefs in justification by multiple sources did not predict the extent of individuals’ query and navigation activities, that is, the number of websites visited or revisited, the number of queries formulated, or the opening of multiple browser tabs; nor did they predict individuals’ think-aloud comments concerning source evaluation and corroboration. These findings were not consistent with our expectations.

While this lack of predictability is in line with findings by Knight et al. (2017) and List and Alexander (2018), who assessed relationships between behavioral metrics obtained during Web search (e.g., number of unique websites visited or revisited and the number of unique queries formulated) and individuals’ Internet-specific justification beliefs assessed with the ISEQ (Bråten et al., 2005) or self-reported corroboration behavior, respectively, it contradicts findings based on self-report and reading time data by Bråten and colleagues (e.g., Bråten, Anmarkrud, et al., 2014; Bråten, Brandmo, et al., 2019). Specifically, the latter studies showed that justification by multiple sources positively predicted the reported or actual time, effort, and persistence put into Web-based learning or multiple document reading. It is conceivable that the fixed time limit of 20 minutes for the Web search might have suppressed interindividual differences concerning the time, effort, and persistence put into the Web search in the present study. Further, it should be noted that we did not consider the quality, trustworthiness, or usefulness of the websites accessed by participants in our analyses. Future work could analyze whether beliefs in justification by multiple sources might predict the number or percentage of trustworthy websites visited or the absolute or relative time spent on such websites. Of course, this would require a classification scheme suitable for defining the trustworthiness of any website accessed by participants.

There is yet another possible interpretation of the finding that beliefs in justication by multiple sources were not related to any of the variables indicating corroboration or cross-checking during Web search. This is that justification by multiple sources might refer less to individuals’ conscious and overt corroboration processes displayed during Web search (at least when searching for unfamiliar issues) and more to their covert, mental consideration and comparison of information from various sources in trying to arrive at an informed decision. Admittedly, this interpretation is speculative at this point, and much further research is needed to clarify both how individuals interpret the questionnaire items concerning justification by multiple sources and how their responses relate to processes or outcomes of corroboration.

The Role of Justification by Authority

As expected and consistent with existing empirical evidence (e.g., Bråten et al., 2013), beliefs in justification by authority were not related to the extent or relevance of individuals’ written justifications. Further, our assumption that beliefs in justification by authority would positively predict comments regarding source evaluation was confirmed. That is, participants who believed that information about natural science topics encountered on the Internet needs to be justified by referring to the competency or expertise of the source, reflected more on source parameters during their Web search. This was the case both while they inspected search results on SERPs and while they processed websites.

Beliefs in justification by authority were also found to positively predict the percentage of visited websites that were listed beyond the first three Google search results. While we had not expected this relationship, it should be noted that increased consideration of websites presented further down on a SERP previously has been interpreted as an indicator of source evaluation (e.g., Kammerer et al., 2013; Kammerer & Gerjets, 2014). Likewise, as reported by Wineburg and McGrew (2019), professional fact checkers typically exhibit “click restraint”, that is, instead of simply trusting the ranking order provided by the search engine, they carefully evaluate multiple search results, also those located further down on a SERP, and then deliberately select the ones that seem most reliable. Accordingly, beliefs in justification of Internet-based knowledge about natural science issues by considering the competency and expertise of the source do not seem to reflect a naïve reliance on external authority, but rather an adaptive view that knowledge claims need to be critically evaluated in the light of their source (i.e., the author or website provider).

In addition, justification by authority predicted the likelihood that individuals opened multiple browser tabs during their Web search, which we had not expected. This is because we considered it more likely that the strategy of opening multiple tabs simultaneously would be more strongly linked to justification by multiple sources. However, similar to selecting search results further down on a SERP, opening websites in multiple browser tabs is a strategy exhibited by professional fact checkers (e.g., Wineburg & McGrew, 2019). The present findings are therefore consistent with the view that university students who believe that Internet-based knowledge claims about natural sience topics need to be justified on the basis of the competency or expertise of the source have a “fact-checking mindset.” This may partly be due to a lower trust in their own competency (Bråten, Brandmo, et al., 2019), which seems particularly reasonable in the context of an unfamiliar topic, such as potential health risks of nanoparticles in sunscreen. Support for this interpretation was recently provided by Bråten, McCrudden, et al. (2018).

In sum, the results of our study suggest that endorsing the view that Internet-based knowledge claims about natural sience topics need to be justified on the basis of the competency or expertise of the source may facilitate source evaluation and cross-checking of multipe documents when searching the Web to learn about unfamiliar socio-scientific issues. Moreover, these results speak for the validity of the ISEJ dimension addressing justification by authority. In contrast, other self-report measures addressing source evaluation have been shown to be unrelated to university students’ actual source evaluation behavior displayed during Web search (e.g., List & Alexander, 2018).

The Role of Personal Justification

Our assumptions regarding beliefs in personal justiﬁcation were partly supported. First, beliefs that Internet-based knowledge claims about natural science topics should be justified based on individuals’ own prior knowledge and reasoning were positively correlated with their prior knowledge regarding nanotechnology, as measured with a knowledge test. Thus, students with stronger beliefs in personal justification might also be somewhat more capable of justifying knowledge claims within the same domain (i.e., natural science) encountered on the Internet. Of note is, however, that the specific topic of potential health risks of nanoparticles in sunscreen still was quite unfamiliar to them, which could make it difficult to judge the validity of topic-specific knowledge claims based on their own knowledge and reasoning. As expected, beliefs in personal justification also negatively predicted corroboration of information across websites, as indicated by participants’ think-aloud comments. This finding is consistent with previous findings by Barzilai et al. (2015). Furthermore, personal justification negatively predicted the number of relevant aspects mentioned in participants’ written justifications. Other than expected, however, participants’ beliefs in personal justification were unrelated to the length of their written justifications, their think-aloud comments concerning source evaluation, and their query and navigation activities. These inconsistent findings suggest that personal justification in terms of own knowledge and reasoning might be less detrimental to the processing and products of multiple document comprehension than is personal justification in terms of own opinion, which has been the main focus of prior research on personal justification beliefs (Bråten & Strømsø, 2020; Bråten et al., 2016).

The Role of Prior Domain Knowledge

Finally, we used prior domain knowledge as an additional predictor variable in all analyses. In accordance with theoretical assumptions and prior research (Monchaux et al., 2015; Sanchiz et al., 2017; White et al., 2009; Zhang et al., 2015), prior domain knowledge was a unique positive predictor of the number of different websites visited and revisited during Web search and the number of unique queries submitted to the search engine. Likewise, participants with higher domain knowledge made more comments indicating source evaluation (cf. Brand-Gruwel et al., 2017; Rouet et al., 1997; von der Mühlen et al., 2016; Wineburg, 1991), particularly while inspecting SERPs, and they were more likely to open multiple tabs simultaneously in the browser. However, prior domain knowledge was unrelated to the proportion of websites selected from the first three Google search results or lower on the SERPs, as well as to the number of comments indicating corroboration and the extent and relevance of their written justifications. These somewhat inconsistent findings regarding prior knowledge may suggest that participants’ prior knowledge about the topic, on average, was too low to be generally facilitative in the context of the Web search task. Furthermore, it should be noted that the internal consistency of the prior knowledge measure (the PKNT test; cf. Gottschling et al., 2019; Lin et al., 2013) was rather low. This might be due to the fact that nanotechnology is a broad knowledge domain about which participants might have some basic knowledge concerning more general aspects but cannot correctly answer other more specialized questions. Still, we believe that the PKNT test is an adequate measure to assess laypersons’ prior knowledge about nanotechnology.

Limitations and Future Directions

As a limitation of the present study, we acknowledge that the results are based on a rather small and homogeneous sample of university students, who were asked to conduct a Web search on one predefined topic in a laboratory setting. Moreover, the data we collected are only correlational. Thus, in order to draw conclusions about causality, further experimental work is required. Future research, for instance, could examine the effects of instructional interventions that aim at strengthening Internet-specific beliefs in justification by multiple sources and justification by authority on individuals’ source evaluation and corroboration behaviors during Web search as well as on the learning outcomes associated with such behaviors. Recently, several intervention studies showed that interventions targeting learners’ source evaluation or confronting learners with conflicts across documents can strengthen (a) beliefs in justification by multiple sources (Ferguson et al., 2013; Kammerer et al., 2015) or (b) actual source evaluation and corroboration behaviors or comprehension outcomes (e.g., Bråten, Brante, et al., 2019; McGrew, 2020; for recent reviews, see Brante & Strømsø, 2018; Bråten, Stadtler, et al., 2018; Kammerer & Brand-Gruwel, 2020). However, future studies should examine both epistemic beliefs and behaviors in conjunction as well as their potentially mediational relationship.

To the best of our knowledge, the present study represents the first attempt to jointly explore self-reported Internet-specific justification beliefs and actual process and outcome measures obtained from individuals’ naturalistic Web searches. In so doing, we took a mainly quantitative approach, considering, for instance, the number of comments, submitted queries, and visited websites, as well as the number of words and relevant aspects included in the post-search written justifications. Future research should also consider qualitative aspects of such endeavours, for instance, concerning the appropriateness of participants’ source evaluation comments as well as the trustworthiness and accuracy of the visited websites and the relevance of individuals’ search queries. However, given that in the context of self-directed Web searches on the open Internet, each participant “[…] creates a unique reading path in an unlimited information space” (Kiili et al., 2018, p. 25), this would be a quite complex and extensive task. Furthermore, with regard to the analysis of the think-aloud data, future research could also examine comments that reflect efforts to justify knowledge claims based on individuals’ own knowledge or beliefs. However, this might be more informative when using a more familiar Web search topic than the one presented in this study. Finally, it should be noted that, as in previous research (e.g., Bråten, Ferguson, et al., 2014), students varied considerably in the overall amount of verbal comments they produced during the Web search (ranging from 10 to 90 comments). This might indicate that some students overall were more skilled, more motivated, or more self-confident in verbalizing their thoughts during the Web search than were other students.

Despite these limitations, we believe that the present research provides a unique contribution to our understanding of the relationships between Internet-speciﬁc justification beliefs and individuals’ processes and products when searching for an unsettled and unfamiliar socio-scientific issue on the Internet.

Supplemental Material

sj-pdf-1-jec-10.1177_0735633120952731 - Supplemental material for The Role of Internet-Specific Justification Beliefs in Source Evaluation and Corroboration During Web Search on an Unsettled Socio-Scientific Issue

Supplemental material, sj-pdf-1-jec-10.1177_0735633120952731 for The Role of Internet-Specific Justification Beliefs in Source Evaluation and Corroboration During Web Search on an Unsettled Socio-Scientific Issue by Yvonne Kammerer, Steffen Gottschling and Ivar Bråten in Journal of Educational Computing Research

Footnotes

Appendix: Items of the Internet-Specific Epistemic Justification Inventory (ISEJ)

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research reported in this paper was partly funded by the Leibniz Competition Fund (SAW-2016-IWM) supporting Yvonne Kammerer and the Leibniz Research Alliance “Nanosafety” supporting Steffen Gottschling.

ORCID iDs

Yvonne Kammerer

Ivar Bråten

Supplemental Material

Supplemental material for this article is available online.

Note

Author Biographies

Yvonne Kammerer is a senior research scientist at the Leibniz-Institut für Wissensmedien, Tübingen, Germany, and assistant professor at the Open University of the Netherlands, Heerlen, NL.

Steffen Gottschling is a junior research scientist at the Leibniz-Institut für Wissensmedien, Tübingen, Germany.

Ivar Bråten is full professor of educational psychology at the Faculty of Educational Sciences at the University of Oslo, Norway.

References

Barzilai

Tzadok

Eshet-Alkalai

(2015). Sourcing while reading divergent expert accounts: Pathways from views of knowing to written argumentation. Instructional Science, 43(6), 737–766. https://doi.org/10.1007/s11251-015-9359-4

Braasch

J. L. G.

Bråten

(2017). The discrepancy-induced source comprehension (D-ISC) model: Basic assumptions and preliminary evidence. Educational Psychologist, 52(3), 167–181. https://doi.org/10.1080/00461520.2017.1323219

Brand-Gruwel

Kammerer

van Meeuwen

van Gog

(2017). Source evaluation of domain experts and novices during Web search. Journal of Computer Assisted Learning, 33(3), 234–251. https://doi.org/10.1111/jcal.12162

Brand-Gruwel

Wopereis

Walraven

(2009). A descriptive model of information problem solving while using internet. Computers & Education, 53(4), 1207–1217. https://doi.org/10.1016/j.compedu.2009.06.004

Brante

E. W.

Strømsø

H. I.

(2018). Sourcing in text comprehension: A review of interventions targeting sourcing skills. Educational Psychology Review, 30, 773–799. https://doi.org/10.1007/s10648-017-9421-7

Bråten

(2011). Personal epistemology in education. In Aukrust

V. G.

(Ed.), Learning and cognition in education (pp. 52–58). Elsevier.

Bråten

Anmarkrud

Ø.

Brandmo

Strømsø

H. I.

(2014). Developing and testing a model of direct and indirect relationships between individual differences, processing, and multiple-text comprehension. Learning and Instruction, 30, 9–24. https://doi.org/10.1016/j.learninstruc.2013.11.002

Bråten

Brandmo

Kammerer

(2019). A validation study of the Internet-Specific Epistemic Justification Inventory with Norwegian pre-service teachers. Journal of Educational Computing Research, 57(4), 877–900. https://doi.org/10.1177/0735633118769438

Bråten

Brante

E. W.

Strømsø

H. I.

(2018). What really matters: The role of behavioral engagement in multiple document literacy tasks. Journal of Research in Reading, 41(4), 680–699. https://doi.org/10.1111/1467-9817.12247

10.

Bråten

Brante

E. W.

Strømsø

H. I.

(2019). Teaching sourcing in upper secondary school: A comprehensive sourcing intervention with follow-up data. Reading Research Quarterly, 54(4), 481–505. https://doi.org/10.1002/rrq.253.

11.

Bråten

Britt

M. A.

Strømsø

H. I.

Rouet

J.-F.

(2011). The role of epistemic beliefs in the comprehension of multiple expository texts: Toward an integrated model. Educational Psychologist, 46(1), 48–70. https://doi.org/10.1080/00461520.2011.538647

12.

Bråten

Ferguson

L. E.

(2014). Investigating cognitive capacity, personality, and epistemic beliefs in relation to science achievement. Learning and Individual Differences, 36, 124–130. https://doi.org/10.1016/j.lindif.2014.10.003

13.

Bråten

Ferguson

L. E.

Strømsø

H. I.

Anmarkrud

Ø.

(2013). Justification beliefs and multiple-documents comprehension. European Journal of Psychology of Education, 28(3), 879–902. https://doi.org/10.1007/s10212-012-0145-2

14.

Bråten

Ferguson

L. E.

Strømsø

H. I.

Anmarkrud

Ø.

(2014). Students working with multiple conflicting documents on a scientific issue: Relations between epistemic cognition while reading and sourcing and argumentation in essays. British Journal of Educational Psychology, 84(1), 58–85. https://doi.org/10.1111/bjep.12005

15.

Bråten

Gil

Strømsø

H. I.

Vidal-Abarca

(2009). Personal epistemology across cultures: Exploring Norwegian and Spanish university students’ epistemic beliefs about climate change. Social Psychology of Education, 12, 529–560. https://doi.org/10.1007/s11218-009-9097-z

16.

Bråten

McCrudden

M. T.

Stang Lund

Brante

E. W.

Strømsø

H. I.

(2018). Task-oriented learning with multiple documents: Effects of topic familiarity, author expertise, and content relevance on document selection, processing, and use. Reading Research Quarterly, 53(3), 345–365. https://doi.org/10.1002/rrq.197

17.

Bråten

Stadtler

Salmerón

(2018). The role of sourcing in discourse comprehension. In Schober

M. F.

Rapp

D. N.

Britt

M. A.

(Eds.), Handbook of discourse processes (2nd ed., pp. 141–166). Routledge.

18.

Bråten

Strømsø

H. I.

(2010). Effects of task instruction and personal epistemology on the understanding of multiple texts about climate change. Discourse Processes, 47(1), 1–31. https://doi.org/10.1080/01638530902959646

19.

Bråten

Strømsø

H. I.

(2020). On the roles of dispositions and beliefs in learning from multiple perspectives. In Van Meter

List

Lombardi

Kendeou

(Eds.), Handbook of learning from multiple representations and perspectives (pp. 141–163). Routledge.

20.

Bråten

Strømsø

H. I.

Ferguson

L. E.

(2016). The role of epistemic beliefs in the comprehension of single and multiple texts. In Afflerbach

(Ed.), Handbook of individual differences in reading: Text and context (pp. 67–79). Routledge.

21.

Bråten

Strømsø

H. I.

Samuelstuen

M. S.

(2005). The relationship between Internet-specific epistemological beliefs and learning within Internet technologies. Journal of Educational Computing Research, 33(2), 141–171. https://doi.org/10.2190/E763-X0LN-6NMF-CB86

22.

Bråten

Strømsø

H. I.

Samuelstuen

M. S.

(2008). Are sophisticated students always better? The role of topic-specific personal epistemology in the understanding of multiple expository texts. Contemporary Educational Psychology, 33(4), 814–840. https://doi.org/10.1016/j.cedpsych.2008.02.001

23.

Bromme

Goldman

S. R.

(2014). The public’s bounded understanding of science. Educational Psychologist, 49(2), 59–69. https://doi.org/10.1080/00461520.2014.921572

24.

Bromme

Kienhues

Porsch

(2010). Who knows what and who can we believe? Epistemological beliefs are beliefs about knowledge (mostly) to be attained from others. In Bendixen

L. D.

Feucht

F. C.

(Eds.), Personal epistemology in the classroom: Theory, research, and implications for practice (pp. 163–193). Cambridge University Press.

25.

Brossard

(2013). New media landscapes and the science information consumer. Proceedings of the National Academy of Sciences, 110(3), 14096–14101. https://doi.org/10.1073/pnas.1212744110

26.

Cheng

C.-H.

Bråten

Yang

F.-Y.

Brandmo

(2020). Investigating structural relationships among upper-secondary students’ beliefs about knowledge, justification for knowing, and Internet-specific justification in the domain of science. Graduate Institute of Science Education, National Taiwan Normal University, Taiwan/University of Oslo, Norway. [Manuscript submitted for publication].

27.

Chi

M. T.

Bassok

Lewis

M. W.

Reimann

Glaser

(1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13(2), 145–182. https://doi.org/10.1016/0364-0213(89)90002-5

28.

Chiu

Y.-L.

Liang

J.-C.

Tsai

C.-C.

(2013). Internet-specific beliefs and self-regulated learning in online academic information searching. Metacognition and Learning, 8(3), 235–260. https://doi.org/10.1007/s11409-013-9103-x

29.

De La Paz

Monte-Sano

Felton

Croninger

Jackson

Worland Piantedosi

(2017). A historical writing apprenticeship for adolescents: Integrating disciplinary learning with cognitive strategies. Reading Research Quarterly, 52(1), 31–52. https://doi.org/10.1002/rrq.147

30.

Ericsson

K. A.

Simon

H. A.

(1993). Protocol analysis: Verbal reports as data (Rev. ed.). The MIT Press.

31.

Ferguson

L. E.

Bråten

(2013). Student profiles of knowledge and epistemic beliefs: Changes and relations to multiple-text comprehension. Learning and Instruction, 25, 49–61. https://doi.org/10.1016/j.learninstruc.2012.11.003

32.

Ferguson

L. E.

Bråten

Strømsø

H. I.

(2012). Epistemic cognition when students read multiple documents containing conflicting scientific evidence: A think-aloud study. Learning and Instruction, 22(2), 103–120. https://doi.org/10.1016/j.learninstruc.2011.08.002

33.

Ferguson

L. E.

Bråten

Strømsø

H. I.

Anmarkrud

Ø.

(2013). Epistemic beliefs and comprehension in the context of reading multiple documents: Examining the role of conflict. International Journal of Educational Research, 62(6), 100–114. https://doi.org/10.1016/j.ijer.2013.07.001

34.

Flanagin

A. J.

Metzger

M. J.

(2007). The role of site features, user attributes, and information verification behaviors on the perceived credibility of web-based information. New Media & Society, 9(2), 319–342. https://doi.org/10.1177/1461444807075015

35.

Gil

Bråten

Vidal-Abarca

Strømsø

H. I.

(2010). Summary versus argument tasks when working with multiple documents: Which is better for whom? Contemporary Educational Psychology, 35(3), 157–173. https://doi.org/10.1016/j.cedpsych.2009.11.002

36.

Gottschling

Kammerer

Gerjets

(2019). Readers’ processing of source information as a function of its usefulness to explain conflicting scientific claims. Discourse Processes, 56(5–6), 429–446. https://doi.org/10.1080/0163853X.2019.1610305

37.

Greene

J. A.

Azevedo

Torney-Purta

(2008). Modeling epistemic and ontological cognition: Philosophical perspectives and methodological directions. Educational Psychologist, 43(3), 142–160. https://doi.org/10.1080/00461520802178458

38.

Greene

J. A.

Copeland

D. Z.

Deekens

V. M.

S. B.

(2018). Beyond knowledge: Examining digital literacy’s role in the acquisition of understanding in science. Computers & Education, 117, 141–159. https://doi.org/10.1016/j.compedu.2017.10.003

39.

Greene

J. A.

Torney-Purta

Azevedo

(2010). Empirical evidence regarding relations among a model of epistemic and ontological cognition, academic performance, and educational level. Journal of Educational Psychology, 102(1), 234–255. https://doi.org/10.1037/a0017998

40.

Greene

J. A.

S. B.

Copeland

D. Z.

(2014). Measuring critical components of digital literacy and their relationships with learning. Computers & Education, 76, 55–69. https://doi.org/10.1016/j.compedu.2014.03.008

41.

Guo

Crossley

S. A.

McNamara

D. S.

(2013). Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18(3), 218–238. https://doi.org/10.1016/j.asw.2013.05.002

42.

Hofer

B. K.

Bendixen

L. D.

(2012). Personal epistemology: Theory, research, and future directions. In Harris

K. R.

Graham

Urdan

(Eds.), APA educational psychology handbook: Vol. 1. Theories, constructs, and critical issues (pp. 227–256). American Psychological Association.

43.

Hofer

B. K.

Pintrich

P. R.

(1997). The development of epistemological theories: Beliefs about knowledge and knowing and their relation to learning. Review of Educational Research, 67(1), 88–140. https://doi.org/10.3102/00346543067001088

44.

Kammerer

Amann

D. G.

Gerjets

(2015

). When adults without university education search the Internet for health information: The roles of Internet-specific epistemic beliefs and a source evaluation intervention.

Computers in Human Behavior, 48, 297–309. https://doi.org/10.1016/j.chb.2015.01.045

45.

Kammerer

Brand-Gruwel

(2020). Trainings and tools to foster source credibility evaluation during Web search. In Fu

W.-T.

van Oostendorp

(Eds.), Understanding and improving information search (pp. 213–243). Springer. https://doi.org/10.1007/978-3-030-38825-6_11

46.

Kammerer

Bråten

Gerjets

Strømsø

H. I.

(2013). The role of Internet-specific epistemic beliefs in laypersons' source evaluations and decisions during web search on a medical issue. Computers in Human Behavior, 29(3), 1193–1203. https://doi.org/10.1016/j.chb.2012.10.012

47.

Kammerer

Gerjets

(2014). The role of search result position and source trustworthiness in the selection of web search results when using a list or a grid interface. International Journal of Human-Computer-Interaction, 30(3), 177–191. https://doi.org/10.1080/10447318.2013.846790

48.

Kiili

Leu

D. J.

Utriainen

Coiro

Kanniainen

Tolvanen

Lohvansuu

Leppänen

P. H.

(2018). Reading to learn from online information: Modeling the factor structure. Journal of Literacy Research, 50(3), 304–334. https://doi.org/10.1177/1086296X18784640

49.

Kim

Thomas

Sankaranarayana

Gedeon

Yoon

H.-J.

(2015). Eye-tracking analysis of user behavior and performance in web search on large and small screens. Journal of the Association for Information Science and Technology, 66(3), 526–544. https://doi.org/10.1002/asi.23187

50.

Knight

Rienties

Littleton

Mitsui

Tempelaar

(2017). The relationship of (perceived) epistemic cognition to interaction with resources on the Internet. Computers in Human Behavior, 73, 507–518. https://doi.org/10.1016/j.chb.2017.04.014

51.

Latini

Bråten

Anmarkrud

Ø.

Salmerón

(2019). Investigating effects of reading medium and reading purpose on behavioral engagement and textual integration in a multiple document context. Contemporary Educational Psychology, 59, Article 101797. https://doi.org/10.1016/j.cedpsych.2019.101797

52.

Lee

W.-C.

Chiu

Y.-L.

Liang

J.-C.

Tsai

C.-C.

(2014). Exploring the structural relationships between high school students’ Internet-specific epistemic beliefs and their utilization of online academic help seeking. Computers in Human Behavior, 36, 391–400. https://doi.org/10.1016/j.chb.2014.03.069

53.

Lewandowski

(2011). The influence of commercial intent of search results on their perceived relevance. In Proceedings of the 2011 iConference (pp. 452–458). ACM. https://doi.org/10.1145/1940761.1940823

54.

Lin

S. F.

Lin

H. S.

Y. Y.

(2013). Validation and exploration of instruments for assessing public knowledge of and attitudes toward nanotechnology. Journal of Science Education and Technology, 22(4), 548–559. https://doi.org/10.1007/s10956-012-9413-9

55.

List

Alexander

P. A.

(2018). Corroborating students’ self-reports of source evaluation. Behaviour & Information Technology, 37(3), 198–216. https://doi.org/10.1080/0144929X.2018.1430849

56.

Lorigo

Haridasan

Brynjarsdóttir

Xia

Joachims

Gay

Granka

Pellacini

Pan

(2008). Eye tracking and online search: Lessons learned and challenges ahead. Journal of the American Society for Information Science and Technology, 59(7), 1041–1052. https://doi.org/10.1002/asi.20794

57.

Mason

Ariasi

Boldrin

(2011). Epistemic beliefs in action: Spontaneous reflections about knowledge and knowing during online information searching and their influence on learning. Learning and Instruction, 21(1), 137–151. https://doi.org/10.1016/j.learninstruc.2010.01.001

58.

Mason

Boldrin

Ariasi

(2010a). Epistemic metacognition in context: Evaluating and learning online information. Metacognition and Learning, 5, 67–90. https://doi.org/10.1007/s11409-009-9048-2

59.

Mason

Boldrin

Ariasi

(2010b). Searching the Web to learn about a controversial topic: Are students epistemically active? Instructional Science, 38(6), 607–633. https://doi.org/10.1007/s11251-008-9089-y

60.

McGrew

(2020). Learning to evaluate: An intervention in civic online reasoning. Computers & Education, 145, Article 103711. https://doi.org/10.1016/j.compedu.2019.103711

61.

Monchaux

Amadieu

Chevalier

Mariné

(2015). Query strategies during information searching: Effects of prior domain knowledge and complexity of the information problems to be solved. Information Processing & Management, 51(5), 557–569. https://doi.org/10.1016/j.ipm.2015.05.004

62.

Muis

K. R.

Bendixen

L. D.

Haerle

F. C.

(2006). Domain-generality and domain-specificity in personal epistemology research: Philosophical and empirical reflections in the development of a theoretical framework. Educational Psychology Review, 18(1), 3–54. https://doi.org/10.1007/s10648-006-9003-6

63.

Purcell

Brenner

Raine

(2012). Search engine use 2012. Pew Internet & American Life Project. http://pewinternet.org/∼/media/Files/Reports/2012/PIP_Search_Engine_Use_2012.pdf

64.

Rouet

J.-F.

Favart

Britt

M. A.

Perfetti

C. A.

(1997). Studying and using multiple documents in history: Effects of discipline expertise. Cognition and Instruction, 15(1), 85–106. https://doi.org/10.1207/s1532690xci1501_3

65.

Sanchiz

Chin

Chevalier

W.-T.

Amadieu

(2017). Searching for information on the web: Impact of cognitive aging, prior domain knowledge and complexity of the search problems. Information Processing & Management, 53(1), 281–294. https://doi.org/10.1016/j.ipm.2016.09.003

66.

Sandoval

W. A.

Greene

J. A.

Bråten

(2016). Understanding and promoting thinking about knowledge: Origin, issues, and future directions of research on epistemic cognition. Review of Research in Education, 40(1), 457–496. https://doi.org/10.3102/0091732X16669319

67.

Scharrer

Bromme

Britt

M. A.

Stadtler

(2012). The seduction of easiness: How science depictions influence laypeople’s reliance on their own evaluation of scientific information. Learning and Instruction, 22(3), 231–243. https://doi.org/10.1016/j.learninstruc.2011.11.004

68.

Schneider

Schlagmüller

Ennemoser

(2007). LGVT 6-12: Lesegeschwindigkeits- und -verständnistest [LGVT 6-12: A test for the assessment of reading speed and comprehension for high school students]. Hogrefe.

69.

Shah

Hendahewa

González‐Ibáñez

(2016). Rain or shine? Forecasting search process performance in exploratory search tasks. Journal of the Association for Information Science and Technology, 67(7), 1607–1623. https://doi.org/10.1002/asi.23484

70.

Stadtler

Bromme

(2014). The content–source integration model: A taxonomic description of how readers comprehend conflicting scientific information. In Rapp

D. N.

Braasch

J. L. G.

(Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 379–402). MIT Press.

71.

Strømsø

H .I.

Bråten

(2010). The role of personal epistemology in self-regulation of Internet-based learning. Metacognition and Learning, 5(1), 91–111. https://doi.org/10.1007/s11409-009-9043-7

72.

Strømsø

H. I.

Bråten

(2014). Students’ sourcing while reading and writing from multiple documents. Nordic Journal of Digital Literacy, 9(2), 92–111.

73.

Strømsø

H. I.

Bråten

Anmarkrud

Ø.

Ferguson

L. E.

(2016). Beliefs about justification for knowing when ethnic-majority and ethnic-minority students read multiple conflicting documents. Educational Psychology, 36(4), 638–657. https://doi.org/10.1080/01443410.2014.920080

74.

Strømsø

H. I.

Kammerer

(2016). Epistemic cognition and reading for understanding in the Internet age. In Greene

J. A.

Sandoval

W. A.

Bråten

(Eds.), Handbook of epistemic cognition (pp. 230–246). Routledge.

75.

L. Y. F.

Akin

Brossard

Scheufele

D. A.

Xenos

M. A.

(2015). Science news consumption patterns and their implications for public understanding of science. Journalism & Mass Communication Quarterly, 92(3), 597–616. https://doi.org/10.1177/1077699015586415

76.

Ulyshen

T. Z.

Koehler

M. J.

Gao

(2015). Understanding the connection between epistemic beliefs and Internet searching. Journal of Educational Computing Research, 53(3), 345–383. https://doi.org/10.1177/0735633115599604

77.

Van Deursen

A. J.

(2012). Internet skill-related problems in accessing online health information. International Journal of Medical Informatics, 81(1), 61–72. https://doi.org/10.1016/j.ijmedinf.2011.10.005

78.

von der Mühlen

Richter

Schmid

Schmidt

E. M.

Berthold

(2016). The use of source-related strategies in evaluating multiple psychology texts: A student-scientist comparison. Reading and Writing, 29(8), 1677–1698. https://doi.org/10.1007/s11145-015-9601-0

79.

White

R. W.

Dumais

S. T.

Teevan

(2009). Characterizing the influence of domain expertise on Web search behavior. In Baeza-Yates

Boldi

Ribeiro-Neto

Cambazoglu

B. B.

(Eds.) Proceedings of the Second ACM international Conference on Web Search and Data Mining (WSDM ‘09) (pp. 132–142). ACM. https://doi.org/10.1145/1498759.1498819

80.

Wiley

Goldman

Graesser

Sanchez

Ash

Hemmerich

(2009). Source evaluation, comprehension, and learning in internet science inquiry tasks. American Educational Research Journal, 46(4), 1060–1106. https://doi.org/10.3102/0002831209333183

81.

Wineburg

(1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83(1), 73–87. https://doi.org/10.1037/0022-0663.83.1.73

82.

Wineburg

McGrew

(2019). Lateral reading and the nature of expertise: Reading less and learning more when evaluating digital information. Teachers College Record, 121, 1–40.

83.

Zhang

Liu

Cole

Belkin

(2015). Predicting users' domain knowledge in information retrieval using multiple regression analysis of search behaviors. Journal of the Association for Information Science and Technology, 66(5), 980–1000. https://doi.org/10.1002/asi.23218

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.21 MB