Students’ Civic Online Reasoning: A National Portrait

Abstract

Are today’s students able to discern quality information from sham online? In the largest investigation of its kind, we administered an assessment to 3,446 high school students. Equipped with a live internet connection, the students responded to six constructed-response tasks. The students struggled on all of them. Asked to investigate a site claiming to “disseminate factual reports” on climate science, 96% never learned about the organization’s ties to the fossil fuel industry. Two thirds were unable to distinguish news stories from ads on a popular website’s home page. More than half believed that an anonymously posted Facebook video, shot in Russia, provided “strong evidence” of U.S. voter fraud. Instead of investigating the organization or group behind a site, students were often duped by weak signs of credibility: a website’s “look,” its top-level domain, the content on its About page, and the sheer quantity of information it provided. The study’s sample reflected the demographic profile of high school students in the United States, and a multilevel regression model explored whether scores varied by student characteristics. Findings revealed differences in student abilities by grade level, self-reported grades, locality, socioeconomic status, race, maternal education, and free/reduced-price lunch status. Taken together, these findings reveal an urgent need to prepare students to thrive in a world in which information flows ceaselessly across their screens.

Keywords

assessment regression analyses social studies education technology

Teenagers swim in a sea of digital information. Outside of school and homework, they spend more than 7 hours a day online (Rideout & Robb, 2019). Three quarters say they get news from social media “often” or “sometimes,” and nearly 70% do so through internet videos (Dautrich, 2018). Given this reality, teenagers need to be able to sort fact from fiction online to understand issues that affect themselves, their communities, and their country. Are they equipped to do so? Evidence suggests that young people, like many Americans, struggle to evaluate online sources (Gasser et al., 2012; Hargittai et al., 2010; Lurie & Mustafaraj, 2018; McGrew et al., 2018; Pan et al., 2007). However, prior research has rarely asked students to contend with actual digital content and has never yielded a nationwide perspective on students’ abilities. This study is the first to offer a national portrait across demographic groups of how students evaluated online sources.

Background

The health of a democracy depends on people’s ability to access reliable information (Hobbs, 2010; Mihailidis & Thevenin, 2013). If youth consume information without the ability to assess its credibility—unable to tell who is behind a cause and what their motives might be—they are easy marks for rogues of all stripes. On the internet, traditional gatekeepers and hallmarks of authority are largely absent. It is thus imperative to democratic functioning that students know how to assess the quality of information on which they base their decisions (Lynch, 2017; Metzger et al., 2010).

Unfortunately, research has revealed shortcomings in students’ approaches to evaluating digital content. Students who said they would base evaluations on source information rarely did so when observed in real time (Hargittai et al., 2010). Students frequently ignored source information (Bartlett & Miller, 2011; Barzilai & Zohar, 2012), focusing instead on the relevance of the information provided (Walraven et al., 2009) and basing their conclusions on a website’s surface-level features (Coiro et al., 2015; McGrew et al., 2018). These findings may reflect deficiencies in how students are taught to judge the credibility of internet sources. Many of the most widely used website evaluation materials—including those appearing on prestigious university websites—feature outdated strategies that can lead students dangerously astray (Breakstone et al., 2018; Caulfield, 2016; Sullivan, 2019; Wineburg et al., 2020; Wineburg & Ziv, 2020).

Despite research pointing to students’ lack of preparation, few tools exist to measure students’ competency in the digital realm. Extant assessments rarely ask students to evaluate real sources. Instead, they fill out Likert-scale items (Ashley et al., 2013) or respond to scaffolded hypothetical scenarios (Leu et al., 2012). From such proxy measures, it is hard to know what students actually do online.

The limited assessment options spurred us to develop new measures. To do so, we first sought to identify expertise in the evaluation of online sources. We asked three groups to examine a series of websites: Stanford University students, university professors from five different institutions, and professional fact checkers from leading news outlets (Wineburg & McGrew, 2017). Students and academics often approached sites by “reading vertically,” starting at the top of their screen and proceeding to the bottom, pausing to comment on the site’s prose and layout, the presence of references and contact information, and the site’s top-level domain (dot-orgs good, dot-coms bad). Fact checkers took a radically different tack. Landing on an unfamiliar website, they clicked away from it and engaged in lateral reading (Wineburg & McGrew, 2019). They opened up new tabs across the horizontal axis of their browser and searched for information about the organization or individual behind it. Only after surveying what other sites had to say did they return to their original starting place. Using this approach, fact checkers quickly sized up sites that masked their intent and backers. Students and academics, on the other hand, often dwelled on the original site, resulting in confusion about its real agenda or sponsor. Lateral reading allowed fact checkers to answer a crucial question: Who is behind the information? Their evaluations were guided by two other questions: (1) What is the evidence? (2) What do other sources say? Taken together, these strategies allowed fact checkers to better evaluate digital content than either academics or college students.

We refer to fact checkers’ approach to searching for and evaluating social and political information as Civic Online Reasoning. Our purview is more targeted than the expansive fields of media and digital literacy, which can embrace topics ranging from cyberbullying to identity theft. It focuses squarely on how to sort fact from fiction online, a prerequisite for responsible civic engagement in the 21st century (Kahne et al., 2012; Mihailidis & Thevenin, 2013).

In 2015, our research team launched a 2-year project to develop assessments that measured Civic Online Reasoning. We developed a range of tasks through an iterative process of prototyping, expert review, and piloting with students in classrooms across the country. We also used think-aloud protocols to ensure that the assessments tapped the intended skills. These tasks asked students to evaluate real digital sources, ranging from Facebook videos to websites of advocacy groups pretending to be nonpartisan think tanks. Students from middle school to college struggled to perform basic evaluations of digital content (McGrew et al., 2018). Middle school students confused sponsored content with news stories. High school students trusted a photograph posted anonymously to a social media site. College students deemed a website about minimum wage policy credible even though it was run by a public relations firm.

Despite widespread concern about digital misinformation, no prior research has gauged students’ ability to evaluate online sources on a national scale. In this study, students completed a holistic assessment in which they went online to evaluate a range of sources. We analyzed data from 3,446 students, a sample that reflected the demographic profile of high school students in the United States. This approach yielded evidence of students’ ability to evaluate online information and evidence of how student performance on the assessment varied by region, locality, grade level, race/ethnicity, gender, home language, grades in school, maternal education, and free/reduced-price lunch status.

Method

Participating Districts

Using the 2016–2017 Common Core of Data from the National Center for Education Statistics (NCES), we compiled a list of all traditional public and charter schools in the United States that enrolled students through Grade 12 (NCES, n.d.-b). We categorized districts by region (Northeast, Midwest, South, West) and grouped them within regions by locale (urban, suburban, and rural/town), resulting in 12 separate categories.

We recruited at least one district in each of the 12 categories. We prioritized districts with racial and ethnic diversity, and we aimed to maximize the number of states represented.¹ The final sample included 16 districts in 14 states. The participating districts varied considerably in enrollment, racial and ethnic diversity, percentage of students receiving special education services, and proportion of schools eligible for Title I funds under the Elementary and Secondary Education Act (see Table 1).

Table 1

Characteristics of the Participating Districts

Region	Locale	Enrollment	Title I Schools (%)	Hispanic (%)	White (%)	Black/African American (%)	Asian (%)	Native American (%)	Special Education (%)	Limited English Proficiency (%)
Northeast	Urban	22,359	89.7	3.1	32.7	52.9	3.6	0.2	21.9	4.0
Northeast	Suburban	15,878	0.0	15.1	73.1	5.2	4.0	0.1	17.2	1.3
Northeast	Rural	2,089	0.0	1.8	91.0	1.6	5.3	0.3	14.4	1.0
Northeast	Rural	2,034	83.3	3.7	88.9	2.5	1.3	0.1	17.7	0.7
Midwest	Urban	45,455	92.3	14.6	2.5	80.7	1.7	0.2	17.0	12.3
Midwest	Suburban	20,322	0.0	3.1	76.8	4.0	11.7	0.2	11.4	2.2
Midwest	Suburban	21,029	100.0	8.7	44.4	23.7	16.2	0.3	14.2	10.5
Midwest	Rural	5,122	100.0	3.0	89.3	1.3	1.6	0.6	15.5	1.2
Midwest	Rural	5,100	38.5	4.5	85.6	1.8	4.6	1.2	14.0	2.1
South	Urban	12,115	10.0	15.7	51.5	11.2	15.0	0.3	10.2	10.4
South	Suburban	15,942	60.0	13.9	58.5	6.1	3.1	4.6	17.2	4.3
South	Rural	427	100.0	15.2	71.9	6.6	0.9	0.2	10.5	0.7
South	Rural	2,486	100.0	30.0	66.3	1.0	0.6	0.6	9.5	11.1
West	Urban	29,682	9.6	17.8	73.8	1.1	3.2	0.5	^*	5.1
West	Suburban	4,074	0.0	13.7	76.1	0.6	1.3	1.2	9.5	1.7
West	Rural	6,579	68.2	19.3	65.6	0.9	1.2	6.5	14.5	4.6

Note. *The total count of students receiving special education services was missing for the urban district in the West in both 2016–17 and 2015–16 NCES data. Title I status was missing at the school level for 2016–17 for the rural district in the West, so we used data from 2015–16 instead.

Teacher Recruitment and Registration

We randomly selected three high schools (Grades 9–12) to participate within each district; if a district had three or fewer high schools, all were included.² We invited all social studies teachers to participate at each of the selected schools. For each teacher who volunteered, we randomly selected two sections from their schedule and asked them to administer assessments in those sections. As a token of appreciation, we provided $50 gift cards for each of the selected class sections that participated. Teachers could choose to administer the assessment in other sections as well.

Administration

The assessment was administered with an online survey platform. Participation was voluntary. The students completed the survey using internet-connected computers during their regularly scheduled social studies class, and they were allowed to use the internet at any time to answer the questions. The tasks appeared in the same order for all students. The students were not allowed to return to items after they had submitted their responses; however, no time limits were imposed. Teachers were allowed to provide technical assistance to students, but they could not help students evaluate sources, interpret questions, or compose responses.

Analytic Sample

We collected completed assessments from 6,445 students in 79 classrooms at 31 schools across 16 districts. To ensure that our sample reflected the U.S. population of students by region and locale, we used the 2016–2017 Common Core of Data from NCES to calculate the percentages of public school students in each geographical region of the United States (Midwest, Northeast, South, and West) and by locale within each region (rural, suburban, and urban). Next, we generated a random sample of 2,937 students that was proportionate to the distribution of students in each region and locale.

We examined the racial and ethnic composition of this sample and found that the proportions of Hispanic and Black/African American students were lower than in the national population of public school students.³ To improve the representativeness of our sample, we returned to the set of completed assessments and added the remaining 509 students who identified themselves as Hispanic or Black/African American to the analytic sample regardless of their district. Adding these students resulted in a final analytic sample of 3,446 students that better matched the racial/ethnic makeup of American public schools (see Table 2).

Table 2

Analytic Sample by Region, Locale, Race/Ethnicity, Gender, Grade Level, Free/Reduced-Price Lunch Status, and Mother’s Education

Category	% U.S. Enrollment^a	Analytic Sample (n)	Analytic Sample (%)	Difference Between Analytic Sample and % U.S. Enrollment (%)	Analytic Sample Without Missing (%)
Region
Northeast	16	732	21	5^*	—
Midwest	21	852	25	3^*	—
South	38	1,084	31	−6^*	—
West	25	778	23	−3^*	—
Locale
Urban	31	967	28	−3^*	—
Suburban	42	1,659	48	6^*	—
Rural	27	820	24	−3^*	—
Race/ethnicity
Asian or Pacific Islander	5	112	3	−2^*	4
American Indian/Alaska Native	1	29	1	0	1
Black or African American	15	306	9	−6^*	11
Hispanic	26	698	20	−6^*	25
Other	—	42	1	—	2
Two or more races	3	224	7	4^*	8
White	49	1,373	40	−9^*	49
Prefer not to say	—	28	1	—	—
Missing	—	634	18	—	—
Gender
Male	52	1,283	37	−15^*	43
Female	48	1,443	42	−6^*	49
Nonbinary/self-described/prefer not to say	—	230	7	—	8
Missing	—	490	14	—	—
Grade level
8th	—	17	0	—	1
9th	27	820	24	−3^*	28
10th	26	491	14	−12^*	17
11th	24	867	25	1	29
12th	24	761	22	−2^*	26
Missing	—	490	14	—	—
National School Lunch Program status
Eligible/participant	52	850	25	−27^*	29
Not eligible/nonparticipant	48	1,475	43	−5^*	50
I don’t know	—	533	15	—	18
Prefer not to say	—	84	2	—	3
Missing	—	504	15	—	—
Mother’s education level^b
Less than high school	—	222	6	—	8
High school	—	506	15	—	17
Some college	—	460	13	—	16
Bachelor’s degree	—	736	21	—	25
Advanced degree	—	473	14	—	16
I don’t know	—	411	12	—	14
Prefer not to say	—	148	4	—	5
Missing	—	490	14	—	—

Note. Statistical significance of the difference between the analytic sample and the U.S. enrollment target for each student subpopulation was obtained from a one-sample difference in proportion test.

U.S. enrollment data are from the National Center for Education Statistics Common Core of Data (2016–2017).

Common Core of Data does not include data on mother’s education for U.S. public school students.

p < .05.

Adding the Black/African American and Hispanic students changed the composition of the sample by region/locality, but the impact was small. Table 2 shows the geographic distribution of the final analytic sample. Students from suburban schools in the Northeast and Midwest were slightly overrepresented, and students from the South were slightly underrepresented, but the overall sample reflected the geographic distribution of students in the United States. We also considered whether the sample reflected the self-reported grade level, gender, and free/reduced-price lunch eligibility (see Table 2).

Assessment Instrument

The assessment included six constructed-response tasks previously developed by our team (cf. McGrew et al., 2018). The tasks sampled a range of Civic Online Reasoning skills, such as lateral reading and evaluating evidence on social media. Appendix A in the online supplement (available on the journal website) provides descriptions of the tasks, and Appendix B (available on the journal website) includes the instrument. For example, the Website Evaluation task (Task 3 in Appendix B) assessed whether students could investigate who was behind a website. The students were provided a link to the home page of co2science.org, an organization whose About page states that they seek to “disseminate factual reports and sound commentary” on the environmental effects of carbon dioxide. The group is actually funded by fossil fuel companies, including ExxonMobil. The students were asked to judge whether the site was reliable, and a screen prompt reminded them that they could search online to answer.

A second task, Evaluating Evidence (Task 1 in Appendix B), gauged students’ ability to evaluate the trustworthiness of a social media post. The task presented students with a Facebook video from a user named “I on Flicks” and asked if it constituted “strong evidence” of voter fraud during the 2016 Democratic primaries. The video included four silent clips of poll workers surreptitiously stuffing ballots into bins. Captions told viewers that the clips depicted Democratic primary elections in Illinois, Pennsylvania, and Arizona. The post accompanying the video read, “Have you ever noticed that the ONLY people caught committing voter fraud are Democrats?” None of this was true. The clips actually showed voter fraud in Russia, something easily ascertained by a brief Web search.

After the students completed the six tasks, they self-reported a range of demographic characteristics, including age, grade, gender, race, ethnicity, primary language, typical grades in school, mother’s education, and free/reduced-price lunch participation.

Rubric Design and Scoring Procedures

Rubric design was informed by prior research and development of Civic Online Reasoning assessments (McGrew et al., 2018; Wineburg et al., 2016). In Mastery responses, students evaluated online content by investigating the source of information, by interrogating the evidence presented, or by seeking out information from other reliable sources. Emerging responses were partially incorrect or did not fully articulate sound reasoning. Beginning responses included incorrect or irrelevant strategies for evaluating online information. See Appendix C for a sample rubric.

Two independent raters scored the student responses using this three-level rubric (Beginning = 0, Emerging = 1, Mastery = 2).⁴ Estimates of interrater reliability were high, with kappa coefficients between .93 and .97 for each task (Task 1 = .95, Task 2 = .97, Task 3 = .94, Task 4 = .96, Task 5 = .94, and Task 6 = .93).

Multivariate Regression Analysis

We used multivariate regression to explore the relationship between student characteristics and performance on the assessment. We first created an overall composite score for students (n = 3,402) across the six tasks. We dropped 44 students in the analytic sample from the composite score analysis because none of their completed tasks could be scored (e.g., they indicated that they encountered technical problems on the tasks they completed). We then fit a multilevel regression model to explore differences in composite scores by a variety of student characteristics. Below, we outline how we constructed the composite score and specify the model used for the analysis.

Composite Score

The composite score was the average of the students’ scores on the items they completed.⁵ So if a student completed all six items, then their composite score was the mean of their six scores; if they completed three items, their composite score was the mean of those three scores. The average composite score ranged from 0 (a student received a score of 0 on all completed items) to 2 (a student received a score of 2 on all completed items).

Multivariate Analysis

We fit a single two-level hierarchical linear model with random intercepts at the school level.⁶ We included the students’ self-reported grades as a measure of prior academic achievement because higher-achieving students might score systematically higher. The “student grades” ranged from 1 (students reported they earned “mostly Ds”) to 7 (students reported “mostly As”). We also anticipated that students in upper grades might perform better than students in lower grades, so we included grade level in the model.

Additionally, we included students’ geographic region and locale to adjust for any systematic differences by student location. Finally, we included demographic variables that raw scores suggested might be relevant to student performance on the tasks, including race/ethnicity, gender, free/reduced-price lunch status, maternal education, and first language spoken at home.⁷ Appendix D describes these variables and how they were measured; Appendix E provides the percentage of students scoring Emerging or Mastery by task for each of the variables.

Results

Students struggled on all of the tasks. At least two thirds of the student responses were scored as Beginning for each of the six tasks. On four of the six tasks, more than 90% of students’ responses were scored as Beginning. Of all the student responses to the six tasks, fewer than 3% received a Mastery score. Claims on Social Media 1 (Task 4 in Appendix B) had the lowest proportion of Mastery responses, with fewer than 1% of the students demonstrating a strong understanding of the online reasoning competencies measured. Evaluating Evidence had the highest proportion, with 8.7% earning a Mastery score. See Table 3 for the distribution of scores by task.

Table 3

Percentage of Responses at Each Rubric Level by Task

Level	Evaluating Evidence	Webpage Comparison	Website Evaluation	Social Media 1	Social Media 2	Home Page Analysis
Beginning	77.4%	92.1%	96.8%	92.8%	95.4%	68.7%
Emerging	13.9%	6.7%	1.4%	6.9%	2.7%	26.6%
Mastery	8.7%	1.2%	1.8%	0.3%	1.9%	4.7%
Total (N)	3,119	2,678	2,994	2,431	2,431	2,969

Note. Some students in the analytic sample did not complete all the tasks or did not provide responses that could be scored. Consequently, the total number of responses varied by task.

The mean composite score across all students was 0.17 out of a possible score of 2. See Appendix F for the distribution of composite scores. Fifty-eight percent of students had an average composite score of 0, indicating that they scored at the Beginning level on all of the tasks they completed. Of the 3,402 students in the analytic sample with a composite score, only 13 (0.38%) scored at the Mastery level on all six tasks.

Website Evaluation Task

The Website Evaluation task, which presented students with a climate change website funded by fossil fuel companies, had the highest proportion of Beginning scores, with 96.8% of students receiving the lowest score. Instead of leaving the site, the vast majority of students remained glued to the site itself, drawn in by its top-level domain (.org), the recency of its updates, and the sheer quantity of information it included. A student from a suburban district in the Northeast wrote, “This page is a reliable source to obtain information from. You see in the URL that it ends in .org as opposed to .com.” This student was seemingly unaware that, unlike dot-gov or dot-edu sites, dot-org is an “open” domain; any individual or group can obtain one without passing a character test or attesting to benevolent intent (Wineburg & Ziv, 2019). Students also gave undue weight to the site’s About page, not recognizing it as a curated portrait of how a group wished to be perceived. A student from the urban South wrote,

The “about us” tab does show that the organization is a non-profit dedicated to simply providing the research and stating what it may represent. In their position paper on CO₂, they provide evidence for both sides and state matters in a scientific manner. Therefore, I would say they are an unbiased and reliable source.

In contrast, strong student responses used other online sources to discover that co2science.org undermines climate change science on behalf of corporate funders. Establishing this was not hard but did require that students leave the site to investigate. Entering the group’s name in their browsers and reading laterally called up a host of results that shed light on the group’s agenda. Less than 2% of students did this. One student who did wrote,

co2science.org is not a reliable source because it has ties to large companies that want to purposefully mislead people when it comes to climate change. According to USA TODAY, Exxon has sponsored this nonprofit to pump out misleading information on climate change. According to the Seattle Post-Intelligencer, many of their scientists also have ties with energy lobbyists.

Evaluating Evidence Task

More than half of the students (52%) were taken in by the video purportedly showing ballot stuffing in the United States. They concluded that the video provided “strong evidence” of Democratic voter fraud in the primary elections. A student from a rural district in the Midwest took the video at face value: “Yes, it shows video evidence of fraud in multiple different states at multiple different times.” A student from an urban district in the Northeast wrote, “Yes, because the video showed people entering fake votes into boxes.”

A quarter of the students rejected the video but did not provide a coherent rationale for doing so. Students focused on irrelevant features such as the lack of audio, the video’s grainy quality, or insufficient narration from the post’s author. Others accepted the veracity of the video but ended up rejecting it because there were too few examples. A student from the South wrote, “The video only shows a few specific instances, and is not enough to blame the whole Democratic party of voter fraud.” Still others struggled to interpret what was depicted in the clips, ignoring the source and origin of the video altogether. One student explained,

I have no idea what is going on exactly and I see people put papers in the box, but there are cases where the votes are sneaked in. I am not sure this is a strong case of faking votes for a person, but it is a case for vote fraud.

Only 8.7% of students received a Mastery score by rejecting the video and providing a relevant explanation. These students understood that they had no way of knowing if “I on Flicks” was a reliable source or whether the video depicted American voting fraud. A student from an urban district in the West questioned the video’s origin:

It does not provide enough evidence because we don’t know who the people are in the video, and who the people are that made the video. There is very little context and information given, besides the place, and a little bit of commentary making claims to convince you.

Of the 3,119 responses on this task, only three responses actually tracked down the source of the video. Again, doing so required little more than choosing relevant search terms (e.g., “video” and “Democratic ballot stuffing”) and scanning the search engine results page for sites like Snopes or the BBC. A student from a suburban district in the West showed what could be learned by leaving the video and reading laterally:

Doing some research on the Internet, I found the exact same video as shown above. However, the video which I found, which was presented by BBC News, showed that the cam footage had nothing to do with the 2016 Democratic primary elections. Instead, BBC presented the cam footage as that of the Russian 2016 election. According to the announcer of BBC News, the video showed that the people in Russia—not America—were committing voter fraud.

Such responses were few and far between.

Multivariate Results

Our regression analysis revealed significant differences in students’ composite scores for a variety of demographic factors. Table 4 shows the regression coefficients and standard errors for each of the variables in our model. Adjusting for all other included variables, students who reported earning higher grades performed significantly better than students who reported earning lower grades. A one-step increase in self-reported grades (e.g., from “mostly Bs” to “mostly As and Bs”) was associated with a .03 improvement in average composite score (a noticeable improvement in a scale with a maximum of 2). Similarly, grade level was a significant predictor of student performance. Each increase in grade level (e.g., from sophomore to junior) was associated with a .03 increase in students’ composite scores for students who were similar in all other reported characteristics.

Table 4

Results From the Mixed-Effects Regression Model

Category	Composite Score
Student grades	0.029 (.00)***
National School Lunch Program eligible	−0.044 (.02)**
Mother’s education	0.014 (.01)*
Grade level	0.029 (.07)**
Region (compared with Midwest)
Northeast	−0.036 (.04)
South	0.009 (.03)
West	0.001 (.04)
Locality (compared with rural)
Suburban	0.043 (.03)
Urban	0.107 (.03)***
Gender (compared with female)
Male	0.017 (.01)
Nonbinary/prefer to self-describe	−0.020 (.03)
Home language (compared with English)
Other	−0.036 (.03)
Spanish	−0.012 (.02)
Race/ethnicity (compared with White)
Asian	0.019 (.03)
Black or African American	−0.120 (.03)***
Other	−0.083 (.04)*
Two or more races	−0.039 (.02)
Hispanic	−0.014 (.18)
_cons	−0.157 (.05)**
Variance: school	.003 (.001)*
Intraclass correlation (ρ)	.044
Schools: n	31: 1,690
Wald χ²	87.59

Note. Standard errors are in parentheses.

p < .05. ^**p < .01. ^***p < .001.

We also observed significant results for proxy measures of socioeconomic status. Maternal education was a significant predictor of average composite score. For every one-step increase in a mother’s reported education level (e.g., from less than high school to high school, or from some college to a bachelor’s degree), performance improved by about .01 point on average. Free/reduced-price lunch status was also a significant predictor of performance, with students eligible to receive free/reduced-price lunch performing .04 point lower, on average, than students not eligible, adjusting for all other variables in the model.

Our model also revealed significant differences by other characteristics. Urban students significantly outperformed their rural and suburban peers. Table 5 includes contrasts between each locale pair not included in the initial model. Adjusting for all other variables, urban students scored .11 point higher than rural students and .06 point higher than suburban students. Race/ethnicity was also a significant factor in our model. After adjusting for all other variables, Black/African American students in our sample had lower composite scores than students who were Asian/Pacific Islander (−.14 point), Hispanic (−.11 point), multiracial (−.08 point), or White (−.12 point). Table 5 includes the contrasts for each race/ethnicity comparison not included in the initial model. The other variables were not significant predictors of student scores. Gender, geographic region, and home language were not significant predictors when adjusting for all other variables.

Table 5

Contrasts of Marginal Effects for Race/Ethnicity and Locale

Category	Contrast
Race/ethnicity
Black vs. Asian	−0.139 (.04)**
Black vs. White	−0.120 (.03)***
Black vs. other	−0.037 (.04)
Black vs. two or more races	−0.081 (.03)**
Black vs. Hispanic	−0.105 (.03)**
Locality
Urban vs. rural	0.107 (.03)^*
Urban vs. suburban	0.064 (.03)^*

Note. Standard errors are in parentheses.

p < .05. ^**p < .01. ^***p < .001.

Limitations

This study’s sample was not randomly generated; as a result, it was not perfectly representative on all observable characteristics. Assembling a truly random, representative sample of students in the United States for this type of assessment would have presented prohibitive administrative and financial barriers. In contrast to the National Assessment of Educational Progress (NAEP), we had no legislative mandate for administration and far fewer resources for data collection. In contrast to NAEP, we relied only on the goodwill of school districts to participate, negotiated research access with districts individually, and worked with technology administrators to ensure students’ access to the open internet during assessment administration. Additionally, it was a low-stakes assessment. Students were not required to participate, and their performance did not affect course grades. However, the same holds true for NAEP exams, which are held up as the “gold standard” of large-scale testing (NCES, n.d.-c). Despite these limitations, this is the largest study of students’ online reasoning skills to date in the United States.

Measuring students’ Civic Online Reasoning at scale presented a variety of challenges. In the interest of ecological validity, we asked the students to evaluate real online sources about social and political issues using the internet. This openness stands in stark contrast to the tightly controlled environments typical of large-scale assessment. We could not control the sources that the students encountered while searching online. Nor could we directly observe how they arrived at their answers. Giving students a free rein online also meant greater heterogeneity in the responses collected, which made scoring more difficult. Going forward, researchers will need to weigh the benefits of asking students to engage with authentic sources against the measurement challenges that the accompanying freedom creates.

There are important dimensions of evaluating online sources that this study did not address. In particular, it did not account for students’ political beliefs. Although the assessment deliberately included sources from across the political spectrum, students’ beliefs may have influenced their evaluations of sources about contentious social and political issues. The interaction between students’ political orientations and their evaluations of digital sources obviously merits further study.

Discussion

The results of this study should give pause to anyone who cares about the future of democracy. More than half of the students believed that an anonymously posted Facebook video, actually shot in Russia, provided strong evidence of U.S. voter fraud. Students displayed a troubling tendency to remain on a website and take it at face value. More than 96% never learned about the ties between a climate change website and the fossil fuel industry. Instead of investigating the group behind the site, students were duped by weak signs of credibility: the website’s “look,” its top-level domain, or the content on its About page. Each of these features is ludicrously easy to game.

All students would benefit from carefully designed, thoughtfully delivered, and rigorously evaluated curriculum. At the same time, our findings suggest that the students most underserved by our nation’s schools are similarly underserved in terms of access to quality materials for evaluating digital sources. Socioeconomic status affects educational outcomes in the United States (cf. Mandara et al., 2009; Reardon et al., 2015), and indicators of socioeconomic status were predictors of scores here too. After controlling for other variables, eligibility for the National School Lunch Program was negatively associated with student scores; higher maternal educational attainment was associated with higher scores. A pernicious racial opportunity gap persists in American schools (cf. Howard, 2019; Reardon et al., 2015), and our findings suggest that it was at play here as well.

It would be a mistake to see our results as an indictment of students. Presented with something to read, they mostly did what they have been taught. They read. Unless students possessed deep subject matter knowledge, a site like co2science.org—studded with citations to scientific articles, “major reports,” and lists of awards—looked, well, pretty good. The few students who learned that the website was backed by corporate interests did so not because they stayed on the site to engage in “close reading” (Common Core State Standards Initiative, n.d.). They succeeded because they did not.

When browsing the internet is how we become informed citizens, traditional ways of reading are not merely ineffective (Gigerenzer & Gaissmaier, 2011; Hertwig & Engel, 2016; Kozyreva et al., 2020), they’re dangerous (Caulfield, 2018; Warzel, 2021). As far back as 1971, Herbert Simon presciently noted that information overload results in a scarcity of attention. Today, when advertisers, corporations, lobbyists, clickbait sites, conspiracy theorists, hate groups, and foreign governments work overtime to hijack attention, often the wisest thing to do is not to read but to preserve attention by practicing strategic ignoring (Wineburg & McGrew, 2019). Under conditions of limited attention, the most crucial decision to make is where to allocate it. Above all else, lateral reading is an act of attention conservation. One uses the awesome powers of the internet to quickly decide if an unfamiliar group, organization, or claim merits sustained attention. The few students who left a website to navigate the open internet found answers waiting for them at the ready.

We have emphasized the need to leave a site to get an accurate read on it. Students would obviously benefit by mastering a host of other skills: choosing effective search terms, conducting reverse image searches, checking archived versions of webpages in the Internet Archive, and so on. We have not addressed the thorny issue of motivated reasoning, which renders the most sophisticated searches moot if the searcher’s mind is made up from the beginning. But we need to start somewhere. Unlike learning how to apply Boolean operators or decode a video’s metadata, the basics of lateral reading are straightforward. Students open new tabs on their browsers and enter search terms, something they already know how to do. But they will not do so unless they understand, in deep and compelling ways, that the Web plays by a different set of rules from the vetted texts provided to them in school.

A much greater investment in research and development is needed to prepare students to navigate treacherous digital terrain. Small-scale but promising studies suggest that it is possible to improve students’ digital savvy (Breakstone et al., 2021; Brodsky et al., 2019; Kohnen et al., 2020; McGrew, 2020; McGrew et al., 2019). Given the threat to civic and public health posed by digital illiteracy, these preliminary interventions need to be taken up at scale to identify approaches that can be widely adopted. Investigating how teachers across subject areas and grade levels integrate these skills into existing instruction will be critical. It is doubtful that students will become skilled consumers of digital content if instruction is restricted to a single course or one-off workshop.

Conclusion

Across regions and demographic groups, the students in this study fell short. Their credulity makes them—along with the rest of us—easy prey for groups that seek to deceive, mislead, and manipulate. The toxic effects of disinformation have chipped away at the foundations of democracy the world over (Diamond, 2020; Levitsky & Ziblatt, 2018).

In October 1957, a whirling orbital ball known as Sputnik roused Americans from their slumber. It led to the passage of the National Defense Education Act, a massive federal investment in American education. Today, the threat posed to national security by a digitally credulous citizenry warrants a similarly urgent response.

Anything less guarantees the continued erosion of a democratic society.

Supplemental Material

sj-pdf-1-edr-10.3102_0013189X211017495 – Supplemental material for Students’ Civic Online Reasoning: A National Portrait

Supplemental material, sj-pdf-1-edr-10.3102_0013189X211017495 for Students’ Civic Online Reasoning: A National Portrait by Joel Breakstone, Mark Smith, Sam Wineburg, Amie Rapaport, Jill Carle, Marshall Garland and Anna Saavedra in Educational Researcher

Footnotes

ORCID iD

Joel Breakstone

Notes

Authors

JOEL BREAKSTONE, PhD, is the director of the Stanford History Education Group at Stanford University, 485 Lasuen Mall, Stanford, CA 94305; breakstone@stanford.edu. His research focuses on curriculum and assessment development.

MARK SMITH, PhD, is the director of assessment of the Stanford History Education Group at Stanford University, 485 Lasuen Mall, Stanford, CA 94305; msmith4@stanford.edu. His research focuses on history/social studies assessment and curriculum.

SAM WINEBURG, PhD, is the Margaret Jacks Professor of Education at Stanford University, 485 Lausen Mall, Stanford, CA 94305; wineburg@stanford.edu . His research focuses on how people learn to distinguish truth from falsehood on the internet.

AMIE RAPAPORT, PhD, is director of research at Gibson Consulting Group, 1801 South Mopac Expressway, Austin, TX 78746; arapaport@gibsonconsult.com . Her research focuses on intervention implementation and effectiveness.

JILL CARLE, PhD, is a senior research associate at Westat, 1600 Research Blvd., Rockville, MD 20850; JillCarle@westat.com . Her research focuses on methods in large-scale survey operations.

MARSHALL GARLAND, MA, is a senior research scientist at Gibson Consulting Group, 1801 South MoPac Expressway, Austin, TX 78746; mgarland@gibsonconsult.com . His research focuses on the conditions that support students’ successful postsecondary transitions.

ANNA SAAVEDRA, EdD, is a research scientist at the University of Southern California Dornsife Center for Economic and Social Research, 635 Downey Way VPD Suite 505, Los Angeles, CA 90089; asaavedr@usc.edu . Her research focuses on student-centered instruction, particularly in the context of civic education.

References

Ashley

Maksl

Craft

(2013). Developing a news media literacy scale. Journalism & Mass Communication Educator, 68(1), 7–21. https://doi.org/10.1177/1077695812469802

Bartlett

Miller

(2011, September). Truth, lies and the internet. Demos. https://www.demos.co.uk/files/Truth_-_web.pdf

Barzilai

Zohar

(2012). Epistemic thinking in action: Evaluating and integrating online sources. Cognition and Instruction, 30(1), 39–85. https://doi.org/10.1080/07370008.2011.636495

Breakstone

McGrew

Smith

Ortega

Wineburg

(2018). Why we need a new approach to teaching digital literacy. Phi Delta Kappan, 99(6), 27–32. https://doi.org/10.1177/0031721718762419

Breakstone

Smith

Connors

Ortega

Kerr

Wineburg

(2021). Lateral reading: College students learn to critically evaluate internet sources in an online course. Harvard Kennedy School Misinformation Review, 2(1), 1–17. https://doi.org/10.37016/mr-2020-56

Brodsky

Brooks

P. J.

Scimeca

Todorova

Galati

Batson

Grosso

Matthews

Miller

Tachiera

Caulfield

(2019, October 3–5). Teaching college students the four moves of expert fact-checkers [Paper presentation]. Technology, Mind, & Society, an American Psychological Association Conference, Washington, DC, United States.

Caulfield

(2016, December 19). Yes, digital literacy. But which one? Hapgood. https://hapgood.us/2016/12/19/yes-digital-literacy-but-which-one/

Caulfield

(2018, December 19). Recalibrating our approach to misinformation. EdSurge. https://www.edsurge.com/news/2018-12-19-recalibrating-our-approach-to-misinformation

Coiro

Coscarelli

Maykel

Forzani

(2015). Investigating criteria that seventh graders use to evaluate the quality of online information. Journal of Adolescent and Adult Literacy, 59(3), 287–297. https://doi.org/10.1002/jaal.448

10.

Common Core State Standards Initiative. (n.d.). English language arts standards. http://www.corestandards.org/ELA-Literacy/CCRA/R/

11.

Dautrich

(2018, December 5). The future of the First Amendment: 2018 Survey of high school students and teachers. John S. and James L. Knight Foundation. https://knightfoundation.org/reports/the-future-of-the-first-amendment-2018/

12.

Diamond

(2020). Ill winds: Saving democracy from Russian rage, Chinese ambition, and American complacency. Penguin.

13.

Gasser

Cortesi

Malik

Lee

(2012). Youth and digital media: From credibility to information quality. Berkman Klein Center for Internet & Society. https://doi.org/10.2139/ssrn.2005272

14.

Gigerenzer

Gaissmaier

(2011). Heuristic decision making. Annual Review of Psychology, 65, 451–482. https://doi.org/10.1146/annurev-psych-120709-145346

15.

Hargittai

Fullerton

Menchen-Trevino

Thomas

K. Y.

(2010). Trust online: Young adults’ evaluation of web content. International Journal of Communication, 4, 468–494.

16.

Hertwig

Engel

(2016). Homo ignorans: Deliberately choosing not to know. Perspectives on Psychological Science, 11(3), 359–372. https://doi.org/10.1177/1745691616635594

17.

Hobbs

(2010). Digital and media literacy: A plan of action. Aspen Institute. https://assets.aspeninstitute.org/content/uploads/2010/11/Digital_and_Media_Literacy.pdf

18.

Howard

T. C.

(2019). Why race and culture matter in schools: Closing the achievement gap in America’s classrooms. Teachers College Press.

19.

Kahne

Lee

Feezell

J. T.

(2012). Digital media literacy education and online civic and political participation. International Journal of Communication, 6, 1–24.

20.

Kohnen

A. M.

Mertens

G. E.

Boehm

S. M.

(2020). Can middle schoolers learn to read the web like experts? Possibilities and limits of a strategy-based intervention. Journal of Media Literacy Education, 12(2), 64–79. https://doi.org/10.23860/JMLE-2020-12-2-6

21.

Kozyreva

Lewandowsky

Hertwig

(2020). Citizens versus the internet: Confronting digital challenges with cognitive tools. Psychological Science in the Public Interest, 21(3), 103–156. https://doi.org/10.1177/1529100620946707

22.

Leu

D. J.

Coiro

Kulikowich

J. M.

Cui

(2012, November 28–December 1). Using the psychometric characteristics of multiple-choice, open internet, and closed (simulated) internet formats to refine the development of online research and comprehension assessments in science: Year three of the ORCA project [Paper presentation]. Literacy Research Association 62nd Annual Conference, San Diego, CA, United States.

23.

Levitsky

Ziblatt

(2018). How democracies die. Penguin.

24.

Lurie

Mustafaraj

(2018, May). Investigating the effects of Google’s search engine result page in evaluating the credibility of online news sources. In Proceedings of the 10th ACM Conference on Web Science (pp. 107–116). Association for Computing Machinery. https://doi.org/10.1145/3201064.3201095

25.

Lynch

M. P.

(2017). An internet of us: Knowing more and understanding less in an age of big data. W. W. Norton.

26.

Mandara

Varner

Greene

Richman

(2009). Intergenerational family predictors of the Black–White achievement gap. Journal of Educational Psychology, 101(4), 867–878. https://doi.org/10.1037/a0016644

27.

McGrew

(2020). Learning to evaluate: An intervention in Civic Online Reasoning. Computers & Education, 145, 1–13. https://doi.org/10.1016/j.compedu.2019.103711

28.

McGrew

Breakstone

Ortega

Smith

Wineburg

(2018). Can students evaluate online sources? Learning from assessments of Civic Online Reasoning. Theory & Research in Social Education, 46(2), 165–193. https://doi.org/10.1080/00933104.2017.1416320

29.

McGrew

Smith

Breakstone

Ortega

Wineburg

(2019). Improving students’ web savvy: An intervention study. British Journal of Educational Psychology, 89(3), 485–500. https://doi.org/10.1111/bjep.12279

30.

Metzger

M. J.

Flanagin

A. J.

Medders

R. B.

(2010). Social and heuristic approaches to credibility evaluation online. Journal of Communication, 60(3), 413–439. https://doi.org/10.1111/j.1460-2466.2010.01488.x

31.

Mihailidis

Thevenin

(2013). Media literacy as a core competency for engaged citizenship in participatory democracy. American Behavioral Scientist, 57(11), 1611–1622. https://doi.org/10.1177/0002764213489015

32.

National Center for Education Statistics. (n.d.-a). Collecting race and ethnicity data from students and staff using the new categories. https://nces.ed.gov/ipeds/report-your-data/race-ethnicity-collecting-data-for-reporting-purposes

33.

National Center for Education Statistics. (n.d.-b). Information on the Common Core of Data (CCD). https://nces.ed.gov/ccd/files.asp

34.

National Center for Education Statistics. (n.d.-c). NAEP is the leader in large-scale assessment. https://nces.ed.gov/nationsreportcard/about/leader_assessments.aspx

35.

Pan

Hembrooke

Joachims

Lorigo

Gay

Granka

(2007). In Google we trust: Users’ decisions on rank, position, and relevance. Journal of Computer-Mediated Communication, 12(3), 801–823. https://doi.org/10.1111/j.1083-6101.2007.00351.x

36.

Reardon

S. F.

Robinson-Cimpian

J. P.

Weathers

E. S.

(2015). Patterns and trends in racial/ethnic and socioeconomic academic achievement gaps. In Ladd

H. F.

Goertz

M. E.

(Eds.), Handbook of research in education finance and policy (2nd ed., pp. 491–509). Routledge.

37.

Reardon

S. F.

Townsend

J. B.

(2018). SEG: Stata module to compute multiple-group diversity and segregation indices [Statistical software module]. EconPapers. https://econpapers.repec.org/software/bocbocode/s375001.htm

38.

Rideout

Robb

M. B.

(2019). The Common Sense census: Media use by tweens and teens, 2019. Common Sense Media. https://www.commonsensemedia.org/research/the-common-sense-census-media-use-by-tweens-and-teens-2019

39.

Simon

H. A.

(1971). Designing organizations for an information-rich world. In Greenberger

(Ed.), Computers, communications, and the public interest (pp. 38–72). Johns Hopkins Press. https://digitalcollections.library.cmu.edu/awweb/awarchive?type=file&item=33748

40.

Sullivan

M. C.

(2019). Why librarians can’t fight fake news. Journal of Librarianship and Information Science, 51(4), 1146–1156. https://doi.org/10.1177/0961000618764258

41.

Walraven

Brand-Gruwel

Boshuizen

H. P. A.

(2009). How students evaluate information and sources when searching the World Wide Web for information. Computers & Education, 52(1), 234–246. https://doi.org/10.1016/j.compedu.2008.08.003

42.

Warzel

(2021, February 18). Don’t go down the rabbit hole. The New York Times. https://www.nytimes.com/2021/02/18/opinion/fake-news-media-attention.html

43.

Wineburg

Breakstone

Ziv

Smith

(2020). Educating for misunderstanding: How approaches to teaching digital literacy make students susceptible to scammers, rogues, bad actors, and hate mongers (Working Paper A-21322). Stanford History Education Group. https://purl.stanford.edu/mf412bt5333

44.

Wineburg

McGrew

(2017). Lateral reading: Reading less and learning more when evaluating digital information (Working Paper No. 2017-A1). Stanford History Education Group. https://doi.org/10.2139/ssrn.3048994

45.

Wineburg

McGrew

(2019). Lateral reading and the nature of expertise: Reading less and learning more when evaluating digital information. Teachers College Record, 121(11), 1–40.

46.

Wineburg

McGrew

Breakstone

Ortega

(2016). Evaluating information: The cornerstone of Civic Online Reasoning (Working Paper No. 2016-A1). Stanford History Education Group. http://purl.stanford.edu/fv751yt5934

47.

Wineburg

Ziv

(2019, December 5). The meaninglessness of the .org domain. The New York Times. https://www.nytimes.com/2019/12/05/opinion/dot-org-domain.html

48.

Wineburg

Ziv

(2020, November 6). Why can’t a generation that grew up online spot the misinformation in front of them? Los Angeles Times. https://www.latimes.com/opinion/story/2020-11-06/colleges-students-recognize-misinformation

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.66 MB