Effectiveness of a Non-Classroom News Media Literacy Intervention Among Different Undergraduate Populations

News Media Literacy Education

Media literacy is commonly defined as “the ability to access, analyze, evaluate, and create messages in a variety of forms,” a definition that is premised on building skills to become a more savvy media consumer and creator (Livingstone, 2004, p. 5). Media literacy education should promote an understanding of how various media operate and the relationship between media and audiences (Ashley et al., 2013; Fleming, 2014; Hobbs & Jensen, 2009). Most media literacy education focuses on building general skills and skills related to specific media domains, including news (Potter, 2013). Recognizing the importance of digital and media literacy in the 21st century, Hobbs (2010) notes that skills and competencies must be developed in formal classroom settings and informal settings outside of classrooms. In fact, media literacy has been recognized as a fundamental part of education by a number of educational stakeholders and is integrated into the Common Core State Standards for English and Language Arts education (Common Core State Standards, n.d.; Hobbs, 2010). Despite this recognition of the importance of media literacy, integrating media literacy education into K-12 schools remains a challenge as teacher training and resources are limited (Hobbs, 2010; Share, 2015).

“News media literacy” is designed to teach individuals how to apply media literacy skills (e.g., critical thinking, deconstructing arguments) to news consumption and highlights the democratic value of news (Ashley et al., 2013; Hobbs, 2010; Mihailidis, 2012). Most news media literacy education emphasizes the conditions and constraints under which news is produced and the responsibility of the audience to be critical thinkers when consuming news (Ashley et al., 2013; Craft, Maksl, & Ashley, 2013; Fleming, 2014; Mihailidis, 2012; Potter, 2013). Developing news media literacy is particularly important and challenging in our complex, ever-evolving media landscape. Hobbs (2010) contends that educators

[m]ust help people of all ages to learn skills that help them discriminate between high-quality information, marketing hype, and silly or harmful junk. We must raise the visibility and status of news and current events as powerful, engaging resources for both K-12 and lifelong learning while we acknowledge the challenges faced by journalism today and in the future. (p. xii)

This call to action reflects the growing need to incorporate news media literacy into formal and informal education.

Media literacy educators typically focus on developing curricula and classes to produce more media literate students (Craft et al., 2013; Mihailidis, 2012). However, less is known about how to promote media literacy in everyday life. Research suggests that exposure to short media literacy messages outside the classroom can boost perceptions of media literacy and mitigate biased processing of neutral media content (Vraga, Tully, et al., 2012; Vraga et al., 2009). Theoretically, however, short news media literacy messages should function primarily by reminding people of the value of the press and existing media literacy attitudes (Tully & Vraga, in press; Vraga & Tully, 2015). For example, 64% of Americans say they prefer getting unbiased, nonpartisan news—news that reflects the standards promoted by journalism and media educators (Kovach & Rosensteil, 2007; Pew Research Center for the People and the Press, 2012). News literacy messages are designed to reinforce the value of a diverse, fair press and promote critical thinking skills among audiences. Therefore, a media literacy PSA focusing on the role of a press in democratic society and the need for critical consumption should tap into these broadly held values.

However, audiences may not naturally apply these beliefs to their media consumption, particularly younger audiences who have become accustomed to consuming social media “news” and who often face information overload (Hobbs, 2010). Exposure to the media literacy PSA should make these news media literacy attitudes more accessible, priming individuals to use these beliefs in evaluating subsequent media content (Price & Tewksbury, 1997; Zaller, 1992). Thus, the news media literacy PSA should function like a reinforcement message during an inoculation campaign, motivating individuals to apply media literacy values in processing the political news talk show (Pfau et al., 1992). We hypothesize the following:

However, we expect that the PSA will be more effective when it reinforces already-existing media literacy values and knowledge. To uncover the relationship between in-class media literacy education and non-classroom messages, we test the PSA with students currently exposed to media literacy education in their classroom versus students enrolled in an interpersonal communication course (see below for detailed course descriptions). Priming is more successful when it resonates with existing beliefs and attitudes in one’s memory, as well as when people are frequently and recently exposed to similar messages (Price & Tewksbury, 1997; Zaller, 1992). Similarly, reinforcement messages are designed to bolster previously received messages, rather than introduce new concepts (Pfau & Burgoon, 1988; Pfau et al., 1992). Therefore, for the students currently enrolled in media education and effects courses, the PSA should serve in this reinforcement role, reminding them of the importance and applicability of news media literacy to their news consumption (Vraga & Tully, 2015). For students in the non-media class, the message may be more unfamiliar and less resonant, and thus have less ability to affect processing of the news content.

Method

To test our hypotheses, we performed an experiment embedded in a web-based survey in the spring of 2014. We recruited participants from undergraduate students at three American universities, who received extra credit for their participation. ¹ Of roughly 1,550 students invited to participate via email, 831 completed the survey (53.6% response rate). After cleaning the data, we were left with N = 305 valid responses. ²

Participants for this study were recruited from two types of courses: non-media versus media education classes. For the non-media course, we recruited participants from one of two general education courses required of all undergraduate students at George Mason University, located on the East Coast. This course is titled Interpersonal and Group Interactions and was selected for its minimal focus on the news media (n = 137). For the media education classes, we recruited participants from two universities located in the Midwest. At the time these courses were pre-requisites for admission into the journalism and mass communication majors at each university and are titled Media Uses and Effects (University of Iowa, n = 86) and Introduction to Mass Communication (University of Wisconsin–Madison, n = 82). They both focus on the role of the news and other media in U.S. society and pay special attention to “develop[ing] the ability to make better use of media as thoughtful consumers,” according to the syllabus for Media Uses and Effects (p. 1). Therefore, the media education courses should include more media literate students than the non-media course, which we hypothesize will condition their response to the PSA.

For this study, we experimentally manipulated whether participants were exposed to a news media literacy PSA before viewing a political program with a neutral host using a between-subjects experimental design. ³ The PSA reportedly came from “The Media Literacy Coalition” and emphasized the role of the news to inform citizens and fairly represent diverse viewpoints and the role of citizens to critically engage with news content (Ashley et al., 2013; Craft et al., 2013; Vraga & Tully, 2015; Vraga, Tully, et al., 2012). The script was recorded, and an experienced video editor assembled the voice-over with professional stock footage of newsrooms and news consumers to create the video. The PSA appeared as an advertisement before the main video, including a countdown to when the “video would return” for external validity. ⁴

After the PSA, participants watched a simulated political talk show video on the issue of immigration reform. In the program, the host remained politically neutral while moderating the debate between two partisan guests (one conservative, one liberal). Immigration reform was selected as a divisive and important political issue, which has remained among the top priorities for Americans from 2009 to 2014 (Pew Research Center for the People and the Press, 2014) while also dividing Democrats and Republicans in their perceptions of immigrants and their preferences for reform (Pew Research Center for the People and the Press, 2013).

The political talk show was created by researchers to maintain experimental precision (Carr, Barnidge, Rogers, Wise, & Vraga, 2011). Professional actors played the role of a host and two partisan guests, and a television studio with a green screen was used to tape a mock program, similar to “talking head” programming. A professional director and experienced video editor assisted in the development and production of the videos, maintaining quality and realism.

Measures

Results

Differences Between Courses

Before testing our hypotheses, we examined differences between participants from the non-media course versus the media education courses. A series of t tests revealed several differences: Students in the media courses were significantly more Republican, more strongly partisan, lower in conflict avoidance, and more favorable of restrictive policies toward illegal immigrants (see Table 1). In addition, a chi-square test demonstrates that the non-media course had a significantly higher proportion of students whose parents were born outside of the United States. These results demonstrate a limitation of our design: Differences exist between universities in their student population and between course enrollments. Therefore, in all subsequent analyses, we include extensive controls for individual predispositions and attitudes that could influence our results. ⁷

OPEN IN VIEWER

Table 1.

Demographics Differences Between Students in Media Versus Non-Media Courses.

Variable	Test statistic	M for non-media course	M for media course
t test
Party identification (Democrat)	2.31*	4.63	4.26
Political ideology (Conservative)	−0.48	3.56	3.63
Partisan strength	−1.98*	1.10	1.29
Conflict avoidance	2.64**	4.05	3.69
Immigration attitude (restrictive)	−2.72**	3.77	4.25
Age	1.36	19.78	19.30
Year in school	−0.05	1.58	1.59
Chi-square test
Gender (Female)	1.75	60.6%	67.9%
Race (White)	33.34***	47.4%	79.2%
Parents birthplace (U.S.)	37.70***	53.3%	83.9%

Note. For party identification, conflict avoidance, and immigration attitudes, values ranged from 1 to 7. For partisan strength, values ranged from 0 to 3, and for year in school, values ranged from 1 to 5.

*

p < .05. **p < .01. ***p < .001.

Hypothesis Testing

To test our hypotheses, we ran a series of two-way ANCOVAs examining the effects of exposure to the PSA and course enrollment, controlling for host role in the political program, race (White or Other), parents’ birthplace (U.S. or other), gender, age, ⁸ immigration attitude, party affiliation, political ideology, partisan strength, year in school, conflict avoidance, and week of participation in the academic semester. We first examine whether exposure to the PSA affected perceived media literacy and whether its effects were stronger for students in a media education course. We find support for H1a: There is a significant main effect (F = 5.05, p = .03, η² _p = .020) of exposure to the PSA, with those who saw the PSA reporting higher perceived media literacy (M = 5.18) than those who did not see the PSA (M = 4.85). However, in contrast to H4a, no interaction between PSA exposure and course enrollment (F = .01, p = .94) emerges, nor is there a main effect of course enrollment on perceived media literacy (F = 2.04, p = .15), although the results are directionally consistent with our expectations (non-media course: M = 4.89; media course: M = 5.14).

For actual news media literacy, we find no significant effect of exposure to the PSA (F = .01, p = .92) nor a significant interaction between course enrollment and PSA exposure (F = 1.00, p = .32), failing to support both H1b and H4b. Instead, we see a significant main effect of course enrollment (F = 6.94, p = .01, η² _p = .027), with students in the non-media course reporting lower news media literacy (M = 5.44) than students in media courses (M = 5.75).

This pattern is repeated for general media trust: A main effect of course enrollment emerges (F = 28.44, p = .00, η² _p = .101), with students in the media education courses reporting significantly higher media trust (M = 4.32) than students in the non-media course (M = 3.43). In contrast to both H2 and H5, media trust was not affected by exposure to the PSA (main effect: F = 2.36, p = .13; interaction: F = .43, p = .51). Overall, we find no support for our argument that the PSA would more strongly influence the news media literacy attitudes of students enrolled in a media course versus a non-media course. However, particularly for perceived and actual news media literacy, a ceiling effect may limit our ability to find effects.

Turning to processing of the media content that appeared after the PSA, we test whether exposure to the PSA directly boosted perceptions of program credibility, as proposed by H3a. This hypothesis is not supported (F = .11, p = .74). However, the data support H6a, as a significant interaction between course enrollment and exposure to the PSA emerges (F = 6.88, p = .01, η² _p = .027). The pairwise comparisons support our hypothesis: The PSA had a significant impact on credibility ratings (p = .03) among students in the media education courses, boosting perceptions of credibility (see Figure 1). Meanwhile, this effect is not significant among the students in the non-media course (p = .13), although directionally the results suggest that PSA decreased credibility ratings of the program compared with the no-PSA group.

OPEN IN VIEWER

Figure 1.

Interaction between media literacy PSA and course enrollment on program credibility.

We see a similar pattern of results for self-reported attention to the program. Exposure to the PSA did not boost attention to the program overall, as predicted by H3b (F = 1.08, p = .30). Instead, the effects of the PSA on attention depend on course enrollment (F = 5.99, p = .02, η² _p = .023), as predicted by H6b. For students in the media course, exposure to the PSA significantly increased attention to the program (p = .01) compared with the control condition (see Figure 2), an effect that is not significant for students in the non-media course (p = .35).

OPEN IN VIEWER

Figure 2.

Interaction between media literacy PSA and course enrollment on attention to the program.

Discussion

Findings from this research suggest that the ability of news media literacy messages to influence students’ beliefs is conditioned by their preexisting media literacy education. The news media literacy PSA boosted perceived media literacy among all audiences, but the PSA worked differently among students enrolled in media courses compared with students enrolled in a non-media course in their evaluations of the subsequent political news content.

Specifically, the PSA encouraged students enrolled in media-focused courses to more critically evaluate the political talk show. As a result, students viewing the PSA reported paying more attention to the program and rated the program—which upheld norms of journalistic neutrality and balance—as more credible. Meanwhile, these effects are not evident among students enrolled in a non-media course.

However, in contrast to our expectations, the PSA did not have stronger effects on media literacy attitudes or knowledge among students in media-focused courses. As expected, students in media-focused courses reported higher levels of media trust and news media literacy and directionally appeared to have higher perceived media literacy, which may limit our ability to perceive effects. Another possibility is while the news media literacy PSA primed existing media literacy attitudes and highlighted their applicability to judging the subsequent media content (Price & Tewksbury, 1997; Zaller, 1992), it did not offer enough content to enhance these beliefs. Finally, a third possibility—especially for the null finding regarding perceived media literacy—is that the very definition of what it means to be “media literate” differs for these groups, and as such, cannot be directly compared. More effort is needed to understand how individuals define “media literacy,” particularly dependent on characteristics like previous education.

However, although it is promising that the news media literacy PSA enhanced critical consumption among students in media education courses, it is disappointing that it had no effect on students in non-media courses. One of the goals of moving media literacy messages outside the classroom is to reach a broader population, but this study suggests that brief messages may not be effective among those without a background in media literacy. Of course, our media literacy PSA offers only one type of message, one with a relatively informative tone. A message that focuses more on different components of news media literacy, such as its value for democracy rather than news constraints, or that uses a different style, such as a humorous or imploring tone, may be more effective, particularly among younger audiences. More research is needed on the types of news media literacy interventions that are effective outside the classroom and on the types of skills and training that should occur within the classroom to enhance media literacy education and its application to news consumption and information processing.

This study has limitations that must be addressed. Most notably, the study suffers from a lack of random assignment to courses. Students have a choice when selecting which courses to take; therefore, our results might be skewed by students’ preferences in course selection. It is possible that students with an interest in media are more likely to enroll in the media-focused courses and may enter the courses with higher levels of media awareness and literacy. However, this limitation does not undermine our fundamental finding that our news media literacy PSA worked better among students who are already engaged with media literacy in their classes.

Second, although we used extensive demographic controls, it is possible that other differences exist between students in the media education versus non-media courses that contribute to their response to the PSA. We propose that future research needs to continue to test the individual and social characteristics that contribute to responses to media literacy messages.

Finally, this experiment tested only one message and its effects on processing one news clip. As noted above, more research is needed into different types of effective media literacy messages paired with different types of media content (Tully & Vraga, in press). Similarly, the issue of immigration addressed in the political program may not have been particularly interesting to students, especially at universities less involved in the immigration debate. ⁹ A more involving issue may have further limited the effectiveness of the PSA or may have exacerbated differences between groups in its effectiveness.

Ultimately, our research suggests that there is a relationship between classroom education and non-classroom media literacy interventions. As such, we need to consider how exposure to media literacy messages in and outside of the classroom may work together to encourage critical consumption of news and other media content. If students receive media literacy education in their classrooms, small “injections” of media literacy in their lives could help these individuals apply critical thinking skills more regularly, thus extending the effectiveness of their education.