Factual accuracy and the cultural context of science in popular media: Perspectives of media makers,middle school students,and university students on an entertainment television program

Media selection

To select a media program, criteria were developed to find shows representative of where scientist consultants have been utilized. Specifically, media programs were considered if they (1) contained a variety of science-related elements; (2) were from popular television as opposed to movies, since exposure to TV has generally been greater in the public (Rideout et al., 2010); (3) were not widely known to the general public to minimize the influence of prior conceptions, thereby focusing reactions on the actual media presented; (4) had “production value” similar to other shows on cable/network television so that poor quality did not become a distraction; and (5) had availability of the show creators, writers, and science consultants for interviews.

Given these criteria, the television drama Eleventh Hour was selected. This show was about a scientist (Dr Jacob Hood) who teams together with a Federal Bureau of Investigation (FBI) agent (Ms Rachel Young) to solve crimes of a scientific nature. The series was created by Stephen Gallagher and ran for one season on CBS during 2008–2009. The plotlines of Eleventh Hour explored contemporary scientific issues of public concern ranging across physical, life, and medical science. The show, as described by Gallagher, handled topics largely with “real” science as opposed to science fiction.

Once Eleventh Hour was selected, the series was narrowed down to clips to be used in focus groups and interviews. Through piloting, it was determined that three 15-minute segments would provide sufficient continuity and variety while also allowing focus groups to be completed within a reasonable time of about 2 hours. To select the clips, episodes were first eliminated that were not written or co-written by Gallagher. Segments were then identified that were rich in science-related aspects to provide participants a variety of elements to respond to. Specifically, the episodes were coded for science-related elements such as science in society, science professionals, laboratory spaces, science practices, and factual science content. The 15-minute intervals with the highest count for each episode were then identified. These are listed below. It is suggested that the reader watch these video segments to familiarize themselves with them:

Study participants

Two media makers participated in the study. Stephen Gallagher was the creator of Eleventh Hour and writer for many of the episodes. Dr Andrew Bazarko was the scientific consultant who collaborated with the writers, producers, and art directors for the show.

In total, 36 audience representatives also participated. One group consisted of middle school students (N = 17). Students of this age were selected because an interest in science and technical fields is thought to solidify around ages 11–14 (Tai et al., 2006). Understanding the perspectives of these students was therefore relevant to concerns about the pipeline of future scientists. (The comprehensibility of Eleventh Hour for this age group was later confirmed by the fluent discussions about plotlines, science, and characters observed among these participants.) The middle school participants were drawn from a county in California with 36% eligibility for free/reduced school meals, compared to 58% for California as a whole. This somewhat higher level of affluence was partially mitigated by recruiting subjects from a public youth center providing free after school programs suitable for low-income families. Study recruitment was voluntary with a US$25 gift certificate as an incentive. The final sample contained 12 females and five males. In terms of grades, six were in sixth grade, four were in seventh grade, and seven were in eighth grade. Ethnically, 13 were White and four were non-White.

The second group of audience participants were university students (N = 19). Specifically, non-science majors were recruited from an elite private university in California. This pool of participants was selected because (1) they were non-scientist laypersons like the participants in early studies demonstrating media effects on science perceptions (Gerbner et al., 1981) and (2) they were likely to become influential members of the public (e.g. teachers, policymakers) that might in turn affect the views of others. Understanding their perspectives was therefore relevant to understanding the engagement of the lay public with science. Participants were drawn from both the general undergraduate population and the teacher education master’s program. Study recruitment was voluntary with US$50 as an incentive. Among interested students, only non-science majors were invited to join the study. The final sample contained 13 females and six males. In terms of university program, 14 were undergraduates and five were teacher education master’s students. Ethnically, 10 were White and nine were non-White.

Data collection

The overall approach for this study followed an exploratory methodology. Focus groups were selected for their suitability to topics where the response space is not well-characterized (Morgan and Krueger, 1993). For the audience participants, three middle school focus groups and three university focus groups were conducted. Three focus groups have been found to be sufficient for achieving saturation—that is, the point when adding additional groups no longer generates substantial new insights (Krueger and Casey, 2009). Participants were organized into groups of six to eight, as recommended by Finch and Lewis (2003). Due to unexpected absences, one middle school group and one university group ended up with five participants.

The protocol for the focus groups followed the overall structure recommended by Finch and Lewis (2003). Participants first completed a survey on their background, academic interests, and familiarity with Eleventh Hour. No participants indicated more than a passing familiarity with the show. After watching the first clip, the following prompt was given: “What were some of the more memorable things related to science or scientists?” Participants were invited to respond open-endedly, first in writing to establish individual thoughts and then orally in the group. Although the question prompts were about science, participants were allowed to talk about any aspects. The process was then repeated for the second and third clips. The prompt for the second clip was, “In what ways was the show similar or different to real science or scientists?” The prompt for the third clip was, “What science- or scientist-related part was most convincing or unconvincing, and why?” Different prompts were used with each clip to avoid repetitiveness, keep participants engaged, and widen the range of responses. The researchers moderated the focus groups following the “process facilitation” method described by Milward (2006), with gradual retraction of guidance to the extent possible in each group.

For the media makers, semi-structured interviews were conducted with a set of advance questions. As with the audience participants, these questions introduced topics and then invited participants to respond open-endedly. The three clips were also made available prior to the interviews to aid in stimulating recall. The interview with Stephen Gallagher was conducted via email at his request. He was asked about the themes of Eleventh Hour, the process of working with a science consultant, and the science-/scientist-related elements intended for each clip. Gallagher provided over nine pages of written responses to the questions. The interview with Dr Andrew Bazarko was conducted by phone. He was asked about the themes of Eleventh Hour, the process of working as a consultant, his perspectives on the science-/scientist-related elements for each clip, and the aspects he actually provided feedback/input on. The interview with Bazarko lasted 41 minutes and was audio-recorded. The media makers were not offered compensation for their participation.

Analysis

Qualitative content analysis was conducted on the interviews and focus groups to determine the aspects that each group found salient. Specifically, science- and scientist-related codes were determined a priori from aspects identified in the literature as relevant to public knowledge and engagement with science. The codes were then modified emergently based on discussions with a second independent rater. The final codes can be found in Supplementary Appendix A (http://pus.sagepub.com/). All data were then coded at the level of meaning units: cohesive, distinctive ideas which are understandable on their own and preserve the psychological integrity of the notion being expressed (Ratner, 2002). An independent rater coded a random 20% of the units, establishing inter-rater reliability at 83%. Finally, the code frequencies were converted to word count—the number of words coded under each category—to better represent the duration/intensity of discussion on each topic.

Science Content/Practice

The category Science Content/Practice contained the following types of observations: (1) accuracy or inaccuracy of science content (Content Accuracy); (2) noticing science content but not its accuracy (Content Noticed); (3) accuracy or inaccuracy of science practices, as defined by the NGSS (Practice Accuracy); and (4) noticing science practices but not their accuracy (Practice Noticed). Table 1 shows the word count for each code, total count of all Science Content/Practice codes, and the total as a percentage of all codes.

OPEN IN VIEWER

Table 1.

Science Content/Practice word count, total count, and total as percentage of all codes.

Participant group	Content Accuracy	Content Noticed	Practice Accuracy	Practice Noticed	Total Science Content/Practice	Total as % of all codes
Show Creator/Writer	462	33	236	37	768	47
Teacher Ed Master’s	496	198	408	286	1388	23
Undergraduate 1	166	290	407	571	1434	41
Undergraduate 2	57	86	978	1020	2141	42
Middle School 1	458	1752	245	388	2843	74
Middle School 2	16	756	253	501	1526	53
Middle School 3	318	151	236	171	876	65
Science Consultant	415	117	278	320	1130	50
Actual Consult Topic	268	117	223	97	705	58
Total	2656	3500	3264	3391	12,811	50

Factual science Content was one aspect of the show broadly noted by audience members (46% of Science Content/Practice). Sometimes these remarks were about accuracy, such as this critique offered by the Teacher Ed Master’s Group: “They’re talking about the smallpox virus. And then they’re talking about an antibiotic. And I was um, curious why they were talking about. ’Cause antibiotics don’t kill viruses.” It also included approval about the depicted accuracy, such as a member from Middle School Group 3 observing, “Well, similar [to real science] was how people had to get vaccines to not get the disease.” Other Content observations were participants noticing the science without necessarily focusing on correctness. For instance, in Middle School Group 1, a student discussed the science of the DNA films from the cloning episode without necessarily commenting on the factual accuracy of the scene: “The papers things, that’s DNA. … You know what DNA is, right? It’s like [a] person’s genetics, like who they are.”

Although participants clearly noticed the science content of the show, there was a comparable focus on the practices of science (54% of Science Content/Practice). One example of a Practice comment was a critique in Undergraduate Group 1 on how the process of science was depicted: “I think the part that I didn’t like … it didn’t really show like how long of a process it is with research and stuff. … It doesn’t really show like the true nature of, I guess, research.” Other comments were directed at specific Practices. For instance, when the protagonist Dr Hood said that a negative result is just as important as a positive one, a member of Undergraduate Group 2 commented on this practice of Analyzing Data: “Only like someone who constantly does experiments as a scientist would say like: no, like having a negative as just as worthy as having a positive outcome, right?” As with Content, not all Practice-related discussions were about the accuracy of the science. About half (54%) of Practice codes were actually participants noticing science practices without necessarily focusing on their correctness.

Middle School Students also noticed science Practices, but not surprisingly, with less sophistication than university students. For instance, when asked about what they found realistic, a student in Middle School Group 2 said, “It’s similar to real science because he had to go look at the virus from the microscope. From uh, blood that a person died from, and [what] kind of virus was it.” Not all middle school students agreed with the practice depictions though: commenting on the hair drug tests, a student in Middle School Group 2 said, “I don’t think you get that much stuff and how many things she took … out of [that] piece of hair.”

For the Show Creator and Science Consultant of the program, Science Content/Practice was also an important focus. With the media makers, 72% of their Science Content/Practice codes were Accuracy comments. The Show Creator emphasized this explicitly: “The bedrock of the whole thing was that the science had to be actual. … If you want to do science fiction, do science fiction, but Eleventh Hour isn’t that.” Similarly, when asked about his role, the consultant said it was largely to ensure scientific correctness: “Mostly the technical stuff was … where I was supposed to provide feedback.” Not surprisingly, many of the codes for Actual Consult Topic - elements where the consultant actually provided feedback or input - were Science Content/Practice related (58%). Not all Science Content/Practice comments were about ensuring accuracy, however, such as when the Show Creator described the overall plot for the psychiatric episode: “this story hinged on the intersection of psychiatry and drug abuse, with a few forensic flourishes—using hair thickness to chart pregnancy, Reyes syndrome, etc.” As such, Noticed was also included as a coding category for media makers to reflect these science content mentions.

Scientific content and practices were clearly a focus for both the creator of the show and the scientific consultant, which matched the findings from the audience participant groups. Overall among the various participant groups, Science Content/Practice averaged 50% of all codes. This consistent prevalence of scientific content and practices has implications for the debate on the significance of science content in popular media—a finding which will be discussed later.

Image of Science

The category Image of Science contained the following types of observations: (1) visual representations of laboratories and equipment (Labs-Equipment); (2) the virtues or hazards of science for society, such as ethical concerns (Virtue-Ethics); (3) science issues that have entered public awareness mainly through news, events, or controversies (Public Issue); and (4) science portrayed as interesting or engaging (Interesting-Engaging). Table 2 shows the word count for each code, total count of all Image of Science codes, and the total as a percentage of all codes.

OPEN IN VIEWER

Table 2.

Image of Science word count, total count, and total as percentage of all codes.

Participant group	Labs/Equipment	Virtue/Ethics	Public Issue	Interesting/Engaging	Total Image of Science	Total as % of all codes
Show Creator/Writer	15	157	131	48	351	21
Teacher Ed Master’s	1251	1055	323	0	2629	43
Undergraduate 1	823	263	57	384	1527	43
Undergraduate 2	756	423	47	0	1226	24
Middle School 1	818	0	62	91	971	25
Middle School 2	687	70	0	54	811	28
Middle School 3	406	0	34	0	440	33
Science Consultant	466	220	62	0	748	33
Actual Consult Topic	466	47	0	0	513	42
Total	5688	2235	716	577	9216	33

Among audience participants, visual representations of laboratories and equipment were the most frequent observation in this category (62% of Image of Science). For instance, a student from Middle School Group 1 commented on the variety of lab depictions in the clips:

With all three of them, er, each one individually had different kind[s] of laboratory settings. Like, the first one they had, at the beginning with all the clone parts. And then in the second one [had] the people with smallpox in separate rooms. And then, in the third one they had the DNA sample area.

Sometimes, participants judged the labs based on depictions from other popular media. From the Teacher Ed Master’s Group, “It’s a funny thing, it’s reminding me of Cloverfield … where they’re all like at like the … underground quarantine facility?” And from Middle School Group 2, “I kind of thought the lab was like, realistic in a way. But … I’ve never really seen those kinds of labs except in like ET.” Not all participants displayed this pattern though. Some based their assessments on more reliable sources such as formal education, as one member of Undergraduate Group 1 indicated: “Those like, DNA breakdowns … seemed pretty similar to what I’d seen in real life. Because I took like a, like a simple biotech class when I was in high school.”

The other codes within Image of Science: Virtue-Ethics, Public Issue, and Interesting-Engaging appeared mainly among University participants. For instance, one member of the Teacher Ed Master’s Group addressed both Virtue-Ethics and Public Issue with the ethics of forced drug treatments in the psychiatric episode:

People who, you know, kind of I guess, [are] not able to give consent for medication. But they should have it. … It made me think about … discussions we’ve had about um, ADHD. And should kids, like should kids be medicated? … It’s like a really big topic within education, and just overall society.

And another from Undergraduate Group 1 expressed interest in the real-world chemistry behind the hair test in the psychiatric episode (Interesting-Engaging):

I remember the hair follicle test. I didn’t understand specifically what was the, like chemistry behind how it worked. … I understand how they didn’t want to bog in the details. But I would like to know that later on.

In contrast, among Middle School Students, the Image of Science category was mainly represented by Labs-Equipment. The other aspects occurred much less, together accounting for only 4% of all middle school codes. Students of this age group therefore focused on the more visual, tangible aspects of the Image of Science, namely, the laboratories and equipment.

For the Show Creator, few of his Image of Science discussions were about Labs-Equipment (4%). Instead, his larger focus was on how the show depicted the interaction between science and society. For instance, regarding Public Issues, he stated that Eleventh Hour was intended to address current scientific issues of public concern: “One of the basic principles of the show … was to deal with the contemporary facts of a subject like human cloning (i.e. it isn’t yet possible, and any attempt comes with a hidden cost in non-viable embryos).” And regarding Virtue-Ethics, he stated that Eleventh Hour “was to be a pro-science show.”

In contrast, the Science Consultant focused more on the depiction of laboratories and equipment, with 62% of his Image of Science codes being about Labs-Equipment. This trend was even more pronounced for the Actual Consult Topics: 91% of these Image of Science codes pertained to Labs-Equipment. The lack of codes other than Labs-Equipment from the science consultant was a notable divergence from the Show Creator as well as the University Students.

Image of Scientists

The category Image of Scientists contained observations about the scientists in the show regarding their (1) character and morality (Character); (2) intelligence and professional competence (Intellect); (3) lifestyle and social interactions (Status-Lifestyle); (4) social competence, such as being nerdy or awkward (Social Skill); (5) demographic traits such as ethnicity, gender, or age (Demographic); and (6) physical appearance (Appearance). Table 3 shows the word count for each code, total count of all Image of Scientists codes, and the total as a percentage of all codes.

OPEN IN VIEWER

Table 3.

Image of Scientists word count, total count, and total as percentage of all codes.

Participant group	Character	Intellect	Status/Lifestyle	Social Skill	Demographic	Appearance	Total Image of Scientists	Total as % of all codes
Show Creator/Writer	297	67	47	35	77	0	523	32
Teacher Ed Master’s	1231	91	242	142	106	253	2065	34
Undergraduate 1	119	63	402	0	0	0	584	17
Undergraduate 2	1047	298	258	92	81	20	1796	35
Middle School 1	0	28	0	0	0	0	28	1
Middle School 2	423	138	0	0	0	0	561	19
Middle School 3	30	0	0	0	0	0	30	2
Science Consultant	70	180	0	136	0	0	386	17
Actual Consult Topic	0	0	0	0	0	0	0	0
Total	3217	865	949	405	264	273	5973	17

In general, Image of Scientists was not mentioned frequently by middle school participants. Most of these codes instead occurred among university students. For instance, one member of the Teacher Ed Master’s Group noted that both negative and positive images of scientists were shown: “So like I saw the scientist being, sort of glamourous. … But then also like the cloning was presented as, I thought, very evil and scary.” University students also brought up issues such as the intelligence or professional competence of the scientist (Intellect). For example, from Undergraduate Group 2, “it’s like, literally everyone’s incompetent except for this one scientist that they had to bring in from DC.” They also noted Status-Lifestyle qualities and Social Skill portrayals, as in this exchange in the Teacher Ed Master’s Group:

Although less common, a few comments were also made about the demographics of the scientists, such as one participant in the Teacher Ed Master’s Group noting the gender of a scientist: “I thought it was good to include a woman scientist in there.” Least common of all were comments about the physical appearance of the scientists, such as one made by a member of Undergraduate Group 2 on the attractiveness of the main character: “I also like his eyes (laughing).”

For the Show Creator, Image of Scientists figured notably in his thinking (32% of his overall codes). In alignment with the university participants, many of these were Character comments. For instance, he saw the protagonist, Dr Hood, as a courageous and principled individual, drawing from real-life scientists who had shown similar fortitude in the face of political adversity:

In the UK we’d experienced several national situations that showed up politicians’ inability to deal with an immediate scientific reality. … Several outspoken non-government scientists braved political spin and stood their ground on the evidence, eventually to be proved right. … That’s how I saw Hood.

The Show Creator also emphasized other aspects such as Intellect, with the main character being the intellectual savior of the show: “the police don’t know what to make of it; into their midst strides someone who can.” And he included Status-Lifestyle, with the scientist holding a position of importance and authority: “Hood was younger, but that worked; FBI status gave him the necessary gravitas and authority for which there’s no British equivalent.” Indeed, the Show Creator commented on every aspect of Image of Scientists except for Appearance.

In contrast, the Science Consultant had less emphasis on the Image of Scientists, with 17% of his overall codes falling into this category. He did mention some aspects related to Intellect: “They wanted to convey that he was kind of a working level scientist. You know, using his smarts to … save the world.” He also mentioned some aspects related to Social Skill: “They try to make him out to be, I guess a little nerdy. But … it doesn’t, doesn’t really go that far. He’s not, he’s not like some caricature.” Overall though, fewer of the consultant’s comments were related to Image of Scientists than the Show Creator. This trend was even more pronounced when examining the Actual Consult Topics. Among these codes, none were about the Image of Scientists.

Overall trends

The overall pattern of responses can be seen in Table 4, which summarizes the percentage of responses by participant type. The table is shaded by quartile rank so that out of 70 total cells, the 25% (i.e. 17 cells) with the lowest rate of response are colored white, the next 25% shaded light gray, and so forth.

OPEN IN VIEWER

Table 4.

Rate of response for each code as a percentage of all coded words for each participant type.

Participant group	Science Content/Practice (%)					Image of Science (%)					Image of Scientists (%)
	Content Accuracy	Content Noticed	Practice Accuracy	Practice Noticed		Labs/Equipment	Virtue/Ethics	Public Issue	Interesting/Engaging		Character	Intellect	Status/Lifestyle	Social Skill	Demographic	Appearance
Show Creator/Writer	28	2	14	2		1	10	8	3		18	4	3	2	5	0
University Students	5	4	12	13		19	12	3	3		16	3	6	2	1	2
Middle School Students	10	33	9	13		24	1	1	2		6	2	0	0	0	0
Science Consultant	18	5	12	14		21	10	3	0		3	8	0	6	0	0
Actual Consult Topic	22	10	18	8		38	4	0	0		0	0	0	0	0	0

Shading by quartile rank among all cells.

1st quartile (0.0%)

2nd quartile (0.1%–3.1%)

3rd quartile (3.2%–9.9%)

4th quartile (10.0%+)

The Science Content/Practice category was the most consistently discussed among the participant types. Within this category, the generally high prevalence of Accuracy indicated that, overall, both audience members and media makers were attentive to the scientific veracity of the show. Moreover, many of these observations were about similar elements. For instance, regarding the microscope in the smallpox episode, the Science Consultant cautioned the writers: “[a] bit of feedback: … you know, viruses are pretty small. … You’d use an electron microscope to uh, to really get an image of the virus. [But] still he used an optical microscope.” Several audience participants similarly questioned the accuracy of the microscope depiction, such as the Teacher Ed Master’s Group: “[I] doubt like … what they show us under the microscope is really what scientists see, you know.”

Within Science Content/Practice, Noticed codes were also widespread, indicating that audience members and media makers were not solely fixated on factual accuracy. Again, participants of different types often commented on similar aspects. For example, media makers and several participant groups noticed how hair was used to test for pregnancy, yet did not necessarily discuss whether such depictions were accurate. Specifically, the Show Creator said the episode had “a few forensic flourishes [like] using hair thickness to chart pregnancy.” A participant from Middle School Group 1 noted, “she said that she was not pregnant … And then when they took the hair piece … it did say that she was pregnant.” And a member of Undergraduate Group 2 asked, “Did I hear correctly when [they] said like the hair is … someone who carried a baby to full term?” In reality, the thickness of hairs does not indicate pregnancy: perceived thickness increases due to more overall hair, but individual hairs—such as those tested in the episode—do not actually change in width (BabyMed, n.d.).

Within Science Content/Practice, participants also showed an emphasis on science Practice. This was reflected in a generally high prevalence of Practice discussions, although media makers and participants sometimes disagreed on how well it was depicted. For instance, the Science Consultant thought, “they want to kind of show … The scientific process and the fact that, you know, it sort of takes hard work to get to the answer.” In contrast, several audience participants were more critical, such as a member of Undergraduate Group 1 (quoted previously) who remarked, “it didn’t really show like how long of a process it is with research and stuff. … It doesn’t really show like the true nature of, I guess, research.” Despite differing opinions, however, the overall focus on science Practice was nonetheless notable. Therefore, not only was traditional science content a conspicuous feature to both audiences and media makers, but so were the more newly defined science practices from the NGSS. Overall then, media makers and audience members showed a broadly similar focus on Science Content/Practice.

In contrast, differences between the participant types were more evident in the Image of Science and Image of Scientists category. All participant groups except the Show Creator had frequent mentions of lab and equipment depictions, as would be expected with a visual medium. Yet, the other Image elements were generally discussed more by the Show Creator and University Students. For instance, the Show Creator and several University Students discussed the ethics of how science and scientists were portrayed: from the Show Creator, “I don’t think that a drama in which science goes wrong reflects badly on science. … I wanted to establish that this was science’s Good Guy coming in,” and from the Teacher Ed Master’s Group, “There was a … lot of portrayal of the evilness. And, sort of like a mad scientist type of thing. But then also … they were using it to solve problems.” In contrast, the Middle School Students, Science Consultant, and Actual Consult Topic groups discussed Image less frequently. Indeed, the last group—Actual Consult Topic—had no occurrences at all in the Image of Scientists category. This suggests that the Show Creator and University Students were more focused on how science and scientists were being portrayed compared to Middle School Students and the Science Consultant.