The case of #arseniclife: Blogs and Twitter in informal peer review

1. Introduction

Recent high-profile retractions from the prestigious scientific journals Nature (Obokata et al., 2014a; 2014b) and Science (LaCour and Green, 2014; McNutt, 2015), and concerns over the ethics of a recent study on emotional contagion using the social networking site Facebook (Kramer et al., 2014), have increased discussions among the scientific community of post-publication peer review enabled by Web-2.0 technologies (Ghosh et al., 2012; Shema, 2014) and among broader audiences (White, 2014). These technologies have led to new ways of communicating scientific findings (Brossard, 2013), potentially facilitating informal means of legitimizing scientific knowledge. The forum for scientific disputes is increasingly shared by traditional, closed pre-publication models and open post-publication ones enabled by technological progresses. While online spaces for post-publication review (e.g. PubPeer) are proliferating, we illustrate this broader shift from pre- to post-publication review using the “arsenic life” controversy as a case study.

The controversy surrounds an article published on 2 December 2010 in Sciencexpress, the online ahead-of-print medium of the journal Science. The paper was co-authored by 12 scientists led by Felisa Wolfe-Simon of the United States National Aeronautics and Space Administration (NASA). The authors claimed to have discovered and cultured bacteria capable of replacing phosphorus with arsenic in its major physiological building blocks (Wolfe-Simon et al., 2010). If validated, this discovery would have significant implications; phosphorus is thought to be necessary for all living organisms, while arsenic is generally toxic (Reaves et al., 2012). But, after 14 months of informal post-publication discussion, the hypothesis was refuted. Although Wolfe-Simon and her colleagues claimed to grow the bacterium, GFAJ-1, on a medium supplemented with arsenate, a microbiologist from the University of British Columbia, Rosemary Redfield (2011a), was unable to replicate the growth. Redfield was a candid critic of the arsenic life hypothesis and was central to the events. In addition to dissemination via social media, the development of the debate occurred on blogs. Altmetrics, which are data complementary to citation-based metrics, of the Science article show it received the most mentions on blogs (117 mentions), followed by tweets (52 mentions; Altmetric.com, 2016). Altmetrics measure the online activity and mentions of a particular article from sources such as blogs, Wikipedia, and online news. While the numbers for the Science article are not particularly high, they refer only to the paper published in June 2011. These altmetrics do not include mentions of the December 2010 version published in Sciencexpress.

Using this case, we explore how online blogs and Twitter affect the practice of peer review and its salience among wider audiences. To reenact the case study, we present major events chronologically by drawing on selected blog posts. Using automated sentiment analysis of a census of publicly available tweets, we analyze the volume and compare skeptical and non-skeptical tones of discussion on Twitter. Hereafter, we use the term “tone” to refer to our analysis of skeptical and non-skeptical discussion on Twitter. Though our quantitative approach is one of many possible approaches to analyzing Twitter discussion of the arsenic life controversy, researchers have used sentiment analysis to assess opinions in a range of contexts (Bansal et al., 2008; e.g. Chmiel et al., 2011; Conrad and Schilder, 2007; for more on sentiment analysis, see Pang and Lee, 2008). The advantage of this approach is its ability to perform human-based computer-aided content analysis that allows for the examination of a very large volume of online content from within the United States and abroad. We also explore reposted Twitter content and the most prolific users to examine how and by whom content was shared.

2. Method

We present the arsenic life debate chronologically over a period of 16 months (28 November 2010 to 31 March 2012) during which most of the controversy played out. Through inductive thematic analysis of online science blogs, primarily those written by prolific bloggers on the issue (Ivan Oransky, Carl Zimmer, and Rosemary Redfield), we construct a timeline and identify claims made by various actors at each stage of the controversy. Oransky (2010b) is the executive editor of Reuters Health and his blog, Embargo Watch, is his way of “examining trends in embargoes and how they [affect] news coverage [of science].” Zimmer, a blogger and popular science writer for prominent news organizations such as The New York Times, often guest posts on various science blogs and maintains his own for Discover Magazine. Furthermore, we used NASA’s website and Redfield’s blog, RRResearch. Science, the publisher of the original scientific article, was central to the debate and Nature covered the story consistently over the 16 months, therefore the websites of these two scientific journals were also included.

Additionally, we gauged the volume and tone of the controversy on the Twittersphere, relying on sentiment analysis software from the social media monitoring company, Crimson Hexagon. The software, ForSight, uses an algorithm described in Hopkins and King (2010) to analyze all publicly available tweets. Specifically, ForSight’s algorithm uses nonparametric statistical modeling and “does not require classifying individual documents into categories and then aggregating; it estimates the aggregate proportions directly” (Hopkins and King, 2010: 237). Human coders train the algorithm to recognize patterns of words representative of specific concepts by classifying random samples of the tweets manually.

In order to train the algorithm, a series of publicly available tweets were randomly drawn using a keyword search. ¹ These sampled tweets, which included retweets, were then analyzed by trained human coders and classified into specific categories. In this case, we trained the software to recognize posts that discussed the controversy in skeptical and non-skeptical tones. Specifically, tweets that expressed reservations about the arsenic life discovery or the Science publication were coded as skeptical. Posts that lacked skepticism towards the discovery or article were classified as non-skeptical. In addition, coders trained the software to recognize posts as off-topic if they failed to fit into either category (Table 1). Off-topic tweets were excluded from subsequent analysis. Once the program was sufficiently trained, which requires each of the pre-defined categories to have at least 20 exemplar posts, we ran the software, which drew all relevant online content identified by the keyword search and assigned each post to a relevant trained category.

OPEN IN VIEWER

Table 1.

Examples of human-coded tweets.

Category	Example of tweet
Skeptical	Scientists unable to replicate findings in lab environment. RT @newscientist: Arsenic life does not exist after all
Non-skeptical	Bacteria With Arsenic-Based DNA Hint of Alien Life Form on Earth
Off-topic	Love that! Makes me think of all those arsenic based beauty products

While sentiment analysis has limitations, this method allows us to analyze a census of publicly available tweets instead of having to rely on a sample. In general, human-trained computer-aided sentiment analysis is limited by the lack of a standard set of linguistic rules applied in training the algorithm (Gunter et al., 2014). In our specific case, we consistently trained the algorithm using tweets that were exclusively skeptical or non-skeptical. In other words, tweets selected to train the algorithm either contained only skepticism or lacked it entirely, therefore alleviating the potential problem of a lack of a standard set of linguistic rules posed in other contexts. We were able to do this due to the length limitation (140 characters) of Twitter posts.

Additionally, we examined how information traveled across the Twittersphere by analyzing retweets about the arsenic life issue. We also identified the most prolific Twitter authors (those who contributed the most tweets to the discussion) base on their Klout score. A Klout score, a number between 1 and 100, is based on an author’s influence and ability to reach users across social media (Edwards et al., 2013; Stevenson, 2012). This allowed us to identify the 50 “most influential authors” for this issue. Manual coding was used to categorize authors as journalists and/or bloggers, as well as experts or non-experts. Journalists and bloggers were defined, respectively, as individuals who write articles for established media organizations and those who maintain personal or professional blogs containing original content, which allowed some individuals to be categorized as both. If the authors either did not have a PhD or did not have one in a field pertinent to the controversy, they were coded as non-experts.

3. Results and discussion

We obtained 35,860 posts over the study time period. Of these, 26,078 (72.7%) were relevant; 48.2% (12,581) of which were skeptical. Of the relevant posts, 9,845 (37.8%) were retweets. The average number of skeptical and non-skeptical opinions over the entire time period were 25.68 (standard deviation (SD) = 106.63) and 27.54 (SD = 339.87), respectively. We present our results chronologically, emphasizing periods when the volume or tone of discussion changed dramatically.

Like most scientific issues in the media, the arsenic life debate followed an issue attention cycle (Brossard et al., 2004; Nisbet et al., 2003) prompted by NASA’s press release on 29 November 2010: “NASA (2010b) will hold a news conference at 2 p.m. EST on Thursday, Dec. 2, to discuss an astrobiology finding that will impact the search for evidence of extraterrestrial life.” Various online news and blogs began to speculate about the discovery, with headlines such as “‘Life as we don’t know it could prove existence of aliens” (Alleyne, 2010).

The news conference and second press release (NASA, 2010a) coincided with the early release of the peer reviewed paper in Sciencexpress on 2 December 2010, led by Wolfe-Simon, a NASA-funded astrobiologist. The paper presented an isolated bacterium, GFAJ-1, able to substitute arsenic for phosphorus in its genetic material (Wolfe-Simon et al., 2010). This claim was revolutionary and the publication of the article in Science added legitimacy to a controversial hypothesis. On 2 December 2010, opinions on Twitter peaked at 6,621 (Figure 1), many expressing excitement over the findings. Examples of tweets during this peak included “Woot! NASA finds new life form. #ThisChangesEverything! http://bvl.co/12” (2 December 2010), and “#NASA has discovered an alien life-form here on earth: http://gizmodo.com/5704158/nasa-finds-new-life #thischangeseverything #ET” (2 December 2010). Although the hashtag “#arseniclife” was used, many were contextualized by other hashtags, such as “#thischangeseverything,” affecting others’ interpretations of the message (Tewksbury and Scheufele, 2009).

OPEN IN VIEWER

Figure 1.

Timeline of selected events with the proportion of skeptical sentiment (bold grey line) and volume of relevant opinions (shaded area) on Twitter.

Amid the excitement, some experts expressed skepticism. On 4 December 2010, microbiologist Rosemary Redfield (2010a) ( @RosieRedfield ) posted a critique of the Science article on her blog, RRResearch, ² concluding the study was “[l]ots of flim-flam, but [contains] very little reliable information … If these data were presented by a PhD student at their committee meeting, I’d send them back to the bench to do more cleanup and controls.” Other scientists, posting on blogs, echoed Redfield’s concerns. For example, Alex Bradley (2010), a microbiology postdoctoral researcher from Harvard University, wrote, “In the midst of all the excitement, one thing has been overlooked: The claim is almost certainly wrong.”

On 5 December 2010, Wolfe-Simon (2010) responded to the blogged criticisms in a tweet: “Discussion about scientific details MUST be within a scientific venue so that we can come back to the public with a unified understanding.” She was not alone in her insistence that discussion be conducted through a formal peer review process; NASA’s senior public affairs officer, Dwayne Brown, also insisted the debate should occur in scientific publications (Oransky, 2010a).

While mainstream news organizations began their coverage of the arsenic life story on the same day as the NASA press conference, focus quickly shifted from the Science article to the online expert discussion surrounding the controversy, quoting blog posts (Vestergaard, 2016). For example, on 6 December 2010, a news article quoted Redfield’s blog (CBC News, 2010): “Redfield dissected Wolfe-Simon’s molecular biology and microbiology methods and results in detail on her blog, RRResearch, garnering tens of thousands of hits and dozens of comments from other scientists.” Similar to other scientific controversies in which media have taken on the rhetorical role of scientific journals (Brossard, 2009), Redfield received comments from other scientists on her blog, which acted as a scientific venue. Additionally, there is an aspect of this type of informal review that is impossible with the formal process; the transparency of open, informal critique allowed a variety of interested audiences to observe and potentially participate in the process.

The blogosphere immediately produced critiques of the study and NASA’s handling of the debate (e.g. Oransky, 2010a; Zimmer, 2010b). Most referred to the initial press release as misleading as it hinted at the discovery of extraterrestrial life. In response to the enigmatic press release (29 November 2010) and NASA’s desire to have criticisms discussed within the confines of formal peer review, on 8 December 2010, Zimmer (2010a) quoted biologist and fellow blogger Jonathan Eisen of the University of California, Davis, saying, “[NASA] carried out science by press release and press conference […] they are now hypocritical if they say that the only response should be in the scientific literature.”

Although Brown’s statement may have been an attempt to preserve NASA’s reputation, in highlighting the prestige of Science and holding formal pre-publication peer review over informal post-publication review, the online environment as a space for scientific discussion was implicitly delegitimized. In his blog, Oransky (2010a) emphasized Brown’s attempt to discredit blogs and social media by contrasting them to “credible media organizations.” In response to this insistence on conducting additional peer review through official channels, Redfield prepared a letter for submission to Science. She posted a draft of the letter on her blog on 8 December 2010, inviting comments (Redfield, 2010b), and encouraged other scientists to submit letters to Science.

Between the months of January and May 2011, Twitter audiences appeared to lose interest in the issue (Figure 1). Similarly, little content related to arsenic life was seen in the blogs we examined. While the proportion of skeptical opinions fluctuated during this time, a minima was observed in March 2011. During this time, tweets included mentions of the article as a reference for others such as “Fascinating. MT @carlzimmer [1/2]: My report on the possible discovery of a new domain of life (no, not @arseniclife) http:is.gd/RTYmN2” (21 March 2011), as well as retweets of older, non-skeptical posts. Online activity increased in the summer with the publication of eight critiques of the original article and Wolfe-Simon et al.’s (2011) response (Benner, 2011; Borhani, 2011; Cotner and Hall, 2011; Csabai and Szathmary, 2011; Foster, 2011; Oehler, 2011; Redfield, 2011a; Schoepp-Cothenet et al., 2011). However, the critiques did not include follow-up experiments with GFAJ-1. In a blog post on 27 May 2011, Redfield (2011b) realized it was her responsibility to follow-up on GFAJ-1. She wrote,

I’ve been saying that researchers shouldn’t invest the time and resources needed to test Wolfe-Simon et al.’s claims, because of the vanishingly small probability that they are correct. But I’m having second thoughts, because the most important claims can, I think, be very easily tested.

The elapsed time between publication of the original article and Redfield’s follow-up studies was relatively short (Knoepfler, 2015). Redfield began working with GFAJ-1 less than 6 months after Wolfe-Simon’s article was published in Sciencexpress. It is likely that the combined roles of scientists, online audiences (including bloggers), and journalists, particularly in the Twittersphere, kept the issue prominent and was partly responsible for the time frame within which this controversy was resolved.

In addition to reconstructing the timeline of events, we examined the 50 most influential Twitter authors involved in the arsenic life discussion. Our results showed many of them were associated with media. Thirty-two percent of these authors were journalists and 56% were bloggers, many of whom were not experts in the scientific field in question (46% were categorized as non-experts). These results imply that experts actively discussing the controversy on Twitter attracted the attention of journalists and bloggers, many of whom were non-experts, such as science journalist Carl Zimmer (@carlzimmer), who was the fourth most prolific author in the debate. In turn, these journalists and bloggers on Twitter helped reach broader audiences.

Twitter has been called the “microphone of the masses” (Murthy, 2011), and retweets can spread information rapidly in a cascade (Morales et al., 2014). In some cases, scholars have identified Twitter as a major factor in popular mobilizations (e.g. Shirky, 2011). The relatively large number of retweets about the arsenic life controversy (9,845) is indicative of the reach of the content and Twitter’s role in keeping the issue relevant. Interested audiences were able to stay abreast of and express reactions to developments in the debate simply by following “#arseniclife.” Interestingly, the 50 most influential authors had Klout scores that ranged from 83 to 98. The average Klout score on Twitter is approximately 40, indicating these users had relatively high influence (Leary, 2013).

In June 2011, the volume of posts on Twitter reached a second peak at 3,631, with skeptical voices reaching to over 99% in mid-June. For example, “From the shadows to the spotlight to the dustbin—the rise and fall of GFAJ-1 http://bit.ly/lVLOR6 by @RosieRedfield #arseniclife” (16 June 2011) was posted during this time. During summer 2011, Twitter audiences were largely skeptical of the arsenic life hypothesis. Skepticism also dominated during the latter half of 2011; approximately, 75% of posts expressed skepticism over the original findings. While our data do not permit us to do so, it would be interesting to test Mazur’s (1981) hypothesis with similar data. Mazur’s hypothesis posits a relationship between the quantity of media coverage and negative public opinion on scientific and technological issues. In other words, as the amount of media coverage of an issue increases, so does negative public reaction. While there is a significant relationship between the weekly average volume of relevant posts and that of skeptical opinions (Pearson’s r = .96, p ⩽ .001, N = 491), we do not have a representative sample of the volume of online media coverage of the arsenic life debate with which to test this hypothesis. Although previous studies have not found support for Mazur’s hypothesis (Gutteling, 2005), this may be a fruitful area of future research to examine in the context of social media.

Over the course of the debate, we observed the dynamics of the skepticism on Twitter. The NASA announcement and Sciencexpress article initiated excitement and optimistic speculation. As critiques emerged, tweets began to echo skepticism in the scientific community. As Redfield continued to blog about her lack of progress with GFAJ-1, Twitter sentiment was dominated by even clearer skepticism towards the original study. On 30 January 2012, Redfield (2012) published a blog post entitled “The #arseniclife manuscript has been submitted!” On 1 February 2012, in keeping with informal channels of review and open science, Redfield made her research publicly available on arXiv.org, an open access online journal hosted by Cornell University Library. While the prevalence of open access journals is growing, the reaction of the scientific community to publishing in these outlets is still under investigation (e.g. Davis and Walters, 2011; Schroter and Tite, 2006). In choosing to make her research open access and blogging about her progress, Redfield opened the doors of science to broader audiences, allowing a glimpse into a typically opaque process.

After 14 months of open, informal post-publication review in the blogosphere, the arsenic life hypothesis was refuted. In March and April 2012, the volume of posts decreased and online discussion about arsenic life waned. Unlike recent retractions in which data were falsified (e.g. LaCour and Green, 2014), these are valid data refuted by other scientists through non-conventional processes of peer review. Therefore, the original paper by Wolfe-Simon et al. remains in Science. The follow-up experiments on GFAJ-1 conducted by Redfield and her research group were published in July 2012 (Reaves et al., 2012).

4. Conclusion

We used the arsenic life debate as a case study to highlight how the online environment can provide a forum for open and informal post-publication review of a contested scientific issue. In addition to discussion on blogs, audiences on Twitter kept the issue relevant through real-time interactions and sharing. The issue was brought into a broader sphere of audiences who were not directly involved in the controversy, informing them on the subject and providing them with the opportunity to comment on the process.

Our findings can be extended to other instances. For example, in April 2015, researchers at the University of Chicago turned to Twitter to highlight errors in a December 2014 genomics study. One of the co-authors of the critique, Yoav Gilad, said he “took to social media to highlight his work, which might otherwise have been overlooked” (Woolston, 2015: 1). Another example involves the recent retraction of a study in Science (McNutt, 2015) on gay canvassers changing opinions on same-sex marriage in the United States (LaCour and Green, 2014). This retraction was initiated by Oransky (2015), who broke the story in a blog post after receiving a tip via Twitter.

Our results indicate that interactive online communication technologies can enable members in the broader scientific community to perform the role of journal reviewers to legitimize scientific information after it has advanced through formal review channels. In addition, a variety of audiences can attend to scientific controversies through these technologies and observe an informal process of post-publication peer review. Criticizing each other’s research is not novel among scientists. What is interesting in this case is that peer review occurred in the blogosphere and captured the attention of Twitter audiences. Recent research shows online news media can rely on blog posts made salient through social media as sources (Vestergaard, 2016), which underscores journalists’ tendency to use Twitter as a tool for tracking news (Ahmad, 2010; Farhi, 2009; Hermida, 2010a, 2010b; Willnat and Weaver, 2014).

Blogs are forums in which audiences can interact by sharing knowledge, delivering feedback, and developing ideas (Birch and Weitkamp, 2010). However, their reach is limited in terms of public engagement (Scheufele, 2011). Yet, with the aid of social media, scientists’ blogs can be sufficiently influential to set scientific agendas. The popularity of Twitter and its unique technology of categorizing messages with hashtags make issues salient, facilitating spontaneous, real-time sharing of information and interaction among engaged individuals who are otherwise beyond the reach of blogs or traditional media. In this way, communication technologies are creating opportunities to “re-engage the public with science, rather than to patronizingly call for their understanding” (Miah, 2005: 415).

What practical implications of open informal post-publication review are highlighted by the arsenic life case? Are these implications good for science? These questions affect a broad range of scholarly processes and actors, including how academic work is assessed prior to publication, scholarly reputation, and copyright considerations. For decades, science has operated via an opaque process where gatekeepers limit information flow. Open informal processes of review promote transparency, encourage attention to heterogeneity within scientific communities, and facilitate integration of a wide range of voices that may extend beyond traditional expertise to benefit scientific communities.

But, the value of informal post-publication peer review is not in opposition to formal review processes. Instead, interactive online media can be complementary, encouraging transparent reviews that provide non-expert audiences insight into certifying scientific knowledge in various disciplines. In this case, back-and-forth information flow was observed between formal and informal processes, emphasizing the permeable boundaries between scientific journals and mass media that have been stressed elsewhere (Brossard, 2012).

However, even open post-publication review has its flaws. Recent legal actions over online scientific reviews have brought attention to some of its issues, including concerns of libel and incivility (Jump, 2014). Nonetheless, it may encourage audiences’ interest in science, familiarize them with the process of research, and perhaps lead to future involvement in science.

This case study is not without limitations. First, while we examined the volume of tweets and skeptical and non-skeptical tones expressed in the arsenic life debate, we did not assess the quality or themes of the tweets themselves. It would be interesting to further explore the role of Twitter as a source of critical reviewer feedback by examining the quality of links and arguments presented in tweets.

Second, our analysis of tweets may have failed to identify some nuanced elements, such as sarcasm or irony. For example, the tweet “RT @settostun: As arsenic completes nucleic acid, you complete me. #candyheartreject #arseniclife” (14 February 2012) would be difficult for the algorithm, trained for our purposes, to categorize as skeptical or non-skeptical. Future studies should attempt to include more nuanced analyses of sentiments, such as sarcasm, which have long presented challenges for content analysis.

Despite these limitations, our study is not restricted by sample size; the analysis software enabled us to consider a census of tweets about the arsenic life issue over the course of 16 months. Additionally, while many studies of political issues have employed automated content analysis (Gunter et al., 2014), ours is one of the few that applies this technique to approach vast quantities of digital text related to a specific scientific controversy.

It remains to be seen whether academic journals will embrace a more transparent and open peer review process, whatever form it may take. Of course, this is dependent on other influences, such as academic norms, publishers’ economic incentives, and commercial constraints. At the very least, the arsenic life case study has exemplified how traditional peer review has the potential to benefit from open post-publication debates online.