Characteristics of Facebook’s third-party medical fact checkers

Introduction

Social media websites, such as Facebook, have made an effort to monitor and label news stories and op-eds that could be false or misleading. ¹ HealthFeedback.org is one of the dozens of organizations that review multiple types of media articles for Facebook, but HealthFeedback.org focuses specifically on health and medical articles. They solicit two to four reviewers per article, who makes a determination as to which articles are misleading. Reviewers apply to review articles identified by Facebook, as opposed to reviewers for scientific articles who are invited, and reviewers do not disclose conflict of interest. Fact checking for misinformation relies on unbiased and objective reviewer selection and processes to balance the protection of scientific dialogue with the removal of misinformation from Facebook. Hence, we sought to evaluate fact checkers for news stories and op-eds that circulate on Facebook.

Methods

On 3 October 2021, we included and searched all articles on HealthFeedback.org, since the website inception. We abstracted data on names of reviewers and other quoted individuals cited in the article, their professional titles, the article title they reviewed, the number of articles reviewed, and year of review(s). We searched Twitter on 10 March 2021, to see whether the reviewers and quoted individuals had an account and noted the number of Twitter followers. Data are presented in frequencies and are stratified by pre-Covid-19 and post-Covid-19 (January 2020). We tested differences in means with a t-test, medians with Wilcoxon rank-sum test, and differences in categorical variables with Fisher exact test. As data were publicly available, institutional review board approval was not required.

Results

We found that of the 19 articles reviewed by HealthFeedback.org, seven (37%) articles were fact-checked since January 2020. The average number of reviewers per article prior to January 2020 was 3.6, and the average number of reviewers per article since then was 2.7 (p = 0.03). Prior to January 2020, 43% of reviewers (n = 19) were by assistant, associate, or full professors, 14% of reviewers (n = 6) were by fellows or postdocs, and 18% were done by researchers (n = 8). Since January 2020, 63% were assistant, associate, or full professors (n = 12), 21% of reviews were done by fellows or postdocs (n = 4), and one (5%) was done by a researcher.

Prior to January 2020, 64% of reviewers had active Twitter accounts, and the average follower count was 6986 (median 1012; Table 1). Since then, 79% of reviewers have active Twitter accounts, and the average follower count was 42,000 (median 10,000).

OPEN IN VIEWER

Table 1.

Twitter activity among HealthFeedback.org fact checkers and quoted individuals.

	Reviewers since January 2020 (n = 19)	Reviewers prior to January 2020 (n = 44)	p value a
Presence of Twitter account	15 (79%)	28 (64%)	0.38
Average followers	42,000	6986	0.06
Median followers	10,000	1012	0.007
Average number of reviewers per article	3.7	2.7	0.03
	Other quoted individuals since January 2020 (n = 16)	Other quoted individuals prior to January 2020 (n = 2)	p value a
Presence of Twitter acct	14 (88%)	2 (100%)	1.0
Average followers	161,123	314	0.0007
Median followers	153,050	314	0.04

^a

Fisher exact test p value for “presence of Twitter account” independent t-test p value for “average followers” and “average number of reviewers”, and Wilcoxon rank sum test for “median followers”.

The average number of followers on Twitter for quoted individuals was 161,123 (since January 2020) versus 314 (prior to January 2020; p = 0.0007; Table 1). The median number of followers on Twitter for quoted individuals was 153,050 (since January 2020) versus 314 (prior to January 2020; p = 0.04). There was at least one case since January 2020 where the reviewer had already been critical of the topic on Twitter, prior to reviewing the article. We did not find any prior to January 2020.

Discussion

We found that reviewers who were selected to review articles for Facebook and those whose quotes are included have large followings on Twitter and in at least one case, the reviewer had publicly stated their opinion of the article prior to the review. These findings are concerning in light of survey results from another study, which reported that social media was the biggest perceived source of Covid-19 misinformation among researchers, clinicians, and academics. ² Our findings are limited by sample sizes, but we included all articles and reviewers on HealthFeedback.org during our study time frame.

The spread of misinformation on Facebook raises concerns. Current fact-checking processes appear to be strongly associated with large Twitter followings. Further, reviewers appear to be selected based upon a priori viewpoints that reach a large audience. It is unclear how articles are selected for review and how reviewers are chosen since the exact process for determining specific reviewers is not included on the website or in the articles themselves. Greater transparency in the process of determining misinformation is needed.