How fact-checkers delimit their scope of practices and use sources: Comparing professional and partisan practitioners

Abstract

Fact-checkers, as their names suggest, are supposedly agents who check the validity of “facts.” But in reality, how do fact-checkers delimit the scope of their practices? What sources do they use to establish the truth and falsity of the examined materials? Do the practices of different types of fact-checkers vary in these aspects? This study examines how professional and partisan fact-checkers deal with facticity during a protest movement in Hong Kong. The content analysis shows that partisan fact-checkers, when compared to professional fact-checkers, are less likely to restrict themselves to debunking factual claims, and they are more likely to provide no source information for the materials used. Posts addressing “misleading claims,” as opposed to factual claims, are where partisan inclinations are more clearly exhibited. Moreover, partisan fact-checkers used government information in ways consistent with their political predilection. Implications of the findings on our understanding of the fact-checking enterprise are discussed.

Keywords

Fact-checking professional versus partisan fact-checkers facticity information source protest movements

The proliferation of fake news has led to the emergence of a global fact-checking movement aiming at combating political misinformation and disinformation (Amazeen et al., 2018; Nyhan and Reifler, 2010). Some studies found that fact-checking can effectively correct misbeliefs and make people’s grasp of public affairs more accurate (Porter et al., 2018; Walter et al., 2019; but see Nieminen and Sankari, 2021). Other scholars pointed toward the inconsistency in fact-checking practices among practitioners (Lim, 2018; Uscinski, 2015). Some noted the presence of partisan fact-checkers who aimed at advancing the interests of one’s side (Graves, 2016). Without dismissing the potential contribution of the fact-checking movement, more research on the performances of different types of fact-checkers under varying circumstances is needed.

This study aims at contributing to the literature on fact-checking by examining how partisan and professional fact-checkers deal with the fundamental question of facticity. Two interrelated issues are at stake. First, fact-checkers need to demarcate their scope of operation by determining what kinds of claims to be verified. Implicitly or explicitly, fact-checkers have to adopt a certain conceptualization of what constitutes “false information” or “fake news.” Second, fact-checkers need to employ methods to establish their conclusions. That is, once a fact-checker decides to verify a claim, it needs to find ways to establish the claim’s truth or falsity. Therefore, we can ask: what are fact-checkers checking? How do they establish their conclusion? How do professional and partisan fact-checkers handle the issue of facticity? Or, how does fact-checkers’ approach to facticity signify their degrees of professionalism and partisanship?

This article examines such questions by examining the fact-checking scene in Hong Kong during the Anti-Extradition Law Amendment Bill (Anti-ELAB) Movement. The movement provided a context within which fact-checking can become the means of mobilization and counter-mobilization, though even partisan fact-checkers may exhibit different degrees of professionalism in their practices (Feng et al., 2021). This article illustrates how the manifestation of partisanship is related to the extent to which fact-checkers restrict themselves to factual verification and to fact-checkers’ employment of sources in establishing their conclusion about the examined materials.

The following further discusses fact-checkers’ scope of practice and use of sources. We then discuss the notions of professional and partisan fact-checkers. Hypotheses and research questions are set up after elaborating the context. Presentation of methods and findings follows. Implications of the findings are discussed at the end.

The scope of fact-checking

It is generally understood that fact-checking agencies are mostly established for combating misinformation, disinformation, and/or fake news. These latter notions overlap with each other, but are also distinctive. Disinformation and misinformation refer to false information shared with and without the intent to manipulate, respectively (Tandoc et al., 2018), whereas fake news refers to “false or misleading content presented as news and communicated in formats spanning spoken, written, printed, electronic, and digital communication” (Higdon, 2020). Debunking misinformation is important for maintaining meaningful public deliberation, for reasonable societal conversations could be extremely difficult if people cannot agree on facts.

The practices of fact-checking, however, are not uncontroversial. Uscinski and Butler (2013) noted the presence of questionable fact-checking practices, such as attempting to (in)validate predictions even though the predicted events are yet to occur. More fundamentally, they criticized the naïve realism underlying the enterprise, i.e. the assumptions that facts are unambiguous and cannot be subject to interpretation. In response, Amazeen (2015) argued that bad performances by some fact-checkers should not lead one to dismiss the fact-checking enterprise. Besides, while the significance of facts often depends on contexts and interpretations, unambiguous cases of deception exist. Fact-checking has an important role in fighting these practices of deception.

Amazeen’s (2015) argument about the possibility of identifying factual claims that can be objectively (in)validated rhymes with conventional journalism’s emphasis on the fact-opinion distinction. Facts are “assertions about the world open to independent validation,” and they “stand beyond the distorting influences of any individual’s personal preferences” (Schudson, 1978: 5). Fact-checkers informed by journalistic professionalism can be expected to adhere to the fact-opinion distinction and focus primarily on “checkable facts” (Nieminen and Sankari, 2021; Walter and Salovich, 2021), though practitioners might acknowledge the difficulty to draw a sharp fact-opinion distinction in all cases (Graves, 2017).

However, in popular discourses, the term fake news is often appropriated by political actors, who may use it to describe commentaries they disagree with even though those commentaries may not contain false information (Tandoc, 2019). In debates about anti-fake news legislation, authoritarian governments are often criticized for defining the term broadly and vaguely so that relevant laws could facilitate censorship and consolidation of power (Neo, 2020). Meanwhile, there can indeed be cases where the media’s framing of an event can be subjected to reasonable criticisms, and propagandistic materials may be criticized for presenting one-sided narratives (Tandoc et al., 2018). Although questionable framing or propagandistic narratives may not contain false information, the label “fake news” may nonetheless be applied when political actors offer such criticisms.

For this study, the important point is that the practices of fact-checking agencies may be based on different definitions of fake news or disinformation. If political actors could use the term fake news to encompass a wide range of questionable claims, fact-checkers may also do so. Based on the above discussion and for the sake of parsimony, this study focuses on the distinction between (in)validating facts and critiquing misleading claims. The former is what an agent does when it identifies and verifies a checkable fact by referring to unambiguous and objective evidence. The latter is what an agent does when it tries to criticize a claim for being “wrong” or unjustified due not to factual errors but to questionable framing, narration, contextualization of events, and/or argumentation. The empirical question is the extent to which fact-checkers restrict themselves to handling facts that can be unambiguously (in)validated and the extent they venture into handling misleading claims.

Establishing facticity and the use of sources

No matter how broad or narrow their scope is, fact-checkers need to find ways to establish the truth or falsity of the examined claims. Establishing facticity has always been a central issue for journalism. As Anderson and Schudson (2019) argued, the professional jurisdiction of journalism comes from the process of knowledge-making and information collection instead of the expertise of esoteric fields. Journalists have their own methods and routines of establishing facts. Tuchman’s (1978) classic analysis illustrated how journalists weaved together pieces of observations and evidence, common-sense understandings of the world, and information provided by authorities into a web of facticity so that the materials would mutually validate each other.

A news report’s credibility rests on the materials provided by reporters and the sources of those materials. A trustworthy report has to elucidate its choice of sources to persuade readers. Yet scholars have long noted the problematic aspects of journalists’ sourcing practices, especially the tendency to rely on official sources (Carlson, 2017; Fishman, 1980). Although reliance on governmental institutions is inevitable where such institutions hold the monopoly of authoritative information (Kwanda and Lin, 2020), such reliance may reinforce the power holders’ domination in public knowledge production (Van Leuven et al., 2018). The media should retain a degree of skepticism toward the validity, fairness, and comprehensiveness of official information, especially when the official institutions themselves are involved in a political conflict or controversy.

Similarly, sourcing is an important and potentially problematic aspect of fact-checking practice. The Chicago Guide to Fact-checking (Borel, 2016) spent a chapter on sourcing, discussing a range of issues such as anonymous sources handling, the need to confirm with multiple sources, and the need to differentiate the credibility of media organizations. The International Fact Checking Network (IFCN) also highlighted the issue of sources in its codes of principles. Specifically, the IFCN (2021) stated that fact-checkers should identify the source of all significant evidence used and provide relevant links for readers. Fact-checkers should use the best available primary evidence, check multiple sources, and identify the relevant interests of the sources being used whenever possible.

Not surprisingly, some analyses have examined fact-checkers’ sourcing practices. Rogerson’s (2013) study of fact-checkers in England, Canada and the US found varying levels of rigor in fact-checking practices, including whether the fact-checkers would trace the source of the original quote being examined and/or provide evaluation of the source. But there was no quantitative data about the extent to which various practices were adopted. Haigh et al.’s (2018) study of StopFake, a fact-checking project conducted by the independent Media Reforms Center in Ukraine, documented the extent to which the agency employed various kinds of sources. But without a comparison target, the import of the numerical findings remains unclear. This study, in contrast, would document and compare the extent to which different types of sources are used by different types of fact-checkers.

Professional and partisan fact-checkers

The present study focuses on the distinction between professional and partisan fact-checkers. Conceptually, professional fact-checkers embrace the core tenets of journalistic professionalism, including informational accuracy, neutrality, and objectivity (Graves, 2018). They posit themselves as autonomous from political influences. The rise of professional fact-checkers can be seen as the journalism industry’s attempt to reclaim their gatekeeping power undermined by technological and political changes (Waisbord, 2018). In contrast, fact-checking practices can also aim at advancing partisan interests (Graves, 2016). Here, partisanship does not necessarily refer to supporting specific political parties; it can refer to supporting one side in a society’s political cleavage. Partisan fact-checkers typically engage in selective fact-checking to defend their side and criticize their opponents.

In terms of practices, professional fact-checkers share many traits with conventional news outlets. They formalize their methodologies, institutionalize their nonpartisanship, and engage in international collaboration (Graves and Konieczna, 2015; Lowrey, 2017). The IFCN’s code of principles emphasizes compliance to nonpartisanship and fairness, transparency of sourcing, funding and methodology, and open and honest correction policy (IFCN, 2021). To be a verified signatory of IFCN, a fact-checker must explain how their claim selection is non-partisan and get assessed on the provided evidence. For example, FactCheck.org, affiliated with the University of Pennsylvania, maintains its fairness by scrutinizing claims from both sides of the US political spectrum and featuring analysis from experts with opposing ideologies (Graves, 2016).

In contrast, partisan fact-checkers typically engage in selective agenda-setting, i.e. they primarily debunk fake news disfavoring one’s own side. An example in the US is the conservative fact-checker NewsBusters dedicated to combating reports from the liberal media (Graves, 2016). An earlier study (Feng et al., 2021) of two partisan fact-checkers in Hong Kong showed that the claims examined by the two fact-checkers seldom overlapped. Both fact-checkers exhibited discernible partisan biases through their selection of claims to (in)validate. The pro-government fact-checker mainly verified claims targeting the government and the police, spending less effort debunking claims unfavorable toward the movement and its supporters. In contrast, the pro-movement fact-checker debunked anti-movement claims more frequently, spending less effort debunking claims unfavorable toward the government. The study also found that partisan fact-checkers would offer explicit criticisms selectively to highlight the problematic character of fake news disfavoring one’s side.

However, the professional versus partisan distinction is only conceptual. The above discussions highlight how the prototypical professional and partisan fact-checkers could behave, but professionalism and partisanship are not mutually exclusive. While the hallmark of partisanship is a tendency for a fact-checker to primarily debunk disinformation disfavoring one side, one-sidedness is a matter of degree. It is possible for fact-checkers who have a strong commitment to methodological rigor and the idea of public service to exhibit some degree of favoritism to one side of a political divide (Feng et al., 2021). It is also possible for fact-checkers established to defend one’s side and criticize the opponents to adhere to at least part of IFCN’s code of principles, including transparency of operation and open correction policy. In fact, some partisan fact-checkers may see the employment of rigorous methods and valid sources as important for establishing their credibility and ensuring the effectiveness of their work. Therefore, during controversies and/or protest movements, some fact-checkers may exhibit clear political inclinations yet still offer useful fact-checking services (Khaldarova and Pantti, 2016; Lee and Chan, 2018).

Based on the conceptual discussions, this research examines the performance of a professional fact-checker and two partisan fact-checkers (which may nonetheless exhibit varying degrees of professionalism), focusing on their scope of fact-checking practices and use and treatment of sources. The specific research questions and hypotheses are stated after a brief introduction of the case background.

Case background, hypotheses and research questions

Much has been written about the decline of press freedom in Hong Kong in the past two decades. The Chinese state exercised informal control of the Hong Kong press through ownership co-optation, influence over major advertisers, and occasional criticisms of what they regarded as problematic reporting (Lee, 2018). However, for a long period of time, the media system did not completely succumb to political pressure because of the counteracting forces of journalistic professionalism and market pressure (Chan and Lee, 2008). The media scene exhibited a significant degree of political parallelism. Hong Kong has been a hybrid regime with only part of the legislature being directly elected. The major political cleavage existed between the pro-democracy camp and the pro-government camp. News media outlets can also be placed on the pro-democracy versus pro-government spectrum (So, 2016). Similarly, fact-checkers in Hong Kong may be partisan in the sense of showing an affinity toward one side of the political cleavage.

The empirical study focuses on the performance of fact-checkers amidst the Anti-ELAB Movement in Hong Kong. The movement originated as an opposition against the government’s plan of amending the Fugitive Ordinance to allow the extradition of suspects to mainland China. Protester demands later expanded to include democratization and means to address police abuse of power (Lee et al., 2020). Protest activities continued for more than half a year before being subdued by the COVID-19 outbreak in early 2020.

Misinformation and rumors abound during the movement. For instance, a prominent anti-movement rumor claimed that female protesters were deceived into offering sex to male protesters. The claim was hard to verify, but fact-checkers repeatedly noted how the claim was transmitted online together with photos taken from pornographic materials. On the other side, an example of anti-government fake news is a faked video of a police officer throwing a petrol bomb at protesters.

Several fact-checking agencies were active in combating misinformation during the movement. This study focuses on AFP Fact Check Hong Kong (AFPHK), Kauyim, and TrueNews. AFPHK is a regional branch of Agence France-Presse. It is expected to behave as a professional fact-checker given its affiliation with a traditional news organization, certification by IFCN, and lack of ties to local groups.

In comparison, Kauyim and TrueNews can be considered partisan. Kauyim, literally meaning “seeking verification,” was a Facebook page established in September 2014 by five volunteers. The page regularly publishes fact-checking reports and self-identifies as a promoter of digital media literacy. Yet Kauyim also vowed to fight against “the camp attempting to fool the public” in its page introduction. This indicates its self-awareness of being opposed to a “camp” and, in the local context, hints at its critical attitude toward the political establishment. TrueNews is a Facebook page established in September 2019. Its background remains obscure as it does not provide information about its affiliation, a statement of purpose, or contact information. But the list of “similar pages” shown by Facebook contains many other pro-establishment social media outlets. TrueNews could be seen as belonging to the pro-government online media enclave that is part of the Chinese state’s “united front” (Cheng, 2020).

With a content analysis, Feng et al. (2021) noticed that the two fact-checkers' inclinations were evident in selective agenda-setting, explicit criticisms toward the examined claims, and criticisms extending beyond fact-checking. TrueNews exhibited a relative bias toward the pro-government camp by selectively debunking more fake news unfavorable toward the government and offering more explicit criticisms when debunking fake news that undermined the government. Kauyim exhibited the opposite tendencies. Meanwhile, compared to TrueNews, Kauyim exhibited a relatively higher degree of professionalism in its practice. For example, it debunked a significant amount of fake news unfavorable toward the government, i.e. it exhibited a relatively lower degree of selective agenda-setting. It also defended the professional media more proactively. Moreover, Kauyim’s reports have been cited by professional local media such as Ming Pao and overseas media such as BBC. It also won recognition from overseas fact-checkers such as Taiwan FactCheck Center, a verified signatory of IFCN. We take into account the difference between Kauyim and TrueNews in their relative degree of professionalism when setting up the hypotheses below.

A cursory reading of the posts on Kauyim and TrueNews suggests that their fact-checking techniques included tracing the claims’ origin, cross-checking with official accounts, and reverse image searching. AFPHK utilized more resource-intensive methods, such as geolocation tracking and direct contact with claim-makers. Compared to AFPHK, Kauyim and TrueNews do not share visible affiliations with local media, and both seemingly lack the human resources to engage in complex investigations. This difference in resources needs to be kept in mind when understanding the findings. In any case, this study will systematically examine how Kauyim, TrueNews, and AFPHK differed in delimiting their scope of practice and using sources. We expect professional fact-checkers to restrict themselves to handling verifiable factual claims, whereas a partisan practitioner may expand the scope to criticize misleading claims (Graves, 2016). Given our characterization of the three fact-checkers, H1 is stated as:

H1: AFPHK is the most likely to restrict itself to debunking factual inaccuracies, followed by Kayuyim, and TrueNews is the least likely to stick to debunking factual inaccuracies.

Scope of practice has implications on the extent to which partisan fact-checkers exhibit their partisanship. We expect partisan fact checkers’ political biases to be clearer when they criticize misleading claims because doing so allows or even requires the application of judgment and interpretations that are not entirely objective. H2 is therefore stated as:

H2: Partisan fact-checkers’ political inclination is more conspicuous when they engage in criticizing misleading claims (as opposed to debunking factual inaccuracies).

The second issue is the use of sources. IFCN (2021) demands fact-checkers to indicate the sources of materials based on which they declare a claim’s truth or falsity. We pose an open-ended question about the fact-checkers’ use of different types of sources, and we expect the professional fact-checker to be less likely to provide no source information:

Q1: What kinds of sources do AFPHK, Kauyim and TrueNews employ in their fact-checking practices?

H3: AFPHK is the least likely to provide no source information for their fact-checking materials, followed by Kauyim, and TrueNews is the most likely to provide no source information for their fact-checking materials.

We are also interested in whether the use of sources would constrain the manifestation of partisan biases. On the one hand, transparency on the use of sources may have such constraining effects. In this case, partisan fact checkers’ political biases would be more evident among reports without source information. But on the other hand, a partisan fact-checker may be highly selective in its use of sources. Hence the presence of sources may not undermine partisan biases. We therefore pose an open question here:

Q2: Would a partisan fact-checker’s partisan bias be more conspicuous in reports without source attribution?

Moreover, we pay attention to the extent to which fact-checkers use official sources. Use of government information is not necessarily problematic, since government agencies indeed hold important facts. But we can expect pro-government fact-checkers to be the most reliant on government sources, whereas fact-checkers who lean toward the political opposition should be the least reliant on government sources, with professional fact-checkers somewhere in-between. To obtain additional insights into how fact-checkers handle sources, we further explore the qualitative differences in how the fact-checkers make use of government information. H4 and Q3 are stated as follows:

H4: TrueNews is the most likely to use government information in fact-checking, followed by AFPHK, and Kauyim uses government information least frequently.

Q3: How do AFPHK, Kauyim, and TrueNews make use of government information in their fact-checking process?

Method and data

To address the hypotheses and research questions, this study content analyzed posts pertinent to the Anti-ELAB Movement published by the three fact-checkers between 9 June 2019 (generally regarded as the starting date of the movement) and 30 September 2020 (because legal prosecution and political consumption associated with the movement persisted after protest activities subsided in mid-2020). The authors manually screened and collected the posts and all relevant materials such as texts, photos, and video clips. Not all posts were fact-checking reports. Some were mere reposting of information or commentaries. The dataset does not include these materials. A total of 717 posts were collected, with 87 from AFPHK, 273 from Kauyim, and 357 from TrueNews.

Textual analysis based on a close and iterative reading of the posts using government information was conducted to address Q3. The authors identified recurrent frames and patterns of source usage inductively through a preliminary reading (Strauss and Corbin, 1998). This procedure allowed researchers to be sensitive to the actual discourses in the sample. For Q1, Q2, and the hypotheses, a coding scheme was developed for a quantitative content analysis. The coding was built on the preliminary observations and conceptual considerations (e.g. the core distinction between factual and misleading claims). Before formal coding, the first two authors conducted several rounds of trials and then established inter-coder reliability through coding 70 randomly selected posts. Among the various indices that take into account agreement by chance, Scott’s pi was used since all variables were nominal and only coded by two researchers. All variables included in the analysis had a Scott’s pi higher than 0.80. A word list of the key variables was developed to aid consistency in coding.

The key variables are defined as follows:

Target of fake news. During a political conflict, false information could be used to delegitimize a protest movement (Lee and Chan, 2018) or employed as “weapons of the weak” to pressure the state (Scott, 1985). Hence the posts were coded as: (a) verifying/criticizing claims targeting the pro-movement camp (i.e. the movement, protesters, and other movement supporters), (b) verifying/criticizing claims targeting the government or its supporters (i.e. the government, the police, and other movement opponents), and (c) verifying/criticizing claims targeting neither side. This variable is the basis for establishing whether the fact-checkers focused on verifying/criticizing claims targeting one side in the conflict.

Nature of the claim being examined. Central to the issue of fact-checking scope is the distinction between claims that are factually incorrect and claims that are misleading. What are coded here are the claims as presented in the posts. A post highlighting the fakeness of visual materials used in fake news is an example of a post that debunks factual inaccuracies. For an example of a post that criticizes a misleading claim, in October 2019, a student media reported a police officer shooting a citizen. TrueNews pointed out that the officer was trying to deescalate the situation and opened fire only after he was assaulted. TrueNews’ report did not point to factual errors. It criticized the media for constructing a misleading narrative that portrays the police as opening fire unnecessarily. Based on the distinction, the coders register each post as either (a) (in)validating facts or (b) criticizing misleading claims. A post can invalidate facts and criticize misleading claims simultaneously. For parsimony, a post is coded as (a) if it highlights factual inaccuracies or fakeness. The simultaneous presence of criticisms against misleading claims is treated as of secondary importance.

Source of materials used in fact-checking. For the present analysis, the authors classified sources of materials into four types: (1) “original source,” i.e. tracing the original materials to show a claim’s falsity or the fakeness of the materials constituting the disinformation (e.g. doctored photographs); (2) government source, i.e. the use of official records or responses as evidence; (3) media sources, which were further differentiated into pro-government, pro-movement, and other media (i.e. outlets generally understood as politically neutral) following studies on the Hong Kong media (Lee, 2018; So, 2016) and the authors’ understanding of the media scene during the Anti-ELAB protests, aided by relevant literature (e.g. foreign media were treated as neutral as there is no basis for claiming them as pro-movement or pro-government, whereas student media were treated as pro-movement, see Luqiu, 2021); (4) others, which include sources that do not belong to the above categories, such as companies and specific individuals. A post can draw upon materials from multiple types of sources. Besides, posts without providing source information were registered as such for tackling H3 and Q2.

Analysis and findings

Delimiting the scope of fact-checking

H1 is concerned with the extent to which the fact-checkers restricted themselves to verifying facts. As Table 1 shows, all 87 posts of AFPHK dealt with factual inaccuracies. Among the 273 posts by Kauyim, 255 (93.4%) dealt with factual inaccuracies and 18 (6.6%) posts criticized claims for being misleading. In contrast, among the 357 posts published by TrueNews, only 205 (57.4%) dealt with factual inaccuracies, whereas 152 (42.6%) criticized claims for being misleading. The results are consistent with H1. AFPHK adhered strictly to debunking factual inaccuracies or fake information. Kauyim was relatively closer to AFPHK. TrueNews stood out as substantially more likely to venture into criticizing misleading claims.

Table 1.

Nature of the claims being examined.

	AFPHK	Kauyim	TrueNews
Factual inaccuracies	100.0 (87)	93.4 (255)	57.4 (205)
Misleading claims	0.0 (0)	6.6 (18)	42.6 (152)
(χ²/Cramer’s V = 0.444, p < 0.001)

Figures are column percentages. Bracketed figures denote numbers of posts.

H2 predicts that the political predilection of the partisan fact-checkers would be more conspicuous when criticizing misleading claims. Table 2 focuses only on the two partisan fact-checkers. Kauyim is supposed to be relatively supportive toward the protest movement. Indeed, it tended to examine claims disfavoring the movement instead of claims disfavoring the government. Consistent with expectation, this tendency was stronger among the posts that criticize misleading claims (55.6%) as compared to the posts that debunk factual inaccuracies (47.5%), though the difference is not statistically significant (χ² = 4.143, p = 0.13).

Table 2.

Relative bias in agenda-setting by fact-checkers and type of claims.

	Examined claims unfavorable toward
	The movement side	The government side	Neither
Kauyim (χ² = 4.14, p = 0.13)
Factual inaccuracies	47.5 (121)	32.5 (83)	20.0 (51)
Misleading claims	55.6 (10)	11.1 (2)	33.3 (6)
TrueNews (χ² = 6.20, p < 0.05)
Factual inaccuracies	7.3 (15)	73.2 (150)	19.5 (40)
Misleading claims	2.0 (3)	81.6 (124)	16.4 (25)

Figures are row percentages. Bracketed numbers are numbers of posts.

TrueNews was expected to exhibit an overall relative bias toward the government, and its posts indeed overwhelmingly focused on claims disfavoring the government. This relative bias of TrueNews was even more evident when criticizing misleading claims (81.6%) than debunking factual inaccuracies (73.2%). The pattern is statistically significant (χ² = 6.20, p < 0.05). Strictly speaking, H2 is only partly supported, but Kauyim’s pattern is also consistent with expectation.

Sourcing patterns of the fact-checkers

The other research questions and hypotheses are concerned with the sources of materials utilized. Table 3 summarizes the findings. As much as 74.7% of AFPHK’s reports used “original sources.” This is understandable because tracing the original materials is arguably the most convincing way to show how a quote was misreported or an image was manipulated. Kauyim also had a high percentage of posts (62.3%) using original sources. In contrast, only 18.5% of TrueNews’s posts used original sources.

Table 3.

Sourcing pattern.

	AFPHK	Kauyim	TrueNews
Original sources (χ²/Cramer’s V = 165.311, p < 0.001)	74.7 (65)	62.3 (170)	18.5 (66)
Government sources (χ²/Cramer’s V = 37.524, p < 0.001)	19.5 (17)	7.3 (20)	26.3 (94)
Pro-government media (χ²/Cramer’s V = 1.132, p = 0.57)	5.7 (5)	6.6 (18)	8.4 (30)
Pro-movement media (χ²/Cramer’s V = 12.993, p < 0.01)	3.4 (3)	4.0 (11)	10.9 (39)
Other media (χ²/Cramer’s V = 10.912, p < 0.01)	28.7 (25)	18.7 (51)	30 (107)
Other sources (χ²/Cramer’s V = 36.735, p < 0.001)	47.1 (41)	16.1 (44)	30.8 (110)
No source (χ²/Cramer’s V = 0.117, p < 0.01)	0.0 (0)	10.6 (29)	9.2 (33)

Entries are column percentages. Bracketed figures denote numbers of posts.

For TrueNews, replacing the role of “original sources” are the various kinds of media sources: 8.4%, 10.9% and 30.0% of its posts used materials from pro-government media, pro-movement media, and other media outlets, respectively. Interestingly, TrueNews did not restrict itself to using pro-government media materials. A close reading reveals that, for TrueNews, employing pro-movement media materials often served the rhetorical purpose of highlighting how even pro-movement media acknowledged that certain anti-government rumors were ungrounded or false. In fact, Kauyim was also slightly more likely to utilize materials from pro-government media (6.6%) than from pro-movement media (4.0%). AFPHK is least likely to rely on materials from outlets with a known political stance (5.7% and 3.4% for pro-government and pro-movement media, respectively).

AFPHK was most likely to use “other sources” (47.1%) when compared to Kauyim (16.1%) and TrueNews (30.8%). This is related to the aforementioned fact that, with more resources, AFPHK utilized a wider range of techniques in their work. Hence the sources were more diverse. Notably, combining all the percentages in the first row of Table 3, it can be seen that AFPHK used 1.79 types of sources on average for each report. The corresponding figures for Kauyim and TrueNews were 1.26 and 1.34, respectively. In other words, AFPHK was more likely to use multiple types of sources in each fact-checking exercise.

Regarding H3, AFPHK was the least likely to provide no source information: in fact, all 87 AFPHK posts contained source information. However, there was no substantial difference between Kauyim and TrueNews in percentages of source-less posts (10.6 vs 9.2%). This defies H3’s expectation. Yet a scrutiny of Kauyim’s source-less posts shows that a majority of them (20 out of 29) examined facts that were unverifiable at the moment of fact-checking. For example, one post handled a pro-government politician’s claim that the US had provided USD 60 million to support the movement. It was virtually impossible for a fact-checker to validate the stated fact with evidence. The post simply pointed out the politician’s lack of evidence and reminded citizens to be skeptical about such statements. Excluding such “cautionary posts,” only nine Kauyim posts (3.3%) did not contain source information. In comparison, only three of the 33 source-less posts by TrueNews were cautionary posts. Hence 8.4% of TrueNews posts did not contain source information when they arguably should have. This would make the results in line with the expectation of H3.

Q2 asks if partisan fact-checkers’ biases are more prevalent when posts do not contain source information. Table 4 provides the data. For Kauyim, 69.0% of posts without source information and 45.5% of posts with some sources were checking claims unfavorable toward the movement or its supporters. This suggests that, for Kauyim, the use of sources did seem to relate to a lower level of relative selection bias. However, for TrueNews, while 66.7% of the posts without source information were examining claims unfavorable toward the government side, the corresponding percentage is 77.8% for the posts with source information. Relative selection bias is even stronger when sources, especially partisan media sources, were used. This suggests that the implication of source transparency on selection bias could differ across partisan fact-checkers.

Table 4.

Relative bias by source.

	Examined claims unfavorable toward
	The movement side	The government side	Neither
Kauyim (χ² = 5.768, p = 0.056)
Having source information	45.5 (111)	32.8 (80)	21.7 (53)
Government sources	75.0 (15)	15.0 (3)	10.0 (2)
Pro-government media	57.9 (11)	26.3 (5)	15.8 (3)
Pro-movement media	50.0 (6)	25.0 (3)	25.0 (3)
Non-partisan sources	42.7 (97)	34.8 (79)	22.5 (51)
No source information	69.0 (20)	17.2 (5)	13.8 (4)
TrueNews (χ² = 3.695, p = 0.16)
Having source information	5.2 (17)	77.8 (252)	17.0 (55)
Government sources	2.1 (2)	88.3 (83)	9.6 (9)
Pro-government media	2.7 (1)	88.1 (30)	16.2 (6)
Pro-movement media	2.6 (1)	84.6 (33)	12.8 (5)
Non-partisan sources	5.9 (15)	76.8 (195)	17.3 (44)
No source information	3.0 (1)	66.7 (22)	30.3 (10)

Entries are row percentages. Bracketed figures denote numbers of posts.

Use of government sources

Table 3 has provided the evidence supporting H4. The pro-government TrueNews was indeed the most likely to use government materials (26.3%), whereas Kauyim was the least likely to rely on government materials (7.3%). AFPHK is somewhere in-between (19.5%).

Guided by Q3 and for obtaining a more nuanced understanding of the fact-checkers’ performance, we analyzed their use of government information more closely. Among the 20 Kauyim posts that used government information or materials, seven (35%) relied solely on government information. Interestingly, six of them were debunking claims disfavoring the movement, whereas the remaining post was addressing a claim unfavorable to neither side. That is, Kauyim’s use of government information was similar to how it (and TrueNews) used materials from media on the opposite side: to showcase that even the opposite side acknowledged the lack of substance behind a claim unfavorable to one’s side. For example, in July 2019, a Facebook page “Salute to HK Police” accused a protester of throwing an acid bomb at the police. Kauyim posted the police’s response and wrote:

The police statement did not mention the acid attack......The page admin should stick with police information if they really support the police force. But they once again chose to deceive their supporters and created information chaos.

For AFPHK, among the 17 posts that used official information or materials, eight dealt with claims disfavoring the government side, three dealt with claims disfavoring the movement, and six dealt with claims disfavoring neither. The distribution is relatively more balanced. Only five posts (29.4%) relied solely on government sources. Four were related to disinformation about the 2019 District Council elections, and official information could indeed be seen as adequate for the purpose of clarification. In other cases, governmental responses were used together with other materials. For example, in July 2019, a video went viral online with the claim that some police officers broke into a flat to arrest protesters who participated in the siege of the Legislative Council. AFPHK quoted the police response that the officers were arresting suspects in a case of illegal imprisonment and rape. The report bolstered the validity of its conclusion by quoting a pro-movement outlet, responses by councilors, and a reverse image search.

In comparison, TrueNews was the most likely to use and to rely solely on government sources: 44 of the 94 posts (46.8%) that used official information relied only on it. Besides, 34 of the 44 posts (77.3%) dealt with claims unfavorable to the government side. TrueNews often treated official responses as adequate refutation. For instance, in October 2019, residents in a district reported the smell of tear gas and suspected that the police were testing tear gas inside the district police station. TrueNews refuted the claim, but the only “evidence” presented was the police announcement that they had not conducted any testing. For another example, a TrueNews post in July 2020 criticized the pro-democracy Apple Daily for wrongly accusing the police of using unreasonable violence when arresting a protester. The post wrote:

As the police said, online videos clearly show that the driver refused to obey the instructions by police officers…… The Police condemns the Apple Daily for whitewashing the illegal act of the protesters, maligning the police with biased reporting and distorted facts, and disregarding journalism ethics.

Remarkably, while the police referred to publicly available online videos, TrueNews did not check the original videos. Instead, it simply quoted the police’s statement. The post then recited the police’s criticisms against the Apple Daily. Through such practices, TrueNews assumed the validity of official statements and helped spread official discourses.

Concluding discussion

This article studies how fact-checkers handle the issue of facticity. While the previous study confirmed that partisan fact-checkers exhibited their biases through a varying degree of selective agenda-setting and criticisms toward the claims being examined (Feng et al., 2021), in this study we extend the discussion and focus on how fact-checkers delimit the scope of their practices and how they employ different sources to establish their conclusions. While professional bodies provide guidelines on how to tackle these issues (IFCN, 2021), the performance of practitioners may vary. The analysis shows that professional and partisan fact-checkers indeed differ from each other. When dealing with the Anti-ELAB Movement, the professional AFPHK stuck to verifying factual (in)accuracies, whereas both partisan fact-checkers engaged in criticisms against misleading claims. AFPHK invariably provided source information. It was most likely to track the original source of the debunked claims or materials and to use multiple types of sources in one report. These can be the grounds to see AFPHK’s fact-checking as overall speaking more trustworthy and solid.

Nevertheless, partisan and professional fact-checkers do not constitute a simple dichotomy. Consistent with previous findings (Feng et al., 2021), partisan fact-checkers can vary in professionalism. Specifically, Kauyim’s performance, when compared with TrueNews’s, was closer to AFPHK in several extents: in which it stuck to debunking factual inaccuracies, in which it tracked the original sources of the debunked claims and materials, and in which it offered source information (when the “cautionary posts” were put aside). TrueNews fared worse in these regards, and there is evidence that it handled government materials particularly uncritically.

The findings have several implications for our understanding of fact-checking. First, this study extends previous analysis (Feng et al., 2021) and demonstrates additional differences between partisan and professional fact-checkers. This study and Feng et al. (2021) point to the lack of consistency among fact-checkers, at least in Hong Kong. However, we do not intend to question the enterprise of fact-checking due to inconsistent and problematic performances (Uscinski, 2015). As different groups are interested in conducting fact-checking to serve varying purposes, inconsistency is inevitable. Evaluating the quality of fact-checking should be based not on consistency among players, but on whether and how recognized standards are followed.

Second, one should not dismiss the potential contribution of partisan fact-checkers. One could evaluate a partisan fact-checker based on professional principles, and the results may show that some partisan fact-checkers are actually performing reasonably well. The hallmark of a partisan fact-checker is its selective agenda setting (Feng et al., 2021; Graves, 2018), and some partisan fact-checkers may want to adopt the best practices to convincingly combat disinformation attacking one’s side. Normatively speaking, scholars have noted the distinction between internal and external diversity (Loecherbach et al., 2020), i.e. diversity can be achieved by a media outlet offering a diverse range of viewpoints, or by having outlets favoring different views. Similarly, diversity and overall fairness in fact-checking might be achieved by having partisan fact-checkers on different sides of a political divide instead of requiring all fact-checkers to be strictly “neutral.”

The potential contribution of partisan fact-checkers is pertinent to the fundamental question of who can legitimately claim to be a fact-checker. With the current findings, one might question whether TrueNews can be considered a fact-checker, or merely a partisan entity spreading false information. Certainly, if one normatively defines that only an entity exhibiting a minimum degree of professionalism is a fact-checker, then one can evaluate the credibility of TrueNews. However, this study is not based on any normative definition. It begins with the fact that TrueNews (and Kauyim) claimed to conduct fact-checking. The findings thus show how self-professed fact-checkers might vary substantially in their practices. This is an important point given the reality that many organizations may similarly profess to be conducting fact-checking. Meanwhile, the relatively high degree of professionalism exhibited by Kauyim also suggests that one cannot dismiss all partisan fact-checkers as propaganda machines.

Third, this study emphasizes the distinction between debunking factual inaccuracies or fake information and criticizing misleading claims. The findings suggest that professional fact-checkers do tend to stick to the task of examining “checkable facts” (Nieminen and Sankari, 2021; Walter and Salovich, 2021). This is grounded in the distinction between facts and opinions in journalistic practices. Philosophically speaking, one can criticize the fact-opinion distinction as epistemologically naïve (Uscinski and Butler, 2013). However, the distinction can have an important pragmatic value where outright deception is prevalent (Amazeen, 2015). As this study shows, partisan fact-checkers are more likely to criticize misleading claims, and their biases are more conspicuous when they do so. Sticking to facts seemingly helps constrain political biases.

In addition, in debates around the world surrounding the use of legislation to combat fake news, authoritarian states are more likely to support legislation and to adopt a broader definition of disinformation (Neo, 2020). Perhaps non-incidentally, in this study, the pro-government fact-checker is the most likely to treat misleading claims as fake news to be debunked. It is beyond this article’s scope to discuss exactly how a broad definition of disinformation could facilitate the suppression of dissent. It is needed to attend to the political implications of the operational definition of fake news.

Fourth, this study examines the range of sources utilized by fact-checkers. In addition to how the use of sources can represent the extent of professionalism, some findings point to strategic uses of sources. Most interestingly, although it is not the focus of the analysis, we saw that partisan fact-checkers may proactively utilize sources from the other side of the political divide when debunking claims unfavorable to one’s side. We also noted that the pro-movement Kauyim had used government materials in the same way. The logic seems to be that the fact-checking’s conclusion would be seen as particularly convincing when the other side also acknowledges – or provides materials that point to – the wrongfulness of the examined claim. This finding also suggests that, in addition to tracking original sources or using multiple sources for triangulation, fact-checkers of specific backgrounds could have other methods to establish their credibility.

Notably, as this study has identified a prototypical professional fact-checker and two partisan fact-checkers, some of the differences could be seen as unsurprising. But such differences are worth documenting. Moreover, there are also unexpected nuances in the findings, such as how sticking to debunking factual inaccuracies could constrain the expression of partisanship, the tendency for partisan fact-checkers to cite media on the opposite side strategically, the varying implications of the use of sources on the expression of partisanship, and the qualitative findings regarding how different fact-checkers deal with government sources. The latter particularly illustrated that using government sources can be legitimate or problematic depending on how they were used, on what occasions, and together with what other sources.

As noted earlier, fact-checkers’ performance can be influenced not only by their degrees of professionalism and partisanship, but also by resourcefulness. We cannot dismiss the possibility that resourcefulness may partly explain the differences between AFPHK and the two partisan fact-checkers. However, our analysis also shows that Kauyim demonstrated a high degree of professionalism despite the fact that only a few volunteers operated it. Notably, TrueNews published more reports about the Anti-ELAB Movement than Kauyim. Therefore, the differences between the two seem not to be explicable by resource differences. Moreover, one interesting finding is that Kauyim’s relative selection bias was clearer when its posts did not have any sources, whereas TrueNews’s relative selection bias was stronger when the posts had sources. Similarly, having more resources may suppress or amplify partisan biases. Future research can investigate more systematically the impact of resourcefulness on fact-checkers’ performance.

A few limitations and possibilities for future studies should be acknowledged. This study examines only a prominent movement in Hong Kong. It would be interesting to examine if the differences between professional and partisan fact-checkers would be smaller when they handle issues that are less contentious or political. Methodologically, researchers may consider combining content analysis and survey or experimental research on the impacts of fact-checking. While some studies have acknowledged that presentational styles or methodologies can influence the effectiveness of fact-checking (Amazeen et al., 2018; Walter et al., 2019), extant studies rarely tie their “independent variables” to the content of fact-checking reports. Doing so could strengthen the real-world relevance of impact studies.

Lastly, given the emergence of fact-checkers worldwide, it should be possible to construct a coding scheme for fact-checking reports that is applicable across national contexts, e.g. by following generally accepted codes of practices. Comparative analysis of the performance of fact-checkers in varying national and political contexts could provide important insights into our theoretical understanding of fact-checking as a phenomenon and practical information for fact-checkers in different places to improve their practices.

Footnotes

Acknowledgements

The authors are grateful to the editor and anonymous reviewers for their detailed and useful feedback on this manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research reported in this article was supported by a Vice-Chancellor’s One-off Discretionary Fund offered by the Chinese University of Hong Kong to the third author in 2019.

ORCID iDs

Nathan L.T. Tsang

Mengzhe Feng

Francis L.F. Lee

Author Biographies

Nathan L.T. Tsang is a PhD student in sociology at the University of Southern California and was a research assistant at the School of Journalism and Communication, the Chinese University of Hong Kong. His research focuses on social movement studies, cultural sociology, and disinformation studies.

Mengzhe Feng is an Mphil student at the School of Journalism and Communication, the Chinese University of Hong Kong. Her research interests include political communication, gender and communication, and social activism.

Francis L.F. Lee is Professor at the School of Journalism and Communication, the Chinese University of Hong Kong. He is the lead author of Media, Social Mobilisation and Mass Protests in Post-colonial Hong Kong (Routledge, 2011) and Media and Protest Logics in the Digital Era: The Umbrella Movement in Hong Kong (Oxford University Press, 2018). He is also Chief Editor of The Chinese Journal of Communication and Associate Editor of Mass Communication & Society.

References

Amazeen

(2015) Revisiting the epistemology of fact-checking. Critical Review 27(1): 1–22.

Amazeen

Thorson

Muddiman

, et al. (2018) Correcting political and consumer misperceptions: the effectiveness and effects of rating scale versus contextual correction formats. Journalism and Mass Communication Quarterly 95(1): 28–48.

Anderson

Schudson

(2019) Objectivity, professionalism, and truth seeking. In: Wahl-Jorgensen

Hanitzsch

(eds) The Routledge Handbook of Journalism Studies. New York, NY: Routledge.

Borel

(2016) The Chicago Guide to Fact-Checking. Chicago, IL: University of Chicago Press.

Carlson

(2017) Journalistic Authority. New York, NY: Columbia University Press.

Chan

Lee

FLF

(eds) (2008) Media and Politics in Post-Handover Hong Kong. London: Routledge.

Cheng

(2020) United front work and mechanisms of countermobilization in Hong Kong. The China Journal 83: 1–33.

Feng

Tsang

NLT

Lee

FLF

(2021) Fact-Checking as mobilization and counter-mobilization: the case of the anti-extradition bill movement in Hong Kong. Journalism Studies 22(10): 1358–1375.

Fishman

(1980) Manufacturing the News. Austin, TX: University of Texas Press.

10.

Graves

(2016) Deciding What’s True. New York, NY: Columbia University Press.

11.

Graves

(2017) Anatomy of a fact check: objective practice and the contested epistemology of fact checking. Communication, Culture and Critique 10(3): 518–537.

12.

Graves

(2018) Boundaries not drawn. Journalism Studies 19(5): 613–631.

13.

Graves

Konieczna

(2015) Sharing the news: journalistic collaboration as field repair. International Journal of Communication 9: 1966–1984.

14.

Haigh

Kozak

(2018) Stopping fake news. Journalism Studies 19(14): 2062–2087.

15.

Higdon

(2020) The Anatomy of Fake News. California: University of California Press.

16.

IFCN (International Fact-Checking Network) (2021) International fact-checking network code of principles. Available at: https://www.ifcncodeofprinciples.poynter.org/know-more/the-commitments-of-the-code-of-principles (accessed 14 August 2021).

17.

Khaldarova

Pantti

(2016) Fake news. Journalism Practice 10(7): 891–901.

18.

Kwanda

Lin

TTC

(2020) Fake news practices in Indonesian newsrooms during and after the Palu earthquake: a hierarchy-of-influences approach. Information, Communication and Society 23(6): 849–866.

19.

Lee

FLF

(2018) Changing political economy of the Hong Kong media. China Perspectives 3: 9–18.

20.

Lee

FLF

Chan

(2018) Media and Protest Logics in the digital Era. New York, NY: Oxford University Press.

21.

Lee

FLF

Tang

GKY

Yuen

, et al. (2020) Five demands and (not quite) beyond. Communist and Post-Communist Studies 53(4): 22–40.

22.

Lim

(2018) Checking how fact-checkers check. Research and Politics 5(3): 205316801878684.

23.

Luqiu

(2021) Covering the 2019 Hong Kong Protests. London: Palgrave MacMillan.

24.

Loecherbach

Moeller

Trilling

, et al. (2020) The unified framework of media diversity: a systematic literature review. Digital Journalism 8(5): 605–642.

25.

Lowrey

(2017) The emergence and development of news fact-checking sites. Journalism Studies 18(3): 376–394.

26.

Neo

(2020) A cudgel of repression: analyzing state instrumentalisation of the “fake news” label in Southeast Asia. Journalism 2020: 146488492098406.

27.

Nieminen

Sankari

(2021) Checking politifact’s fact-checks. Journalism Studies 22(3): 358–378.

28.

Nyhan

Reifler

(2010) When corrections fail: the persistence of political misperceptions. Political Behavior 32: 303–330.

29.

Porter

Wood

Kirby

(2018) Sex trafficking, Russian infiltration, birth certificates, and pedophilia: a survey experiment correcting fake news. Journal of Experimental Political Science 5(2): 159–164.

30.

Rogerson

(2013) Fact checking the fact checkers: Verification web sites, partisanship and sourcing. In: Campbell

Martin

(eds) American Political Science Association 2013 Annual Meeting. Washington DC. Available at: https://ssrn.com/abstract=2301235

31.

CYK

(2016) How do university students perceive political parties and mass media? Available at:https://theinitium.com/article/20161005-opinion-clementso-umbrellamovement/ (accessed 17 January 2022).

32.

Schudson

(1978) Discovering the News. New York, NY: Basic Books.

33.

Scott

(1985) Weapons of the Weak. New Haven, CT: Yale University Press.

34.

Strauss

Corbin

(1998) Basics of Qualitative Research Techniques. London: Sage.

35.

Tandoc

(2019) The facts of fake news: a research review. Sociology Compass 13(2): e12724.

36.

Tandoc

Lim

Ling

(2018) Defining "fake news". Digital Journalism 6(2): 137–153.

37.

Tuchman

(1978) Making News. New York, NY: Free Press.

38.

Uscinski

(2015) The epistemology of fact checking (is still naìve): rejoinder to amazeen. Critical Review 27(2): 243–252.

39.

Uscinski

Butler

(2013) The epistemology of fact checking. Critical Review 25(2): 162–180.

40.

Van Leuven

Kruikemeier

Lecheler

, et al. (2018) Online and newsworthy. Digital Journalism 6(7): 798–806.

41.

Waisbord

(2018) Truth is what happens to news. Journalism Studies 19(13): 1866–1878.

42.

Walter

Cohen

Holbert

, et al. (2019) Fact-checking: a meta-analysis of what works and for whom. Political Communication 37(3): 350–375.

43.

Walter

Salovich

(2021) Unchecked versus uncheckable: how opinion-based claims can impede corrections of misinformation. Mass Communication and Society 24(4): 500–526.