(Ideo)Logical Reasoning: Ideology Impairs Sound Reasoning

Abstract

Beliefs shape how people interpret information and may impair how people engage in logical reasoning. In three studies, we show how ideological beliefs impair people’s ability to (1) recognize logical validity in arguments that oppose their political beliefs and (2) recognize the lack of logical validity in arguments that support their political beliefs. We observed belief bias effects among liberals and conservatives who evaluated the logical soundness of classically structured logical syllogisms supporting liberal or conservative beliefs. Both liberals and conservatives frequently evaluated the logical structure of entire arguments based on the believability of arguments’ conclusions, leading to predictable patterns of logical errors. As a result, liberals were better at identifying flawed arguments supporting conservative beliefs and conservatives were better at identifying flawed arguments supporting liberal beliefs. These findings illuminate one key mechanism for how political beliefs distort people’s abilities to reason about political topics soundly.

Keywords

belief bias political psychology ideology logical reasoning syllogistic reasoning

Is logical reasoning the antidote to seemingly intractable political disagreements, or might it be a key ingredient in the poison of political partisanship? On the one hand, rationality and logic offer great promise: The application of principles of sound inference has the potential to improve decision-making and promote political consensus. On the other hand, the application of logic is rarely objective or rational (Henle & Michael, 1956; Morgan & Morton, 1944; Thouless, 1959). Decades of research on human judgment—or even a glimpse at modern political dysfunction—remind us that humans are imperfect at applying logical principles. Opposing political partisans often disagree not only about their political beliefs but also over the logical soundness of arguments supporting those beliefs.

In the present article, we examine whether one bias that is endemic to human reasoning—namely, belief bias—interferes in perceptions of logic in political arguments. Belief bias refers to a common tendency for the subjective believability of an argument’s conclusion to influence evaluations of the logical soundness of the entire argument (Evans, Barston, & Pollard, 1983; Feather, 1964; Oakhill & Johnson-Laird, 1985). Consider, for example, the following syllogism (Norenzayan, Smith, Kim, & Nisbett, 2002):

All things made of plants are healthy.

Cigarettes are made of plants.

Therefore, cigarettes are healthy.

Although this argument is logically sound (the conclusion follows logically from the premises), many people will evaluate it as unsound due to the implausibility of its conclusion about the health value of cigarettes. If, however, “cigarettes” are replaced by “salads,” ratings of the logical soundness of the argument will increase substantially even though substituting a plausible conclusion for an implausible one has no effect on whether that conclusion follows logically from the premises. Belief bias (originally referred to as “atmosphere effects”), which has a long history in psychological research (e.g., Morgan & Morton, 1944; Sells, 1936), can degrade logical reasoning by predisposing people to incorrectly judge unsound arguments as logically sound when their conclusions are subjectively believable and to incorrectly judge sound arguments as unsound when conclusions are less believable (Janis & Frick, 1943).

Research has previously shown that motivated reasoning, in various forms, can degrade formal logical reasoning. Gervais, Shariff, and Norenzayan (2011) found that antipathy toward an outgroup increased participants’ likelihood of committing the conjunction fallacy. Kahan, Peters, Dawson, and Slovic (2017) showed that participants’ political motivations interfered with their ability to form valid conclusions about empirical data. Belief bias in particular has been observed across cultures (though to differing degrees; Norenzayan et al., 2002) and in a number of contexts (Evans et al., 1983; Newstead, Pollard, Evans, & Allen, 1992; Oakhill & Johnson-Laird, 1985). Feather (1964), for example, found that people’s religious beliefs biased their likelihood of evaluating pro-religious and anti-religious syllogisms as sound or unsound.

However, there are reasons to suspect that belief bias may present unique problems within the context of political reasoning. First, bias is pervasive during the evaluation of political arguments. Politics evoke motivations to defend the legitimacy and morality of one’s policy views, political party, and cultural connections, and these motivations can bias political judgments in predictable ways. Committed partisans evaluate the same scientific evidence as more valid when it supports rather than challenges their political views (Lord, Ross, & Lepper, 1979), they evaluate the identical policy more favorably if they believe their own party supports it rather than the opposing party (Cohen, 2003), and they even see less violence in the same video of a political demonstration if they support the cause of the protest than if they oppose it (Kahan, Hoffman, Braman, & Evans, 2012). There is evidence that these effects may not be mitigated by superior reasoning skills. In fact, those with higher abilities in analyzing quantitative information can use those abilities to selectively interpret data to suit their desired political outlooks (Kahan, Peters, Dawson, & Slovic, 2017). These reasoning biases appear to occur automatically (Smith, Ratliff, & Nosek, 2012), and a “bias blind spot” may result in underestimating the impact of such biases on one’s own judgments even when those same biases are readily recognizable when committed by others (Pronin, Lin, & Ross, 2002).

A second reason that belief bias causes unique problems in the political domain is that political beliefs are often divided along ideological lines. Individuals or groups with differing beliefs should be biased to perceive logical soundness in opposite directions. In the example above, virtually everyone finds “salads are healthy” a more believable statement than “cigarettes are healthy.” But the same is not true of assertions frequently made in political discourse like “tax increases harm the economy” or “abortion is murder.” Liberals and conservatives are likely to have opposing beliefs about the truth-values of such statements. Belief bias should thus cause arguments with these conclusions to seem logically sound to one group while appearing logically unsound to the other. This, in turn, promotes a specific pattern of “mirror-image” judgment errors in which each side tends to be overly lax in accepting the logic of arguments with politically palatable conclusions and overly critical of the logic of arguments with politically unpalatable conclusions.

In three studies, we examined the impact of belief bias on liberals’ and conservatives’ abilities to evaluate the logical soundness of political arguments. In Study 1, we observed significant belief bias effects among liberals and conservatives from YourMorals.org who evaluated the logical soundness of classically structured logical syllogisms supporting liberal or conservative beliefs. In Study 2, we observed ideological belief bias effects among participants from ProjectImplicit.org who were trained in logical reasoning before evaluating political syllogisms presented in natural language similar to what one might encounter in popular media. In Study 3, we replicated Studies 1 and 2 in a nationally representative sample and again observed belief bias effects among both liberals and conservatives.

Studies

The complete data and analysis scripts for all studies are available at https://osf.io/njcqc/.

Study 1

Materials and Methods

Participants were 1,374 visitors (30.1% female, M_age = 40.09, SD_age = 16.50) to YourMorals.org, a psychological research website where volunteers complete psychological surveys in exchange for personalized feedback about their results. Political ideology was measured along a continuous 7-point scale (1 = very liberal, 7 = very conservative) with additional options for “don’t know/not political,” “libertarian,” and “other.” There were 924 in the final sample (490 liberals, 110 moderates, and 324 conservatives) after excluding participants who did not indicate their political ideology along the 7-point liberal–conservative spectrum.

Participants completed a test of logical reasoning (using Norenzayan et al., 2002 as a guide) consisting of 16 syllogisms in which the terms within the premises and conclusions were political in nature (see Table 1). Participants were asked to judge whether each conclusion logically followed from its premises. The premises were balanced such that half of the arguments were sound and half were unsound. Further, they were equally divided among two classical logical structures: modus ponens (MP; If P, then Q. P. Therefore, Q) and modus tollens (MT; If P, then Q. Not Q. Therefore, not P). The study manipulated what might be called partisan believability, that is, half of the syllogisms contained conclusions that were consistent with liberal ideological beliefs (e.g., “Abortion is not murder”), while the other half contained conclusions that were consistent with conservative ideological beliefs (e.g., “Tax increases harm the economy”).

Table 1.

The Breakdown of Logical Arguments and the Participant Stimulus for the Three Studies.

Syllogism Part	Study 1: Ideological	Study 2 and 3: Ideological	Study 3: Nonideological
Premise 1	All drugs that are dangerous should be illegal	Judge Wilson believes that if a living thing is not a person, then one has the right to end its life	Doctor Simmi believes that if a tumor is not detected, then the surgery will proceed as planned
Premise 2	Marijuana is a drug that is dangerous	She also believes that a fetus is a person	She also believes that a tumor was detected
Conclusion	Therefore, Marijuana should be illegal	Therefore, Judge Wilson concludes that no one has the right to end the life of a fetus	Therefore, Doctor Simmi concludes that the surgery will not proceed as planned
Participant stimulus	All drugs that are dangerous should be illegal. Marijuana is a drug that is dangerous. Therefore, Marijuana should be illegal	Judge Wilson believes that if a living thing is not a person, then one has the right to end its life. She also believes that a fetus is a person. Therefore, Judge Wilson concludes that no one has the right to end the life of a fetus	Doctor Simmi believes that if a tumor is not detected, then the surgery will proceed as planned. She also believes that a tumor was detected. Therefore, Doctor Simmi concludes that the surgery will not proceed as planned

Participants were specifically instructed to judge whether or not the conclusion of each syllogism followed logically from its premises, while assuming that all of the premises were true and limiting themselves only to information presented in the premises. They were asked to “Choose YES if, and only if, you judge that the conclusion can be derived from the given premises. Otherwise, choose NO.” See Supplemental Information (SI) for the manipulation check results for all three studies.

Results

For each of the following three studies, we report two separate regressions, predicting the perceived logical soundness of arguments and the accuracy of each participant’s judgments, respectively. For each model reported below, we first report each significant main effect and interaction term observed in the model, followed by a more detailed explanation of the interaction term testing each study’s primary hypothesis.

For Study 1, a logistic regression with a three-way interaction was conducted to predict perceived logical soundness using the political ideology of the argument’s conclusion (henceforth argument ideology; liberal vs. conservative), soundness of the argument (sound vs. unsound), and participant political ideology (henceforth participant ideology; 1 = very liberal, 7 = very conservative) as predictor variables, while controlling for the logical structure. (For all studies, see SI for the data on demographics [Table SI1], results controlling for demographics [Table SI2], and manipulation checks [Table SI3].)

We observed main effects of participant ideology and argument soundness, such that, for participant ideology, participants at higher levels of conservatism¹ perceived sound arguments as less sound (odds ratio [OR] = 0.93) and, for argument soundness, perceived sound arguments as more sound (OR = 8.59). There was no main effect of argument ideology (OR = 0.92). There was, however, a main effect of argument’s logical structure (OR = 0.75), such that modus tollens arguments were perceived as less sound. We also observed a two-way interaction between participant ideology and argument soundness (OR = 0.78), but no two-way interaction between argument soundness and argument ideology (OR = 0.94).

Most relevant to our hypothesis, there was a two-way interaction between participant ideology and argument ideology (OR = 1.49). Specifically, at higher levels of conservatism, participants were less likely to rate syllogisms with liberal conclusions as sound (b = −0.04, p < .001) and more likely to rate syllogisms with conservative conclusions as sound (b = 0.04, p < .001). This effect was not qualified by the actual soundness of the syllogisms. That is, there was no three-way interaction between participant ideology, argument soundness, and argument ideology (OR = 0.99). See Table 2 for full results. See Table 3 for the summary statistics for all three studies.

Table 2.

Study 1 Logistic Regression With Three-Way Interaction Predicting Perceived Logical Soundness.

Predictor	Odds Ratio	RSE	95% Confidence Interval
PI	0.93	.02 (.003)	0.88	0.97
AS	8.59	.65 (<.001)	7.40	9.96
PI × AS	0.78	.03 (<.001)	0.73	0.84
AI	0.92	.05 (.138)	0.3	1.03
PI × AI	1.49	.04 (<.001)	1.41	1.58
AS × AI	0.94	.06 (.277)	0.83	1.06
PI × AS × AI	0.99	.03 (.621)	0.93	1.04
Logical structure	0.75	.03 (<.001)	0.70	0.80

Note. p Values are given in the parentheses. PI = participant ideology; AS = argument soundness; AI = argument ideology; RSE = Robust Standard Error.

Table 3.

Coefficients for Regressions With Three-Way Interaction Predicting Perceived Logical Soundness for Studies 1–3.

Predictor	Study 1 (Odds Ratio)	Study 2 (b)	Study 3 (b)
PI	0.93 (.003)	−.02 (.577)	.02 (.433)
AS	8.59 (<.001)	.66 (<.001)	.55 (<.001)
PI × AS	0.78 (<.001)	−.11 (.039)	−.08 (.022)
AI	0.92 (.138)	.11 (.157)	.22 (.002)
PI × AI	1.49 (<.001)	.16 (.002)	.08 (.036)
AS × AI	0.94 (.277)	−.04 (.721)	−.07 (.454)
PI × AS × AI	0.99 (.621)	−.04 (.602)	.04 (.404)
Logical structure	0.75 (<.001)	−.53 (<.001)	−.43 (<.001)
Argument’s topic
Abortion	—	.01 (.862)	.06 (.383)
Capital punishment	—	−.99 (<.001)	−.65 (<.001)
Government intervention	—	−.19 (<.001)	−.22 (<.001)

Note. p Values are given in the parentheses. Base argument topic is affirmative action. PI = participant ideology; AS = argument soundness; AI = argument ideology.

On average, participants correctly judged 73% of the syllogisms. To better understand the influence of belief bias on participants’ accuracy, we conducted a second three-way logistic regression to predict the accuracy of each judgment based on the participant’s ideology and argument’s ideology. There was a main effect of participant ideology such that participants at higher levels of conservatism had greater accuracy (OR = 1.08). There was no main effect of argument soundness (OR = 1.16), nor of argument ideology (OR = 1.07). There was, however, a main effect of each argument’s logical structure (OR = 0.55), such that modus tollens arguments had worse accuracy. There was a two-way interaction between participant ideology and argument soundness (OR = 0.67), between participant ideology and argument ideology (OR = 0.67), and between argument soundness and argument ideology (OR = 0.80).

Most importantly, there was a three-way interaction between participant ideology, argument soundness, and argument ideology (OR = 2.20). Specifically, the more conservative the participant, the more likely sound syllogisms with conservative conclusions were to be evaluated correctly as sound (b = 0.01, p = .006), but the less likely unsound syllogisms with conservative conclusions were to be evaluated correctly as unsound (b = −0.06, p < .001). The opposite pattern of errors was found for syllogisms with liberal conclusions. In these cases, the more conservative the participant, the less likely sound syllogisms were to be evaluated correctly (b = −0.06, p < .001) and the more likely unsound syllogisms were to be evaluated correctly (b = 0.02, p = .003). Thus, the accuracy of participants’ judgments depended on participants’ ideologies, the syllogisms’ ideologies, and the soundness of the syllogisms (see Figure 1; Table 4 for full results; Table 5 for the summary statistics for all three studies).

Figure 1.

Study 1 results for the likelihood of correct responses for sound liberal arguments, sound conservative arguments, unsound liberal arguments, and unsound conservative arguments for liberal and conservative participants. Error bars are 95% CI.

Table 4.

Study 1 Logistic Regression With Three-Way Interaction Predicting Accuracy of Participant’s Judgments.

Predictor	Odds Ratio	RSE	95% Confidence Interval
PI	1.08	.03 (.003)	1.03	1.14
AS	1.16	.09 (.066)	0.99	1.35
PI × AS	0.67	.03 (<.001)	0.62	0.73
AI	1.07	.06 (.198)	0.96	1.19
PI × AI	0.67	.02 (<.001)	0.63	0.71
AS × AI	0.80	.07 (.011)	0.68	0.95
PI × AS × AI	2.20	.11 (<.001)	1.99	2.43
Logical structure	0.55	.02 (<.001)	0.52	0.60

Note. p Values are given in the parentheses. PI = participant ideology; AS = argument soundness; AI = argument ideology.

Table 5.

Coefficients for Logistic Regression With Three-Way Interaction Predicting Accuracy of Participant’s Judgments for Studies 1–3.

Predictor	Study 1	Study 2	Study 3
PI	1.08 (.003)	1.00 (.818)	0.97 (.416)
AS	1.16 (.066)	4.01 (<.001)	2.80 (<.001)
PI × AS	0.67 (<.001)	0.89 (.024)	0.98 (.761)
AI	1.07 (.198)	1.00 (.984)	0.83 (.019)
PI × AI	0.67 (<.001)	0.88 (.009)	0.95 (.201)
AS × AI	0.80 (.011)	1.03 (.780)	1.33 (.016)
PI × AS × AI	2.20 (<.001)	1.26 (<.001)	1.17 (.011)
Logical structure	0.55 (<.001)	0.44 (<.001)	0.48 (<.001)
Argument’s topic
Abortion	—	1.17 (.057)	0.99 (.958)
Capital punishment	—	1.05 (.576)	0.82 (.052)
Government intervention	—	1.06 (.395)	1.00 (.951)

Note. p Values are given in the parentheses. Base argument topic is affirmative action. All coefficients are odds ratios. PI = participant ideology; AS = argument soundness; AI = argument ideology.

Discussion

Results from Study 1 indicate that participants showed ideological belief bias—they perceived arguments as more logically sound to the extent that their conclusions were congenial with their ideologies, and this relationship was mediated by participants’-specific ideological beliefs. This led them to be overly harsh evaluators of the soundness of politically challenging arguments (i.e., more likely to judge a sound argument as unsound) and overly lax evaluators of the soundness of politically congenial arguments (i.e., more likely to judge an unsound argument as sound).

Although Study 1 provided preliminary evidence for ideological belief bias, the rigid, syllogistic structure of the logical arguments limits the generalizability of these findings. In Study 2, we aimed to better understand the extent to which ideological belief bias influences everyday political judgments by examining ideological belief bias effects using stimuli designed to be more similar to the language one encounters conversationally. In Study 2, we also provided participants with a training task prior to the actual task to ease the difficulty of the task.

Study 2

Materials and Methods

One thousand, seven hundred and forty-three participants (65.4% female; M_age = 34.25, SD_age = 13.39) entered the study at Project Implicit, a public research and education website (https://implicit.harvard.edu/). Participants choose to participate for a variety of reasons (e.g., class requirements, personal curiosity, research recruitment) and were randomly assigned to this study among a pool of studies. Political ideology was again measured along a continuous 7-point scale (1 = very liberal, 7 = very conservative). There were 1,489 participants in the final sample (622 liberals, 515 moderates, and 353 conservatives) after excluding those who did not indicate their political ideology along the 7-point spectrum.

Before completing a test for logical reasoning, participants completed a training session that reviewed the essentials of logical reasoning and introduced the structure of the logical arguments. The training reinforced participants’ knowledge about logical reasoning with four arguments that participants should easily recognize as sound or unsound (e.g., “If a person is pregnant, then that person is a woman. Taylor is pregnant. Therefore, Taylor is a woman”). Participants read the arguments (half MP, half MT), evaluated their soundness, and then received immediate feedback with the correct answer.

After the training session, participants completed a test with four syllogisms in which the terms within the premises and conclusions were political in nature (as in Study 1) and in addition were framed in a less formal and more conversational language (see Table 1). Similar to Study 1, the syllogisms were balanced according to soundness, ideology, and logical structure. Finally, for the four syllogisms presented to the participant, one was on the topic of capital punishment, one was on abortion, one was on government intervention, and one was on affirmative action. See the SI for the full set of arguments.

After reading the arguments, to elicit their evaluation of the logic and the confidence of their answer participants were asked to, “Evaluate whether the claims are logically sound or not, and how confident you are about your assessment,” using a 6-point scale (−3 = logically unsound, very confident; −2 = logically unsound, confident; −1 = logically unsound, slightly confident; 1 = logically sound, slightly confident; 2 = logically sound, confident; 3 = logically sound, very confident).

Results

Given the continuous outcome variable in Study 2, a linear mixed effects model with three-way interactions was conducted to predict logical soundness ratings using the argument ideology, the soundness of the argument, participant ideology, logical structure, and argument topic as predictor variables.

Consistent with Study 1, there was a main effect of argument soundness such that participants perceived sound arguments as more sound (b = 0.66). However, there was no main effect of participant ideology (b = −0.02) or argument ideology (b = 0.11). There was, however, a main effect of argument’s logical structure such that modus tollens arguments are perceived as less sound (b = −0.53) and of argument’s topic such that arguments concerning capital punishment were perceived as the least sound (b = −0.99). There was also a two-way interaction between participant ideology and argument soundness (b = −0.11), but no two-way interaction between argument soundness and argument ideology (b = −0.04).

Most relevant to our hypothesis, we again observed that participants’ evaluations of the logical syllogisms depended, in part, on their ideological leanings and the arguments ideology, as evidenced by a significant interaction between participant ideology and argument ideology (b = 0.16). Specifically, the more conservative the participant, the more syllogisms with conservative conclusions were rated as sound (b = 0.06, p = .026) and the more syllogisms with liberal conclusions were rated as unsound (b = −0.07, p = .006). This was again found regardless of the actual soundness of the arguments, that is, there was no three-way interaction between participant ideology, argument soundness, and argument ideology (b = −0.04). See Table 6 for full results.

Table 6.

Study 2 Linear Regression With Three-Way Interaction Predicting Perceived Logical Soundness.

Predictor	b	SE	95% Confidence Interval
PI	−.02	.04 (.577)	−0.09	0.05
AS	.66	.08 (<.001)	0.50	0.81
PI × AS	−.11	.05 (.039)	−0.21	−0.01
AI	.11	.08 (.157)	−0.04	0.27
PI × AI	.16	.05 (.002)	0.06	0.26
AS × AI	−.04	.12 (.721)	−0.27	0.18
PI × AS × AI	−.04	.07 (.602)	−0.18	0.10
Logical structure	−.53	.06 (<.001)	−0.64	−0.42
Argument’s topic
Abortion	.01	.08 (.862)	−0.14	0.17
Capital punishment	−.99	.08 (<.001)	−1.15	−0.84
Government intervention	−.19	.08 (<.001)	−0.34	−0.03

Note. p Values are given in the parentheses. Base argument topic is affirmative action. PI = participant ideology; AS = argument soundness; AI = argument ideology.

Despite the training session, on average, participants only judged 53.0% of the logical arguments correctly—reaffirming that logical reasoning is cognitively difficult, perhaps especially so when evaluating less formal, conversational arguments. To further examine the factors influencing judgment accuracy, we conducted a three-way logistic regression with accuracy as the dependent variable. There was no main effect of participant ideology (OR = 1.00) or argument ideology (OR = 1.00). There was, however, a main effect of argument soundness such that there was a greater accuracy for sound arguments (OR = 4.01) and each argument’s logical structure such that there was a worse accuracy for modus tollens arguments (OR = 0.44). We did not observe a main effect of argument topic (OR = 1.01; see SI for breakdown of correctness based on the structure and the topic of the argument). There was a two-way interaction between participant ideology and argument soundness (OR = 0.89), between participant ideology and argument ideology (OR = 0.88), but not between argument soundness and argument ideology (OR = 1.03).

Most importantly, there was a three-way interaction between participant ideology, argument soundness, and argument ideology (OR = 1.26). Thus, as political conservatism increased, syllogisms with conservative conclusions were less likely to be evaluated correctly if they were unsound (b = −0.03, p = .005), and syllogisms with liberal conclusions were less likely to be evaluated correctly if they were sound (b = −0.02, p = .003). However, as political conservatism increased, there was no difference for syllogisms with conservative conclusions that were sound (b = 3.82 × 10⁻³, p = .626) and for syllogisms with liberal conclusions that were unsound (b = −1.32 × 10⁻⁴, p = .988; see Figure 2; Table 7 for full results).

Figure 2.

Study 2 results for the likelihood of correct responses for sound liberal arguments, sound conservative arguments, unsound liberal arguments, and unsound conservative arguments for liberal and conservative participants. Error bars are 95% CI.

Table 7.

Study 2 Logistic Regression With Three-Way Interaction Predicting Accuracy of Participant’s Judgments.

Predictor	Odds Ratio	RSE	95% Confidence Interval
PI	1.00	.04 (.818)	0.94	1.08
AS	4.01	.34 (<.001)	3.40	4.74
PI × AS	0.89	.05 (.024)	0.80	0.98
AI	1.00	.08 (.984)	0.86	1.17
PI × AI	0.88	.04 (.009)	0.81	0.97
AS × AI	1.03	.18 (.780)	0.83	1.29
PI × AS × AI	1.26	.09 (<.001)	1.11	1.45
Logical structure	0.44	.02 (<.001)	0.39	0.49
Argument’s topic
Abortion	1.17	.09 (.057)	0.99	1.38
Capital punishment	1.05	.09 (.576)	0.88	1.25
Government intervention	1.06	.08 (.395)	0.92	1.23

Note. p Values are given in the parentheses. Base argument topic is affirmative action. PI = participant ideology; AS = argument soundness; AI = argument ideology.

Discussion

Study 2 provides further evidence for ideological belief bias, with an advantage of a large, but distinct internet sample and with the use of more conversational language in the stimuli. The results in Studies 1 and 2 both showed that participants evaluated entire political arguments based on whether or not the arguments’ conclusions aligned with participants’ ideologies and that this effect was driven by participants’ ideological beliefs. In Study 3, a similar logical reasoning task was administered to a nationally representative sample. Additionally, participants in Study 3 were administered a logical reasoning task that is devoid of political content in order to test whether liberals and conservatives differ in logical reasoning skills.

Study 3

Materials and Methods

Study 3 was conducted on a nationally representative sample of 1,109 participants (48.3% female; M_age = 49.81, SD_age = 17.16). Time-Sharing Experiments for Social Sciences funded the sampling for the study and GfK Group implemented the survey. Due to funding constraints with sample sizes, potential participants who selected “4” on 7-point political ideology scale (1 = very liberal, 7 = very conservative) were screened out, and the final sample was 552 liberals and 557 conservatives.

In Study 3, participants completed two logical reasoning tasks. Both tasks were similar to those described in Study 2; however, the arguments in the first task were nonpolitical in nature. (“Chief Pava believes that anyone who has a khntzor can participate in the Ump Festival. She also believes all Lonbums have khntzors. Therefore, Chief Pava concludes that Lonbums can participate in the Ump Festival”; Norenzayan et al., 2002) After reading extensive directions, participants read four arguments and evaluated their soundness on a 6-point factor scale (1 = logically unsound, very confident to 6 = logically sound, very confident). The first two arguments tested the logic of, “If P, then Q. P. Therefore, Q” (MP), and the next two tested the logic of “If P, then Q. Not Q. Therefore, not P” (MT; see SI).

After the nonideological arguments task, participants completed a test with four syllogisms in which the terms within the premises and conclusions were political in nature and identical to the arguments in Study 2. See the SI for the full set of arguments. The response options and manipulation check were identical to those of Study 2.

Results

As in Study 2, a linear mixed effects model with three-way interactions was conducted to predict logical soundness ratings using the argument ideology, the soundness of the argument, participant ideology, logical structure, and argument topic as predictor variables. Results indicate a main effect of argument soundness such that participants perceived sound arguments as more sound (b = 0.55). Also, there was a main effect of argument ideology (b = 0.22) such that conservative arguments were considered more sound. However, there was no main effect of participant ideology (b = 0.02). Finally, we observed a main effect of the argument’s logical structure such that modus tollens arguments were perceived as less sound (b = −0.43) and of argument’s topic such that arguments about capital punishment were perceived as the least sound (b = −0.65). For two-way interactions, the interaction between participant ideology and argument soundness was significant (b = −0.08) and the interaction between argument soundness and argument ideology was not significant (b = −0.07).

As in Studies 1 and 2, participants in Study 3’s nationally representative sample displayed evidence of ideological belief bias, although the effect was somewhat less pronounced than in the previous two studies. In line with Studies 1 and 2, there was a significant two-way interaction between participant ideology and argument ideology on judgments of argument soundness, b = 0.08, suggesting that evaluations of the logical syllogisms depended, in part, on their ideological leanings. In Study 3, however, participants with greater conservatism rated arguments with conservative conclusions as more sound (b = 0.08, p < .001) but did not differ on their ratings for arguments with liberal conclusions (b = −0.02, p = .316). These effects were again independent of the soundness of the argument, as indicated in the three-way interaction (b = 0.04). See Table 8 for full results.

Table 8.

Study 3 Linear Regression With Three-Way Interaction Predicting Perceived Logical Soundness.

Predictor	b	SE	95% Confidence Interval
PI	.02	.03 (.433)	−0.03	0.07
AS	.55	.07 (<.001)	0.42	0.69
PI × AS	−.08	.04 (.022)	−0.16	−0.01
AI	.22	.07 (.002)	0.08	0.35
PI × AI	.08	.04 (.036)	0.01	0.15
AS × AI	−.07	.10 (.454)	−0.26	0.18
PI × AS × AI	.04	.05 (.404)	−0.06	0.14
Logical structure	−.43	.05 (<.001)	−0.53	−0.33
Argument’s topic	−.14	.02 (<.001)	−0.18	−0.09
Abortion	.06	.07 (.383)	−0.07	0.19
Capital punishment	−.65	.07 (<.001)	−0.79	−0.52
Government intervention	−.22	.07 (<.001)	−0.35	−0.09

Note. p Values are given in the parentheses. Base argument topic is affirmative action. PI = participant ideology; AS = argument soundness; AI = argument ideology.

On average, participants correctly judged 55.93% of the syllogisms. We again conducted a three-way logistic regression to predict the accuracy of participant’s judgment. There is no main effect of participant ideology (OR = 0.97). There was, however, a main effect of argument ideology where arguments with conservative conclusions had worse accuracy (OR = 0.83), argument soundness where sound arguments had greater accuracy (OR = 2.80), and argument’s logical structure where modus tollens arguments had worse accuracy (OR = 0.48). And, there was no main effect of argument topic (OR = 0.98). There was no two-way interaction between participant ideology and argument soundness (OR = 0.98) and between participant ideology and argument ideology (OR = 0.95). There was a significant interaction between argument soundness and argument ideology (OR = 1.33).

Most importantly, there was a three-way interaction between participant ideology, argument soundness, and argument ideology (OR = 1.17). As political conservatism increased, syllogisms with conservative conclusions were more likely to be evaluated correctly if they were sound (b = 0.01, p = .054) and less likely to be evaluated correctly if they were unsound (b = −0.02, p = .020). However, as political conservatism increased, there was no difference for syllogisms with liberal conclusions that were sound (b = −0.01, p = .230) or unsound (b = −0.01, p = .485; see Figure 3; Table 9 for full results).

Figure 3.

Study 3 results for the likelihood of correct responses for sound liberal arguments, sound conservative arguments, unsound liberal arguments, and unsound conservative arguments for liberal and conservative participants. Error bars are 95% CI.

Table 9.

Study 3 Logistic Regression With Three-Way Interaction Predicting Accuracy of Participant’s Judgments.

Predictor	Odds Ratio	RSE	95% Confidence Interval
PI	0.97	.03 (.416)	0.91	1.04
AS	2.80	.27 (<.001)	2.33	3.38
PI × AS	0.98	.05 (.761)	0.89	1.09
AI	0.83	.07 (.019)	0.71	0.97
PI × AI	0.95	.04 (.201)	0.87	1.03
AS × AI	1.33	.16 (.016)	1.05	1.68
PI × AS × AI	1.17	.07 (.011)	1.04	1.33
Logical structure	0.48	.03 (<.001)	0.43	0.54
Argument’s topic	0.98	.03 (.497)	0.94	1.04
Abortion	0.99	.09 (.958)	0.83	1.20
Capital punishment	0.82	.08 (.052)	0.68	1.00
Government intervention	1.00	.08 (.951)	0.86	1.18

Note. p Values are given in the parentheses. Base argument topic is affirmative action. PI = participant ideology; AS = argument soundness; AI = argument ideology.

Discussion

The key ideological belief bias effects observed among liberals and conservatives in Studies 1 and 2 replicated in a nationally representative sample. Participants’ evaluations of the logical soundness of political arguments were influenced by their ideologies, and these judgments were driven by participants’ specific ideological beliefs.

Our analysis of belief bias by liberals and conservatives may naturally raise the question, “are liberals and conservatives differently susceptible to the bias”? However, our data are not well suited to answer such questions conclusively. In a recent meta-analysis comparing biased assimilation between liberals and conservatives, Ditto and colleagues (2018) identified a study’s stimulus match as a critical experimental factor necessary for generalizing ideological differences in bias. In other words, the stimuli evaluated by liberals and conservatives must be equivalent in their informational content, separate from their partisan framing. Although we attempted to match the stimuli that measured liberal and conservative belief bias in as many ways as possible, it is likely that the arguments we chose were not equally polarizing for liberals and conservatives. For example, in Study 3, the mean difference in agreement ratings between arguments with liberal and conservative conclusions for liberal participants was 0.39, t(1,050) = 7.40, p < .001, 95% CI [0.29, 0.49], and the corresponding difference for conservatives was 1.48, t(1,050) = 30.00, p < .001, 95% CI [2.29, 3.77]. Further, we observed inconsistent ideological differences in our samples: Study 1 participants with increasing degrees of political conservatism were more likely to evaluate syllogisms correctly (OR = 0.94), but there was no main effect of participant ideology in Study 2 (OR = 1.00) or Study 3 (OR = 0.97). Finally, participants from the nationally representative sample in Study 3 did not show differences in their correctness on nonideological content, b = 0.01, t(1,109) = 0.41, p = .965, 95% CI [−0.04, 0.07]. Taken together, our results are well suited to show that there is ideological belief bias on both the left and right, but the evidence is not sufficient to generalize relative differences in the degree to which this bias occurs between liberals and conservatives.

General Discussion

In three high-powered studies (total N = 2,898), we observed evidence that people’s political beliefs impact their ability to reason logically about political issues. Biased beliefs about political arguments’ conclusions caused liberals and conservatives to make predictable patterns of errors. Specifically, participants evaluated the logical structure of entire arguments based on whether they believed in or agreed with the arguments’ conclusions. Although these effects were modest in magnitude, they were persistent: We observed these biases in evaluations of both classically structured logical syllogisms and conversationally framed political arguments, across a variety of polarized political issues, and in large Internet and nationally representative samples.

These results demonstrate that belief bias is a pervasive problem in political reasoning that affects both liberals and conservatives. Participants failed to overcome ideological belief bias effects even after a training session on logical reasoning and explicit instructions on how to evaluate logical soundness. These studies also emphasize that belief bias can be particularly problematic in the political domain because of preexisting differences in partisans’ political beliefs. That is, political opponents’ judgments of logical soundness were biased in opposite directions, meaning that liberals and conservatives came to disagree not only about their political beliefs but also in their perceptions of what it means to be logical at all.

Future research should examine whether and how individual differences might mitigate political belief bias. It is possible that indicators of cognitive ability, such as high numeracy, would improve overall performance. However, there is also evidence that enhanced cognitive abilities may exacerbate, rather than mitigate, our biases in a politically motivated setting (e.g., Kahan et al., 2017).

Returning to our introductory question, is logical reasoning the antidote to political disagreement, or is it the poison? Our results suggest that it might be both. On the one hand, logical reasoning led participants to evaluate a majority of arguments in each study correctly, regardless of their political orientation. On the other hand, liberals and conservatives frequently and predictably disagreed in their evaluations of logical soundness. Conclusions and arguments that appear believable and therefore logically sound to liberals appear unbelievable and therefore unsound to political conservatives, and vice versa, regardless of the actual soundness of the arguments. While partisanship alone may push liberals and conservatives apart in their beliefs, ideological belief bias then pushes liberals and conservatives apart even in the perceived logic underlying those political beliefs.

Despite this, a more optimistic view of our results is that understanding these predictable biases could ultimately improve political reasoning. Consistent with bias blind spot research (Pronin et al., 2002), reasoners appear to be better at identifying biased reasoning in others than in themselves. That is, liberals were better at identifying flawed arguments supporting conservative beliefs and conservatives were better at identifying flawed arguments supporting liberal beliefs. A takeaway from this research, then, may be that reasoners should strive to be epistemologically humble. If logical reasoning is to serve as the antidote to the poison of partisan gridlock, we must begin by acknowledging that it does not merely serve our objectivity, but also our biases.

Supplemental Material

SPPS829059_suppl_mat - (Ideo)Logical Reasoning: Ideology Impairs Sound Reasoning

SPPS829059_suppl_mat for (Ideo)Logical Reasoning: Ideology Impairs Sound Reasoning by Anup Gampa, Sean P. Wojcik, Matt Motyl, Brian A. Nosek and Peter H. Ditto in Social Psychological and Personality Science

Footnotes

Authors’ Note

Anup Gampa and Sean P. Wojcik contributed equally to the work and share joint first authorship.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The nationally representative study was made possible by funding from Time-Sharing Experiments for Social Sciences (TESS; NSF Grant 0818839, Jeremy Freese and James Druckman, Principal Investigators).

Supplemental Material

The supplemental material is available in the online version of the article.

Note

References

Cohen

G. L.

(2003). Party over policy: The dominating impact of group influence on political beliefs. Journal of Personality and Social Psychology, 85, 808.

Ditto

P. H.

Liu

Clark

C. J.

Wojcik

S. P.

Grady

Chen

E. E.

… Zinger

. (2018). At least bias is bipartisan: A meta-analytic comparison of partisan bias in liberals and conservatives. Perspectives in Psychological Science. doi: 10.1177/1745691617746796.

Evans

J. S. B.

Barston

J. L.

Pollard

(1983). On the conflict between logic and belief in syllogistic reasoning. Memory & Cognition, 11, 295–306.

Feather

N. T.

(1964). Acceptance and rejection of arguments in relation to attitude strength, critical ability, and intolerance of inconsistency. The Journal of Abnormal and Social Psychology, 69, 127.

Gervais

W. M.

Shariff

A. F.

Norenzayan

(2011). Do you believe in atheists? Distrust is central to anti-atheist prejudice. Journal of Personality and Social Psychology, 101(6), 1189–1206.

Henle

Michael

(1956). The influence of attitudes on syllogistic reasoning. The Journal of Social Psychology, 44, 115–127.

Janis

I. L.

Frick

(1943). The relationship between attitudes toward conclusions and errors in judging logical validity of syllogisms. Journal of Experimental Psychology, 33, 73.

Kahan

D. M.

Hoffman

D. A.

Braman

Evans

(2012). They saw a protest: Cognitive illiberalism and the speech-conduct distinction. Stanford Law Review, 64, 851.

Kahan

D. M.

Peters

Dawson

E. C.

Slovic

(2017). Motivated numeracy and enlightened self-government. Behavioural Public Policy, 1, 54–86.

10.

Lord

C. G.

Ross

Lepper

M. R.

(1979). Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37, 2098.

11.

Morgan

J. J.

Morton

J. T.

(1944). The distortion of syllogistic reasoning produced by personal convictions. The Journal of Social Psychology, 20, 39–59.

12.

Newstead

S. E.

Pollard

Evans

J. S. B. T.

Allen

(1992). The source of belief bias effects in syllogistic reasoning. Cognition, 45, 257–284.

13.

Norenzayan

Smith

E. E.

Kim

Nisbett

R. E.

(2002). Cultural preferences for formal versus intuitive reasoning. Cognitive Science, 26, 653–684.

14.

Oakhill

J. V.

Johnson-Laird

P. N.

(1985). The effects of belief on the spontaneous production of syllogistic conclusions. The Quarterly Journal of Experimental Psychology, 37, 553–569.

15.

Pronin

Lin

D. Y.

Ross

(2002). The bias blind spot: Perceptions of bias in self versus others. Personality and Social Psychology Bulletin, 28, 369–381.

16.

Sells

S. B.

(1936). The atmosphere effect: An experimental study of reasoning. Archives of Psychology (Columbia University), 200, 72.

17.

Smith

C. T.

Ratliff

K. A.

Nosek

B. A.

(2012). Rapid assimilation: Automatically integrating new information with existing beliefs. Social Cognition, 30, 199–219.

18.

Thouless

R. H.

(1959). Effect of prejudice on reasoning. British Journal of Psychology, 50, 289–293.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB