Abstract
This study reports the effects of two debiasing strategies on the complexity of people’s thinking on a controversial policy issue – the question of Scottish independence. I start from the well-researched assumptions of motivated reasoning theory that individuals tend to protect their beliefs, are often not willing to hear the other side and fail to integrate contrasting arguments and different perspectives in their political considerations – although considering different viewpoints is a fundamental normative requirement for democratic decision-making. Two different debiasing techniques, which are meant to counteract this tendency and to evoke more integrative and complex thinking, were tested experimentally: a cognitive and a motivational strategy. The experiment was situated in the context of the Scottish independence referendum. The expectation of accountability – having to justify one’s opinion in front of unknown others – significantly enhanced integrative complexity of thinking about the issue, while inducing subjects to consider the opposite had no significant effect. Opinion strength and political knowledge did not affect the treatment effects significantly.
Biases in political thinking have received much attention in political psychology, and their pervasiveness is by now well documented (Lau and Redlawsk, 2001; Leeper and Slothuus, 2014; Redlawsk, 2002). As Taber and Lodge (2006: 576) have noted, in politics, ‘all reasoning is motivated’. However, new questions arise now: What is a considered and unbiased opinion? Which psychological mechanisms underlie motivational and cognitive biases? And how can such biases be counteracted? These questions have received much less attention so far (Bolsen et al., 2014). As Druckman (2012) pointed out, motivated reasoning occurs under certain conditions and depends on an individual’s motivation and the context. He considers the role of different conditions in prompting accurate versus motivated reasoning a ‘particularly fruitful area in need of more study’ (Druckman, 2012: 206). This study approaches these questions by testing two different debiasing mechanisms. To this purpose, a laboratory opinion experiment was run in the forefront of the Scottish independence referendum. I found that announcing that subjects will have to justify their opinions in a group discussion at the end of the experiment (‘accountability’) has a positive effect on the considerateness of subjects’ opinions. A considered opinion is defined as one which integrates different perspectives of an issue and which is well justified with substantive arguments. This study adds to a recent debate in political psychology on the conditions, under which motivated reasoning occurs in offering possible strategies to reduce it. This article proceeds as follows: I start by explaining the main concepts and by developing hypotheses. Next, I describe the experimental design. Finally, I present and discuss the results.
Motivated Reasoning and Considered Opinions
Citizens’ participation in political decisions constitutes the very essence of democracy. Since democracy exists, citizen competence to participate in political decisions has been questioned though. This is true for representative democracy, but the scepticism is even greater for direct democratic decisions where citizens decide over policy matters themselves. Citizens’ low political knowledge levels have often been criticised in public opinion research (Converse, 1964; Delli Carpini and Keeter, 1996). More recently, however, the question arose whether high political knowledge is an adequate measure of citizen competence at all – as political scientists discovered the concept of motivated reasoning (Kunda, 1990; Lau and Redlawsk, 2006; Lodge and Taber, 2013; Redlawsk, 2002; Redlawsk and Lau, 2005; Taber and Lodge, 2006). Motivated reasoning describes a process whereby (political) reasoning is not necessarily led by a goal to be accurate (an accuracy goal) – which would correspond to a normative ideal – but more often by directional goals, such as a personal motivation to protect existing beliefs or to follow a certain party line. A directional goal is defined as ‘when a person is motivated to arrive at a particular conclusion’ (Kunda, 1999: 236). This means that citizens select and process information in a biased way, preferring information that is consistent with their existing knowledge and disregarding and/or devaluating incompatible information.
Recent studies have found evidence for motivated reasoning processes on political opinions on different issues such as the 2003 Iraq war (Gaines et al., 2007), political corruption in Spain (Anduiza et al., 2013), climate change policies (Hart and Nisbet, 2011), perceptions of the economy (Evans and Andersen, 2006; Gerber and Huber, 2010; Ramirez and Erickson, 2014) or welfare and health-care policies (Bullock, 2011; Cohen, 2003). Leeper and Slothuus (2014) give an excellent review of the recent debates on motivated reasoning.
Given the intense research interest in motivated reasoning and biased thinking, it is surprising that only little thought has been devoted to the question how to counteract these biases and how to foster considered opinions. So far, political scientists have concentrated on analysing various factors moderating biased thinking – in particular political sophistication, opinion strength, elite polarisation and message repetition (Druckman et al., 2013). Only very recently, the question about the impact of motivations and cognitive mechanisms at the time of opinion formation was raised (see Bolsen et al., 2014; Leeper and Slothuus, 2014). This study adds to this current debate by testing the effect of two different experimental stimuli on the considerateness of political opinions.
Why is it important to study ‘considered opinions’? This is a concept, which has not received much attention in political psychology and public opinion research so far. I think it is worth exploring this concept more in depth for at least two reasons. First, on a theoretical level, a political decision taken by citizens who hold well-considered opinions is more legitimate. On one side, considered opinions are supposed to be more resistant to manipulation by elites and news media. The concern that elites and opinion leaders might manipulate public opinion is an old one (Katz, 1957; Schattschneider, 1975; Schumpeter, 1950). More recently, the large literature on framing effects has shown (Chong, 2013; Chong and Druckman, 2007; Druckman et al., 2013; Sniderman and Theriault, 2004) how citizens’ opinions are often malleable and susceptible to the way political information is presented to them. The problem then is that citizens’ opinions, when they are manipulated, do not anymore reflect their values and political predispositions.
But not only elite manipulation or distorted information can be problematic: as the literature on motivated reasoning summarised above shows, we often follow ‘in-built’ cognitive biases and heuristics when we form our opinions, such as trying to protect our own beliefs or blindly following party cues. These cognitive heuristics can constitute efficiency-enhancing shortcuts, which bring us to a reasonable decision with less effort, as has been argued by some (Bowler and Donovan, 1998; Lupia, 1994; Lupia et al., 2000). But they may also lead us astray, if we blindly rely on our party’s position or if we are not ready to update our opinions in light of new evidence.
Second and related to the first point, on an empirical level, the study of citizen competence has often focused on factual political knowledge (Lavine et al., 2012). The gold standard of the knowledgeable citizen is called in question though, by studies, which find that often the more knowledgeable are more prone to cognitive biases such as motivated reasoning. Thus, it is important that opinions are not only informed but also well considered (Fishkin, 2009; Yankelovich, 1991). Therefore, I present an attempt to move in the direction of measuring ‘considered opinions’ in order to assess citizen competence instead of relying exclusively on factual knowledge scores. This study presents a suggestion of how such a considered opinion can be operationalised empirically.
The questions how to define and measure considered opinions are not trivial. One option, which is compatible with a deliberative model of democracy, is to set the consideration of different viewpoints and alternative arguments, as well as the ability to justify one’s position, as a desirable standard for citizens’ participation in democratic decisions (see, for example, Bächtiger and Wyss, 2013; Fishkin, 2009; Gerber et al., 2014; Lavine et al., 2012; Luskin et al., 2002; Minozzi et al., 2012). Thus, for this study, a considered opinion is defined as one which integrates arguments of different sides and one which can be well justified by substantive reasons.
Debiasing Strategies
On the level of individual citizens, we lack knowledge on how exactly – through which psychological mechanisms – exposure to differing viewpoints affects opinion formation, and in particular, through which mechanism belief protection biases are reduced. There are at least two possible experimental stimuli, which might foster considered opinions. Both have been discussed mainly in social psychology under the heading of ‘debiasing strategies’ (Fischhoff, 1982). Debiasing strategies are defined as techniques that eliminate bias or diminish its intensity or frequency. Thereby ‘the goal of debiasing techniques should be to help people grasp and appreciate alternative points of view, not necessarily to accept them as equally valid or moral’ (Lilienfeld et al., 2009). First, the fact that participants are explicitly induced to consider different viewpoints on an issue might enhance the cognitive accessibility of opposite information. Social psychologists have argued that a biased preference for pre-existing opinions can be counteracted by a ‘consider-the-opposite’ strategy (Fischhoff, 1982; Hirt et al., 2004; Lord et al., 1984; Mussweiler et al., 2000). Thereby, individuals are instructed to think explicitly about the arguments of the opposite side, and this process is supposed to make this information accessible in memory, leading to more balanced, unbiased and well-justified opinions. From this research, I derive a first hypothesis:
H1: Being induced to consider arguments of the opposite side leads to more considered opinions.
The consider-the-opposite strategy is a cognitive debiasing mechanism, as it is assumed to enhance the cognitive accessibility of diverging information in memory. The assumption behind H1 is that individuals in general are motivated to be accurate, but they often disregard attitude-incongruent information simply because it requires more cognitive effort to process.
Second, the conventional approach within psychology for inducing an accuracy motivation is to tell them they would later have to justify the reason for their judgements. Accountability – that is, the pressure to justify one’s opinion to others – was found to ‘motivate complex (effort-demanding) information processing by increasing the importance of avoiding “bad” judgements (embarrassment, loss of self-esteem) and of making “good” judgements (praise, status)’ (Tetlock, 1983: 74). Subjects expecting accountability were found to be more likely to consider various options, more receptive to new evidence, more tolerant for inconsistency and to focus more on the content of a message than on its source (Bolsen et al., 2014; Chaiken, 1980; Green et al., 2000; Klahr, 2013; Kruglanski and Freund, 1983; Tetlock, 1983). Based on these findings, I derive a second hypothesis:
H2: The expectation of accountability leads to more considered opinions.
In contrast to the first strategy, accountability constitutes a motivational debiasing strategy, as it induces a motivation to be accurate. The assumption behind hypothesis 2 is that individuals in general are rather motivated to protect existing beliefs than to be accurate. Inducing an accuracy motivation will lead subjects to ‘evaluate political arguments with the hope of reaching an outcome that is the correct or otherwise best conclusion’ (Bolsen et al., 2014: 238). As the authors further note, individual motivations in evaluating information in the context of opinion formation constitute ‘one factor that has not been examined by political scientists’ (Bolsen et al., 2014: 237). Furthermore, Leeper and Slothuus (2014: 149) argue, ‘demonstrating the impact of motivations is critical to disentangling whether goals [i.e. motivations] as opposed to some other cognitive processes are at work’. This is what this study aims at.
Beyond the general effects of the two debiasing techniques, the differential effects dependent on subjects’ opinion strength and political sophistication are interesting. Some studies have found motivated reasoning and belief protection to be just as strong (Cohen, 2003; Taber et al., 2009) or even stronger in highly politically knowledgeable individuals than in low knowledgeable (Meffert et al., 2006; Mutz, 2006; Taber and Lodge, 2006), while others found a reduction of bias among highly knowledgeable (Anduiza et al., 2013; Arceneaux, 2008; Kam, 2005). Furthermore, various studies have found an attitude strength effect, meaning that citizens holding stronger attitudes on an issue, because the issue is of high personal relevance to them, are more prone to motivated reasoning (Druckman et al., 2012; Holbrook et al., 2005; Taber and Lodge, 2006). It is not clear, however, how debiasing effects might be moderated by these two variables.
I expect a negative political sophistication effect, such that subjects with high political knowledge react less strongly to the treatments than low-knowledge subjects. Because they dispose of more political knowledge resources and because they are better capable of processing complex information, I expect more politically sophisticated subjects to be better able to protect and justify their priors even when induced to more balanced and considerate thinking.
In addition, I expect a positive attitude strength effect, that is, I expect subjects who hold strong opinions on the issue at stake to react more sensibly to the debiasing treatments. Subjects holding strong opinions are the ones who consider the issue as personally relevant, so these are the ones who are most involved and who have most at stake. For these reasons, I expect them to be more readily willing to react to the stimuli which induce consideration and complex thinking on the issue:
H3: Both debiasing mechanisms have a weaker effect on subjects with high political knowledge.
H4: Both debiasing mechanisms have a stronger effect on subjects for whom the issue at stake is of high personal relevance.
Summing up the preceding paragraphs, ideally, citizens would consider different arguments before making political judgements, but in reality, belief protection motivation often hinders consideration of different viewpoints. The question is then, how can citizens’ readiness to think through and integrate different arguments be enhanced, and thus how can considered opinion be fostered? 1 The issue considered in this study, the question of Scottish independence, is a real-life, salient and controversial issue – an issue which motivates people and where listening to the other side is particularly difficult. For such an issue, it might also be more difficult to find debiasing effects. Nevertheless, it is important to understand how citizens reason on such divisive issues.
Research Design
A laboratory experiment was conducted before the referendum on Scottish independence, taking place in September 2014. The Scottish independence referendum was one of few national-level referendums, where people decide over a substantial political change, which affects their lives. The independence question is a highly salient issue, as well as a very controversial one where there are two clearly contrasting positions – a situation in which listening to the other side and holding considered opinions is difficult. Moreover, the strong emotional nature of the debate, with arguments on Scottish versus British identity figuring prominently, makes this case an exceedingly hard test of being able to change complexity of thinking through experimental intervention. 2 The question posed to citizens was ‘Should Scotland be an independent country?’. The debate involved not only questions of Scottish identity but also economic concerns and other pragmatic issues such as EU-membership, the future currency, national defence, higher education funding and welfare questions.
The experiment took place during 1 week in April 2014 and involved 179 students from the University of Edinburgh. All subjects were eligible, and 75% reported their intention to vote in the referendum. In order to avoid testing effects by asking subjects for their opinion before and after the treatment, a between-group design involving a control group and two treatment groups was employed. On arrival in the laboratory, the subjects were randomly assigned to the different treatment conditions (the treatments were randomised within sessions). The experiment was programmed with Qualtrics. 3 The experiment consisted of four parts: a pre-treatment questionnaire containing socio-demographic information and a political knowledge quiz; the treatment (reading a set information material on Scottish independence under different instructions); a set of opinion measures on Scottish independence (outcome measures); and, finally, a post-treatment questionnaire containing additional measures. The entire session lasted between 45 and 60 minutes.
Stimulus Material 4
All subjects were exposed to a balanced set of four pro- and four contra-arguments on Scottish independence. Through these readings, a baseline condition was created, where all subjects shared at least a minimal common knowledge base, in order to avoid the problem that the variance in opinion complexity is determined mainly by individuals’ prior knowledge of the issue. In other words, the fact that all subjects shared the information in these readings makes it possible to attribute differences in post-treatment opinion complexity to the respective treatments. Following the procedure used in various framing experiments (Druckman et al., 2013) the relevant arguments were identified through a content analysis of media coverage and expert interviews. In total, 10 UK political scientists working at the European University Institute were asked to rate the direction and strength of various arguments and to list their own thoughts on Scottish independence. Considering their answers, the four most prominent arguments were selected for each side. The arguments were all drawn from print and online publications of Scottish and British newspapers and from official documents by the Scottish and British government. The arguments were of comparable complexities and all were about 200 words long in order to avoid confounding effects on opinions.
Treatments
Subjects in the control condition were instructed to read the stimulus material carefully and to report their opinion afterwards. In order to avoid rank order effects, the arguments were presented in a randomised order in all treatments. In addition, there were two different debiasing treatments. In the ‘consider-the-opposite’ condition, subjects received the same set of arguments to read with an additional instruction, which read:
While reading the articles, please write down all the arguments that would be used by the opposite side, i.e. all the arguments that a proponent of the side opposite to yours would use to convince you. That is, if you tend to support independence, write down the arguments of a pro-unionist and if you tend to oppose independence, write down the arguments of an independence supporter. Please use the paper and pencil on the table in front of you.
In the last treatment group, the At the end of the study, you will be asked to justify your opinion on the issue of Scottish independence to the other participants in a short group discussion. The discussion will take place in this room once everyone is done with the computer tasks. The discussion will help us understand the interpersonal communication of attitudes. Please note that the lab managers will take notes on this discussion, keeping the identity of participants completely anonymous.
A brief group discussion took place at the end of the session, and participants had the chance to express their opinions. 5
Measures
I am interested in two substantive outcome measures indicating considered opinions: first, integrative complexity (IC) of justification and, second, argument strength ratings.
IC of Justifications. 6
The main outcome measure is a justification paragraph written by the subjects, which was coded for IC (Suedfeld et al., 1992). All subjects across all three groups were asked to justify their position on Scottish independence by the following instruction: ‘Please justify your position in one paragraph (4/5 sentences)’. Note that the justification paragraph was written after reading the arguments but before the discussion took place.
High complexity scores indicate considered opinions. IC is a measure composed of two cognitive structural properties: differentiation and integration. While differentiation refers to the number of characteristics or dimensions of a problem that an individual takes into account, integration refers to the development of complex connections among these different characteristics (Suedfeld and Tetlock, 1977; Tetlock, 1983). In this study, I am mainly interested in the occurrence of integration. Low differentiation is reflected by a tendency to focus on only one theme or aspect in the analysis and to rely on simple, one-dimensional rules without qualification (e.g. ‘Independence is economically unviable’), while higher levels of differentiation are reflected in the recognition and acceptance as valid and legitimate of either alternative perspectives (i.e. pro- and contra-independence arguments) or different dimensions of an issue (e.g. support for independence for reasons of national identity, financial wealth and democratic representation). Integration is inferred when different elements are linked conceptually. Integration can be expressed through the recognition of interactive causality, such as the idea that which position is taken depends on the perspective. The recognition of value-trade-offs and the suggestion of solutions in the form of overarching principles of perspective are also signals of integration (e.g. the recognition of a trade-off between stronger democratic representation through independence and the short-term economic risks which can be expected is a sign of integration).
The assessment of IC typically proceeds on a 7-point scale, where 1 implies neither differentiation nor integration, 3 indicates differentiation but no integration and 4–7 signal varying levels of integration of perspectives and/or dimensions. Differentiation is thus a necessary but not sufficient prerequisite for integration. More than 40% of the subjects reached a score of 3, indicating high differentiation (see online appendix), while only few subjects reached a score of 4 or higher, which is indicative of integration of perspectives. In our student sample, thus, giving a highly differentiated justification appears to be an easy task – thus, the interesting question in the present sample is whether subjects were able to integrate different perspectives in their justifications. Therefore, I use the binary distinction between justifications with and without integration as a dependent variable. For this purpose, scores of 4 and higher were coded as 1, and all other scores were coded as 0 (see Tadmor and Tetlock, 2009). Over the entire sample, 81.5% of subjects score 0 (no integration), while 18.5% score 1 (indicative of integration).
The coding was conducted by a certified coder following the coding manual for cognitive complexity (Baker-Brown et al., 1992). 7 To test for the reliability and objectivity of the coding, a second, independent, certified coder re-coded a random sample of 50% of all justification paragraphs. The two coders achieved a high interrater agreement of α = 0.89.
Argument Strength Rating
After reading each argument, subjects were asked to rate its strength on a 10-point scale (where 0 means extremely weak and 10 extremely strong). Thereby, subjects were explicitly instructed to focus on the strength of the argument and not on whether they agree or disagree with it. This measure was used in previous studies to measure a ‘prior attitude effect’ on argument ratings (Petty and Krosnick, 1995; Taber and Lodge, 2006). With regard to the debiasing treatments, it captures whether subjects are willing to give some credit to the arguments of the opposite side, even if they do not agree with them, which is taken as an indicator of a considered opinion. The mean strength rating for the whole sample was 6.04 for pro-arguments and 6.15 for con-arguments.
Additional measures included political knowledge, measured by an index based on a political knowledge quiz which was part of the pre-treatment questionnaire (ranging from 0 to 17 points). General interest in politics was measured on a 5-point scale. Opinion strength was measured with a conventional item asking: ‘Compared to how you feel about other public issues – such as immigration, environment, foreign policy, etc. – how strong are your feelings regarding the issue of Scottish independence?’ (0- to 10-point response scale with 0 indicating not at all strong and 10 very strong). Table A1 (online appendix) shows summary statistics of the additional variables.
Balance Tests
To check whether the randomisation procedure resulted in a balanced distribution of individuals over the experimental groups, I conducted a series of balance tests on pre-treatment variables. 8 These tests did not yield any significant differences between treatment groups, except for gender, which is not balanced over groups. As none of the hypotheses concerns gender effects, I decided to run the experiment on the whole sample and to introduce gender as a control variable in the robustness tests. Table A2 in the online appendix shows description of the composition of experimental groups.
Analysis
Integrative Complexity (IC)
In the first part of the analysis, I test whether the treatments have an effect on the IC of subjects’ justifications for their position on Scottish independence. Let us first examine the difference in IC across experimental groups for all subjects in the sample: As Figure 1 shows, in the accountability group, as compared to the control group, there is more than double the share of high IC scorers (12% vs 25%, p > 0.04). 9 This difference is substantive and statistically significant at the 5% level (Table A3 in the online appendix summarises the results of the difference test between groups). In the consider-the-opposite group, in contrast, there are 7 percentage points more subjects with a high IC score as compared to the control group, this difference is not statistically significant. 10
Proportion of High IC Scores over Experimental Groups.
Looking at subsamples of subjects divided by their political knowledge levels shows a similar picture (see Figure 2 and Table A3): The consider-the-opposite treatment has small positive effects on IC, with an 8% rise in high IC scores for highly knowledgeable and a 5% for less knowledgeable individuals (not significant). Accountability increases cognitive complexity from 10% to 22% (not significant) for the less knowledgeable and from 14% to 28% (p < 0.1) for the highly knowledgeable subjects. Even though the treatment effect is significant only for high knowledge subjects, the substantive effect is comparable for both subgroups. While more knowledgeable subjects show higher IC scores in general, there are no substantial differences in the treatment effects between subjects with high versus low political knowledge levels.
Proportion of High IC Scores over Experimental Groups by Political Knowledge.
The reaction to treatment looks different, however, for subjects with strong opinions, who react more strongly to both treatments (see Figure 3 and Table A3). While there are no significant differences across treatment groups for subjects with weak opinions, subjects holding strong opinions in the consider-the-opposite group show 16 percentage points more high IC scores (p < 0.04) than in the control group. In the accountability group, this share even rises by 18%–26% (p < 0.03). This result has to be interpreted with caution though, as, due to the small sample size, none of the interaction effects between the treatments and the knowledge and opinion strength variables are statistically significant 11 (see Table A4 in the online appendix).
Proportion of High IC Scores over Experimental Groups by Opinion Strength.
Robustness of Findings on IC
The difference-in-proportion tests presented above provide an easily understandable technique to assess the differences in outcome variables across treatment groups. In addition, I tested the robustness of these findings, controlling for gender, the only variable that was not balanced over experimental groups, as the balance test above has shown. This procedure is suitable to increase the precision of the estimation (Angrist and Pischke, 2008).
The results are presented in Table 1. Model 1 presents the results of the treatment effects without controls, while in model 2 I include gender. Table 1 shows that accountability has a strong and robust significant positive effect on IC.
Robustness Tests Integrative Complexity.
SE: standard error.
Logit estimates of binary IC measure (high = 1/low = 0); standard errors are in parentheses. Statistically significant differences (p < 0.1) are in bold. The left-hand panel shows the results for the accountability treatment, and the right-hand panel shows the results for the consider-the-opposite treatment. The samples include only the control group and the respective treatment group.
Argument Strength Ratings
In the second part of the analysis, I test for an attitude bias on argument strength ratings (0- to 10-point scale), by comparing the average strength ratings for attitudinally congruent and incongruent arguments. I expect subjects to rate attitudinally congruent arguments as stronger than attitudinally incongruent arguments. The attitude is defined through the subjects’ reported vote intention with subjects answering yes labelled as ‘Pro’ and subjects answering no as ‘Con’ independence (the undecided were excluded in order to draw a clear distinction in attitude).
Figure 4 displays the mean pro- and con-argument ratings by attitude, broken down by opinion strength (left side) and political knowledge (right side). Dark bars represent average strength ratings for pro-arguments, light bars for con-arguments; the first pair of bars shows the responses of proponents of the issue, and the second pair shows responses of opponents. As we can see, over all subsamples, congruent arguments are rated as substantively stronger than incongruent arguments, indicating an attitude congruence bias on argument ratings as expected. The differences in mean strength ratings range from 1.3 to 3.1 on an 11-point scale. The effect is stronger for subjects with high political knowledge and holding strong opinions.
Argument Strength Ratings, by Opinion Strength and Political Knowledge.
Table 2 reports regression analyses of independence attitude on argument strength ratings. Subjects’ overall argument strength rating (the dependent variable in Table 2) was computed as the sum of ratings of the pro-arguments minus the sum of ratings of the con-arguments (ranging from −40 to +40). To test for an attitude effect, these gaps in argument strength ratings were regressed on attitude. In all models, the coefficient of independence attitude on argument strength ratings is positive and highly statistically significant, indicating that subjects rate attitudinally congruent arguments as significantly stronger than incongruent ones (see Taber and Lodge, 2006, for an analogous procedure). Note that these results are obtained despite the explicit instruction to abstract from the own attitude and concentrate on the strength of arguments and despite the fact that over the whole sample, as well as in pre-tests, the pro- and con-arguments were rated as almost equally strong.
OLS Regressions of Argument Strength Ratings on Independence Attitude, Including Interaction Effects of Treatments and Moderators. a
OLS: ordinary least squares; SE: standard error.
Dependent variable: argument strength rating (−40/+40). Standard errors are in parentheses. Statistically significant differences (p < 0.1) are in boldface; the effect of the treatment groups was calculated as compared to the control group only, excluding the subjects in the other treatment group.
Three-way interactions between moderators, treatments and attitude did not yield any significant effects. They are not reported here, as the small sample size makes it difficult to get reliable results for three-way interactions. Test on subsamples within treatment groups, however, yielded a stronger debiasing effect among low-knowledge subjects in both treatment groups, indicating that here debiasing is stronger for subjects with low knowledge. Also, these subsamples are very small though, which renders the regression results less reliable.
p<0.05; **p<0.01; ***p<0.001.
Models 2 and 3 include coefficients for the effects of the treatments as well as interaction terms between treatments and attitude. Looking at the interaction terms, we see that the consider-the-opposite treatment did not reduce the gap in mean strength ratings between pro- and con-independence subjects. Accountability reduced the gap by almost 4 points (on a range of −40/+40), but this result is not statistically significant. There was a reduction of the gap in the accountability group.
By how much was the gap in ratings reduced? The difference in mean argument strength rating between subjects who are pro- and con-independence is 17.6 points in the control group and 14.3 points in the accountability group. Thus, accountability reduces the gap in predicted ratings by almost 4 points (on a range of −40/+40). 12 This interaction effect is not statistically significant however, and therefore, we cannot conclude with sufficient certainty that accountability leads subjects to give the other side more credit.
Discussion
This study finds that the expectation of accountability has a strong positive effect on the IC of subjects’ justifications – being held accountable led to more considered opinions as defined in the Introduction. Considering-the-opposite, however, does not have a statistically significant effect on cognitive complexity. I conclude that hypothesis 2 can be accepted, while hypothesis 1 needs further testing.
The finding that accountability increases cognitive complexity confirms that there is clearly a motivational component to political thinking – as soon as we are motivated to be accurate because we expect to justify ourselves to others (and want to avoid embarrassment), we start integrating different perspectives in our opinions, making them more complex. It is thus not merely the presence and availability of even-handed information per se (which was given in all three groups) that enables people to make complex judgements – the informational environment must meet the appropriate motivation or goal in an individual – that is a goal to be accurate.
Debiasing works in the same way for highly politically knowledgeable subjects as for the less knowledgeable. Thus, hypothesis 3 has to be rejected, and political knowledge does not moderate the effect of debiasing techniques. At the same time, we find some preliminary evidence for hypothesis 4: The treatments have stronger effects on subjects with strong opinions. This latter finding has to be interpreted with caution though, as due to the small sample size, we are not able to conclusively determine the conditional treatment effect. Replication studies are necessary on this point.
As to the second outcome measure, the analysis of argument strength ratings confirms the presence of an attitude congruence effect in argument strength ratings, a finding in line with previous studies (e.g. Lord and Ross Lepper, 1979; Taber and Lodge, 2006). Independence supporters consistently rate pro-independence arguments as stronger than con-arguments, and vice versa, independence opponents rate con-arguments as stronger. This occurs even though subjects were explicitly instructed to abstract from their opinion and rate argument strength in an objective way, and given the fact that in expert pre-tests, pro- and con-arguments were rated as equally strong. The bias is particularly strong for subjects with high political knowledge and with strong opinions – again in line with previous findings. Here again, in the accountability group, subjects rated arguments in a more balanced way. This suggests that accountability had a minor debiasing effect, even though not a statistically significant one, and thus provides some support for hypothesis 2 but not for hypothesis 1.
The fact that debiasing strategies did not have a stronger effect on strength ratings is in line with Taber and Lodge’s (Taber and Lodge, 2006; Lodge and Taber, 2013) repeated finding that instructions to be balanced and even-handed in the evaluation of argument strength are ineffective. On one hand, this ineffectiveness might be taken as evidence for the pervasiveness of directional reasoning (see Bolsen et al., 2014). However, it might also be attributed to the difficulty of the task: Abstracting from our own opinion while reading arguments, and rating them as strong or weak from an ‘objective’ or ‘neutral’ perspective, seems like a very ‘unnatural’ or ‘artificial’ requirement – and is thus probably too difficult as an experimental task. Seen from yet another perspective, one could question whether this can be called a ‘bias’ at all – the result can simply mean that subjects clearly prefer one option (independence or union) over the other and that they, therefore, believe consistently that the arguments in favour of their preferred option are stronger than the counterarguments – not so surprising after all. I suggest to be cautious with labelling confirmatory argument strength ratings as ‘prior attitude effects’ as some scholars have done (Bolsen et al., 2014; Druckman et al., 2012; Taber and Lodge, 2006). Certainly, a longitudinal study would be better suited to demonstrate that a cognitive bias is at work.
How can we combine the findings that that accountability has a strong effect on IC of justifications but a less clear effect on the differential bias in argument strength ratings? One possible explanation is that accountability affects more the ‘talk’ than the attitude of individuals. When held accountable, subjects adjust the complexity of their justifications and integrate arguments of the other side, while in reality, they have not come to agree more with the other side. This might mean that they try to appear more conciliatory than they really are – a sort of ‘keeping up the appearances’ – reaction.
In sum, however, when being held accountable, subjects start to integrate contrasting arguments in their own opinion formation – in particular when expecting to justify themselves in a group. Exactly, this – the consideration of diverse arguments and perspectives and their integration in the opinion formation process – is a crucial normative requirement in democratic decision-making. 13
Conclusion
Motivated reasoning and belief protection are pervasive phenomena in political thinking. Bolsen et al. (2014: 245) have defined directional reasoning as ‘the default method to forming evaluations in political contexts’. At the same time, the authors emphasise that ‘it is time scholars move beyond testing moderators and/or documenting the presence of partisan motivated reasoning and work towards a more complete theory of […] motivated reasoning’ (Bolsen et al., 2014: 252). So far, little is known about the psychological mechanisms underlying the phenomenon of motivated political thinking. And even less is known about the question how to counteract it.
This study takes a step in this direction by testing strategies to counteract motivated reasoning. Accountability to others – which in turn induces an accuracy motivation – is key in enhancing the complexity of political thinking (see also Bolsen et al., 2014; Leeper and Slothuus, 2014). Mere exposure to balanced information is not enough – a balanced information environment must encounter an individual motivated to be accurate. More complex opinions come closer to a central normative requirement of democratic decision-making.
Thus, creating a social environment, where subjects have to discuss their views with diverse others will enhance the considerateness of their political opinions – a claim which has been made for a long time by deliberative democrats (e.g. Mutz, 2006). However, the cognitive and motivational underpinnings of this claim have not been analysed yet. In fact, so-called deliberative mini-publics (Farrar et al., 2010; Fishkin and Luskin, 2005; Niemeyer, 2011; Smith, 2009) entail both debiasing elements tested in this study: the explicit consideration of opposite arguments, as participants are invited to listen to exponents of different camps, as well as group discussions, where participants themselves have to justify their own positions. Different authors have emphasised the importance of better understanding the psychological components of such deliberative settings (Bächtiger et al., 2010; Goodin and Niemeyer, 2003; Mutz, 2006). The present findings can contribute something to this question as well.
By offering a definition of what a considered opinion might be and how to measure it, this study also aims at contributing to a normative debate on citizens’ political competence, which has not been led by political psychologists yet. According to Bolsen et al. (2014: 253), there is ‘a lack of consensus among scholars as to what constitutes a normatively appealing opinion’ (see also Druckman, 2012). I suggest that political psychology might link to democratic theory, in particular deliberative democracy theory, in search for answers to the question of normative requirements for citizens’ participation. This link has rarely been made yet, and a more detailed exchange between empirical and normative scholars is highly desirable. Finally, conducting a controlled experiment in the context of a real-world political decision, where subjects take part and which affects their life, constitutes an interesting variation of previous experiments, which mostly used political questions taken from the general public debate.
As to the limitations of this study, first, it looks at only at one single policy decision in a certain context, thus the results remain to be replicated under different circumstances. It would be interesting to see whether the effects are different with less controversial and less polarising issues. It might be that the selection of a highly contentious issue such as the Scottish referendum has made it more difficult to find debiasing effects and considered opinions, as motivated reasoning is particularly likely with such issues (see Leeper and Slothuus, 2014). A second limitation concerns the justification paragraphs written by subjects. It would have been desirable to have more written material for the measurement of IC, in order to enhance the validity and reliability of the measurement. However, time limits and resource constraints did not allow this. Furthermore, written material can naturally provide only an indirect measure of thought complexity. There is the possibility, for example, that a subject has very well considered the issue in the past, but decides to write down only a one-sided justification which corresponds to her actual position.
Third, as with every laboratory experiment using student samples, generalisability is a question. As Druckman and Kam (2011) state, the advantages of representative samples over student samples have not been empirically proved so far. However, other authors have come to less optimistic conclusions (e.g. Peterson, 2001). Student samples produce biased estimates only if we cannot assume a homogeneous treatment effect for the whole population. This is an empirical question and remains to be tested on different samples. Fourth, as mentioned above, in order to conclusively test whether there is a conditional treatment effect dependent on attitude strength and/or political knowledge, a replication with a larger sample, yielding more statistical power would be desirable. A closer analysis of the inconclusive findings on the consider-the-opposite treatment would be worthwhile, as some evidence points in the direction of cognitive accessibility – and not only motivation – as a cause for opinion considerateness. Finally, a critical question needs to be asked as to whether and to what extent voters base their decisions on policy information and arguments at all. Several recent contributions have made the point that group belonging and social identities constitute the very basis of political reasoning and that identities and emotions affect voting behaviour often without conscious consideration (Achen and Bartels, 2016; Lodge and Taber, 2013).
In sum, this study tries to link older findings on debiasing from social psychology to the recent debate on the determinants of motivated reasoning versus considered political opinions in political psychology. The strong evidence for the positive effect of having to justify one’s political opinion to others constitutes a starting point for further research on how to enhance considered opinions and citizen competence.
Footnotes
Acknowledgements
I am grateful to Hanspeter Kriesi, Alexander Trechsel, Marco Steenbergen, Johanna Gereke and the members of the EUI Political Behaviour Colloquium for their valuable comments and support. I thank Diego Gambetta for his organisational support and Davidi Morisi for the pleasant and fruitful collaboration in Scotland. I thank the BLUE laboratory at the University of Edinburgh and in particular Michèle Belot and Ivan Salter for their kind welcome and support in the laboratory. Finally, I thank Lela Chakhaia for her help with data coding and Mathias Birrer for his many invaluable inputs.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: European University Institute, Florence, Italy, Mission Funding 2014.
Supplementary Material
Supplementary material for this article is available online.
Notes
Author Biography
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
