Breaking Negative Stereotypes of Climate Activists: A Conjoint Experiment

Abstract

We tested 44 variations in profiles of climate change activists to see what affected willingness to associate with them. The largest effects were from activists’ perspectives on climate change, how often they pressure others, gun control views, and party affiliation. If implemented as a traditional factorial experiment, this experiment would require 648,000 conditions and an infeasibly large sample. We obtained our results much more efficiently via an experimental design rare in communication research. Conjoint experiments will be useful to science communication researchers who wish to simultaneously test many factors of complex stimuli, such as individuals, organizations, technologies, or policies.

Keywords

stereotypes activism civic engagement climate change conjoint experiment

Making it possible for a broad range of citizens to participate in collective action is widely seen as a public good. Collective action is one of the most important forms of public engagement with science, since it can potentially lead to citizens having a greater influence on policy than other forms of engagement (Grossmann, 2012; Scheufele, 2011). For climate change specifically, social movement participation has the potential to help maintain media attention to the issue and increase the likelihood of policy action (Priest, 2016). In addition, the recent March for Science events throughout the United States drew large amounts of participation and led to controversy over their potential positive and negative impacts (Motta, 2018; Myers, Kotcher, Cook, Beall, & Maibach, 2018; Ross, Struminger, Winking, & Wedemeyer-Strombel, 2018). This highlights the importance of understanding the causes and consequences of collective action on scientific issues like climate change.

There are many barriers to participation in collective action. Some of these barriers, such as the required time investment, are inherent to collective action itself. However, in other cases, it may not be features of collective action itself that is the problem but negative perceptions of those engaging in action.

Past research has shown that many people hold negative stereotypes of activists (Bashir, Lockwood, Chasteen, Nadolny, & Noyes, 2013) and that even people who are personally interested in collective action sometimes worry about being associated with extreme activists (Stuart, Thomas, & Donaghue, 2018). Although one study found perceptions of environmentalists to be positive as well as negative, this study also showed evidence that participants distinguished between environmentalists and environmental activists and viewed activists more negatively (Klas, Zinkiewicz, Zhou, & Clarke, 2018). Perceptions of activists as unpleasant extremists may partly be a result of a phenomenon that Sobieraj (2011) identified and that Feinberg, Willer, and Kovacheff (2017) refer to as the activist’s dilemma. Activist groups will sometimes engage in extreme actions in order to gain more media attention. However, when most media coverage of activism is of more extreme forms of activism, this creates a stereotype of all activists as extremists.

Bashir et al. (2013) found evidence suggesting that negative stereotypes of activists reduce participation because people are unwilling to associate with unpleasant individuals. People fear that they will face social sanctions or a less positive self-concept from being associated with extremists (Sedikides & Gregg, 2008). These fears cause reduced interpersonal attraction to activists, with attraction in this case meaning willingness to affiliate, as opposed to romantic attraction (Montoya & Horton, 2004). This reduced attraction, in turn, reduces participation. Therefore, increasing attraction by breaking stereotypes might increase participation.

Besides potential benefits of increased participation, this research helps broaden understanding of how communication influences engagement with scientific issues. Similarly to how portrayals of scientists can influence people’s willingness to become scientists themselves (Long et al., 2010), people’s willingness to get involved with controversial scientific issues will likely be affected by portrayals of the people participating in other forms of engagement.

Reducing the Impact of Negative Stereotypes: The Continuum Model

Understanding how stereotype-breaking portrayals of activists might be effective requires understanding how stereotypes influence information processing. The continuum model (Fiske, Lin, & Neuberg, 1999) explains how encountering people with stereotype-inconsistent attributes can affect overall impression formation. In the continuum model, individuals who encounter a person will use rapid, heuristic processing in an attempt to fit the person into a category, based on initially salient features of the person. If the person is successfully categorized, attributes commonly associated with that category will guide further processing. If there are attributes present that contradict the initial categorization, however, it is more likely that the individual will devote greater cognitive resources to processing the individual’s attributes. This greater attention makes it more likely that the specific person’s attributes, rather than features of the stereotypical category they were initially assigned to, will guide the overall impression that is formed. This suggests two complementary ways in which portraying activists with stereotype-inconsistent attributes might increase attraction. First, these attributes might directly cause more positive impressions on their own. Second, they might shape information processing and increase the attention paid to the activist’s other stereotype-breaking attributes.

Although they did not use the continuum model to guide their research, Bashir et al. (2013) did test whether activists with stereotype-breaking attributes would be received positively. In four out of the five studies included in their article, they portrayed activists as counterstereotypical by saying that their activism consisted of tactics like social events and fundraising, as opposed to protest rallies. In the fifth study, no specific attributes were mentioned; instead, the activists were simply described as “atypical.” In all studies, participants were more attracted to activists portrayed with counterstereotypical attributes.

In addition, Bashir et al. (2013) found that these portrayals of atypical activists did not increase attraction simply by being atypical per se. Their results suggested that stereotypical activists are seen as militant, eccentric, and unfriendly. Portrayals of atypical activists significantly reduced these perceptions, and these changes were associated with corresponding increases in attraction and collective action intentions.

The results of Bashir et al. (2013) suggest that people might be more attracted to activism if they encounter portrayals of activists who seem nonmilitant, noneccentric, and friendly. However, knowing which perceptions to try and change does not make it clear how best to change them. Knowing that we ought to reduce perceived militantness, for example, does not give us a lot of guidance on what kind of portrayals of activists are likely to reduce militantness. Should they be portrayed as having a calm speaking style? Or is it more important to alter the content of what they say? If so, which specific topics should they speak about? It is not difficult to generate reasonable guesses about what kinds of content might reduce perceptions of militantness. However, guesses are not the same as empirical data. As O’Keefe (2003) has noted, in order to understand more fully how communication can shift behavior through changing psychological states, we need to systematically study the relationship between features of message content and psychological states, as well as studying the relationships between psychological states and behavior. Thus, one crucial step in understanding what makes activists more or less attractive will be to get a more systematic understanding of how message content affects perceived militantness, eccentricity, friendliness, and attraction.

The Bashir et al. (2013) studies give us a very limited understanding of how message content affects these outcomes. They used only two ways of portraying counterstereotypical activists: They either mentioned activists’ use of less confrontational tactics (social events and fundraising) or simply described them as “atypical.” Because Bashir et al. measured only two types of variation in message content, our knowledge of how to change perceptions is limited to those specific types of variation. Broadening the types of content tested in any form will improve our understanding. In addition, there is particular reason to test the effects of personal attributes of activists, as opposed to the differences in tactics for achieving social change examined by Bashir et al. (2013). The personal attributes of activists are on display any time activists are portrayed in the media or when people encounter them. If these attributes have a substantial effect on public support for activism, this would be a very different world from the world where activists’ tactical choices are the only thing that substantially affects public support.

Selecting What to Vary From an Infinite Population of Possibilities

We have shown how our understanding of citizens’ responses to activists will be improved by testing the effects of a wide variety of counterstereotypical attributes. Based on the findings of mediation analyses by Bashir et al. (2013), it makes sense to test attributes that are likely to shift perceptions of militantness, eccentricity, and friendliness. However, existing theories and empirical evidence provide little guidance on how specific types of content affect these target perceptions. Some prior work has been conducted on perceptions of aggression, which is similar to militantness (e.g., Třebický, Havlíček, Roberts, Little, & Kleisner, 2013); on perceptions of eccentricity (e.g., Van Tilburg & Igou, 2014); and on perceptions of warmth or friendliness (e.g., Fiske & Dupree, 2014). Even in well-researched areas such as perceptions of warmth, however, there have been few studies that systematically compare the effect sizes of different types of portrayals of individuals on perceived warmth. This means that it is difficult to be sure which attributes of activists are likely to lead to the largest changes in each of the three perceptions. In this situation, we should aim to test as many different characteristics as possible.

It would also be wise to increase the number of attributes tested to account for a confounding factor. Some of the personal attributes we manipulate could have spurious effects on the outcomes due to causing people to infer other attributes (Dafoe, Zhang, & Caughey, 2018). Perhaps the most important of these, and a factor not addressed in the Bashir et al. (2013) studies, is political party identification. Party identification is one of the most important influences on U.S. citizens’ views about climate change (Merkley & Stecuła, 2018), as well as being one of the most important and divisive factors in how Americans see each other more generally (Mason, 2018). Furthermore, environmentalism and climate activism are strongly associated with the Democratic party (Bashir et al., 2013; Merkley & Stecuła, 2018). Because of this, any attributes that change perceptions of what kind of an activist a person is may cause people to infer their party allegiance based on that activism. For example, people expressing support for more radical actions on climate change may be inferred to be likely Democratic partisans; people expressing less radical perspectives may be inferred to be more likely to be Independents, or at least less strong partisans. Explicitly including party identification in the portrayal of the activists would mean that participants would not need to infer it from other characteristics. This would enable greater confidence that the observed effects of those other characteristics were not simply due to them shifting participants’ judgments of party ID. Similarly, including demographic characteristics would reduce the distorting impact of potential inferences about them.

Testing all our factors of interest with a factorial experiment would either result in low statistical power or require an expensive sample. Our experimental design described below would have required 648,000 experimental conditions if implemented as a factorial experiment (calculated by multiplying attribute values for each of the different attributes, 9 × 5 × 5 × 8 × 3 × 3 × 5 × 2 × 4). We were able to obtain relatively precise estimates of effects for these variations with a sample size of only 1,652. As we explain further below, the features of conjoint experiments make this possible.

Conjoint Experiments

Conjoint experiments have been used for decades, but the design used in the present study is based on more recent work in political science (Hainmueller, Hopkins, & Yamamoto, 2014) and is still rare in communication (though see Knudsen & Johannesson, 2018). In a typical example of this design, each stimulus consists of a two-column table that profiles two individuals, with the rows of the table corresponding to different attributes of the person. In the example in Figure 1, from the present study, each column corresponds to a climate activist, and the rows correspond to different attributes. Each attribute can take several values. For example, in our study the values for Party ID were “Democrat,” “Republican,” and “Independent,” and the values for gender were “Male” and “Female.” In the simplest conjoint design, the two profiles are generated completely randomly, with both profiles randomly assigned a value for each attribute, and the order of attributes randomized as well.

Figure 1.

Example of how profile tables were displayed to participants.

Participants are asked to read the table and respond to questions relating to the individuals profiled there. These questions serve as outcome measures. For example, one of the outcome measures in the present study asked participants to rate on a 7-point scale how much they would like to meet each of the two individuals profiled in the table.

In traditional experiments, each experimental condition is shown to many participants. In conjoint experiments, there is no requirement that any specific combination of attributes occurs, and most specific combinations will not be presented more than once across all participants. The marginal effects of each attribute value are calculated, instead of the effects of whole profiles. Participants’ responses are used to calculate a quantity known as the average marginal component effect or AMCE (Hainmueller et al., 2014). As an example, consider the effect of an activist being male as opposed to being female. The AMCE for being male consists of the average difference in reported willingness to meet the activist in all profiles where the activist is male, compared to all profiles where the activist is female. The other attributes in the profiles will vary widely and be present in many combinations. However, because the attributes are all randomly assigned, averaging provides an unbiased estimate of the AMCE of gender (Hainmueller, Hangartner, & Yamamoto, 2015; Hainmueller et al., 2014; Knudsen & Johannesson, 2018).

Calculating the AMCE of different components in this way requires several assumptions, which are described in more detail in Hainmueller et al. (2014). Briefly, one assumption is that when rating multiple profiles, participants’ responses do not change for later profiles as opposed to earlier profiles and that the attributes seen in earlier profiles do not affect how they respond to later profiles. A second assumption is that participants will not respond differently due to the side of the table (left or right) on which a profile appears. A third assumption is that profiles are truly randomly generated. Whether the first two assumptions hold can be checked empirically, as we do in the Results section. The third assumption, of random profile generation, is guaranteed if the computer program that generates the profiles in fact does generate them randomly. It is also necessary to adjust for the fact that each individual participant responds to multiple profiles; this can be addressed by using cluster-robust standard errors (Hainmueller et al., 2014).

Strengths of Conjoint Experiments

The main strength of conjoint experiments is that they make it possible to test for the causal effects of a large number of factors simultaneously, and to do so efficiently with feasible sample sizes. With traditional experiments, each experimental condition is presented to multiple participants. With conjoint experiments, there is no requirement that any discrete stimulus—in this case, a table of profiles—is repeated across multiple participants. In addition, many conjoint experiments (e.g., Bansak, Hainmueller, Hopkins, & Yamamoto, 2018; Bechtel & Scheve, 2013; Hainmueller et al., 2014) have each participant rate multiple profiles, and pool the results. In effect, if the assumptions hold, having each of your N participants rate one extra profile is equivalent to adding another N participants to your experiment. This is very attractive, since the marginal cost of adding one extra profile rating per participant is much lower than adding N participants. One recent study suggests that data quality does not reduce substantially even if participants rate dozens of profiles (Bansak et al., 2018). To sum up, conjoint experiments make it possible to achieve high statistical power for a large number of effects at a low price.

The ability to test multiple message components simultaneously also makes conjoint experiments ideal for addressing an important challenge of communication research: accounting for message heterogeneity as well as variability (Slater, Peter, & Valkenburg, 2015). Message variability refers to the elements of a stimulus that are deliberately varied. In a test of climate change fear appeals, for example, message variability might consist of the comparison between messages that do and do not include fear-arousing content. Message heterogeneity refers to the variations in message content other than those variations directly manipulated across conditions. For example, there are a wide variety of climate change messages that include fear-arousing content. As Slater et al. (2015) note, we will get stronger evidence about the effectiveness of fear appeals if we account for this within-category heterogeneity, in addition to the between-category variability of whether fear is included. Because conjoint experiments allow many different stimulus variations to be tested, it is possible to test multiple instances within each category.

Including many factors simultaneously may also increase external validity. Because real-world stimuli are complex, people’s reactions to the multidimensional stimuli in conjoint experiments may be more similar to their reactions to real-world phenomena. Hainmueller et al. (2015) conducted conjoint experiments where Swiss participants viewed profiles of hypothetical immigrants to Switzerland and rated their willingness to admit each immigrant. Because some Swiss immigration was formerly decided by citizens being mailed profiles of immigrants and voting on who to admit, a comparison to real-world behavior was possible. The authors found that the results of the conjoint experiments were very similar to the results of the real-world voting.

Research Questions

We included four attributes targeted at specific mediating perceptions, with 27 possible values across these attributes. Five attributes were included to prevent participants from inferring those attributes based on other attributes, with 17 possible values across these attributes. Details of each attribute, the possible values for that attribute, and the reason for including it are given below.

Our research questions were the following:

Research Question 1: Which attribute values increase (or decrease) attraction to climate change activists?

Research Question 2: Are the results for militantness, eccentricity, and friendliness consistent with these variables mediating the effects of the attribute values on attraction?

Method

Sample

The initial sample consisted of 2,000 participants recruited via Amazon Mechanical Turk, an online platform where participants complete various tasks for payment, on March 1 and 2, 2018. The participants were restricted to those registered in the Mechanical Turk system as being U.S. residents over 18 years of age and having a 98% or higher approval rate for tasks they had participated in on Mechanical Turk. Participants were paid $1.25 for completing the study. Of the 2,000 participants who completed some part of the study, 348 were dropped from the final analysis due to potentially being people who had already taken the experiment or due to completing the experiment too quickly. Full details of reasons for case deletion are available in the online Supplemental Material.

Dropping the participants who completed the study too quickly left a final sample of 1,652 for analysis. These individuals had a mean age of 38 years. Individuals identifying as female made up 51% of the sample, 48% identified as male, and less than 1% gave any other answer. In terms of race and ethnicity, 83% were White, 8% were Black or African American, 9% were Hispanic/Latino, and 7% were Asian. Politically, 56% identified as Democrats or Democratic-leaning Independents, 16% were Independents close to neither party, and 28% were Republicans or Republican-leaning Independents. See the online Supplemental Material for party ID statistics for the full sample of 2,000.

Procedure

Participants were given a brief introductory message explaining that they were about to be shown several profiles of climate change activists, and asking them to answer honestly, as if they were real individuals that they might meet in the future. Following this, participants were shown eight pairs of profiles. These were presented to participants in the format displayed in Figure 1. The attribute values for each profile were randomly selected for every single profile. The order in which each attribute appeared in the table was the same across all pairs of profiles for each participant. This meant that, for example, someone whose first pair of profiles had gender as the attribute displayed in the top row would always have gender displayed in the top row. However, the order of attributes was randomized for each participant.

For the first four pairs of profiles, participants were asked to indicate their willingness to meet the activists and their willingness to get to know them. For the second four pairs of profiles, participants were asked to indicate how militant, eccentric, and friendly they thought the activists were. After viewing the eight pairs of profiles, the experiment ended, and participants were presented with the code that enabled them to receive payment via Amazon Mechanical Turk.

Stimuli

All profile attributes are described below. For four of the attributes, we note which of the mediators that attribute was targeted at. In doing this, we are not claiming that manipulating activists’ jobs with the intention of affecting friendliness, for example, is equivalent to directly manipulating friendliness. Following O’Keefe (2003), we distinguish between two types of relationship—relationships between message content and mediating variables, and relationships between mediating variables and outcomes. We agree with O’Keefe that both types of evidence are important for developing theoretical understanding of communication. However, we can provide direct evidence only for the effects of content on mediators and content on attraction. As discussed below, we address the relationships between the mediating states and attraction indirectly.

In addition to including attribute values designed to shift stereotypes of activists, we also had to include values that would be likely to confirm the stereotype. This is because the effects of different attribute values are not defined in comparison to a control group. Instead, effects of each value of an attribute are measured in comparison to one of the attribute’s values that is selected as a baseline category. For each attribute, we wanted the baseline value to be one that confirmed the stereotype of activists. This would mean that the effect of each attribute value would be in comparison to an activist with a stereotype-confirming value on that attribute. Full text of all attribute values is available in Table 1.

Table 1.

Full Text of All Profile Attributes and Attribute Values.

Attribute	Value
Perspective on climate change	We must turn from a liberal agenda of reform to embrace a radical agenda of revolt. If you’re not part of a radical solution, you’re part of the problem.
	Fossil fuel and coal company executives are responsible for horrible human suffering, and should be put on trial for crimes against humanity.
	The fight against climate change is a battle for the soul of America.
	Climate change is not that different from the risk issues dealt with in financial markets every day. No outcome is certain, but it’s rational to pay money to reduce the risk.
	I care about climate change, but I always try to be tolerant of people who disagree with me. Having real discussions with people who are skeptical makes me smarter.
	I love to talk about global warming all the time because it makes me a really big hit at parties. (Ha ha, not really).
	Climate change activism is my way of making up for my other hobby: Making delicious meals out of endangered animals. (Just kidding!)
	As George Carlin said, I don’t want to save the planet. The planet is gonna be fine. It’s the people who are f***ed!
	I heard that climate change will make beer production more difficult. Don’t mess with my beer.
How often they pressure friends and family to support action on climate change	All the time
	Often
	Occasionally
	Rarely
	Never
Preferred clothing style	Unconventional, odd accessories
	Scruffy, worn pants and old shirt
	Causal, jeans and t-shirt
	Tidy, button-down shirt and slacks
	Formal, business suit
Job	College student (graduate school)
	Lawyer
	Taxi driver
	Construction worker
	Office clerk
	Farmer
	Doctor
	Nurse
Political party	Democrat
	Independent
	Republican
Views on gun laws in the United States	Gun laws should be more strict.
	Gun laws are currently just about right.
	Gun laws should be less strict.
Age bracket, years	20-29
	30-39
	40-49
	50-59
	60-69
Gender	Female
Gender	Male
Race	White
	Black
	Hispanic/Latino
	Asian

Note: Boldfaced words in perspectives on climate change indicate the word used to identify that perspective in Figures 2 -5; these words were not boldfaced in actual stimuli. The top value displayed for each attribute is the baseline.

Perspective (Targeted at Perceptions of Militantness)

The attribute we called “Perspective on climate change” was not used because we thought giving individuals’ perspectives per se would have powerful effects. Instead, it served as a convenient vehicle for what we saw as two other potentially powerful influences. First, demonstration of cool-headed rationalism. Someone who discusses using logic and evidence, and not expressing strong emotion, seems like they should be perceived as very different to a militant extremist. In addition, the director of a prominent libertarian think tank in Washington, D.C., has reported that a rational presentation of the climate issue was part of what convinced him of its seriousness (Roberts, 2015). We included two examples of rational perspectives. One was based on statements by that think tank director (“Rational”). Another was based on tolerance for disagreement and wish for dialogue (“Tolerant”).

We thought that humor might be another powerful way of reducing perceived militantness. Bashir et al. (2013) reported that participants often described environmentalists as “self-righteous” and “zealous,” words we normally associate with an inflated sense of seriousness and importance. Making a joke seems like the opposite of this, as it involves being not completely serious, and making light of the situation. In addition, some research has shown humor messages to increase climate activism intentions (Skurka, Niederdeppe, Romero-Canyas, & Acup, 2018). Finally, we included radical-sounding perspectives to function as comparison points for the rational and humorous perspectives.

Pressure (Targeted at Perceptions of Militantness)

One other attribute that seemed likely to influence perceptions of militantness was the frequency of an activist pressuring friends and family to contribute to the cause. Two descriptors of activists frequently mentioned by participants in the Bashir et al. (2013) study, which fell into the category of militant descriptors, were “forceful” and “annoying.” Pressuring friends and family to take part seemed likely to be one activity common to many activists that would influence how militant they seemed.

Clothing (Targeted at Perceptions of Eccentricity)

The Bashir et al. (2013) top 30 free-response answers included “dresses like a man” and “bad dresser” for feminists and “unfashionable” for environmentalists, suggesting that clothing might be one important component of what makes activists seem eccentric. Due to an oversight, the attribute value that was meant to begin with the word “Casual” had this word misspelled as “Causal.” As we explain in the Discussion section, we have reason to believe that this did not substantially affect results.

Job (Targeted at Perceptions of Friendliness)

Quantitative data on the perceived warmth of different occupations within the United States were available (Fiske & Dupree, 2014), and describing people as having high-warmth occupations seemed likely to make them appear friendlier. In choosing which jobs to use, we selected occupations from Fiske and Dupree (2014) that had a range of both warmth and competence, including those who were highly warm and highly competent (e.g., nurse), highly warm and more moderately competent (e.g., farmer), low on warmth and high on competence (e.g., lawyer), and low on both warmth and competence (e.g., taxi driver). We also added one job that was not measured in Fiske and Dupree’s (2014) study. This was the baseline category, “College student (graduate school),” which we expected to match the stereotype of climate activists.

Political Party

As described above, one major reason we included political party ID was to ensure that our estimates of other attributes’ effects would not be contaminated by participants using those attributes to infer party ID. In addition, we were interested in comparing the effects of party ID on attraction to the effects of other attributes. Affective polarization—polarization in opposite partisans’ feelings of warmth toward each other—is very strong and is likely to be a large influence on attraction (Iyengar & Westwood, 2015). In addition, because most people likely assume that environmental activists are Democrats (Bashir et al., 2013), portraying a Republican activist may be one way to positively break stereotypes for Republicans.

Views on Gun Laws

We included this attribute because politics are so central to attitudes on climate change and collective action, and we wanted to include another attribute connected to where the activists stand politically. Gun control views were appealing for several reasons. Gun control is one of the most politically polarized issues in the United States (Pew Research Center, 2017). There is also evidence that issue positions can still make a difference to how people are evaluated when their partisanship is known (Lelkes, 2018), suggesting that issue positions are cues that affect people’s impressions at least partly independently from their effect on inferred partisanship. Also, gun attitudes seemed like they could potentially be harder to reconcile with the stereotype of climate activists than party ID. It seemed possible that participants who are told a climate activist is Republican might infer they are only weakly Republican—in popular parlance, a RINO, or Republican in Name Only. However, if someone is pro-gun, their divergence from the stereotype of liberal, Democratic environmentalists is based on concrete position rather than a label. This seemed like it might be harder to reinterpret the attribute value in a way that is consistent with the stereotype. Third, using positions on one issue to appeal to those that would otherwise be outside the typical base of a cause is a common strategy. For example, Bob Inglis, former South Carolina congressman and leader of conservative climate organization RepublicEn, has often used his high ratings from the National Rifle Association as a way of appealing to Republicans (e.g., Breslow, 2012). We based the wording of the values on a 2017 Pew survey on gun views (Pew Research Center, 2017).

Age Bracket, Gender, Race

We included activists’ age bracket, gender, and race in order to control for potential effects of participants inferring these qualities based on other attributes of the activists.

Measures

Party Identification

Participants’ party ID was measured with a single item: “Generally speaking, when it comes to political parties in the United States, how would you best describe yourself?” Answer options were 1 (a strong Democrat), 2 (a not strong Democrat), 3 (Independent, lean toward Democrat), 4 (Independent (close to neither party)), 5 (Independent, lean toward Republican), 6 (a not strong Republican), and 7 (a strong Republican).

Attraction

Attraction to activists was measured as the mean of two items adapted from the Interpersonal Attraction Questionnaire by Montoya and Horton (2004). One item consisted of the statement “I would like to meet Person A.” The second read “I would like to get to know Person A better.” These were repeated for the other profile in the table (“Person B”). Answer options were 1 (strongly disagree), 2 (moderately disagree), 3 (slightly disagree), 4 (neither agree nor disagree), 5 (slightly agree), 6 (moderately agree), and 7 (strongly agree).

Militantness, Eccentricity, Friendliness

Militantness, eccentricity, and friendliness were each measured with a single item. Militantness was measured with the statement “Please indicate how militant each person seems to you.” Answer options ranged from 1 (not at all militant) to 7 (extremely militant), with the other scale points numbered but not labeled. Measures for eccentricity and friendliness were identical, with the words “eccentric” and “friendly” replacing the word “militant.”

Results

Figure 2 displays the results for attraction, and Figures 3 to 5 display the results for militantness, eccentricity, and friendliness. Each graph includes three panels, with the left-hand panel (“Unconditional”) representing the effects averaged across all participants, the middle panel representing the effects for Strong Democrats only, and the right-hand panel representing the effects for Strong Republicans only. The effects for Strong Democrats and Strong Republicans are displayed in order to see the maximum possible variation in effects for the effects that do vary with party ID; the effects for people with other levels of party ID are omitted for space reasons but can be viewed in the online Supplemental Material. The asterisks displayed next to some of the attribute value labels represent that the effect of that attribute value had a significant interaction with participants’ party ID. When the interaction is not significant, readers should focus on unconditional effects, since we cannot be confident that any difference across Democrats and Republicans is not due to chance.

Figure 2.

Effects of attribute values on attraction.

Figure 3.

Effects of attribute values on militantness.

Figure 4.

Effects of attribute values on eccentricity.

Figure 5.

Effects of attribute values on friendliness.

The dots represent the estimated AMCE for an attribute value compared to the baseline category (with these categories not displayed on the graph). The x-axis units are the original scale points for each variable—attraction, militantness, eccentricity, and friendliness. The bars on either side of the dots are 95% confidence intervals for these effects. For all attribute values where the confidence interval does not include the zero point on the x-axis, the effect compared to the baseline value is statistically significant.

Effects on Dependent Variable: Attraction

Several of the perspectives on climate change resulted in large positive effects on attraction. The “Tolerant” perspective had the biggest positive effect on attraction of all attribute values. There were also substantial effects for the “Soul,” “Rational,” “Parties,” and “Animals” perspectives. The effects of other perspectives differed by party. The “Crimes” perspective had the second largest positive effect for strong Democrats but performed worse than the baseline perspective for strong Republicans. The “Planet” and “Beer” perspectives, while among the most effective for strong Republicans, were among the worst for strong Democrats.

When analyzing the results in terms of categories of perspective, as opposed to individual perspectives, the perspectives focused on rationality (“Rational” and “Tolerant”) performed the best overall, regardless of political party. The effectiveness of the other categories of perspective varied across participants of different political parties. Radical perspectives (“Radical,” “Crimes,” “Soul”) often performed well for Democrats, with two out of three of these perspectives (“Crimes” and “Soul”) leading to roughly the same level of attraction as the rational perspectives, and higher levels of attraction than the humorous perspectives. For Republicans, humorous perspectives performed almost as well as the rational perspectives, and better than all of the radical perspectives.

Large effects on attraction were also found for the activists’ views on gun control. These effects were larger than the effect of most perspectives on climate change. People strongly prefer to associate with climate activists who share their party’s views on gun control. For Republicans, a climate activist who favors keeping gun laws the same, or who favors making them less strict, is much more attractive than one who favors stricter gun control. For Democrats, these views make the activist substantially less attractive.

The effects of party identification are similar to the effects of gun control views, though the magnitude of the effects is smaller. Republicans are more attracted to other Republicans, and Democrats are more attracted to Democrats. However, while Republicans prefer Independents to Democrats, Democrats are equally attracted to an Independent or Democratic climate activist.

How often people pressure their friends and family has quite a divergent effect on Republicans and Democrats. For strong and not strong Democrats, the frequency of pressure has very little or no effect on willingness to associate. For everyone else, including Democratic-leaning Independents, all lower levels of pressure have a positive effect. For Republicans, the positive effect of never pressuring friends and family, compared to pressuring them all the time, is one of the biggest effects observed—for example, it is larger than the effect of the activist being a Republican.

Participants’ clothing and jobs, for the most part, had little effect on willingness to associate with them. None of the forms of clothing had a significant effect on attraction. The effect of being a construction worker on attraction varied with participants’ party ID, with a significant positive effect observed for participants who identified as not strong and strong Republicans, and no significant effect for any other participants. There was a small positive effect on attraction for farmers, which did not vary across participants’ party ID.

All age brackets from 40 to 49 upward had small negative effects on attraction. As discussed below, the effect of the 30 to 39 age bracket is indeterminate due to this attribute value having a significantly different effect depending on which side of the table it was presented. While there was a significant interaction between participants’ party ID and the activist being Hispanic/Latino, none of the effects of being Hispanic/Latino were significant for any specific level of party ID. None of the other demographic attribute values had significant effects on attraction.

Effects on Potential Mediators: Militantness, Eccentricity, Friendliness

We could not perform a traditional mediation analysis, as attraction was never measured together with the mediators for any of the profiles. We made this choice for three reasons. First, we know of no analysis software that facilitates analysis of mediation in conjoint experiments. Second, we thought that measuring attraction and the potential mediators separately would reduce the risk of participants’ responses on the mediators contaminating their responses on the outcome variable, or vice versa. Third, while it is commonly assumed that mediation analyses in a randomized experiment can provide strong evidence for the causal chain linking treatment to mediator to outcome, this is true only in designs where both the treatment and the mediator are randomized (Bullock, Green, & Ha, 2010; Pirlott & MacKinnon, 2016). If only the treatment is randomized, the relationship between mediator and outcome is correlational.

Due to all these factors, we decided that it would be better to obtain clean measures of causal effects on both the outcome variable and potential mediator variables, and see to what extent these effects are consistent with mediation. We pay special attention to instances where there are significant increases in the mediator with no significant changes in attraction. Especially if the changes in the mediator are substantively large, the lack of associated changes in the outcome is evidence against a consistent mediating relationship.

Looking at the effects displayed in Figures 3 to 5, the overall pattern is more or less consistent with militantness, eccentricity, and friendliness mediating effects on attraction. The attribute values that cause significant changes in attraction usually also cause changes in one, and often all three, of the mediators. However, the evidence is stronger for militantness and friendliness than it is for eccentricity. Most attributes that lead to significant decreases in militantness or increases in friendliness also lead to increases in attraction (with the opposite effect for significant increases in militantness or decreases in friendliness). There are exceptions to this. For example, the “Beer” perspective substantially reduces militantness compared to the Radical perspective for strong Democrats but does not increase attraction for strong Democrats. However, there are few exceptions like this. For eccentricity, there is a larger exception. All forms of clothing led to significant and relatively large decreases in eccentricity but were not accompanied by a significant increase in attraction. Since the only attribute values causing large changes in eccentricity but not other mediators did not lead to any significant change in attraction, this is inconsistent with what we would expect of an important mediator.

Analysis of Assumption Violations

We tested whether the assumptions of no carryover effects or profile order effects held in practice, by conducting analyses where the round of presentation or the side of the table on which an attribute value was presented were included as variables interacting with the attribute effects. There were almost no instances where accounting for these factors made any meaningful difference to the interpretation of the effects displayed in Figures 2 to 5. We list the two types of exception here. First, the effect of the 30 to 39 age bracket on attraction is not reliable. This age bracket had a significant negative effect compared to the baseline category only for profiles displayed on the left side of the table. For profiles displayed on the right side, there was no significant effect. Because it is unclear which of these would more closely represent the effect of a single profile displayed on its own, the effect of this age bracket on attraction is indeterminate. Similar patterns were observed for activists whose job was listed as nurse or construction worker, with effects on attraction or militantness, respectively. In these cases, there was a significant effect for profiles displayed on the right side of the table but not the left. Therefore, these effects are also indeterminate.

Second, there were several instances where an attribute value had a small but statistically significant effect on friendliness in the fifth round (which was the first round that the effects on friendliness were measured) but no significant effect when averaged across all rounds. In the fifth round, three jobs had a negative effect on friendliness: lawyer (−0.24), construction worker (−0.18), and nurse (−0.23). In addition, Black activists were seen as more friendly than White activists in the fifth round (0.13) but not overall. While these effects may simply be due to chance, they may also indicate small but real effects that fade after repeated exposures. If so, the null effects displayed in Figure 5 for these attribute values would be inaccurate.

Discussion

The main aim of our study was to discover whether it was possible to make people more willing to associate with climate activists by changing their personal attributes, as opposed to changing the kind of political action they engage in. Our results show that this is indeed possible. We should specify that not all ways of breaking stereotypes had positive effects; in multiple cases, we found that attributes that differed from stereotypical portrayals had no significant effect, or even negative effects. However, we also found many positive effects. People wishing to encourage participation from a broader range of citizens may be able to do so by breaking negative stereotypes in the specific ways we found to be successful.

The strengths of the conjoint experiment design were fundamental to obtaining many of the most important parts of our results. For example, our results enable us to compare the relative effectiveness of many different profile attributes. We found that perspectives on climate change, views on gun control, activists’ party ID, and frequency of pressuring others had the largest effects on attraction, with small effects for activists’ age and little to no effects due to activists’ clothing, job, gender, or race. It would have required a much greater amount of resources to obtain measures of all these causal effects in a conventional experiment.

Testing multiple factors also made it possible to identify ideal combinations of attributes that are likely to lead to larger positive effects. For Democrats or Democrat leaners, practitioners should consider portraying activists who express rational or radical (but not too radical) perspectives, who do not have any attributes that could make them seem like they are a Republican or against stricter gun control, and who are farmers. For Republicans or Republican leaners, practitioners should consider portraying activists who express rational or humorous perspectives, who never pressure their friends and family to act on climate change, who are explicitly Republicans, who are against stricter gun control, and who are farmers.

The conjoint design not only allowed us to test multiple forms of content but also provided evidence toward multiple theoretical questions. One of the most important ways we build on the work of Bashir et al. (2013) is by addressing the role of party identification, the driving force in polarization over climate change (Merkley & Stecuła, 2018). Our results show that indicators of activists’ partisan political leanings have some of the biggest effects on how those activists are received. Addressing climate activism in a depoliticized manner without considering the ways in which messages might be implicitly signaling partisanship seems unwise. In addition, we demonstrated that reception of nonpolitical attributes sometimes varied based on the partisan identification of the participant. This confirms that the public is not uniform in how they respond to stereotype-breaking portrayals of activists.

Another theoretical question we addressed was which perceptions mediate the effects of different attributes on attraction. The conjoint design made it easy to include multiple manipulations of content directed at each of three different mediators. This provided some evidence that eccentricity might be less important than militantness and friendliness in increasing attraction to activists. If this were true, breaking stereotypes by portraying activists as noneccentric would be less effective than making them seem nonmilitant and friendly. The consistent similarity of effects on militantness and friendliness is also interesting—future research could further probe whether other attributes of activists affects these perceptions differently, as well as testing which of the two perceptions has larger effects.

Including many content variations also made it possible to account for message heterogeneity—intracategory variation in effects. This was demonstrated clearly in our results for perspective on climate change, where we included at least two of each of the categories of perspective—radical, rational, and humorous. If we had tested only the baseline “Radical” perspective, for example, our results would have made it seem as if radical perspectives generally perform very poorly, as all other perspectives were more attractive to participants. However, because we included three radical perspectives, it demonstrated that two of the radical perspectives (“Crimes” and “Soul”) were among the best performing perspectives for Democrats, and one of the radical perspectives (“Soul”) performed reasonably well even for Republicans. Making conclusions about categories of message based on single instances would have been severely misleading.

One limitation of our study was that the textual profiles presented were relatively sparse and lacking in vivid detail. Presenting a personal attribute of a person in just a few words assigned to one cell of a table is a much less rich way of encountering that attribute than might be achieved by meeting that individual in person, or even reading a lengthier textual portrayal of them. Thus, the effects we observed should be understood to measure the effects of those attributes as they were presented. These same effects might not generalize to other forms of presenting the same attributes.

As our study sample was from Mechanical Turk, the effects we observed may not generalize to the broader U.S. population. Although one conjoint experiment with a Mechanical Turk sample found results similar to a nationally representative sample (Hainmueller & Hopkins, 2015), another study with a nonrepresentative student sample found results that diverged substantially from a representative sample (Hainmueller et al., 2015).

Finally, our misspelling of the word “Casual” as “Causal” may have affected our results. We suspect that most participants did not notice this error, as not a single participant mentioned this issue in the feedback box that most participants were offered at the end of the study. Any participant that noticed the misspelling may have responded differently to that attribute, and potentially to other parts of the survey. However, we would expect that this would only increase overall measurement error, rather than systematically distorting specific effects.

Conclusion

In addition to addressing our specific questions of interest, our study demonstrates the value of conjoint experiments in general. Conjoint experiments are useful in any area where researchers want to test how people react to multiple facets of complex phenomena. Many areas of science communication research fit this description. For example, one conjoint experiment tested how different features of international climate policy affect public support (Bechtel & Scheve, 2013). Future research could test how social media posts varying in their language, virality, source, recency, and use of images or video are perceived as more or less credible. Another possibility would be testing how multiple variations in the ways in which gene-editing technologies are applied affect public support.

For any scholars interested in implementing conjoint experiments themselves, there are free tools to do so available online. An experiment similar to the one we conducted can be implemented simply by copying and pasting freely available code (Meyer & Rosenzweig, 2016) into the Qualtrics survey platform. Analysis of the results can be performed with cjoint, a free R package (Barari et al., 2017).

Supplemental Material

Conjoint_SciComm_Supplementary_Material_Corrected_2 – Supplemental material for Breaking Negative Stereotypes of Climate Activists: A Conjoint Experiment

Supplemental material, Conjoint_SciComm_Supplementary_Material_Corrected_2 for Breaking Negative Stereotypes of Climate Activists: A Conjoint Experiment by Neil Stenhouse and Richard Heinrich in Science Communication

Footnotes

Acknowledgements

The authors wish to thank Nicole Bonoff for suggesting a conjoint experiment for this research; Kirk Bansak, Anton Strezhnev, and Alex C. Meyer for technical assistance; and the following individuals for helpful feedback and suggestions: Dominique Brossard, Jennifer Chung, Sarah Clifford, April Eichmeier, Keith Franke, Hahrie Han, Emily Howell, Patrice Kohl, John Kotcher, Nicole Krause, Adam S. Levine, Daniel Maliniak, Hannah Monroe, Julian Mueller-Herbst, Tomoko Okada, Kathleen Rose, Dietram Scheufele, Chris Wirz, Mike Xenos, and Shiyu Yang.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The authors acknowledge the Office of the Vice Chancellor for Research and Graduate Education at the University of Wisconsin-Madison (with funding from the Wisconsin Alumni Research Foundation) for its support of this research, in addition to funding provided by the College of Agricultural and Life Sciences at the University of Wisconsin-Madison in support of this research.

ORCID iD

Neil Stenhouse

Supplemental Material

Supplemental material for this article is available online at http://journals.sagepub.com/doi/suppl/10.1177/1075547019848766 and

Author Biographies

Neil Stenhouse is an assistant professor in the Department of Life Sciences Communication at the University of Wisconsin-Madison. His research program focuses on the intersection of science communication and the politics of scientific issues, especially energy development and climate change. Specifically, he is interested in testing whether insights from research on political mobilization and other forms of civic engagement can help increase and diversify participation in scientific issues, as well as helping groups that are already active work together better.

Richard Heinrich is a PhD student in the Department of Life Sciences Communication at the University of Wisconsin-Madison. His work focuses on the role of social identity in influencing perceptions of politicized science. One current project looks at which factors cause a person to be seen as an extremist in the context of a controversial scientific issue.

References

Bansak

Hainmueller

Hopkins

D. J.

Yamamoto

(2018). The number of choice tasks and survey satisficing in conjoint experiments. Political Analysis, 26, 112-119.

Barari

Berwick

Hainmueller

Hopkins

Liu

Strezhnev

Yamamoto

(2017). cjoint: AMCE estimator for conjoint experiments. Retrieved from https://cran.r-project.org/web/packages/cjoint/cjoint.pdf

Bashir

N. Y.

Lockwood

Chasteen

A. L.

Nadolny

Noyes

(2013). The ironic impact of activists: Negative stereotypes reduce social change influence. European Journal of Social Psychology, 43, 614-626.

Bechtel

M. M.

Scheve

K. F.

(2013). Mass support for global climate agreements depends on institutional design. Proceedings of the National Academy of Sciences of the United States of America, 110, 13763-13768.

Breslow

(2012). Bob Inglis: Climate change and the Republican party. PBS Frontline. Retrieved from https://www.pbs.org/wgbh/frontline/article/bob-inglis-climate-change-and-the-republican-party/

Bullock

J. G.

Green

D. P.

S. E.

(2010). Yes, but what’s the mechanism? (don’t expect an easy answer). Journal of Personality and Social Psychology, 98, 550-558.

Dafoe

Zhang

Caughey

(2018). Information equivalence in survey experiments. Political Analysis, 26, 399-416.

Feinberg

Willer

Kovacheff

(2017). Extreme protest tactics reduce popular support for social movements (Working Paper No. 2911177). Toronto, Ontario, Canada: Rotman School of Management. doi:10.2139/ssrn.2911177

Fiske

S. T.

Dupree

(2014). Gaining trust as well as respect in communicating to motivated audiences about science topics. Proceedings of the National Academy of Sciences of the United States of America, 111(Suppl. 4), 13593-13597.

10.

Fiske

S. T.

Lin

Neuberg

(1999). The continuum model. In Chaiken

Trope

(Eds.), Dual-process theories in social psychology (pp. 321-254). New York, NY: Guilford.

11.

Grossmann

(2012). The not-so-special interests: Interest groups, public representation, and American governance. Stanford, CA: Stanford University Press.

12.

Hainmueller

Hangartner

Yamamoto

(2015). Validating vignette and conjoint survey experiments against real-world behavior. Proceedings of the National Academy of Sciences of the United States of America, 112, 2395-2400.

13.

Hainmueller

Hopkins

D. J.

(2015). The hidden American immigration consensus: A conjoint analysis of attitudes toward immigrants. American Journal of Political Science, 59, 529-548.

14.

Hainmueller

Hopkins

D. J.

Yamamoto

(2014). Causal inference in conjoint analysis: Understanding multidimensional choices via stated preference experiments. Political Analysis, 22, 1-30.

15.

Iyengar

Westwood

S. J.

(2015). Fear and loathing across party lines: New evidence on group polarization. American Journal of Political Science, 59, 690-707.

16.

Klas

Zinkiewicz

Zhou

Clarke

E. J.

(2018). “Not all environmentalists are like that . . .”: Unpacking the negative and positive beliefs and perceptions of environmentalists. Environmental Communication. Advance online publication. doi:10.1080/17524032.2018.1488755

17.

Knudsen

Johannesson

M. P.

(2018). Beyond the limits of survey experiments: How conjoint designs advance causal inference in political communication research. Political Communication. Advance online publication. doi:10.1080/10584609.2018.1493009

18.

Lelkes

(2019). Policy over party: Comparing the effects of candidate ideology and party on affective polarization. Forthcoming, Political Science Research and Methods.

19.

Long

Steinke

Applegate

Knight Lapinski

Johnson

M. J.

Ghosh

(2010). Portrayals of male and female scientists in television programs popular among middle school-age children. Science Communication, 32, 356-382.

20.

Mason

(2018). Uncivil agreement: How politics became our identity. Chicago, IL: University of Chicago Press.

21.

Merkley

Stecuła

D. A.

(2018). Party elites or manufactured doubt? The informational context of climate change polarization. Science Communication, 40, 258-274.

22.

Meyer

Rosenzweig

(2016). Conjoint tools for developing country contexts. The Political Methodologist. Retrieved from https://thepoliticalmethodologist.com/2016/02/18/conjoint-analysis-tools-for-developing-country-contexts/

23.

Montoya

R. M.

Horton

R. S.

(2004). On the importance of cognitive evaluation as a determinant of interpersonal attraction. Journal of Personality and Social Psychology, 86, 696-712.

24.

Motta

(2018). The polarizing effect of the March for Science on attitudes toward scientists. PS: Political Science & Politics, 51, 782-788.

25.

Myers

Kotcher

Cook

Beall

Maibach

(2018, April). March for Science 2017: A survey of participants and followers. Fairfax, VA: George Mason University Center for Climate Change Communication.

26.

O’Keefe

D. J.

(2003). Message properties, mediating states, and manipulation checks: Claims, evidence, and data analysis in experimental persuasive message effects research. Communication Theory, 13, 251-274.

27.

Pew Research Center. (2017). America’s complex relationship with guns: An in-depth look at the attitudes and experiences of US adults. Retrieved from http://www.pewsocialtrends.org/2017/06/22/americas-complex-relationship-with-guns/

28.

Pirlott

A. G.

MacKinnon

D. P.

(2016). Design approaches to experimental mediation. Journal of Experimental Social Psychology, 66, 29-38.

29.

Priest

(2016). Ingredients of a successful climate movement. In Communicating climate change: The path forward (pp. 137-159). London, England: Palgrave Macmillan.

30.

Roberts

(2015). The arguments that convinced a libertarian to support aggressive action on climate. Retrieved from https://www.vox.com/2015/5/12/8588273/the-arguments-that-convinced-this-libertarian-to-support-a-carbon-tax

31.

Ross

A. D.

Struminger

Winking

Wedemeyer-Strombel

K. R.

(2018). Science as a public good: Findings from a survey of March for Science participants. Science Communication, 40, 228-245.

32.

Scheufele

D. A.

(2011). Modern citizenship or policy dead end? Evaluating the need for public participation in science policy making, and why public meetings may not be the answer (Paper No. R-34, Joan Shorenstein Center on the Press, Politics and Public Policy Research Paper Series). Retrieved from https://shorensteincenter.org/modern-citizenship-or-policy-dead-end-dietram-scheufele/

33.

Sedikides

Gregg

A. P.

(2008). Self-enhancement: Food for thought. Perspectives on Psychological Science, 3, 102-116.

34.

Skurka

Niederdeppe

Romero-Canyas

Acup

(2018). Pathways of influence in emotional appeals: Benefits and tradeoffs of using fear or humor to promote climate change-related intentions and risk perceptions. Journal of Communication, 68, 169-193.

35.

Slater

D. M.

Peter

Valkenburg

P. M.

(2015). Message variability and heterogeneity: A core challenge for communication research. Annals of the International Communication Association, 39, 3-31.

36.

Sobieraj

(2011). Soundbitten: The perils of media-centered political activism. New York: NYU Press.

37.

Stuart

Thomas

E. F.

Donaghue

(2018). “I don’t really want to be associated with the self-righteous left extreme”: Disincentives to participation in collective action. Journal of Social and Political Psychology, 6, 242-270.

38.

Třebický

Havlíček

Roberts

S. C.

Little

A. C.

Kleisner

(2013). Perceived aggressiveness predicts fighting performance in mixed-martial-arts fighters. Psychological Science, 24, 1664-1672.

39.

Van Tilburg

W. A. P.

Igou

E. R

. (2014). From Van Gogh to Lady Gaga: Artist eccentricity increases perceived artistic skill and art appreciation. European Journal of Social Psychology, 44, 93-103.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.61 MB