Abstract
Inbar and Lammers (2012, this issue) conducted two surveys of a sample of Society for Personality and Social Psychology listserv members to empirically examine two questions: How many social psychologists are politically liberal, and is there evidence that liberal social psychologists are biased against their conservative colleagues and/or research taking a conservative perspective? They conclude that the field is overwhelming liberal, and that there is ample evidence of bias against both conservatives and conservatively motivated research. Biased sampling, conversational norm and question context effects, missing control conditions, and a focus on hypothesis confirmation instead of hypothesis testing, however, undermine the scientific confidence one can place in their findings. Is there liberal bias? Probably. Is the evidence scientifically sound? Not so much. The field needs to adhere to the same standards of scientific rigor when conducting self-critical research as it requires of normal science.
Jonathan Haidt stirred considerable debate at the 2011 annual conference of the Society for Personality and Social Psychology (SPSP) when he accused the field of liberal bias in a widely attended and subsequently distributed presentation (Haidt, 2011). Professor Haidt had asked some of his fellow social psychologists to nominate conservative colleagues and received very few nominees. He also asked the audience at his talk to raise their hands to identify first whether they were liberal and second if they were conservative: Most in the audience raised their hands in response to the first request; few responded to the second. Finally, Professor Haidt provided anecdotal examples of conservatives who felt uncomfortable “outing” themselves to their liberal counterparts for fear of possible reprisals.
In a related vein, a number of programs of social psychological research have been the target of published critiques of possible liberal bias in recent years (see, for example, the critiques posed by Greenberg & Jonas, 2003; Mullen, Bauman, & Skitka, 2003; Sniderman & Tetlock, 1986; Tetlock, 1994; Tetlock & Mitchell, 2008. For various replies, see Jost, Glaser, Kruglanski, & Sulloway, 2003; Jost et al., 2009; Sears, 1994). Some topics we study in social psychology are inherently more politicized or potentially ideologically loaded than others. Most researchers strive to be value neutral—they consistently make an effort to create a level playing field for hypothesis testing and explore both positive and negative normative spins for empirical results. Value neutrality, however, can be a difficult standard to apply when the scientist is as potentially vulnerable to the same cognitive biases as those s/he studies (e.g., Nickerson, 1998).
Given this backdrop, Inbar and Lammers (2012, this issue) decided to collect data to empirically test whether the field is predominantly liberal and biased against its conservative colleagues and research taking a conservative perspective. They found that social psychologists are more likely to be liberal than conservative and that conservatives experience the field as unwelcoming. Moreover, conservatives’ perceptions have some basis in reality, given a significant proportion of social psychologists sampled openly admitted they would (for example) give greater preference to an openly liberal job candidate than they would to an openly conservative one. Less problematic in my view, those sampled also reported a willingness to negatively review grants and papers with a suspected conservative agenda (one just hopes that the field would similarly “discriminate” against grants and papers with a suspected liberal agenda, something Inbar and Lammers did not test).
Inbar and Lammers’ findings are an important reminder that intolerance and prejudice comes in many forms. Although I am inclined to believe that ideological disparities in the field of social psychology are due much more to self-selection than overt discrimination, these findings nonetheless raise suspicions that discrimination may be a contributing factor. In addition to being just plain wrong to discriminate against people committed to doing good science whose ideological beliefs happen to be different from one’s own, the underrepresentation of conservatives in social psychology no doubt comes with a host of opportunity costs for scientific advancement (see Crawford et al., 2012).
I personally believe Inbar and Lammers’ (2012) description of the problem is a pretty accurate characterization of the field. I say that, however, as a person who lives in a particular social context and who is looking into a subjective mirror, not as a scientist evaluating the quality of the evidence at hand. Regardless of my personal beliefs about the truth value of a certain amount of liberal bias in social psychology, it is important to use the same rules to evaluate self-critical research that we use to evaluate usual science, even when—perhaps especially when—the research confirms our private suspicions. Taken from this perspective, the Inbar and Lammers’ studies are not persuasive science, as I will now explain.
Population Estimation
Social psychologists frequently test hypotheses using convenience samples, and few of us are trained to care very much about sample representativeness and bias. Social psychologists are usually interested in hypothesis testing and discovering what can happen under various controlled circumstances; they are less concerned with deriving population estimates. When theory testing, it is enough simply to discover what can occur under specific circumstances (e.g., the Asch conformity studies; Mook, 1983). One purpose of Inbar and Lammers’ study, however, was to make a population estimate (“How many social psychologists are politically liberal/conservative?”). Sampling—and attendant concerns about sampling bias, response rates, etc.—therefore become essential criteria for assessing the scientific merit of the work.
The sampling frame 1 for the Inbar and Lammers study was SPSP members. There are three flaws to this approach: (a) not all social psychologists are members of SPSP and the choice to use SPSP alone to recruit participants disproportionately represents North American social psychologists, (b) SPSP members who participate on the SPSP listserv may be different from those who do not in unknown ways (e.g., age and ideology), and (c) large proportions of those who responded to the survey were students.
Although the SPSP sampling frame has limitations, let’s nonetheless assume that the total pool of social psychologists worldwide equals the SPSP membership (N = 7,583), of whom 1,939 are SPSP listserv members (25.5%). Of those 1,939 listserv members, 508 (26%) responded to the survey for Study 1, which is 6.6% of the possible “population” of social psychologists. The participation rate is even worse for Study 2 (approximately 4%).
Including nonfaculty in these already very small samples is problematic for a number of reasons (43% in Study 1 and 34% in Study 2 were students or post-docs). Students have not been completely socialized into the field, and a good number of them will never be in a position to make serious decisions about grant reviews or hiring (attrition rates in psychology graduate programs in the U.S. are substantial, see the Council of Graduate Schools, 2008). Thus, even though the survey accurately represents the number of nonfaculty in SPSP, it nonetheless overrepresents the young in its population estimates. Age is also consistently correlated with political orientation (people over 30 tend to be conservative, Saad, 2012) and with listserv and technology adoption (Charness, 2008). Combined, the inclusions of nonfaculty in the sample means that the true number of conservative social psychologists and liberal social psychologists in a position of sufficient power to discriminate against their conservative peers are no doubt underrepresented. Taken together, these problems suggest serious sampling bias (Groves et al., 2004). Even if the demographics of these proportionally small samples match the demographics of the population of SPSP, it is nonetheless impossible to make scientifically credible generalizations about the “true” population of social psychologists given these combined limitations in sampling.
Question Context Effects and Conversational Norms
It is a truism in survey research that people use surrounding questions as a conversational context to understand any given question (see Schwarz, Groves, & Schuman, 1998; Schwarz & Sudman, 1992, for reviews). Inbar and Lammers (2012) asked four questions in the same block (emphasis added—responses to all four items were on a scale of 1, not at all, to 7, very much):
If you were reviewing a research grant application that seemed to you to take a politically conservative perspective, do you think this would negatively influence your decision on the grant application?
If you were reviewing a paper that seemed to you to take a politically conservative perspective, do you think this would negatively influence your decision on the paper?
If you were organizing a symposium, do you think you would be reluctant to invite a colleague who is generally known to be politically quite conservative?
If two job candidates (with equal qualifications) were to apply for an opening in your department, and you knew that one was politically quite conservative, do you think you would be inclined to vote for the more liberal one?
Although Inbar and Lammers (2012) argue that respondents understood the differences between questions that asked about research that took a politically conservative perspective and politically conservative psychologists, the fact that people use surrounding questions as information to inform them about what researchers are really asking muddies the interpretation of what, exactly, the research reveals about bias: Is it bias against conservatives (persons), or bias against politically motivated research agendas? Bias against someone for their political views is patently discriminatory; bias against biased research or researchers, however, is professionally appropriate. If the questions were asked in the order presented by Inbar and Lammers, then it is especially easy to imagine respondents interpreted the questions in terms of the latter frame.
Missing Control Conditions
A closely related problem to the context effects noted above is that Study 2 lacked control conditions. Without knowing how people would respond to liberals and liberally motivated research, or politically neutral scientists and research, we have no way to know for sure what respondents’ responses really mean. With these control conditions, it might be possible to disentangle whether participants were responding to conservatives as people, research with a conservative agenda, or research and persons with any political agenda (whether it is liberal or conservative).
Hypothesis Confirmation
The exclusive focus on bias against conservatives is suggestive that the authors were motivated by an explicit desire to confirm their hypothesis of bias rather than by a desire to provide a level playing field to test their hypothesis. Ideally, the studies should have been designed to allow for a number of various possibilities to emerge: bias, lack of bias, or reciprocal bias. Although an incomplete test, one could argue Inbar and Lammers simply rediscovered ingroup favoritism and outgroup disdain (see Robbins & Krueger, 2005, for a review).
Conclusion
The question of whether the field demonstrates evidence of a liberal bias is an important and serious one. Despite having a number of problems, the Inbar and Lammers article represents an important reminder that reviewers, editors, grant review panelists, and others who serve as the professional gate keepers of the field need to be sensitive to the possibility of political biases in research, regardless of ilk. Moreover, even if the evidence provided by Inbar and Lammers is limited and open to criticism that naysayers may use to system justify the current status quo in social psychology, it nonetheless is a very important reminder that social psychologists need to be tolerant and welcoming of not only demographic diversity in our field, but ideological diversity as well. Going forward, however, we need to be careful to apply the same degree of empirical rigor and burden of proof when engaging in critical self-examination as we bring to our other empirical work.
Footnotes
Declaration of Conflicting Interests
The author declared no conflicts of interest with respect to the authorship or the publication of this article.