Panel Conditioning in the General Social Survey

Abstract

Does participation in one wave of a survey have an effect on respondents’ answers to questions in subsequent waves? In this article, we investigate the presence and magnitude of “panel conditioning” effects in one of the most frequently used data sets in the social sciences: the General Social Survey (GSS). Using longitudinal records from the 2006, 2008, and 2010 surveys, we find convincing evidence that at least some GSS items suffer from this form of bias. To rule out the possibility of contamination due to selective attrition and/or unobserved heterogeneity, we strategically exploit a series of between-person comparisons across time-in-survey groups. This methodology, which can be implemented whenever researchers have access to at least three waves of rotating panel data, is described in some detail so as to facilitate future applications in data sets with similar design elements.

Keywords

panel conditioning longitudinal surveys time-in-survey effects General Social Survey panel data

Sociologists have long recognized that longitudinal surveys are uniquely valuable for making causal assertions and for studying change over time. Scholars have also long been aware of the many special challenges that accompany the use of such surveys: they are more expensive to administer, they raise greater data disclosure concerns, and they suffer from additional forms of nonresponse bias. Nevertheless, researchers have generally been content to assume that longitudinal surveys do not suffer from the sorts of “testing” or “reactivity” biases that sometimes arise in the context of experimental or intervention-based research. The implicit assumption is that answering questions in one round of a survey in no way alters respondents’ reports in later waves. If this assumption is false, scholars risk mischaracterizing the existence, magnitude, and correlates of changes across survey waves in respondents’ attitudes and behaviors.

In this article, we investigate the presence and magnitude of “panel conditioning” effects in the General Social Survey (GSS).¹ The GSS is a foundational data resource in the social sciences, surpassed by only the U.S. Census and the Current Population Survey in terms of overall use (Smith 2008). In 2006, the survey made the transition from a replicating cross-sectional design to a design that uses rotating panels. Respondents are now asked to participate in up to three waves of survey interviews, with an identical set of core items appearing in each wave. The core GSS questionnaire touches on a variety of social and political issues, including abortion, intergroup tolerance, crime and punishment, government spending, social mobility, civil liberties, religion, and women’s rights (to name just a few). Basic sociodemographic information is also collected from each respondent at the time of their interview and then recollected in subsequent rounds.

Our primary objective is to determine whether panel conditioning influences the overall quality of these data.² Along the way, we provide a useful methodological framework that can be used to identify panel conditioning effects in other commonly used data sets. Simply comparing response patterns across individuals who have and have not participated in previous waves of a survey is a good first step, but more sophisticated techniques are needed to convincingly differentiate between panel conditioning and biases introduced by panel attrition (Das, Toepoel, and van Soest 2011; Warren and Halpern-Manners 2012). As we describe in more detail below, our approach (which can be implemented in any longitudinal data set that contains at least three waves of overlapping panel data) resolves this issue by strategically exploiting between-person comparisons across rotation groups.

We believe that this is an important contribution to the emerging literature on panel conditioning effects in social science surveys. Most prior research on this subject, including our own, has focused on the incidence and magnitude of panel conditioning using a narrow subset of attitudinal or behavioral measures (e.g., employment status or life satisfaction). These analyses have tended to use weaker methods to measure panel conditioning effects and have rarely considered the prevalence of the problem across topical domains. In this article, we offer a general assessment of panel conditioning in an omnibus survey that is heavily used by social scientists for a wide variety of research purposes. Our results should be valuable to users of the GSS and to researchers who are interested in identifying panel conditioning effects in other data sets that also include an overlapping panel component.

The remainder of this article is organized into four main sections. In the section that follows, we summarize the literature on panel conditioning and provide a theoretical rationale for examining the issue within the context of the GSS. Next, we describe the methodology we use to identify panel conditioning effects. This discussion is meant to be nontechnical, so as to facilitate future applications in data sets with similar design elements. In the third section, we present our main findings and then subject these findings to a falsification test. Finally, we conclude by discussing the implications of our research for scholars who work with the GSS, as well as other sources of longitudinal social science data.

Panel Conditioning and the GSS

When does survey participation change respondents’ actual attitudes and behaviors? When does survey participation change merely the quality of their reports about those attitudes and behaviors? Elsewhere, we have developed seven theoretically motivated hypotheses about the circumstances in which panel conditioning effects are most likely to occur (Warren and Halpern-Manners 2012).³ These hypotheses are grounded in theoretical perspectives on the cognitive processes that underlie attitude formation and change, decision making, and the relationship between attitudes and behaviors (see, e.g., Feldman and Lynch 1988). In short, responding to a survey question is a cognitively and socially complex process that may or may not leave the respondent unchanged and/or equally able to provide accurate information when reinterviewed in subsequent waves. Five of these hypotheses suggest that panel conditioning effects could potentially arise within the context of the GSS.

First, respondents’ attributes may at least appear to change across waves when items (like many of those featured on the GSS) require them to provide socially nonnormative or undesirable responses (Torche et al. 2012). The experience of answering survey questions can force respondents to confront the fact that their attitudes, behaviors, or statuses conflict with what mainstream society regards as normative or appropriate (Schaeffer 2000; Toh, Lee, and Hu 2006; Tourangeau, Rips, and Rasinski 2000).⁴ Some respondents may react by bringing their actual attitudes or behaviors into closer conformity with social norms. Others may simply avoid cognitive dissonance and the embarrassment associated with offering nonnormative responses by bringing their answers into closer conformity with what they perceive as socially desirable.⁵ In both cases, the end result would be the same: researchers would observe changes over time in respondents’ attributes that would not have occurred had the initial interview not taken place.

Second, respondents’ attributes may appear to change across waves as they attempt to manipulate the survey instrument in order to minimize their burden (see, e.g., Bailar 1989). Respondents sometimes find surveys to be tedious, cognitively demanding, and/or undesirably lengthy (Krosnick 1991; Krosnick et al. 2002; Tourangeau et al. 2000). To get around these hassles, respondents in longitudinal studies may learn how to direct or manipulate the survey experience in such a way that minimizes the overall amount of time or energy that they have to devote to it (Duan et al. 2007; Wang, Cantor, and Safir 2000).⁶ In the GSS, for example, a respondent may learn during their first interview that they are asked to provide many additional details about their job characteristics and work life. In order to reduce the duration of follow-up surveys, some respondents may subsequently report that they are out of the labor force or unemployed.⁷ The result would be the appearance of change across waves when no change has actually occurred.

Third, as hypothesized by Waterton and Lievesley (1989:324), it is possible that some respondents change their answers to survey questions as they gain an “improved understanding of the rules that govern the interview process.” When first interviewed, participants in the GSS may not have had full access to the information requested from them, may not have known how to make use of various response options, or may not have known how or when to ask clarifying questions. Upon reinterview, these individuals may be better prepared and more cognizant of “how surveys work.” While this may translate into undesirable manipulation of the survey instrument, as posited previously, it may also lead to more accurate and complete responses over time. This would again result in the appearance of change over time when respondents’ underlying attributes remain entirely the same (see, e.g., Mathiowetz and Lair 1994; Sturgis, Allum, and Brunton-Smith 2009).

Fourth, respondents may become more comfortable with and trusting of the survey experience after being exposed to the survey process and interviewers (van der Zouwen and van Tilburg 2001). Survey methodologists have found that respondents’ judgments about the relative benefits and risks associated with answering survey questions are significantly related to the chances that they provide complete and accurate answers (Dillman 2000; Krumpar 2013; Rasinski et al. 1999; Willis, Sirken, and Nathan 1994). As respondents become more familiar with and trusting of the survey process and with interviewers and interviewing organizations, they may become less suspicious and their confidence in the confidentiality of their responses may grow. Participating in the GSS may provide evidence about the survey’s harmless nature, reduce suspicion, or increase respondents’ comfort level. Any of these effects could lead to changes in respondents’ reported attitudes or behaviors.

Finally, respondents’ answers to factual questions may change over time as they acquire more and better information about the topic at hand (Toepoel, Das, and van Soest 2009). After an initial interview, respondents may “follow-up” on unfamiliar items by consulting external sources and/or people who are knowledgeable in the area. In this scenario, prior questions serve as stimuli for obtaining the type of information that is needed to give correct responses in later waves. In many cases, it may not even be necessary that respondents remember that they encountered the item during a previous interview. As Cantor (2008:136) points out, all that matters is that “the process of answering the question the first time changes what is eventually accessible in memory the next time the question is asked.” The GSS includes a number of “knowledge tests” that may be especially prone to this form of panel conditioning.

Unfortunately, these hypotheses have not been well validated using the sorts of data sets social scientists typically rely on. One consequence of this is that we know very little about the nature and magnitude of panel conditioning in important data resources like the GSS.⁸ Whereas most large-scale surveys provide users with methodological documentation about issues like sampling, attrition, and missing data, we know of none that routinely provides information about panel conditioning based on strong methods for understanding such biases. In the short run, we hope that our empirical estimates of panel conditioning in the GSS will improve the scholarship that is based on analyses of these data. In the longer run, we intend for our research design to serve as a methodological model for assessing panel conditioning in surveys like the GSS that employ rotating panel designs.⁹

Data and Research Design

The GSS is a large, full-probability survey of noninstitutionalized adults in the United States. It has been administered annually (1972–1993) or biennially (1994 onward) since 1972 by NORC at the University of Chicago. In 2006, the GSS switched from a cross-sectional design to a rotating panel format. Under the new setup, subsets of about 2,000 respondents are randomly selected in each wave for reinterview two and four years later. The longitudinal panel that began the GSS in 2006 was reinterviewed in 2008 and 2010; the panel that began in 2008 was reinterviewed in 2010 and 2012. As described below, our focus is on responses to the 2008 survey by two groups of individuals: those who were interviewed for the first time in 2006 (or cohort A) and those who were interviewed for the first time in 2008 (or cohort B).

At first glance, it might seem that the easiest way to identify panel conditioning effects in these data would be to compare the responses given by individuals who were new to the survey in 2008 (cohort B) to those given by individuals who first participated in 2006 (cohort A). The problem with this approach is its inability to distinguish the effects of panel conditioning from the effects of panel attrition. Whereas the new rotation group may be representative of the original target population (i.e., noninstitutionalized adults living in the United States at the time of the 2008 survey), the 2006 cohort may have suffered from nonrandom attrition between the 2006 and 2008 waves. Unless credible steps are taken to adjust for the resulting panel selectivity, differences in responses between cohorts cannot be clearly attributed to panel conditioning (Halpern-Manners and Warren 2012).

Various methodologies have been proposed to deal with this issue (see, e.g., Das et al. 2011; Warren and Halpern-Manners 2012). One of the most common involves the use poststratification weights (Clinton 2001; Nukulkij et al. 2007). Under this approach, attrition is assumed to be random conditional on a predetermined set of observable characteristics, which are then used to generate weights that correct for discrepancies between different cohorts of respondents. As others have pointed out, the overall effectiveness of this technique depends entirely on whether or not assumptions concerning “ignorability” are met (Das et al. 2011; Sturgis et al. 2009; Warren and Halpern-Manners 2012). If the two cohorts under consideration (i.e., the 2006 and 2008 cohorts) differ in ways that are not easily captured by the variables used to construct the weights, contamination due to panel attrition cannot be ruled out.

One way around this problem is to “preselect” individuals who have the same underlying propensity to persist in the sample. Consider, for example, cohorts A and B as defined previously. These groups of respondents began the GSS in 2006 and 2008, respectively. If we systematically select individuals from both cohorts who participated in at least the first two waves of survey interviews, and then examine their responses in 2008, we can accurately identify the effects of panel conditioning in that year. Both sets of respondents experienced the same social and economic conditions at the time of their interview in 2008, and both exhibited the same propensity to persist in (or attrite from) the GSS panel (because both participated in the same number of waves).¹⁰ The key difference between the groups is that members of the 2006 cohort were experienced GSS respondents in 2008 and members of 2008 cohort were not.¹¹

This is the approach that we use in our analysis. Using panel data from the 2006, 2008, and 2010 waves of the GSS, we were able to identify 3,117 respondents who completed at least the first two rounds of survey interviews.¹² Of these respondents, 1,536 entered the sample in 2006 (the 2006 cohort) and 1,581 entered the sample in 2008 (the 2008 cohort). If the responses given by individuals in the first group are significantly different than the responses given (in the same year) by individuals in the second, we can infer that these differences came about from panel conditioning. No adjustments for panel attrition are necessary and person weights are not needed to correct for subsampling and/or nonresponse.¹³ By design, the 2006 and 2008 cohorts have already been equated on both observed and unobserved characteristics.¹⁴

As we noted previously, items on the GSS span a wide variety of substantive topics (Smith et al. 2007). Although theory suggests that some of these topics may be more or less prone to panel conditioning effects, we feel it is important (for the sake of completeness) to examine every instance in which such biases could possibly occur. For this reason, we considered all 2008 GSS variables that met two very basic requirements: (1) the item had to be answered by the respondent and not the survey interviewer and (2) the variable in question had to be empirically distinct from other measures in our analysis. The first rule meant that items like “date of interview” and “sex of interviewer” were excluded from the study. The second rule meant that we considered variables like “age” and “year of birth,” but not both.¹⁵

After eliminating items that did not satisfy these criteria, we were left with a total of 310 variables. To analyze panel conditioning effects in each of these measures, we carried out hypothesis tests comparing the response patterns in 2008 across cohorts. For continuous measures, we used t-tests to compare group means; for categorical measures, we used chi-square tests (if all cell sizes were in excess of 5) and Fisher’s exact tests (if they were not).¹⁶ Because the GSS employs a split-ballot design, where certain items are only asked of certain individuals in a given year, there is no guarantee that experienced respondents had prior exposure to all of the variables in our sample.¹⁷ Such cases were removed from the analysis using pairwise deletion. See online Appendix Table A1 for complete information on all measures, including sample sizes disaggregated by cohort.

Results

Our analysis includes significance tests for 310 different items; this makes it extremely susceptible to multiple comparison problems. Even if the null hypothesis (of no panel conditioning) is true for every item in our data set, the probability of finding at least one statistically significant effect just by chance is 1 − (1 − 0.05)³¹⁰ ≈ 1, assuming a standard α level of .05. To address this issue, we examined the distribution of test statistics across all items in our sample. Under the null, the p values obtained from our tests should be uniformly distributed between 0 and 1 (Casella and Berger 2001). Approximately 5 percent of the test statistics should be below 0.05, another 5 percent should fall between 0.05 and 0.09, and so on throughout the entire [0, 1] interval. Depending on where they occur in the distribution, departures from this pattern could indicate an overabundance of significant results.

Figure 1 gives a visual summary of the main findings. In the panel on the left, we provide a simple histogram of the p values we obtained from our comparisons of the 2006 and 2008 cohorts. In the panel on the right, we provide a quantile–quantile (Q-Q) plot comparing the empirical distribution of these values (as indicated by the black circles) to a theoretical null distribution (as indicated by the diagonal line).¹⁸ In both instances, there is clear clustering of estimates in the extreme low end of the distribution.¹⁹ Overall, 63 of the 310 tests that we conducted were significant at a .10 level (whereas 31 would be expected by chance); 37 were significant at a .05 level (whereas 16 would be expected by chance); and 22 were significant at a .01 level (whereas 3 would be expected by chance). We take this as good evidence that panel conditioning exists in the GSS among certain subsets of items.

Figure 1.

Histogram and Q-Q plot of observed p values. The panel on the left shows the observed distribution of p values for all items in our sample (n = 310). Under the null, the values should be uniformly distributed between 0 and 1. The panel on the right compares the observed distribution to a theoretical (null) distribution. If p values are more significant than expected, points will move up and away from the diagonal line. If p values are uniformly distributed, the circles will track closely with the line throughout the entire range. See text for further details.

In order to confirm this interpretation, we calculated p values that have been adjusted for the false discovery rate (FDR) using the algorithm of Benjamini and Hochberg (1995). Many techniques exist for dealing with multiple comparison problems and there is some debate over which is the most appropriate (Gelman, Hill, and Yajmia 2012). The FDR is generally thought to be more powerful than Bonferroni-style procedures and is frequently used when the volume of tests is high. Instead of controlling for the chances of making even a single type 1 error, the FDR controls for the expected proportion of type 1 errors among all significant results. In total, the FDR-adjusted estimates include eight significant results at the p < .05 level and 19 significant results at the p < .10 level (see online Appendix Table A1). If we set the FDR threshold to 5 percent, we can say with confidence that only one of these “discoveries” occurred by chance.

The Direction and Magnitude of Panel Conditioning Effects

These results suggest that some people may respond differently to GSS questions based on whether or not they have previously participated in the survey. Although this is an important finding in its own right, users of these data should also be interested in knowing which variables are subject to panel conditioning, in what direction the observed effects operate, and how big they are from a substantive standpoint. In this section, we describe the direction and magnitude of panel conditioning biases in the 2008 survey and provide some preliminary thoughts about possible mechanisms. To be appropriately conservative when interpreting the results for individual variables, we focus on items that (with a few exceptions) produced FDR-adjusted p values < .10. The exceptions to this rule are noted in the text below.

First, members of the 2006 and 2008 cohorts sometimes differed in their responses to attitudinal questions about “hot-button” issues. Examples include items dealing with premarital sex, first amendment rights and racism, and governmental aid to minorities. As indicated in Table 1, members of the 2006 cohort were 14 percent more likely to say that sex before marriage is always or almost always wrong, 10 percent more likely to say that people have a right to make hateful speeches in public, and 23 percent more likely to say that current levels of assistance for African Americans are neither too high nor too low. These effect sizes are generally in line with estimates that have been produced in past panel conditioning research (see, e.g., Torche et al. 2012).

Table 1.

Differences Between Cohorts in 2008, Illustrative Results.

		Estimate (Percentage or Mean)
Variable	Description of Response Options/Measure	2006 Cohort	2008 Cohort
phone	Respondent refuses to give information about their phone	1.17	7.93
visitors	Average number of visitors in the household	0.05	0.01
parsol	Respondent’s standard of living is higher than their parents’ standard of living	66.21	59.50
rplace	The respondent is the householder or their spouse	91.70	88.21
adults	Average number of adults in the household	1.97	1.87
natracey	Respondent thinks current levels of public assistance for blacks are about right	53.51	43.60
marital	Respondent is divorced or widowed	25.88	21.56
spkrac	Respondent agrees that people have a right to make hateful speeches in public	67.08	60.81
rincom06	Respondent refuses to report income	4.27	6.05
famgen	Reports that there is only one generation in household	53.26	57.12
premarsx	Respondent reports that sex before marriage is always or almost always wrong	34.94	30.75
radioact	Correctly answers question about the source of radioactivity	84.79	79.40
viruses	Correctly answers question about efficacy of antibiotics	65.64	59.35
condrift	Correctly answers question about plate tectonics	91.34	87.21
electron	Correctly answers question about sizes of electrons/atoms	75.77	70.45

Note: The 2006 cohort is restricted to respondents who were interviewed in 2006 and 2008; the 2008 cohort is restricted to respondents who entered the panel in 2008 and were also interviewed in 2010. Comparisons between cohorts are made in 2008, the year that they overlap in the sample. All of the variables presented in this table produced false discovery rate (FDR)-adjusted p values below .10, except for the science knowledge items. We included these items because of the consistency across measures (all four were significant by conventional standards and all four effects were in the same, theoretically sensible, direction). See text for more details, and online Appendix Table A1 for the full set of results.

Second, panel conditioning effects emerged in several questions related to household composition. These include items dealing with the respondent’s relationship to the household head (members of the 2006 cohort were more likely to be the head or spouse), the number of adults present (members of the 2006 cohort reported more adults), the number of visitors present (members of the 2006 cohort reported more visitors), and the number of family generations that live with the respondent (members of the 2006 cohort reported more generations). The fact that experienced GSS respondents reported higher numbers in all of these cases may be related to our hypothesis concerning survey skill and/or trust. After completing the survey for the first time, respondents may have become more willing to open up, to report on more people, or to ask follow-up questions about who qualifies as living in their household.²⁰

Third, members of the 2006 and 2008 cohorts frequently differed in their responses to questions about demographic and economic attributes. Respondents with prior survey experience were 20 percent more likely to be divorced or widowed, 11 percent more likely to be upwardly mobile relative to their parents, and 31 percent less likely to refuse to answer questions about their personal income. Although we cannot provide definitive tests, these patterns could also be attributable to differences in respondents’ trust. As we discussed earlier, being interviewed repeatedly may make the interview process seem less threatening to the respondent, which could decrease their need to give guarded and/or socially desirable responses in the follow-up wave (van der Zouwen and van Tilburg 2001). That this would occur for potentially sensitive items like those listed above makes good theoretical sense.²¹

Finally, we found large and consistent differences between groups with respect to their knowledge about science. Although these differences were typically not below the FDR-adjusted p < .10 threshold, the frequency with which they occurred is at the very least suggestive of a “true” effect. As shown in Table 1, respondents in the treatment group were markedly more likely to answer correctly questions about the source of radioactivity, the efficacy of antibiotics in killing viruses, the ongoing process of plate tectonics (condrift), and the relative sizes of electrons and atoms. One possible explanation for these results is the “learning hypothesis” that we proposed earlier: if respondents who previously participated in the GSS seek out information about questions that have one objectively correct answer, we would expect to see differences between cohorts on precisely these sorts of items.

A Note on Exceptions

Although the empirical patterns that we present in Table 1 are generally consistent with theoretical expectations, there are also plenty of counterexamples where the two cohorts of respondents did not differ in predictable or meaningful ways. We did not always find differences between cohorts when examining questions about socially charged issues, nor did we observe significant effects for all items that required factual knowledge or increased levels of respondent trust (for the complete set of results, see online Appendix Table A1). These interitem inconsistencies do not invalidate our findings, but they do suggest the need for more finely grained analyses that are capable of isolating and carefully testing the various hypotheses that we outlined out earlier. We will return to this idea later on in the Discussion section.

Falsification Test

In the final part of our analysis, we carry out a falsification test to confirm the adequacy of our empirical approach. As a part of its mission to provide up-to-date information about a wide variety of topics, the GSS frequently introduces new survey content through the use of special topical modules. This allows us to perform an important methodological check. Using the same analytic setup as before, we can test for differences between cohorts on items that have not previously been answered by anyone in the sample, regardless of which cohort they belong to. In the absence of any contaminating influences, we would expect to see a similar distribution of responses across groups for these measures. Any other result (e.g., nonzero differences between the two cohorts on items that should not, in theory, differ) would call into question the internal validity of our empirical estimates.

We present results from these comparisons in Table 2. In total, there are 19 variables that (1) were not asked in 2006, (2) were asked of both cohorts in 2008, and (3) meet the selection criteria that we defined earlier. Among these items, only one (autonojb) shows any evidence of variation between cohorts, and that evidence disappears when corrections are made for multiple comparisons.²² None of the estimated tests are significant at a .01 level and only two reach significance at the .10 level (with 19 comparisons, we would expect to see ∼1 significant result by chance, assuming a type 1 error rate of 0.05).²³ This is a reassuring finding for our purposes, as it minimizes the possibility that the two cohorts differ in ways that could spuriously produce some or all of what we previously deemed to be panel conditioning effects.

Table 2.

Results From Falsification Tests.

		Tests for Differences Between Cohorts
Variable Description	Name	p	FDR-adjusted p
Trying to start a business	startbiz	.50	.75
Number of full-time jobs since 2005	work3yrs	.67	.78
Number of years worked for current employer	curempyr	.54	.75
Amount of pay change since started job	paychnge	.40	.75
Was pay higher/lower/the same in previous job?	pastpay	.28	.68
Why did the respondent leave their previous job?	whyleave	.35	.75
Does more trade lead to fewer jobs in the United States?	moretrde	.27	.68
Computer use at work	wkcomptr	.12	.66
Can job be done without a computer?	wocomptr	.82	.82
Have any coworkers been replaced by computers?	autonojb	.02	.22
Frequency of meetings with customers, clients, or patients	meetf2f1	.15	.66
Frequency of meetings with coworkers	meetf2f2	.28	.68
Frequency of communication with coworkers outside the United States	intlcowk	.22	.68
Does the respondent receive health insurance from their employer?	emphlth	.82	.82
Is there another name for the respondent’s insurance or HMO policy?	othplan	.58	.75
Gender of sex partners	sexsex18	.09	.66
Ever been the target of sexual advances by a coworker/supervisor?	harsexjb	.55	.75
Has respondent been the target of a sexual advance by a religious leader?	harsexcl	.66	.78
Do they know others who have been the target of sexual advances?	knwclsex	.47	.75

Note: HMO, Health Maintenance Organization; These items were not asked of the 2006 cohort in 2006, but were asked of both cohorts in 2008. Variable names are given in the “name” column. The FDR-adjusted p is the p value adjusted for the false discovery rate. Adjustments were made using the procedures of Benjamini and Hochberg (1995). See text for more details.

Discussion

Sociologists who work with longitudinal data typically assume that the changes they observe across waves are real and would have occurred even in the absence of the survey. Whether or not this assumption is justified is an important empirical question, one that should be of concern to methodologists and nonmethodologists alike. In this article, we provided an analytic framework for detecting “panel conditioning effects” in longitudinal surveys that include a rotating panel component. To demonstrate the utility of our approach, we analyzed data from recent waves of the GSS. Results from these analyses suggest that panel conditioning influences the quality of a small but nontrivial subset of core survey items. This inference was robust to a falsification test and cannot be explained by statistical artifacts stemming from panel attrition and/or differential nonresponse.

What should applied researchers make of these findings? Our analysis suggests that panel conditioning exists in the GSS on a broad scale, but it is much less clear about the specific content domains that are most affected by this form of bias. As we mentioned at the outset, panel conditioning is a complex interactive phenomenon that involves a range of cognitive processes and subjective individual assessments. Predicting when and where it will occur is a difficult theoretical exercise. We have attempted to provide some guidance to users of the GSS by listing the variables that show the most evidence of possible effects. We would advise researchers to weigh this information carefully when conducting studies with these data. Although panel conditioning does not always present itself in an intuitive or internally consistent manner, it would be wrong to dismiss it as an unimportant methodological issue.

There is obviously still work to be done in this area. The analytic techniques described herein can be usefully applied in any longitudinal data set that contains overlapping panels. An interesting future application would be to examine heterogeneity in panel conditioning among different subgroups of respondents. In our analysis, we sought to identify the average treatment effect taken over all members of the sample. In reality, these effects may vary considerably across individuals, across social contexts, and across topical domains (see, e.g., Zwane et al. 2011). A treatment effect of zero in the population may nevertheless be nonzero for certain subgroups with particular experiences and/or predispositions. Identifying who these individuals are, and how they differ from others, would go a long way toward refining our theoretical understanding of why panel conditioning occurs.

Another worthwhile extension would be to conduct stand-alone experiments that allow for a closer examination of possible mechanisms. These experiments would not need to be complicated; it would probably be enough to assign individuals at random to receive alternate forms of a baseline questionnaire and then to ask all questions of all individuals in a follow-up. To speak to the issue in a way that is broadly useful to sociologists, the questions would need to be similar or identical to those that routinely appear in other widely used surveys, like the GSS, and would need to be carefully selected in order to isolate the various social and psychological processes that we described earlier. This would obviously require considerable effort and careful planning, but we believe it is the best way to produce a general and theoretically informed understanding of panel conditioning in longitudinal social science research.

Footnotes

Acknowledgments

We would like to thank Eric Grodsky, Michael Davern, Phyllis Moen, and three anonymous reviewers for their helpful comments and suggestions.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Support for this project was provided in part by a grant from the National Science Foundation (SES-0647710).

Supplemental Material

The online appendix is available at .

Notes

References

Bailar

Barbara A.

1975. “Effects of Rotation Group Bias on Estimates from Panel Surveys.” Journal of the American Statistical Association 70:23–30.

Bailar

Barbara A.

1989. “Information Needs, Surveys, and Measurement Errors.” Pp. 1–24 in Panel Surveys, edited by Kasprzyk

Duncan

G. J.

Kalton

Singh

M. P.

. New York: John Wiley.

Benjamini

Yoav

Hochberg

Yosef

. 1995. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society, Series B (Methodological) 57:289–300.

Cantor

David

. 2008. “A Review and Summary of Studies on Panel Conditioning.” Pp. 123–38 in Handbook of Longitudinal Research: Design, Measurement, and Analysis, edited by Menard

. Burlington, MA: Academic Press.

Casella

George

Berger

Roger L.

. 2001. Statistical Inference. New York: Duxbury Press.

Clinton

Joshua D.

2001. “Panel Bias from Attrition and Conditioning: A Case Study of the Knowledge Networks Panel.” Unpublished manuscript. Department of Political Science, Stanford University. Retrieved April 5, 2006 (http://www.princeton.edu/∼clinton/WorkingPapers/C_WP2001.pdf).

Corder

Larry S.

Horvitz

Daniel G.

. 1989. “Panel Effects in the National Medical Care Utilization and Expenditure Survey.” Pp. 304–18 in Panel Surveys, edited by Kasprzyk

Duncan

G. J.

Kalton

Singh

M. P.

. New York: John Wiley.

Das

Marcel

Toepoel

Vera

van Soest

Arthur

. 2011. “Nonparametric Tests of Panel Conditioning and Attrition Bias in Panel Surveys.” Sociological Methods & Research 40:32–56.

Dillman

Don A.

2000. Mail and Internet Surveys: The Tailored Design Method. New York: John Wiley.

10.

Duan

Naiuhua

Alegria

Margarita

Canino

Glorisa

McGuire

Thomas G.

Takeuchi

David

. 2007. “Survey Conditioning in Self-reported Mental Health Service Use: Randomized Comparison of Alternative Instrument Formats.” Health Services Research 42:890–907.

11.

Feldman

Jack M.

Lynch

John G.

. 1988. “Self-generated Validity and Other Effects of Measurement on Belief, Attitude, Intention, and Behavior.” Journal of Applied Psychology 73:421–35.

12.

Gelman

Andrew

Hill

Jennifer

Yajmia

Masanao

. 2012. “Why We (Usually) Don’t Have to Worry about Multiple Comparisons.” Journal of Research on Educational Effectiveness 5:189–211.

13.

Godin

Gaston

Sheeran

Paschal

Conner

Mark

Germain

Marc

. 2008. “Asking Questions Changes Behavior: Mere Measurement Effects on Frequency of Blood Donation.” Health Psychology 27:179–84.

14.

Halpern-Manners

Andrew

Warren

John Robert

. 2012. “Panel Conditioning in the Current Population Survey: Implications for Labor Force Statistics.” Demography 49:1499–519.

15.

Kessler

Ronald C.

Wittchen

Hans-Ulrich

Abelson

Jamie A.

McGonagle

Katharine

Schwarz

Norbert

Kendler

Kenneth S.

Knäuper

Bärbel

Zhao

Shanyang

. 1998. “Methodological Studies of the Composite International Diagnostic Interview (CIDI) in the US National Comorbidity Survey (NCS).” International Journal of Methods in Psychiatric Research 7:33–55.

16.

Kreuter

Frauke

McCulloch

Susan

Presser

Stanley

Tourangeau

Roger

. 2011. “The Effects of Asking Filter Questions in Interleafed Versus Grouped Format.” Sociological Methods & Research 40:88–104.

17.

Krosnick

Jon A.

1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology 5:213–36.

18.

Krosnick

Jon A.

Holbrook

Allyson L.

Berent

Matthew K.

Carson

Richard T.

Michael Hanemann

Kopp

Raymond J.

Mitchell

Robert Cameron

Presser

Stanley

Ruud

Paul A.

Kerry Smith

Moody

Wendy R.

Green

Melanie C.

Conaway

Michael

. 2002. “The Impact of ‘No Opinion’ Response Options on Data Quality: Non-attitude Reduction or an Invitation to Satisfice?” Public Opinion Quarterly 66:371–403.

19.

Krumpar

Ivar

. 2013. “Determinants of Social Desirability Bias in Sensitive Surveys: A Literature Review.” Quality & Quantity 47:2025–47.

20.

Mathiowetz

Nancy A.

Lair

Tamra J.

. 1994. “Getting Better? Changes or Errors in the Measurement of Functional Limitations.” Journal of Economic & Social Measurement 20:237–62.

21.

McCormick

Maryann K.

Butler

Derrick M.

Singh

Rajendra P.

. 1992. “Investigating Time in Sample Effect for the Survey of Income and Program Participation.” Pp. 554–59 in Proceedings of the Survey Research Methods Section of the American Statistical Association.

22.

Nukulkij

Poom

Hadfield

Jadfield

Subias

Lewis

Evan

. 2007. “An Investigation of Panel Conditioning with Attitudes toward U.S. Foreign Policy.” Retrieved August 25, 2013 (http://www.knowledgenetworks.com/ganp/docs/jsm2009/Panel%20Conditioning%20and%20Attrition_JSM_2009_submitted.pdf).

23.

Pearson

Thomas A.

Manolio

Teri A.

. 2008. “How to Interpret a Genome-wide Association Study.” JAMA 299:1775–877.

24.

Rasinski

Kenneth A.

Willis

Gordon B.

Baldwin

Alison K.

Yeh

Wenchi

Lee

Lisa

. 1999. “Methods of Data Collection, Perceptions of Risks and Losses, and Motivation to Give Truthful Answers to Sensitive Survey Questions.” Applied Cognitive Psychology 13:465–84.

25.

Schaeffer

Nora Cate

. 2000. “Asking Questions about Threatening Topics: A Selective Overview.” Pp. 105–22 in The Science of Self-report: Implications for Research and Practice, edited by Stone

Turkkan

J. S.

Bachrach

C. A.

Jobe

J. B.

Kurtzman

H. S.

Cain

. Hillsdale, NJ: Lawrence Erlbaum.

26.

Sherman

Steven J.

1980. “On the Self-erasing Nature of Errors of Prediction.” Journal of Personality and Social Psychology 39:211–21.

27.

Smith

Tom W.

2008. “Repeated Cross-sectional Research: The General Social Surveys.” Pp. 33–48 in Handbook of Longitudinal Research: Design, Measurement, and Analysis, edited by Menard

. London, England: Elsevier.

28.

Smith

Tom W.

Kim

Jibum

Koch

Achim

Park

Alison

. 2007. The GSS Model of Social-science Research. GSS Project Report No. 27. Chicago, IL: National Opinion Research Center, University of Chicago.

29.

Smith

Tom W.

Son

Jaesok

. 2010. An Analysis of Panel Attrition and Panel Change on the 2006-2008 General Social Survey Panel. GSS Methodological Report No. 118. Chicago, IL: National Opinion Research Center, University of Chicago.

30.

Solon

Gary

. 1986. “Effects of Rotation Group Bias on Estimation of Unemployment.” Journal of Business & Economic Statistics 4:105–09.

31.

Spangenberg

Eric R.

Greenwald

Anthony G.

Sprott

David E.

. 2008. “Will You Read This Article’s Abstract? Theories of the Question-behavior Effect.” Journal of Consumer Psychology 18:102–06.

32.

Sturgis

Patrick

Allum

Nick

Brunton-Smith

Ian

. 2009. “Attitudes over Time: The Psychology of Panel Conditioning.” Pp. 113–26 in Methodology of Longitudinal Surveys, edited by Lynn

. New York: John Wiley.

33.

Toepoel

Vera

Das

Marcel

van Soest

Arthur

. 2009. “Relating Question Type to Panel Conditioning: Comparing Trained and Fresh Respondents.” Survey Research Methods 3:73–80.

34.

Toh

Rex S.

Lee

Eunkyu

Michael Y.

. 2006. “Social Desirability Bias in Diary Panels is Evident in Panelists’ Behavioral Frequency.” Psychological Reports 99:322–34.

35.

Torche

Florencia

Warren

John Robert

Halpern-Manners

Andrew

Valenzuela

Eduardo

. 2012. “Panel Conditioning in a Longitudinal Study of Chilean Adolescents’ Substance Use: Evidence from an Experiment.” Social Forces 90:891–918.

36.

Tourangeau

Roger

Rips

Lance J.

Rasinski

Kenneth

. 2000. The Psychology of Survey Response. New York: Cambridge University Press.

37.

Tourangeau

Roger

Yan

Ting

. 2007. “Sensitive Questions in Surveys.” Psychological Bulletin 133:859–83.

38.

van der Zouwen

Johannes

van Tilburg

Theo

. 2001. “Reactivity in Panel Studies and its Consequences for Testing Causal Hypotheses.” Sociological Methods & Research 30:35–56.

39.

Wang

Kevin

Cantor

David

Safir

Adam

. 2000. “Panel Conditioning in a Random Digit Dial Survey.” Pp. 822–27 in Proceedings of the Survey Research Methods Section of the American Statistical Association.

40.

Warren

John Robert

Halpern-Manners

Andrew

. 2012. “Panel Conditioning Effects in Longitudinal Social Science Surveys.” Sociological Methods & Research 41:491–534.

41.

Waterton

Jennifer

Lievesley

Denise

. 1989. “Evidence of Conditioning Effects in the British Social Attitudes Panel Survey.” Pp. 319–39 in Panel Surveys, edited by Kasprzyk

Duncan

G. J.

Kalton

Singh

M. P.

. New York: John Wiley.

42.

Willis

Gordon B.

Sirken

Monreau G.

Nathan

Gad

. 1994. “The Cognitive Aspects of Responses to Sensitive Survey Questions.” Working Paper Series, No. 9, National Center for Health Statistics, Hyattsville, MD.

43.

Zwane

Alix Peterson

Zinman

Jonathan

Van Dusen

Eric

Pariente

William

Null

Clair

Miguel

Edward

Kremer

Michael

Karlan

Dean S.

Hornbeck

Richard

Giné

Xavier

Duflo

Esther

Devoto

Florencia

Crepon

Bruno

Banerjee

Abhijit

. 2011. “Being Surveyed Can Change Later Behavior and Related Parameter Estimates.” Proceedings of the National Academy of Sciences 108:1821–26.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.21 MB