Abstract
The problem of assessing opinions through surveys with double-barreled items has long been recognized, but nonfield experiments that examine their construction have been limited. Opinions about evolution based on an international multidecade Gallup survey were examined through an experimental design in which answers to the Gallup alternatives, one of which was double-barreled, were compared to those obtained when the components were separated. Random assignment of the original and deconstructed versions to 302 university students resulted in relatively few students who received the deconstructed version providing a pattern of answers that replicated the original item. The Harris organization had previously examined the Gallup survey, but the problem of deconstructing a double-barreled alternative in combination with an experimentally controlled procedure had not been demonstrated. The discussion includes a distinction between complex and compound (double-barreled) survey items and briefly examines the difference between explicit and implicit double-barreled items.
Keywords
Introduction
The problem of ambiguous survey questions is well recognized (Krosnick & Presser, 2010; Pew Research Center, 2017). One category of such questions has been labeled double-barreled. An example of a double-barreled question that has attained some attention from the nonacademic media (Fox News, 2009) was contained in a Gallup survey that examined opinions about evolution (Jones, 2005; Jones & Saad, 2014; Neuport, 2014). Although this survey has been subjected to critical analysis (Bishop, Thomas, Wood, & Gwon, 2010; Gwon, 2012), the methodology of deconstructing a double-barreled item into its components in combination with an experimentally controlled environment has not been evident. The present research developed a deconstruction of the original Gallup survey into its components and compared the relative frequency of the pattern of answers from this version to that obtained from the original. The term compound has been substituted for double-barreled, as there may be more than two components to the wording being examined.
The Gallup survey consisted of the following three alternatives: 1.) God created human beings pretty much in their present form at one time within the last 10,000 years or so, 2.) Human beings have developed over millions of years from less advanced forms of life, but God guided this process, 3.) Human beings have developed over millions of years from less advanced forms of life, but God had no part in this process.
Gallup reported that over 40% of U.S. respondents chose the first alternative and that this percentage has been consistent, with small variations, since 1982 when the survey was first conducted. An inference from the results was that, “In U.S., 42% Believe Creationist View of Human Origins” (Neuport, 2014). The creationist interpretation of the results was an inference because the Gallup survey simply asked a question about human origins. Although the selection of alternative one was consistent with Genesis, it did not establish that the respondent’s answer was Genesis based. In the deconstructed version, additional questions related to the Genesis description were added, such as the age of the universe and when dinosaurs disappeared, to examine the Genesis inference.
Alternative one is a compound item that includes (a) a 10,000-year time frame, (b) the role of the supernatural, and (c) “pretty much in their present form” which may be interpreted as representing biological and/or social development. As the time frame of alternative one differs substantially from that in the other two, a respondent’s choice might conceivably have been based on this factor. Neuport’s inference is based on Genesis chapters 1 and 2 for the universe and the planet and chapters 4, 5, and 11 for the genealogies since the era of Adam and Eve (New King James Version, 1982). On this basis, “pretty much in their present form” consists of physical and mental development as well as social norms. The Gallup survey did not ask about the universe.
Materials and procedure
Two forms of the questionnaire were distributed randomly to 302 university students in Introductory Psychology and in Introduction to Social Behavior over several academic terms from September 2013 through January 2018. The courses were taught at a large urban state university in the Midwest. All respondents were explicitly informed that their participation was anonymous and voluntary. The purpose of the study was described within the appropriate topic in the course. After elimination of fewer than 4% of the forms that were blank or incomplete, 169 responses were obtained to the Gallup form and 139 to the deconstructed version. (An experimental variation in which related materials were presented before the respondent received the deconstructed version was judged to possibly bias the data and was excluded from the analysis.) Data collection occurred at the very beginning of the first meeting of the class prior to any other statements about the course and student anonymity was preserved during the collection of the responses. The Gallup survey attempted to assess a population’s parameter about human origins from a representative sample. The experiment reported here required randomization into a control and an experimental group among a set of subjects. Of course, results from any experiment that is not based on a representative sample of a larger population limit any generalization. An experiment simply establishes that the effects do exist among the group represented by the sample. The instructions for the two forms differed somewhat in the response requested. In the Gallup survey, the respondent selected the preferred alternative among the three provided. In the deconstructed version, for each part, the respondent selected an alternative that was closest to that individual’s opinion. Edlund and Sagarin (2009) have distinguished forced choice from continuous variables. The Gallup survey involved a forced choice. The deconstructed version included a number of alternatives that differed quantitatively within an item. Table 1 presents the two forms.
Original and deconstructed questionnaires.
aOnly the alternatives are from the original. The instructions were added for this study.
Results
The pattern in the deconstructed version that reflected the full set of conditions in alternative one of the original required (a) or (b) in item 1, no in item 3, and yes in item 4. Since yes in item 4 could also be selected by those who believed in evolution, the full pattern was required. The other items in the deconstructed form assess those who might believe in a strict interpretation of Genesis creation which occurred in six days before God rested a day and then created humans. The Genesis consistent answers, therefore, required an approximately 10,000-year frame.
Original and deconstructed comparisons
Table 2 presents the relationship between questionnaire form and alternative selected (or the deconstructed pattern in that form). Observed frequencies are above and expected below the cell diagonal. The probability that form and selection were independent was less than .00003.
Association between questionnaire form and item selection.
Ages of universe, planet, and species
An inference that the selection of the first alternative in the Gallup form was based on a belief in the Genesis account of creation was examined through other items in the deconstructed form. These included the ages of the planet, the time at which dinosaurs disappeared, the appearance of human tool-making (Harmon, 2011), and the age of the universe (Burgess & Quevedo, 2017). Answers based on Genesis required time periods no longer than 10,000 years. The results are summarized in Table 3.
Respondent beliefs about ages of the universe and the earth.
Note: ▪, alternative not offered, refer to Table 1.
ak represents thousands, m, millions and b, billions of years.
bItem added later in data collection, Fall 2016, as last item in the deconstructed form.
Although 12 respondents who received the deconstructed form provided a selection for the age of humans that was consistent with the first alternative in the Gallup survey, only four selected the planet’s age within the same time frame. Among 46 individuals who responded to the age of the universe (this item was not present in early versions), only one identified its age within the 10,000-year period. In contrast, 75 respondents identified tool-making within the Genesis time frame suggesting the possibility that the phrase “in their present form” may have been interpreted as not necessarily purely physical.
Summary and discussion
Summary
A deconstructed form of a double-barreled alternative about evolution resulted in a substantially lower rate of responses consistent with the double-barreled version of human creation. Although 28% chose the first alternative in the original form (consistent with the Gallup report for the national sample when education and age were separately evaluated), only 9% provided a pattern in the deconstructed form consistent with the original alternative, and that was further reduced to 3% when the age of the planet was included with only one respondent selecting the universe within the 10,000-year time period.
Evaluation of the experimental procedure
Experimental procedures have frequently been employed in survey studies. The effects of question wording, manner in which the questionnaire is constructed, respondent motivation, interviewer characteristics, and other nuances of respondents and the data collection process have been extensively studied (Engelbrecht & Edlund, 2016; Kalton, Collins, & Brook, 1978; Krosnick, 1999; Krosnick, Narayan & Smith, 1996; Krosnick & Presser, 2010). Field studies employ standard sampling techniques and are often conducted by telephone or in the field but at the sacrifice of total anonymity of the respondent since the interviewer has identified the person to be interviewed. Laboratory experiments have also been evident. (Rosenthal (2003) presents some subtle, generally nonverbal, communications that may distort results in laboratory experiments.) This study involved a minimal role for the experimenter and was so short in time, a minute of the first activity of the first class meeting, that no verbal or nonverbal communications were observed. Further, self-administration and anonymity reduces artifacts such as social acceptability and social desirability (Engelbrecht & Edlund, 2016; Krosnick, 1999; Krosnick & Presser, 2010).
Although the research was consistent with experimental procedures, with a very simple and direct design, the full sample was based on an accumulation of data over a number of academic terms. Relationships between form and item response were similar throughout the period of data collection.
This study relied on a moderate base rate. Were the proportion of respondents who chose alternative one in the Gallup version very small, much larger samples would be required. Thus, replication in many sub-populations and other countries might result in Type II error.
Form and context or framing of the question has been related to respondent selection and this was specifically demonstrated in the Harris Interactive Survey (Bishop et al., 2010; Gwon, 2012). However, the Harris examination of the Gallup evolution study was conducted by phone and requested a true or false response. Information questions are subject to social influences including social acceptability or social approval (Krosnick & Presser, 2010; Krumpal, 2013) that may include a desire to appear factually competent even when the facts conflict with personal beliefs. For example, a person may believe that global warming is not a result of human activity but is aware that the scientific community is overwhelmingly of the judgment that it is. If interviewed, the respondent might provide an answer consistent with scientific opinion to appear informed about the subject. This study emphasized opinions and excluded in the instructions a recognition of a correct answer. In addition, the Harris study did not deconstruct the compound alternative and that was the fundamental question in this study: Do the parts (deconstruction) add up to the whole?
Logical structure of complex compared to compound items
A general principle is that the respondent’s opinion is more difficult to assess the greater the amount of information processing required. Confusion reduces the accuracy of judgment (Beatty & Willis, 2007). However, there is a distinction between complexity defined by “or” and that derived by “and” structures. The logical structure of “or” complexity is that it may be agreed to if any of the components are true. Nevertheless, a sequence of “or” statements may result in confusion because of difficulty in processing all of the information. Cannell, Oksenberg, Kalton, and Bischoping (1989) present an exemplar of this. Compound or double-barreled questions include a logical structure of a set of “and” statements. If any component is not true, the statement is false. However, a respondent might still select such an item because some portion of it is evaluated as extremely important relative to other choices. The Gallup alternative of this study is a clear example. Even if there was disagreement with some aspect of the compound item, the other alternatives may have been even less acceptable.
Implications of the results and logical analysis
Two related implications may be derived from this research. The first is with respect to applications. The data reinforce the requirement that a respondent be presented with an unambiguous referent. Thus, “How would you rate the overall environment of your workplace?” is implicitly compound and ambiguous. The respondent, on some unascertained basis, may choose among various aspects of the workplace such as relations with supervisory personnel, opportunity for independent and/or creative contributions, recognition for employee contributions, freedom from sexual harassment, and many others. These difficulties may be present in any overall evaluation, ranging from teaching effectiveness to political performance. A second implication is to the instructions to respondents which may contain complexities and ambiguities. For example, the research by Buss, Larsen, Westen, and Semmelroth (1992) contained instructions that included any one of three conditions, the past (with no time period specified), the present, and a hypothetical future (and a respondent could qualify for more than one of the categories.) The assessment of the hypothetical future contains difficulties which were identified by LaPiere’s (1934) research which led to the detailed study of the relationship between attitudes and behavior.
Concrete and implicit compound questions
The evolution question represented a concrete or explicit compound survey alternative. Even when two alternatives are presented that differ in one component, as in the second and third alternatives, the basis for the selection may be unclear. A respondent presented with the second and third alternatives might choose one based on the role of God, but this does not by itself indicate an opinion about evolution. Of course, alternative one might have been chosen, but that included a number of conditions, such as the time frame, that the individual might strongly reject.
Many compound questions are implicit and frequently are of the form of general statements such as job approval of the president or the direction in which the country is heading. An implicit compound item is one in which a number of factors may be assessed in an “and” judgment, but these factors are not identified and may differ both in substance and relative importance among those surveyed. The answers may be adequate to assess the mood of the respondent and may also predict voter preference. Understanding, however, requires additional information (often included in such surveys). In social science, there is sometimes an inference about the reason for a response that is not supported by the information provided. The evidence from this research is consistent with the general principles: “Use simple syntax” and “Ask about one thing at a time…” (Krosnick & Presser, 2010, p. 264).