Measuring risk/benefit perceptions of emerging technologies and their potential impact on communication of public opinion toward science

Abstract

This study presents a systematic comparison of two alternative measures of citizens’ perceptions of risks and benefits of emerging technologies. By focusing on two specific issues (nanotechnology and biofuels), we derive several insights for the measurement of public views of science. Most importantly, our analyses reveal that relying on global, single-item measures may lead to invalid inferences regarding external influences on public perceptions, particularly those related to cognitive schema and media use. Beyond these methodological implications, this analysis suggests several reasons why researchers in the area of public attitudes toward science must revisit notions of measurement in order to accurately inform the general public, policymakers, scientists, and journalists about trends in public opinion toward emerging technologies.

Keywords

public consultations public understanding of science risk perception science communication science policy

1. Introduction

Taking into account citizens’ views of emerging technologies early on in the issue cycle has become a priority for science communication, in part due to lessons learned from public reactions to agricultural biotechnology (Brossard and Shanahan, 2007), but many questions remain regarding the optimal methods to do so. In particular, the lack of familiarity and interest among the public poses challenges to researchers trying to measure public attitudes, and measurement is even more problematic when policymakers and journalists are interested in dichotomized descriptions of public opinion. As a result, a tension exists between the priority of gauging well-informed, deliberative opinions while—at the same time—reporting an “approve-or-disapprove” message that fits a headline format and assumes a singular “public.”

One example of such summative judgments is a measure that is ubiquitous in research on public reactions to emerging technologies: citizens’ assessments of the relative risks and benefits of emerging technologies. This study focuses on at least two problems associated with measuring perceptions in a single-item question format with few response categories (for an overview, see Cacciatore, Scheufele and Corley, in press). First, responses may be biased due to “response order effects” (e.g., Schuman and Presser, 1981). Asking respondents whether the benefits outweigh the risks, for instance, is a much different question than one that offers the “risks outweighing the benefits” as the initial response option, and research suggests that “the earlier in the list [of response options] an acceptable answer appears, the more popular it will be” (Tourangeau, Rips and Rasinski, 2000: 250) among respondents. Second, this form of measurement forces respondents to make subjective judgments about the relative importance of several risks and benefits. Such judgments are often skewed, given people’s tendency to remember unfavorable information about an issue to a greater extent than favorable information (e.g., Gilovich, 1991).

Our study takes both a methodological and a conceptual approach to exploring the appropriateness of survey measures and the proper ways to construct predictive models of public perceptions, with implications for policymakers on issues related to science, technology, and risk. As a starting point, we emphasize the differences between “explanatory” and “descriptive” research (Babbie, 2007: 89–90). Whereas reliability and validity of measurement have received abundant attention in the former domain (e.g., Carmines and Zeller, 1979), the latter domain—where we commonly find policymakers and journalists interacting with researchers—has given much less attention to these methodological concerns (e.g., Tankard, 1976), with detrimental implications for public communication of science and technology.

2. Assessing public opinion of science and technology

In order to outline how measurement can impact models of public opinion of science and technology, it is important to position our study within the context of current work in this area. First, there is a normative component to any assessment of public opinion about emerging technologies. In other words, what is the place of science in democracy and of democracy in science? Some scholars stress an ideal scientific citizen who is both willing and able to take part in the policy decision-making process (e.g., Irwin, 2001), although others have questioned precisely what is meant by citizen participation (Rowe and Frewer, 2005). Strategies for realizing these normative ideals, following from work in political science (e.g., Fishkin, 1991), have focused on consensus conferences and deliberative models of generating informed public opinion (Kleinman, Delborne and Anderson, in press; Powell and Kleinman, 2008).

Second, there is the descriptive component of public opinion, notably the realization that much of the public is neither interested nor capable of taking part in the debate on these issues (Scheufele, 2006). Empirical research suggests that the American public may not possess the basic levels of scientific knowledge and literacy to arrive at the informed opinions that result from deliberative contexts (Miller, 2004). Moreover, differences in levels and types of knowledge within different sub-groups of the public (Corley and Scheufele, 2010) highlight the ongoing tensions between the normative and descriptive components of public opinion of science.

One method of rectifying this tension is to adopt a more strategic approach to public engagement (Nisbet and Mooney, 2007). This perspective is based on research from psychology and social psychology dealing with the “availability heuristic” (Tversky and Kahneman, 1973), i.e., people’s tendency to rely on the pieces of information that are most easily retrievable from memory to make judgments. Often, the only relevant pieces of scientific information are transmitted through mass media, which can have a strong influence on both the accessibility and applicability of particular issue attributes (Scheufele and Tewksbury, 2007). Accordingly, risk perceptions associated with specific technologies vary considerably throughout the public, often in direct contradiction to the quantitative definitions offered by risk experts (Slovic, 1987).

These aspects of the ongoing scholarly debate shed light on our central question of interest: What is the best method of measuring citizens’ perceptions of risks versus benefits of an emerging technology? One approach to achieving the normative goal of citizen involvement is the systematic consultation of public views (Rowe and Frewer, 2005), most often through surveys. In order for this collection of views to be useful to scientists, policymakers, and journalists, however, the opinions must be measured both precisely and in a way that adequately represents broader populations. If they are not, whatever insights might be gleaned from the data may actually misinform those who view effective communication as their central goal. Thus, prior to presenting our empirical examination, we provide a brief review of the nature of survey response and the centrality of proper measurement to good social science investigations.

3. Nature of survey response and measuring risk/benefit perceptions

The cognitive processes involved with responding to a survey question have been researched extensively for decades, and they merit revisiting as we consider how best to tap respondents’ evaluations of emerging technologies. At their simplest, these cognitive processes involve seven steps: (a) interpreting the question, (b) generalizing an opinion, (c) accessing relevant information, (d) deciding how to use that information, (e) “computing” the judgment, (f) formatting a response, and (g) editing the response (Sudman, Bradburn and Schwarz, 1995: 58). The availability of relevant considerations in forming an opinion to report (i.e., steps c through e) has received the greatest amount of attention among theorists (e.g., Zaller and Feldman, 1992), and echoes the concerns outlined above. Importantly, if individuals have not encountered a particular object before reporting their attitude, they rely on the most salient considerations—what’s at the top of their heads—to form that response.

This process has implications for measuring attitudes for at least two reasons. First, if researchers do not offer an identical definition to all respondents, there is no baseline across which to compare responses. Recent surveys of public attitudes toward nanotechnology (e.g., Brossard et al., 2009; Cobb and Macoubrie, 2004; Kahan et al., 2009) and agricultural biotechnology (e.g., Brossard and Nisbet, 2007; Crne-Hladnik et al., 2009) have therefore strived to provide such information. Second, especially for questions asking for global assessments of benefits versus risks, specific personality traits or demographic characteristics may influence responses. For example, since women and men differ systematically in their subjective assessment of risk (Flynn, Slovic and Mertz, 1994), responses to such questions might reflect this overall tendency. Consequently, while the aim of the question is to assess the judgments of a specific scientific or technological issue, responses may in fact reflect more arbitrary tendencies that have little to do with considerations of the attitude object.

In addition, there are potential problems related to the survey instrument. Researchers always confront a conflict between the expediency of space and time (i.e., maximizing the number of constructs measured) versus precision of measurement (i.e., minimizing the potential for random error by including multiple indicators of the same construct). We address the importance of reliable measurement below. Problems associated with the length of questionnaires are well documented (Brehm, 1993), and we address their implications for assessing public views of science in our conclusions.

Potential problems associated with measurement

These criticisms related to measurement are certainly not new to the literature, but it bears repeating that they can become particularly problematic as issues evolve in the policy arena. We choose to focus in particular on the trade-offs inherent in relying on single-item, global attitude reports and more fine-grained variables based upon multiple (and independently measured) items. Thus, a central concern is measurement, which “focuses on the crucial relationship between the empirically grounded indicator(s)—that is, the observable response—and the underlying unobservable concept(s)” (Carmines and Zeller, 1979: 10). We focus on reliability (i.e., that the same result would be obtained with repeated observations) and validity (i.e., accuracy in reflecting the concept intended to be measured) (Babbie, 2007). The latter measurement criterion is commonly considered to be contingent upon the former (Chaffee, 1991).

Assessing validity is more difficult than assessing reliability, in part because no standard coefficient of comparison exists (Carmines and Zeller, 1979). Of the many types of validity relevant to survey measures (Babbie, 2007: 146–147), our study focuses on construct validity (i.e., “the degree to which a measure relates to other variables as expected within a system of theoretical relationships”) (Babbie, 2007: 147). This is the most straightforward type of validity to assess empirically because it involves comparing the focal variables to a variety of other variables available in a given data set (McLeod and Pan, 2005).

The role of attitude formation

So far, our discussion has focused primarily upon methodological concerns. And, while measurement is the primary focus of our study, prior to introducing our specific research questions we reiterate the importance of theory in assessing public views of science. Of particular concern are theories of attitude formation, as well as how responses to survey questions reflect some underlying, latent attitude.

The general approach to the study of attitudes relies on the notion that evaluations of attitude objects are made while considering that object within a specific context (Eagly and Chaiken, 1993). The relationship between beliefs about the object and the accessibility of those beliefs is treated as central in the expectancy value model of attitude formation. Specifically, “a person’s attitude toward an object is determined by the subjective values or evaluations of the attributes associated with the object and by the strength of these associations” (Ajzen and Fishbein, 2008: 2224). From an investigator’s point of view, theory is crucial to understanding which attributes may be relevant to the scientific or technological issue under scrutiny, as questions phrased by a researcher may not represent beliefs about an object that are accessible to a respondent (Ajzen, 2001).

The implications for the present investigation should be clear: there is a trade-off in measuring public attitudes toward these objects within the context of interest to the investigator and measuring them within the context of the respondent’s familiarity with the object. Consequently, while we have criticized survey questions that require respondents to make complex judgments comparing risks and benefits of an issue, improving such measures to tap latent evaluations of these attributes depends upon sound theoretical considerations. Applying theories of attitude formation to the collection of public views about science therefore requires an acknowledgment of “risk” and “benefit” as empirically distinct attributes of an issue that are intimately tied to the context in which they are accessible to an individual.

Research questions

The foregoing overview leads us to several research questions that we will evaluate in this study. We have placed particular emphasis on trade-offs associated with different measures to assess citizens’ views of benefits and risks associated with emerging technologies. This emphasis leads to the following research questions:

RQ1: How does the reliability of a multi-item benefits-versus-risks construct compare to that of a single-item construct?

RQ2: How does the validity of a multi-item benefits-versus-risks construct compare to that of a single-item construct?

With the latter question, we focus on how the validity of a multi-item construct may allow for more valid inferences related to public views of emerging technologies. In addition, because each emerging technology will bring with it a unique set of public concerns and opinion dynamics, it is important to assess these measures across multiple issues.¹ We therefore pose a third research question:

RQ3: How do the reliability and validity of these measures compare across multiple issue contexts?

4. Methodology

Because our study is fundamentally about measurement and its implications for models predicting public opinion, we rely on two secondary data sets that focus on separate emerging technologies. The first is a nationally representative survey of American adults conducted from August to October of 2004 concerning nanotechnology (response rate = 25.6%). The second was carried out from April to June of 2009 regarding biofuels and alternative energy (response rate = 38.8%). This survey was conducted with a representative sample of adults living in one of the top biofuels-producing and -consuming states in the U.S. Both surveys were conducted via a random-digit-dialing procedure through the University of Wisconsin Survey Center. Response rates correspond to the RR3 definition from the American Association for Public Opinion Research Standard Definitions (AAPOR, 2009).

These two surveys provide several advantages for our study. First, with the issue of nanotechnology, we are able to compare our findings with past research findings from national samples. Second, with the issue of biofuels, we capitalize on higher levels of salience and familiarity with this technology among our respondents, as well as complexity of risks and benefits in terms of trade-offs of the technology. The more localized sample is appropriate for these reasons, especially considering that these individuals may have personal experience with biofuels even if they have not considered some of the risk–benefit trade-offs that surround it. Moreover, this allows us to capitalize on the unique situation of having data with different risks/benefits evaluations across the two issues.

Respondents’ evaluations of benefits versus risks

The two surveys included a question asking respondents to make an overall judgment of the risks versus benefits of the respective emerging technology, as well as a battery of questions where respondents provided an evaluation of various risk and benefit statements independently.² We describe these questions and the construction of multi-item indices in this section.

Measures from 2004 nanotechnology survey

The question for the single-item measure of benefits versus risks of nanotechnology was worded as, “Do you think the benefits of developing nanotechnology further outweigh the risks, or do you think the risks outweigh the benefits, or do you think the risks and benefits are about equal?” There were three response categories: “Risks outweigh benefits” (14.0%), “Risks and benefits equal” (56.8%), and “Benefits outweigh risks” (29.2%).

The survey also included a battery of eight items assessing respondents’ evaluations of specific risks and benefits of nanotechnology. Responses to these questions were measured on a ten-point scale (1 = “Strongly disagree,” 10 = “Strongly agree”). The four benefits question were: (1) “Nanotechnology may lead to new and better ways to treat and detect human disease” (M = 7.28, SD = 2.49); (2) “Nanotechnology may help us develop increased national security and defensive capabilities” (M = 6.71, SD = 2.60); (3) “Nanotechnology may lead to new and better ways to clean up the environment” (M = 6.56, SD = 2.65); and (4) “Nanotechnology may give scientists the ability to improve human physical and mental abilities” (M = 6.50, SD = 2.68). The four risk questions were: (1) “Because of nanotechnology we may lose more U.S. jobs” (M = 4.89, SD = 2.95); (2) “Nanotechnology may lead to an arms race between the US and other countries” (M = 4.99, SD = 2.91); (3) “Nanotechnology may lead to the loss of personal privacy because of tiny new surveillance devices” (M = 6.43, SD = 2.88); and (4) “Nanotechnology may lead to the uncontrollable spread of very tiny self-replicating robots” (M = 3.29, SD = 2.64).

Measures from 2009 biofuels survey

Our single-item measure evaluating benefits-versus-risks of biofuels was worded, “And, overall, do you think that the risks associated with biofuels outweigh the benefits, that the benefits outweigh the risks, or that the risks and benefits are about equal?” Responses were again measured in three categories: “Risks outweigh benefits” (12.6%), “Risks and benefits about equal” (58.5%), and “Benefits outweigh risks” (28.9%).

The biofuels survey included sixteen indicators of risk and benefit evaluations. These were grouped, with two benefits and two risks each, into four separate domains where biofuels are relevant: environmental impacts, economic consequences, social/ethical implications, and political ramifications. The benefits questions were: (1) “Biofuels are less damaging to the environment than petroleum-based fuels” (M = 6.35, SD = 2.60); (2) “Biofuels burn cleaner than regular gasoline” (M = 6.21, SD = 2.67); (3) “Biofuels production will create more jobs” (M = 6.22, SD = 2.53); (4) “Developing domestic biofuels will help strengthen the U.S. economy” (M = 6.30, SD = 2.62); (5) “Biofuels will help the U.S. maintain global leadership in science and technology” (M = 5.98, SD = 2.61); (6) “Biofuels enable us to turn agricultural waste into energy” (M = 6.87, SD = 2.50); (7) “Increasing production of biofuels will reduce our dependence on foreign oil” (M = 6.52, SD = 2.80); and (8) “By investing in biofuels, the U.S. government can join the international fight against global warming” (M = 5.63, SD = 2.91).

The risks questions were: (1) “Biofuels will have negative environmental impacts” (M = 4.48, SD = 2.44); (2) “Biofuels production will threaten plants and wildlife” (M = 4.28, SD = 2.52); (3) “Biofuels will increase fuel costs” (M = 5.39, SD = 2.63); (4) “Biofuels production will lead to an increase in the price of food” (M = 5.80, SD = 2.72); (5) “Biofuel plants reduce the quality of life in surrounding communities” (M = 4.02, SD = 2.37); (6) “Recent increases in biofuels production have contributed to world hunger” (M = 4.16, SD = 2.68); (7) “Developing biofuels takes resources away from other renewable energy solutions, such as wind and solar” (M = 4.79, SD = 2.67); and (8) “Government mandates to use more biofuels put unfair restrictions on U.S. industry” (M = 5.42, SD = 2.58).

Construct validity measures

For the purposes of assessing the validity of respondents’ evaluations of benefits versus risks of nanotechnology and biofuels, we compared how the two benefits-versus-risks variables were related to several external criteria.

Measures from 2004 nanotechnology survey

Demographic indicators included age (M = 50.01, SD = 17.72), sex (42.0% male) and education (Mdn = 5.00, “College 1 year to 3 years”; ordinal scale ranged from 1 = “Never attended school or only attended kindergarten” to 6 = “College 4 years or more”). Cognitive schema included political ideology (two-item mean index with “conservative” coded high: M = 4.23, SD = 1.38; r = .52, p < .001) and factual nanotechnology knowledge (six-item additive index: M = 3.75, SD = 1.55). Media use measures included attention to political news on television (two-item index M = 5.47, SD = 2.87; r = .80, p < .001) and in newspapers (two-item index M = 6.42, SD = 2.51; r = .70, p < .001), as well as attention to science news on television (two-item index M = 4.63, SD = 2.69; r = .82, p < .001) and in newspapers (two-item index M = 5.49, SD = 2.61; r = .85, p < .001). Media attention was measured on an 11-point scale (0 = “No attention at all,” 10 = “A great deal of attention”).

Measures from 2009 biofuels survey

Similar measures were included from the biofuels survey: age (M = 57.38, SD = 16.24), sex (42.3% male) and education (Mdn = 5.00, “College 1 year to 3 years (some college or technical school)”; ordinal scale ranged from 1 = “Never attended school or only attended kindergarten” to 8 = “Graduate degree”). Political partisanship was assessed with two filter questions, resulting in a single seven-point measure (M = 4.15, SD = 2.08; 1 = “Strongly Republican,” 7 = “Strongly Democrat”). Knowledge about biofuels was assessed with nine true/false questions (additive index, M = 4.95, SD = 1.64). Attention to political news on television (two-item index, M = 6.62, SD = 2.65; r = .70, p < .001) and in newspapers (two-item index, M = 5.83, SD = 3.05; r = .81, p < .001) and attention to science news on television (two-item index, M = 5.80, SD = 2.65; r = .76, p < .001) and in newspapers (two-item index, M = 5.03, SD = 2.89; r = .77, p < .001). These indicators were measured on an 11-point scale (0 = “No attention at all,” 10 = “A lot of attention”).

Analytic strategy

The analysis proceeded in two phases. The first phase involved comparing responses to single-item versus multi-item indicators of a benefits-versus-risks construct. We focused on assessing how well respondents were able to fulfill a request for an overall, global evaluation versus one based upon more fine-grained responses to specific benefits and risks. In the second phase, we assessed the validity of both measures. This involved comparing the relationships with external criteria through zero-order correlations and multiple regression.

5. Results

There are two recommended approaches to assessing reliability: internal consistency and alternative-form method (Carmines and Zeller, 1979: 50). Since evaluating internal consistency requires at least two or more measures, we can only evaluate a multi-item index in this way. We can also, however, simulate an alternative-form comparison by looking at the distributions of responses on our single-item measures and a recoded version of our multi-item variables.

Internal consistency

We relied on two coefficients for internal consistency: Cronbach’s alpha (Cronbach, 1951) and the average inter-item correlation (Carmines and Zeller, 1979). The latter is particularly valuable for our study because the value of Cronbach’s alpha tends to inflate as the number of indicators increases. From the statements assessing respondents’ views of the risks and benefits (four each for nanotechnology and eight each for biofuels) we constructed two independent indices for each issue: perception of risks and perception of benefits. The nanotechnology risks index (M = 4.91, SD = 2.16; alpha = .71, average inter-item r = .38) and benefits index (M = 6.75, SD = 2.12; alpha = .81, average inter-item r = .53) each displayed adequate levels of internal consistency. Similarly, the biofuels risks index (M = 4.79, SD = 1.64; alpha = .79, average inter-item r = .32) and benefits index (M = 6.22, SD = 2.05; alpha = .90, average inter-item r = .52) were also highly reliable.

These levels of internal consistency for each index suggest that their constituent indicators tap the underlying attitude constructs very well. The next step in constructing our overall benefits-versus-risks variable was to calculate a relative measure, by subtracting the risks variable from the benefits variable. In both cases, the resulting index has a possible range from -9.00 (“risks outweigh benefits completely”) to +9.00 (“benefits outweigh risks completely”). We define this variable, based upon highly reliable measures of the latent risk and benefit evaluations, as a continuous and highly reliable version of the single-item measure.

Alternative-form comparison

An alternative-form evaluation of reliability relies on evaluating the same underlying construct using two different, independent tests. These two “tests” corresponded to our independent measurement of benefits-versus-risks with the single-item versus multi-item variables. An important drawback to the alternative-form method is its inability “to distinguish between true change from unreliability of measure” (Carmines and Zeller, 1979: 40). For our purposes, this is less of a concern because we assume that no true change occurs while a respondent is answering the survey questionnaire.

Comparison of means

We first examined the mean value on our multi-item index of benefits-versus-risks across the three response categories on the single-item measure. The results of this comparison were fairly ambiguous. In the nanotechnology survey, those responding to the single-item question that “the risks outweigh the benefits” scored, on average, 0.40 (SD = 2.27; N = 82) on the multi-item index, while those responding that “the risks and benefits are about equal,” and that “the benefits outweigh the risks” had average multi-item index response scores of 1.62 (SD = 2.33; N = 333) and 3.22 (SD = 2.38; N = 176), respectively. For the biofuels survey, responses that “the risks outweigh the benefits” corresponded to an average of -2.57 (SD = 2.82; N = 71) on the multi-item index, “risks and benefits are about equal” to a mean of 1.24 (SD = 2.08; N = 330), and “the benefits outweigh the risks” to a mean of 3.73 (SD = 2.24; N = 163).

Mean differences were evaluated by comparing confidence intervals on the multi-item index within each of the three response categories, which provided a range of likely values for the population parameter (Wooldridge, 2003: 145). Computing the 95% confidence interval revealed substantial overlap among mean values on the multi-item measure and the three categories of the single-item measure. For nanotechnology, this variation was substantially less (−4.14 < µ_risks < 4.94; −3.04 < µ_equal < 6.28; −1.54 < µ_benefits < 7.98) than for biofuels (−8.21 < µ_risks < 3.07; −2.92 < µ_equal < 5.40; −0.75 < µ_benefits < 8.21).³ Nonetheless, in all three groups for both issues the confidence interval overlapped with the zero-point, suggesting that certain respondents commit errors in reporting a summative judgment of benefits-versus-risks. Of course, this comparison has two limitations. First, it does not allow us to evaluate which individuals are committing these errors. Second, a true alternative-form reliability test would require exactly equivalent forms of the two measurements (i.e., comparing a three-category measure with another three-category measure).

Comparison of distributions

With the above limitations in mind, we undertook an additional comparison by recoding the multi-item benefits-versus-risks indices to correspond roughly to the single-item (i.e., three-category) measure. We employed this method to approximate an alternative-form comparison in two steps. First, we recoded the multi-item index so that all scores below zero corresponded to “risks outweigh benefits,” all scores above zero corresponded to “benefits outweigh risks,” and scores equal to zero corresponded to “risks and benefits about equal.” This negative/positive split was highly restrictive because it assumed survey respondents who voluntarily assign themselves to the “about equal” middle category responded to a battery of risk and benefit items such that, on balance, they really did come out equal.

In spite of this restrictive assumption, the results of our comparison demonstrated just how fraught with error self-categorization on the single-item measure can be (see Table 1). Respondents self-selecting into the “benefits outweigh risks” category were the least susceptible to err; for both issues, at least 90% of respondents correctly placed themselves. For biofuels, a majority of respondents (81.7%) correctly placed themselves in the “risks outweigh benefits” category. For nanotechnology, however, within the same category a majority of respondents (50.0%) reliably reported seeing more benefits than risks, according to our multi-item index. Across the two issues, those placing themselves in the middle category (“risks and benefits about equal”) were more likely to fall on the positive than negative side of the multi-item index. The disparity for this category, however, is likely linked to our restrictive comparison method.

Table 1.

Comparison of single-item distributions with reduced multi-item categories based on simple negative/positive split.

Single-item response	Recoded multi-item index
	Risks outweigh benefits	Risks and benefits equal	Benefits outweigh risks	Total
Nanotechnology
Risks outweigh benefits	35.4%	14.6%	50.0%	100.0%
Risks and benefits equal	18.9%	7.5%	73.6%	100.0%
Benefits outweigh risks	6.3%	2.8%	90.9%	100.0%
Total	17.4%	7.1%	75.5%	100.0%
Biofuels
Risks outweigh benefits	81.7%	1.4%	16.9%	100.0%
Risks and benefits equal	23.3%	2.7%	73.9%	100.0%
Benefits outweigh risks	3.1%	.0%	96.9%	100.0%
Total	24.8%	1.8%	73.4%	100.0%

Note: N = 607 (nanotechnology), 564 (biofuels). Cell entries are row percentages.

The second comparison relaxed the restriction on “about equal” evaluations. We recoded the multi-item index according to its distribution within each survey sample, i.e., based upon where respondents scored relative to each other. Thus, the top third were assigned to the category “benefits outweigh risks,” the bottom third to “risks outweigh benefits,” and the middle third to “about equal.”

The results of this second comparison, while less restrictive, replicated several of the same errors of self-categorization from our first comparison (see Table 2). For nanotechnology, the highest level of correct self-categorization was only 58.0% (“benefits outweigh risks”). For biofuels, respondents fared better (84.5% correct on “risks outweigh benefits”). Across the two data sets, we saw the most errors in self-categorization in the middle category, with fewer than half of respondents correctly categorizing themselves (34.2% for nanotechnology and 43.9% for biofuels) and wrong self-categorizations split roughly equally between the two remaining categories.

Table 2.

Comparison of single-item distributions with reduced multi-item categories based on three-way split of sample distribution.

Single-item response	Recoded multi-item index
	Risks outweigh benefits	Risks and benefits equal	Benefits outweigh risks	Total
Nanotechnology
Risks outweigh benefits	56.1%	28.0%	15.9%	100.0%
Risks and benefits equal	35.7%	34.2%	30.0%	100.0%
Benefits outweigh risks	11.4%	30.7%	58.0%	100.0%
Total	31.3%	32.3%	36.4%	100.0%
Biofuels
Risks outweigh benefits	84.5%	11.3%	4.2%	100.0%
Risks and benefits equal	33.3%	43.9%	22.7%	100.0%
Benefits outweigh risks	5.5%	22.7%	71.8%	100.0%
Total	31.7%	33.7%	34.6%	100.0%

Note. N = 607 (nanotechnology), 564 (biofuels). Cell entries are row percentages.

Overall, our approximated alternative-form method for assessing the reliability of the single-item, three-category risks-versus-benefits item revealed substantial measurement error. According to our less restrictive second comparison fewer than half (44.3%) of respondents to the nanotechnology survey correctly self-categorized, while slightly more than half (57.1%) of respondents to the biofuels survey reported correct judgments. In sum, there existed substantial error in respondents’ abilities to self-categorize their overall judgments according to the three response categories of the single-item survey measure.

Assessing construct validity

As outlined earlier, there are several types of validity we might evaluate regarding benefits-versus-risks evaluations of an emerging technology. The present empirical analysis focuses on construct validity, i.e., how the variable of interest itself relates to other theoretically relevant variables. The aim here is to determine if using a less precise (i.e., unreliable) measure inhibits us from making accurate (i.e., valid) inferences regarding the relationships between variables.

First, we revisit the original responses to the two single-item measures across data sets—the distributions are strikingly similar. Just over half of respondents (56.8% and 58.5% for nanotechnology and biofuels, respectively) selected the middle category, with the remainder split unevenly between “risks outweigh benefits” (14% for nanotechnology, and 12.6% for biofuels) and “benefits outweigh risks” (29.2% for nanotechnology, and 28.9% for biofuels). A key question for this single-item measure should be whether the similarities in distributions are due more to the question format than its content.

Second, we address the validity of the single-item measure of benefits versus risks.⁴ To do so, we compare the means on a series of external criteria across the three possible response categories and test for differences using an omnibus one-way analysis of variance. In Table 3, we report descriptive statistics on nine criterion variables for the two technologies. Looking first at the results for nanotechnology, across the three categories of the single-item measure we find significant differences on sex (males are more likely to self-categorize into “benefits outweigh risks”), education, and attention to news media (where higher levels of these variables correspond to an increased likelihood of self-categorizing into the same “benefits outweigh risks” category). There were no significant between-groups differences on age, political ideology, or issue-specific knowledge. For biofuels, we found significant differences that are not at all similar. For this issue, age (younger respondents more likely to self-categorize in “benefits outweigh risks” category), sex (males much more likely to self-categorize into “risks outweigh benefits”), education (lower levels correspond to greater likelihood of responding “risks and benefits about equal”), and issue-specific knowledge (lower levels associated with “benefits outweigh risks”) yielded significant differences. There were no differences in political predispositions or across the four measures of attention to news media.

Table 3.

Comparison of means on criterion variables by single-item category.

	Nanotechnology				Biofuels
	Risks outweigh benefits	Risks and benefits equal	Benefits outweigh risks	ANOVA Sig.	Risks outweigh benefits	Risks and benefits equal	Benefits outweigh risks	ANOVA Sig.
Age	50.44	50.32	46.51	n.s.	58.10	58.97	53.90	**
	(18.27)	(18.16)	(16.34)		(14.31)	(16.83)	(15.14)
Sex (male)	40.00%	36.23%	60.45%	***	70.42%	38.79%	40.49%	***
Education	4.71	4.84	5.35	***	5.46	4.77	5.36	***
	(0.92)	(1.02)	(0.92)		(1.44)	(1.24)	(1.28)
Ideology (conservative)	4.18	4.24	4.08	n.s.	n/a	n/a	n/a	n/a
	(1.48)	(1.33)	(1.41)		n/a	n/a	n/a
Partisanship (Democrat)	n/a	n/a	n/a	n/a	3.69	4.16	4.30	n.s.
	n/a	n/a	n/a		(1.81)	(2.10)	(2.12)
Knowledge	3.69	3.94	3.98	n.s.	5.06	5.15	4.72	*
	(1.55)	(1.44)	(1.27)		(1.43)	(1.59)	(1.51)
Attention to politics (TV)	5.16	5.14	6.21	***	6.23	6.71	6.60	n.s.
	(3.02)	(2.85)	(2.79)		(3.00)	(2.52)	(2.72)
Attention to science (TV)	4.39	4.40	5.56	***	5.54	5.92	5.81	n.s.
	(2.99)	(2.63)	(2.54)		(3.00)	(2.48)	(2.77)
Attention to politics (NP)	6.00	6.13	7.14	***	6.18	5.85	5.81	n.s.
	(2.71)	(2.53)	(2.33)		(3.18)	(2.94)	(3.09)
Attention to science (NP)	5.26	5.12	6.68	***	5.43	5.06	5.05	n.s.
	(2.91)	(2.50)	(2.31)		(3.09)	(2.80)	(2.96)
(N)	82	333	176		71	330	163

Note: *p ≤ .05, **p ≤ .01, ***p ≤ .001. Means and standard deviations (in parentheses) are reported for continuous variables. Percentages are reported for dichotomous variables. The final columns (ANOVA Sig.) for each technology are omnibus F-test for significant differences across the three groups.

Taken together, these across-issue comparisons provided no discernible patterns of response likelihood across the three categories of the single-item measure. Respondents seemed to be basing their answers to these questions, at least in part, on considerations related to the attitude object (nanotechnology or biofuels), suggesting some level of construct validity even in the absence of precise measurement.

A second comparison concerned the validity of the single-item and multi-item measures as dependent variables in an ordinary least squares regression model. In spite of the evident lack of reliability in the former measure, we present this comparison to illustrate the possibly erroneous inferences that may be drawn from such analyses. We are therefore primarily interested in differences in models within each issue rather than across issues.

The comparison models for the nanotechnology survey are reported in Table 4. As expected, both the zero-order correlations and upon-entry regression coefficients are attenuated when comparing the single-item to multi-item dependent variable models. These slight differences in magnitude, however, are less of a concern than are the missing significant relationships between the two models. The model predicting the highly reliable multi-item dependent variable unambiguously revealed positive relationships with nanotechnology knowledge and attention to science news on television. These findings are fairly consistent with past research in this area (e.g., Brossard et al., 2009; Cacciatore et al., in press; Lee, Scheufele and Lewenstein, 2005), but are missing in the model predicting the alternative dependent variable.

Table 4.

Zero-order correlations and results of OLS regressions predicting benefits versus risks for nanotechnology based on single-item and multi-item measures.

	Single-item dependent variable		Multi-item dependent variable
	Zero-order	β	Zero-order	β
Block 1: Demographics
Age	−.08*	−.07	−.10**	.00
Sex (male)	.17***	.13**	.22***	.23***
Education	.23***	.23***	.28***	.24***
Incr. R2 (%)	–	9.9 ***	–	14.4 ***
Block 2: Cognitive schema
Ideology (conservative)	−.04	−.03	−.05	−.04
Knowledge	.06	−.01	.14***	.09*
Incr. R2 (%)	–	0.0	–	1.3 *
Block 3: Attention to media
Political news (TV)	.14**	.03	.13**	−.06
Science news (TV)	.17***	.02	.27***	.22**
Political news (NP)	.17***	.00	.17***	.07
Science news (NP)	.21***	.13	.23***	−.03
Incr. R2 (%)	–	2.5 *	–	3.7 ***
Model R2 (%)	–	12.5	–	19.3

Note: *p ≤ .05, **p ≤ .01, ***p ≤ .001. N = 607. Cell entries under “zero-order” heading are Pearson’s r correlation coefficients and standardized regression betas under “β” heading.

Results for the pair of biofuels models are reported in Table 5. The most notable discrepancy between these models is the absence of a significant effect for political partisanship in the first model, but its strong presence in the second. The absence is not trivial, considering the major role that partisanship plays in determining how citizens view biofuels and alternative energy technologies in the U.S. (Brigham Schmuhl et al., 2010).

Table 5.

Zero-order correlations and results of OLS regressions predicting benefits versus risks for biofuels based on single-item and multi-item measures.

	Single-item dependent variable		Multi-item dependent variable
	Zero-order	β	Zero-order	β
Block 1: Demographics
Age	−.11*	−.14**	−.07	−.13**
Sex (male)	−.14**	−.14**	−.12**	−.13**
Education	.05	.04	−.03	−.04
Incr. R2 (%)	–	3.5 ***	–	2.9 **
Block 2: Cognitive schema
Partisanship (Democrat)	.08	.06	.18***	.15***
Knowledge	−.09*	−.12**	−.09*	−.12**
Incr. R2 (%)	–	1.6 *	–	3.7 ***
Block 3: Attention to media
Political news (TV)	.03	.04	.07	.07
Science news (TV)	.02	.03	.09*	.04
Political news (NP)	−.03	−.05	−.03	−.13
Science news (NP)	−.03	−.01	.01	.08
Incr. R2 (%)	–	0.5	–	1.2
Model R2 (%)	–	5.6	–	6.4

Note: *p ≤ .05, **p ≤ .01, ***p ≤ .001. N = 593. Cell entries under “zero-order” heading are Pearson’s r correlation coefficients and standardized regression betas under “β” heading.

6. Discussion

The goals of this study were two-fold. First, we evaluated single-item versus multi-item measures of perceptions of risks-versus-benefits of two emerging technologies. Second, we sought to demonstrate how differences in measures can influence the validity of inferences drawn from statistical models. The results have important implications for undertaking public opinion surveys, and for communication of survey results to a wide variety of different audiences, including policymakers, scientists, and journalists. Before elaborating upon these implications, we discuss several potential limitations of this study.

One potential limitation is assuming that errors associated with measurement are intrinsic to the survey respondent rather than considering extrinsic factors (e.g., the structure of the survey questionnaire). Such effects have been postulated elsewhere (e.g., Cacciatore et al., in press) and may hamper our comparison across two surveys of different populations. We are confident, however, that extrinsic factors had little influence on our analysis for two reasons. First, the questionnaire structures for our external criterion variables were uniform, i.e., both surveys began with questions about news media attention and ended with measurement of demographic characteristics. Second, the batteries of risks and benefits items were randomized within blocks to minimize question-order effects. While the order of these blocks differed between surveys, analyses looking at the potential effects of question wording (not reported here) revealed no systematic differences due to questionnaire structure.

A second limitation is our evaluation of measures in the context of two specific technologies. We believe this in-depth analysis is a valuable return to the fundamentals of survey measurement and how it can influence policy decisions. Further research is certainly needed, particularly beyond the single-item versus multi-item comparison we conducted. Future inquiries may also benefit from incorporating a diverse set of methodological tools for assessing respondents’ perceptions of risk and attitudes toward scientific issues. These methodological tools may include, but should not be limited to, factor and principal components analyses (e.g., Klop and Severiens, 2007; Pardo and Calvo, 2002), as well as cluster analysis (e.g., Klop and Severiens, 2007).

Finally, the present study focuses on comparisons between a single, global measurement of benefit vs. risk perceptions with a measurement consisting of multiple risk and benefit indicators from several contexts. The latter form of measurement may not be suitable for determining overall evaluations toward emerging technologies. Rather, it may be described more accurately as a measure of attitudes toward particular classes of risks and benefits (e.g., economic risks and benefits or environmental risks and benefits). Future research should explore how specific classes of risks and benefits determine more global evaluations toward science issues.

Measurement implications

Given the results of our investigation, the implications for survey measurement are relatively clear: single-item measures that ask respondents to make complex judgments of a potentially unfamiliar attitude object can be plagued by random error. Of course, we saw a great deal more error present in evaluations of nanotechnology. At least part of this unreliability is likely due to a lack of familiarity with this issue compared to biofuels, though the general public also seems quite unfamiliar with the different facets of this latter issue as well (Wegener and Kelly, 2008).

It is also problematic in terms of the specific risks and benefits that come to mind among respondents. Concerns related to face and content validity should receive more attention from researchers considering the very different notions that underlie experts’ and scholars’ perception of risk versus lay publics’ (Slovic, 1987). As emerging technologies become more familiar, it becomes necessary to account for distinct domains in which they may differ in their risks and benefits (Cacciatore et al., in press). A multi-item measure does little to overcome these validity shortcomings unless it is based upon a battery of theoretically derived indicators.

The overall implications of using either of the two measures for assessing the dynamics of public opinion toward emerging technologies, however, was made most clear with our joint assessment of construct validity. Violations of the assumptions of multiple regression notwithstanding (i.e., using a categorical measure as dependent variable; see Wooldridge, 2003), existing research that has used a single-item measure of a “benefits-versus-risks” construct may have led to flawed conclusions.

Implications for communicating public opinion of science

Such inferential errors have direct implications for disseminating results of studies examining public opinion of science. We find ourselves in an era of press/politics, where the reporting and production of survey results has become a cottage industry among media outlets in addition to scholarly endeavors (von Hoffman, 1980). While professional associations can provide some oversight of survey-results-as-news, it is ultimately an ethical responsibility for researchers to employ the best measures available. This includes assessing reliability and validity of measurement, as well as distilling results for a variety of audiences, including journalists—who may have little experience with science (Friedman, Dunwoody and Rogers, 1999), let alone social science—in reporting summaries of public opinion research results. It also includes further theoretical work that might delineate appropriate contexts in which to measure evaluations of emerging technologies, such as political, economic, or social/ethical domains. Multivariate analysis of latent constructs requires the interchangeability of indicators from within the same universe of items (McLeod and Pan, 2005), and accounting for issue contexts should therefore be a priority for future research.

While our focus with this study has important implications for public opinion researchers, a final additional concern is communicative in nature. Although the social researcher and the journalist, for example, may be interested in an examination of the same phenomenon, their approaches may be very different from one another (Tankard, 1976). As a result, we make two recommendations for researchers. First, as our results indicate, they should avoid using single-item measures of complicated judgments when respondents are unlikely to be able to categorize their views correctly. Second, even if researchers are able to rely on more valid and reliable measures of public views, they have a responsibility to translate them into results that are meaningful to journalists, policymakers, and—ultimately—the general public. Limits on available resources for including multiple questions undoubtedly pose problems for the first recommendation. We chose to use a large number of indicators for our examination in part because they were available to us. Assessing public views based upon even two indicators will be better than relying on a single-item measure that may yield results fraught with error.

To the extent that scholars are focusing on the systematic consultation of public views (Rowe and Frewer, 2005), we should always keep measurement in mind as a fundamental component of theory-building and refinement (Chaffee, 1991; McLeod and Pan, 2005). We view our study as a valuable step toward prioritizing measurement in order to further our knowledge of fluctuating public opinion in the area of scientific and technological risk.

Footnotes

Acknowledgements

This work was supported by the National Science Foundation (grant number SES-0531194); and the United States Department of Agriculture (grant number MSN-120801). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the United States Department of Agriculture.

Notes

Author Biographies

Andrew R. Binder is an assistant professor in the Department of Communication and associate director of the Public Communication of Science and Technology (PCOST) Project at North Carolina State University. He researches the mechanisms underlying public opinion toward scientific controversies, including the roles of attitude formation, media use, and news coverage.

Michael A. Cacciatore is a doctoral student in the Department of Life Sciences Communication at the University of Wisconsin-Madison. His research interests include risk communication and attitudes toward new technologies, including nanotechnology and alternative energies.

Dietram A. Scheufele is the John E. Ross Chaired Professor of Science Communication and Director of Graduate Studies in the Department of Life Sciences Communication at the University of Wisconsin-Madison. His work focuses on public opinion, political communication, and public attitudes toward new technologies, including nanotechnology, stem cell research, and GMOs.

Bret R. Shaw is an assistant professor in the Department of Life Sciences Communication at the University of Wisconsin-Madison. He has done extensive work in the areas of social marketing and communication related to environmental campaigns, as well as technology-based education and support for people facing chronic health problems.

Elizabeth A. Corley is the Lincoln Professor of Public Policy, Ethics & Emerging Technologies and an Associate Professor in the School of Public Affairs at Arizona State University. Her research focuses on technology policy and environmental policy. She serves as a Co-Principal Investigator for the NSF-funded Center for Nanotechnology in Society at Arizona State University (CNS-ASU).

References

AAPOR (2009) Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys, 6th edn. Lenexa, KS: AAPOR (American Association for Public Opinion Research).

Ajzen

(2001) Nature and operation of attitudes. Annual Review of Psychology 52: 27–58.

Ajzen

Fishbein

(2008) Scaling and testing multiplicative combinations in the expectancy-value model of attitudes. Journal of Applied Social Psychology 38(9): 2222–2247.

Babbie

(2007) The Practice of Social Research, 11th edn. Belmont, CA: Thomson Wadsworth.

Brehm

(1993) The Phantom Respondents: Opinion Surveys and Political Representation. Ann Arbor, MI: University of Michigan Press.

Brigham Schmuhl

Cacciatore

Binder

Shaw

Scheufele

(2010) Technological innovation in America’s backyard: The intersection of media use, political partisanship, and public opinion of biofuels. Paper presented at the annual meeting of the American Association for Public Opinion Research, May Chicago, IL.

Brossard

Nisbet

(2007) Deference to scientific authority among a low-information public: Understanding U.S. opinion on agricultural biotechnology. International Journal of Public Opinion Research 19(1): 24–52.

Brossard

Shanahan

(2007) Perspectives on communication about agricultural biotechnology. In: Brossard

Shanahan

Nesbitt

(eds) The Public, the Media, and Agricultural Biotechnology: An International Casebook. Cambridge, MA: CABI, 3–20.

Brossard

Scheufele

Kim

Lewenstein

(2009) Religiosity as a perceptual filter: Examining processes of opinion formation about nanotechnology. Public Understanding of Science 18(5): 546–558.

10.

Cacciatore

Scheufele

Corley

(in press) From enabling technology to applications: The evolution of risk perceptions about nanotechnology. Public Understanding of Science. doi: 10.1177/0963662509347815

11.

Carmines

Zeller

(1979) Reliability and Validity Assessment. Thousand Oaks, CA: Sage.

12.

Chaffee

(1991) Explication. Newbury Park, CA: Sage.

13.

Cobb

Macoubrie

(2004) Public perceptions about nanotechnology: Risks, benefits and trust. Journal of Nanoparticle Research 6(4): 395–405.

14.

Corley

Scheufele

(2010) Outreach going wrong? The Scientist 24(1): 22.

15.

Crne-Hladnik

Peklaj

Kosmelj

Hladnik

Javornik

(2009) Assessment of Slovene secondary school students’ attitudes to biotechnology in terms of usefulness, moral acceptability and risk perception. Public Understanding of Science 18(6): 747–758.

16.

Cronbach

(1951) Coefficient alpha and the internal structure of tests. Psychometrika 16(3): 297–334

17.

Eagly

Chaiken

(1993) The Psychology of Attitudes. Fort Worth, TX: Harcourt Brace.

18.

Fishkin

(1991) Democracy and Deliberation. New Haven: Yale University Press.

19.

Flynn

Slovic

Mertz

(1994) Gender, race, and perception of environmental-health risks. Risk Analysis 14(6): 1101–1108.

20.

Friedman

Dunwoody

Rogers

(1999) Communicating Uncertainty: Media Coverage of New and Controversial Science. Mahwah, NJ: Erlbaum.

21.

Gilovich

(1991) How We Know What Isn’t So: The Fallibility of Human Reason in Everyday Life. New York: Free Press.

22.

Irwin

(2001) Constructing the scientific citizen: Science and democracy in the biosciences. Public Understanding of Science 10(1): 1–18.

23.

Kahan

Braman

Slovic

Gastil

Cohen

(2009) Cultural cognition of the risks and benefits of nanotechnology. Nature Nanotechnology 4(2): 87–90.

24.

Kleinman

Delborne

Anderson

(in press) Engaging citizens: The high cost of citizen participation in high technology. Public Understanding of Science. doi:10.1177/0963662509347137

25.

Klop

Severiens

(2007) An exploration of attitudes towards modern biotechnology: A study among Dutch secondary school students. International Journal of Science Education 29(5): 663–679.

26.

Lee

Scheufele

Lewenstein

(2005) Public attitudes toward emerging technologies: Examining the interactive effects of cognitions and affect on public attitudes toward nanotechnology. Science Communication 27(2): 240–267.

27.

McLeod

Pan

(2005) Concept explication and theory construction. In: Dunwoody

Becker

McLeod

Kosicki

(eds) The Evolution of Key Mass Communication Concepts: Honoring Jack M. McLeod. Cresskill, NJ: Hampton Press, 13–76.

28.

Miller

(2004) Public understanding of, and attitudes toward, scientific research: What we know and what we need to know. Public Understanding of Science 13(3): 273–294.

29.

Nisbet

Mooney

(2007) Science and society: Framing science. Science 316(5821): 56.

30.

Pardo

Calvo

(2002) Attitudes toward science among the European public: A methodological analysis. Public Understanding of Science 11(2): 155–195.

31.

Powell

Kleinman

(2008) Building citizen capacities for participation in nanotechnology decision-making: The democratic virtues of the consensus conference model. Public Understanding of Science 17(3): 329–348.

32.

Rowe

Frewer

(2005) A typology of public engagement mechanisms. Science, Technology and Human Values 30(2): 251–290.

33.

Scheufele

(2006) Messages and heuristics: How audiences form attitudes about emerging technologies. In: Turney

(ed.) Engaging Science: Thoughts, Deeds, Analysis and Action. London: The Wellcome Trust, 20–25.

34.

Scheufele

Tewksbury

(2007) Framing, agenda setting, and priming: The evolution of three media effects models. Journal of Communication 57(1): 9–20.

35.

Schuman

Presser

(1981) Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. New York: Academic Press.

36.

Slovic

(1987) Perception of risk. Science 236: 280.

37.

Sudman

Bradburn

Schwarz

(1995) Thinking about Answers: The Application of Cognitive Processes to Survey Methodology. San Francisco: Jossey-Bass.

38.

Tankard

(1976) Reporting and scientific method. In: McCombs

Shaw

Grey

(eds) Handbook of Reporting Methods. Boston: Houghton Mifflin, 42–77.

39.

Tourangeau

Rips

Rasinski

(2000) The Psychology of Survey Response. New York: Cambridge University Press.

40.

Tversky

Kahneman

(1973) Availability: A heuristic for judging frequency and probability. Cognitive Psychology 5(2): 207–232.

41.

von Hoffman

(1980) Public opinion polls: Newspapers making their own news. Public Opinion Quarterly 44(3): 572–573.

42.

Wegener

Kelly

(2008) Social psychological dimensions of bioenergy development and public acceptance. Bioenergy Research 1(2): 107–117.

43.

Wooldridge

(2003) Introductory Econometrics. Mason, OH: South-Western College Publishing.

44.

Zaller

Feldman

(1992) A simple theory of the survey response: Answering questions versus revealing preferences. American Journal of Political Science 36(3): 579–616.