The State of Survey Methodology

Interviews

Researchers have long recognized that respondents to interviews, whether in-person or by telephone, give answers that are influenced by the interviewer. They are more likely to give socially desirable answers (e.g., “Yes, I voted in the last election”) and less likely to give socially undesirable answers (e.g., “No, I do not regularly drive my car after having several alcoholic drinks;” Belli et al. 1999; Davis et al. 2010). People also tend to be agreeable, or acquiesce, in interview situations (Schuman and Presser 1981).

Interviewer effects notwithstanding, over the past two decades, we have seen a steady decline in response rates for telephone interviews (de Leeuw 2008; de Leeuw and de Heer 2002). Moreover, with an increasing number of gated communities and a decreasing willingness to respond to in-person interviewers—in combination with the excessive costs—there are problems for researchers employing in-person methodologies (Groves and Couper 1998). With the telephone, the heavy use of answering machines as well as caller identification has presented formidable barriers to achieving contact with households and individuals (Groves and Lyberg 1988).

However, today an even more profound challenge has been the decrease in coverage for the traditional random digit dialing (RDD) samples, which only include landline telephone numbers (Blumberg et al. 2011). The decline in landline connections, particularly for young people, households with children, and people with changing social relationships such as separation and divorce, make coverage a greater problem (Keeter et al. 2007). Although mobile phone numbers are often added to landline sample frames to increase coverage, this dual frame approach introduces new challenges. For instance, dual sample frames are more prone to the sources of error in comparison to RDD-only frames. Moreover, in the United States, there are instances where incoming calls are counted against the respondent’s prepaid minutes or data plans (Brick et al. 2011a). In addition, mobile phones produce geographic location challenges because individuals commonly keep the same mobile phone number after they relocate, rendering the area code a poor indicator of residential area.

The Internet

With the decline in interview response rates, there was much optimism that the Internet would become the survey replacement for the telephone. For many reasons, that has not happened. For instance, in the United States, household Internet coverage is only about 75% (National Telecommunications and Information Administration [NTIA] 2011). Moreover, in those households with connections, some individuals lack the skills to use it, are uncomfortable with it, or use it infrequently (Stern et al. 2009).

Achieving a high response rate by e-mail contact only is also problematic. Specialized populations notwithstanding (e.g., attendees at conferences or recent doctorate graduates), very few e-mail-only surveys have been found effective at achieving high response rates. In addition, there is no acceptable general population sample frame for e-mail addresses, and professional norms prevent sending survey requests to individuals with whom there is no prior relationship (e.g., client, student, and association member). One can sometimes use mail (or even telephone) contact to ask people to respond over the web, but doing so adds to the cost and also has to be effective in convincing people to go from that mode of contact to a computer where they enter a URL and password (Anderson and Tancreto 2011).

An even bigger challenge is now emerging. There is evidence that significant numbers of people find they do not need a desktop or even a laptop computer (Miller et al. 2012). Instead, a smartphone (or other mobile device) is considered adequate for all of their electronic communication needs. For instance, Pew Internet and American Life Research Center (Miller et al. 2012:7) recently reported that nearly half of U.S. adults own a smart phone and 17% of them do the majority of their web searching on this device instead of on a computer. Asking questions on a smartphone requires utilizing formats that require very little space (Callegaro 2010). Thus, some traditional question formats (e.g., matrices) may be unusable when a significant portion of one’s sample relies only on such devices (Buskirk and Andrus 2012).

Mail

Mail surveys have continued to do reasonably well both as a singular mode or when used in conjunction with another form of data collection despite the challenges discussed previously (Dillman et al. 2009). However, researchers have expressed several concerns.

First, it is now considered a higher cost mode of self-administered data collection in comparison to the web (Dillman et al. 2009) due to the expenditures associated with postage, processing, and printing (Groves et al. 2009).

A second reason for the reluctance to use mail surveys is that it is seen as less flexible given it is difficult to use branching or probe questions as effectively as in interviews or on the web, thus limiting the complexity and, in some cases, depth of a survey instrument (Biemer et al. 2004; Bourque and Fielder 2003; Lyberg and Kasprzyk 1991). Furthermore, it takes longer to obtain responses, and evidence exists that responses are less complete compared to other modes (Dillman et al. 2009). In addition, item nonresponse is often slightly higher even in well-designed mail surveys when compared to web and telephone (Dillman 2012).

Finally, despite the availability of the U.S. Postal Service’s Delivery Sequence File, which now makes attainable nearly all addresses of residential households where mail is delivered, there are still coverage and nonresponse issues. For example, similar to landline telephones, mail surveys have been shown not to obtain adequate response rates for: (1) younger people who do not use the mail system readily; (2) individuals in transitions such as when relocating; (3) those living at “drop points” (e.g., incomplete addresses for multiple family dwellings); (4) people living in new construction; and (5) illiterate and English as a Second Language (ESL) populations (Fowler 2009; Messer and Dillman 2011). Additionally, postal surveys suffer from being seen as junk mail when not adequately designed (Dillman et al. 2009).

Different Modes for Different Survey Situations

Unlike in the past, today it is clear that no single data collection mode is adequate for a majority of survey situations. That is, different populations, sponsorships, client needs, and circumstances require different modes and implementation procedures to reduce the major sources of survey error and keep survey costs in check.

Interview modes: Despite low response rates and coverage problems, we are going to continue to do single-mode interview surveys in some situations. Researchers will still use telephone-only surveys with mobile phone augmentation of landline lists, particularly for political polling due to the need for repeated timely results. In addition, they likely will still be used in surveys of the general population where the likelihood that a respondent will be eligible to respond is low, such as in the case of National Immunization Survey (NIS), where the eligible respondent must have a child between the ages of 19 and 35 months and the eligibility rate is less than 3%. With millions of households screened in any given year, using mail in this case would be time inefficient. Thus, in some instances, there is simply no suitable alternative yet that gives the needed speed of reporting, cost efficiencies, and coverage (AAPOR Cell Phone Task Force 2010).

In election studies, we typically have exit poll data that allow statistical adjustment. However, in many cases, statistics produced through telephone-only surveys simply will not be convincing because of our lack of knowledge regarding the correlates for most data, which effectively prohibits post-stratification weighting. As a result, we will also continue to use in-person interviews as is done for the Current Population Survey and American Community Survey that collect critical societal information and thus cannot tolerate the coverage and nonresponse errors found with other modes.

Self-administered modes: In addition to the current challenges associated with interviewing discussed previously, there are two more significant reasons why self-administered modes are growing in popularity for general population surveys with mail perhaps being the predominant self-administered mode used for the near future. First, there is the availability of adequate mail sample frames (e.g., U.S. Postal Service’s Delivery Sequence File and household registers in Europe). Second, there is still uneven diffusion of high-speed Internet access combined with low Internet proficiency among certain segments of the population and reluctance among some individuals to provide personal information online (Stern et al. 2009; Stern et al. 2012). What is more, household survey tests of mail-only survey methods show that it may obtain response rates of 40–70% in nonfederal government surveys (Messer and Dillman 2011; Smyth et al. 2010). Studies have also shown that mail can be effective for screening a household’s characteristics and following up with a detailed set of questions, which is comparable with the two-stage telephone request (Brick et al. 2011b; Mathiowetz et al. 2010).

Are Shared Online Panels a Viable Alternative to One-time Samples?

One of the proposed solutions to the current challenges associated with individual mode surveys is to recruit panels of respondents who agree to complete a series of surveys over the Internet. The idea finds its basis in the work of Saris and de Pijper (1986) who, in 1984, began development of the first telepanel, a representative sample of Dutch citizens using computer-assisted surveying (Saris 1998). With the rise of the Internet over the past two and a half decades, two sampling models have dominated the developments in this area.

In the nonprobability model, individuals are asked to join the study, most often when they are visiting a web site. The sponsoring organizations tend to offer per survey incentives to increase the attractiveness of panel membership. The approach has two issues with which it must contend. First, the samples are not representative with an overrepresentation of, for example, Internet users, students, and homemakers seeking to earn money. Second, when payments are the driving force for staying in the panel, the motivation for providing optimal answers must be questioned.

The second model relies on a probabilistic approach using RDD or ABS samples. Several research firms (e.g., CentERdata, InterSurvey, Gallup, and Knowledge Networks) have been using probabilistic methods in their Internet panels. Many of the current probabilistic online panel surveys follow the model of the CentERdata panels from the Netherlands, using a combination of traditional probability sampling and stratified refreshment samples to ensure representativeness (see Scherpenzeel and Das 2010 for a full description). For instance, Knowledge Network recruits their members via an ABS sampling approach and then provides Internet access and a laptop computer with the Windows operating system for the sampled individuals who do not have a computer or Internet access in their homes. Gallup recruits through RDD and then assigns respondents to web or mail based on user characteristics. Although these approaches potentially aim to solve the issue of web coverage, they are costly and dependent on web proficiency and preference. In addition, as far as we know, there is no experimental evidence that providing Internet access leads to more representative and better data among probability-based panels.

Both models present unique challenges. For instance, both probability- and nonprobability-based panels are subject to conditioning effects; however, we need further investigation whether this is more the case with Internet panels. Although research has shown that the accuracy of the probability-based online panels exceeds that of nonprobability online panels, they still are not able to obtain the level of precision of traditional methods such as RDD surveys (Chang and Krosnick 2009; Yeager et al. 2011).

Assessing the magnitude of the differences between traditional modes of survey data collection, including face-to-face and RDD surveys and online panels, requires consideration of two interdependent factors—nonrandom selection and differences between Internet users and the general population. As Chang and Krosnick (2009) and others have noted, when considering an opt-in and even some probability-based online panels as a survey data source, it must be understood that not every member of the population has an equal chance for inclusion in the study. Recent attempts have been made to use various post-data adjustments, such as weighting, and sampling strategies to offset this nonrandom propensity for inclusion as well as examining various ways to assess the differences between online panels and other modes (e.g., absolute average error, largest absolute error, number of significant differences, etc.). ¹ A confounding factor tends to be that Internet users still differ on key sociodemographic variables that lead to differences in their substantive responses; this is true on both behavioral and attitudinal measures (Duffy et al. 2005). For example, proficient Internet users tend to be more highly educated and wealthier, personal characteristics that covary with measures of political affiliation, views toward community change, and civic/political engagement (Stern and Dillman 2006). Although research has shown that some post-stratification strategies—including combined demographic and propensity weighting and alternative selective sampling approaches—can potentially offset some of the differences, this is not universally the case (Duffy et al. 2005; Terhanian and Bremer 2012).

Their representativeness notwithstanding, nonprobability surveys are inexpensive, and fast, which makes them appealing to marketing firms and researchers with a specific, targeted sample, but they are not based on well-established statistical or methodological theory (Dennis et al. 2011). We can envision strong opposition among many governmental and nongovernmental organizations to include their survey in someone else’s panel, thus making this approach only feasible for select survey situations. Still, we are seeing an increasing number of these panels used in the marketing and nonprofit survey sector to supplement and augment existing modes when responses have not reached expectations; thus, more research in this area is certainly forthcoming (Terhanian and Bremer 2012).

When One Survey Mode Is Not Enough

Although mixing modes is not new (see de Leeuw 2005; Lyberg and Kasprzyk 1991; Payne 1964), it is only in the last decade that the strategy has become well established. This point is evidenced by the increasing number of empirically based studies on the effects of mixing modes as well as the more pervasive use of two or three modes by government and large survey organizations (Dillman et al. 2009). Recent research has helped bring to light effective ways and reasons for survey researchers to implement self-administered mixed-mode surveys.

One question that has arisen is whether different modes should be offered concurrently or sequentially. Studies have found that offering a concurrent choice in mode does little to improve response (Dillman et al. 2009; Gentry and Good 2008). In fact, Fulton and Medway (2012) found in their meta-analysis, which included 19 experimental comparisons, that offering a concurrent web option in mail surveys significantly reduced response rates. Similarly, Millar and Dillman (2011) found among college students that this concurrent approach did not improve response; but it did not significantly diminish response rates either. However, a sequential approach of web then mail did improved response rates.

Nonetheless, mixed-mode surveys can improve the representativeness of a sample. For instance, Messer and Dillman (2011) show that web-only approaches exclude important and often underrepresented segments of the population. However, they also found that web surveys supplemented by a subsequent mail questionnaire produced results similar to those of mail-only surveys, but at an increased monetary cost to the researcher.

There are some additional concerns researchers must consider in using a mixed-mode approach. For instance, mixing modes can lead to differences in substantive responses due to one mode favoring a certain sociodemographic over another mode (e.g., RDD attracts older homeowners whereas web-only surveys are more apt to obtain responses from younger people). Another source of differences stems from mode of presentation, with visual self-administered surveys often obtaining different answers than aurally administered questionnaires (de Leeuw and Hox 2011; Dillman and Christian 2005).

Additionally, with a web component, surveyors must attempt to understand the way different segments of society use the web. Whereas 96% of the U.S. population has access to and can use a landline telephone (NTIA 2011), web proficiency is much more variable. When using a web survey component, one must design for ease and the ability to replicate the design in another mode. We simply do not know what effects new possible question structures will have on measurement error in different segments of the population based on web proficiency. Finally, today we must consider how the visual design in a web component translates to mobile devices and the consequences for measurement error, response burden, and the interface of smartphone and web (Buskirk and Andrus 2012).

Accepting a Culture of Self-administration

The way people communicate is changing and surveyors need to adapt to that. As we have discussed previously, a substantial amount of research has documented the increasing number of households without a landline telephone continues to rise, rendering traditional RDD frames incomplete (e.g., Blumberg and Luke 2012). In addition to this technological change, there has been a cultural shift away from synchronous communication and reliance on human intermediaries. This move is most often discussed in sociologically based theories such as “networked individualism” or “glocalization,” where distance and localized personal communication is increasingly less needed for meaningful matters (e.g., Hampton and Wellman 2003; Stern and Dillman 2006; Wellman 2001). We now use automated systems to do much of our banking and bill paying as well as to get answers from local or governmental agencies, make reservations or make appointments. Perhaps this explains why we are less tolerant of interviews—they are increasingly nonnormative. There is some empirical evidence of this form in survey research. Smyth et al. (2010) included a question in a general population mixed-mode survey asking about the mode in which respondents preferred to complete a survey (telephone, web, or mail). Their results show a full 98.8% of respondents chose a self-administered mode.

One question to come out of this social change is whether interactive voice recognition (IVR) has a future (see Steiger and Conroy 2008). With IVR, people are given an automated aural stimulus and respond either aurally or using a keypad on their phones. Although the systems can be cumbersome, an entire generation does not recall when a live operator was the first level of connection. As a result, society is perhaps primed for this mode of delivery and response. In addition, in the developing world, many more people own mobile phones than have access to the Internet or a landline connection (Lerer et al. 2010). Technologically, the transition from aural to visual may be less pronounced than ever, thanks to smartphones. People across many generations are now avidly text messaging, meaning keypad entry is less problematic than it once was. Therefore, as smartphones increasingly become the normative tool of communication, IVR may be well situated. Recent research has shown that keypad entry as part of an IVR approach is possible when there is little mixing between the need to vocalize responses and manually enter them (Kuun 2009; Lerer et al. 2010). Although the future of IVR remains unclear, there are compelling reasons to continue to explore its utility.

Understanding How Visual Layout Affects Answers

Concomitant with the trend toward self-administered surveys has been the need to understand the way respondents are affected by the visual design of questions and the questionnaire. Twenty years ago, survey methodologists focused more on the wording of questions than overall questionnaire design and how people responded to it. During the 1990s, cognitive interviewing became the primary means for evaluating both questions and questionnaires (Lessler and Forsyth 1996; Tourangeau et al. 2000). Survey methodologist began to uncover how small manipulations in the visual design of questions influenced respondents’ answers (Jenkins and Dillman 1997). Over the past decade, research has shown that the visual design of a survey and its various components are as influential to responses as any documented interviewer or wording effect.

The body of experimental work in this area has shown that visual layout may dramatically affect the reporting of numbers and selection of response categories; font size, spacing, location of information, and symbols affect answers. For example, Christian and Dillman (2004), Smyth et al. (2006), and Stern et al. (2007) have all shown how arrows and answer boxes operate independently and in concert to communicate information to respondents on the navigational path required for completing self-administered questionnaires. In related work, Tourangeau et al. (2004, 2006) have shown that there are interpretive heuristics based on spatial or visual features that influence respondents.

Putting this research in perspective brings to light two important issues for the future. First, in a time when mixed-mode surveys are seen as a necessity as opposed to a novelty, we must be cognizant of differences in the visual design across different modes and platforms. Survey researchers simply cannot design a web and paper version of the survey; instead, there must be an attempt at common design. Second, web survey designers must now evaluate how their questionnaires appear on smartphones and tablets. Not doing so is no longer option.