Item Location,the Interviewer–Respondent Interaction,and Responses to Battery Questions in Telephone Surveys

Abstract

Survey researchers often ask a series of attitudinal questions with a common question stem and response options, known as battery questions. Interviewers have substantial latitude in deciding how to administer these items, including whether to reread the common question stem on items after the first one or to probe respondents’ answers. Despite the ubiquity of use of these items, there is virtually no research on whether respondent and interviewer behaviors on battery questions differ over items in a battery or whether interview behaviors are associated with answers to these questions. This article uses a nationally representative telephone survey with audio-recorded interviews and randomized placement of items within four different batteries to examine interviewer and respondent behaviors and respondent answers in battery questions. Using cross-classified random-effects models, the authors find strong evidence that there is more interviewer–respondent interaction on items asked earlier in the battery. In addition, interviewer and respondent behaviors are associated with both substantive and nonsubstantive answers provided to battery items, especially if the interviewer decided to reread or probe with the response options. These results suggest that survey designers should follow recommendations to randomize battery items and consider the importance of standardization of question administration when designing battery questions.

Keywords

survey research telephone surveys interviewer–respondent interaction battery items

1. Introduction

Battery items are commonly used in interviewer-administered surveys, such as the General Social Survey (Smith et al. 2017) and the American National Election Studies (American National Election Studies 2017), and are fundamental to the creation of most scales, such as the Conflict Tactics Scales (Straus 1979). In a battery, a common question stem is presented before a list of items with a shared set of response options, often mentioning that additional items are forthcoming (Dillman, Smyth, and Christian 2014; Saris and Gallhofer 2007; Siminski 2008). The order of items within the battery may be randomized or fixed. The respondent may see this common stem and response categories on a show card in face-to-face surveys, but such a visual aid is not possible in telephone surveys.

Standardized interviewing guidelines instruct interviewers to deliver each survey question as written in an attempt to avoid off-script conversation and, in turn, to minimize interviewer influences on responses. Conversational interviewing, on the other hand, allows interviewers flexibility to assist respondents, with the goal of increasing understanding (e.g., Schober and Conrad 1997). Although the foundational texts on standardized interviewing (e.g., Fowler and Mangione 1990) describe how to administer individual items, they are silent about the special situation of batteries. In survey practice, interviewers are required to read the battery’s question stem and response options with the initial item or two, and then are allowed discretion to repeat the stem and/or response options as they think it is needed for later items (Dykema et al. forthcoming; Fowler 1995). Interviewers may also probe in a standardized way by repeating the question or the response categories rather than stating the question stem or response categories initially (Fowler and Mangione 1990). As such, the survey industry administers battery items as a compromise between strictly standardized interviewing (Fowler and Mangione 1990) and conversational interviewing (Beatty 1995; Schaeffer 1991; Suchman and Jordan 1990). In particular, battery items deviate from the prototypical standardized interviewing procedures (Cannell, Miller, and Oksenberg 1981; Fowler and Mangione 1990) in that interviewers are not clearly instructed in the questionnaire itself about when they should reread the introduction or the response options to the respondent (Dykema et al. forthcoming) and from prototypical conversational interviewing in that their follow-up options are scripted in a standardized interviewing fashion. This deviation is important because survey researchers may assume that the question stem for a battery is read as scripted only on the first item in a battery in a fully standardized manner. Yet survey practitioners train interviewers to administer batteries differently from how the questionnaire suggests, with optional rereading of the question stem and response options throughout the battery, similar to conversational interviewing. That is, the commonly accepted and ostensibly standardized procedure for administering battery items in today’s surveys is not clearly standardized or conversational, but somewhere in between.

Battery items provide an interesting case study for a type of item that requires being attuned to conversational cues (e.g., Suchman and Jordan 1990). For later items in the battery, repeating the question stem and response options on each item may be perceived as unnecessarily repetitive. In this case, interviewers may be perceived as breaking a fundamental rule of cooperative communication—that one should not give more information than is required (Grice 1975)—and the application of strict standardization by reading the full question stem on each item may undermine measurement quality. However, the conversational interviewing literature largely has been silent about batteries, primarily focusing instead on factual and autobiographical reports (Belli, Shay, and Stafford 2001; Conrad and Schober 2000; West, Conrad, Kreuter, and Mittereder 2018).

In addition, battery items put heavy demands on respondents’ working memory, especially in telephone surveys, where information is presented orally. In telephone interviews, respondents have to remember information from the introductory question stem and response options while processing subsequent items (Tourangeau, Rips, and Rasinski 2000). Respondents mentally insert each new item into the recalled introductory stem to have a complete question and then generate a response mapped to a recalled response option. There are no visual aids in a telephone survey, unlike self-administered surveys or show cards in face-to-face surveys. As a result, respondents may request help in remembering the question stem or the response categories. Thus, in telephone surveys especially, giving interviewers the discretion to reread the question stem and response options as needed allows them to react to any confusion or forgetfulness stemming from this high memory demand and burdensome question format.

Thus, survey practitioners face a quandary: how to design and administer battery items in telephone surveys to minimize unnecessary repetition, help respondents remember necessary information, and reduce the potential for interviewer effects on responses. To date, there is little research on how interviewers and respondents administer and answer these questions and how this administration may be associated with respondents’ answers. In this article, we empirically examine how interviewers and respondents administer and answer four different sets of battery items in a nationally representative telephone survey.

2. Background

2.1. Interview Behaviors and Item Location

It is unknown when interviewers use the discretion they are given in administering batteries or when respondents request help in answering these items. Thus, this is our first research question:

Research Question 1: Do interviewer and respondent behaviors in battery questions differ by the location of items in the battery?

Standardized survey interviewers train respondents on how to answer survey questions and otherwise be a “good” respondent (Fowler and Mangione 1990). In battery questions, interviewers may need to read or repeat the question stem and/or response options on items early in the battery to help train respondents for later items. After a few repetitions, reading this information may no longer be needed as information that has been repeated is easier to recall (Peterson 1966). If interviewers do not read the optional information, respondents may have to ask for assistance (Dijkstra and Ongena 2006). In general, respondents need to remember and integrate multiple pieces of information, and they will have less practice with this on early items. On the other hand, interviewer rereadings and respondent requests for clarification may occur more frequently on later items as respondents get farther from having heard the question stem and response options (i.e., memory decay) and are more fatigued. Thus, (Hypothesis 1a) if respondent learning/training is occurring, we expect that there will be more conversational turns and higher rates of interviewers reading the introductory question stem and/or response options, probing by rereading the question stem and response categories, and respondents asking for the question to be repeated on items presented earlier in a battery than on items presented later in a battery. We also expect that the decline will be steeper for earlier items than for later items. Conversely, (Hypothesis 1b) if forgetting is occurring, we expect more of these behaviors on later items.

2.2. Item Location and Respondent Answers

Ideally, the location of the item in a battery is independent of the answers that are given to the item. Thus, this is our second research question:

Research Question 2: Do answers provided by respondents in battery questions differ across the location of items in the battery?

Generally, respondents are thought to take cognitive shortcuts (i.e., satisfice) when they experience increased burden, working-memory challenges, and/or a series of items that are very similar to one another, all of which commonly occur in batteries (Alwin and Beattie 2016; Krosnick 1991; Krosnick and Alwin 1987), although empirically assessing satisficing is challenging (Alwin and Beattie 2016). There may be higher rates of “don’t know” (DK) or refusal (REF) responses on early items in the battery because respondents are doing extra cognitive work to learn the response options. Alternatively, because of fatigue, respondents may be more likely to acquiesce (i.e., say yes or agree with a statement regardless of the content of the item; Schuman and Presser 1981) or otherwise answer in ways that shortcut the response process (such as providing DK/REF answers) to reduce their burden on later items. Empirically, previous work has shown inconsistent or null findings on item content and location (Krosnick and Presser 2010).

Thus, (Hypothesis 2a) if respondent learning is occurring, we expect higher rates of DK/REF on early items in a battery but no clear pattern for item endorsement. (Hypothesis 2b) If satisficing and respondent fatigue is occurring, we expect higher rates of DK/REF and answers indicating acquiescence for later items in a battery. Furthermore, questionnaire design texts (e.g., Sudman, Bradburn, and Schwarz 1996) often suggest randomizing the order of items to avoid systematic context effects; (Hypothesis 2c) if this recommendation holds, we do not expect to see systematic differences in the levels of DK/REF or endorsement of particular items depending on their location.

2.3. Interview Behaviors and Respondent Answers in Battery Items

Understanding whether interviewer and respondent behaviors are associated with respondents’ answers is fundamentally important for understanding the implications of the quasistandardized battery question administration strategy that the survey industry has adopted. As such, this is our third research question:

Research Question 3: Do interviewer and respondent behaviors predict respondent answers to items in the battery?

Battery items’ unique features of high respondent working-memory demands and interviewer discretion in administering them interact to create a potentially problematic measurement situation. Battery items consistently have been shown to have lower reliability and validity than other items in a questionnaire (Alwin 2007, chap. 8; Alwin 2010; Alwin and Beattie 2016; Dykema, Schaeffer, and Garbarski 2016; Saris and Gallhofer 2007). For instance, Alwin (2007) compares questions in battery items that share a question stem and response options, questions in a topically related series that do not share question stems and response options, and questions that are not topically related to the surrounding items, and finds that items in a battery have lower reliability than those that are in a topically related series, which in turn have lower reliability than stand-alone questions. We know surprisingly little about why this is the case.

Battery questions are unique in that the variation in initial question reading is likely to be more constrained (e.g., reading the scripted question stem or the response options) than in other types of questions, and they may be used to head off potential respondent problems or confusion about the question. For instance, Dykema, Schaeffer, Garbarski, et al. (2016) found that respondents are less likely to exhibit problematic response behaviors when interviewers read optional parenthetical information in question stems versus not reading that information, a decision process similar to that for battery items. In addition, prior research has shown that question-reading behaviors can have downstream effects on respondent behaviors and answers and on interviewer behaviors, and they may influence rapport between the respondent and interviewer (e.g., Fowler and Mangione 1990; Garbarski, Schaeffer, and Dykema 2016; Holbrook et al. 2015, 2016). Thus, (Hypothesis 3a) we hypothesize that including the question stem and response options in the item administration reminds the respondent about the response task, decreasing the rate of final DK/REF responses.

To remedy respondent problems, interviewers often turn to probing behaviors. For instance, behaviors reflecting respondent problems with comprehension and mapping have been found to positively predict interviewer probing (Holbrook et al. 2016), and probing is associated with higher interviewer variance (Fowler and Mangione 1990; Mangione, Fowler, and Louis 1992) and response inaccuracies (e.g., Belli and Lepkowski 1996). Standard practice when interviewers receive a DK or, sometimes, REF response is to probe for a substantive answer, thus adding additional conversational turns because of the DK response itself, not causing it. As a result, (Hypothesis 3b) we expect that the number of conversational turns and interviewer probing will be associated with higher rates of DK/REF answers. Finally, (Hypothesis 3c) we expect respondent requests for help to be associated with lower rates of final DK/REF answers because respondents are likely to ask for help if they want to provide a substantive answer but are having difficulty doing so.

For substantive answers, it is harder to anticipate how the interviewer–respondent interaction will affect responses, but we recognize that it is critically important to understand whether these interviewer and respondent decisions are associated with substantive responses. Ideally, the initial reading of the question stem or the response options is not associated with substantive answers. To the extent that it is, reading the response options, especially when they are not scripted, may either make it easier to provide a less thoughtful response (reading the response options cues the respondents that a quick response that maps into one of those categories is desired) or may make it easier to provide a more thoughtful response (the time spent reading the response options can be spent thinking about the content of the question). For probing and respondent assistance, people who feel positively (or negatively) toward a particular attitude object may be more inclined to ask for assistance or answer in a way that requires probing. Alternatively, off-script interaction between interviewers and respondents may contribute to the attitude formation itself, although it is difficult to anticipate the direction in which this occurs. Furthermore, the interaction with the interviewer may motivate the respondents to do the hard work of answering survey questions and reduce acquiescence. For these reasons, we (Hypothesis 3d) evaluate whether measures of interviewer–respondent interaction are associated with survey responses but do not have concrete directional hypotheses.

3. Data

The data come from the U.S.–Japan Newspaper Opinion Poll, a dual-frame random-digit-dial telephone survey of U.S. adults in landline and cell phone households conducted during November 2013 (American Association for Public Opinion Research Response Rate 1 = 7.4%, N = 1,005). Gallup conducted the 13-minute-long general public opinion poll, which included questions about confidence in American institutions, attitudes toward various Asian countries, and demographics.

We examine four battery questions ranging in length from 6 to 14 items (Figure 1). Each respondent received the items in each battery in a different randomized order. Because of this important design feature, the item location within each battery can be disentangled from the content of the items themselves. The first battery (B1), the first question of the survey, had 14 items asking about confidence with institutions in American society. The next two batteries, asking about countries that will become a military threat (B2, 13 items) and issues about China (B3, 8 items), were positioned in the middle of the survey. The fourth battery, asking about North Korea (B4, 6 items), was administered at the end of the substantive questions before the demographic questions. All of the items within each battery have the same set of dichotomous substantive response options and standard (unread) DK and REF response options, although the substantive response options vary across the batteries (B1: do, do not; B2: yes, no; B3: concerned, not concerned; B4: yes, no).

Figure 1.

The wording in four survey battery questions.

3.1. Item Location

The location of the items within the battery is a key independent variable. For each respondent in each battery, the first item that was asked received a 1, the second item asked received a 2, and so forth, resulting in a variable that captures the item’s randomly assigned cardinal position in the battery (Alwin 2007).

3.2. Interviewer and Respondent Behaviors

We examine six interviewer and respondent behaviors using behavior coding, a systematic coding of the interviewer and respondent behaviors in a survey interview (Fowler and Cannell 1996; Ongena and Dijkstra 2006). Due to the cost of transcribing and coding interviews, we selected a random subset of 467 audio recordings of the 1,005 interviews. Ten of these interviews were partial interview recordings, and 19 were conducted in Spanish, leaving 438 English-language interviews completed by 31 interviewers who had conducted at least 10 interviews, which facilitated model estimation (van Breukelen and Moerbeek 2013; Vassallo, Durrant, and Smith 2017). Each recorded interview was transcribed, and the transcripts were behavior coded by a team of trained undergraduates using the Sequence Viewer program (Dijkstra 1999).

We examine behaviors for each item answered across the four batteries. Although 438 interviews were processed, partial recordings, backups (respondents answering questions that had been asked previously), or interviewers not asking all of the battery items yielded slightly smaller analytic sample sizes in B1 (433 respondents), B2 (431 respondents), and B4 (432 respondents).

We code each conversational turn within a question-asking sequence on the actor (interviewer or respondent), his or her initial action (e.g., probing), and an assessment of the initial action (e.g., probing using response options). This initial behavior coding scheme did not adequately capture whether interviewers chose to read the question stem or response options in their asking of each item after the first (where it was required). Thus, a second set of independent coders evaluated whether the initial reading of each item within each battery included the question stem, the response options, or no additional information. Table 1 shows a behavior coding example from the first battery.

Table 1.

Behavior Coding Example

Conversational Turns	Behavior Codes
Conversational Turns	Actor	Initial Action	Assessment
Interviewer (I): Regarding North Korea, which issues should the U.S. and Japanese governments, working in cooperation, give priority to resolving? How about normalizing diplomatic relations between the U.S. and North Korea?	Interviewer	Question asking	Read exactly as worded; included initial question stem
Respondent (R): Uh, uh, what were the choices again?	Respondent	Clarification	Ask for repeat response options
I: Uh, yes or no. Would it be a priority?	Interviewer	Probing	Repeat response options
R: Oh. Uh, no.	Respondent	Answer provided	Adequate answer

Two master coders independently coded 10 percent of the transcripts to assess reliability of the behavior codes. Kappas were 1.00 for the actor, 0.90 for the initial action for the respondents, and 0.95 for the initial action for the interviewers. For the specific sets of behaviors that we examine in this article, defined by combinations of initial actions and assessments, kappa values were 0.98 for whether the question stem or response options were included in the initial question reading, 0.76 for probing by reading the question, 0.69 for probing by reading the response options, and 0.47 for the respondent requesting the question or response options be repeated, all above the recommended threshold of kappa = 0.40 (Bilgen and Belli 2010).

First, we count the total number of conversational turns for each question–answer sequence for each administration of each battery item (Table 2). For paradigmatic sequences, we expect two or three conversational turns—a question administered by the interviewer, a respondent answer, and potential interviewer recognition of that answer (Schaeffer and Maynard 1996). The average number of turns (and range) for items in each battery is as follows: B1, 2.96 (2–31); B2, 2.54 (2–27); B3, 2.76 (2–23); and B4, 2.84 (2–35).

Table 2.

Mean Number of Conversational Turns and Interviewer Behavior Rates by Battery

Variable	M (SD) or %
Variable	B1	B2	B3	B4
Mean number of conversational turns	2.96 (1.87)	2.54 (1.39)	2.76 (1.58)	2.84 (1.85)
Introductory question reading
Question stem included
Second and later items in battery	3.25%	0.89%	2.51%	1.20%
Response options included
Second and later items in battery	3.22%	1.68%	6.46%	9.31%
Probing
Repeat all or part of the question stem	4.07%	3.30%	4.42%	6.48%
Repeat all of question exactly
Repeat all of question with changes Repeat all of question with repairs
Repeat part of the question exactly
Repeat part of the question with changes Repeat part of the question with repairs
Repeat all or part of the response options	12.36%	5.07%	8.13%	7.72%
Ask for explicit response
Repeat response options
Respondent requests repeat	4.29%	2.84%	5.31%	6.91%
Asks to repeat the question
Asks to repeat response options
Asks, “What?”
Number of respondents	433	431	438	432
Number of observations	6,062	5,603	3,504	2,592

Note: For introductory question reading, the question stem was required for the first item in the battery but not the second and later items. B = battery.

Next we create two dichotomous variables indicating if the question stem (= 1) or response options (= 1) were included versus not included (= 0) when the item was initially asked. Between 0.89 percent and 3.25 percent of initial item readings contained the question stem, and between 1.68 percent and 9.31 percent included the response options. The question stem was read for the first item in each battery for 100 percent of the respondents (although between 20 and 40 percent of the administrations were not read exactly as worded). The response options were a scripted part of the question stem for B1 and B4, and thus were read in all of the first question readings; B2 and B3 had response options of yes and no, which interviewers chose to include when reading the first item in the battery for 53.13 percent of the cases in B2 and 55.79 percent of the cases in B3 (an unscripted reading of the initial question stem for the first item).

We use two dichotomous indicators for whether the interviewer ever probed by repeating the question stem (1 = interviewer probed using question stem, 0 = no probe using question stem) or response options (1 = interviewer probed using response options, 0 = no probe using response options) on any conversational turn for each item in the battery. The specific behavior codes used are shown under the Probing heading of Table 2. Probing using the question stem occurred between 3.30 percent and 6.48 percent of items; probing using the response options occurred between 5.07 percent and 12.36 percent of items. Virtually every instance of probing using the response options repeated both of the dichotomous response options (less than 0.37 percent of items were probed using only one response option).

The final variable is a dichotomous indicator for whether respondents ever asked for the question or response options to be repeated or stated “What?” (= 1) versus not engaging in these behaviors (= 0). Respondents requested the question or response options be repeated on 2.84 percent to 6.91 percent of the items.

3.3. Respondent Answers

For respondent answers, we first create an indicator variable for whether the respondent provided a nonsubstantive answer of DK or REF (= 1) versus any substantive answer (= 0) for each item in the battery. Overall, item missing rates were quite low (less than 2.5 percent of all answers in any battery were DK/REF). Then, for each battery item with a substantive answer, we create an indicator of providing a final endorsement answer (do, yes, concerned = 1) versus a nonendorsement (do not, no, not concerned = 0) answer; nonsubstantive answers are set to missing. Overall endorsement rates varied substantially across batteries.

3.4. Control Variables

Other factors may also influence interviewer and respondent behaviors or respondent answers. First, we statistically adjust for a set of measures of the complexity of each item in the battery. Items are typically similar to one another in length within a battery, an important predictor of reliability and validity in battery items (e.g., Alwin 2007, 2010; Alwin and Beatty 2016). To account for this, we include the number of words in each item. Because the items are randomized, and thus respondents differ in the “first item” where the question stem is always read, our measure of question length includes the full question stem for each item. The average number of words ranges from 20 to 30 words (see Table A1 in the online appendix). We include the Flesch-Kincaid reading grade level, measured using Microsoft Word (Olson and Smyth 2015; but see Lenzner 2012, 2014). The average item’s Flesch-Kincaid reading level ranges from 6.29 (B3), indicating a reading level around sixth grade, to 12.48 (B4). Three linguistic measures obtained using the online Question Understanding Aid (QUAID) tool (Dykema et al. forthcoming; Graesser et al. 2006) include whether the item contains an unfamiliar technical term (21.5–100 percent of items), a vague or imprecise relative term (including vague quantifiers; approximately 15 percent of items in three of the four batteries; no items in the fourth battery), or a vague or ambiguous noun phrase (including nouns with multiple meanings; approximately 20 percent of items in two batteries and none in the other two). None of the items were identified to have complex syntax as defined by QUAID, and QUAID identified only one item as “working memory overload”; these question characteristics are omitted.

Second, respondents with lower working-memory capacity and cognitive ability, including older adults and those with lower education levels, tend to show more interactional problems and lower data quality (Knäuper et al. 1997; Krosnick 1991; Narayan and Krosnick 1996). Thus, we control for age and education. Regarding age, respondents were asked, “What is your age?” and answers were coded into categories of less than 40 (21.4 percent), 40 to 65 (48.0 percent), and greater than 65 (30.6 percent). For education, respondents were asked, “What is the highest level of education you have completed?” Answers were categorized as less than high school graduate (5.3 percent), high school graduate (21.0 percent), some college (21.7 percent), or college degree or higher (52.1 percent). The missing data rate on both age and education was 1.6 percent; we imputed the modal category for missing values on these variables.

Third, interviewer experience—both overall and accumulated over the course of a field period—is associated with shorter survey length (Olson and Peytchev 2007). In addition, interviewers with more experience elicit more acquiescent responses than those with less experience (Olson and Bilgen 2011). Thus, we control for two types of interviewer experience. The first experience variable is length of employment at Gallup, measured as less than one year at the job (= 0; 48.9 percent) versus one year or more at the job (= 1; 51.1 percent). The second is experience on this particular survey, measured by a count variable capturing the order in which each interviewers’ surveys were completed (i.e., 1 = first interview, 2 = second interview; mean = 15.8; range = 10–25).

3.5. Analysis Methods: Cross-classified Random-effects Models

The behaviors and answers for each battery item are answered by each respondent, and respondents are nested within interviewers. Thus, four-level cross-classified random-effects models allow us to take into account the nesting of behaviors or responses to items in a battery within interviewers, questions, and respondents. For analyses of the behaviors and DK/REF responses, these models are estimated by combining all four batteries together, yielding a total of 17,761 items (16,027 items when the first item is excluded) across the four batteries and 438 respondents.

To evaluate Research Question 1, we use cross-classified random-effects logistic regression models to predict introductory question reading, probing, and respondent requests for assistance on any item within a battery, and cross-classified random-effects linear regression models to predict the total number of conversational turns.¹ To examine Research Questions 2 and 3, we predict nonsubstantive answers and substantive answers using slightly different approaches. First, we examine whether respondents gave a final nonsubstantive answer (DK/REF = 1) versus any substantive answer (= 0) through a cross-classified random-effects logistic regression model, including all of the batteries in the same model. Among respondents who offered a substantive response, we predict that they will provide a final endorsement (do, yes, concerned; coded 1) versus a nonendorsement (do not, no, not concerned; coded 0) response using cross-classified multilevel logistic regression models. For this set of analyses, because each battery has a different set of response options, we estimate models for the four batteries separately. We estimate these models using Stata 15.0 and the mixed command with restricted maximum likelihood (number of conversational turns) and meqrlogit command with a QR decomposition to estimate the variance components (interviewer and respondent behaviors; survey responses). Variance tests overall and for individual variance components were conducted using both likelihood ratio tests and a mixture of chi-square distributions, as shown in Table 3 (Rabe-Hesketh and Skrondal 2012a, 2012b). Stata code is available from the authors on request.

Table 3.

Variance Parameter Estimates for Interviewer and Respondent Behaviors, Base Models with No Covariates and Full Models

Variable	Interviewer	Question	Respondent	Response	Likelihood Ratio Test
Number of turns
Variance: Base model	.062****	.050****	.284****	2.435	1738.09****
Variance partition coefficient	.022	.018	.100	.860
Variance: Full model	.068****	.022****	.268****	2.340	1558.03****
Percentage reduction in variance	–9.7%	56.0%	5.6%	3.9%
Introductory question reading:Included stem
Variance: Base model	1.797****	.574****	1.005****		607.24****
Variance partition coefficient	.270	.086	.151
Variance: Full model	1.845****	.117**	1.012****		523.00****
Percentage reduction in variance	–2.6%	79.6%	–.8%
Introductory question reading:Included response options
Variance: Base model	4.223****	.730****	1.417****		1523.29****
Variance Partition coefficient	.437	.076	.147
Variance: Full model	4.385****	.000	1.425****		1257.79****
Percentage reduction in variance	–3.8%	100.0%	–.5%
Probe: Question stem
Variance: Base model	.016	.175****	.650****		251.11****
Variance partition coefficient	.004	.042	.157
Variance: Full model	.029	.099****	.592****		189.16****
Percentage reduction in variance	–87.1%	43.5%	9.0%
Probe: Response options
Variance: Base model	.086***	.197****	.654****		661.93****
Variance partition coefficient	.020	.047	.155
Variance: Full model	.069**	.006	.655****		474.30****
Percentage reduction in variance	20.6%	97.0%	–.2%
Respondent requests repeat
Variance: Base model	.092**	.247****	.657****		304.33****
Variance partition coefficient	.021	.058	.153
Variance: Full model	.043	.106****	.651****		203.41****
Percentage reduction in variance	53.8%	57.0%	.7%

Note: Variance tests overall and for individual variance components conducted using a likelihood ratio test and a mixture of chi-square distributions. Variance parameters for the number of conversational turns estimated from a linear model; variance parameters for the interviewer and respondent behaviors estimated from a logistic regression model. The variance partition coefficient is the proportion of the total variance associated with each level in the model from the base (empty) model. The full model contains respondent, interviewer, and question characteristics.

p < .05. **p < .01. ***p < .001. ****p < .0001.

For our base model for number of conversational turns, for example, we predict $NumTurn s_{i (j_{1}, j_{2}) k} = β_{0} + u_{j_{1}} + u_{j_{2}} + v_{k} + e_{i (j_{1}, j_{2}) k}$ for behavior i to item j₁, respondent j₂, and interviewer k, where $u_{j_{1}} ~ N (0, σ_{u (1)}^{2}), u_{j_{2}} ~ N (0, σ_{u (2)}^{2}), ν_{k} ~ N (0, σ_{ν}^{2}), and e_{i (j_{1}, j_{2}) k} ~ N (0, σ_{e}^{2})$ .

In our base models, we are interested in the variance partition coefficients (VPCs)—the proportion of variance for each level out of the total of the variance components (Goldstein, Browne, and Rasbash 2002)—and the intraclass correlation coefficients (ICCs)—the level of homogeneity of behaviors within each item, respondent, and interviewer). For example, the VPC for interviewers is calculated as follows:

{VPC}_{I} = \frac{σ_{v}^{2}}{σ_{u (1)}^{2} + σ_{u (2)}^{2} + σ_{v}^{2} + σ_{e}^{2}}

By assuming the error variance $σ_{e}^{2}$ follows a logistic distribution, we can calculate the VPCs and ICCs using π²/3 as the Level 1 error variance for logistic regression models (Snijders and Bosker 1999). The ICCs are reported in Table A2 in the online appendix.

We start by examining base (empty) models for each of the behaviors. First, we examine whether the question stem was included (= 1) versus not included (= 0) when the item was initially asked, and the same for the response options. Interviewers vary widely in introductory question reading behaviors of including the question stem (VPC = 27.0 percent, p < .0001) and response options (VPC = 43.7 percent, p < .0001) for Items 2 and later in the battery, reflecting interviewer discretion on these items. Interviewers vary to a lesser degree in the total number of conversational turns (p < .0001), the use of probes with response options (p = .0002), and for respondents requesting a repeat of the question (p = .002) (VPCs range from 0.4 percent to 2.2 percent), and interviewers do not vary for probes with the question stem (p = .28). Respondents contribute a significant portion of variability in all of the behaviors (VPCs range from 10.0 percent to 15.7 percent, p < .0001), as do the individual items (VPCs range from 1.8 percent to 8.6 percent, p < .0001). These findings suggest that a multilevel framework is appropriate; although the interviewer variance term is not significantly different from zero in the base model for probing using the question stem, we include interviewer, question, and respondent random effects in all analyses for consistency.

The key independent variable for Research Questions 1 and 2 is item location. We include a log-transformed variable for item location to examine whether there was a decline or increase in any of the outcomes over the course of the battery items (the log transformation showed better model fit than a linear term). We also include indicators for the four different batteries to account for differences in battery content, an interaction term between battery and item location to permit differences across the batteries in the rate of decline or increase, the question complexity measures, respondent age, respondent education, and both measures of interviewer experience. For instance, using the above notation, the model predicting the total number of conversational turns for item j₁, respondent j₂, and interviewer k, is

\begin{matrix} {NumTurns}_{i (j_{1}, j_{2}) k} = β_{0} + β_{1 b} \ln ({Location}_{i (j_{1}, j_{2}) k}) + \sum_{b = 1}^{4} β_{2 b} {Battery}_{j_{1} k} \\ + \sum_{c = 1}^{4} β_{3 c} {Battery}_{j_{1} k} * \ln ({Location}_{i (j_{1}, j_{2}) k}) + \sum_{d = 1}^{D} β_{4 d} {QuestionComp}_{j_{1} k} \\ + \sum_{e = 1}^{E} β_{5 e} {Age}_{j_{2} k} + \sum_{f = 1}^{F} β_{6 f} {Educ}_{j_{2} k} + β_{7} {N th Survey}_{j_{2} k} \\ + β_{8} {IExp}_{k} + u_{j_{1}} + u_{j_{2}} + v_{k} + e_{i (j_{1}, j_{2}) k} \end{matrix}

Appropriate adaptations are made to the model to predict $logit [P (I' {wBeh}_{i (j_{1}, j_{2}) k} = 1)]$ , $logit [P (DK / {REF}_{i (j_{1}, j_{2}) k} = 1)]$ , and $logit [P ({Endorsement}_{i (j_{1}, j_{2}) k} = 1)]$ .

For Research Question 3, we expand the models from Research Question 2 to include the interviewer and respondent behavior variables as predictors of final DK/REF and endorsement responses. The complete model for DK/REF answers is

\begin{matrix} logit [P (DK / {REF}_{i (j_{1}, j_{2}) k} = 1)] = β_{0} + β_{1 b} \ln ({Location}_{i (j_{1}, j_{2}) k}) + \sum_{b = 1}^{4} β_{2 b} {Battery}_{j_{1} k} \\ + \sum_{c = 1}^{4} β_{3 c} {Battery}_{j_{1} k} * \ln ({Location}_{i (j_{1}, j_{2}) k}) + \sum_{d = 1}^{D} β_{4 d} {QuestionComp}_{j_{1} k} \\ + \sum_{e = 1}^{E} β_{5 e} {Age}_{j_{2} k} + \sum_{f = 1}^{F} β_{6 f} {Educ}_{j_{2} k} + β_{7} {NthSurvey}_{j_{2} k} + β_{8} {IExp}_{k} \\ + β_{9} {NumTurns}_{i (j_{1}, j_{2}) k} + β_{10} {InitialStem}_{i (j 1, j 2) k} + β_{11} {InitialRO}_{i (j_{1}, j_{2}) k} \\ + β_{12} ProbeSte m_{i (j_{1}, j_{2}) k} + β_{13} ProbeR O_{i (j_{1}, j_{2}) k} + β_{14} RRepea t_{i (j_{1}, j_{2}) k} + u_{j_{1}} + u_{j_{2}} + v_{k} \end{matrix}

Because the total number of turns and the individual behaviors are correlated (e.g., probing occurs only on an additional conversational turn), we first estimate models with just the individual interviewer and respondent behaviors (Model 1) and then add the total number of conversational turns to the model (Model 2). Because the initial question-stem asking behaviors are constant for all respondents for all items asked as the first in the battery for all batteries, and including the response options in the initial asking are constant for all respondents for two batteries, we estimate models separately for responses to all items and for the second and later items in the battery (Models 3 and 4). The four batteries are examined separately for substantive responses.

4. Findings

4.1. Research Question 1: Do Interviewer and Respondent Behaviors in Battery Questions Differ by the Location of Items in the Battery?

Figure 2 shows the average number of conversational turns for each item in the battery by item location. Although the mean number of turns differs across the batteries, in all four batteries, the pattern is identical: items administered earlier in the battery have more conversational turns than items administered later in the battery (Table 4; log[location], p < .0001; battery*log[location], p < .0001). The trend is especially strong for B1, the first and longest battery administered and the first question in the questionnaire, and it is somewhat attenuated for the other batteries. This pattern is independent of the content of the items, due to the randomization of items across locations. Thus, the learning/training hypothesis (Hypothesis 1a) is supported: interviewers and respondents need more conversation on early items in the battery than on later items.

Figure 2.

The mean number of conversational turns by item location.

Table 4.

Test Statistics for Item Location Predicting Interview Behaviors and DK/REF Responses

Variable	Log(location)	Battery	Log(location)*Battery
Number of conversational turns	z = −22.55p < .0001	χ²(3) = 67.69p < .0001	χ²(3) = 118.27p < .0001
Introductory question reading
Question stem	z = −8.34p < .0001	χ²(3) = 10.82p = .0128	χ²(3) = 6.85p = .0768
Response options	z = −6.57p < .0001	χ²(3) = 26.29p < .0001	χ²(3) = 8.05p = .0449
Probing
Question stem	z = −9.01p < .0001	χ²(3) = 1.83p = .6086	χ²(3) = 3.83p = .2808
Response options	z = −19.07p < .0001	χ²(3) = 83.35p < .0001	χ²(3) = 26.49p < .0001
Respondent requests repeat	z = −10.35p < .0001	χ²(3) = 4.29p = .2316	χ²(3) = 8.25p = .0410
DK/REF	z = −3.84p < .0001	χ²(3) = 13.77p = .0032	χ²(3) = 9.24p = .0262

Note: Tests reported are from multilevel models accounting for clustering of responses within items, interviewers, and respondents, and account for question characteristics, respondent characteristics, and interviewer experience. DK/REF = “don’t know”/refusal.

Now we turn to individual interviewer and respondent behaviors. We present the full models predicting the initial question reading behavior in Table 5. (Other behaviors are presented in the online appendix.) Once again, our hypotheses about differences in interviewer and respondent behaviors across item location in the batteries are confirmed. In all four batteries, there is significant decline (p < .0001) for all of the behaviors across item location in the battery (see again Table 4). Figure 3 shows that there are higher rates of including the question stem in the initial reading, reading the response options, probing with the question stem, and probing with the response options for items asked early in the battery. Respondent requests for repeats of the question follow the same pattern (Figure 4). This decline in all of the behaviors is also consistent with the learning hypothesis (Hypothesis 1a). In addition, interviewer probing and respondent requests to repeat questions or response options increase the number of conversational turns, helping explain why the early items have longer interactions.

Table 5.

Cross-classified Logistic Regression Coefficients and Standard Errors Predicting Interviewer Introductory Question Reading

Variable	Stem Included		Response Options Included
Variable	Coefficient	SE	Coefficient	SE
Intercept	−2.225*	1.099	−2.516	1.550
Log(location)	−1.113****	.133	−.898****	.137
Battery
B1
B2	−1.704**	.659	.099	.454
B3	−1.522**	.543	1.732****	.382
B4	−.224	.865	.950*	.465
Battery*Log(location)
B1*Log(location)
B2*Log(location)	.423	.286	−.250	.235
B3*Log(location)	.668*	.298	−.644**	.230
B4*Log(location)	−.279	.542	−.126	.263
Question complexity
Number of words in question	.088	.054	.058*	.026
Flesch-Kincaid reading score	−.221*	.106	.049	.052
Unfamiliar term	−.189	.216	−.148	.134
Imprecise term	.030	.280	−.220	.155
Ambiguous noun	−.866*	.425	.201	.175
Age
<40	—	—	—	—
40–65	.060	.266	.308	.264
65+	.609*	.284	.732**	.278
Education
<High school graduate	—	—	—	—
High school graduate	−.788+	.429	−.526	.524
Some college	−.442	.417	−.783	.524
College +	−.691+	.391	−.787	.504
Interviewer experience
<1 year	—	—	—	—
1 year +	.546	.546	.072	.809
Within-survey experience	−.036	.061	−.073	.089
Variance components
Interviewer variance	1.845****	.596	4.385****	1.471
Question variance	.117**	.066	7.16 × 10⁻¹³	2.17 × 10⁻⁷
Respondent variance	1.012****	.229	1.425****	.231
Likelihood ratio test for variance components	523.00****		1257.79****
Model fit statistics
Log-likelihood	−1241.67		−1876.11
Wald χ²	159.61****		414.89****
AIC	2529.35		3798.21
BIC	2706.04		3974.94
Number of …
Interviewers	31		31
Respondents	438		438
Observations	16,027		16,027

Note: AIC = Akaike information criterion; BIC = Bayesian information criterion.

p < .05. **p < .01. ***p < .001. ****p < .0001.

Figure 3.

Percentage of respondent behaviors by item location.

Figure 4.

Percentage of respondent requests for repeat by item location.

Looking again at Table 3, we see the variance components for the full models and the percentage reduction in variance from the base model. Between 43.5 percent and 100 percent of the variance due to questions was explained by the covariates included in the models. The covariates included in these models failed to explain much of the interviewer-level and respondent-level variation, and even increased either the interviewer-level or respondent-level variation for five of the six behaviors.

4.2. Research Question 2: Do Answers Provided by Respondents in Battery Questions Differ across the Location of Items in the Battery?

4.2.1. DK/REF answers

As shown at the top of Table 6, in the base model, there is significant question and respondent-level variance for final DK/REF answers (p < .0001), but the interviewer-level variance in DK/REF answers is not different from zero (likelihood ratio, χ² = 0.00, p = 1.00).

Table 6.

Variance Parameter Estimates for DK/REF and Substantive Answers, Base Models with No Covariates and Full Models

Variable	Interviewer	Question	Respondent	Likelihood Ratio Test
DK/REF (all batteries)
Variance: Base model	5.65 × 10⁻¹⁸	.289****	5.587****	868.50****
VPC: Base model	.000	.032	.610
Variance: Full model	1.28 × 10⁻⁹	.161****	4.265****	615.70****
Percentage reduction in variance	—	44.2%	23.7%
B1: Do
Variance: Base model	.049*	1.321****	.922****	1385.67****
VPC: Base model	.009	.237	.165
Variance: Full model	.043+	.854****	.895****	986.64****
Percentage reduction in variance	12.3%	35.3%	2.9%
B2: Yes
Variance: Base model	.009	2.658****	1.694****	1953.05****
VPC: Base model	.001	.347	.221
Variance: Full model	.006	1.749****	1.669****	1378.34****
Percentage reduction in variance	36.3%	34.2%	1.5%
B3: Are concerned
Variance: Base model	9.85 × 10⁻¹²	.285****	2.130****	463.47****
VPC: Base model	.000	.050	.373
Standard deviation: Full model	5.54 × 10⁻¹⁶	.054***	1.992****	353.27****
Percentage reduction in variance	—	81.1%	6.5%
B4: Yes
Variance: Base model	.050	1.712****	.806****	624.23****
VPC: Base model	.009	.292	.138
Standard deviation: Full model	.042	.238****	.750****	122.70****
Percentage reduction in variance	17.1%	86.1%	7.0%

Note: All items included in models. Variance tests conducted using both a likelihood ratio test and a mixture of chi-square distributions. Base models are estimated without any covariates. Full models correspond to Model 2. DK/REF = “don’t know”/refusal; B = battery; VPC = variance partition coefficient, or the proportion of the total variance associated with each level in the model.

p < .05. **p < .01. ***p < .001. ****p < .0001.

Figure 5 shows the percentage of DK/REF responses at each item location by battery. For all items in all locations, the DK/REF rate does not exceed 4.2%, making a DK/REF answer a rare event (virtually all of these nonsubstantive responses are recorded as DK responses). There is a negative significant association between log(location) and DK/REF answers for B1, B2, and B3 (log[location] coefficient = −0.466, p < .0001), but a positive association for B4 (B4*Log[location] coefficient = 0.801, p < .01; see again Table 4). That is, in the first three batteries, items presented later in the battery have lower rates of DK/REF responses, consistent with a learning hypothesis (Hypothesis 2a). In the last battery, B4, DK rates increase for items asked later in the battery, perhaps because this battery was difficult and later in the questionnaire, triggering fatigue (Hypothesis 2b).

Figure 5.

Percentage of “don’t know”/refusal responses at each item location based on four battery questions.

4.2.2. Substantive answers

Across the four batteries, there is virtually no interviewer-level variance in the substantive answers provided (VPC < 0.01 for all batteries; p < .05 for B1 only; Table 6, base models). However, there are significant item-level and respondent-level effects on responses (p < .0001).

The association between item location and providing an endorsement response varies across batteries (Figure 6). Endorsement (saying “do”) increases across items in B1 (coefficient = 0.095, p = .025) but decreases across items in B2 (coefficient = −0.134, p = .01) and B4 (coefficient = −0.198, p = .031), with yes responses indicating endorsement in B2 and B4. The increased rates of do responses in B1 may indicate increased acquiescence for later items (Hypothesis 2b), but we see less acquiescence on later items in the two batteries asking for a yes response. In sum, there is evidence that the order in which items are presented affects DK/REF and substantive responses, independent of the content of those items. The common recommendation for questionnaire designers to randomly rotate items spreads the “location error” across items, rather than concentrating in a particular item.

Figure 6.

Percentage of substantive responses by item location, based on four battery questions.

4.3. Research Question 3: Are the Answers Provided by Respondents in Battery Items Associated with Interviewer and Respondent Behaviors on Those Items?

4.3.1. DK/REF answers

Now we examine whether respondents provide different answers for different types of interviewer and respondent behaviors. Table 7 shows model coefficients from four models predicting final DK/REF answers across the four batteries.

Table 7.

Cross-classified Logistic Regression Model Coefficients and Standard Errors Predicting Response of DK/REF (= 1) versus Substantive Response (= 0) with Interviewer and Respondent Behaviors across the Four Batteries

Variable	All Items				Items 2+
	Model 1		Model 2		Model 3		Model 4
	Coefficient	SE	Coefficient	SE	Coefficient	SE	Coefficient	SE
Intercept	−5.536****	.903	−6.378****	.892	−5.685****	1.023	−6.710****	1.101
Number of conversational turns	—	—	.270****	.030	—	—	.332****	.036
Introductory question reading
Included question stem	.270	.232	.136	.240	.468	.322	.341	.334
Included response options	.109	.226	.084	.233	.301	.295	.272	.301
Probing
Repeated question stem	.868****	.242	−.137	.274	.906***	.276	−.252	.312
Repeated response options	1.789****	.149	.987****	.175	1.804****	.165	.831****	.197
Respondent requests repeat	−.407	.256	−.745**	.264	−.459	.297	−.933**	.308
Log(location)	−.055	.148	−.027	.151	−.001	.186	.004	.189
Battery
B1	—	—	—	—	—	—	—	—
B2	−.259	.560	−.045	.563	−.208	.717	−.109	.724
B3	−.426	.492	−.121	.494	.610	.706	1.053	.715
B4	−2.153**	.748	−2.074**	.757	−2.122*	1.003	−1.879	1.015
Battery*Log(location)
B1*Log(location)	—	—	—	—	—	—	—	—
B2*Log(location)	−.101	.184	−.123	.190	−.135	.270	−.075	.276
B3*Log(location)	.223	.252	.112	.256	−.394	.387	−.588	.396
B4*Log(location)	.626*	.289	.640*	.298	.653	.484	.555	.492
Question complexity
Number of words in question	−.079	.057	−.072	.057	−.085	.062	−.080	.062
Flesch-Kincaid reading score	.290*	.117	.302**	.117	.298*	.127	.315*	.127
Unfamiliar term	.559*	.229	.485*	.229	.474	.245	.370	.245
Imprecise term	−.570	.354	−.683	.360	−.634	.383	−.664	.384
Ambiguous noun	−.949	.491	−.947	.498	−1.090*	.536	−1.021	.533
Age
<40	—	—	—	—	—	—	—	—
40–65	.494	.403	.450	.394	.471	.447	.428	.441
65+	.838*	.419	.814*	.409	.791	.466	.760	.459
Education
<High school graduate	—	—	—	—	—	—	—	—
High school graduate	−.285	.659	−.173	.641	−.787	.741	−.637	.722
Some college	−.654	.660	−.558	.642	−1.002	.735	−.873	.717
College +	−.871	.624	−.797	.607	−1.111	.691	−1.020	.675
Interviewer experience
<1 year (reference)	—	—	—	—	—	—	—	—
1 year +	−.416	.296	−.417	.289	−.372	.330	−.393	.325
Nth interview	−.025	.031	−.018	.030	−.022	.034	−.014	.034
Variance components
Interviewer variance	1.48 × 10⁻¹⁰	.000	1.28 × 10⁻⁹	.000	1.63 × 10⁻¹¹	.000	6.46 × 10⁻¹⁶	.000
Question variance	.167****	.058	.161****	.058	.189****	.066	.181****	.067
Respondent variance	4.629****	.884	4.265****	.802	5.922****	1.372	5.503****	1.237
Likelihood ratio test	654.51****		615.70****		589.01****		561.12****
Model fit statistics
Log-likelihood	–1297.65		–1255.81		–1123.60		–1078.33
Wald χ²	210.13****		276.40****		166.33****		235.18****
AIC	2651.30		2569.62		2303.20		2214.67
BIC	2869.27		2795.38		2518.30		2437.44
Number of …
Interviewers	31		31		31		31
Respondents	438		438		438		438
Observations	17,761		17,761		16,027		16,027

Note: DK/REF = “don’t know”/refusal; B = battery; AIC = Akaike information criterion; BIC = Bayesian information criterion.

p < .05. **p < .01. ***p < .001. ****p < .0001.

First, initial question asking behavior, including either the question stem or the response options, is not associated with DK/REF answers, counter to Hypothesis 3a. Next, probing by repeating the question stem, probing by repeating the response options, and larger numbers of conversational turns are associated with higher rates of DK/REF reports, confirming Hypothesis 3b. For example, each additional conversational turn is associated with a 31 percent increase (Model 2: e^0.270 = 1.310, p < .0001) in the odds of providing DK/REF reports. In addition, probing using the question stem or the response options during the question yields a 138 percent and 498 percent increase (Model 2: probe question stem, e^0.868 = 2.382, p < .0001; probe response options, e^1.789 = 5.983, p < .0001) in the odds of DK/REF responses. As noted earlier, these effects may be somewhat circular—interviewers are trained to probe when an initial DK/REF is offered by the respondent, thus adding conversational turns and occurring because of the initial DK/REF answer. When we include the total number of conversational turns, the association between probing by restating the question stem and DK/REF answers disappears, but the association with probing using the response options stays. Thus, even with additional probing by restating the response options and more conversation, the interaction still resolves in a final DK/REF response.

Finally, as hypothesized (Hypothesis 3c), when respondents ask for help with the response options or question stem, they are significantly (p < .01) less likely to provide a DK/REF answer than a substantive answer. For example, the odds of a DK/REF answer are (Model 2: e^−0.745 = 0.475, p = .005) about 53 percent smaller on items in which the respondent requests assistance than those in which they do not request it. Thus, respondents who request assistance are trying to answer with a substantive response rather than a nonsubstantive response.

The included set of covariates explained 44.2 percent of the question-level variation in DK/REF final answers and 23.7 percent of the respondent-level variation (see again Table 6).

4.3.2. Substantive answers

We now look at whether the behaviors during the interview affect the substantive answers (Hypothesis 3d). Because the four batteries are examined separately, the sample size is reduced in each model, especially for less frequent behaviors. However, understanding the risk for behaviors on these battery items to affect survey answers is important. Thus, for this analysis, we report p < .10 as statistically significant, and report the odds ratios to interpret the magnitude of the effect. Results are shown in Table 8.

Table 8.

Coefficients from Cross-classified Logistic Regression Models Predicting Substantive Responses of “Do” (B1), “Yes” (B2), “Concerned” (B3), and “Yes” (B4), Excluding DK/REF Responses, with Interviewer and Respondent Behaviors

Yes/No	Number of Turns	Introductory Question Reading		Probing		Respondent Requests Repeat
Yes/No	Number of Turns	Question Stem	Response Options	Question Stem	Response Options	Respondent Requests Repeat
B1
All items
Model 1		.201	.257	.088	−.084	−.148
Model 2	.013	.198	.256	.052	−.122	−.164
Item = 2+
Model 3		.273	.402†	.124	−.088	−.239
Model 4	.011	.270	.401†	.093	−.120	−.251
B2
All items
Model 1		−.405*	.222	.006	.860****	.071
Model 2	−.040	−.403*	.225	.120	.972****	.128
Item = 2+
Model 3		.925+	.607†	.109	.950****	−.069
Model 4	−.054	.957*	.615†	.268	1.103****	.013
B3
All items
Model 1		−.355	.125	.114	−.152	−.347
Model 2	−.099*	−.332	.157	.358	.120	−.208
Item = 2+
Model 3		−.273	.209	.025	−.493*	−.253
Model 4	−.175**	−.280	.234	.423	.018	−.002
B4
All items
Model 1		−.051	−.268	−.299	−.839	.281
Model 2	−.045	−.021	−.260	−.141	−.698**	.303
Item = 2+
Model 3		−.169	−.477†	−.245	−.876***	−.041
Model 4	−.054	−.107	−.470†	−.102	−.719*	.006

Note: B = battery: DK/REF = “don’t know”/refusal.

†

p < .10. *p < .05. **p < .01. ***p < .001. ****p < .0001.

First, simply restating the introductory text and/or the response options affected the endorsement rate in three of the four batteries (p < .07). In B1, the odds of endorsement increased by 49 percent when the interviewer restated the response options (Model 4: e^0.401 = 1.493, p = .06). In B2, the odds that respondents endorsed an item after the second item increased by 160 percent (Model 4: e^0.957 = 2.604, p = .048) when the interviewer included the question stem and by 85 percent when the interviewer included the response options (Model 4: e^0.615 = 1.850, p = .069). In B4, the odds of endorsement decreased by 37.5 percent (Model 4: e^−0.470 = 0.625, p = .057) when the interviewer included the response options. This is striking.

Probing by repeating the response options is related to substantive answers, but there is no association between probing by repeating the question stem or between respondent requests for the question to be repeated and substantive answers. In B2, when respondents are probed with the response options, the odds double (Model 2: e^0.972 = 2.643, p < .0001) that they will answer that a given country will become a military threat. In B4, when respondents are probed using the response options, the odds are (Model 2: e^-0.698 = 0.498, p = .008) about 50 percent less that the respondent will indicate that the U.S. and Japanese governments should work together to give priority to resolving a particular policy item about North Korea. Both of these items have yes/no response options, although other characteristics of the batteries are quite different. Thus, probing by providing the response options is associated with differences in substantive responses, but requests by respondents for the question to be repeated seem to come equally from the positive and negative sides of the attitudinal domains.

The number of conversational turns is significantly associated with substantive responses in only one battery. In B3, more conversational turns are associated with lower rates of reports of being concerned about China (Model 2: coefficient = −0.099, p = .029).

Finally, we examine the proportion of the variance components at each level that were explained by the included covariates. Although there was no interviewer-level variance for B2, B3, and B4, 12.3 percent of the (small) interviewer-level variance in substantive responses was explained for B1. In B1 and B2, just over one third of the question-level variance was explained by the included covariates; this increased to over 80 percent of the question-level variance for B3 and B4. Finally, between 1.5 percent and 7.0 percent of the respondent-level variance was explained across the four batteries.

5. Discussion

This article examined the association between item location, interviewer and respondent behaviors, and answers provided to battery questions in a telephone survey. Because the items were randomized within batteries, we can disentangle the location of a battery item from its content. In general, the results for item location support a learning hypothesis. All of the interviewer–respondent behaviors occurred at higher rates on items earlier in the battery than on items later in the battery. In addition, respondents had higher rates of DK/REF answers to items presented early in the battery and lower rates of endorsement (less acquiescence), with one exception (one battery showed higher rates of endorsement for later items).

Importantly, interview behaviors are associated with respondent answers in battery questions. Items on which there is more interaction overall have higher rates of DK/REF responses but similar or slightly lower endorsement rates (on one battery). Additional conversational turns can be driven by inadequate responses (such as DK/REF) as standardized interviewers are trained to probe these answers. Yet even with this higher rate of interaction, the answers still resolve to a DK/REF rather than a substantive answer. Thus, it appears that respondents are not forgetting the question stem or response options to these questions but genuinely do not know (or do not want to provide) an answer. Interviewer reading of the question text or the response options when presenting the items was significantly associated with endorsement rates in three of the four batteries. For example, in the first battery about confidence in American institutions, confidence in federal government agencies increased from 38.2 percent when the introductory question stem was not included to 53.3 percent on when it was included.² Items on which interviewers probed using the response options had both higher DK/REF rates and different substantive responses (albeit with opposite directions across batteries). For example, when asked whether India will become a military threat to the United States, 12.2 percent of respondents who were not probed with the response options indicated that it would, compared to 56.3 percent who were probed using the response options. It is not clear why the interaction is associated with respondent answers, but it is clear that respondents with increased interaction with the interviewer provided different answers. Respondents who asked the interviewers to repeat the question stem or the response options were less likely to provide a DK/REF answer than those who did not ask the interviewers to repeat this information, but this action was not associated with substantive answers. This suggests that requests for help with the question indicate that the respondent is trying to provide a substantive report rather than having problems.

Although there is variation in the associations between the behaviors and the substantive responses, the takeaway is that both DK/REF and substantive responses were significantly associated with at least one of the interviewer behaviors—whether question reading, probing, or simply more conversational turns. We encourage future research on this topic to further unpack this result.

So, what does this mean for data quality? Battery items as they are currently administered allow a degree of discretion over reading the question stem and/or response options not seen in other types of questions. Reports to battery questions that respond to simple behaviors by the interviewer or are associated with behaviors by the respondent are likely to be less reliable and/or valid (Krosnick and Abelson 1992) than responses that are not affected by these behaviors. As noted earlier, battery items have lower reliability and validity than other types of items; some of this lack of reliability or validity could arise because of these types of interviewer behaviors. Unfortunately, we cannot directly test this hypothesis; we do not have reinterviews with the same respondents, so we cannot directly evaluate reliability of attitudes over time with these data (for examples, see Alwin 2007; Hout and Hastings 2016). We might anticipate seeing more changes in answers over time to battery questions where the interviewer probed respondent answers using the response options, suggesting weaker or less crystallized attitudes (Smith 1985). Alternatively, increased interaction through reading or repeating the question stem may have led to a more complete understanding of the question; thus reports to these items may be more “accurate” and thus more reliable. Future research on reliability of responses to battery questions over time would benefit from incorporating indicators of these behaviors.

Although this study was the first of its kind to examine the location of items in a battery and interviewer and respondent behaviors, we examine the behaviors largely in isolation, and not as part of a sequence of behaviors that unfold during the conversation for each item. The majority of items have a paradigmatic question-answer sequence (68.2 percent), with a sizable group having a paradigmatic question-answer-feedback sequence (15.1 percent). Across all of the items and across all of the batteries, 16.7 percent of the items have more than three conversational turns, indicating deviations from a paradigmatic interviewer–respondent interaction sequence. This interaction could be used to navigate the complexities of the battery items, but it could also be used to build rapport (Garbarski et al. 2016). A sequential examination of how interviewer discretion affects respondents’ behaviors is a useful step for future research.

As an actual production survey, this article provided insights into how these items are actually administered. Although the items within each battery were randomized, the batteries themselves were not randomly presented within the survey. Thus, we cannot evaluate whether differences across the batteries occur because of the placement of the batteries or their content. Future experiments should rotate both items within batteries and batteries within the survey. In addition, only interviewers with 10 or more interviews were selected for behavior coding. This assists with model estimation, but it limits the inference to these interviewers. Furthermore, although the location of the items within the batteries is randomly assigned, the interviewer and respondent behaviors are not randomly assigned. We control for the theoretically selected effects of respondent age and education and interviewer experience, and a variety of measures of question complexity, but there could be other interviewer and respondent factors associated with both the behaviors and the survey responses that are contributing to these results. For instance, some interviewers may feel that rereading the question stem or response options or probing helps with building rapport, and thus a future study that has an interviewer-level measure of desire for rapport may provide insights (Garbarski et al. 2016; West and Blom 2017). Although this survey provided a useful case study, the batteries all used dichotomous response categories and had topics that may not be familiar for some respondents. Future work should examine batteries with ordinal scales and with other battery topics. We also focused primarily on interviewer behaviors here, with one respondent behavior related to requests for question information to be repeated. Future work should examine respondent behaviors in more detail, including how they may be associated with or trigger interviewer behaviors. Finally, this analysis was conducted on a dual-frame telephone survey; future research should examine how batteries operate in face-to-face surveys, the mode used in many of the existing evaluations of reliability and validity (Alwin 2007).

The implications for questionnaire design are clear. First, items within batteries should be randomized as interviewer and respondent behaviors and survey responses vary by location of the item. Although this is a common practice, it is not ubiquitous. Second, if keeping the question stem and response options in mind is important, questionnaire designers should write the questions as they want them to be asked, rather than relying on the interviewer’s judgment or discretion for restating the question stem or response options, keeping in mind the balance between information and repetitiveness. This study suggests that scripting these questions may, in some instances, shift overall distributions of responses. Yet the shift will correspond to an intentional decision by a survey designer and apply to all respondents equally rather than an ad hoc decision by an interviewer. Future studies should examine what happens to the interaction between interviewers and respondents and responses to survey questions when the question stem and/or response options are read for every question or scripted for a subset of item locations, keeping in mind the challenges of programming a computerized telephone instrument when the items themselves are randomized.

The implications for training and monitoring interviewers are less clear. Significant interviewer-level variance remains in initial question asking behaviors, suggesting that interviewers strongly deviate from standardization in their administration of battery items. If survey organizations want to minimize this variation without scripting question wording, interviewers could be more closely monitored on administering battery items. Monitoring data could be used to estimate these types of models during field operations, with the goal of minimizing interviewer variance. Interviewers with particularly large random interviewer effects could be targeted for retraining or intervention. In addition, standardized interviewing practice dictates that interviewers should probe DK responses, but probing using the response options was associated with substantive responses in two batteries. Developing effective interviewer training will require future insights into the specific cues provided by respondents during their interaction with the interviewer when providing uncodable responses that might resolve to a (valid) substantive response. In sum, this article was the first to study how battery questions are currently being administered in telephone surveys, including examining the roles of item location and interviewer–respondent behaviors. It provides us with an initial insight into potential causes of reduced reliability and validity in battery questions. Future research should continue to explore these questions.

Supplemental Material

Battery_Tables_REV2_APPENDIX – Supplemental material for Item Location, the Interviewer–Respondent Interaction, and Responses to Battery Questions in Telephone Surveys

Supplemental material, Battery_Tables_REV2_APPENDIX for Item Location, the Interviewer–Respondent Interaction, and Responses to Battery Questions in Telephone Surveys by Kristen Olson, Jolene D. Smyth and Beth Cochran in Sociological Methodology

Footnotes

Acknowledgements

An earlier version of this article was presented at the 2015 Annual Meeting of the Midwest Association for Public Opinion Research and the 2016 Annual Meeting of the American Association for Public Opinion Research conference. We thank participants at these meetings for helpful comments. We also thank four anonymous reviewers for comments that substantially improved the article. The data and code used in this article will be made available through the Inter-university Consortium for Political and Social Research by December 2018.

Funding

This work was supported by the National Science Foundation Grant No. SES-1132015. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Notes

Author Biographies

Kristen Olson is the Leland J. and Dorothy H. Olson Associate Professor in Sociology at the University of Nebraska–Lincoln. Her research focuses on nonsampling errors in household surveys, including coverage, measurement, and nonresponse errors. Her current projects examine interviewer effects in telephone surveys, the joint role of nonresponse and measurement errors in data quality, within-household selection in mail surveys, and questionnaire design issues in mail, web, and mobile surveys. She has published work on these topics in sociological, methodological, and public opinion research journals.

Jolene D. Smyth is an associate professor in the Department of Sociology and the director of the Bureau of Sociological Research at the University of Nebraska–Lincoln. Her research focuses on improving data collection processes through the reduction of measurement and nonresponse error. Her current projects examine how questionnaire design impacts interviewer–respondent interactions and responses in telephone surveys, how various questionnaire design features (question wording, visual emphasis, response scales, etc.) impact responses in both interviewer and self-administered surveys, and mobile web survey design.

Beth Cochran is an assistant professor in the Department of Mathematics and Statistics and statistics coordinator for the Science and Math Learning Center at James Madison University, Harrisonburg, Virginia.

References

Alwin

Duane F.

2007. Margins of Error: A Study of Reliability in Survey Measurement. Hoboken, NJ: Wiley.

Alwin

Duane F.

2010. “How Good Is Survey Measurement? Assessing the Reliability and Validity of Survey Measures.” Pp. 405–36 in Handbook of Survey Research, edited by Marsden

P. V.

Wright

J. D.

Bingley, UK: Emerald Group.

Alwin

Duane F.

Beattie

Brett A.

2016. “The KISS Principle in Survey Design: Question Length and Data Quality.” Pp. 121–52 in Sociological Methodology, vol. 46, edited by Alwin

D. F.

Thousand Oaks, CA: Sage. doi:10.1177/0081175016641714.

American National Election Studies. 2017. User’s Guide and Codebook for the ANES 2016 Time Series Study. University of Michigan and Stanford University. Retrieved October3, 2017 (http://www.electionstudies.org/studypages/anes_timeseries_2016/anes_timeseries_2016_userguidecodebook.pdf).

Beatty

Paul

. 1995. “Understanding the Standardized/Non-standardized Interviewing Controversy.”Journal of Official Statistics 11(2):147–60.

Belli

Robert F.

Lepkowski

James M.

1996. “Behavior of Survey Actors and the Accuracy of Response.” Pp. 69–74 in Health Survey Research Methods: Conference Proceedings. Washington, DC: U.S. Department of Health and Human Services.

Belli

Robert F.

Shay

William L.

Stafford

Frank P.

2001. “Event History Calendars and Question List Surveys: A Direct Comparison of Interviewing Methods.”Public Opinion Quarterly 65(1):45–74.

Bilgen

Ipek

Belli

Robert F.

2010. “Comparison of Verbal Behaviors between Calendar and Standardized Conventional Questionnaires.”Journal of Official Statistics 26(3):481–505.

Cannell

Charles F.

Miller

Peter V.

Oksenberg

Lois

. 1981. “Research on Interviewing Techniques.” Pp. 389–437 in Sociological Methodology, vol. 12, edited by Leinhardt

San Francisco: Jossey-Bass.

10.

Conrad

Frederick G.

Schober

Michael F.

2000. “Clarifying Question Meaning in a Household Telephone Survey.”Public Opinion Quarterly 64(1):1–28.

11.

Dillman

Don A.

Smyth

Jolene D.

Christian

Leah M.

2014. Internet, Phone, Mail, and Mixed-mode Surveys: The Tailored Design Method. Hoboken, NJ: Wiley.

12.

Dijkstra

Wil

. 1999. “A New Method for Studying Verbal Interactions in Survey Interviews.”Journal of Official Statistics 15(1):67–85.

13.

Dijkstra

Wil

Ongena

Yfke P.

2006. “Question-answer Sequences in Survey-interviews.”Quality and Quantity 40(6):983–1011.

14.

Dykema

Jennifer

Schaeffer

Nora Cate

Garbarski.

Dana

2016. “The Role of Question Characteristics in Designing and Evaluating Survey Questions.”American Association for Public Opinion Research webinar, presentedJanuary21, 2016.

15.

Dykema

Jennifer

Schaeffer

Nora Cate

Garbarski

Dana

Hout

Michael

. Forthcoming. “The Role of Question Characteristics in Designing and Evaluating Survey Questions.” In Advances in Questionnaire Design, Development, Evaluation, and Testing, edited by Beatty

Collins

Kaye

Lyn

Padilla

J.-L.

Willis

Wilmot

Hoboken, NJ: Wiley.

16.

Dykema

Jennifer

Schaeffer

Nora Cate

Garbaski

Dana

Nordheim

Erik V.

Banghart

Mark

Cyffka

Kristen

. 2016. “The Impact of Parenthetical Phrases on Interviewers’ and Respondents’ Processing of Survey Questions.”Survey Practice 9(2). doi:10.29115/SP-2016-0008.

17.

Fowler

Floyd J.

Jr.

1995. Improving Survey Questions: Design and Evaluation. Thousand Oaks, CA: Sage.

18.

Fowler

Floyd J.

Jr. Cannell

Charles F.

1996. “Using Behavioral Coding to Identify Cognitive Problems with Survey Questions.” Pp. 15–36 in Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research, edited by Schwarz

Sudman

San Francisco: Jossey-Bass.

19.

Fowler

Floyd J.

Jr. Mangione

Thomas W.

1990. Standardized Survey Interviewing: Minimizing Interviewer-related Error. Thousand Oaks, CA: Sage.

20.

Garbarski

Dana

Schaeffer

Nora Cate

Dykema

Jennifer

. 2016. “Interviewing Practices, Conversational Practices, and Rapport: Responsiveness and Engagement in the Standardized Survey Interview.” Pp. 1–38 in Sociological Methodology, vol. 46, edited by Alwin

D. F.

Thousand Oaks, CA: Sage Publications. doi:0081175016637890.

21.

Goldstein

Harvey

Browne

William

Rasbash

Jon

. 2002. “Partitioning Variation in Multilevel Models.”Understanding Statistics 1(4):223–31.

22.

Graesser

Arthur C.

Cai

Zhiqiang

Louwerse

Max M.

Daniel

Frances

. 2006. “Question Understanding Aid (QUAID): A Web Facility That Helps Survey Methodologists Improve the Comprehensibility of Questions.”Public Opinion Quarterly 70:1–20.

23.

Grice

H. P.

1975. “Logic and Conversation.” Pp. 41–58 in Syntaxt and Semantics, vol. 3, Speech Acts, edited by Cole

Morgan

J. L.

New York: Academic Press.

24.

Holbrook

Allyson L.

Johnson

Timothy P.

Cho

Young Ik

Shavitt

Sharon

Chavez

Noel

Weiner

Saul

. 2015. “The Effect of Question Characteristics on Respondent and Interviewer Behaviors.”Presented at the annual meeting of the American Association for Public Opinion Research, Hollywood, FL.

25.

Holbrook

Allyson

Timothy

Johnson

Cho

Young Ik

Shavitt

Sharon

Chavez

Noel

Weiner

Saul

. 2016. “Do Interviewer Errors Help Explain the Impact of Question Characteristics on Respondent Difficulties.”Survey Practice 9(2). Retrieved September25, 2017 (http://www.surveypractice.org/index.php/SurveyPractice/article/view/350).

26.

Hout

Michael

Hastings

Orestes P.

2016. “Reliability of the Core Items in the General Social Survey: Estimates from the Three-wave Panels, 2006–2014.”Sociological Science 3(43):971–1002. doi:https://dx-doi-org.web.bisu.edu.cn/10.15195/v3.a43.

27.

Knäuper

Bärbel

Belli

Robert F.

Hill

Daniel H.

Herzog

A. Regula

. 1997. “Question Difficulty and Respondent’s Cognitive Ability: The Effect on Data Quality.”Journal of Official Statistics 13(2):181–99.

28.

Krosnick

Jon A.

1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.”Applied Cognitive Psychology 5:213–36.

29.

Krosnick

Jon A.

Abelson

Robert P.

1992. “The Case for Measuring Attitude Strength in Surveys.” Pp. 177–203 in Questions about Questions: Inquiries into the Cognitive Bases of Surveys, edited by Tanur

J. M.

New York: Russell Sage Foundation.

30.

Krosnick

Jon A.

Alwin

Duane F.

1987. “Satisficing: A Strategy for Dealing with the Demands of Survey Questions.” General Social Survey Technical Report No. 74; Methodological Report No. 46. Chicago: National Opinion Research Center.

31.

Krosnick

Jon A.

Presser

Stanley

. 2010. “Question and Questionnaire Design.” Pp. 263–313 in Handbook of Survey Research, 2nd ed., edited by Marsden

P. V.

Wright

J. D.

Bingley, UK: Emerald Group.

32.

Lenzner

Timo

. 2012. “Effects of Survey Question Comprehensibility on Response Quality.”Field Methods 24(4):409–28. doi:10.1177/1525822x12448166.

33.

Lenzner

Timo

. 2014. “Are Readability Formulas Valid Tools for Assessing Survey Question Difficulty?”Sociological Methods and Research 43(4):677–98. doi:10.1177/0049124113513436.

34.

Mangione

Thomas W.

Fowler

Floyd J.

Jr. Louis

Thomas A

. 1992. “Question Characteristics and Interviewer Effects.”Journal of Official Statistics 8(3):293–307.

35.

Narayan

Sowmya

Krosnick

Jon A.

1996. “Education Moderates Some Response Effects in Attitude Measurement.”Public Opinion Quarterly 60(1):58–88.

36.

Olson

Kristen

Bilgen

Ipek

. 2011. “The Role of Interviewer Experience on Acquiescence.”Public Opinion Quarterly 75(1):99–114.

37.

Olson

Kristen

Peytchev

Andy

. 2007. “Effect of Interviewer Experience on Interview Pace and Interviewer Attitudes.”Public Opinion Quarterly 71(2):273–86.

38.

Olson

Kristen

Smyth

Jolene D.

2015. “The Effect of CATI Questions, Respondents, and Interviewers on Response Time.”Journal of Survey Statistics and Methodology 3(3):361–96. doi: 10.1093/jssam/smv021.

39.

Ongena

Yfke P.

Dijkstra

Wil

. 2006. “Methods of Behavior Coding of Survey Interviews.”Journal of Official Statistics 22(3):419–51.

40.

Peterson

Lloyd. R.

1966. “Short-term Memory.”Scientific American 215(1):90–95.

41.

Rabe-Hesketh

Sophia

Skrondal

Anders

. 2012a. Multilevel and Longitudinal Modeling Using Stata, 3rd ed., vol. 1, Continuous Responses. College Station, TX: Stata Press.

42.

Rabe-Hesketh

Sophia

Skrondal

Anders

. 2012b. Multilevel and Longitudinal Modeling Using Stata, 3rd ed., vol. 2, Categorical Responses, Counts, and Survival. College Station, TX: Stata Press.

43.

Saris

Willem E.

Gallhofer

Irmtraud N.

2007. Design, Evaluation, and Analysis of Questionnaires for Survey Research. Hoboken, NJ: Wiley.

44.

Schaeffer

Nora Cate

. 1991. “Conversation with a Purpose—or Conversation? Interaction in the Standardized Interview.” Pp. 365–91 in Measurement Errors in Surveys, edited by Biemer

Groves

R. M.

Lyberg

L. E.

Mathiowetz

N. A.

Sudman

New York: Wiley.

45.

Schaeffer

Nora Cate

Maynard

Douglas W.

1996. “From Paradigm to Prototype and Back Again: Interactive Aspects of Cognitive Processing in Standardized Survey Interviews.” Pp. 65–88 in Answering Questions Methodology for Determining Cognitive and Communicative Processes in Survey Research, edited by Schwarz

Sudman

San Francisco: Jossey-Bass.

46.

Schober

Michael F.

Conrad

Frederick G.

1997. “Does Conversational Interviewing Reduce Survey Measurement Error?”Public Opinion Quarterly 61:576–602.

47.

Schuman

Howard

Presser

Stanley

. 1981. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Content. New York: Academic Press.

48.

Siminski

Peter

. 2008. “Order Effects in Batteries of Questions.”Quality and Quantity 42(4):477–90.

49.

Smith

Tom W.

1985. “Nonattitudes: A Review and Evaluation.” Pp. 215–55 in Surveying Subjective Phenomena, edited by Turner

C. F.

Martin

New York: Russell Sage.

50.

Smith

Tom W.

Davern

Michael

Freese

Jeremy

Hout

Michael

. 2017. General Social Surveys, 1972–2016. [MRDF]. Chicago: NORC.

51.

Snijders

Tom A.

Bosker

Roel J.

1999. Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. London: Sage.

52.

Straus

Murray A.

1979. “Measuring Intrafamily Confilct and Violence: The Conflict Tactics (CT) Scales.”Journal of Marriage and Family 41(1):75–88.

53.

Suchman

Lucy

Jordan

Brigitte

. 1990. “Interactional Troubles in Face-to-face Survey Interviews.”Journal of the American Statistical Association 85(409):232–53.

54.

Sudman

Seymour

Bradburn

Norman M.

Schwarz

Norbert

. 1996. Thinking about Answers: The Application of Cognitive Processes to Survey Methodology. San Francisco: Jossey-Bass.

55.

Tourangeau

Roger

Rips

Lance J.

Rasinski

Kenneth

. 2000. The Psychology of Survey Response. Cambridge: Cambridge University Press.

56.

van Breukelen

Gerard

Moerbeek

Mirjam

. 2013. “Design Considerations in Multilevel Studies.” Pp. 183–200 in The Sage Handbook of Multilevel Modeling, edited by Scott

M. A.

Simonoff

J. S.

Marx

B. D.

London: Sage.

57.

Vassallo

Rebecca

Durrant

Gabriele

Smith

Peter

. 2017. “Separating Interviewer and Area Effects by Using a Cross-classified Multilevel Logistic Model: Simulation Findings and Implications for Survey Designs.”Journal of the Royal Statistical Society, Series A: Statistics in Society 180(2):531–50. doi:10.1111/rssa.12206.

58.

West

Brady T.

Blom

Annelies G.

2017. “Explaining Interviewer Effects: A Research Synthesis.”Journal of Survey Statistics and Methodology 5(2):175–211. doi:10.1093/jssam/smw024.

59.

West

Brady T.

Conrad

Frederick G.

Kreuter

Frauke

Mittereder

Felicitas

. 2018. “Can Conversational Interviewing Improve Survey Response Quality without Increasing Interviewer Effects?”Journal of the Royal Statistical Society, Series A: Statistics in Society 181(1):181–203. doi:10.1111/rssa.12255.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.36 MB