Paradata for Nonresponse Adjustment

Observations about sampled individual’s neighborhood

Interviewer observations about neighborhood characteristics for a sampled unit are collected in large-scale face-to-face surveys. Neighborhood observations, for example, are routinely collected in the European Social Survey (ESS), the Los Angeles Family and Neighborhood Study (LAFANS), the Survey of Consumer Finances (SCF), the National Survey of Family Growth (NSFG), the British Crime Survey (BCS), and the Health and Retirement Study (HRS) (see Callens and Croux 2003; ESS 2008; Groves and Heeringa 2006; HRS 2009; Kennickell 2000; Kreuter, Lemay, and Casas-Cordero 2007; Lynn 2003; Peterson, Sastry, and Pebley 2007). Some surveys, such as the Project on Human Development in Chicago Neighborhoods (Interuniversity Consortium for Political and Social Research 2010) or LAFANS (Sastry and Pebley 2004), are specifically designed to collect neighborhood observations to evaluate the effects of neighborhood conditions on important social and economic outcomes. In these surveys, field interviewers observe and record characteristics such as the mix of residential and business units, the upkeep of the neighborhood, the wealth of the neighborhood, or the presence of trash and graffiti in the neighborhood surrounding the sampled housing units.

Neighborhood observations are theoretically available for all sampled persons, including noncontacted units, and are easily added to any in-person survey. However, ecological data for telephone, mail, or Web surveys may have to be obtained through other sources, such as merging census tract information to a sample frame (Campanelli, Sturgis, and Purdon 1997; Johnson et al. 2006; Kennickell 2005) or purchasing data from commercial sources (Schräpler, Schupp, and Wagner 2010). Field interviewers frequently visit a neighborhood before attempting initial contact with a sampled household (Morton-Williams 1993), and their observations are thus unobtrusive and require no interaction with the sampled unit. Not surprisingly, interviewers report little difficulty making these types of assessments (Lynn 2003).

Interviewers may fail to complete the observations altogether, of course (Matsuo, Billiet, and Loosveldt 2010); and different interviewers may perceive neighborhoods differently (Peterson, Sastry, and Pebley 2007; Raudenbush and Sampson 1999), which can lead to measurement error in the observational data. Item nonresponse rates for neighborhood observations vary greatly across surveys, ranging from under 5 percent (Casas-Cordero 2010; Lynn 2003) to more than 50 percent (Kreuter, Lemay, and Casas-Cordero 2007; Matsuo, Billiet, and Loosveldt 2010). One measure of the quality of neighborhood observations is the similarity in observations among different interviewers for the same neighborhood. Raudenbush and Sampson (1999, 7) found “modest” interobserver correlation coefficients ranging from 0.13 to 0.36 for a variety of scales created from neighborhood observations. Such measurement error naturally attenuates the association between the observations and survey outcomes of interest (Fuller 1987; Raudenbush and Sampson 1999). Attenuation can be reduced by increasing the number of neighborhood observers (Raudenbush and Sampson 1999), although using multiple raters quickly increases costs (Sastry and Pebley 2004).

Studies of “neighborhood effects” on social, economic, psychological, and health outcomes are becoming increasingly popular (Gee and Devon 2004; Leventhal and Brooks-Gunn 2003; Raudenbush and Sampson 1999), and recent reviews have shown that neighborhood circumstances affect a variety of outcomes, including crime-related behaviors, school achievement indicators, and, to a more limited degree, mental health measures (Leventhal and Brooks-Gunn 2000; Sampson, Morenoff, and Gannon-Rowley 2002). These findings suggest that neighborhood observations are particularly useful for making nonresponse adjustments in crime surveys or studies of educational achievement among children, as neighborhood observations are most useful for nonresponse adjustment when the association between observations and survey outcomes is large (R² ≥ .48). Such an effect size, however, is not often found (Diez Roux 2001), which may require the development of new neighborhood observations for purposes of nonresponse adjustment that focus on variables related to socioeconomic status (Leventhal and Brooks-Gunn 2000), alcohol or tobacco use, and the frequency of adults and children playing together (Sampson, Morenoff, and Gannon-Rowley 2002, 465–71).

Neighborhoods are hypothesized to affect survey cooperation through mechanisms such as urbanicity, population density, crime, and social disorganization (Groves and Couper 1998, 176–78). In the ESS, for example, observations about the presence of litter or vandalism in the neighborhood are significantly associated with survey participation (Kreuter, Lemay, and Casas-Cordero 2007) as well as respondent contactability and cooperation (Matsuo, Billiet, and Loosveldt 2010). The strength of these associations varies tremendously across countries, however. Interviewer observations about an area’s safety are also significantly associated with both contactability and cooperation in six surveys in the United Kingdom (Durrant and Steele 2009) and with participation in the main interview in the NSFG (Lepkowski, Mosher, et al. 2010). Although aggregate information on tracts or zip code areas is often used for nonresponse adjustment in both face-to-face and telephone surveys (Campanelli, Sturgis, and Purdon 1997; Johnson et al. 2006), paradata about the neighborhood gathered by interviewers or listers are infrequently used for adjustment purposes (Kreuter, Lemay, and Casas-Cordero 2007; see also review in Casas-Cordero 2010), even in surveys that use interviewers to capture neighborhood characteristics (Sastry and Pebley 2003).

The form of models used to estimate response propensities from neighborhood data has received little empirical attention in the survey methodological literature. While studies of ecological effects on social, economic, and health outcomes use hierarchical or random effects models, propensity models that draw on area observations often ignore potential clustering effects. Those studies that do use hierarchical models tend to look at area characteristics as predictors of participation, contactability, or cooperation, but do not use estimates for adjustment purposes (Callens and Croux 2003; Durrant and Steele 2009; O’Muircheartaigh and Campanelli 1999; Steel, Tranmer, and Holt 2003). Although neighborhood observations have been used in adjustment models, little attention has been paid to the nested nature of the observations and how models should account for geographic clustering.

Neighborhood observations can be collected relatively easily in face-to-face surveys, but they are virtually impossible to obtain (without incurring huge costs) in other survey modes. Neighborhood observations may be particularly effective as adjustment variables in studies about crime or educational achievement, but item nonresponse in the neighborhood observations and variability across observers potentially limits their usefulness. When multiple sampled units are in the same neighborhood, it may be necessary to account for clustering in propensity models, although this remains an empirical question.

Observations about the sampled housing unit

Observations about the sampled housing unit eliminate analytic issues related to clustering while still being fairly unobtrusive. Such observations typically include an assessment of the housing unit’s quality relative to others in the neighborhood, whether the unit is in a multiunit structure, if it has access impediments, and whether it is in a safe condition. These sorts of observations have been made on surveys as diverse as the American National Election Studies (ANES), the British Survey of Social Attitudes (BSSA), the ESS, the NSFG, the SCF, the BCS, the Residential Energy Consumption Survey (RECS), and the National Survey of Drug Use and Health (NSDUH) (see Groves and Heeringa 2006; Kennickell 2000; Kreuter et al. 2010; Lepkowski, Mosher, et al. 2010; Lynn 2003; Peytchev and Olson 2007; Sinibaldi 2008, 2010; U.S. Energy Information Administration 2009; Wang et al. 2005). Some surveys also make housing unit observations specific to the content of the questionnaire, such as observations about the presence of political lawn signs in the ANES (National Election Studies 2004) or the presence of bars on the windows or burglar alarms in the BCS (Lynn 2003). As with neighborhood observations, housing unit observations can be made in face-to-face surveys but not in telephone, mail, or Web surveys; and they can easily be made for all sampled units, including noncontacts and refusals, for relatively little additional marginal cost (Lynn 2003). Unlike area characteristics, however, proxies cannot be directly obtained from census data.

Since interviewers in this case serve as reporters, correlated interviewer variance (Kish 1962) and item-missing data on the housing unit observations (Kennickell 1999; Lynn 2003) is possible. In addition, missing data on some housing units can be a problem when they are located in a gated community or locked building and cannot be directly observed. Surprisingly few surveys that collect housing unit observations report missing data rates on these observations. Nonetheless, in the case of the 2004 ANES, observation of whether the sampled person lived in a single-family home was complete for all respondents and nonrespondents (Peytchev and Olson 2007). Similarly, item nonresponse rates for housing unit observations were less than 2 percent in the BCS (Lynn 2003). In contrast, observations of the same characteristic were missing for all sampled units in Norway on round one of the ESS, although item-missing data rates were lower in other countries and varied by type of housing unit observation (Kreuter, Lemay, and Casas-Cordero 2007). Little empirical evidence exists about correlated interviewer variance and other forms of measurement error.

Characteristics of housing units often predict both contactability and cooperation in sample surveys. Ubiquitous predictors of contactability in face-to-face surveys include whether the sampled unit is in a multiunit structure versus a single-family home, or is in a locked building or has other access impediments (Blohm, Hox, and Koch 2006; Campanelli, Sturgis, and Purdon 1997; Groves and Couper 1998; Kennickell 2003; Lynn 2003; Maitland, Casas-Cordero, and Kreuter 2009; Sinibaldi 2008; Stoop 2005). The condition of the housing unit relative to others in the area is also significantly associated with noncontact and refusal rates (Durrant and Steele 2009; Lynn 2003; Sinibaldi 2008). In the ESS and ANES, however, the correlation between being in a multiunit structure versus a single-family house and survey participation is small, explaining less than 4 percent of the variance. A two-stage approach to modeling survey participation, in which contact and cooperation are modeled separately, may yield stronger associations, especially in contact models (Groves and Couper 1995). Few other studies report the degree to which variation in response propensity is explained by housing unit characteristics alone.

Certain survey variables have been shown to be related to characteristics of the housing unit, such as socioeconomic status, energy use, technological sophistication, and exposure to environmental hazards. For example, single-family detached housing units are more likely to have occupants who own rather than rent the housing unit; those occupants also tend to have higher median incomes and higher housing costs than persons living in multiunit housing (U.S. Census Bureau 2008). Persons living in multiunit structures are more likely to be heated by electricity than natural gas, to use a personal computer, and to own more than one television (U.S. Energy Information Administration 2008). Housing unit characteristics also affect exposure to lead paint and environmental triggers for asthma (Jacobs et al. 2002; Saegert et al. 2003). In the ANES, the strongest associations for living in a single-family home and survey variables were observed for number of years living in the home (r = .29), age (r = .22), and putting off medical treatment (r = –.20), but political activity variables had much weaker associations (Peytchev and Olson 2007). In round one of the ESS, the correlation between housing unit observations and survey variables varied tremendously across eight survey variables and did not exceed an absolute value of r = .3 (Kreuter, Lemay, and Casas-Cordero 2007). Thus, housing unit characteristics predict many survey outcomes, although the strength of the association is often quite weak.

In sum, with sufficient monitoring, housing unit observations offer a potentially rich source of information for nonresponse adjustments. Unfortunately, observations about housing units can be made only for in-person surveys. Analytically, housing unit observations are available for each sampled unit and are free from concerns about neighborhood clustering, so that standard logistic regression methods and weighting class adjustments can be used. Although housing unit observations can be recorded for all sampled units, contacts and noncontacts, interviews and refusals, they often have modest predictive power for important survey variables. Although significantly associated with survey participation, housing unit observations tend to explain little variance in overall survey participation.

Observations about members of the housing unit

Potentially rich sources of paradata for nonresponse adjustment come from observations about the members inhabiting selected housing units. These observations fall into two categories: observations on demographic characteristics and observations that proxy for key survey variables. If measured without error, interviewer observations about characteristics of housing unit members will be identical to the respondent’s report of those same characteristics. Information from interviewer observations of key survey variables could be used directly to assess nonresponse bias for statistics computed from those characteristics. Since a strong association with the survey outcome variables is key to being a useful auxiliary variable (Kreuter and Olson 2011; Little and Vartivarian 2005), proxy observations of important survey variables meet that goal, at least in theory. An interviewer-observed Y variable would also be the most effective variable for creating weighting classes that are homogeneous on that characteristic of interest (Kalton 1983; Kalton and Flores-Cervantes 2003).

Demographic characteristics are ubiquitously measured on sample surveys and are often used in nonresponse adjustments because they provide information on individuals and subgroups whose representation in the final sample pool is important (Bethlehem 2002; Cobben and Schouten 2008; Groves 2006). The ESS, the GSS, the SCF, the BCS, the National Health Interview Survey (NHIS), and the National Health and Nutrition Examination Survey (NHANES) incorporate observations of age (Matsuo, Billiet, and Loosveldt 2010; Sinibaldi 2010), gender (Matsuo, Billiet, and Loosveldt 2010), race (Burns et al. 2001; Lynn 2003; Saperstein 2006; Smith 1997, 2001), income (Burns et al. 2001; Kennickell 2000), and the presence of non-English speakers (Bates, Dahlhamer, and Singer 2008; National Center for Health Statistics 2009) for both respondents and nonrespondents. Although many surveys use demographic characteristics reported in the survey questionnaire for poststratification, these variables are collected in the survey questionnaire and will not be considered here.

Interviewer observations of demographic characteristics are not free from item nonresponse or measurement errors, of course. Item nonresponse on observations of demographic characteristics can vary from virtually nonexistent to fairly high. For example, item nonresponse rates for observations about gender and age in the 2004 ESS range from no missing data in Finland to almost 65 percent missing in Germany (Matsuo, Billiet, and Loosveldt 2010). Interviewer observations of age in the United Kingdom’s National Survey of Sexual Attitudes and Lifestyles (NatSAL) were missing for 37 percent of households (Sinibaldi 2010). In the BCS, observations of race were missing on 76 percent of the cases, primarily on respondents (Lynn 2003).

Measurement error in observations of demographic characteristics has been investigated by comparing interviewer observations to reports from respondents themselves. These comparisons have focused largely on observations about household members’ race, in which interviewer observations match about 95 percent of white and black respondents’ reports of race, but only around 50 percent for respondents who identify with other races (Hahn, Truman, and Barker 1996; Saperstein 2006; Smith 1997, 2001). The match rate is also affected by the interviewer’s race (Hill 2002). Less is known about measurement error in observations of householder age and gender. In the NatSAL, about 76 percent of the observations of age matched the reported age within 10-year age groups (Sinibaldi 2010). Although interviewers are frequently instructed to observe respondent gender, surprisingly little research has been conducted on the accuracy of these assessments (Kenney McCulloch, Kreuter, and Calvano 2010).

Demographic characteristics have frequently been examined as predictors of survey participation. Lower response rates have been found among the young and the elderly, men, households of lower and higher socioeconomic status, racial-ethnic-language minorities, immigrants, and persons with lower levels of education (Brehm 1993; Callens and Croux 2003; Campanelli, Sturgis, and Purdon 1997; Collins et al. 2000; Goyder 1987; Goyder, Warriner, and Miller 2002; Green 1996; Groves and Couper 1998; Holbrook, Krosnick, and Pfent 2008; Kennickell 1999; Korinek, Mistiaen, and Ravallion 2005, 2007; Moonesinghe, Mitchell, and Pasquini 1995; Purdie et al. 2002; Smith 1984; Stoop 2005; Voogt 2004). Although frequently examined, these demographic characteristics explain little of the variation in response propensity (Kreuter and Olson 2011; Kreuter et al. 2010). Additionally, a recent meta-analysis revealed that a difference between respondents and nonrespondents on demographic variables may not necessarily indicate nonresponse bias on key survey estimates (Peytcheva and Groves 2009).

A growing set of observations about members of the housing unit come from interviewer-recorded proxy measures of characteristics that form key survey estimates. For example, on the NSFG, interviewers are asked to report whether the selected person is in an active sexual relationship and whether there are children in the household (Kreuter et al. 2010; West 2010). At the National Centre for Social Research in the United Kingdom, interviewers in the NatSAL dress rehearsal were asked to record whether there was evidence of smoking in the household, as well as the two characteristics observed in the NSFG (Sinibaldi 2010). Related to these characteristics is whether a contacted household refuses because of a topic-related issue, such as lack of interest in politics in the ANES (Peytchev and Olson 2007), or health-related reasons in the NHIS (see Dahlhamer and Simile 2009).

As the observation of proxies of important survey variables is quite new, very limited research has been conducted on the quality of these auxiliary variables. However, one measure of quality is item nonresponse. Item nonresponse on these observed characteristics can arise from multiple sources. First, housing units that have never been contacted will not have an observation about members of the sample unit or a selected respondent; these observations are available only for contacted households. Second, interviewers may choose not to complete their observation forms owing to perceptions that the observation is superfluous or irrelevant (Lynn 2003) or because of discomfort about making the observations (Sinibaldi 2010). Third, question wording and computerization of the observation instrument may encourage item nonresponse in the observations. The NSFG and NatSAL, for example, measure identical characteristics about sampled households (presence of children and sexual relationships), but the NSFG had no item nonresponse, whereas in the NatSAL item nonresponse rates ranged from about 10 percent to almost 45 percent, although some of these item-missing cases can be attributed to noncontacts (Sinibaldi 2010). On the NSFG survey, interviewers were required to provide a valid answer to these questions; ¹ on the NatSAL, interviewers were permitted to record that a characteristic was not observed, that they never met the respondent, or that they “can’t say” what the answer would be.

Another measure of quality is the association between an interviewer’s observation and the respondent’s report of the survey variable. Evaluations of observations that are proxies of important survey characteristics show significant associations with the survey characteristics they were intended to measure (Groves, Wagner, and Peytcheva 2007; Kreuter et al. 2010; Sinibaldi 2010; West 2010). The associations, however, are far from perfect (Groves, Wagner, and Peytcheva 2007; Kreuter et al. 2010; Sinibaldi 2010; West 2010). In the NSFG, for example, 79 percent of interviewer guesses about whether a selected female was in a sexually active relationship matched reports obtained through the interview, with “yes” guesses (the selected respondent was assessed as being in a relationship) being better than “no” guesses (the selected respondent was assessed as not being in a relationship).

Householder statements of not being interested in politics were significantly, but modestly, associated with being interested in following campaigns, voting intentions, and caring about who will win the elections (Peytchev and Olson 2007). These assessments are limited because information on these characteristics is not known for the nonrespondents; whether the interviewer guesses for nonrespondents are equivalent in quality cannot be assessed through this method. Additionally, interviewers can vary substantially in the quality of their observations, leading to an increased variance in these observations and potentially limiting the usefulness of these characteristics (West 2010). What makes certain interviewers better at recording these observations than others is unknown.

The novelty of these observations of proxy survey variables means that little is known about their performance as nonresponse adjustment variables. In the NSFG, observations of sexual activity and having children in the household are significantly associated with survey participation (Groves, Wagner, and Peytcheva 2007; West 2010). However, these extra variables explain only 0.6 percent additional variance when added to existing propensity models (West 2010). In the ANES, observations about not being interested in politics were made by only 5 percent of the contacted households and were only weakly correlated (r = –.08) with survey participation (Peytchev and Olson 2007). Thus, these variables may not be effective for reducing nonresponse bias in survey estimates (Little and Vartivarian 2005).

Multiple implementation challenges arise for the observation of either demographic characteristics or observations of proxy survey variables. To be useful for nonresponse adjustment, the measurement error properties of these characteristics should be the same for respondents and nonrespondents (Groves and Couper 1995). Thus, recording the observations must occur before an interview is conducted as the respondent’s answers may influence the interviewer’s observation. Practically, the number of observations of important survey variables that can be made is limited to minimize disrupting the flow of the interview. In a multipurpose survey with dozens of key survey variables, only a few variables can be observed. Additionally, to maintain a constant measurement environment, interviewers’ ability to change the observation after conducting the interview must be restricted. This restriction is much easier to accomplish in a computer-assisted interview than in a paper-and-pencil interview.

Another implementation challenge arises due to the need to train interviewers on the task of completing these observations (Groves et al. 2009; Lepkowski, Axinn, et al. 2010). This training is auxiliary to the task of contacting a case, obtaining cooperation, and administering a survey interview. Proxy observations are directly counter to the task of standardized survey interviewing in which all questions must be asked as written. As such, many interviewers and field managers resist including proxy observations of survey variables, especially for sensitive characteristics (Sinibaldi 2010).

Analytically, observations of either demographic characteristics or proxy survey variables can be made on contacted units, but may be quite difficult to make for noncontacted units. This means that item-missing data on interviewer observations of the sampled person for noncontacts needs to be imputed or a two-stage adjustment method used in which the predictors for contacting a case are different from those for obtaining cooperation (Groves and Couper 1995). Additionally, evidence of interviewer variance and interviewer bias has been found in observations of members of sampled housing units. How these measurement errors affect adjustments and potential fixes require more research. Nevertheless, as a potentially strong predictor of important survey variables, thus fulfilling one of the key criteria for a useful auxiliary variable, interviewer observations on characteristics of housing unit members have great promise for paradata for nonresponse.

Information from call records

Call record data are the traditional form of “process data” related to survey participation, sometimes called “level of effort” data (Biemer 2009). Call records consist of information recorded in sample management systems about the attempts made to a sampled unit as part of the recruitment process (Couper 1998; Hansen 2008; Heerwegh 2002; Kreuter and Casas-Cordero 2010; Laflamme 2008a). These data typically include a record for each recruitment attempt made to a sampled case (sometimes labeled a call), the time of the attempt, the date of the attempt, identification codes for the interviewer (in interviewer-administered surveys), and attempt-level result codes. Although these data are collected for a survey conducted at one point in time, call record data typically are not rectangular and have a structure that resembles longitudinal data (Singer and Willett 2003). That is, sampled cases with different numbers of call attempts have different numbers of observations. A case with only two attempts, for example, will have two observations, whereas a case with twenty attempts will have twenty observations. Call record data may also indicate other call-level events, such as the mode of the attempt (telephone, in-person, Web, or mail), whether an answering machine message was left (in telephone surveys), whether the interviewer left a “sorry I missed you” card (in face-to-face surveys), or whether the case was offered incentives, among other information (Chearo and Van Haitsma 2010; Stussman, Dahlhamer, and Simile 2005).

Although call record data were traditionally recorded through paper-and-pencil forms (Bates 2003), the growth of computerized sample management systems and the usefulness of call record data for sample management purposes (Lepkowski, Axinn, et al. 2010) have made such data routinely available in most surveys (Couper 1998; Groves and Couper 1998; Groves and Heeringa 2006; Hansen 2008; Scheuren 2000; Stoop 2005; Taylor 2008). Information about recruitment attempts and their outcomes can be kept for any survey mode, whether interviewer-administered or self-administered. What data are recorded in call records varies tremendously across organizations (Couper 1998), however, leading some to appeal for call records to be standardized (Chearo and Van Haitsma 2010).

The measurement properties of call record data are largely unstudied. Call record data are collected for the purposes of sample management (Hansen 2008), not analysis, and, as such, may be filled with measurement errors (Couper 1998). Secondary analyses of call record data have encountered duplicate call attempts, missing call attempts, call attempts recorded at improbable times (e.g., 2:00 a.m.), and inconsistencies in outcome codes (e.g., a final outcome of “never contacted” when call attempts with outcomes of refusal were listed for the case). A recent survey of interviewers at a large research organization found that at least 15 percent report making some kind of error in call records, such as recording calls for the incorrect case or failing to record “drive-by” attempts (Wang and Biemer 2010). The same study also found that interviewers are more likely to underreport calls than overreport them (Wang and Biemer 2010). How often this happens and why it happens require future empirical investigation.

The main purpose of keeping call record data is to monitor response rates so as to understand how decisions made in the field affect contact and cooperation and to implement changes and decisions in the course of data collection. It has been found, for example, that the largest changes in response rates happen during the first few contact attempts, with diminishing returns over the course of the field period (Bates 2003; Jocelyn et al. 2009; Laflamme 2008a). Likewise, it has also been found that first and second calls made during weekday evenings and on weekends are more positively associated with contact propensity compared with calls made during the day on weekdays, in both telephone and face-to-face studies (Bates 2003; Groves and Couper 1998; Hoagland, Warde, and Payton 1988; Laflamme 2008b; Odom and Kalsbeek 1999; Weeks et al. 1980; Weeks, Kulka, and Pierson 1987).

Electronic call records also facilitate the implementation of responsive survey designs (Groves and Heeringa 2006; Laflamme 2008a, 2008b; Laflamme, Maydan, and Miller 2008; Lepkowski, Axinn, et al. 2010; Sinibaldi 2008; Wagner 2010). A critical assumption in responsive designs is that response propensities are changeable or dynamic (Olson and Groves 2012). In responsive designs, propensities change when new design features are targeted toward sample units who are less likely to participate, such as members of underrepresented demographic subgroups. Call record data feature prominently in monitoring these potential changes in nonresponse rates and nonresponse bias in key survey estimates over the course of the data collection field period (Tangdahl 2004, 2005, 2006; Wagner 2010). They also figure importantly when making decisions about which cases to subsample (Elliott, Little, and Lewitzky 2000).

Use of call record data for purposes of nonresponse adjustment requires aggregating information across all calls made to a case or using different modeling approaches (Couper 1998; Heerwegh 2002; Kreuter and Casas-Cordero 2010; Kreuter and Kohler 2009; Olson 2006). At this time, there is no single agreed on method for how to incorporate call record data into nonresponse adjustments. Almost all of the adjustment methods that use call record data draw on one of two theoretical models—either a “continuum of resistance model” or a “classes model” (Lin and Schaeffer 1995).

The continuum of resistance model hypothesizes that the number of call attempts to obtain a completed interview is linearly related to a sampled person’s unwillingness to participate in an interview and is also linearly related to important survey characteristics. When this model holds, the “last” or “most difficult” respondents can be used to represent information that would have been obtained from nonrespondents (Bates and Creighton 2000; Filion 1975; Fitzgerald and Fuller 1982; Lahaut et al. 2003; Lin and Schaeffer 1995).

The classes model, in contrast, uses both the number of call attempts and call outcomes to categorize respondents and nonrespondents into groups of easy and hard refusals and easy and hard noncontacts (Lin and Schaeffer 1995). Unlike the strength of the relationship between contact attempts and survey participation, very little variance in important survey variables is explained by the number of call attempts (Kreuter and Kohler 2009; Lin and Schaeffer 1995). Furthermore, a wide literature examining differences between “late” and “early” and “easy” versus “difficult” respondents shows that some estimates differ for these two groups while others do not (see reviews in Groves 2006; Groves and Peytcheva 2008).

One of the first approaches to using information from call attempts for nonresponse adjustment, the Politz-Simmons method, showed that reports of the number of nights that a respondent had been at home during the past week could be used to account for the “not at homes” (Kalton 1983; Politz and Simmons 1949). Limitations in the Politz-Simmons method include potential misreporting by respondents and failure to meet the model’s assumptions in practical situations (Ward, Russick, and Rudelius 1985); these have led to the development of alternative statistical models based on the number of calls made to a sampled person, sometimes incorporating characteristics of the sampled unit.

One approach that uses call record information in propensity models simply includes the number of call attempts made to a case as a predictor in a logistic regression propensity model (Little 1982, 1986). The number of call attempts is clearly related to survey participation; more resistant or difficult-to-contact respondents generally receive more call attempts (Blom 2009; Campanelli, Sturgis, and Purdon 1997; Groves and Couper 1998; Lin, Schaeffer, and Seltzer 1999; Olson 2006). In fact, when added to models predicting survey participation, information from the call records tends to explain more variability in participation than any other predictor (Beaumont 2005; Dahlhamer and Simile 2009).

An alternative approach identifies patterns of call outcomes made to different cases. An example of these patterns is the proportion of noncontacts out of all calls made, rather than simply number of call attempts (Kreuter and Kohler 2009). Concerns about endogeneity of the number of contact attempts with the outcome of survey participation and “censoring” for cases that are never contacted or interviewed have led to the use of discrete time hazard models that change the outcome to the conditional probability of an interview on a given call, given no contact or participation on prior calls (Durrant and Steele 2009; Groves and Heeringa 2006; Kennickell 1999; Olson and Groves 2009).

Other probability-based models have been used to estimate response probabilities at each call attempt as a function of respondent characteristics, sometimes permitting a “hard core” nonresponding group (Alho 1990; Anido and Valdés 2000; Colombo 1992; Drew and Fuller 1980; Potthoff, Manton, and Woodbury 1993; Wood, White, and Hotopf 2006). A recent expansion of the callback models uses latent class models, characteristics of respondents and nonrespondents, and reports from the survey to create weights based on the level of effort exerted to the case (Biemer 2009; Biemer and Wang 2007; Biemer and Link 2006). There is little consensus about how these models can be incorporated into adjustment, however.

Overall, although call record data are consistent with the definition of paradata, they are the most problematic for purposes of nonresponse adjustment. With multiple observations per sampled unit, concerns about causality, and the need for complicated data reduction or transformation decisions, information contained in call records is not necessarily easily incorporated into nonresponse adjustments. Yet call records are invaluable for monitoring progress during data collection and for understanding how design decisions affect contact and cooperation. Recent research into the use of call record data, however, has shown promising possibilities for nonresponse adjustments for univariate estimates and for potentially incorporating information about design decisions into adjustments.

Observations about the interviewer-respondent interaction

Survey researchers have long been interested in the reasons that sample units choose not to participate in surveys (Deming 1953; Morton-Williams 1993). Systematic collection of these reasons, sometimes called “doorstep statements,” has become widespread in computerized sample management systems. Typical comments given include statements such as “I’m not interested” and “I’m too busy”; or questions asked about the survey itself, such as “What is this all about?” Although not as common as call records themselves, many surveys, including the NHIS, the Consumer Expenditure Interview Survey (CEIS), the NSFG, the British Household Panel Study (BHPS), the National Survey of Health and Stress (NSHS), and the ANES record householder questions or statements to the interviewer at the time of recruitment (Bates, Dahlhamer, and Singer 2008; Campanelli, Sturgis, and Purdon 1997; Couper 1997; Groves and Couper 1998; Groves et al. 2009; Peytchev and Olson 2007; Safir and Tan 2009; Taylor 2008). An alternative type of contact observation is the interviewer’s subjective assessment of the householder’s reluctance or willingness to participate on future contacts (Campanelli, Sturgis, and Purdon 1997; Stoop 2005). These contact observations may be recorded on the first contact with the sampled household or on every contact with the household. Contact observations can be recorded by interviewers in face-to-face or telephone surveys but are not possible in self-administered surveys.

Contact observations are usually recorded in closed ended or “check all that apply” questions answered by interviewers when the interaction with a selected householder is finished. In general, little is known about the measurement properties of these interviewer observations. As with all the observations, interviewers may choose not to complete the observations at all, leading to item-missing data. Evaluating the accuracy of the recorded information requires audio recordings, an expensive and rare procedure (Couper and Groves 2002; Morton-Williams 1993). Campanelli, Sturgis, and Purdon (1997) compared taped interviewer-householder interactions to record of the interaction and found low concordance between the two sources. Between 31 and 68 percent of the statements agreed between the two sources, depending on how narrowly the statements were categorized. Both the interviewer observation and the coded tape predicted participation, although the strength of the relationship differed across the two sources. The contact observations as recorded by interviewers, then, may be picking up general impressions of the householder from nonverbal cues as well as verbal statements.

Complications arise when using contact observations as nonresponse adjustment variables. First, similar to observations of demographic or other characteristics of householders, contact observations are available only for contacted households and missing for noncontacted households. Adjustment models thus either need to impute contact observations for noncontacted households or separately model contact and cooperation when creating adjustment weights (Groves and Couper 1995).

Second, although detailed statements collected in contact observations may be useful for understanding heterogeneity across householders in their recruitment concerns, each individual statement may be uttered infrequently, leading to the need to aggregate across multiple statements. For example, time delay statements such as “I’m too busy,” “I don’t have time right now,” or “I need time to think about it” may be aggregated into an overall “Time delay statements” indicator (Campanelli, Sturgis, and Purdon 1997; Groves and Couper 1998; Maitland, Casas-Cordero, and Kreuter 2009).

Third, some households may have only one contact with a contact observation recorded, usually for the attempt on which an interview is obtained. This leads to endogeneity concerns when predicting survey participation. Last, other households may have multiple contacts with the interviewer, with a different observation made at each contact, requiring simplification such as taking the first or last contact observation or aggregation across multiple contacts (Henly and Bates 2006; Lepkowski, Mosher, et al. 2010; Lepkowski et al. 2006; Maitland, Casas-Cordero, and Kreuter 2009; Moon, Rose, and Steel 2005).

Contact observations are nonetheless strong predictors of survey participation. In general, householders who ask questions tend to be more likely to ultimately participate in the survey, whereas statements such as “I’m not interested” or “I’m too busy” and interviewer predictions of unwillingness tend to be negatively associated with participation (Campanelli, Sturgis, and Purdon 1997; Couper and Groves 2002; Dahlhamer and Simile 2009; Groves and Couper 1998; Maitland, Casas-Cordero, and Kreuter 2009; Morton-Williams 1993; Olson et al. 2006).

This pattern is not surprising, as contact observations are designed to be predictors of survey participation, not of survey variables themselves (Groves and Couper 1998). Although there are demographic differences in the characteristics of household members who make different types of statements (Campanelli, Sturgis, and Purdon 1997; Couper 1997; Groves and Couper 1998), contact observations of the interviewer-householder interaction tend to be weak predictors of key survey variables (Maitland, Casas-Cordero, and Kreuter 2009; Peytchev and Olson 2007). For example, across items in the National Health Interview Survey, Maitland and colleagues (2009) found that the average correlations between contact observations and the survey variables were r = .02 to .03, with maximum correlations no greater than r = .12. Similar magnitudes were found by Peytchev and Olson (2007) in the ANES. Thus, contact observations may be limited on a key criterion for being a good auxiliary variable.

In general, contact observations of statements made by the householder about why he or she may or may not want to participate in a sample survey request, not surprisingly, strongly predict participation. Thus, they meet one criterion for being a useful auxiliary variable for nonresponse adjustment, but limitations arise because they are not strong predictors of survey variables and are not measured for all sampled survey units. Additionally, as with call records, analytic decisions have to be made about simplifying, summarizing, or aggregating observations recorded at multiple contacts.