Abstract
Recording interviews is a key feature of quality control protocols for most survey organizations. We examine the effects on interview length and data quality of a new protocol adopted by a national panel study. The protocol recorded a randomly chosen one-third of all interviews digitally, although all respondents were asked for permission to record their interview and interviewers were blind as to whether or not interviews were recorded. We find that the recording software slowed the interview slightly. Interviewer knowledge that the interview may be recorded improved data quality, but this knowledge also increased the length of the interview. Interviewers with higher education and performance ratings were less reactive to the new recording protocol. Survey managers may face a trade-off between higher data quality and longer interviews when determining recording protocols.
Introduction
The panel study of income dynamics (PSID) is a longitudinal study of a nationally representative sample of U.S. families that has collected information on economic and social behavior for nearly five decades. Since the study’s inception, interviewers employed by the Survey Research Center (SRC) at University of Michigan have collected data via interview with one respondent in each family. The mode of data collection is computer-assisted telephone interview for 98.5% of the sample and in-person interviews for the remaining families (see McGonagle et al. [2012] for more information).
As part of the management of the field effort and consistent with survey industry standards (e.g., Couper et al. 1992; Nebeker and Tatum 1993), SRC monitors interviewers in their conduct of administering interviews to guard against falsification and to provide feedback so as to collect high-quality data. After obtaining consent from the respondent, a portion of all interviews taken by each interviewer is recorded, and these recordings are then evaluated by supervisors on several dimensions, including reading each question verbatim and at an appropriate pace and conducting follow-up probes for some questions.
Until 2011, interviewers manually recorded interviews and had a role in choosing which cases to record. Beginning in 2011, a new quality control protocol was implemented that used digital recording software that randomly selected the cases eligible for recording, leaving the interviewer blind as to whether or not an interview was being recorded, except when a respondent refused to provide consent for recording. Routine project analyses conducted during data collection revealed that interviews in 2011 were substantially longer than expected. In this study, we examine whether components of the new quality control protocol had the unintended consequence of increasing interview length as well as the intended positive effect of increasing data quality.
New Quality Control Protocol and Potential Impact
The traditional protocol for interview verification and quality control used by the PSID required interviewers to manually tape-record a set of interviews. For example, in 2009, interviewers were instructed to record the first five interviews and every 10th interview thereafter. A key disadvantage of this approach was that it was not possible to determine whether the interviewer correctly counted the number of interviews they had already administered or whether difficult cases were deliberately skipped from recording. If interviewers overrode the selection criteria and purposely skipped recording respondents who were expected to be reluctant, guarded, or hostile, then the evaluative feedback might miss areas of needed improvement. Additionally, because interviewers knew when they were and were not being taped, adherence to training protocols might be higher during the taped sessions. These limitations, as well as the resource intensive nature of using cassette tapes and their resulting poor quality, yielded a small number of taped interviews (7% in 2009), providing few cases for evaluation.
In 2011, a new protocol for verification and quality control was implemented using digital recording software to capture the verbal interaction between respondent and interviewer. As others have described, the advantages of digital recordings are numerous (e.g., Biemer et al. 2000; Thissen and Rodriguez 2004; Thissen et al. 2008). Using a random selection algorithm, the software selects the case for recording; if the respondent provides consent, the recording occurs without any action by the interviewer. This eliminates the potential for interviewers to make errors and/or override the selection criteria of cases for recording.
During the 2011 wave, a subset of the recordings was transferred to supervisors who reviewed a portion for each interviewer and provided an overall performance score based on whether the interviewer read each question as scripted, read all response options, used a clear, moderately paced tone, used standard follow-up question probing, and conducted himself or herself professionally. Scores were communicated to team leaders who in turn provided corrective feedback, if needed, to interviewers.
The new approach for recording interviews represented a significant change in a key data collection protocol that had been in place for many waves. Instead of having interviewers ask a small number of respondents of their choosing for permission to record their interview, the 2011 protocol had them request permission from all respondents, with the software choosing which cases were actually recorded. Thus, unless the respondent did not provide consent, the interviewer was blind to whether or not a particular interview was being recorded. The software selected the first five interviews for each interviewer and then randomly selected one-third of subsequent interviews for recording. About half of the survey instrument was recorded, emphasizing new or challenging content. Approximately 6% of all respondents providing an interview refused to allow their interview to be recorded, which is comparable to other studies reporting consent rates (Basson 2005; Hicks et al. 2010; Wrenn-Yorker and Thissen 2005). Therefore, interviewers were blind to whether a particular interview was being recorded for 94% of their cases. This was in dramatic contrast to the procedure in the prior wave, where for all cases, the interviewer knew whether or not the interview was being recorded and 93% of all cases were, in fact, not recorded.
During 2011, 110 field interviewers, who typically conduct interviews from their residences across the United States, received three and a half days of training in administering the PSID interview. The number of interviews completed by each interviewer ranged from 1 to 171, with an average of 62 completed interviews.
The new quality control protocol is hypothesized to have increased interview length through two mechanisms. First, the recording software may have a mechanical effect that slows the flow of the interview. This could occur as a consequence of adding the recording software to the laptop, which also includes software to run the questionnaire and the sample management system, requiring substantial computer memory. The recording software itself, or its interaction with the questionnaire or sample management software, may slow the overall processing of the instrument. Over the course of an interview that averaged about 88 minutes, the slowdown may have been imperceptible to interviewers but yielded an overall nontrivial increase in length.
Another possibility is that interviewers changed their behavior in response to the new protocol. Specifically, interviewers who believed they were being recorded—and therefore evaluated—may adhere more closely to training guidelines that emphasize the diligent use of required follow-up question probing, reading each question in full and exactly as scripted, and speaking more slowly than usual conversation. This possibility is consistent with theoretical frameworks based on social facilitation theory (Zajonc 1965) and research finding effects of employee monitoring on performance (e.g., Aiello and Douthitt 2001; Bhave 2013; Stanton 2000). While greater adherence to training guidelines might lengthen the interview, they were purposely created to enhance the quality of the data. Thus, we also examine the effects of being recorded on data quality as represented by item nonresponse.
All analyses include only cases providing interviews in both the current and prior wave in order to include characteristics of respondents and their prior wave interview length in the multivariate models. We also remove the first five interviews obtained by each interviewer because these were all recorded for quality control purposes. Finally, we exclude any interviews obtained in person (<1.5%) or tape-recorded in 2009 (7%) to abstract from the impact of this experience on interview length in 2011.
Effect on Interview Length
Digital Recording Software
We first examine whether the digital recording software itself slows the interview. One-third of the cases were randomly assigned to be digitally recorded. Therefore, an unbiased estimate of the effects of the recording software is the simple difference in average interview length between the cases that were recorded and those that were not recorded.
This analysis is restricted to the 94% of cases (N = 6,314) that provided consent to be recorded because recording cannot be conducted without consent. The recorded cases have an average interview length that is statistically significantly longer than cases that were not recorded, but the differential is modest at 3.5% (Table 1).
2011 Average Interview Length (Minutes) and Item Missing Data Rate among Cases Providing Consent.
Note: N = 6,319.
***p < .0001.
Interviewer Knowledge that the Interview May Be Recorded
The second possibility we examine is that the interviewers’ knowledge that the interview may be recorded alters their behavior in a manner that lengthens the interview. Since all respondents are asked to give consent for recording, nonexperimental approaches for identifying causal effects are used. A series of four regression models of interview length are estimated.
The modeling approach adjusts for the potential clustering of observations by interviewers, allowing for random differences between interviewers that affect interview length but that are not correlated with the explanatory variables. In the first model, we simply compare average interview length for respondents who provided consent for recording with respondents who declined consent for recording. A limitation of this approach is that there may be systematic differences between respondents who provide consent and those who do not, and these differences may be correlated with interview length. For example, impatient or guarded respondents may be less inclined to provide consent and may also give brief answers that reduce interview length. Moreover, the simple mean difference may also be biased because interviewers may interpret a respondent’s refusal to be recorded as a lack of cooperation, which could result in nonresponse and may consequently conduct the interview as quickly as possible to ensure its completion. More generally, it is well documented that there are systematic differences between interviewers in a variety of behaviors (e.g., Loosveldt and Beullens 2013; Olson and Peytchev 2007; Pickery and Loosveldt 2000, 2004) that influence survey response, including their ability to obtain consent (Sala et al. 2012). If these differences are also correlated with the length of the interview, ignoring interviewer effects would induce a bias in the estimates of the effects of consenting on interview length.
To address these concerns, we estimated two additional models. First, we control for prior wave interview length. To the extent that families have persistently long (or short) interviews, adjusting for prior wave length would capture important variation that may be correlated with consent status. 1 We also control for observable characteristics of the families that are known to be associated with interview length (Andreski and Schoeni 2011). The PSID was initially drawn from two independent samples, a nationally representative sample (i.e., the SRC sample) and an oversample of low-income and African American families (i.e., the Survey of Economic Opportunity [SEO] sample). We control for sample membership by including an indicator variable for being part of the SEO sample (1) or the SRC sample (0). Indicator variables are also included to adjust for changes in family composition between waves, and the household head being younger than the sample average, having more than one job, at least some college education, being married, having children, and having total family income above the sample median.
Additionally, we add controls for characteristics of the current wave field effort known to be associated with interview length (e.g., McGonagle 2013) and that may differ between respondents as a function of providing consent, including the number of calls to finalize the interview, whether the case was assigned for tracking because it was difficult to locate, using a cell phone, and needing multiple sessions to complete the interview.
Second, to adjust for differences between interviewers on a variety of characteristics including their potential variation in obtaining consent from respondents, a subsequent model controls for interviewer fixed effects by adding separate dummy variables for each interviewer. This approach effectively controls for all characteristics of interviewers that are correlated with interview length, even if these unmeasured characteristics happen to be correlated with the other explanatory variables. The estimate of the effects of consenting on interview length from this model provides our preferred estimate because this model controls for respondent characteristics, characteristics of the field effort, and interviewer fixed effects.
A final objective is to test whether the effects of consenting differ by particular characteristics of interviewers. All else being equal, the greater the resources and skills that interviewers bring to their role, the less affected they should be by the possibility of being recorded. Thus, we augment our preferred model with interactions between indicators (yes = 1, no = 0) for whether or not the respondent consented for recording and three interviewer characteristics: having prior experience as an SRC interviewer, but no experience conducting a PSID interview prior to 2011; completion of a college degree; and receiving the highest performance rating at the end of the field period by a supervisor.
Average interview length, item missing data rate, and descriptive statistics of all explanatory factors are reported in Table 2 separately, according to whether consent for recording was provided. The average length of the 2011 interview was 9.4 minutes longer for families who provided consent—regardless of whether they were actually recorded—representing a statistically significant difference of 11.9%. However, interview length in 2009 did not diverge significantly between these two groups: 73.3 minutes for those refusing consent in 2011 versus 72.2 minutes for those who did not. Table 2 also displays statistically significant differences between the two groups on several socioeconomic and demographic characteristics. Those refusing consent were more likely to be SEO sample members, older and not married, and with lower educational attainment and income.
Descriptive Characteristics by Consent Status.
Note: N = 6,778.
***p ≤ .0001.
**p ≤ .01.
*p ≤ .05.
Table 3 reports results from the multivariate regression analysis where the dependent variable is 2011 interview length (minutes). Model 1 includes the single explanatory variable of whether the respondent agreed to have their interview recorded in 2011. The intercept represents the average interview length for respondents who refused consent, which is 78.70 minutes. Respondents who provided consent for recording had interviews lasting 9.40 minutes longer, on average, or 88.10 minutes in total. These estimates re-create the average interview length by consent status reported in the first row of Table 2.
Multivariate Models of 2011 Interview Length.
Note: N = 6,778. aAdds dummy variables for each of the interviewer identification variables, yes = 1 and no = 0.
***p ≤ .001.
**p ≤ .01.
*p ≤ .05.
Not surprisingly, respondents with longer interviews in 2009 had longer interviews in 2011. The coefficient estimate implies that 2009 interviews that are 1 minute longer are 0.42 minutes longer in 2011 (model 2). Many of the family socioeconomic and demographic characteristics have large and statistically significant effects on interview length. Interviews are longer for families who have children, a married household head, income above the median, and a female respondent. Furthermore, several characteristics of the 2011 field effort are strongly correlated with interview length, including interviews conducted on cell phones and over multiple sessions. Adjusting for prior wave interview length and family and field characteristics reduces the difference in interview length between consenting and nonconsenting respondents to 7.68 minutes.
Model 3 adds dummy variables for each interviewer to control for variability in characteristics such as their differing propensities in obtaining consent. Adjusting for interviewer fixed effects reduces the difference in interview length between consenting and nonconsenting respondents to 5.94 minutes. Finally, model 4 adds interaction terms between consenting status and interviewer characteristics. The coefficient estimate on the indicator for consenting represents the effect of consenting among interviewers who are not new hires, not college educated, and who were not among the highest performers during the 2011 field period. For this group, consenting is estimated to increase interview length by 10.50 minutes. Interviewers with a college degree (interaction effect of −6.34) and those with the highest performance ratings (interaction effect of −8.47) are significantly less reactive to whether or not consent for recording is provided.
Effects on Data Quality
If interviewers are, in fact, conforming more closely to their training because they believe that all of their consented interviews have a positive probability of being recorded, adherence to training protocols such as diligently probing respondents to report valid responses, thereby reducing item missing data rates, may be occurring. Such rates have historically been very low in the PSID, at 1.2% in 2009 and 0.8% in 2011 across all 2,325 items collected in both years. Therefore, for the majority of variables, even a substantial reduction between waves in item nonresponse would change the overall number of cases without missing data by a small amount.
Regardless, some important variables do have nontrivial missing data rates. Thus, we examined the effects of providing consent on item nonresponse for the 25 items with the highest rates of “don’t know” responses that were asked of at least 500 respondents in each wave. These variables include complex items involving mental calculations and/or recall over a period of two or more years, such as questions about expenditures, investment income, and pensions. Our measure of the extent of item missing data for each respondent is the proportion of these 25 items that had nonresponse. For this outcome, we follow the approach used to examine interview length in the preceding section. First, we examine whether the digital recording software is independently related to data quality by comparing the item missing data rates in 2011 of consented interviews that were actually recorded with those that were not. Second, we estimate unadjusted differences in the item missing data rates in 2011 between cases that did and did not provide consent in 2011. We then estimate these differences after accounting for socioeconomic and demographic factors, item missing data rates in 2009, field effort characteristics, and interviewer fixed effects. Furthermore, we test whether the effect of consenting varies by key interviewer characteristics.
Digital Recording Software
As shown in Table 1, while the digital recording software was related to a modest increase in interview length, there is no evidence that the software affected data quality. Among consented cases, the item nonresponse rate was 8.8%, regardless of whether or not the interview was actually recorded.
Interviewer Knowledge that the Interview May Be Recorded
As shown in Table 2, the 2011 rate of item missing data is substantially and statistically significantly lower among cases providing consent. That is, on average, among cases providing consent, 8.8% of these 25 items experienced item nonresponse compared to a much higher 13.8% item nonresponse among those refusing consent. However, there is no evidence that those refusing consent in 2011 have had high rates of item missing data historically. Specifically, there is no statistically significant difference on the item missing data rate in 2009 between those who did and did not provide consent in 2011.
Table 4 reports estimates from the multivariate models. Model 1 demonstrates that the rate of item missing data is 0.138 for those declining consent (i.e., the intercept), while the rate is 0.050 percentage points lower for those providing consent. Estimates from model 2 imply that rates of item nonresponse are higher for younger household heads and female respondents. Interviews with higher rates of item nonresponse in 2009 have much higher rates in 2011, and all four variables representing field effort significantly influence rates of missing data in 2011. However, the effects of providing consent on the item nonresponse rate change only slightly when these factors are included, to 0.047. Including interviewer fixed effects (model 3) has very little influence on the estimated effect of providing consent, reducing it to 0.041. None of the interaction terms for interviewer characteristics are significant. In sum, even after accounting for an extensive set of factors associated with the difficulty of the field effort, item nonresponse in the prior wave, family socioeconomic characteristics, and interviewer fixed effects, item nonresponse rates remain roughly 0.041 lower for those providing consent for recording, which translates into a 30% reduction given that the item missing data rate is 0.138 when consent is declined.
Multivariate Models of 2011 Item Missing Data Rate.a
Note: N = 6,778. aOutcome variable is calculated as “count of don’t know responses” divided by “count of all responses”; bAdds dummy variables for each of the interviewer identification variables, yes = 1 and no = 0; cParameter estimate has been multiplied by 1,000 to portray as nonzero.
***p ≤ .001.
**p ≤ .01.
*p ≤ .05.
To more thoroughly examine potential data quality gains with increasing length, we also examined the effects of consent provision on the amount of information respondents provide when asked to describe in an open format their occupation and its most important activities. A similar pattern of results emerged, with interviews providing consent having character fields that are 17% longer on average (343 characters) than those denying consent (292 characters; see ancillary Table 5). In multivariate models, the parameter for consent was of marginal significance (p = .06). Its magnitude was reduced only modestly by family and field characteristics and more substantially with the addition of interviewer fixed effects. Not surprisingly, interviewers with previous experience conducting a PSID interview and those rated as high performers were less responsive to whether or not the respondent gave consent.
Multivariate Models of 2011 Length of Occupation Open-ended Text Field.a
Note: N = 5,438. aOutcome variable is calculated as count of the characters in the text field for the question asking respondent to describe household heads’ current or main occupation, the sort of work they do, and the most important activities or duties among household heads with some employment during the reporting period; bAdds dummy variables for each of the interviewer identification variables, yes = 1 and no = 0.
***p ≤ .001.
**p ≤ .01.
*p ≤ .05.
Discussion
The time that respondents spend answering survey questions is a precious resource that must be managed carefully. Survey directors continually look for ways to streamline interviews to reduce respondent burden and maximize the scientific value of the time that respondents spend providing information. At the same time, some data collection strategies that reduce interview length may jeopardize data quality, such as not asking respondents if they have any other important activities or duties within a question about occupation. This study examined the impact of a new quality control protocol adopted by a national longitudinal survey and reports on four key findings.
First, we find that the use of digital recording software increased the length of the interview. While the estimated effect is fairly modest at 3.5%, additional analyses are currently being undertaken to understand why interviews that were digitally recorded were longer. However, these effects may be unique to this study, and assessment of the potential effects in other studies is also needed.
The second key finding is that interviews take substantially longer if interviewers know there is a chance that the interview is being recorded. In the 2011 PSID, we estimate this effect to be roughly 6 minutes, implying an interview that is 6.8% longer than if interviewers know with certainty that an interview is not being recorded. This effect persists even after controlling for a large number of characteristics of respondents, field effort, and interviewer fixed effects.
A third finding is that the effect of consenting on interview length differs by observable characteristics of interviewers. In particular, interviewers with higher levels of education and performance are less responsive to whether or not the interview is recorded. Consistent with predictions made by social facilitation theory and other research (e.g., Bhave 2013), this suggests that interviewers who perceived they may be monitored, especially those with the greatest room for improvement, changed their behavior, leading to closer adherence to training guidelines and concomitantly increasing interview length.
A fourth finding that provides additional suggestive evidence that greater adherence by interviewers to their training during potentially recorded interviews increases overall interview length is the emergence of a similar pattern of results for data quality. Rates of item missing data were significantly lower—and the amount of information obtained on an open-ended text field was greater—when interviewers thought the interview was possibly being recorded. As with interview length, this finding persisted even after controlling for a large number of respondent characteristics, field effort, and interviewer fixed effects. The results for the effects of consenting on length of the occupation text field were similar to those for interview length with high performers less reactive to whether or not interviews were being recorded. Interviewer characteristics were unrelated to the effect of consenting on rates of item missing data, with consented interviews having less missing data on average, regardless of interviewer experience, education, or performance.
One way to assess the value of the six additional minutes due to consenting respondents is to calculate the number of survey questions that could be asked if this time was instead used to collect additional information. Many important modules in the PSID are shorter than 6 minutes, including those on expenditures, pensions, and wealth. On average, roughly 3–5 questions can be asked each minute, implying that the impact on interview length of consenting respondents in the PSID is roughly equal to adding 18–30 survey questions. However, given that an individual’s prior wave survey experience is known to influence each successive survey request (e.g., Lepkowski and Couper 2002; Uhrig 2008), increasing the survey length may lead to undesirable outcomes such as increased interview break-offs that require significant field effort to finalize (McGonagle 2013).
At the same time, the increase in interview length due to obtaining consent for recording must be weighed against the beneficial effects found on data quality. Specifically, interviewers who believed there was a chance their interview was being recorded had interviews with average item missing data rates roughly 30% lower on the 25 survey items with the highest rates of item nonresponse. Moreover, the quantity of information collected for one of the most important open-ended questions in the PSID—occupation status—was increased substantially.
Knowledge that the interview may be recorded may alter interviewer behavior by causing interviewers to more closely follow training protocols. If training protocols are optimally designed, then closer adherence to these protocols should be unequivocally beneficial. This study demonstrates the need to design training protocols that consider the potential trade-off between accurately assessing the survey construct of interest—which in some cases may require a lengthy question read slowly and with extensive probes—and minimizing respondent burden as measured by interview length.
In sum, the recording of survey interviews is a commonly used tool for interview verification and quality control. After observing an unexpected increase in interview length during a wave in which a new quality control protocol was introduced, this study found that the recording protocol increased data quality but had the unintended consequence of lengthening the interview. Survey managers should explicitly consider both the costs and benefits of recording when developing quality control protocols.
Footnotes
Acknowledgments
We are grateful for the feedback and assistance of Patricia Andreski, April Beaule, Sara Freeland, Shonda Kruger-Ndiaye, Eva Leissou, Mohammad Mushtaq, and Dan Zahs.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the main sponsors of the Panel Study of Income Dynamics, including the National Science Foundation (SES0518943), the National Institute on Aging (R01AG019802), and the Eunice Kennedy Shriver National Institute of Child Health and Human Development (R01HD033474).