Using Prior Wave Information and Paradata: Can They Help to Predict Response Outcomes and Call Sequence Length in a Longitudinal Study?

Abstract

In recent years the use of paradata for nonresponse investigations has risen significantly. One key question is how useful paradata, including call record data and interviewer observations, from the current and previous waves of a longitudinal study, as well as previous wave survey information, are in predicting response outcomes in a longitudinal context. This article aims to address this question. Final response outcomes and sequence length (the number of calls/visits to a household) are modelled both separately and jointly for a longitudinal study. Being able to predict length of call sequence and response can help to improve both adaptive and responsive survey designs and to increase efficiency and effectiveness of call scheduling. The article also identifies the impact of different methodological specifications of the models, for example different specifications of the response outcomes. Latent class analysis is used as one of the approaches to summarise call outcomes in sequences. To assess and compare the models in their ability to predict, indicators derived from classification tables, ROC (Receiver Operating Characteristic) curves, discrimination and prediction are proposed in addition to the standard approach of using the pseudo R² value, which is not a sufficient indicator on its own. The study uses data from Understanding Society, a large-scale longitudinal survey in the UK. The findings indicate that basic models (including geographic, design and survey data from the previous wave), although commonly used in predicting and adjusting for nonresponse, do not predict the response outcome well. Conditioning on previous wave paradata, including call record data, interviewer observation data and indicators of change, improve the fit of the models slightly. A significant improvement can be observed when conditioning on the most recent call outcome, which may indicate that the nonresponse process predominantly depends on the most current circumstances of a sample unit.

Keywords

Survey nonresponse interviewer call record data paradata call sequence responsive and adaptive survey designs

References

Agresti, A. 2013. Categorical Data Analysis. New Jersey: John Wiley & Sons.

Altman, D.G. 1991. Practical Statistics for Medical Research. London: Champan & Hall.

Akaike, H. 1974. “A New Look at Statistical Model Identification.” IEEE Transactions on Automatic Control AC-19: 716–723. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1109/TAC.1974.1100705.

Asparouhov, T. and B.O. Muthén. 2013. Auxiliary Variables in Mixture Modeling: 3-Step Approaches Using Mplus. Mplus Web Notes: No. 15. Available at: http://www.statmodel.com/download/AppendicesOct28.pdf (accessed 9 June 2017).

Bartholomew, D., F. Steele, I. Moustaki, and J. Galbraith. 2008. Analysis of Multivariate Social Science Data. London: CPC Press.

Bates, N., J. Dahlhamer, and E. Singer. 2008. “Privacy Concerns, too Busy, or just not Interested: Using Doorstep Concerns to Predict Survey Nonresponse.” Journal of Official Statistics 24: 591–612.

Beadnell, B., S. Baker, K. Knox, S. Stielstra, D.M. Morrison, E. DeGooyer, L. Wickizer, A. Doyle, and M. Oxford. 2003. “The Influence of Psychosocial Difficulties on Women’s Attrition in an HIV/STD Prevention Program.” AIDS Care 15(6): 807–820. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1080/09540120310001618658.

Bethlehem, J., C. Fannie, and B. Schouten. 2011. Handbook of Nonresponse in Household Surveys. New Jersey: Wiley.

Biemer, P.P., P. Chen, and K. Wang. 2013. “Using Level-of-Effort Paradata in Non-Response Adjustments with Application to Field Surveys.” Journal of Royal Statistical Society: Serie A 176: 147–168. Doi: https://dx-doi-org.web.bisu.edu.cn/10.2307/23355181.

10.

Bozdogan, H. 1987. “Model Selection and Akaike’s Information Criterion (AIC): the General Theory and its Analytical Extensions.” Psychometrika 52: 345–370. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1007/BF02294361.

11.

Buck, N. and S. McFall. 2012. “Understanding Society: Design Overview.” Longitudinal and Life Course Studies 3(1): 5–17. Doi: https://dx-doi-org.web.bisu.edu.cn/10.14301/llcs.v3i1.159.

12.

Campanelli, P. and C. O’Muircheartaigh. 1999. “Interviewers, Interviewer Continuity, and Panel Survey Nonresponse.” Quality and Quantity 33: 59–76. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1023/A:1004357711258.

13.

De Leeuw, E. and W. de Heer. 2002. “Trends in Household Survey Nonresponse: a Longitudinal and International Comparison.” In Survey Nonresponse, edited by R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little, 41–54. New York: John Wiley and Sons.

14.

Dias, J.G. 2001. Components of Knowledge on AIDS in Brazil: Identifying Information Needs Using a Segmented Approach. Population Research Centre Working Paper. University of Groningen: Population Research Centre.

15.

Durrant, G.B., J. D’Arrigo, and G. Müller. 2013a. “Modeling Call Record Data: Examples from Cross-Sectional and Longitudinal Surveys.” In Improving surveys with paradata: Analytic uses of process information, edited by F. Kreuter, 281–308. New Jersey: Wiley and Sons.

16.

Durrant, G.B., J. D’Arrigo, and F. Steele. 2011. “Using Field Process Data to Predict Best Times of Contact Conditioning on Household and Interviewer Influences.” Journal of Royal Statistical Society: Series A 174: 1029–1049. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1111/j.1467-985X.2011.00715.x.

17.

Durrant, G.B., J. D’Arrigo, and F. Steele. 2013b. “Analysing Interviewer Call Record Data by Using a Multilevel Discrete-Time Event History Modelling Approach.” Journal of Royal Statistical Society: Series A 176: 251–269. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1111/j.1467-985X.2012.01073.x.

18.

Durrant, G.B., R.M. Groves, L. Staetsky, and F. Steele. 2010. “Effects of Interviewer Attitudes and Behaviors on Refusal in Household Surveys.” Public Opinion Quarterly 74(1): 1–36. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1093/poq/nfp098.

19.

Durrant, G.B., O. Maslovskaya, and P.W.F. Smith. 2015. “Modelling Final Outcome and Length of Call Sequence to Improve Efficiency in Interviewer Call Scheduling.” Journal of Survey Statistics and Methodology 3: 397–424. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1093/jssam/smv008.

20.

Durrant, G.B., O. Maslovskaya, and P.W.F. Smith. 2016. “Investigating Call Record Data Using Sequence Analysis to Inform Adaptive Survey Designs.” National Centre for Research Methods (NCRM) Working Paper. University of Southampton.

21.

Durrant, G.B. and F. Steele. 2009. “Multilevel Modelling of Refusal and Noncontact Nonresponse in Household Surveys: Evidence from Six UK Government Surveys.” Journal of the Royal Statistical Society: Series A 172(2): 361–381. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1111/j.1467-985X.2008.00565.x.

22.

Eckman, S., J. Sinibaldi, and A. Montmann-Hertz. 2013. “Can Interviewers Effectively rate the Likelihood of Cases to Cooperate?” Public Opinion Quarterly 77(2): 561–573. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1093/poq/nft012.

23.

Field, A. 2009. Discovering Statistics Using SPSS. Los Angeles: SAGE.

24.

Groves, R.M. 2006. “Nonresponse Rates and Nonresponse Bias in Household Surveys.” Public Opinion Quarterly 70(5): 646–675.

25.

Groves, R.M. and M.P. Couper. 1996. “Contact-Level Influences on Cooperation in Face-to-Face Surveys.” Journal of Official Statistics 12: 63–83.

26.

Groves, R. and M. Couper. 1998. Nonresponse in Household Interview Surveys. New York: Wiley.

27.

Groves, R.M. and S.G. Heeringa. 2006. “Responsive design for Household Surveys: Tools for Actively Controlling Survey Errors and Costs.” Journal of Royal Statistical Society: Series A 169: 439–459. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1111/j.1467-985X.2006.00423.x.

28.

Hagenaars, J.A. and A.L. McCutcheon. 2002. Applied Latent Class Analysis. Cambridge: Cambridge University Press.

29.

Hanly, M. 2014. “Improving Nonresponse Bias Adjustments with Call Record Data.” Conference paper. 25th International Workshop on Household Survey Nonresponse, Iceland.

30.

Hanly, M., P. Clarke, and F. Steele. 2015. “Sequence Analysis of Call Record Data: Exploring the Role of Different Cost Settings.” Journal of the Royal Statistical Society: Series A (Statistics in Society). Doi: https://dx-doi-org.web.bisu.edu.cn/10.1111/rssa.12143.

31.

Haunberger, S. 2010. “The Effects of Interviewer, Respondent and Area Characteristics on Cooperation in Panel Surveys: a Multilevel Approach.” Quality and Quantity 44: 957–969. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1007/s11135-009-9248-5.

32.

Huber, P.J. 1967. “The Behavior of Maximum Likelihood Estimates under Nonstandard Conditions.” In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Berkeley, CA: University of California Press, vol. 1: 221–233. Available at: https://projecteuclid.org/euclid.bsmsp/1200512988 (accessed 31 March 2016).

33.

Kreuter, F. (ed.). 2013. Improving Surveys with Paradata: Analytic Uses of Process Information. New Jersey: Wiley and Sons.

34.

Kreuter, F., M. Couper, and L. Lyberg. 2010a. “The Use of Paradata to Monitor and Manage Survey Data Collection.” In Proceedings of the Joint Statistical Meeting, Section of Survey Research Methods, 282–296. Vancouver, Canada.

35.

Kreuter, F. and U. Kohler. 2009. “Analyzing Contact Sequences in Call Record Data. Potential and Limitations of Sequence Indicators for Nonresponse Adjustments in the European Social Survey.” Journal of Official Statistics 25: 203–226.

36.

Kreuter, F., K. Olson, J. Wagner, T. Yan, T.M. Ezzati-Rice, C. Casas-Cordero, M. Lemay, A. Peytchev, R.M. Groves, and T.E. Raghunathan. 2010b. “Using Proxy Measures and Other Correlates of Survey Outcomes to Adjust for Non-Response: Examples from Multiple Surveys.” Journal of Royal Statistical Society: Series A 173: 389–407. Doi: https://dx-doi-org.web.bisu.edu.cn/0.1111/j.1467-985X.2009.00621.x.

37.

Krueger, B.S. and B.T. West. 2014. “Assessing the Potential of Paradata and Other Auxiliary Information for Nonresponse Adjustments.” Public Opinion Quarterly 78(4): 795–831. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1093/poq/nfu040.

38.

Lagorio, C. 2016. “Call and Response: Modelling Longitudinal Contact and Cooperation Using Wave 1 Call Records Data.” Understanding Society Working Paper Series No. 2016-01.

39.

Lepkowski, J.M. and M.P. Couper. 2002. “Nonresponse in the Second Wave of Longitudinal Household Surveys.” In Survey nonresponse, edited by R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little, 259–272. New York: John Wiley and Sons.

40.

Magidson, J. and J.K. Vermunt. 2004. “Latent Class Models.” In The SAGE handbook of Quantitative Methodology for Social Sciences, edited by D. Kaplan, 175–198. Thousand Oaks: SAGE publications.

41.

McFall, S.L. (ed.). 2012. Understanding Society: Findings 2012. Colchester: Institute for Social and Economic Research, University of Essex.

42.

McFall, S.L. (ed.). 2013. Understanding Society – UK Household Longitudinal Study: Wave 1–3, 2009–2012, User Manual. Colchester: University of Essex.

43.

Muthén, B.O. 1998–2004. Mplus Technical Appendices. Los Angeles, CA: Muthén & Muthén. Available at: http://www.statmodel.com/download/techappen.pdf (accessed 10 February 2016).

44.

Muthén, B.O. and L.K. Muthén. 2000. “Integrating Person-Centered and Variable-Centered Analyses: Growth Mixture Modeling with Latent Trajectory Classes.” Alcoholism: Clinical and Experimental Research 24(6): 882–891. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1111/j.1530-0277.2000.tb02070.x.

45.

Muthén, L.K. and B.O. Muthén. 2012. Mplus User’s Guide. Los Angeles, CA: Muthén and Muthén.

46.

Nagelkerke, N.J.D. 1991. “A Note on a General Definition of the Coefficient of Determination.” Biometrika 78: 691–692. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1093/biomet/78.3.691.

47.

Olson, K. and R.M. Groves. 2012. “An Examination of Within-Person Variation in Response Propensity over the Data Collection Field Period.” Journal of Official Statistics 28: 29–51.

48.

Olson, K., J.D. Smyth, and H.M. Wood. 2012. “Does Giving People Their Preferred Survey Mode Actually Increase Survey Participation rates? An Experimental Examination.” Public Opinion Quarterly 76: 611–635. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1093/poq/nfs024.

49.

Pepe, M.S. 2003. The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford: Oxford University Press.

50.

Pickery, J., G. Loosveldt, and A. Carton. 2001. “The Effects of Interviewer and Respondent Characteristics on Response Behavior in Panel Surveys – a Multilevel Approach.” Sociological Methods and Research 29: 509–523. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1177/0049124101029004004.

51.

Plewis, I., S. Ketende, and L. Calderwood. 2012. “Assessing the Accuracy of Response Propensities in Longitudinal Studies.” Survey Methodology 38(2): 167–171.

52.

Potthoff, R.F., K.G. Manton, and M.A. Woodbury. 1993. “Correcting for Nonavailability Bias in Surveys by Weighting Based on Number of Callbacks.” Journal of the American Statistical Association, Applications and Case Studies 88(424): 1197–1207. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1080/01621459.1993.10476399.

53.

Schwarz, G. 1978. “Estimating the Dimension of a Model.” The Annals of Statistics 6(2): 461–464. Doi: https://dx-doi-org.web.bisu.edu.cn/10.2307/2958889.

54.

Sinibaldi, J. 2014. “Using Call-Level Interviewer Observations to Improve Response Propensity Models.” In Evaluating the Quality of Interviewer Observed Paradata for Nonresponse Applications (PhD Thesis). München: Ludwig-Maximilian-Universität.

55.

Sinibaldi, J., G.B. Durrant, and F. Kreuter. 2013. “Evaluating the Measurement Error of Interviewer Observed Paradata.” Public Opinion Quarterly, Special issue: Topics in Survey Measurement and Public Opinion 77(1): 173–193. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1093/poq/nfs062.

56.

Sinibaldi, J., M. Trappmann, and F. Kreuter. 2014. “Which is the Better Investment for Nonresponse Adjustment: Purchasing Commercial Auxiliary Data or Collecting Interviewer Observations?” Public Opinion Quarterly 78(2): 440–473. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1093/poq/nfu003.

57.

Storr, C.L., H. Zhou, K.Y. Liang, and J.C. Anthony. 2004. “Empirically Derived Latent Classes of Tobacco Dependence Syndromes Observed in Recent-Onset Tobacco Smokers: Epidemiological Evidence from a National Probability Sample Survey.” Nicotine and Tobacco Research 6(3): 533–545. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1080/14622200410001696493.

58.

Wagner, J. 2013a. “Adaptive Contact Strategies in Telephone and Face-to-Face Surveys.” Survey Research Methods 7: 45–55. Doi: https://dx-doi-org.web.bisu.edu.cn/10.18148/srm/2013.v7i1.5037.

59.

Wagner, J. 2013b. “Using Paradata-Driven Models to Improve Contact Rates in Telephone and Face-to-Face Surveys.” In Improving Surveys with Paradata: Analytic Use of Process Information, edited by F. Kreuter, 145–170. New Jersey: Wiley and Sons.

60.

Watson, N. and M. Wooden. 2009. “Identifying Factors Affecting Longitudinal Survey Response.” In Methodology of Longitudinal Surveys, edited by P. Lynn, 157–182. New York: John Wiley and Sons.

61.

West, B.T. 2013. “An Examination of the Quality and Utility of Interviewer Observations in the National Survey of Family Growth.” Journal of the Royal Statistical Society, Series A 176(1): 211–225. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1111/j.1467-985X.2012.01038.x.

62.

West, B.T., F. Kreuter, and M. Trappmann. 2014. “Is the Collection of Interviewer Observations Worthwhile in an Economic Panel Survey? New Evidence from the German Labor Market and Social Security (PASS) Study.” Journal of Survey Statistics and Methodology 2(2): 159–181. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1093/jssam/smu002.

63.

West, B.T. and R.M. Groves. 2013. “A Propensity-Adjusted Interviewer Performance Indicator.” Public Opinion Quarterly 77: 352–374. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1093/poq/nft002.

64.

White, H. 1980. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica 48: 817–830. Doi: https://dx-doi-org.web.bisu.edu.cn/10.2307/1912934.

65.

White, H. 1984. Asymptotic Theory for Econometricians. Orlando, FL: Academic Press.

66.

White, H. 1994. Estimation, Inference and Specification Analysis. New York: Cambridge University Press.