Abstract
We discuss an evidence-based approach to guiding real-time design decisions during the course of survey data collection. We call it responsive and adaptive design (RAD), a scientific framework driven by cost-quality tradeoff analysis and optimization that enables the most efficient production of high-quality data. The notion of RAD is not new; nor is it a silver bullet to resolve all the difficulties of complex survey design and challenges. RAD embraces precedents and variants of responsive design and adaptive design that survey designers and researchers have practiced over decades. In this paper, we present the four pillars of RAD: survey process data and auxiliary information, design features and interventions, explicit quality and cost metrics, and a quality-cost optimization tailored to survey strata. We discuss how these building blocks of RAD are addressed by articles published in the 2017 JOS special issue and this special section. It is a tale of the three perspectives filling in each other. We carry over each of these three perspectives to articulate the remaining challenges and opportunities for the advancement of RAD. We recommend several RAD ideas for future research, including survey-assisted population modeling, rigorous optimization strategies, and total survey cost modeling.
Keywords
References
1.
Altman, R., P. Pizzo, R. Gibbons, K. Hudson, R. Jenkins, B. Lee, M. Lichtveld, M.L. Miranda, C. Perry, H. Zoghbi, and L. Jorgenson. 2014. National Children’s Study . Working Group NIH Advisory Committee to the Director Final Report. Bethesda, MD: National Institutes of Health. https://acd.od.nih.gov/documents/reports/NCS_WG_FINAL_REPORT.pdf.
2.
Brick, M. and R. Tourangeau. 2017. “Responsive Survey Designs for Reducing Nonresponse Bias.” In A.Y. Chun, B. Schouten, and J. Wagner (Eds), Special Issue on Responsive and Adaptive Design, Journal of Official Statistics 33(3): 735–752. Doi: https://doi.org/10.1515/jos-2017-0034.
3.
Burger, J., K. Perryck, and B. Schouten. 2017. “Robustness of Adaptive Survey Designs to Inaccuracy of Design Parameters.” In A.Y. Chun, B. Schouten, and J. Wagner (Eds), Special Issue on Responsive and Adaptive Design, Journal of Official Statistics 33(3): 687–708. Doi: https://doi.org/10.1515/jos-2017-0032.
4.
Calinescu, M. and B. Schouten. 2016. “Adaptive Survey Designs for Nonresponse and Measurement Error in Multi-Purpose Surveys.” Survey Research Methods 10(1): 35–47. Doi: https://dx-doi-org.web.bisu.edu.cn/10.18148/srm/2016.v10i1.6157.
5.
Chun, A.Y. 2009. Nonparticipation of the 12th graders in the National Assessment of Educational Progress: Understanding Determinants of Nonresponse and Assessing the Impact on NAEP Estimates of Nonresponse Bias According to Propensity Models. University of Maryland, College Park, USA. http://hdl.handle.net/1903/9916.
6.
Chun, A.Y. 2012. “What Counts as Group Quarters? – A Glimpse of Census Cost-Data Quality Models.” Paper presented at the Joint Statistical Meetings, San Diego, U.S.A. (accessed July 30, 2012).
7.
Chun, A.Y. and M. Kwanisai 2010. “A Response Propensity Modeling Navigator for Paradata.” Proceedings of the Survey Research Methods Section of the American Statistical Association, Joint Statistical Meetings, Vancouver, Canada, 356–369. http://ww2.amstat.org/sections/SRMS/Proceedings/y2010/Files/306125_55196.pdf.
8.
Chun, A.Y., B. Schouten, and J. Wagner. 2017. JOS Special Issue on Responsive and Adaptive Design: “Looking Back to See Forward – Editorial.” In A.Y. Chun, B. Schouten, J. Wagner (Eds), JOS Special Issue on Responsive and Adaptive Design, Journal of Official Statistics , 33(3): 571–577. Available at: https://www.degruyter.com/downloadpdf/j/jos.2017.33.issue-3/jos-2017-0027/jos-2017-0027.pdf.
9.
Dillman, D.A., J.D. Smyth, and L.M. Christian. 2014. Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method (4th Ed.), Wiley.
10.
Durrant, G., O. Maslovskaya, and P. Smith. 2017. “Using Prior Wave Information and Paradata: “Can They Help to Predict Response Outcomes and Call Sequence Length in a Longitudinal Study?” In A.Y. Chun, B. Schouten, and J. Wagner (Eds), Special Issue on Responsive and Adaptive Design, Journal of Official Statistics 33(3): 801–833. Doi: https://doi.org/10.1515/jos-2017-0037.
11.
Early, K., J. Mankoff, and S. Fienberg. 2017. “Dynamic Question Ordering in Online Surveys.” In A.Y. Chun, B. Schouten, and J. Wagner (Eds), Special Issue on Responsive and Adaptive Design, Journal of Official Statistics 33(3): 625–657. Doi: https://doi.org/10.1515/jos-2017-0030.
12.
Felligi, I.P. and A.B. Sunter. 1974. “Balance Between Different Sources of Survey Errors – Some Canadian Experiences.” Sankhya Series C Vol. 36(3): 119–142.
13.
Government Accountability Office 2015. 2020 Census: Progress Report on Using Administrative Records to Control Enumeration Costs. Testimony before the Subcommittees on Government Operations and information Technology , Committee on Oversight and Government Reform, House of Representatives, Washington DC. https://oversight.house.gov/wp-content/uploads/2015/11/Goldenkoff-GAO-Statement-11-3-Census-2020.pdf.
14.
Groves, R.M. 1989. Survey Errors and Survey Costs. New York: John Wiley and Sons.
15.
Groves, R.M. 2011. “Three Eras of Survey Research.” The Public Opinion Quarterly 75(5): 861–871. https://doi.org/10.1093/poq/nfr057.
16.
Groves, R.M. and S.G. Heeringa. 2006. “Responsive Design for Household Surveys: Tools for Actively Controlling Survey Errors and Costs.” Journal of the Royal Statistical Society Series A: Statistics in Society 169(3): 439–457. Doi: https://doi.org/10.1111/j.1467-985X.2006.00423.x.
17.
Hansen, M.H. and W.N. Hurwitz 1946. “The Problem of Nonresponse in Surveys.” Journal of the American Statistical Association 41(236): 517–529. Doi: 10.1080/01621459.1946.10501894.
18.
Kaminska, O. and P. Lynn 2017. “The Implications of Alternative Allocation Criteria in Adaptive Design for Panel Surveys.” In A.Y. Chun, B. Schouten, and J. Wagner (Eds), Special Issue on Responsive and Adaptive Design, Journal of Official Statistics 33(3): 781–799. Doi: https://doi.org/10.1515/jos-2017-0036.
19.
Klausch, L.T. 2014. “Informed Design of Mixed-Mode Surveys: Evaluating Mode Effects on Measurement and Selection Error.” PhD Thesis, University Utrecht, The Netherlands. https://dspace.library.uu.nl/handle/1874/300673.
20.
Kreuter, F. 2013. Improving Surveys with Paradata: Analytic Uses of Process Information . New Work, U.S.A.: John Wiley & Sons.
21.
Lewis, T. 2017. “Univariate Tests for Phase Capacity: Tools for Identifying When to Modify a Survey’s Data Collection Protocol.” In A.Y. Chun, B. Schouten, and J. Wagner (Eds), Special Issue on Responsive and Adaptive Design, Journal of Official Statistics 33(3): 601–624. Doi: https://doi.org/10.1515/jos-2017-0029.
22.
McCarthy, J., J. Wagner, and H. Sanders 2017. “The Impact of Targeted Data Collection on Nonresponse Bias in an Establishment Survey: A Simulation Study of Adaptive Survey Design.” In A.Y. Chun, B. Schouten, and J. Wagner (Eds), Special Issue on Responsive and Adaptive Design, Journal of Official Statistics 33(3): 857–871. Doi: https://doi.org/10.1515/jos-2017-0039.
23.
Morganstein, D. and D.A. Marker. 1997. “Continuous Quality Improvement in Statistical Agencies.” In Survey Measurement and Process Quality edited by L.E. Lyberg, P.P. Biemer, M. Collins, E. De Leeuw, C. Dippo, N. Schwarz, and D. Trewin, 475–500. New York, U.S.A.: John Wiley & Sons.
24.
Mulry, M. and B. Spencer. 2012. “A Framework for Empirical Cost Modeling Relating Cost and Data Quality.” Paper presented at the 2012 International Total Survey Error Workshop (accessed September 3, 2012).
25.
Paiva, T. and J. Reiter. 2017. “Stop or Continue Data Collection: A Nonignorable Missing Data Approach for Continuous Variables.” 579–599. In A.Y. Chun, B. Schouten, and J. Wagner (Eds), Special Issue on Responsive and Adaptive Design, Journal of Official Statistics 33(3): 579–599. Doi: https://doi.org/10.1515/jos-2017-0028.
26.
Plewis, I. and N. Shlomo 2017. “Using Response Propensity Models to Improve the Quality of Response Data in Longitudinal Studies.” In A.Y. Chun, B. Schouten, and J. Wagner (Eds), Special Issue on Responsive and Adaptive Design, Journal of Official Statistics 33(3): 753–779. Doi: https://doi.org/10.1515/jos-2017-0035.
27.
Särndal, C. and P. Lundquist 2017. “Inconsistent Regression and Nonresponse Bias: Exploring Their Relationship as a Function of Response Imbalance.” In A.Y. Chun, B. Schouten, and J. Wagner (Eds), Special Issue on Responsive and Adaptive Design, Journal of Official Statistics 33(3): 709–734. Doi: https://doi.org/10.1515/jos-2017-0033.
28.
Schouten, B., A. Peytchev, and J. Wagner 2017. Adaptive Survey Design . Chapman and Hall.
29.
Thompson, K.J. and S. Kaputa 2017. “Investigating Adaptive Nonresponse Follow-up Strategies for Small Businesses through Embedded Experiments.” In A.Y. Chun, B. Schouten, and J. Wagner (Eds), Special Issue on Responsive and Adaptive Design, Journal of Official Statistics 33(3): 835–856. Doi: https://doi.org/10.1515/jos-2017-0038.
30.
Tourangeau, R., M. Brick, S. Lohr, and J. Li. 2016. “Adaptive and Responsive Survey Designs: a Review and Assessment.” Journal of the Royal Statistical Society: Series A 180: 203–223. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1111/rssa.12186.
31.
U.S. Census Bureau. 2017. 2020 Census Operational Plan v3.0: A New Design for the 21st Century . Washington D.C. https://www2.census.gov/programs-surveys/decennial/2020/program-management/planning-docs/2020-oper-plan3.pdf.
32.
Vandenplas, C., G. Loosveldt, and K. Beullens. 2017. “Fieldwork Monitoring for the European Social Survey: an illustration with Belgium and the Czech Republic in Round 7.” In A.Y. Chun, B. Schouten, and J. Wagner (Eds), Special Issue on Responsive and Adaptive Design, Journal of Official Statistics 33(3): 659–686. Doi: https://doi.org/10.1515/jos-2017-0031.
33.
Wagner, J. 2008. “Adaptive Survey Design to Reduce Nonresponse Bias.” PhD thesis, University of Michigan, Ann Arbor, USA.
34.
Wagner, J., K. Olson, and R. Anderson. 2016. “Survey Costs: The Missing Half of the “Cost-Error.” Tradeoff, Paper presented at the Joint Statistical Meetings, Chicago, IL, USA.
35.
Wald, A. 1947. Sequential Analysis . Toronto, Canada: General Publishing Co.