Abstract
This article discusses the critical and complex design decisions associated with transitioning an interviewer-administered survey to a self-administered, postal, web/paper survey. Our approach embeds adaptive, responsive, and tailored (ART) design principles and data visualization during a multi-phased data collection operation to project the outcomes of each phase in preparation for subsequent phases. This requires rapid decision making based upon experimental results using a data visualization system to monitor critical-to-quality (CTQ) metrics and facilitate projections of outcomes from the current phase of data collection to inform the design of the subsequent phase. We describe the objectives of the overall design, the features designed to address these objectives, components of the visual adaptive total design (ATD) system for monitoring quality components and relative costs in real time, and examples of the visualization elements and functionalities that were used in one case study. We also discuss subsequent initiatives to develop an interactive version of the monitoring tool and applications for other studies, including those employing adaptive, responsive, and tailored (ART) designs. Our case study is a series of pilot studies conducted for the Residential Energy Consumption Survey (RECS), sponsored by the U.S. Energy Information Administration (EIA).
References
1.
AAPOR (The American Association for Public Opinion Research). 2015. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys , 8th edition. Oakbrook Terrace, IL: AAPOR.
2.
Amaya, A., P. Biemer, and D. Kinyon. 2017. “Total Error in a Big Data World with Applications to the Residential Energy Consumption Survey.” Presented at the American Association for Public Opinion Research Annual Conference, New Orleans, LA.
3.
Biemer, P. 2010. “Total Survey Error: Design, Implementation, and Evaluation.” Public Opinion Quarterly 74(5): 817–848. Doi: https://doi.org/10.1093/poq/nfq058.
4.
Biemer, P., K.H. Harris, B. Burke, K. Considine, C. Halpern, and C. Suchindran. 2017a. “Transitioning an In-Person Longitudinal Survey to a Mixed-Mode, Two-Phase Survey Design: Preliminary Results.” Presented at the Annual Conference of the American Association for Public Opinion Research. New Orleans, LA.
5.
Biemer, P., J. Murphy, S. Zimmer, C. Berry, G. Deng, and K. Lewis. 2017b. “Using Bonus Monetary Incentives to Encourage Web Response in Mixed-Mode Household Surveys.” Journal of Survey Statistics and Methodology . Doi: https://doi.org/10.1093/jssam/smx015.
6.
Breyfogle, F. 2003. Implementing Six Sigma: Smarter Solutions Using Statistical Methods . Hoboken, NJ: John Wiley & Sons.
7.
Camoes, J. 2008. How to Create a Thematic Map in Excel . Available at: http://www.excelcharts.com/blog/how-to-create-thematic-map-excel/ (accessed November 26, 2017).
8.
Chun, A.Y., B. Schouten, and J. Wagner. 2017. “JOS Special Issue on Responsive and Adaptive Survey Design: Looking Back to See Forward – Editorial.” Journal of Official Statistics 33(3): 571–577. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1515/JOS-2017-0027.
9.
Cleveland, W. 1993. Visualizing Data . Summit, NJ: Hobart Press.
10.
Cramér, H. 1946. Mathematical Methods of Statistics . Princeton: Princeton University Press.
11.
Dillman, D., J.D. Smyth, and L.M. Christian. 2014. Internet, Phone, and Mail, and Mixed-Mode Surveys: The Tailored Design Method , 4th Edition. Hoboken, NJ: Wiley.
12.
Dillman, D.A. and M.L. Edwards. 2016. “Designing a Mixed-Mode Survey.” In Wolfe, Christof, Joye, Dominique, Smith, Tom W. and Fu, Yang-chih, Sage Handbook of Survey Methodology. Sage Publications Wolf, Joye, Smith and Fu. Thousand Oaks. CA, 255–268.
13.
Duprey, M., J. Murphy, P. Biemer, and R. Chew. 2017. “Veni, Vidi, Vici: Interactive Data Visualizations for Adaptive Total Design.” Presented at the 5th Workshop on Adaptive and Responsive Survey Design. Ann Arbor, MI.
14.
Eddy, W.F. and Marton, K., Editors. 2012. Effective Tracking of Building Energy Use: Improving the Commercial Buildings and Residential Energy Consumption Surveys . Washington D.C.: The National Academies Press.
15.
Edgar, J., J. Murphy, and M. Keating. 2016. “Comparing Traditional and Crowdsourcing Methods for Pretesting Survey Questions.” SAGE Open 6(4): 1–14. Doi: https://doi.org/10.1177/2158244016671770.
16.
Groves, R.M. 1989. Survey Errors and Survey Costs . New York: Wiley.
17.
Groves, R. and S. Heeringa. 2006. “Responsive Design for Household Surveys: Tools for Actively Controlling Survey Errors and Costs.” Journal of the Royal Statistical Society, Series A 169(3): 439–457. Doi: https://dx-doi-org.web.bisu.edu.cn/10.1111/j.1467-985X.2006.00423.x.
18.
Hardin, M., D. Horn, R. Perez, and L. Williams. 2012. “Which Chart or Graph is Right for You? Telling Impactful Stories with Data.” Tableau Software. Available at: http://theathenaforum.org/sites/default/files/WHich%20chart%20is%20right%20for%20you.pdf (accessed November 26, 2017).
19.
Hornbaek, K. and E. Frokjaer. 2003. “Reading Patterns and Usability in Visualizations of Electronic Documents.” ACM Transactions on Computer-Human Interaction 10(2): 119–149. Doi: https://doi.org/10.1145/772047.772050.
20.
Howden, L., S. Joestl, and R. Cohen. 2015. Improving Response Rates using a Mixed-Mode Approach: Results from the National Health Care Interview Survey. Presented at the 2015 FedCASIC Conference. Available at: https://www.census.gov/fedcasic/fc2015/ppt/27_howden.pdf (accessed November 21, 2017).
21.
Kahneman, D. 2011. Thinking Fast and Slow . New York: Farrar, Straus, and Giroux.
22.
Laflamme, F. and J. Wagner. 2016. “Responsive and Adaptive Designs.” In The SAGE Handbook of Survey Methodology , edited by C. Wolf, D. Joye, T. Smith, and Y. Fu. Los Angeles: Sage.
23.
Link, M. and A. Mokdad. 2005. “Alternative Modes for Health Surveillance Surveys: an Experiment with Web, Mail, and Telephone.” Epidemiology 16: 701–704. Doi: 10.1097/01.ede.0000172138.67080.7f.
24.
Luiten, A. and B. Schouten. 2013. “Tailored Fieldwork Design to Increase Representative Household Survey Response: an Experiment in the Survey of Consumer Satisfaction.” Journal of the Royal Statistical Society A 176: 169–189. Doi: https://doi.org/10.1111/j.1467-985X.2012.01080.x.
25.
Morganstein, D.R. and D.A. Marker. 1997. “Continuous Quality Improvement in Statistical Agencies.” In Survey Measurement and Process Quality , edited by L.E. Lyberg, P. Biemer, M. Collins, E.D. de Leeuw, C. Dippo, N. Schwarz, and D. Trewin. (pp. 475–500). New York: John Wiley & Sons.
26.
Murphy, J., D. Mayclin, A. Richards, and D. Roe. 2016. “A Multi-method Approach to Survey Pretesting.” In 2015 FCSM Research Conference Proceedings . Available at: https://fcsm.sites.usa.gov/files/2016/03/D3_Murphy_2015FCSM.pdf. (accessed November 26, 2017).
27.
Murphy, J., P. Biemer, M. Duprey, and R. Chew. 2017. “Interactive Adaptive Total Design Reports for Near Real-Time Survey Monitoring.” Presented at the 2017 Conference of the European Survey Research Association. Lisbon, Portugal.
28.
Schouten, B., F. Cobben, and J. Bethlehem. 2009. “Indictators of Representativeness of Survey Nonresponse.” Survey Methodology 35: 101–113.
29.
Schouten, B., A. Peytchev, and J. Wagner. 2017. Adaptive Survey Design . Boca Raton, FL: Chapman and Hall/CRC.
30.
Tufte, E. 2001. The Visual Display of Quantitative Information (2nd ed.). Cheshire, CT: Graphics Press. ISBN 0-9613921-4-2.
31.
U.S. Census Bureau. 2015. American Community Survey (ACS) 2014 Data Release New and Noteable. Available at: https://www.census.gov/programs-surveys/acs/news/data-releases/2014/release.html#par_textimage_12. (accessed November 21, 2017).
32.
Zimmer, S., P. Biemer, P. Kott, and C. Berry. 2016. “Testing a Model-Directed, Mixed Mode Protocol in the RECS Pilot Study.” In 2015 FCSM Research Conference Proceedings . Available at: https://s3-amazonaws-com-443.web.bisu.edu.cn/sitesusa/wp-content/uploads/sites/242/2016/03/G2_Zimmer_2015FCSM.pdf. (accessed November 26, 2017).