Examining Variation in Survey Costs Across Surveys

Abstract

Self-administered surveys may be administered with a single mode or mixed data collection modes. How mixing modes of data collection affects survey costs is not well understood. We examine whether cost structures differ for mail-only versus web+mail mixed-mode surveys, what design features are associated with costs, and whether survey costs are associated with response rates. Using administrative survey cost data from two academic survey centers, we find that survey costs per sampled unit and per complete vary substantially across individual surveys. The average cost per sampled unit is surprisingly similar across mail-only and web+mail surveys. How the budget is allocated across printing, postage, incentive, and staff time varies across these designs: printing and postage costs are higher in mail-only surveys, and more of the budget is allocated to incentive costs and project management costs in web+mail surveys. Furthermore, higher cost surveys are associated with higher response rates, particularly for incentive costs.

Keywords

survey costs survey methods mail surveys mixed-mode surveys response rates

Introduction

Survey organizations face multiple decisions in designing surveys, but perhaps no design decision is more substantial than the mode or modes of data collection (de Leeuw 2005). Although possible coverage, nonresponse, and measurement errors factor into selecting a mode, survey costs are also pivotal in the decision process (Olson 2021; Olson et al. 2021; Olson, Wagner, and Anderson 2020). As response rates to interviewer-administered surveys, including telephone and in-person surveys, have fallen and survey costs have risen (Dutwin and Buskirk 2020; Luiten, Hox, and de Leeuw 2020), survey organizations are increasingly moving to self-administered modes—namely, web and mail (Olson et al. 2021).

In household-based surveys that use a mailed recruitment letter, sample members may be asked to complete an enclosed paper survey or alternatively to complete a web survey online, often referred to as a “push-to-web” design (Dillman, Smyth and Christian 2014). Previous examinations of costs of mail and mixed-mode web+mail surveys focus largely on the variable costs of printing and mailing materials for recruitment and survey completion, sometimes including the variable labor costs for data entry and assembling the mailings (Greenlaw and Brown-Welty 2009; Griffis, Goldsby and Cooper 2003; Han, Montaquila and Brick 2013; Olson 2021; Sinclair et al. 2012). Printing and postage are not the only costs that are incurred, however, and may not be a good reflection of fixed costs related to technology, project management-related labor costs, or other costs in mail and mixed-mode web+mail surveys (which may be a large part of the total budget; see, for instance, Roberts and Vandenplas 2017; Zuidgeest et al. 2011). Fixed costs are those which do not vary with sample size, and may include overall project management, questionnaire design and development, mode-specific recruitment protocols, programming of instruments and data entry forms, sample design, and costs of web survey software, among others. Variable costs vary with sample size and—in addition to costs for printing and mailing questionnaires and recruitment letters and incentive costs—include data entry of paper surveys and incentives, among others. In a mixed-mode survey, the fixed costs may be higher because both modes need to be fully designed, but the variable costs may be lower in a mixed-mode design with potentially fewer paper questionnaires included in mailings and lower printing and data entry costs. The need for higher incentives to offset lower response rates in surveys that contain a web mode (Daikeler, Bošnjak and Lozar Manfreda 2020), however, could mitigate the potential savings in variable cost. Surprisingly, whether both fixed and variable costs differ across mail and mixed-mode web+mail surveys are largely unknown.

Furthermore, different survey designs may yield different numbers of completes. Thus, the cost reflected in a total budget per sampled case (e.g., what is the budget to field a survey with a total fielded sample size of n) may differ from the cost per responding unit (or cost per complete; e.g., what is the budget to field a survey with an ultimate respondent data set size of size r). How the cost per sampled unit and cost per complete vary across different types of survey costs differs for mail-only surveys versus web+mail mixed-mode surveys is not well-understood. Finally, whether more expensive surveys yield higher response rates, one indicator of the risk of survey nonresponse error, is also unknown. To address these uncertainties, we examine survey costs across multiple mail-only and mixed-mode web+mail surveys conducted by two academic survey centers between January 1, 2018 and December 31, 2019. In this paper, we address the following research questions:

RQ1: What is the fixed and variable cost structure for single-mode mail-only and mixed-mode web+mail self-administered survey data collections overall?

RQ2: Do costs differ between single-mode mail-only and mixed-mode web+mail self-administered survey data collections?

RQ3: Are costs for these single-mode mail-only and mixed-mode web+mail self-administered surveys associated with other design features? Do the effects of design features on costs differ by mode?

RQ4: Are costs related to survey response rates for single-mode mail-only and mixed-mode web+mail self-administered surveys? Does the association between costs and response rate differ by mode?

Background

Survey costs are an important constraint on survey design decisions (Groves 1989). For a given budget, survey researchers select many aspects of a survey design, including mode of data collection, length of a questionnaire, target population and sample frame, and incentives, among many other design features. Perhaps the biggest driver of both survey costs and the risk for survey errors is the mode of data collection, leading researchers to seek less expensive modes when other costs rise (Link et al. 2008; Olson et al. 2021). As more resources are required to counteract declining response rates, survey practitioners increasingly have been turning to self-administered modes of data collection (Dillman, Smyth, and Christian 2014; Olson et al. 2021), including mail-only and mixed-mode surveys that include both web and mail modes, sometimes called a “push-to-web design.”¹

Despite a generally understood hierarchy of survey modes by costs (e.g., face-to-face is more expensive than telephone which is more expensive than self-administered surveys) (de Leeuw 2005), how different types of self-administered survey designs vary in costs is less well-studied. One reason for limited research on survey costs across studies has to do with different operationalizations of survey costs (Olson, Wagner, and Anderson 2020). Comparing costs across the limited studies available is difficult because they often operationalize costs differently.

The most commonly reported cost measures across studies are costs per sampled unit and costs per complete, dividing expenditures on a study by the overall fielded or responding sample size. For instance, if a survey designer knows that a first-class stamp costs 63 cents, an envelope costs 13 cents, and the cost of printing a letter on letterhead is 20 cents, then the materials cost for printing and mailing one letter is $0.63+$0.13+$0.20 = $0.96 per sampled unit. Some of the inputs into the costs per sampled unit are constant across studies (e.g., the postage rate is set in the US by the US Postal Service), and thus may be easily transported from one study to another. Other inputs—such as the amount of time spent by project staff or by data entry or other production staff or costs related to technology—vary across studies and across organizations and are likely also a function of design features of the study itself.

Costs per complete inflate the costs per sampled unit to account the proportion of sampled units who end up in the final data set. For instance, if the final response rate for the above example study is 25%, then the cost per complete for printing and mailing one letter is inflated by the inverse of the response rate (1/0.25) ($0.96/0.25 = $3.84 per complete). Because the study protocol and other design features (e.g., population, topic, burden, sponsor) can affect response rates, costs per complete may be more likely to vary across different design features such as data collection modes than costs per sampled unit. Thus, transporting costs per complete across studies may be difficult without knowing more about study design features.

A third metric is relative costs, the proportion of the total costs that are spent in each category of expenditures. It is largely unknown how much of a total budget is allocated to variable costs–the focus of most cost studies–versus fixed costs. We may be interested in the proportion of the total costs spent on variable costs (incentives, postage, or printing, for instance) as measures that are commonly used to evaluate the costs of different self-administered surveys, and the proportion that went to fixed costs. Only a couple of studies examine fixed versus variable costs, finding large proportions of a survey budget allocated to fixed costs. For instance, Zuidgeest et al. (2011) report that 41.5% of total costs in a mail-only survey of breast cancer patients went to fixed costs, including general costs, information technology, and post-survey processing costs. In the mixed-mode web+mail version of the survey, 58.4% went to fixed costs. The complement in each budget (58.5% in the mail-only survey and 41.6% in the mixed-mode web+mail survey) was spent on variable costs, including printing, postage, and labor for preparing the mailings. Thus, changes in mode may change the proportion of the budget allocated to fixed and variable costs.

Costs also can be reported using either monetary or nonmonetary measures (Olson 2021; Olson, Wagner, and Anderson 2020). Monetary cost measures are reported in the currency appropriate to the study (e.g., US dollars; Euros); nonmonetary cost measures include all other ways of evaluating costs, including labor hours or numbers of contact attempts. Because some survey organizations consider monetary costs to be proprietary, nonmonetary cost measures are reported to avoid revealing sensitive information such as individual pay rates. Additionally, nonmonetary measures are likely to be more transportable across organizations for cost categories such as staff labor time where different labor markets and job tenure may yield quite different pay rates for the same type of work. For instance, we may be interested in how many hours in a project are spent by higher-level permanent project staff versus production data staff. Thus, following Olson, Wagner, and Anderson (2020), we will examine measures that include monetary and nonmonetary costs per sampled unit, costs per complete, and relative costs in different categories of expenditures.

Cost Study Designs

To understand how different survey costs are associated with design features and field outcomes, researchers need multiple survey implementations that vary in design features. Such studies are difficult to conduct and rare to find. As a compromise, researchers often examine survey costs within one survey administration, with experimental examinations the most common method for exploring how variation in single or packages of design features may affect costs (Biemer et al. 2018; Dykema et al. 2013; Wagner 2019). For instance, Dykema et al. (2021) randomly assigned physicians to different incentive conditions, finding that a prepaid $5 incentive in the first mailing with a prepaid $5 in the second mailing was the most cost-effective incentive allocation when examining variable costs per complete among the study's experimental conditions. Observational examinations of costs across time for repeated survey administrations are another method of obtaining variation in survey costs. For instance, Wagner, Guyer and Evanchek (2020) examined seasonal versus over time effects on costs per interview using interviewer timesheets and expense reports across almost seven years of data collection for the National Survey of Family Growth (NSFG). They find evidence of both seasonality of costs and an increase over time in the per-unit monetary cost of interviewers on the NSFG.

Although these survey- and experiment-specific cost studies are informative, they are necessarily limited. First, single-survey cost studies accumulate costs on one survey at a time and often on one design feature at a time. Thus, how costs differ across design features that vary across studies but not within the survey generally are not examined. Second, single-survey examinations tend to focus on variable costs related to data collection, assuming similar fixed costs for data collection (Dykema et al. 2021). As a result, how time is spent by different types of personnel on a project–often considered part of fixed costs—is not well understood (Olson, Wagner, and Anderson 2020; Olson 2021) and likely is differentially impacted by the inclusion of multiple modes. Finally, aggregating costs across single-survey cost studies is difficult because they vary substantially in how and what costs are reported (Olson, Wagner, and Anderson 2020). As a result, cross-survey examinations may facilitate general statements about trends in costs (e.g., self-administered surveys tend to be no more expensive than a telephone survey for the same study; Olson et al. 2021), but not more specific statements about the relative contributions of different design features.

An alternative method for understanding how survey design features drive costs is to accumulate multiple surveys that vary in design features and use an observational approach to examine costs (Groves 1989; Olson 2021). Observational studies require a common operationalization of the cost metrics and design features, a difficulty that results in few and limited examinations of cross-survey costs. For instance, Presser and McCulloch (2011) examined budgets for federal statistical agencies and number of respondent-hours spent on surveys (measured in over 20 years using data extracted from Office of Management and Budget submission packages), finding increases in both survey respondent-hours and in survey costs over the time period. Gelman, Stevens, and Chan (2003) make a series of model-based assumptions about survey costs to meta-analyze the relationship between incentives, response rates, and costs in telephone surveys, and then use that meta-analysis to simulate the cost savings incurred through use of a prepaid incentive on reducing the number of telephone calls. Beyond these studies, a more thorough examination of costs across surveys with different design features has not been done and may be most easily facilitated within a survey organization that can extract cost information with a consistent definition.

Costs for Mail Versus Mixed-Mode Web+Mail Surveys

Web surveys are thought to reduce survey costs relative to other modes (Tourangeau, Conrad, and Couper 2013), especially mail surveys. It may be that mixed-mode web+mail surveys require more project management—for instance, both a web survey and data entry software need to be programmed, data across modes needs to be merged, and tracking mechanisms require double checking multiple systems—possibly shifting costs from one category of expenditures (e.g., data entry staff) to another (e.g., project management staff).

To help understand how costs may vary across modes of data collection, Table 1 lists types of infrastructure, variable, and fixed costs that are incurred at each stage of survey development and data collection in mail-only and mixed-mode web+mail surveys, along with where costs may differ across mail-only and mixed-mode web+mail surveys. Unfortunately, studying costs at the level of detail described in Table 1 is very difficult or impossible to do without a specific tracking study (Judkins, Waksberg, and Northrup 1990) or development of special cost management systems. Nevertheless, because of the paucity of information publicly available on survey costs, it is useful to consider what costs may be incurred for different tasks in as finely grained detail as possible, even if the current measurement systems do not permit disentangling costs in these categories.

Table 1.

Types of Cost Drivers for Self-Administered Surveys by Survey Development and Collection Stages and Tasks.

Infrastructure Costs (Examples)	Study-Specific Fixed Costs (Examples)	Study-Specific Variable Costs (Examples)	Similar or different for mail-only vs. mixed-mode web+mail?
Sampling
Development of sampling frame	Designing sample Sample processing and formatting Loading sample into sample management system (Infrequent) Assigning sample to experimental conditions	Creating or updating sample frame Tracking or locating known sample members Purchasing sample frame	Similar
Questionnaire Development
Software for mail questionnaire layout Software for web questionnaire	Concept mapping Question drafting Pretesting questionnaire (may be budgeted/paid for separately) Mail: Formatting/designing questionnaire layout Web: Programming questionnaire Web: Testing questionnaire (Infrequent) Developing experimental conditions	Mail: Questionnaire printing	Different—Higher project staff costs related to questionnaire design, layout, and programming in mixed-mode data collection than mail-only surveys (H2).
Recruitment and Data Collection
Sample management system software Software to send recruitment emails	Developing IRB package Mail: Development of recruitment letters and postcards Web: Development of emails Development of incentive protocol Preparing protocol of recruitment material packages Running progress reports on data collection effort (Infrequent) Administration of experimental conditions	Printing for recruitment letter(s) Printing for recruitment postcard(s) Incentives Stuffing mailings Envelopes for mailing survey packages Postage for mailing for survey packages Return postage for completed mail surveys Tracking returns into sample management system Answering telephone help desk and/or email questions from sample members	Different—Mixed-mode web+mail surveys will have lower printing, postage, mailing, and data entry costs (H1). Incentive costs will depend on the study, perhaps reallocating production costs to incentives (H3).
Data Processing
Statistical software	Programming of data entry instrument/Adapting web instrument for data entry of paper questionnaires Data cleaning Harmonizing data across modes into one dataset Developing design weights Developing nonresponse adjustment and/or poststratification weights Creating analytic variables	Data entry for mail surveys Coding open-ended responses	Different—higher labor costs in mixed-mode web+mail surveys for project staff because of more cleaning and harmonizing data sets across modes (H2). Lower labor costs for data entry staff in mixed-mode surveys (H1).
Reporting
Word processing software Publishing software	Writing methodology reports Writing reports of findings		Similar

Cost drivers in self-administered surveys are a mix of supply, personnel, and infrastructure costs, as well as design-specific factors such as incentives, sampling frames, and postsurvey processing (Table 1). Supply and mailing costs—including postage for mailing out and return mailing of paper questionnaires, printing of letters and paper questionnaires, and outgoing and inbound reply envelopes—are perhaps the best understood portions of costs for self-administered surveys. Overall, cost studies have shown that including a web option in a mixed-mode survey reduces variable “production” costs per complete compared to mail alone (Delnevo and Singh 2021; Greenlaw and Brown-Welty 2009; Kaplowitz, Hadlock, and Levine 2004; McMaster et al. 2017; Olson et al. 2021). As shown in the last column of Table 1, we anticipate that printing, postage, and production labor costs per complete, costs per sampled unit, and relative costs will be lower in mixed-mode surveys than in mail-only surveys (H1).

Self-administered surveys require work by project staff to lay out a paper questionnaire, program and debug a web questionnaire, and write recruitment materials for each contact attempt. Personnel can include more senior project managers whose time may reflect project complexity related to sample design, questionnaire design, and/or programming or formatting a questionnaire and more junior data processing staff whose time may reflect data entry or coding of open-ended responses. Personnel costs, including the costs of project managers, data processing staff, and IT staff, among others, are difficult to measure and thus less understood than supply (printing and mailing) costs, and can vary over labor markets. Thus, it is useful to examine labor costs in terms of hours that are spent by different types of staff, allowing comparability across job categories within and across organizations for positions that may differ in pay rates.

In reporting of costs for mail-only and mixed-mode web+mail surveys, project staff labor costs (traditionally considered to be fixed costs) are often omitted or assumed to be constant across designs (Dykema et al. 2021). This omission may give a distorted view of the relative costs of mail-only and mixed-mode web+mail surveys. If the additional project management costs of administering two modes of data collection and of programming a web instrument along with layout of the questionnaire is greater than cost savings on printing and postage, then it is possible to infer that mixed-mode surveys are less expensive than single-mode mail-only surveys when cost savings is allocated to that of printing and postage, but not total costs for the project. We anticipate that project staff costs per sampled case and relative costs for project staff will be higher for mixed-mode surveys than for mail-only surveys because of the extra complexity of managing these surveys (H2a). To the extent that mixed-mode surveys bring in more respondents than mail-only surveys, project staff costs per complete may be lower in mixed-mode surveys than mail-only surveys (H2b).

Many mail-only and mixed-mode web+mail surveys use incentives to increase participation rates (Olson et al. 2021; Singer and Ye 2013). If these self-administered surveys reallocate funds that would have been used for data entry costs to incentive costs, then mixed-mode surveys will have higher incentive costs per sampled case and per complete as well as higher relative costs for incentives (H3a). As a result, the total costs per sampled unit (that is, the total budget) may be similar across these two sets of studies (H3b), but higher incentive costs may increase response rates and—in combination with lower printing and postage costs per complete—thus yield mixed-mode studies that have lower costs per complete than mail-only surveys (H3c). In sum, costs are likely to differ across project staff time, production staff time, printing, postage, and incentives for mail-only and web+mail surveys.

Costs and Design Features for Mail Versus Mixed-Mode Web+Mail Surveys

Surveys vary dramatically in design features, all of which may affect costs. Longer surveys, for example, likely yield higher printing and postage costs for self-administered surveys. Similarly, because different sample frames permit different types of studies (topics that are more salient; different types of contact attempts), it is important to understand how sample frames affect cost structures overall and for costs in surveys with different modes of data collection. We consider a few design features here.

Number of pages

Overall, we expect that longer surveys will yield higher costs, but that the association between length and costs will differ by mode (H4). Because the cost of printing a paper survey increases as a function of the number of pages in a questionnaire, survey length is likely to be a large driver of printing costs; longer surveys will be heavier and cost more to mail. Thus, we anticipate that longer surveys (measured by the number of pages in a paper questionnaire) will have higher printing and postage costs, especially in mail-only surveys. Longer paper surveys also require more data entry, and thus we anticipate higher production staff costs for longer questionnaires, especially in mail-only surveys. In mixed-mode surveys, longer surveys require more programming costs and more complex data merging across modes of data collection. Thus, we anticipate higher project staff costs for longer questionnaires, especially for mixed-mode surveys. Because longer surveys are anticipated to have lower response rates, survey designers may strategically use incentives to overcome the possible deleterious effect of longer surveys. Thus, incentive costs may be higher with long surveys, regardless of mode.

Number of contact attempts

Contacting respondents in mail-only surveys requires sending a mailing to the sampled person. Mailings often contain a cover letter, a questionnaire, and possibly other information such as FAQs or an incentive. Each additional contact attempt in a mail-only survey increases total costs of printing and mailing materials. In a mixed-mode web+mail survey, mailed recruitment materials often include the same materials as in a mail-only survey; in some instances, however, a contact attempt may include an emailed notification if email addresses are available on the frame (e.g., for listed samples) or a mailed letter containing only login information without a paper questionnaire, possibly reducing the costs of contact. Unlike in interviewer-administered surveys, the maximum number and content of these mailings, including prepaid incentives, are often planned in advance before data collection launches. Additionally, survey organizations may tend to use similar planned study designs across studies, such that the maximum number of contact attempts sent to sampled units may have little variation over different types of study designs (e.g., a four-mailing design for users of the Tailored Design Method (Dillman, Smyth, and Christian 2014)). Thus, the maximum number of contacts used for a study may not be strongly related to survey costs per sampled case or per complete in either mail-only or web+mail surveys (H5a). Indirectly, mail or mixed-mode surveys with more planned contact attempts may anticipate having lower response rates overall and thus offer higher incentive values as a mitigating design feature, thus increasing costs. In a competing hypothesis, in a survey that uses mailed recruitment materials, additional contact attempts are likely to require additional printing and postage costs in order to print the materials and project and production staff hours to get the contact attempts ready and send out the materials, increasing the costs per sampled unit and relative costs allocated to printing and postage, especially in mail-only surveys (H5b).

List versus address-based sample (ABS) frame

The sample frame determines what kind of information can be used for recruitment. Names, addresses, or email addresses are all common in list frames, but ABS frames of the general population tend to have only addresses that are reliable and complete; names and email addresses are sometimes merged on by sample providers, but often have low levels of accuracy or completeness (Harter et al. 2016). Because of the presence of multiple types of contact information, list frames may require slightly more project staff time to effectively use when tailoring contact materials but cost less overall if the frame does not need to be purchased (e.g., it is supplied by the researcher). List frames may be particularly likely to save costs when the email address available on the frame is used for at least one contact attempt.

In addition to contact information availability, the use of a list frame may package a collection of other design features that affect survey participation rates, and thus affect costs per complete in several domains. For instance, list frames are often used for surveys that are topically targeted to subgroups (e.g., medical professionals; students; teachers), possibly yielding higher response rates because of greater topic interest. List frames may also be used for target populations of individuals whose characteristics tend to be associated with higher response rates—and possibly selection of modes—across studies (e.g., higher levels of education; greater affiliation with the sponsor; more likely to have internet access). Furthermore, studies using list frames of professionals may be more likely to use higher incentive levels to persuade sample members. Thus, we expect differences in costs per sampled unit and costs per complete across total, printing, postage, and incentive costs as well as production staff across list frames and ABS frames, with the greatest savings coming in costs per complete for list frames, because of the associated package of design features that occur when studies use list frames (H6).

Sample size

Clearly, it is more costly overall to field larger samples. Because our costs are per-unit costs (costs per sampled unit; costs per complete) or relative costs, however, we do not expect an association between our cost measures and sample size across studies (H7).

Year of data collection

Although data collection costs have changed (and increased) over time (Presser and McCulloch 2011), year-to-year fluctuations are likely to be minor unless there are substantial exogenous shocks (e.g., a global pandemic). In the years considered in the analysis here (2018–2019), we do not anticipate any differences in costs across the two years of data collection (H8).

Survey Costs and Survey Errors

Surprisingly little research has examined how survey costs and survey errors are related to each other across studies (Gelman, Stevens, and Chan 2003). This lack of research likely arises because of the difficulty identifying an error measure that can be defined consistently across studies and in obtaining similar cost measurements across studies.

Survey response rates are a well-established data quality measure with a common definition that permits comparisons across studies (American Association for Public Opinion Research 2023). Although response rates are not a good indication of nonresponse error on individual survey estimates (Brick and Tourangeau 2017; Groves 2006; Groves and Peytcheva 2008), response rates are a data quality metric that is commonly used as a proxy for evaluating the risk of survey errors. Additionally, response rates themselves are a good measure of sample yield—how much of a fielded sample returns a questionnaire—and thus are of interest in their own right. In general, whether higher cost mail-only or web+mail mixed-mode surveys yield higher response rates (addressing RQ4)—and how that varies by types of costs—is an important and unanswered question.

In sum, this paper examines how different survey costs relate to each other, how they vary by survey design features, and whether survey costs are related to survey response rates. We now turn to an empirical evaluation of these questions.

Data and Methods

Surveys

This study examines survey costs across two academic survey organizations—the Bureau of Sociological Research (BOSR) at the University of Nebraska-Lincoln (UNL) and the University of Wisconsin Survey Center (UWSC) at the University of Wisconsin-Madison (UW-Madison). These survey organizations share many attributes: they are located at state universities in the Midwest; they conduct surveys in many modes, including telephone and in-person studies (not examined here); and they serve a similar client base of university faculty and state agency researchers. Clients typically hire these organizations for their methodological knowledge and advice regarding designing studies to obtain high response rates with cost-effective designs.

Despite extensive similarities between these two organizations, they differ in several ways. First, wages for both professional staff and part-time staff are slightly higher at the UWSC based on economic and political differences (e.g., higher cost of living in Madison, living wage policies at UW-Madison). Second, the organizations manage certain aspects of mixed-mode surveys differently and in ways that affect costs, including differences in what services are used for printing and mailing. Third, two design factors vary between the centers that affect costs: BOSR's studies typically include business reply envelopes for the return envelopes in paper survey mailings, while the UWSC uses first-class stamped return envelopes; while use of incentives is largely determined by the client, the UWSC is much more likely to use cash pre-incentives in survey mailings and include post-incentives for both mail and web surveys than BOSR. These differences in environment, staffing, and survey implementation may lead to differences in costs across the surveys that are fielded by these two survey organizations. Thus, our analyses below control for the survey organization in multivariable models.

The target populations of surveys for our examination are mail, web, or web+mail mixed-mode surveys conducted between January 1, 2018 and December 31, 2019 that had at least one mailed recruitment letter. We excluded web surveys that only included recruitment by email. As a result, the sample frames for each study included mailing addresses from list-based or ABS samples. Although the list-based samples may include those employed in specific professions (e.g., medical professionals, teachers), the target population for each of these surveys are persons, not establishments. Because of the fundamentally different cost drivers for interviewer-administered phone and in-person surveys (e.g., costs related to interviewer and supervisor wages and travel), these types of surveys are also excluded from this analysis.

Data

We identified 16 studies at BOSR and 21 studies at the UWSC that met the criteria of being conducted with at least one paper mailing during 2018–2019, yielding a total of 37 studies on which we can examine total costs and 36 on which we can examine other cost measures. One study only had total costs available. For consistency, we report only on the 36 studies for which all costs are available (conclusions do not change with one additional study included).

Cost data are from three administrative sources. First, the accounting systems maintain records for costs incurred in several categories. This information forms the dependent variables for our cost analysis. We obtained the total costs charged for the project, total printing cost, total postage costs, and total incentive costs as monetary cost measures. The total cost is all costs charged to a project, including labor costs, as recorded in the accounting system and thus are what Olson, Wagner, and Anderson (2021) refer to as observed costs. Because labor costs vary across employee categories and survey shops, to have a measure that is comparable across studies, we obtained the total number of staff hours charged to the project, including the total number of project staff (e.g., project manager; questionnaire design; sample design) and production staff (e.g., data entry; data collection) hours. We examine monetary costs overall and on printing, postage, and incentives, all in US dollars and non-monetary costs measured in hours spent by study staff overall and for project staff (e.g., senior staff and project managers) versus production staff (e.g., data entry staff). Because of the variation in size of studies, we examine costs per sampled unit (the total number of sampled units that were sent the initial invitation; referred to as costs per sampled case) and per responding unit (the total number of completed questionnaires; referred to as costs per complete). We also calculate the proportion of total monetary costs charged to printing, postage, and incentives as a measure of relative costs for supplies and the proportion of total staff hours charged by project staff and production staff as a measure of relative labor costs.

Second, we coded several design features from the methodology reports produced for each project, including mode(s), year of data collection, sample frame, number of pages for the questionnaire, maximum number of contact attempts, study dates, fielded (total sample) and achieved (total respondents) sample size, and number of completes. This information forms the independent variables for our analysis. Other design information was coded, but was empirically confounded with mode of data collection or other design features (e.g., only web+mail surveys used email addresses which were only available on list frames) and thus omitted from our analyses.

Finally, we obtained response rates (AAPOR RR2) from the sample tracking paradata files for each survey. This information allows us to examine whether survey costs are associated with response rates as an error indicator overall. Because of the small number of studies, all analyses combine data from the two survey organizations.

Table 2 contains descriptive statistics on study characteristics. More than half of the studies were conducted in 2018 (58.3%) (versus 41.7% in 2019). One-third of the studies were administered as mixed-mode studies (conducted using either a web questionnaire with mailed contact attempts or sequential web+mail mixed-modes, following de Leeuw (2005) and Dillman, Smyth, and Christian (2014)'s definitions of mixed-mode surveys; also see Online Supplement Tables A1 and A2) (30.6%). The average study fielded 4650 (SD = 5604) sampled units (not completed surveys). Roughly half of the studies used an ABS frame. On average, the surveys were 9 pages long (average of 143 questions). All studies had at least two contact attempts, with an average of about four contact attempts (3.9) and a range of two to eight attempts. The average response rate was 35.1% (AAPOR RR2).

Table 2.

Survey Design Features for Mail and Mixed-Mode Surveys at BOSR and UWSC with at Least One Mail Contact Attempt, 2018 and 2019.

Variable	n	Mean/%	SD
Mode
Mail-only	25	69.4%
Mixed mode web+mail	11	30.6%
Survey Design Features
Number of questionnaire pages	36	9.0	6.8
Number of contact attempts (including postcards)	36	3.9	1.3
Frame
ABS	18	50.0%
List	18	50.0%
Year
2018 (−2019)	21	58.3%
2019 (−2020)	15	41.7%
Fielded sample size	36	4649.9	5603.6
Survey Error Indicator
Response rates	36	35.1	16.1

Analysis Plan

The analysis examines how costs vary across survey design features using Stata 17. To address RQ1, we start with a univariate examination of each of the individual cost measures to explore to what extent costs vary across studies overall. Given the small number of studies, we then conduct two-sample independent t-tests to examine how the cost measures differ across mail-only and mixed-mode surveys to address RQ2.

We then examine how costs vary across design features, and across mail-only and mixed-mode web+mail surveys, examining RQ3. Using linear regression models, we predict our cost measures with an indicator for mode of data collection and the survey design feature of interest, controlling for survey organization (UWSC = 0, BOSR = 1). We then add an interaction effect between the design feature and mode to evaluate whether the effect of the design feature differs across mail-only and web+mail surveys, estimating:

{Cost}_{i} = β_{0} + β_{1} {MixedMode}_{i} + β_{2} {DesignFeature}_{i} + β_{3} {MixedMode}_{i} \times {DesignFeature}_{i} + β_{4} {Shop}_{i} + e_{i}

We calculate average marginal effects overall and for each type of mode. To test the difference in the effect of the design feature across modes, we used the mchange command (part of SPost13; Long and Freese 2014), margins, and mlincom in Stata 17.

Finally, we examine whether the cost measures are associated with final response rates (RQ4). We estimate linear models predicting the final response rate (from 0 to 100) as a function of mode, the survey cost measure, and an interaction effect between modes and survey costs. The cost models control for survey organization to account for overall differences in both response rates and costs across the two shops, using marginal effects to evaluate the association between costs and response rates overall and for mail-only and web+mail surveys:

{RespRate}_{i} = β_{0} + β_{1} {MixedMode}_{i} + β_{2} {Costs}_{i} + β_{3} {MixedMode}_{i} \times {Costs}_{i} + β_{4} {Shop}_{i} + e_{i}

Results

RQ1: What is the fixed and variable cost structure for single-mode mail-only and mixed-mode web+mail self-administered survey data collections overall?

Table 3 contains descriptive information on survey costs. Costs per sampled unit allow us to understand how the initial budget varies across surveys. That is, if a researcher is planning a new survey, in what areas are costs likely to differ across these two mode decisions or other design features. Costs per complete allow us to understand how costs to achieve a desired respondent pool may differ across design decisions and are a function of all inputs that combine to yield a final respondent pool. Because costs per complete reflect survey participation differences across studies and are thus dividing costs by a random variable, studies with the same costs per sampled unit can have different costs per complete. Additionally, we expect that costs per complete will be more variable than costs per sampled unit.

Table 3.

Survey Costs for Mail-Only and Web+Mail Mixed-Mode Surveys with at Least One Mail Contact Attempt.

		Per sampled unit ($)		Per complete ($)		Percent of total costs
	N	Mean	SD	Mean	SD	Mean	SD
Total costs	36	16.89	15.73	41.46	22.18	100.0%	--
Printing costs	36	1.68	1.10	6.15	5.24	15.1%	12.5
Postage costs	36	2.17	1.36	7.02	4.67	17.1%	8.4
Incentive costs	36	3.26	6.89	4.84	7.70	10.4%	14.2
Non-labor variable costs	36	7.11	7.53	18.01	10.64	42.6%	13.1
Residual fixed costs and labor costs	36	9.78	8.86	23.44	13.70	57.3%	13.1
Staff hours	36	0.23	0.16	0.66	0.40	100.0%	--
Project staff hours	36	0.10	0.09	0.26	0.21	40.6%	19.1
Production staff hours	36	0.13	0.09	0.41	0.25	59.4%	19.1

Note: n = 36 studies. Non-labor variable costs are printing, postage, and incentive costs. Residual fixed costs and labor costs are Total costs—Non-labor variable costs.

The average cost per sampled unit was $16.89 (SD = $15.73), with $1.68 spent on printing (SD = $1.10), $2.17 spent on postage costs (SD = $1.36), and $3.26 spent on incentive costs (SD = 6.89). This means that a total of $7.11 per sampled unit (SD = $7.53) was spent on variable costs related to mailing and incentives and $9.78 (SD = $8.86) per sampled unit on fixed costs and labor costs. On average, staff spent about 0.23 h per sampled unit (SD = 0.16), with about 0.10 h for project staff (SD = 0.09) and 0.13 h for production staff (SD = 0.09) per sampled unit.

Costs per complete are higher and more variable than costs per sampled unit, as expected. The average overall cost per complete is $41.46 (SD = $22.18), with $6.15 spent on printing (SD = $5.24), $7.02 spent on postage (SD = $4.67), and $4.84 spent on incentives (SD = $7.70), totaling $18.01 per complete (SD = $10.64) on printing, postage, and incentive costs and $23.44 (SD = $13.70) per complete on other costs. On average, staff spent 0.66 h per completed questionnaire (SD = 0.40), with 0.26 h for project staff (SD = 0.21) and 0.41 h for production staff (SD = 0.25).

Finally, we examine the percentage of total costs for each of the monetary cost categories and the percentage of labor hours by types of staff. For these studies, about 15% of the total budget is allocated to printing costs, 17% to postage costs, and 10% to incentive costs. Overall, about 42.6% of the total budget is made up of variable costs from printing, postage, and incentives. Staff hours are made up of about 40.6% of project staff time and about 59.4% of production staff time.

As can be seen in the standard deviations reported in Table 3, costs vary across studies (Groves 1989). This heterogeneity is critical to our understanding of how survey design features may contribute to variation in costs. Figures 1 and 2 show the distribution of monetary costs and staff hours across the studies. The variation of these cost measures is large, from $3.81 to $74.76 per sampled case and from $9.77 to $110.50 per complete for total costs and from 0.06 to 0.93 total staff hours per sampled case and from 0.12 to 1.76 total staff hours per complete. Thus, there is no single cost for a self-administered study, reflecting a variety of design features, study budgets, and real-world conditions.

Figure 1.

Total observed, printing, postage, incentive, variable non-labor (printing+postage+incentive), and fixed costs per sampled case and per complete in US dollars (n = 36 studies).

Figure 2.

Total staff hours, project staff hours, and production staff hours per sampled case and per complete (n = 36 studies).

RQ2: Do costs differ between single-mode mail-only and mixed-mode web+mail self-administered survey data collections?

We now turn to the next research question—whether costs differ between mail-only surveys and web+mail mixed-mode surveys (Table 4). First, overall costs per sampled case do not statistically differ between mail-only and web+mail mixed-mode surveys (mail-only: $14.78; web+mail mixed-mode: $21.71; p = .28). Thus, for the same sample size, we find no evidence that the overall budget for a survey will vary across mail-only and mixed-mode web+mail surveys. However, the cost components do differ. We hypothesized that printing, postage, and production labor costs per sampled unit would be lower for mixed-mode surveys than mail-only surveys (H1), and that project staff hours per sampled unit would be higher for mixed-mode than mail-only surveys (H2a). Consistent with H1, printing and postage costs in mail-only surveys are about twice those in web+mail mixed-mode surveys (Printing: $2.01 for mail-only vs. $0.95 for web+mail, p = .005; postage: $2.56 for mail-only vs. $1.27 for web+mail, p = .001). We anticipated that mixed-mode surveys would either have higher incentive costs (H3a) or the same incentive costs (H3b) per sampled case than mail-only surveys. Incentive costs are statistically similar (p = .15), inconsistent with H3a but consistent with H3b, although there is a trend toward higher incentive levels in mixed-mode surveys ($1.77 mail-only; $6.65 web+mail). Inconsistent with H2a, overall staff hours are similar for mail-only and mixed-mode web+mail surveys (p = .62), as are project staff hours (p = .28). Consistent with H1, production staff hours differ—in mail-only surveys, production staff work about 0.15 h per sampled case compared to 0.09 h per sampled case in web+mail surveys (p = .03).

Table 4.

Survey Costs for Mail-Only and Web+Mail Mixed-Mode Surveys with at Least One Mail Contact Attempt.

	Per sampled case			Per complete			Percent of total costs
	Mail Only (mean/SE)	Web+mail (mean/SE)	p-Value	Mail Only (mean/SE)	Web+mail (mean/SE)	p-Value	Mail Only (mean/SE)	Web+mail (mean/SE)	p-Value
Total costs ($)	14.78 (2.70)	21.71 (5.98)	.31	47.13 (4.46)	28.56 (4.91)	.01	100.0%	100.0%
Printing costs ($)	2.01 (0.21)	0.95 (0.27)	.005	7.45 (0.90)	3.21 (1.75)	.05	18.0% (2.3)	8.5% (3.8)	.05
Postage costs ($)	2.56 (0.28)	1.27 (0.24)	.001	8.88 (0.79)	2.80 (0.97)	.0001	20.1% (1.2)	10.5% (3.0)	.01
Incentive costs ($)	1.77 (0.87)	6.65 (3.05)	.15	4.27 (1.47)	6.14 (2.62)	.54	6.5% (1.7)	19.4% (6.1)	.07
Total nonlabor variable costs	6.34 (1.16)	8.87 (3.20)	.47	20.59 (1.98)	12.15 (3.10)	.03	44.5% (2.3)	38.4% (4.7)	.26
Total fixed costs (total-variable nonlabor)	8.44 (1.60)	12.84 (3.11)	.23	26.54 (2.94)	16.41 (2.36)	.01	55.6% (2.3)	61.6% (4.7)	.26
Staff hours	0.24 (0.04)	0.21 (0.04)	.62	0.79 (0.07)	0.37 (0.09)	.001	100.0%	100.0%
Project staff	0.09 (0.02)	0.12 (0.03)	.28	0.29 (0.05)	0.18 (0.03)	.06	34.3%	54.9%	.02
Production staff	0.15 (0.02)	0.09 (0.02)	.03	0.50 (0.04)	0.19 (0.19)	.002	65.7%	45.1%	.02

Notes: p-Values from two-sided t-tests. Mail-only n = 25; Web+mail n = 11.

We see notable differences, however, in the costs per complete and the proportion of the total costs spent on different types of expenditures across these two types of studies. On average, mail-only surveys have higher overall costs ($47.13 vs. $28.56, p = .0095), higher printing costs ($7.45 vs. $3.21, p = .05), and higher postage costs per complete ($8.88 vs. $2.80, p = .0001) than web+mail mixed-mode surveys. This difference is consistent with the prevailing understanding of cost savings in mixed-mode surveys over mail-only surveys and with H1 and H3c, which hypothesized higher costs per complete for mail-only surveys compared to web+mail mixed-mode surveys. Inconsistent with H3a, however, which hypothesized higher incentive costs for mixed-mode surveys, incentive costs, are similar ($4.27 vs. $6.14, p = .54). We hypothesized that project staff and production staff costs per complete would be lower for mixed-mode surveys than mail-only surveys (H1, H2b). Consistent with H1 and H2b, mail-only surveys also require more staff hours per complete (0.79 vs. 0.37, p = .001), primarily in production staff (0.50 vs. 0.19, p = .006); the time spent by the higher-level project staff is similar across these two types of designs, with a trend toward more project staff time in mail-only surveys (0.25 vs. 0.18, p = .06), inconsistent with H2b.

When examined as the percent of the total costs (i.e., relative costs) (Figure 3), 18.0% of the total costs, on average, is spent on printing costs for mail-only surveys, compared to 8.5% of the total costs in mixed-mode web+mail surveys, consistent with H1 (p = .05). Postage costs make up 20.1% of the total costs for mail-only surveys and 10.5% for mixed-mode surveys, consistent with H1 (p = .01). In contrast, and consistent with H3a, incentives are 6.5% of the total costs for mail-only surveys and 19.4% for mixed-mode web+mail surveys (p = .07). Overall, the amount spent on printing, postage, and incentives totals a similar percentage of the budget for these two types of surveys (consistent with H3b, p = .26), but how the budget is allocated differs dramatically. Allocation of staff time varies across the two modes of data collection. In mail-only surveys, project staff make up only 34.3% of staff time, whereas project staff make up 54.9% of staff time in web+mail surveys (p = .02), consistent with H2a. Conversely, production staff contribute 65.7% of staff time in mail-only surveys, reflecting higher mailing and data entry costs, falling to 45.1% of staff time in web+mail surveys (p = .02), consistent with H2b.

Figure 3.

Relative monetary and nonmonetary costs by category overall for mail-only (n = 25) and web+mail (n = 11) surveys.

RQ3: Are costs for these single-mode mail-only and mixed-mode web+mail self-administered surveys associated with other design features? Do the effects of design features on costs differ by mode?

Next, we examine whether design features are associated with survey costs overall and differentially across mail-only and mixed-mode web+mail surveys. We report average marginal effects from the linear models—which can be interpreted as the combined effect of the first-order terms and interaction term in models with interaction effects—to help with interpretation (Figures 4, 5, and 6, and Online Supplement Table A3). It is important to note that these are observational analyses; design features covary in important ways that cannot be easily disentangled in multivariable analyses with only 36 studies. We note these limitations in our interpretation of the model results below.

Figure 4.

Average marginal effects of survey design features on survey costs overall (n = 36) and for mail (n = 25) and web+mail (n = 11) surveys.

Figure 5.

Average marginal effects of survey design features on staff hours overall (n = 36) and for mail (n = 25) and web+mail (n = 11) surveys.

Figure 6.

Average marginal effects of survey design features on percent of total costs overall (n = 36) and for mail (n = 25) and web+mail (n = 11) surveys.

Number of pages

We hypothesized that longer surveys would be more expensive, but that the association would vary by mode (H4). Consistent with H4, questionnaires with additional pages cost more per sample case and per complete overall (Figure 4). For instance, one additional page increases the overall cost per sampled case by $2.12 (represented by circles) and the overall cost per complete by $2.62 for these studies. In the studies examined here, an additional page in a questionnaire increased printing costs by $0.16 per case and $0.35 per complete, postage costs by $0.12 per case, but had no overall effect on postage costs per complete and was associated with having higher incentive costs ($0.74 per case and $0.97 per complete). One additional page increased staff hours by 0.026 h per case and by 0.042 per complete (Figure 5). These associations differ across mail-only (represented by squares) and web+mail (represented by diamonds) mixed-mode surveys. One additional page increases costs per sampled case by $2.45 in a mail-only survey, but by $1.39 in a web+mail mixed-mode survey; this difference increases to $3.45 per additional page in a mail-only survey but only $0.75 per additional page in a mixed-mode survey when examining costs per complete, a statistically significant difference (p = .024). An additional page costs $0.22 more in printing costs in a mail-only survey per sampled case, but only $0.02 more in a web+mail survey (Figure 7, p < .0001), a difference that is exacerbated when examining costs per complete (mail-only: $0.53 vs. web+mail: $0.05, p = .018).

Figure 7.

Predicted printing costs per case by number of pages in the questionnaire, mail-only and web+mail surveys.

Other costs—including labor hours (Figure 5)—follow this pattern of larger increases for additional pages. We find higher costs per sampled case in mail-only surveys than in web+mail surveys (thus indicating higher budgets for both types of studies with additional pages), but with substantial increases in costs per complete for an additional page of the questionnaire for mail-only surveys and little to no increases for additional pages in web+mail surveys (consistent with H4). Interestingly, additional questionnaire pages generally are not associated with the percent of total costs (i.e., relative costs, Figure 6).

Number of Contact Attempts

We next examined whether the maximum number of contact attempts was associated with costs. We had competing hypotheses for the number of contact attempts and costs. We hypothesized that either the number of contact attempts would not be associated with costs because there is little variation in contact attempts across studies and the total number of attempts typically are preplanned (H5a) or that the number of contact attempts would be associated with higher printing, postage, and production staff hours in order to get these materials ready (H5b). In these analyses, we discovered a strange pattern—mail-only surveys with more contact attempts had lower costs but web-mail mixed-mode surveys had higher costs with more contact attempts, inconsistent with H5a and H5b. There appeared to be two phenomena at play. First, the mail-only surveys had a restricted range—only 2, 3, or 4 contact attempts were used—whereas the web+mail surveys had between 2 and 8 contact attempts, including emailed contact attempts. Thus, a model that averaged over all of these contact attempts was borrowing strength beyond the range of possible contact attempts for mail-only surveys. Second, one mail-only survey had only two contact attempts but had extremely high costs. This one survey was driving the negative association between the number of contact attempts and costs. When we exclude studies that had two contact attempts (one mail-only survey; one web-mail survey), these counterintuitive associations disappear, yielding virtually no association between the number of contact attempts and the monetary costs as observed here (consistent with H5a but not H5b).

Relative costs should be somewhat immune to this problem, and indeed we see that more contact attempts yield slightly more costs allocated to printing overall—one additional contact attempt increases the relative printing cost by 3.7 percentage points and especially for mail-only surveys (an increase of 5.8 percentage points of the total cost to printing) but not for mixed-mode surveys (a difference that is statistically significant, p = .011). Similarly, one additional contact attempt is associated with a 4.7 percentage point increase in relative postage costs for mail-only surveys but a 2.8 percentage point decrease for web-mail surveys, also statistically different (p = .007). Mixed-mode surveys with more contact attempts use incentives (one additional contact attempt is associated with a 5.5 percentage point increase in incentive costs), but not mail-only surveys (p = .009). We do not see a shift in allocation of staff hours with one additional contact attempt in either mode.

List-Based Versus ABS Sample Frames

We expected lower costs per complete and lower labor hours for studies that used list frames compared to ABS frames, largely due to the package of information available on list frames (H6). Unexpectedly, there were no clear differences in costs per sampled case between list-based and ABS sample frames overall or in mail-only or web+mail mixed-mode surveys. We also see this pattern for costs per complete, where the difference in costs per complete were not significantly different between list-based and ABS frames in either mail-only or web+mail mixed-mode surveys. However, the difference between list versus ABS frames across mail-only and web+mail surveys is statistically significant for total costs, printing costs, postage costs, total variable costs, staff hours, and production staff hours; list frames have lower costs per complete than ABS frames in web+mail mixed-mode surveys than in mail-only surveys, consistent with H6. Additionally, the relative costs for printing, postage, and production staff hours are lower for list frames in web+mail surveys than for mail-only surveys, but the relative costs for project staff are higher (Figure 6, p < .05). There are no clear relative differences in incentive costs for list versus ABS frames for mail-only surveys (p > .05) or across the modes and frames (p = .195).

In addition to compositional differences in who is contacted on a list frame versus a general population ABS frame and topic salience differences, interpretation of this difference between list-based versus ABS sample frames is confounded by an important design feature—the presence of email addresses on list-based frames—that are exclusively used in web+mail surveys. There are only three list-frame studies in our sample that do not have email addresses available, reflecting the general nature of list frames for known populations. When we statistically control for the presence of an email address, the general patterns maintain—lower costs for list frames in web+mail surveys than in mail-only surveys, likely reflecting the general collection of respondent characteristics and study design features that differentiate these frames from each other.

Sample Size

We expected no clear association between sample size and costs per sampled case across any of the modes of data collection (H7), and that is what we see, consistent with H7. We also expected no clear association between sample size and costs per complete or relative costs, but this expectation is not supported. An increase of 100 sample units is associated with a decrease in the per-complete sample cost of $0.07 overall and of $0.15 in mail-only surveys. This seems to be largely driven by decreases in staff hours—project staff and production staff each have 0.001 fewer hours for every increase of 100 sample units, possibly reflecting some gains of scale. When examining relative costs, every 100 additional sample units increases relative postage costs for mixed-mode surveys by 0.19% and increases the relative production staff hours by 0.56% (similarly, decreasing relative project staff hours by 0.56%).

Year

We expected no differences in costs across the two years considered here (H8), and that is largely what we observe. Only incentive costs were lower in 2019 (by $8.56) than in 2018 in web+mail surveys, a difference that is statistically significantly different (p = .033) from the $1.71 (nonsignificant) increase in mail-only surveys. This then is associated with a change in total nonlabor variable costs over the two years in web-mail surveys (a decrease of $10.09), but not in mail-only surveys ($1.25 nonsignificant increase over the years), a pattern which differs across the two modes (p = .038). Similar percentage decreases are seen in relative costs. No other differences between the two years are found.

RQ4: Are costs related to survey response rates for single-mode mail-only and mixed-mode web+mail self-administered surveys? Does the association between costs and response rate differ by mode?

The association between costs and final response rates varies across cost categories and operationalizations of costs (Table 5). We did not state directional hypotheses for the associations examined here, although we are interested in whether higher cost surveys also have higher response rates. Overall costs per case are strongly associated with the response rate. Each additional dollar spent per case is associated with a 0.45 percentage point increase in response rates, an association that is similar for both mail-only and web+mail mixed-mode surveys. This is largely driven by incentive costs; each additional dollar spent on incentives per sampled case is associated with a 1.20 percentage point increase in response rates. One additional hour per case spent by staff (a unit that is equal to 6 standard deviations of staff hours) is associated with a 29.2 percentage point increase in response rates. Furthermore, when we examine residual fixed costs (including all labor costs) outside of printing, postage, and incentive costs per case, there is a positive association between costs and response rates. Each additional dollar spent on costs outside of printing, postage, and incentives is associated with a 0.78 percentage point increase in response rates, a rate that does not vary across modes. Thus, higher survey costs per case are indeed associated with higher response rates.

Table 5.

Marginal Effects Predicting Overall Response Rates by Survey Costs.

	Overall	Mail-only	Web+Mail	p-Value mail vs. web+mail
Per case
Total cost	0.453**	0.424*	0.514**	0.686
Printing cost	3.175	4.208	0.827	0.482
Postage cost	3.229	0.379	9.705*	0.078
Incentive cost	1.199**	1.283*	1.008*	0.626
Non-labor variable cost	0.924**	0.910*	0.955**	0.924
Residual fixed costs and labor costs	0.781***	0.726*	0.905**	0.669
Total staff hours	29.180*	28.797*	30.049	0.969
Project staff hours	45.491	50.471	34.173	0.730
Production staff hours	43.774	50.010	29.602	0.742
Per complete
Total cost	−0.007	−0.039	0.059	0.725
Printing cost	−0.580	−0.221	−1.397	0.154
Postage cost	−1.662**	−1.598*	−1.809	0.883
Incentive cost	0.727**	0.529	1.176**	0.182
Non-labor variable cost	0.065	−0.021	0.262	0.556
Residual fixed costs and labor costs	−0.072	−0.074	−0.067	0.989
Total staff hours	−9.950	0.833	−34.455*	0.021
Project staff hours	−17.442	2.832	−63.061	0.061
Production staff hours	−12.159	−1.130	−37.225*	0.108
Relative costs
Printing	−0.221	−0.038	−0.636	0.095
Postage	−0.732*	−0.772*	−0.639	0.799
Incentives	0.395	0.344	0.513**	0.592
Non-labor variable costs	0.120	0.063	0.249	0.564
Residual fixed costs and labor costs	−0.120	−0.063	−0.249	0.564
Project staff hours	0.179	0.198	0.137	0.818
Production staff hours	−0.179	−0.198	−0.137	0.818

Note: *p < .05, ** p < .01, ***p < .001; Models control for the survey shop. Mail-only n = 25; Web+mail n = 11.

The association between survey costs per complete and response rates are quite different from costs per sampled case. We would expect that more efficacious designs would yield lower costs per complete; that is, higher response rates would be associated with lower costs per complete. Yet we do not see any clear association here. Incentive costs per complete are associated with higher response rates—one additional dollar spent per sample complete is associated with a 0.727 percentage point increase in the response rate overall, and is higher for mixed-mode web+mail surveys. Higher postage costs are associated with lower response rates—each additional dollar spent on postage is associated with lower response rates by 1.7 percentage points. Overall, survey costs per complete generally are not associated with response rates.

Conclusion

The goal of this study was to examine whether survey costs differed across mail-only and mixed-mode web+mail surveys across two academic survey organizations in the Midwest. We also aimed to understand how design features are associated with survey costs and examine whether costs are associated with response rates. The hypotheses and results are summarized in Table 6.

Table 6.

Summary of Hypotheses and Findings for Survey Costs.

	Mixed-mode vs. mail-only	Survey length (H4)	Number of contact attempts (H5a, H5b)	List vs. ABS (H6)	Sample size (H7)	Year (H8)	Response Rates
Overall
Hypothesis	No association (H3b)/Mixed-mode lower costs than mail (H3c)	Longer surveys have higher costs	No association (H5a)	List lower costs than ABS	No association	No association	Higher costs have higher response rates
Result—Per Case	Not significant	Longer surveys have higher costs	Not significant	Not significant	Not significant	Not significant	Higher costs have higher response rates
Result—Per complete	Mixed-mode lower costs than mail	Longer surveys have higher costs, especially mail	Not significant	Not significant	Larger samples have lower costs, especially in mail	Not significant	Not significant
Hypothesis confirmed?	Yes (H3c)	Yes	Yes (H5a)	No	No	Yes	Yes
Printing
Hypothesis	Mixed-mode lower costs than mail (H1)	Longer surveys have higher costs, especially mail	No association (H5a)/More contacts have higher costs, esp. mail (H5b)	List lower costs than ABS	No association	No association	Higher costs have higher response rates
Result—Per Case	Mixed-mode lower costs than mail	Longer surveys have higher costs, especially mail	Not significant	Not significant	Not significant	Not significant	Not significant
Result—Per complete	Mixed-mode lower costs than mail	Longer surveys have higher costs, especially mail	Not significant	Not significant	Not significant	Not significant	Not significant
Hypothesis confirmed?	Yes	Yes	Yes (H5a)	No	Yes	Yes	No
Postage
Hypothesis	Mixed-mode lower costs than mail (H1)	Longer surveys have higher costs	No association (H5a)/More contacts have higher costs, especially mail (H5b)	List lower costs than ABS	No association	No association	Higher costs have higher response rates
Result—Per Case	Mixed-mode lower costs than mail	Longer surveys have higher costs, esp. mail	Not significant	Not significant	Not significant	Not significant	Higher costs have higher response rates
Result—Per complete	Mixed-mode lower costs than mail	Not significant	Not significant	Not significant	Not significant	Not significant	Higher costs have lower response rates
Hypothesis confirmed?	Yes	Yes	Yes (H5a)	No	Yes	Yes	Yes/No
Incentive
Hypothesis	Mixed-mode higher costs than mail (H3a)	Longer surveys have higher costs	No association (H5a)	List higher costs than ABS	No association	No association	Higher costs have higher response rates
Result—Per Case	Not significant	Longer surveys have higher costs	Not significant overall, but more contacts have higher costs in mixed-mode	Not significant	Not significant	Not significant, but differences in mixed-mode	Higher costs have higher response rates
Result—Per complete	Not significant	Longer surveys have higher costs, especially mail	Not significant	Not significant	Not significant	Not significant	Higher costs have higher response rates
Hypothesis confirmed?	No	Yes	No (H5a)	No	Yes	Yes	Yes
Project staff
Hypothesis	Mixed-mode higher (per case, H2a)/lower (per complete, H2b) costs than mail	Longer surveys have higher costs, especially mixed-mode	No association (H5a)/More contacts have higher costs, especially mail (H5b)	List lower costs than ABS	No association	No association	Higher costs have higher response rates
Result—Per Case	Not significant	Longer surveys have higher costs, especially mail	Not significant	Not significant	Not significant	Not significant	Higher costs have higher response rates
Result—Per complete	Mixed-mode lower costs than mail	Longer surveys have higher costs, especially mail	Not significant	Not significant	Not significant overall, but larger samples have lower costs for mail	Not significant	Not significant
Hypothesis confirmed?	Yes (H2b)	Yes	Yes (H5a)	No	No	Yes	Yes
Production staff
Hypothesis	Mixed-mode lower costs than mail (H1)	Longer surveys have higher costs, especially mail	No association (H5a)/More contacts have higher costs, esp. mail (H5b)	List lower costs than ABS	No association	No association	Higher costs have higher response rates
Result—Per Case	Mixed-mode lower costs than mail	Longer surveys have higher costs, especially mail	Not significant	Not significant	Not significant	Not significant	Not significant
Result—Per complete	Mixed-mode lower costs than mail	Longer surveys have higher costs, especially mail	Not significant	Not significant overall, but list lower cost than ABS in mixed-mode	Not significant overall, but larger samples have lower costs for mail	Not significant	Not significant
Hypothesis confirmed?	Yes	Yes	Yes (H5a)	Yes	Yes	Yes	No

Our first research question was about the cost structure across fixed and variable costs in these two organizations. Costs vary considerably across surveys. Printing, postage, and incentive costs are among the most commonly examined measures of costs in self-administered and mixed mode surveys. These factors combined made up about 40% of the total budget for these studies, with more of the total budget in the surveys examined here allocated to incentives in the mixed-mode surveys. The time spent by project staff is rarely accounted for in cost analyses and is notably higher per complete in mail-only surveys than in web+mail mixed-mode surveys, despite the added complexity of study design. Thus, cost analyses that examine only printing and production-related costs will have a substantially incomplete picture of survey costs; the “fixed” costs of doing a survey by the more senior project staff make up a large fraction of the total costs associated with total costs per case. It is worth examining other drivers of fixed costs, including the complexity of the study design, challenges of frame construction or item development, or other factors that are not accounted for here.

Importantly, costs do differ across mixed-mode surveys. Costs per complete—but not necessarily costs per sampled case—are lower in mixed-mode surveys overall, and for printing, postage, project staff, and production staff, but not for incentives. This means that survey designers will have a more cost-efficient design combining both web and mail modes—at least, as a sequential web+mail mixed mode design or a web survey with mailed recruitment materials—but the overall budget may not differ between the two modes of data collection. Few of the survey design features that we examined were systematically associated with survey costs; only survey length was associated with costs per case and costs per complete across the different cost categories. The maximum number of mailings, use of a list frame, sample size, and the year of data collection generally were not associated with survey costs.

Higher response rate surveys generally have higher incentive costs, upholding the longstanding finding of use of incentives and higher response rates. Additionally, when examining costs per case, postage costs, total fixed costs (including labor costs), and total staff hours are associated with higher survey response rates, suggesting that allocating more resources overall to surveys is associated with higher response rates. When we examine costs per complete, however, these cost measures are generally not associated with response rates (other than incentives). The difference in the associations here suggests that how one measures costs is important when examining whether costs and errors are associated.

We took advantage of having two survey shops to examine cost structures. Here, the association between design features and costs is largely similar across these two shops, although the modal design features are quite different across these two shops. We accounted for this statistically with the survey organization as a control variable in our multivariable models; however, other design features are confounded with mode of administration and cannot be effectively evaluated here. For instance, only list samples had email addresses available, and only mixed-mode surveys used email recruitment. Thus, we cannot disentangle the use of a list sample from the use of email for at least one of the recruitment attempts. Furthermore, the mail-only surveys examined here capped the total number of contact attempts at four, whereas the mixed-mode surveys had up to eight contact attempts. This restricted range means that we cannot effectively examine how the maximum number of contact attempts is related to survey costs in mail-only surveys. None of the mixed-mode surveys examined here used concurrent mixed-mode designs in which a questionnaire and URL are sent to the sampled household simultaneously beginning with the first mailing; survey practice at the time of these studies largely followed the advice of Dillman, Smyth, and Christian (2014, Guideline 11.16) to use sequential mixed-mode designs. We also combined sequential mixed-mode designs that mailed a paper questionnaire and those that used web questionnaires with mailed recruitment materials; both are mixed-mode designs and are generally similar until a paper questionnaire is sent. Additional analyses show similarity in the costs for these two types of mixed-mode designs for the included studies in almost all of the categories examined here other than printing costs, but the number of studies of each of those types of designs examined is quite small (see Online Supplement Tables A1 and A2). Furthermore, with only 36 studies under consideration, any single outlying study can have a strong effect on our conclusions. This is especially true when examining the effect of design features on survey costs. For instance, a counterintuitive negative association between maximum number of contact attempts and costs was explained entirely by one outlying survey. Additionally, we examined only two years—2018 and 2019—prior to the COVID-19 pandemic. Because data collection procedures in 2020 and 2021 were different due to COVID-19 lockdown requirements, we did not consider studies conducted then to be from an equivalent target population of surveys. Future research could examine how cost structures for surveys changed during the COVID-19 pandemic years and beyond.

There are three takeaways for survey practitioners from these findings. First, there is no single “cost” for conducting a mail-only or web+mail mixed-mode survey. Surveys are heterogeneous in target population, topic, length, and implementation protocol. Furthermore, sampled units vary in their decisions to participate, leading to greater variation in costs per complete than costs per sampled case. Survey practitioners can use this heterogeneity to their advantage when planning budget estimates. If an average project uses about 0.1 h per sampled case of project management time, then for a sample size of n = 1500, a typical budget might assume 150 (=0.1*1500) hours of project management time. As seen in Figure 2, these data are not normally distributed; assuming the median of this distribution (0.06 h per case) would yield an estimate of 90 h (=0.06*1500) of project management time and the 75^th percentile (0.13 h per case) would yield an estimate of 195 h (=0.13*1500). This range is wide, but it provides survey organizations with concrete information about the risk of “getting it wrong” in project estimates. As we have seen, cost information varies with design features; additional inputs about these design features could refine the initial estimate and range. Survey organizations with cost and design information in hand for their own projects could empirically evaluate the costs incurred for projects with a given design or sample size. When data are aggregated across time and multiple surveys, this kind of data-driven planning could help survey organizations provide more accurate budget estimates to clients as well as anticipate the risk of cost overruns.

Second, survey designers often use the design features examined here—modes, number of questionnaire pages, number of contact attempts, type of frame, and sample size—as inputs into a survey budget. The results from these analyses confirm some common budgetary assumptions—mixed-mode surveys have lower costs and longer questionnaires have higher costs—and question other common assumptions—surveys with more planned contact attempts (among the limited range in the studies examined here) do not necessarily have higher costs per case or per complete. Empirical evaluation of how design features affect costs can help survey organizations advocate for designs that might improve response rates or other measures of data quality while not dramatically affecting survey costs.

Finally, survey practitioners should consider how these different design decisions may act together to affect response rates. Incentives—both a cost driver and a design decision—are associated with higher response rates. In the studies examined here, longer questionnaires had higher costs in virtually every domain, including having higher incentive costs. Thus, one might expect even more deleterious effects of longer questionnaires on survey costs, especially costs per complete, were incentives not used for these longer surveys. Survey practitioners should anticipate these joint effects of design features on costs and response rates; future analyses with more studies that examine these joint effects of design features and costs on response rates are needed.

Survey costs are understudied, but a critically important driver of survey design decisions. In mixed-mode surveys versus single-mode mail-only surveys, costs can be reallocated from the costs of processing paper surveys to other design features such as incentives that help increase response rates. As one of the only examinations of survey costs across surveys and survey organizations, this study is a first step to understanding how important design features such as mode of data collection are associated with survey costs. Future research should examine additional cost drivers, sources of survey costs, and modes of data collection.

Supplemental Material

sj-docx-1-smr-10.1177_00491241241298914 - Supplemental material for Examining Variation in Survey Costs Across Surveys

Supplemental material, sj-docx-1-smr-10.1177_00491241241298914 for Examining Variation in Survey Costs Across Surveys by Kristen Olson, John Stevenson, Nadia Assad, Lindsey Witt-Swanson, Cameron P.E. Jones, Amanda Ganshert and Jennifer Dykema in Sociological Methods & Research

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Kristen Olson

Lindsey Witt-Swanson

Amanda Ganshert

Jennifer Dykema

Data Availability Statement

The datasets generated during and/or analyzed during the current study are available in the Open Science Framework repository, https://osf.io/vr43w/?view_only=775fac084d504d9897387adefb79485a (). The individual survey cost data are confidential and cannot be shared publicly.

Supplemental Material

Supplemental material for this article is available online.

Notes

Author Biographies

Kristen Olson is the Leland J. and Dorothy H. Olson Professor in Sociology and Director of the Bureau of Sociological Research at the University of Nebraska-Lincoln. Her research examines errors that occur in surveys, including mixed-mode surveys, interviewer effects, within-household selection in self-administered surveys, and questionnaire design.

John Stevenson is Emeritus Associate Director of the University of Wisconsin Survey Center. His research is on methods to increase survey response rates, effective use of incentives and survey costs.

Nadia Assad is a Senior Project Director at the University of Wisconsin Survey Center. Her research is on survey participation in self-administered surveys.

Lindsey Witt-Swanson is a Research Director at NORC at the University of Chicago. Her research is on survey error and mode choice.

Cameron Jones is a Ph.D. student in the Department of Statistics at the University of Wisconsin. His research is on dimensionality reduction and unsupervised learning.

Amanda Ganshert is the Assistant Director for Research and Methods at the Bureau of Sociological Research at the University of Nebraska-Lincoln. Her research focuses on how survey design features such as incentives, modes, and invitation messages affect survey participation and sample composition in mail and mixed-mode surveys.

Jennifer Dykema is Associate Professor of Sociology at the University of Wisconsin and Faculty Director at the University of Wisconsin Survey Center. Her research is on identifying sources of errors produced in the process of gathering standardized measurements and developing and implementing methods to reduce those errors with a focus on interviewer-respondent interaction, questionnaire design, and methods to increase response rates.

References

American Association for Public Opinion Research. 2023. “Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys.” American Association for Public Opinion Research.

Biemer

Paul P.

Murphy

Joe

Zimmer

Stephanie

Berry

Chip

Deng

Grace

Lewis

Katie

. 2018. “Using Bonus Monetary Incentives to Encourage Web Response in Mixed-Mode Household Surveys.” Journal of Survey Statistics and Methodology 6(2):240‐61.

Brick

J. Michael

Tourangeau

Roger

. 2017. “Responsive Survey Designs for Reducing Nonresponse Bias.” Journal of Official Statistics 33(3):735‐52.

Daikeler

Jessica

Bošnjak

Michael

Manfreda

Katja Lozar

. 2020. “Web Versus Other Survey Modes: An Updated and Extended Meta-Analysis Comparing Response Rates.” Journal of Survey Statistics and Methodology 8(3):513‐39.

de Leeuw

Edith

. 2005. “To Mix or Not to Mix Data Collection Modes in Surveys.” Journal of Official Statistics 21(2):233‐55.

Delnevo

Cristine D.

Singh

Binu

. 2021. “The Effect of a Web-Push Survey on Physician Survey Responses Rates: A Randomized Experiment.” Survey Practice 14:1. no pp.

Dillman

Don A.

Smyth

Jolene D.

Christian

Leah Melani

. 2014. Internet, Phone, Mail, and Mixed Mode Surveys: The Tailored Design Method. Hoboken: John Wiley & Sons.

Dutwin

David

Buskirk

Trent D.

. 2020. “Telephone Sample Surveys: dearly Beloved or Nearly Departed? Trends in Survey Errors in the Era of Declining Response Rates.” Journal of Survey Statistics and Methodology 9(3):353‐80.

Dykema

Jennifer

Stevenson

John

Assad

Nadia

Kniss

Chad

Taylor

Catherine A.

. 2021. “Effects of Sequential Prepaid Incentives on Response Rates, Data Quality, Sample Representativeness, and Costs in a Mail Survey of Physicians.” Evaluation & the Health Professions 44:235‐44.

10.

Dykema

Jennifer

Stevenson

John

Klein

Lisa

Kim

Yujin

Day

Brendan

. 2013. “Effects of E-Mailed Versus Mailed Invitations and Incentives on Response Rates, Data Quality, and Costs in a Web Survey of University Faculty.” Social Science Computer Review 31(3):359‐70.

11.

Gelman

Andrew

Stevens

Matt

Chan

Valerie

. 2003. “Regression Modeling and Meta-Analysis for Decision Making.” Journal of Business & Economic Statistics 21(2):213‐25.

12.

Greenlaw

Corey

Brown-Welty

Sharon

. 2009. “A Comparison of Web-Based and Paper-Based Survey Methods: Testing Assumptions of Survey Mode and Response Cost.” Evaluation Review 33(5):464‐80.

13.

Griffis

Stanley E.

Goldsby

Thomas J.

Cooper

Martha

. 2003. “Web-Based and Mail Surveys: A Comparison of Response, Data, and Cost.” Journal of Business Logistics 24(2):237‐58.

14.

Groves

Robert M

. 1989. Survey Errors and Survey Costs. New York: John Wiley & Sons, Inc.

15.

Groves

Robert M

. 2006. “Nonresponse Rates and Nonresponse Bias in Household Surveys.” Public Opinion Quarterly 70(5):646‐75.

16.

Groves

Robert M.

Peytcheva

Emilia

. 2008. “The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis.” Public Opinion Quarterly 72(2):167‐89.

17.

Han

Daifeng

Montaquila

Jill M.

Michael Brick

. 2013. “An Evaluation of Incentive Experiments in a Two-Phase Address-Based Sample Mail Survey.” Survey Research Methods 7(3):207‐18.

18.

Harter

Rachel

Battaglia

Michael P.

Buskirk

Trent D.

Dillman

Don A.

English

Ned

Fahimi

Mansour

Frankel

Martin R.

Kennel

Timothy

McMichael

Joseph P.

McPhee

Cameron

Montaquila

Jill M.

Yancey

Tracey

Zukerberg

Andrew

. 2016. “Task Force Report on Address-Based Sampling.” American Association for Public Opinion Research. Available from https://aapor.org/wp-content/uploads/2022/11/AAPOR_Report_1_7_16_CLEAN-COPY-FINAL-2.pdf

19.

Judkins

David

Waksberg

Joseph

Northrup

Doris

. 1990. “Cost Functions for NHIS and Implications for Survey Design.” Proceedings of the American Statistical Association, Survey Research Methods Section: 34‐43. Available from http://www.asasrms.org/Proceedings/papers/1990_006.pdf.

20.

Kaplowitz

Michael D.

Hadlock

Timothy D.

Levine

Ralph

. 2004. “A Comparison of Web and Mail Survey Response Rates.” Public Opinion Quarterly 68(1):94‐101.

21.

Link

Michael W.

Battaglia

Michael P.

Frankel

Martin R.

Osborn

Larry

Mokdad

Ali H.

. 2008. “A Comparison of Address-Based Sampling (ABS) Versus Random-Digit Dialing (RDD) for General Population Surveys.” Public Opinion Quarterly 72(1):6‐27.

22.

Long

J. Scott

Freese

Jeremy

. 2014. Regression Models for Categorical Dependent Variables Using Stata. 3rd ed. College Station: Stata Press.

23.

Luiten

Annemieke

Hox

Joop

de Leeuw

Edith

. 2020. “Survey Nonresponse Trends and Fieldwork Effort in the 21st Century: Results of an International Study Across Countries and Surveys.” Journal of Official Statistics 36(3):469‐87.

24.

McMaster

Hope Seib

LeardMann

Cynthia A.

Speigle

Steven

Dillman

Don A

and Team Millennium Cohort Family Study. 2017. “An Experimental Comparison of Web-Push vs. Paper-Only Survey Procedures for Conducting an In-Depth Health Survey of Military Spouses.” BMC Medical Research Methodology 17(1):73.

25.

Olson

Kristen

. 2021. “Unpacking the Black Box of Survey Costs.” Research in Social and Administrative Pharmacy 17(7):1342‐46.

26.

Olson

Kristen

Smyth

Jolene D

Horwitz

Rachel

Keeter

Scott

Lesser

Virginia

Marken

Stephanie

Mathiowetz

Nancy A

McCarthy

Jaki S

O’Brien

Eileen

Opsomer

Jean D

Steiger

Darby

Sterrett

David

Jennifer

Suzer-Gurtekin

Z Tuba

Turakhia

Chintan

Wagner

James

. 2021. “Transitions From Telephone Surveys to Self-Administered and Mixed-Mode Surveys: Aapor Task Force Report.” Journal of Survey Statistics and Methodology 9(3):381‐411.

27.

Olson

Kristen

Wagner

James

Anderson

Raeda

. 2021. “Survey Costs: Where are We and What is the Way Forward?” Journal of Survey Statistics and Methodology 9(5):921‐42.

28.

Olson

Kristen

Stevenson

John

Assad

Nadia

Witt-Swanson

Lindsey

Jones

Cameron P.E.

Ganshert

Amanda

Dykema

Jennifer

. 2024. Data and Stata Code for “Examining Variation in Survey Costs across Surveys”. doi: https://doi.org/10.17605/OSF.IO/VR43W.

29.

Presser

Stanley

McCulloch

Susan

. 2011. “The Growth of Survey Research in the United States: Government-Sponsored Surveys, 1984–2004.” Social Science Research 40(4):1019‐24.

30.

Roberts

Caroline

Vandenplas

Caroline

. 2017. “Estimating Components of Mean Squared Error to Evaluate the Benefits of Mixing Data Collection Modes.” Journal of Official Statistics 33(2):303.

31.

Sinclair

Martha

O’Toole

Joanne

Malawaraarachchi

Manori

Leder

Karin

. 2012. “Comparison of Response Rates and Cost-Effectiveness for a Community-Based Survey: Postal, Internet and Telephone Modes With Generic or Personalised Recruitment Approaches.” BMC Medical Research Methodology 12(1):132.

32.

Singer

Eleanor

Cong

. 2013. “The Use and Effects of Incentives in Surveys.” The Annals of the American Academy of Political and Social Science 645(1):112‐41.

33.

Tourangeau

Roger

Conrad

Frederick G.

Couper

Mick P.

. 2013. The Science of Web Surveys. New York: Oxford University Press.

34.

Wagner

James

. 2019. “Estimation of Survey Cost Parameters Using Paradata.” Survey Practice 12:1. no pp.

35.

Wagner

James

Guyer

Heidi

Evanchek

Chrissy

. 2021. “Using Time Series Models to Understand Survey Costs.” Journal of Survey Statistics and Methodology 9(5): 943-60.

36.

Zuidgeest

Marloes

Hendriks

Michelle

Koopman

Laura

Spreeuwenberg

Peter

Rademakers

Jany

. 2011. “A Comparison of a Postal Survey and Mixed-Mode Survey Using a Questionnaire on Patients’ Experiences With Breast Care.” Journal of Medical Internet Research 13(3):e68.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB