Are Prepaid Monetary Incentives Sufficient for Reducing Panel Attrition and Optimizing the Response Rate? An Experiment in the Context of a Multi-Wave Panel with a Sequential Mixed-Mode Design

Abstract

French

In this contribution, we evaluate the short- and long-term effects of a prepaid cash incentive on young people’s cooperation and response rate in the fourth and fifth wave of a panel with sequential mixed-mode design (online questionnaire, CATI). Analyses are based on a survey experiment of students from randomly selected school classes of equal shares, which have participated in the third wave. Findings show that a monetary incentive has a direct and positive effect on the response rate in the fourth but not in the subsequent wave. However, the effect of the incentive is not persistent, since the effect weakens and fades away during the field phase and cannot be directly transferred to the second survey mode. As emphasized in the tailored design method (TDM), a monetary incentive can contribute to a shorter field phase and hence lower costs, but it is an insufficient instrument against panel attrition and the optimization of the retention rate when other strategies are disregarded.

Keywords

incitations attrition de panel taux de réponse expérimentation incentives panel attrition sequential mixed-mode design response rate survey experiment

Introduction

In sociological research on enrolment in education, acquisition of skills, and the attainment of certificates, panel studies became prominent in gathering longitudinal data. These panel studies, such as TREE (Transition from education into employment; TREE, 2016), DAB (Determinanten der Ausbildungswahl und der Berufsbildungschancen; Glauser, 2015), or NEPS (National Educational Panel Study; Blossfeld et al., 2011), provide detailed information on educational trajectories, including states, events, and their time references. However, observing individuals and their trajectories in a panel design is challenging and costly in terms of the number of individuals lost in subsequent panel waves (Andreß et al., 2013). For analysing these educational careers without vacancies, it is pivotal to avoid either missing values (due to systematic unit nonresponse in each of the panel waves) or panel attrition, since incomplete information such as left-censored data usually results in biased findings and estimations with minor statistical precision (for example, Groves and Peytcheva, 2008). In panel studies, therefore, several strategies are used against panel attrition to optimize the response rate and retention rates. Most of them, including a personalized advance letter, prepaid monetary incentive, reminders, and mixed-mode designs, are based on the rationale of the tailored design method (TDM) suggested by Dillman et al. (2014).¹ Prepaid monetary incentives and reminders, as well as sequential mixed-mode designs, are considered particularly efficient, not only in increasing the willingness to participate but also to enhance the response quality of less motivated respondents.

In the DAB panel study on young people’s educational choices and opportunities in German-speaking Switzerland, prepaid monetary incentives and reminders were used for the first time after three successfully finalized waves. In the first three waves, students were interviewed within classes using a web-based questionnaire. Response rates were between 90 to 95 percent (in contrast, see Dillman, 2000: 153). Non-participation occurred mainly due to illness or activities related to career choice, but not due to refusal. Since classroom interviews were no longer feasible after respondents had finished compulsory education after 9th grade, data had to be collected by individual surveys from the fourth wave onwards. To motivate respondents to continue participating in the panel study without the commitment mediated previously by the teachers, the respondents received in the setting of a survey experiment a prepaid monetary incentive enclosed in an advance letter. While the prepaid monetary incentive had been the random treatment under our control, the reminders were an additional treatment, although not at random, since they were self-selectively distributed by the non-responding interviewees who were generally less motivated for survey participation. Finally, we have extended the previous single mode design to a sequential mixed-mode design (Couper, 2011), with the implementation of computer-assisted telephone interviews (CATI) in addition to the online questionnaire (De Leeuw et al., 2008). This proceeding is considered to be more efficient to increase the cooperation of respondents and effective to convert non-cooperating individuals for survey participation (Dillman et al., 2014; Hox et al., 2015; Kreuter et al., 2010) than the simultaneous offer of different modes (Dillman et al., 2009; Krug et al., 2017; Millar and Dillman, 2011).

While previous research has extensively analysed whether monetary incentives have a positive effect on the response rate at the first contact in the first wave of a panel survey (Ryu et al., 2005), the aim of this paper is to reveal the long-term consequences of using monetary incentives to increase survey participation after three waves without using incentives in a multi-wave panel study with sequential mixed-mode design. The following questions are addressed. Does the first-time use of a prepaid cash incentive increase the response rate in the fourth and fifth waves of a panel? Does the incentive influence the duration until respondents start the survey? Are benefitted respondents more likely to fill out the questionnaire completely? Does the incentive influence the individuals’ likelihood to participate in the CATI mode if they did not fill in the online questionnaire?

The remainder of the article is organized as follows. In the second chapter, the theoretical background and hypotheses will be discussed, after a brief presentation of the state of research related to the efficiency of techniques to increase response rates in panel surveys. The third chapter comprises the description of the data set, design, and variables. The findings are presented in the fourth chapter, while conclusions are derived in the final chapter.

Theoretical Background

State of the Art and Open Questions

According to current findings in survey research, unconditional gifts, in particular prepaid monetary incentives, are the most efficient and effective strategy for reducing unit nonresponse and strengthening the cooperation of respondents in social scientific surveys (Becker and Mehlkop, 2011; Groves et al., 2000; Pforr et al., 2015). Laurie and Lynn (2009: 230) suggest that the use of such incentives is an important element of the strategy to minimize attrition for different types of longitudinal surveys. They report that incentives positively affect retention rates in later waves, that incentives in later panel waves increase response rates, and that lowering incentives in later waves does not reduce retention rates (the “crowding effect”: Scherpenzeel and Toepoel, 2012; see also Pforr et al., 2015). There is obviously a positive effect on early survey response and response speed, resulting in increased fieldwork efficiency through the reduction of the intense and costly follow-up efforts that would otherwise be necessary (Kretschmer and Müller, 2017: 11). Lipps (2010) provides empirical evidence of positive cooperation effects due to the positive effects of prepaid incentives on the response rate in household panel surveys. In a recent study, Kretschmer and Müller (2017) report a positive but rather small effect of the unconditional prepaid cash incentive in later waves of the National Educational Panel Study (NEPS).

A large number of meta-analyses and reviews (Church, 1993; Mercer et al., 2015; Pforr et al., 2015; Singer et al., 1999; Singer and Ye, 2013; Szelényi et al., 2005), as well as experiments (Arzheimer and Klein, 1998; Castiglioni et al., 2008; Scherpenzeel and Toepoel, 2012; Singer et al., 2000), confirm these recent findings for panel surveys. In sum, regarding the deficient theoretical background and the small amount of empirical evidence, the questions on long-term consequences of prepaid monetary incentive for the participation in panel surveys have only been partially answered. In particular, empirical evidence is very rare in German-speaking countries (for example, Kretschmer and Müller, 2017).

The Short- and Long-term Effects of Prepaid Monetary Incentives in a Panel Survey

Several theoretical models exist to explain the effect of prepaid cash incentive on the interviewees’ cooperation (effect on the individual level) and the response rate of surveys (effect on the aggregate level) (Becker and Mehlkop, 2011). Particularly worth mentioning are the extended rational action approach (Diekmann and Jann, 2001; Mehlkop and Becker, 2007) or the social exchange approach (Dillman, 2000; Dillman et al., 2014; Groves et al., 1992), in which the roles of social norms and institutions for subjectively rational actions in social exchange between researchers and their subjects are considered. It has often been experimentally confirmed that subjects normally do not act as predicted by the restrictive economic approach. Acting subjects regularly do not take the money and throw away the questionnaire without any return (Becker and Mehlkop, 2011). In general, they are more likely to cooperate with researchers than respondents from the control group (Arzheimer and Klein, 1998; Diekmann and Jann, 2001; Mehlkop and Becker, 2007). This is true for mail surveys, but also for other modes, such as online survey or CATI.² This phenomenon has three explanations. First, on the one hand, the prepaid monetary incentive seems to compensate for costs anticipated by the interviewees. Otherwise, it strengthens the interviewees’ extrinsic motivation by increasing the subjectively anticipated benefits of survey participation (Groves et al., 2004; Groves et al., 2000). This might be particularly the case for wavering subjects, while for determined respondents the incentive is extra income (Scherpenzeel and Toepoel, 2012). Second, the unconditionally given gift endows trust in the social exchange between researchers and their subjects to be interviewed. Due to this symbol of trust, the subjects are more likely to be ready for answering questions of an unknown interviewer or researcher (Dillman, 2000).

According to both these arguments, it could be assumed that low-achieving individuals with less education and low educational aspirations, who are usually from lower social classes, are less likely to participate in surveys than well-educated individuals (hypothesis 1). This behaviour has been observed consistently (Mehlkop and Becker, 2007). On the one hand, less-educated individuals might perceive the costs of participating in a survey to exceed the benefits since responding to questions requires higher effort levels from them. On the other hand, due to their relatively low levels of self-esteem and persistence, this group is less likely to cooperate with researchers they do not know and to respond to their questions in spite of prepaid incentives (Becker, 2006). Therefore, we have used a combination of various procedures proposed by TDM (Dillman et al., 2014).

Third, it is assumed that the prepaid cash incentive activates the universal norm of reciprocity in the social exchange framework of a social scientific survey (Diekmann and Jann, 2001). Reciprocity seems to be the decisive social mechanism in this setting, explaining cooperation in social scientific surveys (Becker and Mehlkop, 2011; Berger and Rauhut, 2015; Diekmann and Voss, 2008; Ryu et al., 2005; Singer et al., 2000: 172).³ In particular, altruistic reciprocal subjects have social preferences for reciprocity, and intend to respond to the friendly actions of other actors (Berger, 2013; Berger and Rauhut, 2015: 731; Diekmann, 2004; Falk and Fischbacher, 2006: 309). In this case, emotions are additional mechanisms for the efficiency of the altruistic reciprocity in social science surveys. A present regularly results in emotions. In our case, the respondent feels obliged to participate in the survey and to cooperate with the researcher to discharge the advance payment. Additionally, the possible normative sanction for nonresponse, and the shame due to defection, results in the subjects’ cognitive dissonance (Festinger, 1957). This dissonance will be solved by accepting the norm of reciprocity and the actual cooperation with the researcher (Homans, 1961; see also Groves et al., 2000: 303). Cooperation is not determined by the prepaid incentive, however, for two reasons. First, it should be stressed that such a gift is not highly institutionalized in contrast to presents on one’s birthday or at Christmas, or on the occasion of a state visit. Furthermore, the gift is given by researchers who are not familiar with the interviewees. Second, it should be noted that the researchers have no power to sanction subjects’ defection of the principle of reciprocity in terms of non–cooperation and unit nonresponse (Axelrod, 1984). On these grounds, and compatible with a sociological rational action approach aiming at mechanism-based explanations such as the most recent theory of subjectively expected utilities (Becker and Mehlkop, 2011), the prepaid monetary incentive often increases the subjects’ propensity for survey participation, resulting in higher response rates. Benefitted individuals are more likely to cooperate in a social science survey than the reference group usually does (hypothesis 2). Since reciprocity is a universal norm the effect of the incentive does not correlate with the benefitted individuals’ social background.

For dissolving the uncomfortable pressure of the obligation and the cognitive dissonance, the benefitted individuals are assumed to respond earlier (hypothesis 3) and complete the questionnaire faster than the respondents without an incentive (hypothesis 4). This means that respondents who received donations and who have already started filling out the questionnaire will anticipate higher costs from premature abandonment. Regarding the “sunk costs”, the cooperation correlates with irreversible costs, which do not matter in the running interview.

Across the survey time and because of the low degree of institutionalization of social exchange in a survey, however, the cognitive dissonance and the effect of the social reciprocity norm fade away provided that reciprocity is not refreshed. On the one hand, it is assumed that the response rate among benefitted respondents therefore decreases across the field phase (hypothesis 5) and across panel waves (hypothesis 6).

On the other hand, in accordance with the TDM, the reminder by the researcher moderates the respondents’ cooperation and will increase their response rate again. This might be particularly true for respondents who have received the prepaid cash incentive, since the reminder reactivates the respondents’ fading feeling of obligation to respond (hypothesis 7).

Panel Study, Experimental Design, Variables, and Statistical Procedures

The DAB Panel Study

The experiment has been conducted in the context of the DAB panel study on determinants of educational choice and vocational training opportunities (Glauser, 2015; Glauser and Becker, 2016). Apart from their parents, the data collection was limited to individuals born around 1997 and who were enrolled in regular classes in public schools in German–speaking Switzerland. The students were interviewed in the middle of the 8th grade (Wave 1: January–February 2012), as well as at the beginning (Wave 2: August–October 2012) and at the end (Wave 3: May–June 2013) of the 9th grade. The method experiment was conducted in the fourth wave (October–November 2014), during which respondents were interviewed about their educational pathways within the first 15 months of leaving compulsory school. The fifth wave took place three years after the completion of compulsory education (June 2016).

The panel data is based on a random stratified gross sample of 296 school classes (8th grade), out of a total universe of 3,045 classes (Glauser, 2015). After contacting the headmasters and teachers, 215 out of 296 school classes were ready to participate in the online survey in the first wave ( Table 1 ). At a class level, the response rate was 73 percent; 95 percent of 3,894 students participated in the first wave.

Table 1.

Samples and response in the DAB panel

	Wave 1	Wave 2	Wave 3	Wave 4	Wave 5
	Jan/Feb 2012	Aug–Oct 2012	May/Jun 2013	Oct/Nov 2014	Jun/Aug 2016
Sample size
Gross sample (school classes)	296	215	203
Net sample (school classes)	215	203	199
Gross sample (individuals)	3,894	3,707	3,436	3,242	3,094
Contactable individuals				2,652	2,861
Type of survey
Online survey of individuals	yes	yes	yes	yes	yes
CATI survey of individuals	no	no	no	yes	yes
Incentive for individuals	no	no	no	yes	yes
Realized interviews
Individuals	3,680	3,343	3,302	2,237	2,229
of whom: Online				1,227	1,330
CATI				1,010	899
Parents	2,302	–	–	–	–
Response rate in %
School classes	73%	94%	98%	–
Individuals	95%	90%	96%	81%	72%
Contactable individuals				84%	78%
of whom: Online				46%	46%
CATI				38%	32%

If the respondents’ changes of school or grade, as well as their withdrawal from the sample, are taken into account, the response rates of the students were 90 percent in the second and 96 percent in the third wave.⁴ In the fourth wave, 81 percent of the students in the gross sample (n = 3,242), who had already left the context of their previous grade and school, could be contacted individually. As in the previous waves, they were asked to complete the web–based online questionnaire. In the case of nonresponse, they were asked to take part at the CATI survey. About 84 percent (n = 2,237) participated in the survey (an online survey using “UniPark EFS Survey”, realized by the Department of Sociology of Education at the University of Bern: 46 percent, and 38 percent per CATI, realized by a commercial polling agency, M.I.S Trend in Lausanne). Finally, in the fifth wave, about 78 percent of contactable individuals responded, 46 percent of them in the online mode and 32 percent in the CATI mode. Both modes were also realized by the Department of Sociology of Education at the University of Bern.

The fieldwork was done according to TDM suggested by Dillman (2000; Dillman et al., 2014). Due to efficiency and methodological advantages, the online mode was chosen by us in each of the waves (Couper, 2000; Couper and Bosnjak, 2010). For example, it is evident that this mode is preferred by the youth (Smyth, 2014: 142), and it is less vulnerable to issues connected with delicate questions or social desirability (for example, grades, achievement: Kreuter et al., 2008). The social selection inherent in having access to the internet (Couper, 2000: 484; Couper et al., 2007; Schonlau et al., 2009), or the preference for another mode, should be compensated by the CATI – in other words, to be more precise, survey nonresponse should be reduced by a mix of modes (De Leeuw, 2005: 240).⁵

The Experimental Design

The experimental design is classic and rather simple. For detecting the consequences of prepaid monetary incentives on the individuals’ cooperation and the total response rate, we have randomly selected school classes (and not the students) which have participated in the third wave in equal shares into the treatment and control groups. This procedure was necessary to avoid envy from the non–benefitted classmates and the shame of the benefitted classmates. The treatment is a MIGROS cash card of 10 Swiss francs.⁶ The general, not earmarked fund is equal to universal cash, which could be used in each of the affiliates or in the online shop. This gift has been enclosed in the advance letter sent to the individuals by regular mail (Edwards et al., 2014). In this polite letter, we inform the respondents in detail that they are being interviewed in the context of the DAB panel study. In this way, we have demonstrated the reputation of the survey and the researchers emphasizing the fact that the project is affiliated to a local university and that the Swiss State Secretary for Education, Research and Innovation (SERI) is the sponsor of the panel study.

Another treatment is the reminder sent to the interviewees in the case that they have not already started filling out the online questionnaire. Interviewees receive the reminder one week after being invited to participate in the online survey. In contrast to the unconditionally prepaid incentive, the treatment is not given at random, but to individuals who are obviously less motivated to participate in the fourth wave. In the empirical analysis, the aim is to test whether the consequences of both treatments are independent or whether an interaction between them can be observed. For example, it could be assumed that the effect of the prepaid incentive increases, since the reminders indirectly appeal to the norm of reciprocity.

The Dependent and Independent Variables

The first dependent variable is the likelihood of participation in the fourth wave of our panel study. Hereby, we distinguish for the response in two different modes offered sequentially (online first and CATI second). We are also interested in the duration between contact and the start of the completion of the questionnaire. The second dependent variable is the likelihood of completing the web–based online questionnaire after beginning to fill it out. The third dependent variable is the likelihood of responding in the CATI mode. The propensity for participation in the fifth wave is the fourth dependent variable.

Both treatments (the prepaid monetary incentive at random and the non–random reminders) and their interaction are the most important explaining variables. For reasons of statistical control, we consider the individuals’ gender, educational aspiration, previous schooling, GPA in the 9th grade (achievement), and the social class of their parents (EGP class schema by Erikson and Goldthorpe, 1992) as well as their internal and external control beliefs, including the individual’s persistence. These beliefs indicate the subjects’ self–efficacy (Bandura, 1995), an integral component of the rational action approach (Bandura, 1986) for explaining survey participation and response to the questionnaire.

Statistical Procedures

For the multivariate analysis, we apply the comparative-static binary logistic regression (Long, 1997) for the estimations of AME, the average marginal effects (Best and Wolf, 2012; Hinz and Auspurg, 2011). The dynamic procedures of event–history analysis have also been utilized (Blossfeld et al., 2007). The effects of incentives and reminders will be estimated by the semi–parametrical hazard rate model, such as the Cox regression, a piecewise constant rate model, and (in the context of the episode splitting procedure) the parametric exponential model and the Gompertz model. It is the aim of the episode splitting procedure to reveal the time dependencies of the treatment effects.

Empirical Findings

In the following steps we analyse the experimental design including a randomized assignment of subjects to the treatment, namely the prepaid incentive, as well as a controlled administration of the treatment. We test the hypotheses with appropriate statistical procedures such as comparative–static logit models or dynamic event history models. The specifications of these models depend on both the hypothesis to be tested and the processing of the survey.

The Effect of Prepaid Cash Incentives on Participation in the Fourth Panel Wave

At a glance, there seems to be no significant effect from the incentive if the propensity for participation in the fourth wave of the DAB panel is estimated without differentiating between the online and the CATI modes (Model 1 in Table 2 ). This might be a statistical artefact. Considering the structure of the sequential mixed mode design, a significant incentive effect is revealed for participation in the online survey offered first to individuals (Model 2). Hypothesis 2 is thus empirically supported. However, it should be noted that the effect is statistically significant, but rather small compared to other experiments reported above (see also: Kretschmer and Müller, 2017). The increase in the response rate overall is just about 2 percentage points and 7 percentage points for the online survey.

Table 2.

The effect of prepaid incentive on response in a sequential mixed–mode design

Model	1	2	3	4	5
Survey mode	Online or CATI	Online survey only
Incentive (reference: no incentive)
Received	0.020	0.072	0.071	0.074	0.061
	(0.014)	(0.019)***	(0.019)***	(0.021)***	(0.022)**
Gender (reference: male)
Female			0.078	0.040	0.062
			(0.020)***	(0.021)	(0.023)**
Aspiration in Wave 3 (reference: other options)
Vocational education and training (VET)			0.248	–0.026	–0.029
Vocational education and training (VET)			(0.161)	(0.280)	(0.305)
General school (Gymnasium)			0.424	0.009	0.003
			(0.162)**	(0.281)	(0.306)
Orientation phase			0.212	–0.030	–0.030
			(0.164)	(0.281)	(0.306)
Indifference			0.230	–0.075	–0.127
			(0.180)	(0.296)	(0.322)
Previous schooling (reference: no selection)
Basic secondary school				–0.047	–0.035
				(0.061)	(0.064)
Advanced secondary school				0.089	0.102
				(0.060)	(0.062)
Gymnasium				0.228	0.246
				(0.065)***	(0.068)***
Previous school achievement
Grade point average in German (9th class)				0.080	0.078
Grade point average in German (9th class)				(0.016)***	(0.016)***
Social class of parents (reference: Unskilled workers VII)
Upper service class (I)				0.040	0.036
				(0.049)	(0.050)
Lower service class (II)				0.081	0.092
				(0.047)	(0.048)
Non–manual routine services (III a/b)				0.075	0.087
Non–manual routine services (III a/b)				(0.046)	(0.047)
Petty bourgeoisie and farmers (IV a/b/c)				0.035	0.042
Petty bourgeoisie and farmers (IV a/b/c)				(0.055)	(0.056)
Engineer, foreman, and skilled worker (V/VI)				0.026	0.018
Engineer, foreman, and skilled worker (V/VI)				(0.048)	(0.048)
Control belief and persistence
Internal control belief					–0.010
					(0.006)
External control belief					0.002
					(0.004)
Persistence					0.013
					(0.004)***
Number of cases	2,652	2,652	2,646	2,177	2,008
Pseudo–R²	0.001	0.004	0. 026	0.060	0.067

*p ≤ 0.05; **p ≤ 0.01; ***p ≤ 0.001.

Average marginal effects (in parenthesis: standard error), estimated by binary logistic regression

This result is in line with theoretical assumptions and remains unchanged when other variables are controlled for, such as gender and educational aspiration (Model 3), previous schooling, school achievement and social origin (Model 4), and personal characteristics such as control beliefs and persistence (Model 5). The results show that female subjects are more likely to participate in the fourth wave than male subjects. This is also true for interviewees visiting the Gymnasium (grammar school) or who plan to be enrolled on this track or who provide for excellent school achievements, while social origin is statistically insignificant. These results are in line with hypothesis 1 assuming that in particular less-educated and low-achieving subjects often do not cooperate with researchers and interviewers. That means that the prepaid incentive does not compensate the mechanisms behind their social situation. This results in a significant panel attrition of a specific group being very interesting for the aim of the DAB panel. However, the higher the subjects’ persistence, the higher is their propensity to participate in the next wave, whereas individuals with a more pronounced external control belief are less likely to participate. Prepaid incentives do not compensate for their negative impact on interviewees’ likelihood for survey participation.

Duration of the Latency and Completion of the Questionnaire

Is there a decreasing latency (i.e. the duration until the interviewee start with filling out the online questionnaire) due to a prepaid monetary incentive? Subjects that have received the incentive start much earlier with filling out compared to the unconsidered interviewees (Cox model in Table 3 ). The survival curves of the treatment and control groups estimated by the first model are presented in Figure 1 .

Table 3.

The effect of prepaid incentive on duration until the start of the online questionnaire – semi-parametrical hazard rate models (Cox regression), piecewise constant rate model (PCR), and after episode splitting: exponential model (EXPSPLIT) and Gompertz model (COMPSPLIT)

Model type	Cox		PCR		EXPSPLIT		COMPSPLIT
Model No.	(1)		(2)		(3.1)		(3.2)
Incentive (reference: no incentive)
Received	1.26***	(4.07)			1.38***	(4.00)	1.30**	(2.99)
Received (time dependent γ-term)							0.99	(–0.50)
Reminder (ref.: not yet received)
Received					3.28***	(13.9)	0.19***	(–4.96)
Received (time dependent γ-term)							1.30***	(7.77)
Interaction term
Incentive · Reminder					0.84	(–1.48)
Time period
1: 0–3			0.065***	(–42.6)
2: 3–7			0.017***	(–34.4)
3: 7–10			0.042***	(–35.0)
4: 10–30			0.001***	(–54.3)
5: >= 30			0.0001***	(–18.8)
Effect of incentive in time period
1: 0–3			1.29**	(3.03)
2: 3–7			1.37*	(2.01)
3: 7–10			1.15	(1.15)
4: 10–30			1.20	(1.53)
5: >= 30			1.32	(.55)
Intercept					0.01***	(–75.0)	0.06***	(–43.2)
Intercept (Gamma γ)							0.87***	(–18.7)
Number of cases (sub episodes)	2,652		2,652		2,652 (66,411)		2,652 (66,411)
Number of events	1.227		1.227		1.227		1.227
LR / Wald Chi² (d.f.)	16.53 (1)		15074.50 (10)		360.53 (3)		33.37 (2)

*p ≤ 0.05; **p ≤ 0.01; ***p ≤ 0.001; in parenthesis: z-value of α-coefficients

Figure 1.

Start of filling out the online questionnaire (estimation based on Model 1 in Table 3)

The median time of the experimental group’s participation is about three weeks, while for the control group it is not possible to estimate any median time, since fewer than 50 per cent of them responded to the questionnaire. This finding is in line with hypothesis 3.

The estimation of the incentive effect on the latency by the piecewise constant rate model (PCR in Table 3 ) shows two important results. First, it is evident that the subjects’ propensity for survey participation declines with increasing time after contact. Second, the effect of the incentive fades in time. After seven days, the incentive has no significant effect on the benefitted respondents’ inclination for participating in the survey. This means hypothesis 5 is supported empirically by our findings.

Is it possible to stop the fading of the incentive effect with a reminder? To investigate this question, the risk time of the episode for the online mode will be divided into sub episodes on a daily basis. The findings of the exponential model are clear (EXSPLIT). The positive effects of the incentive and reminder on the subjects’ response are independent. Of course, the reminder refreshes the individuals’ inclination for survey participation, but there is no statistically significant interaction between incentive and reminder, indicating that the effect of the incentive will not be strengthened by the reminder. Obviously, this result contradicts hypothesis 7.

This finding is in favour of the TDM suggested by Dillman (2000): the exclusive use of an incentive is not enough to increase response rates or to maintain retention rates in a panel study. Additional measures are necessary to underpin the altruistic reciprocity initiated by the incentive; however, incentives are pivotal while the reminders are sufficient. According to the TDM, they are additive regarding their effects on the response rate.

In the next step, we seek to reveal the time dependent effects of both the incentive and the reminder by utilizing the Gompertz model (COMPSPLIT). The estimation supports the previous model specifications. The effect of the incentive is limited to the first seven days, while the effect is insignificant for the later sub episodes. This finding is in line with hypotheses 2 and 4. The time dependencies of the treatments mean that the effect of the incentive fades over time, while the effect of the reminder intensifies. In sum, regarding the response rate, the prepaid monetary incentive and reminder amend their effect by addition in different time periods.

Are the benefitted respondents more likely to completely fill out the online questionnaire than respondents from the control? Taking into account their previous schooling, achievement and personality, there is no empirical evidence that benefitted respondents are more likely to fill out the questionnaire faster without any abort ( Table 4 ). The results do not support hypothesis 4 (Model 1). Respondents enrolled in the upper secondary school (Gymnasium) and who have achieved advantageous grades are the subjects who regularly complete the questionnaire (Model 2). Female respondents, however, needs more time for completing the questionnaire than male respondents.

Table 4.

The effect of prepaid incentive on duration of completing the online questionnaire (vs. abandonment) – Gompertz model

Model No.	1		2
Incentive (reference: no incentive)
Received	0.99	(–0.07)	1.07	(1.01)
Gender (reference: male)
Female			0.74	(–4.12)
Previous schooling (reference: no selection)
Basic secondary school			1.30	(0.78)
Advanced secondary school			1.79	(1.80)
Gymnasium			2.66***	(2.97)
Grades
Grade point average in German (9th class)			1.17***	(2.90)
Social class of parents (reference: Unskilled workers VII)
Upper service class (I)			1.63	(1.79)
Lower service class (II)			1.17	(0.60)
Non–manual routine services (III a/b)			1.22	(0.73)
Petty bourgeoisie and farmers (IV a/b/c)			1.32	(0.98)
Engineer, foreman, and skilled worker (V/VI)			1.48	(1.52)
Control belief and persistence
Internal control belief			1.01	(0.39)
External control belief			1.00	(0.12)
Persistence			1.01	(0.66)
Intercept	0.0026	(–113.1)	0.0004	(–14.0)
Gamma (γ)	0.0007	(–3.53)	0.0012	(5.78)
Number of cases	1,208		954
Number of events	942		749
Wald Chi² (d.f.)	0.01 (1)		88.52 (14)

*p ≤ 0.05; **p ≤ 0.01; ***p ≤ 0.001; in parenthesis: z–value of α–coefficients

Effects of Incentives on Response in the CATI Mode

Now we investigate the question of whether subjects who have not participated in the online survey despite their previous gratification are ready to participate in the CATI mode. In the perspective of recent survey research, the results are sobering. However, they are in line with hypothesis 4. The prepaid monetary incentive has no long–term effect in the final stage of the sequential mixed mode design. The incentive does not increase the interviewees’ likelihood to participate in CATI ( Table 5 ).

Table 5.

The effect of prepaid incentive on CATI response (binary logit; AME; in parenthesis: z–value of β–coefficients) and on the duration of contact and start of CATI – semi–parametrical and parametrical hazard rate models (β–coefficient; in parenthesis: z–value of β–coefficients)

Model type	Binary logit	Binary logit1	Cox1	Exponen– tial1	Gom– pertz1	Gom– pertz
Model No.	1	2	3	4	5.1	5.2
Incentive	0.02	0.01	0.04	0.04	0.04
	(1.00)	(.51)	(.41)	(.56)	(.53)
β: Incentive						0.05
						(0.66)
β: Personality
Internal control belief						0.02
						(1.31)
External control belief						–0.03*
						(2.34)
Persistence						0.0004
						(0.03)
γ: Incentive						0.004
						(0.57)
γ: Personality
Internal control belief						–0.0009
						(–.039)
External control belief						0.002
						(1.37)
Persistence						–0.002
						(–1.59)
Intercept				–3.26*	–2.93*
				(–6.65)	(–6.54)
β: Intercept						–2.72*
						(–9.63)
γ: Intercept						0.0007
						(0.02)
Gamma					–0.03*
					(–7.62)
Number of cases	1,447	1,059	1,059	1,059	1,059	1,316
Number of events	1,010	745	745	745	745	927
LR / Wald Chi² (d.f.)	0.01 (1)	25.65 (14)	24.06 (14)	26.70 (13)	25.81 (14)	7.99 (4)

*p ≤ 0.05 at least; ¹controlled for gender, education, GPA, social origin, and control beliefs

This finding is reliable, because we have utilized different parametric specifications such as binary logit regression, Cox regression, and the exponential model and Gompertz model. For the risk set, it means that the art of persuasion (but not the incentive) is pivotal for the response rate. The impact of individuals’ characteristics is statistically insignificant. However, for the CATI mode, it is interesting to find that – in contrast to individuals with a significant internal control belief – individuals with a pronounced external control belief hesitate to respond at once (Model 5.2), while individuals with a higher persistence are more likely to participate in the CATI than individuals whose persistence is less pronounced. However, it has to be noticed that only the effect of external control belief is statistically significant provided that other variables such as previous schooling, school achievement and social origin are not taken into account.

Effects of Incentives on Response in the Fifth Panel Wave

Finally, the possible long–term effects are analysed of prepaid cash incentives given in wave 4 on the response rate in wave 5 ( Table 6 ). The results are consistent with hypothesis 6. There are no long–term effects for each of the two different modes across the panel waves.

Table 6.

The long–term effect of incentive given in wave 4 on response in wave 5

Survey mode	Online & CATI	Online only	CATI only
Incentive (reference: no incentive)	0.002	–0.019	0.019
	(0.015)	(0.020)	(0.024)
Participation in wave 4	0.477	0.296	0.509
	(0.027)***	(0.025)***	(0.029)***
Number of cases	2,495	2,495	1,314
Pseudo–R²	0.1397	0.0324	0.1361

*p ≤ 0.05; **p ≤ 0.01; ***p ≤ 0.001; AME (in parenthesis: standard error), estimated by binary logistic regression

We find, however, that previous survey participation is a significant predictor for future participation. This means that the investment in the response rate of the most recent panel wave is indirectly an investment in the retention rate of subsequent waves. In this respect, this could be an indirect long–term effect of the prepaid incentive resulting in a persistent retention rate since these respondents are socialized to be “good panellists”.

Summary

Starting from general research on survey methodology (who is participating and why in social science surveys), it is the aim of this contribution to evaluate the long–term effects of a prepaid cash incentive (a voucher for 10 Swiss francs) on the response rate in the fourth and fifth wave of a panel with sequential mixed–mode design (online and CATI). Subjects were young students in German–speaking Switzerland, who were interviewed regarding their educational aspirations and trajectories (Glauser, 2015). We executed a randomized and controlled experiment after the students finished their compulsory schooling. According to the norm of (altruistic) reciprocity, it was assumed that the panellists who received donations were more likely to cooperate with the interviewers. Furthermore, we expected that they would start earlier in filling out the online questionnaire and were more likely to complete it. Since the pressure of cognitive dissonance induced by the norm of reciprocity weakens over time, the effect of the incentive fades across the fieldwork period and the waves. Finally, it was assumed that the reminder would refresh the fading reciprocity so that the interaction of prepaid incentive and reminder would contribute to increasing response rates.

The empirical findings are significant. The first use of a prepaid monetary incentive had a positive effect on the response rate in the fourth wave of our panel survey. The latency and the fieldwork period shortened compared to the respondents not considered, since the interviewees who received donations started earlier in filling out the questionnaire. The effect of the prepaid incentive is not persistent, however; the impact of reciprocity weakened across the fieldwork period, and the incentive effect thus faded over time. In line with other experimental findings, the prepaid incentive in a previous wave had no effect on the response rate of the next panel wave. Finally, it was found that the reminder did not interact with the prepaid incentive in terms of refreshing or strengthening reciprocity.

The findings of the methodological experiment are in line with previous findings, with the theoretically assumed mechanisms of reciprocity norms, and with the related cognitive dissonance. It should, however, be noted that we were unable to measure the acceptance of the norm of reciprocity and the cognitive dissonance directly. In terms of a mechanism-based explanation, the theoretical background is still plausible, since the theoretically assumed mechanisms are empirically confirmed. In methodological terms, the prepaid monetary incentive should be an integral part of each social science survey. However, expectations should not be overrated. A monetary incentive is no panacea for reducing panel attrition, optimizing the retention rate, and increasing the response rate. It contributes to the response and retention rate in a panel with several waves, but the effects are not deterministic, nor did we observe direct long–lasting effects across waves. On the contrary, the effect of the monetary incentive is fading. In our case, the effect of a prepaid cash incentive of 10 Swiss francs faded away after a week, and it could not be transferred to another survey mode. Prepaid incentives work. They are necessary, but insufficient. It is therefore necessary to use a fine-tuned bundle of arrangements to optimize response rates, avoid panel attrition, and preserve the retention rate in longitudinal studies. In this respect, the TDM suggested by Dillman et al. (2014) might be the most promising way.

Footnotes

Acknowledgements

For helpful comments on earlier drafts, we wish to thank Ben Jann, Oliver Lipps, Guido Mehlkop, and Volker Stocké. The manuscript is dedicated to the memory of our colleague and friend Volker Stocké (1966–2017).

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The DAB panel study is substantially financed by the (Swiss) State Secretariat for Education, Research and Innovation (SERI). The interpretations and conclusions are those of the authors and do not necessarily represent the views of the SERI.

Notes

References

AAPOR (The American Association for Public Opinion Research) (2015) Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys (eight edition). AAPOR.

Andreß

Golsch

Schmidt

(2013) Applied Panel Data Analysis for Economic and Social Surveys. Heidelberg: Springer.

Arzheimer

Klein

(1998) Die Wirkung materieller Incentives auf den Rücklauf einer schriftlichen Befragung. ZA–Information 43: 6–31.

Axelrod

(1984) The Evolution of Cooperation. New York, NY: Basic Books.

Bandura

(1986) Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs, NJ: Prentice–Hall.

Bandura

(1995) Self-Efficacy in Changing Societies. Cambridge: Cambridge University Press.

Becker

(2006) Selective response to questions about delinquency. Quality & Quantity 40(4): 483–498.

Becker

Mehlkop

(2011) Effects of prepaid monetary incentives on mail survey response rates and on self–reporting about delinquency – empirical findings. Bulletin of Sociological Methodology 111(1): 5–25.

Berger

(2013) Altruistische Reziprozität. Theoretische Überlegungen und experimentelle Evidenz. Kölner Zeitschrift für Soziologie und Sozialpsychologie 65(1): 31–48.

10.

Berger

Rauhut

(2015) Reziprozität und Reputation. In: Braun

Saam

(eds) Handbuch Modellbildung und Simulation in den Sozialwissenschaften. Wiesbaden: Springer, 715–742.

11.

Best

Wolf

(2012) Modellvergleich und Ergebnisinterpretation in Logit– und Probit–Regressionen. Kölner Zeitschrift für Soziologie und Sozialpsychologie 64(2): 377–395.

12.

Blossfeld

Golsch

Rohwer

(2007) Event History Analysis with Stata. New York, NY: Lawrence Erlbaum.

13.

Blossfeld

Rossbach

von Maurice

(eds) (2011) Education as a Lifelong Process: The German National Educational Panel Study (NEPS). Wiesbaden: Springer.

14.

Bosnjak

Tuten

(2003) Prepaid and promised incentives in web surveys. Social Science Computer Review 21(2): 208–217.

15.

Castiglioni

Pforr

Krieger

(2008) The effect of incentives on response rates and panel attrition: results of a controlled experiment. Survey Research Methods 2(3): 151–158.

16.

Cernat

(2015) The impact of mixing modes on reliability in longitudinal studies. Sociological Methods & Research 44(3): 427–457.

17.

Church

(1993) Estimating the effect of incentives on mail survey response rates: a meta-analysis. Public Opinion Quarterly 57(1): 62–79.

18.

Couper

(2000) Web surveys: a review of issues and approaches. Public Opinion Quarterly 64(4): 464–494.

19.

Couper

(2011) The future of modes of data collection. Public Opinion Quarterly 75(5): 889–908.

20.

Couper

Bosnjak

(2010) Internet surveys. In: Marsden

Wright

(eds) Handbook of Survey Research (second edition). Bingley: Emerald, 527–549.

21.

Couper

Kapteyn

Schonlau

Winter

(2007) Noncoverage and nonresponse in an Internet survey. Social Science Research 36(1): 131–148.

22.

De Leeuw

(2005) To mix or not to mix data collection modes in surveys. Journal of Official Statistics 21(2): 233–255.

23.

De Leeuw

Hox

Dillman

(2008) Mixed-mode surveys: when and why. In: De Leeuw

Hox

Dillman

(eds) International Handbook of Survey Methodology. New York, NY: Lawrence Erlbaum, 299–316.

24.

De Leeuw

Hox

Scherpenzeel

(2011) Mode effect or question wording? Measurement error in mixed mode surveys. In: JSM (Joint Statistical Meetings) Proceedings, Miami Beach, Florida, July 30–August 4, 2011 (Section on Survey Research Methods, ASA), 5959–5967. Alexandria, VA: American Statistical Association.

25.

De Leeuw

Mellenberg

Hox

(1996) The influence of data collection method on structural models. Sociological Methods & Research 24(4): 443–472.

26.

Diekmann

(2004) The power of reciprocity: fairness, reciprocity, and stakes in variants of the dictator game. Journal of Conflict Resolution 48(4): 487–505.

27.

Diekmann

Jann

(2001) Anreizformen und Ausschöpfungsquoten bei postalischen Befragungen. Eine Prüfung der Reziprozitätshypothese. ZUMA–Nachrichten 25(48): 19–27.

28.

Diekmann

Voss

(2008) Soziale Normen und Reziprozität – Die Bedeutung „sozialer” Motive für die Rational–Choice–Erklärung sozialer Normen. In: Diekmann

Eichner

Schmidt

Voss

(eds) Rational Choice: Theoretische Analysen und empirische Resultate. Wiesbaden: Springer, 83–100.

29.

Dillman

(2000) Mail and Internet Surveys. The Tailored Design Method. New York, NY: Wiley.

30.

Dillman

Phelps

Tortora

Swift

Kohrell

Berck

Messer

(2009) Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Social Science Research 38(1): 1–18.

31.

Dillman

Smyth

Christian

(2014) Internet, Phone, Mail, and Mixed–Mode Surveys: The Tailored Design Method. New York, NY: John Wiley & Sons.

32.

Edwards

Dillman

Smyth

(2014) An experimental test of the effects of survey sponsorship on Internet and mail survey response. Public Opinion Quarterly 78(3): 734–750.

33.

Erikson

Goldthorpe

(1992) The Constant Flux. A Study of Class Mobility in Industrial Societies. Oxford: Clarendon Press.

34.

Falk

Fischbacher

(2006) A theory of reciprocity. Games and Economic Behavior 54(2): 293–315.

35.

Festinger

(1957) A Theory of Cognitive Dissonance. Stanford, CA: Stanford University Press.

36.

Gajic

Cameron

Hurley

(2012) The cost-effectiveness of cash versus lottery incentives for a web-based, stated-preference community survey. European Journal of Health Economics 13(6): 789–799.

37.

Glauser

(2015) Berufsausbildung oder Allgemeinbildung. Wiesbaden: Springer.

38.

Glauser

Becker

(2016) VET or general education? Effects of regional opportunity structures on educational attainment in German-speaking Switzerland. Educational Research in Vocational Education and Training 8(8): 1–25.

39.

Göritz

(2004) The impact of material incentives on response quantity, response quality, sample composition, survey outcome, and cost in online access panels. International Journal of Market Research 46(3): 327–345.

40.

Göritz

(2006) Incentives in web studies: methodological issues and a review. International Journal of Internet Science 1(1): 58–70.

41.

Gouldner

(1960) The norm of reciprocity: a preliminary statement. American Sociological Review 25(2): 161–178.

42.

Greene

Speizer

Wiitala

(2008) Telephone and web: mixed-mode challenge. Health Research and Educational Trust 43(1): 230–248.

43.

Groves

Cialdini

Couper

(1992) Understanding the decision to participate in a survey. Public Opinion Quarterly 56(4): 475–495.

44.

Groves

Peytcheva

(2008) The impact of nonresponse rates on nonresponse bias: a meta-analysis. Public Opinion Quarterly 72(2): 167–189.

45.

Groves

Presser

Dipko

(2004) The role of topic interest in survey participation decisions. Public Opinion Quarterly 68(1): 2–31.

46.

Groves

Singer

Corning

(2000) Leverage-saliency theory of survey participation: description and an illustration. Public Opinion Quarterly 64(3): 299–308.

47.

Heerweegh

(2009) Mode differences between face-to--ace and web surveys: an experimental investigation of data quality and social desirability effects. International Journal of Public Opinion Research 21(1): 111–121.

48.

Heerwegh

Loosveldt

(2008) Face-to-face versus web surveying in a high-Internet-coverage population. Public Opinion Quarterly 72(5): 836–846.

49.

Hinz

Auspurg

(2011) Gruppenvergleiche bei Regressionen mit binären abhängigen Variablen – Probleme und Fehleinschätzungen am Beispiel von Bildungschancen im Kohortenverlauf. Zeitschrift für Soziologie 40(1): 62–73.

50.

Homans

(1961) Social Behavior: Its Elementary Forms. New York: Harcourt, Brace & World.

51.

Hox

De Leeuw

Jijmans

EAO

(2015) Measurement equivalence in mixed mode surveys. Frontiers in Psychology 6(87): 1–11.

52.

Kretschmer

Müller

(2017) The Wave 6 NEPS adult study incentive experiment. Methods, Data, Analyses 11(1): 7–28.

53.

Kreuter

Müller

Trappmann

(2010) Nonresponse and measurement error in employment research: making use of administrative data. Public Opinion Quarterly 74(5): 880–906.

54.

Kreuter

Presser

Tourangeau

(2008) Social desirability bias in CATI, IVR, and web surveys: the effects of mode and question sensitivity . Public Opinion Quarterly 72(5): 847–865.

55.

Krug

Carstensen

Kriwy

(2017) Die richtige Mischung? Ein randomisiertes Experiment zur Datenqualität bei der Kombination von Telefon– und Onlineerhebung in der empirischen Sozialforschung . Zeitschrift für Soziologie 46(2): 89–106.

56.

LaRose

Tsai

(2014) Completion rates and non-response error in online surveys: comparing sweepstakes and pre-paid cash incentives in studies of online behavior. Computers in Human Behavior 34: 110–119.

57.

Laurie

Lynn

(2009) The use of respondent incentives on longitudinal surveys. In: Lynn

(ed.) Methodology of Longitudinal Surveys. Chichester: Wiley & Sons, 205–233.

58.

Lipps

(2010) Effects of different incentives on attrition and fieldwork effort in telephone household panels surveys. Survey Research Methods 4(2): 81–90.

59.

Long

(1997) Regression models for categorical and limited dependent variables. Thousand Oaks, CA: SAGE Publications.

60.

Mauss

(1984) Die Gabe: Form und Funktion des Austauschs in archaischen Gesellschaften (The Gift: The Form and Reason for Exchange in Archaic Societies). Frankfurt am Main: Suhrkamp.

61.

Mehlkop

Becker

(2007) Zur Wirkung monetärer Anreize auf die Rücklaufquote in postalischen Befragungen zu kriminellen Handlungen: Theoretische Überlegungen und empirische Befunde eines Methodenexperiments. Methods, data, analyses 1(1): 5–24.

62.

Mercer

Caporaso

Cantor

Townsend

(2015) How much gets you how much? Monetary incentives and response rates in household surveys. Public Opinion Quarterly 79(1): 105–129.

63.

Millar

Dillman

(2011) Improving response to web and mixed-mode surveys. Public Opinion Quarterly 75(2): 249–269.

64.

Pforr

, et al. (2015) Are incentive effects on response rates and nonresponse bias in large-scale, face-to-face surveys generalizable to Germany? Evidence from ten experiments. Public Opinion Quarterly 79(3): 740–768.

65.

Ryu

Couper

Marans

(2005) Survey incentives: cash vs. in-kind; face–to–face vs. mail; response rate vs. nonresponse error. International Journal of Public Opinion Research 18(1): 89–121.

66.

Sanchez–Fernándenz

Munoz–Leiva

Montoros–Rios

Ibanez–Zapata

(2010) An analysis of the effect of pre-incentives and post-incentives based on draws on response to web surveys. Quality & Quantity 44(2): 357–373.

67.

Scherpenzeel

Toepoel

(2012) Recruiting a probability sample for an online panel: effects of contact mode, incentives, and information. Public Opinion Quarterly 76(3): 470–490.

68.

Schonlau

Van Soest

Kapteyn

Couper

(2009) Selection bias in web surveys and the use of propensity scores. Sociological Methods & Research 37(3): 291–318.

69.

Schouten

Van den Brakel

Buelens

Van der Laan

Klausch

(2013) Disentangling mode-specific selection and measurement bias in social surveys. Social Science Research 42(6): 1555–1570.

70.

Singer

Van Hoewyk

Gebler

Raghunathan

McGonagle

(1999) The effect of incentives on response rates in interviewer-mediated surveys. Journal of Official Statistics 15(2): 217–230.

71.

Singer

(2013) The use and effects of incentives in surveys. The Annals of the American Academy of Political and Social Science 645(1): 112–141.

72.

Singer

Van Hoewyk

Maher

(2000) Experiments with incentives in telephone surveys. Public Opinion Quarterly 64(2): 171–188.

73.

Smyth

Olson

Millar

(2014) Identifying predictors of survey mode preference. Social Science Research 48: 135–144.

74.

Szelényi

Bryant

Lindholm

(2005) What money can buy: examining the effects of prepaid monetary incentives on survey response rates among college students. Educational Research and Evaluation 11(4): 385–404.

75.

TREE (2016) Documentation on the first TREE cohort (TREE1), 2000–2016. Bern: TREE.

76.

Vannieuwenhuyze

Loosveldt

Molenberghs

(2012) A method to evaluate mode effects on the mean and variance of a continuous variable in mixed-mode surveys. International Statistical Review 80(2): 306–322.

77.

Young

O’Halloran

McAulay

Pirotta

Forsdike

Stacey

Currow

(2014) Unconditional and conditional incentives differentially improved general practitioners’ participation in an online survey: randomized controlled trial. Journal of Clinical Epidemiology 68(6): 693–697.