Abstract
Inferential statistics teach us that we need a random probability sample to infer from a sample to the general population. In online survey research, however, volunteer access panels, in which respondents self-select themselves into the sample, dominate the landscape. Such panels are attractive due to their low costs. Nevertheless, recent years have seen increasing numbers of debates about the quality, in particular about errors in the representativeness and measurement, of such panels. In this article, we describe four probability-based online and mixed-mode panels for the general population, namely, the Longitudinal Internet Studies for the Social Sciences (LISS) Panel in the Netherlands, the German Internet Panel (GIP) and the GESIS Panel in Germany, and the Longitudinal Study by Internet for the Social Sciences (ELIPSS) Panel in France. We compare them in terms of sampling strategies, offline recruitment procedures, and panel characteristics. Our aim is to provide an overview to the scientific community of the availability of such data sources to demonstrate the potential strategies for recruiting and maintaining probability-based online panels to practitioners and to direct analysts of the comparative data collected across these panels to methodological differences that may affect comparative estimates.
Keywords
Introduction
There are compelling reasons to expect that Internet interviewing will become the dominant survey mode in the social sciences over the next few decades, largely replacing written, face-to-face, and telephone interviewing. According to Baker et al. (2010, p. 7), about 85% of online research in 2009 replaces research that previously would have been conducted in traditional modes, primarily by telephone or face-to-face. In addition, Callegaro, Villar, Yeager, and Krosnick (2014) point out that global expenditures on online research, as a percentage of total expenditures on quantitative research, increased from 19% in 2006 to 35% in 2012. Internet penetration, including the use of smartphones and tablets, is increasing across all countries and all socio-economic groups. The adoption of Internet surveys has spread rapidly, driven by the promise of faster and cheaper data collection (Couper, 2008).
Online panels typically pre-recruit their sample members for regular online interviews on diverse topics. Because online panels can invite their members by means of inexpensive e-mail messages and do not require the employment of interviewers, this mode of data collection is a cost-efficient alternative to the traditional modes (Dillman & Bowker, 2001). In addition, panels allow us to regularly re-interview the same respondents, thus enabling longitudinal research programs that investigate changes over time. For respondents, online panels are attractive, since participants can fill in the online questionnaires at their own pace and at times that are most convenient to them. Due to the self-completion format without interviewers present, social desirability biases can be reduced (e.g., Kreuter, Presser, & Tourangeau, 2008).
As with any mode of data collection, we distinguish between online panels that are based on a probability sample of the target population (i.e., probability panels) and those that recruit their respondents by means of a convenience sample (i.e., nonprobability panels; e.g., Couper & Bosnjak, 2010). According to statistical theory, drawing a probability sample is a necessary precondition to make inferences about the target population. However, volunteer access panels, in which respondents self-select themselves into the sample, still dominate the survey landscape. Such panels are attractive due to their low recruitment and maintenance costs. Their quality in terms of representativeness and measurement error, however, remains questionable (e.g., Callegaro et al. 2014; Krosnick, MacInnis, Suh, & Yeager, 2013; Yeager et al., 2011).
In addition to the sampling strategy (probability vs. nonprobability), the coverage of the target population and nonresponse is important for the representativeness of a sample (Groves et al., 2009). In online surveys, noncoverage is especially worrisome for those who do not have a computer and Internet access. Being “online” is typically related to age, education, and employment status and thus of relevance to many key social and economic research questions (see Bandilla, Kaczmirek, Blohm, & Neubarth, 2009; Leenheer & Scherpenzeel, 2013; Blom, Gathmann, & Krieger, 2015). In Europe, there are, to date, four probability-based online and mixed-mode panels that include the offline population by either providing respondents without computers and/or Internet access with the necessary equipment or interviewing them in mail surveys, as part of a mixed-mode strategy. These panels are—in chronological order of their establishment—the Longitudinal Internet Studies for the Social Sciences (LISS) Panel in the Netherlands, the German Internet Panel (GIP), the Longitudinal Study by Internet for the Social Sciences (ELIPSS) Panel in France, and the GESIS Panel in Germany. 1 All four panels are recruited via offline contact modes, with intensive recruitment efforts, and invest in incentives and careful panel management, in order to maintain high response rates and low attrition rates over time.
Across Europe, there is currently considerable interest in setting up probability-based online panels. For example, in the United Kingdom, an initiative supported by the National Centre for Research Methods gauged the viability of a probability-based online panel for the general population, 2 followed by an expertise into this issue commissioned by the U.K. Economic and Social Research Council. Similar efforts are currently being undertaken in Norway; several other countries are still investigating the viability of such initiatives for themselves.
This article aims to outline the range of methodological options available when setting up a probability-based online panel that includes (previously) offline persons. By showcasing the choices made by the four existing panels in Europe—the LISS Panel, the GIP, the ELIPSS Panel, and the GESIS Panel—we introduce their methodological similarities and differences to the scientific community.
In the following section, we survey each panel briefly. This overview presents key characteristics of the panels, such as sample sizes, wave frequency, and data access. The third section describes the target populations and corresponding sampling procedures. It includes details about how the part of the target population that does not use computers and/or the Internet is included in the panels. The fourth section looks into the details of the offline recruitment process and reports on response rates obtained so far. The fifth section describes how the panels are managed and which panel care measures are adopted. Finally, the sixth section concludes by discussing the implications of the design choices taken by the four panels, both for survey practitioners involved in similar projects and for analysts of the four panels’ cross-national data collections.
LISS Panel, GIP, ELIPSS Panel, and GESIS Panel—An Overview
The LISS Panel (the Netherlands) 3 was first established in 2007 and is the central resource in An Advanced Multidisciplinary Facility for Measurement and Experimentation in the Social Sciences (MESS) funded by the Netherlands Organisation for Scientific Research and led by CentERdata, Tilburg University. MESS is designed to stimulate, as well as integrate, research in the social sciences, life sciences, and behavioral sciences in the Netherlands and abroad. The infrastructure, which includes a large household panel and a data archive, is open to academic researchers and policy makers all over the world.
The GIP 4 (Germany) was setup in 2012 and is part of the Collaborative Research Center Political Economy of Reforms (SFB 884) at the University of Mannheim and is funded by the German Research Foundation. The GIP is the central data collection of the research center, which is composed of approximately 15 project teams of political scientists, economists, and sociologists. Although the data are made available to the scientific community (worldwide) and free of charge, submissions to the questionnaire can only be made by the project teams of the research center.
The ELIPSS Panel (France) 5 is part of the larger project entitled Data, Infrastructure, Methods of Investigation in the Social Sciences and Humanities (DIME-SHS), led by Sciences Po and bringing together seven French research institutions. The panel currently fields a large-scale pilot study, which was recruited in 2012; the start of the main study is scheduled for 2015. During the pilot study, the ELIPSS Panel primarily served a defined group of social scientists at the DIME-SHS research institutions in terms of questionnaire submissions. After the conclusion of the pilot study, calls for proposals can be submitted by the scientific community (worldwide). Implementing surveys in the ELIPSS Panel is currently free of charge.
The GESIS Panel (Germany) 6 is located at the GESIS—Leibniz Institute for the Social Sciences and funded by the German Federal Ministry of Education and Research. The GESIS Panel recruited its current sample in 2013 and is a general survey research infrastructure. Questionnaire proposals are accepted from researchers across Germany and worldwide. Questionnaire implementation in the panel as well as the data are free of charge.
All four panels make their data publicly available to the scientific community. The LISS Panel data can be found in the LISS Data Archive. 7 The GIP and the GESIS Panel both grant access to their data through the GESIS Data Archive for the Social Sciences. 8 The Socio-Political Data Centre is responsible for data collection and documentation in the ELIPSS Panel, and the data are disseminated through the portal of the French data archives for social sciences (Réseau Quetelet). 9 Thus, while in terms of accepting research proposals, the GIP primarily serves researchers based at the Collaborative Research Centre (SFB 884) at the University of Mannheim, the LISS Panel, the ELIPSS Panel, and the GESIS Panel are more general research infrastructure projects that allow researchers worldwide to submit proposals for data collection in their panels.
There are key similarities across the four panels: the random probability samples of the general population, the inclusion of (previously) offline respondents in the panel (see also Section 3), and strategies for recruiting and maintaining a high-quality sample. However, the approaches chosen to achieve probability-based online panels for the general population differ across the panels. Table 1 provides an overview.
Overview of Approaches.
The LISS Panel was first initiated in 2007, then the GIP and the ELIPSS Panel followed its example in early and late 2012, respectively, and the GESIS Panel in 2013. The LISS Panel is, at the same time, the largest of the panels with approximately 8,000 active panel members in July 2014, followed by the GESIS Panel, with more than 4,800, and the GIP, with more than 1,500 panel members in July 2014 (a refreshment sample of another 3,500 panel members has been recruited in 2014). The ELIPSS pilot study is based on an initial sample size of 1,026 panel members; the recruitment of the main study aims to reach 4,000 panel members in 2015.
The frequency and intensity with which the panel members are interviewed differ across panels. In the LISS Panel and the ELIPSS Panel, they are invited to interviews that take approximately 30 min every month, while the GIP and the GESIS Panel conduct interviews lasting approximately 20–25 min every other month. For the LISS Panel, the GIP, and the ELIPSS Panel, the fieldwork period for each wave is 1 month, while the GESIS Panel maintains a 2-month fieldwork period to account for the returning of the paper questionnaires in the offline mode (see Section 3). The start dates of the waves also vary: While the LISS Panel starts on the first Monday of each month, the ELIPSS Panel launches on the first Thursday of each month, the GIP starts on the first day of each uneven month (e.g., January 1, March 1, through November 1), and the GESIS Panel commences on the 15th day of each even month (e.g., February 15, April 15, through December 15).
All panels outsourced the recruitment of panel members to professional face-to-face and telephone fieldwork agencies. 10 However, the panels take different approaches to running the online data collections. The LISS Panel and the ELIPSS Panel manage the complete online survey operation at their respective institutes. This includes the submissions of questionnaires, programming and testing of the online questionnaires, and the management of panel members, including the maintenance of a telephone hotline. The GESIS Panel conducts almost all survey operations in-house. However, it outsources the printing and posting of the paper questionnaires and maintains an external hotline that panel members can access 24/7. The GIP, in contrast, coordinates the questionnaire development and testing of online questionnaires in-house, but all programming and panel maintenance measures are outsourced to a single data collection agency, which cooperates closely with the GIP research team.
The differences in design across panels stem, to some extent, from differences in methodological convictions about optimal survey strategies and, to a further extent, from differences in funding structures and the institutional settings of the panels. For example, whether a panel encourages questionnaire submissions from the general research community depends on the overall purpose of the panel and the reasons why the respective national research council funded the study. Decisions about start date and frequency of waves (every month or every second month) depend on the panel’s capacity for managing frequent questionnaire design and programming cycles, as well as on convictions about the relationship between attrition rates, interview frequency, and timing of wave invitations. Unfortunately, while there is empirical evidence about some design decisions, such as the effect of contact mode and incentives at recruitment (see Section 4), the effect of organizational decisions on the general design of surveys, such as the outsourcing of survey operations and the timing of each wave, remains untested.
Target Populations, Coverage, and Sampling
The key goal of all four panels is to repeatedly survey a sample of the general population in a self-completion mode over time, yet also including persons who have no private access to a computer and/or the Internet, or who hesitate to participate online, despite having access to the necessary equipment. Table 2 displays how the panels chose to accomplish this goal in terms of coverage and sampling. The strategies vary across panels; some of this variation is due to differences across countries in available sampling frames. However, the numerous differences also reflect the array of options available to achieve this one goal.
Target Populations, Coverage, and Sampling.
One key difference across panels is the unit of analysis. While the LISS Panel is a panel of households, the GIP, the ELIPSS Panel, and the GESIS Panel are panels of individuals. In household panels, researchers are primarily interested in the household as a whole and in interactions between household members. Household panels thus typically aim to interview all household members, which allows for analyzing household dynamics and individual characteristics (e.g., Cherchye, De Rock, & Vermeulen, 2012). In panels of individuals, by contrast, the research questions focus on individual characteristics, attitudes, and behaviors. The choice of a household versus an individual panel has operational consequences. The LISS Panel follows all original household members also when a household splits up and when new members enter an existing household. Household members who become age eligible (i.e., turn 16 years old) are invited into the panel. The GIP also interviews all household members who are age eligible and live in a selected household at recruitment. However, it is a panel of individuals who are clustered in households rather than a household panel. The key difference is that the GIP only follows those persons who were part of the sample at the time of recruitment. New household members and children who become age eligible during the lifetime of the panel are not invited to participate in the GIP. Both the ELIPSS and the GESIS Panels recruit individuals, typically one person per household, and follow these individuals, rather than the entire household, over time.
The four panels follow different strategies regarding how sample units that do not have a computer and/or Internet at the time of recruitment are included in the panel. The LISS Panel equips previously offline households with broadband Internet and a special computer, the so-called simPC. It is operated by large “buttons” for the most frequently used functions and has screens that are designed to be easily readable for people who are sight impaired. In addition, a large button with an LISS logo is available for easy access to the questionnaires. Within offline households, all household members can participate in the panel via this equipment. When a household moves, all the equipment is re-installed by a service company or the respondent at the household’s new address. In case a split-off household is followed, new equipment is installed at the new household, if necessary.
The GIP also equips previously offline households with a user-friendly computer and broadband (3G or LTE) Internet. In 2012, the computer equipment consisted of a BenPC, a specially programmed touch-screen desktop computer, which is very similar in its operation to the LISS simPC. Given the rapid technological advances in user-friendly tablet computers, the GIP changed its equipment for the previously offline households in the 2014 refreshment sample to tablets with a special GIP interface and an external keyboard. When original sample members in an equipped household move, the GIP ensures that all equipment is reinstalled at the new location and, if necessary, new equipment is provided. Since the GIP is a panel of individuals, only the originally sampled persons are followed over time and, if needed, equipped with a computer/tablet and Internet access.
The ELIPSS is a sample of individuals and provides every panel member whether previously online or offline with an A5-sized tablet computer and free 3G Internet access. The equipment is thus a personal device, independent of household structures and moves. Interviews are conducted through a special ELIPSS app installed on each tablet, thus eliminating differences in the online display of questions across different types of computer systems (see Callegaro, 2010; Couper, 2008, ch. 4; Tourangeau, Conrad, & Couper, 2013, see also the unimode design principle put forward by Dillman, Smyth, & Christian, 2009, ch. 6; Link et al., 2014).
Finally, the GESIS Panel surveys persons who are unable or unwilling to participate online via mailed paper questionnaires. Because it is a sample of individuals, the GESIS Panel does not follow household splits, but regularly updates its address database for panel members who move, to ensure that letters and mailed paper questionnaires reach their destination.
The age range of the target population is another aspect that any survey of the general population needs to define. In social surveys, the youngest age-group typically commences at 16 or 18 years, depending on both the research aims and the national legal restrictions regarding interviews with minors. In the case of our four panels, two chose a lower age bound of 16 years (the LISS Panel and the GIP) and two interview persons who are at least 18 years old (the ELIPSS Panel and the GESIS Panel). The upper age bound is typically defined by survey practicalities. Older age cohorts are difficult to survey in any mode and pose particular difficulties in self-completion modes, in which the respondents receive no help from an interviewer to read out question texts and answer options. For this reason, three of the four panels chose an upper age bound, that is, 70 years in the case of the GESIS Panel and 75 years in the case of the GIP and the ELIPSS Panel. The LISS Panel does not have an upper age bound and includes all ages, starting at 16 years. Both the lower and upper age bounds are in place at the time of recruitment for all four panels. As the panel ages, the target population thus ages in parallel with it. The exception is the LISS Panel, which, as a household panel, adds household members when they turn 16. None of the three panels that use an upper age bound excludes those panel members who reach the upper age bound.
The types of sampling frames used differ across surveys. While the LISS Panel and the GESIS Panel draw their samples from national and municipal population registers, the GIP and the ELIPSS Panel use area probability samples. The motivation for these choices was access to suitable frames as well as sample efficiency. According to Lynn, Häder, Gabler, and Laaksonen (2007), different sampling strategies for probability samples across countries lead to equivalent samples, as long as strict probability sampling is adhered to and each sampling unit has a known non-zero selection probability. The different sampling frames and probability sampling strategies adopted by the four panels thus allow for comparative survey research.
In the Netherlands, a detailed population register is centrally available. The Dutch national statistical office (Statistics Netherlands) drew a simple random sample of addresses for the LISS Panel recruitment. For each address, a name was selected from the register and the letter and envelope were addressed to this selected name (to avoid letters addressed to “the inhabitants of this address,” since these are likely to be thrown away unopened). After the recruitment, all household members aged 16 years or older are identified as eligible to be panel members. Including the complete household in the sample reduced the costs of equipping previously offline panel members because a complete household could be included with one set of equipment (see also Scherpenzeel & Das, 2011).
In France, the national statistical office (INSEE) granted ELIPSS access to its list of housing units from the rotating census. For this purpose, INSEE drew a stratified two-stage probability sample of housing units. Within the households, one person was randomly selected to participate in the panel.
In Germany, access to registers can only be granted through local municipalities. This means that, first, a random sample of municipalities needs to be drawn. Selected municipalities are then approached and asked to draw a sample of individuals. Since the GESIS Panel aimed to include one person per household, this strategy promised the highest quality and sample efficiency. In addition, paper questionnaires are mailed to GESIS Panel members who do not have access to the Internet or who do not wish to participate in surveys online. Thus, household clustering would not have generated significant cost-savings.
In the case of the GIP, however, the research team decided to sample all household members. Since household contexts were relevant to the research questions and previously offline households were to be equipped with computers and Internet access, including all household members seemed prudent. In such a situation, however, the municipal registers of individuals are not the most efficient sampling strategy in Germany, due to regional clustering and unequal selection probabilities that depend on household size. Therefore, an area sampling strategy with areas that were randomly selected and households that were listed without interval along a predefined route prior to the fieldwork was chosen. From the resulting list of household addresses, the gross household sample was drawn and all household members included with equal selection probabilities.
To ensure that the number of panel members stays about the same over time and to correct for selective dropout, three refreshment samples were added to the LISS Panel in 2009, 2011, and 2013. As with the initial sample, Statistics Netherlands drew the samples from the population register. In 2009, a stratified sample was drawn to improve the representativeness of the panel by oversampling the difficult-to-reach groups that had a below-average response in the main recruitment. In 2011, a simple random sample was used, while in 2013, an oversampling approach was again taken.
The GIP refreshed its sample and increased its sample size with a refreshment sample in 2014. The refreshment sample follows the same strategy as the sample in the initial recruitment: No segments of the population are oversampled. The reason for this is two-fold. On the one hand, methodologists have different opinions on oversampling for selective dropout, since bias due to dropout might well be related to factors that cannot be accounted for by the oversampling. On the other hand, oversampling certain segments of the population is not practically implementable in the area probability sampling strategy adopted by the GIP.
In 2015, the ELIPSS sample size will be increased by recruiting around 3,000 new panel members for the main panel. The sampling strategy will be based on a sample frame of addresses similar to that used for the pilot in 2012. As with the pilot study, one person will be randomly selected from the list of eligible members within each household.
This section demonstrates how various design choices follow from each other and how these choices influence different survey errors. For example, choices that may initially stem from considerations regarding measurement, such as surveying offline households by mail, equipping them with computers and Internet, or providing every panel member with the same survey device, may be connected with the sampling strategy chosen. This, in turn, might influence the recruiting modes available, as the following section showcases.
Offline Recruitment
As mentioned earlier, all four panels recruited panel members offline via face-to-face, telephone interviews or mail recruitment. In all countries, the recruitment strategy was based on a face-to-face sampling frame. For telephone recruitment, this sampling frame was augmented with telephone numbers. The panels chose this recruitment strategy because samples for face-to-face surveys typically show fewer coverage problems (Busse & Fuchs, 2012; Joye, Pollien, Sapin, & Ernst Stähli, 2012; Lynn, 2003) and allow for higher response rates (Lipps & Kissau, 2012; Peytchev, Carley-Baxter, & Black, 2011). Furthermore, all four panels made substantial efforts to recruit a sample with high response rates and low nonresponse bias, adopting key response-enhancing measures such as advance letters and information material, well-trained interviewers, several reapproaches for noncontacted sample units, and refusal conversion measures as well as monetary incentives at various stages of the recruitment process (see Groves et al., 2009 for an overview). A summary of the recruitment strategies used by the four panels is provided in Table 3.
Recruitment Strategies at First Sample Recruitment.
Note. Due to differences in sampling frames, survey design, and available information on the cases with unknown eligibility, the panels report slightly different response rates, but all follow the Standard Definitions the American Association for Public Opinion Research (AAPOR, 2011). The final disposition codes used to calculate the response rates are based on most-recent coding in the LISS Panel and the ELIPSS Panel, while the GIP and the GESIS Panel implemented priority-coded disposition codes (see Blom, 2014 for a discussion). The LISS Panel reports household response rates, since it is a household panel. The GIP, the ELIPSS Panel, and the GESIS Panel report person response rates, since their units of analysis are persons. The overall response rates at panel registration are equivalent to RECR × PROR, that is, the recruitment rates times the profile rates, as defined by AAPOR (2011, pp. 36–37).
aAAPOR RR2, including short recruitment interview as partial interviews; bAAPOR RR3; cAAPOR RR4 assuming 1.78 eligible persons per household for households in which the exact number of household members is unavailable; dAAPOR RR5.
The exact approaches taken, however, differ across the panels. Many of these design choices of the four panels were not based on empirical evidence across panels. However, several experiments were performed within some of the panels to optimize certain panel strategies. In the LISS Panel, methodological experiments were performed on factors related to recruitment—contact mode, incentive amount, timing of the incentive, content of the advance letter, and timing of the panel participation request (see Scherpenzeel & Toepoel, 2012). An experiment on the use of conditional and unconditional incentives at two stages during the recruitment was conducted in the GIP (see Blom et al., 2015). In the ELIPSS pilot study, an experiment was conducted on the use of unconditional incentives. Because the experiments were conducted only within panels and not between panels, it is not possible to evaluate their (potential) differential effect on the recruitment across the panels.
During the offline recruitment phase, all four panels sent out advance letters to announce the visit or call of an interviewer. In addition, information leaflets were used to explain the purpose of the panels to the target persons.
While the LISS Panel recruited almost all respondents through both telephone and face-to-face interviews, and the ELIPSS pilot study additionally used recruitment via mail, the two German panels limited recruitment to face-to-face interviews. The three approaches exemplify cross-national differences in research culture and, potentially, survey climate (see Lyberg & Dean, 1992). In France and the Netherlands, the approach was to recruit respondents through a variety of channels, including less expensive postal invitations and phone interviews. For the LISS Panel, the choice for the multi-mode recruitment strategy for the initial recruitment was based on the outcome of an experiment during a pilot study (see Scherpenzeel & Toepoel, 2012). In Germany, survey researchers are concerned about low cooperation rates when the target persons are first contacted by telephone, because this typically leads to more refusals (see Blohm, Hox, & Koch, 2007). The ELIPSS pilot study was based on a sequential recruitment design (mail, then telephone, then face-to-face) and confirmed that recruitment efficiency is highest via face-to-face interviews (see Cornilleau, Cousteaux, & Legleye, 2015). For this reason, the sample of the ELIPSS main study in 2015 will be based on personal visits at the addresses by professional interviewers.
Decisions regarding the moment at which target persons are officially invited to the online panel differed across studies. In the LISS Panel, face-to-face and telephone interviewers requested an e-mail address for the household’s contact person; for other household members, the e-mail address was registered when completing the first online questionnaire on the household composition. Among other things, these e-mail addresses are used for inviting and reminding panel members to complete new online questionnaires. The German studies gave the interviewers a smaller role. The GIP and the GESIS Panel sent the actual panel invitation via letter from the research teams after the interviewer had left the household. The interviewers mentioned the panel when conducting the face-to-face interviews, but the official invitation came by postal mail approximately a week later. In the case of the ELIPSS Panel, depending on the mode in which target persons had been contacted, a formal agreement was either sent by the office (in case of telephone or mail contact) or handed over by the interviewer (in case of face-to-face contact). This formal agreement to be part of the ELIPSS Panel had to be signed by each respondent, in order to receive the tablet.
Because several studies had demonstrated positive effects of respondent incentives on response rates in face-to-face and telephone surveys (for example, Singer, van Hoewyk, Gebler, Trivellore, & McGonagle, 1999; Singer & Ye, 2013), all the panels used incentives during the recruitment. Typically, incentives of 5€ or 10€ were implemented either conditionally or unconditionally upon participation in the offline recruitment interview. Because the recruitment interviews were relatively short—approximately 10–15 min—this incentive was regarded as sufficient for obtaining good response rates.
In addition, the LISS Panel, the GIP, and the ELIPSS Panel conducted their own incentives and recruitment design experiments. The experiments in the LISS Panel showed that the response rate increases with the incentive level, but incentives above the 10€ level did not substantially increase response rates beyond those seen at the 10€ level (Scherpenzeel & Toepoel, 2012). The GIP found that a 5€ unconditional cash incentive sent in the advance letter yielded significantly higher response rates than a promise of 10€ in cash after the recruitment interview (Blom et al., 2015). In the ELIPSS pilot study, an unconditional incentive of 10€ enclosed with the advance letter was offered to half of the sample. It significantly increased the chance of participation in the panel, compared to receiving no incentive (odds ratio [OR] = 1.4; p = .0004). Additionally, all panels reviewed the effectiveness of various methods and procedures to minimize nonresponse in self-administered surveys (see, for example, Church, 1993; Cook, Heath, & Thompson, 2000; Fox, Crask, & Jonghoom, 1988; Lozar Manfreda, Bosnjak, Berzelak, Haas, & Vehovar, 2008; Shih & Fan, 2007) and implemented tested procedures accordingly.
Because the survey designs, available resources, timing, interviewer staff, and national survey cultures differed across countries, the response rates that were achieved also differed (see also Blom, de Leeuw, & Hox, 2011; Johnson, O’Rourke, Burris, & Owens, 2002; Lyberg & Dean, 1992).
In the following, we report the response rates for the recruitment of the four panels. For the LISS Panel, we report the household response rates, because the unit of analysis here is the household. For the GIP, the ELIPSS Panel, and the GESIS Panel, person response rates are reported (see Table 3). Two recruitment stages can be distinguished in all panels: (1) the offline recruitment stage and (2) the registration for the online panel. Since the two stages may be related to different nonresponse bias mechanisms, we report response rates both for the offline recruitment interview and for the overall registration to the online panel (based on the full initial gross sample). All reported response rates are based on the American Association for Public Opinion Research (AAPOR) Standard Definitions (2011) for face-to-face surveys, as the sampling frames of all panels were most similar to the sampling frames of face-to-face data collections. Due to the differences in survey design (e.g., whether the design included short doorstep interviews that were counted as partial interviews) and available information on the gross sample (e.g., whether cases of unknown eligibility existed and how they were recorded), it is necessary to report slightly different AAPOR response rates across the panels. In the LISS Panel, the 2007 recruitment yielded an offline household response rate of 73.2% (AAPOR RR2); the overall response rate at online panel registration was 48.3%, based on the full gross sample (AAPOR RR3). In 2012, the GIP reached a response rate of 52.1% (AAPOR RR2) at the offline stage and an overall person response rate at online panel registration of 18.1% (AAPOR RR4). The ELIPSS 2012 pilot recruitment yielded a person response rate at the offline recruitment interview of 31.3% (AAPOR RR3) and an overall person response rate for the online panel of 27.3% (AAPOR RR3). In the GESIS Panel, the 2013 recruitment yielded a face-to-face person response rate of 38.6% (AAPOR RR5); the overall response rate for the panel was 25.1%, based on the full gross sample (AAPOR RR5). Note that, for all panels, the overall online response rate is equivalent to RECR × PROR, that is, the recruitment rate times the profile rate, as defined by AAPOR (2011, pp. 36–37).
The LISS Panel, the GIP, and the ELIPSS Panel have conducted analyses into the sample composition of the online panels, that is, of those respondents who became panel members. The samples show biases regarding age (over-representing younger and under-representing older persons) and education (over-representing higher educated persons). Challenges related to the representation of the (previously) offline population in the online panel are being met. The LISS Panel, the GIP, and the ELIPSS Panel all include 7–10% panel members who were previously offline. It is difficult to quantify how this compares to the proportion of offliners in the population at the time of recruitment. Judging from information gathered during the offline recruitment interviews, the group of offliners is typically twice as large. 11 So far, detailed analyses into representativeness have only been conducted and published on the LISS Panel (see Scherpenzeel & Bethlehem, 2011). For all other panels such analyses are still in train and conclusions are tentative. Importantly, insights into how representative these panels are compared to both offline probability-based panels conducted face-to-face or by telephone and online panels based on nonprobability samples are still lacking.
Maintaining the Online Panels
Retaining respondents in a panel after they have been recruited requires sustained effort (see, for example, Göritz, 2006; Millar & Dillman, 2011). For this reason, all four panels implement several panel care measures, including regular interaction with the panel members through various channels, as well as incentives for participating in the online surveys.
At the beginning of each wave, the panels invite each panel member personally and, after 1 or 2 weeks, send reminders to the nonrespondents. The modes in which this happens differ slightly across panels (see Table 4), depending on the mode in which panel members are typically reached. For example, whereas for the LISS Panel, the GIP, and the ELIPSS Panel, e-mail or push messages are the typical mode of contact with panel members, the GESIS Panel uses postal communication for those panel members who participate via paper questionnaires and invites all panel members at each wave via postal mail. In addition, the GIP has recently switched to inviting previously offline households via postal mail, since they proved difficult to reach electronically.
Panel Care Activities.
The incentives used at each wave differ slightly across panels. The LISS Panel, the GIP, and the GESIS Panel use similar incentive amounts of 4€–5€ per 20-min interview. However, while the GESIS Panel and the GIP use fixed size incentives, independent of the actual length of the questionnaire in any month, the LISS Panel pays 15€ per hour interview time. The questionnaire length, and thus also the incentive amount, is estimated by the LISS team before the questionnaire is fielded, based on the average completion time of respondents. Furthermore, the GIP rewards a yearly bonus of 10€ if a panel member participated in all waves and 5€ if a panel member participated in all but one wave during that year. LISS panel members who have not participated for 3 or more consecutive months, so-called “sleepers”, are offered a conditional incentive of 10€ if they participate again. Finally, the LISS Panel and the GIP pay out incentives that are conditional on participation in a particular wave, while the GESIS Panel sends unconditional incentives, in cash, together with the postal invitation to the wave.
Paying out incentives to panel members is also handled differently across panels. In the LISS Panel, incentives earned are registered on the personal LISS Panel pages of the respondents and transferred to their bank account once every 3 months. In the GIP, respondents can choose between a bank transfer, an Amazon voucher, or a donation to a charity. Incentives are also accumulated and paid out every half year. The GESIS Panel sends unconditional cash incentives to panel members with the invitation letter at each wave. The ELIPSS Panel does not work with monetary incentives in the online panel. Because all participants receive a tablet computer with a 3G Internet connection that they can also use for other purposes, this is considered as an incentive for their participation (see Cornilleau et al., 2015). Indeed, for more than 60% of panel members, the free tablet was the primary motivation to participate in the panel. Additionally, 90% of panel members reported that they also use the tablet for purposes other than answering questionnaires.
Although its effect is difficult to quantify, all panels consider personal and close contact with panel members to be crucial for their long-term participation. Attrition from a probability-based online or mixed-mode panel may be caused by two distinct events. First, panelists may actively dropout by notifying the research team that they do not wish to be invited to any further surveys. Second, through rule-based exclusion, that is, panelists who have not participated in a certain number of successive waves are excluded from further invitations to the panel. With the help of intensive contact and communication procedures, retention rates of above 90% after several waves of data collection typify the four panels presented in this article (see Figure 1). To implement such close contact, all four panels have setup channels through which panel members can reach the panel management teams directly via e-mail, phone, or messages on the internal part of the study website (see Table 4). Furthermore, the LISS Panel, the GIP, and the ELIPSS Panel conduct personalized actions (letters, phone calls, etc.) to re-activate panel members who missed several waves. In addition, the panels aim to stay in touch with the sample members by providing study results via e-mail, mail, or on a website, and by introducing researchers involved with the study, thus giving the questionnaires a “face” to associate with.
Retention rates during the first year. Notes. The basis for the retention rates is the first online or mixed-mode wave (100%). Rates are reported per month for the first year. Because the LISS Panel and the ELIPSS Panel collect data every month and the GIP Panel and the GESIS Panel collect data every 2 months, different numbers of data points are reported. The LISS Panel reports the household retention rates, while the GIP, the ELIPSS Panel, and the GESIS Panel report person retention rates. The LISS Panel retention rates refer to the first year of the panel recruited in 2008, the GIP retention rates to the first year of the panel recruited in 2012, the ELIPSS Panel retention rates to the first year of the pilot panel recruited in 2012, and the GESIS Panel retention rates to the first three waves of the panel recruited in 2013/2014.
The approaches to reaching a high-quality online or mixed-modes panel of the general population differed across the four panels, in terms of both the recruitment and regular panel maintenance measures adopted. However, the similarities with regard to high efforts in obtaining and maintaining the panel are also apparent. As shown in Figure 1, as a result, the achieved retention rates (i.e., the proportion of the original panel that is still a panel member and can be approached for interviews) are highly similar across panels and above 90% even after a year of data collection.
Discussion
Many roads can lead to a high-quality probability online and mixed-mode panel covering the general population. Using the LISS Panel, the GIP, the ELIPSS Panel, and the GESIS Panel as examples, this contribution showcases the variety of design choices taken along the way and demonstrates how these choices interact with each other. We thereby aim to demonstrate to survey practitioners embarking on similar endeavors the array of available tools and the operational implications that these have.
One key design choice for all panels representing the general population is the way in which persons without computer and/or Internet access are included in the panel. This decision is driven by considerations regarding both measurement equivalence and coverage. The ELIPSS Panel decided to aim for maximum measurement equivalence by subjecting all panel members to exactly the same stimulus. For this purpose, all panel members receive a tablet computer with 3G Internet connection. All questionnaires are fielded through an applet on the tablet and are thus displayed in exactly the same way to all respondents. The GESIS Panel, in contrast, aims for maximum coverage. Since persons without computers and Internet access are typically reserved with respect to new technologies, the GESIS Panel offers the possibility of participating in the panel via mailed paper questionnaires. Although this method is attractive to persons with low technical affinity, it results in a mixed-mode design, with one group of panel members interviewed online and the other interviewed on paper. The LISS Panel and the GIP chose an in-between option. They conduct online surveys, which respondents with access to the Internet can complete through the web, and provide previously offline respondents with the necessary equipment. By ensuring that the computers provided are specifically devised for persons without prior computer experience and by supplying various personal support channels, the LISS Panel and the GIP aim to minimize the burden for panel members and, thus, maximize coverage. At the same time, the web survey mode for all panel members aims for high measurement equivalence. However, a single stimulus, as generated in the ELIPSS Panel, is not achieved in the LISS Panel and the GIP, due to differences across the devices (computers, tablets, and smartphones) and browsers that are used.
Another design choice is whether the panel is to serve as an open research infrastructure for the general scientific community and, consequently, to encourage external researchers to submit questionnaires, or whether it is a topic-oriented panel where a limited group of researchers determines the questionnaire content. This decision is likely to impact on whether the infrastructure is of a more longitudinal or cross-sectional nature because, with a general infrastructure, it is more difficult to coordinate a long-term longitudinal research program, as the group of external scientists involved is continuously changing. By contrast, involving a wide range of disciplines in an open infrastructure creates new (possibly yet unknown) opportunities for secondary analyses by linking variables from different cross-sectional studies in a longitudinal context.
The longitudinal versus cross-sectional character of an infrastructure may, in turn, influence decisions on initial sample sizes and refreshment samples. If longitudinal research is the basic goal of the panel, it makes sense to initially recruit a large sample that can be followed for several years. Although the sample becomes smaller over time, due to attrition, the large initial sample ensures sufficient respondents whose characteristics, attitudes, and behavior can be followed over time, even after years of data collection. With the aging of the panel and the associated dropout, however, there may be increasing mismatches between the panel and the population. For longitudinal research, this mismatch is less of a concern, since the main interest is in observing changes over time. However, if the research questions are predominantly of cross-sectional nature, it makes sense to draw a somewhat smaller sample at the outset and regularly (e.g., every year or every other year) refresh the sample with a newly recruited set of participants. One might even consider a rolling cross-sections design, where panel members—by design—leave the panel after a set number of waves, while new panel members enter it. With such regular updates of the sample, a continued representation of the population can be ensured. The cross-sectional research thus becomes more accurate; however, longitudinal research questions across several years encounter difficulties in such a design.
Since the design choices were not experimentally tested across panels, we are unable to make statements about their differential effect on our data. By describing the similarities and differences across panels, we intend to make analysts aware of them. We encourage analysts who use these data to consider the methodological implications of the differences presented. In April/May 2014, the LISS Panel, the GIP, the ELIPSS Panel, and the GESIS Panels concurrently implemented a joint wave of data collection, for the first time. In this joint wave, key questions of mutual interest, borrowed from existing cross-national surveys and adapted to the online mode, were fielded in all four panels. The data are available to the scientific community for cross-national analyses. In the coming years, we aspire to implement further joint waves, thus generating more comparative data. In the long run, our aim is to work toward a more elaborate and integrated cross-national data collection across the four panels and across new panels emerging throughout Europe and the rest of the world.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The LISS Panel data were collected by CentERdata (Tilburg University, The Netherlands) through its MESS project funded by the Netherlands Organization for Scientific Research (grant number 176.010.2005.017). The German Internet Panel is the central data collection (project Z1) of SFB 884 at the University of Mannheim and is funded by the German Research Foundation (DFG). The ELIPSS Panel is part of the larger project entitled Data, Infrastructure, Methods of Investigation in the Social Sciences and Humanities (DIME-SHS) led by Laurent Lesnard and supported by a public grant from the French National Research Agency (ANR-10-EQPX-19-01). The GESIS Panel is funded the German Federal Ministry of Education and Research.