Response Quality of Self-Administered Questionnaires

Abstract

The objective of this article is to analyze the quality of the information collected by a self-administered survey addressed to citizens of Andalusia, who were offered the possibility of answering using the post or Internet. The study showed the advantages of using web-based self-administered questionnaires. Web surveys showed a low number of unanswered questions, more detailed answers to open questions, and longer answers to questions than those generated from paper questionnaires. The five open questions with text showed longer answers from the web survey, around 63 characters more. In the open questions with numerical answers, the use of drop-box (select list) generated better response than the use of a blank space left in the paper questionnaire.

Keywords

self-administered questionnaire mail survey web survey response quality

Introduction

When surveys were first conducted, questionnaires were administered through face-to-face interviews, the leading mode of data collection in surveys from the initial applications of this technique until the 1980s (Fowler & Mangione, 1990; Groves, 2012; Rossi, Wright, & Anderson, 1983; Wright & Mardsen, 2010). Since then, new innovations and social conditions (Couper, 2005, 2011; Dillman, Smyth, & Christian, 2009) have produced a notable decline in the use of face-to-face surveys in favor of quicker and more economical modes, such as telephone interviews, up to the end of the 20th century, and the self-administered web surveys conducted today.

Several researchers suggest that web surveys attract lower item non-response rates (Boyer, Olson, Calatone, & Jackson, 2002; Denscombe, 2006; Kwak & Radler, 2002; Schaefer & Dillman, 1998; Tourangeau, Rips, & Rasinski, 2000), although there is no unanimous position on this issue. Some studies placed higher rates of non-response in web surveys (Lozar Manfreda, Bosnjak, Berzelak, Haas, & Vehovar, 2008), as well as noting that there is a higher number of interviewees who leave the survey incomplete (Brecko & Carstens, 2006). Other experts consider that the quality of response varies according to the type of questions used (Smyth, Dillman, Christian, & Stern, 2005), whether the questions require qualitative or quantitative answers (Lozar Manfreda & Vehovar, 2002; Stangl, 2004), or whether the questionnaire contains open or closed questions (Reja, Lozar Manfreda, Hlebec, & Vehovar, 2003). Concerning this point, a large number of researchers think that too little attention has been paid to the quality of responses of web questionnaires and, more specifically, to “the way that Web questionnaires might affect respondents’ willingness to complete individual items within a questionnaire” (Denscombe, 2009, p. 282).

The objective of this article is to analyze the quality of the information collected by a postal and web survey given to a general population, something not very common in the literature, as the majority of the studies have used specific samples (students, etc.). The population that was the object of this study are citizens of Andalusia (a region in Southern Spain) with the right to vote while being resident in a foreign country, who were offered the possibility of answering using the post or Internet.

As a starting point, the fact that Internet users do not read the questions in a questionnaire carefully but instead scan the text was considered. This happens because respondents have a number of windows open at the same time (software and websites), and due to the need to read quickly in order to process the large amount of information available on the Internet. This can lead to problems in understanding the meaning of the questions, which results in the incorrect completion of the questionnaire.

The proposition to be investigated is to show that the greater “pace” (or speed) of the web mode, together with the large number of distracting elements, will produce a poor completion of the questionnaire (De Leeuw, 2005; Heerwegh, 2009; Heerwegh & Loosveldt, 2008; Holbrook, Green, & Krosnick, 2003; Tourangeau et al., 2000). This would produce responses of lower quality than that achieved with the “traditional” postal survey received in paper form, which is completed by interviewees in a calmer manner, with fewer distractions around them. The quality of response to the questionnaire will be defined by the number of non-responses, the amount of information given in response to open questions (in which respondents, in principle, are not limited in their answers), and the choice—in closed questions—of the “easiest” categories.

Theoretical Background and Expectations

One of the most used theoretical reference points concerning the quality of response to questionnaires is the Survey Satisficing by Krosnick (1991, 1999), based on the proposals of Tourangeau (1984) on the cognitive process that takes place when answering a questionnaire. From his point of view, answering a survey requires a significant cognitive effort, to the extent that, in each question, the interviewee must pass through four stages: (1) interpreting the meaning of each question, (2) searching and retrieving all the information saved in their memory, (3) integrating the information into an opinion or judgment, and (4) expressing this opinion appropriately (Krosnick, 1991; Tourangeau & Rasinski, 1988). The Survey Satisficing theory postulates that some people who have expressed a wish to participate in a survey change their mind as the interview progresses, “their motivation to work hard has evaporated” (Krosnick, 1991, p. 214) so that—instead of withdrawing their cooperation—they opt to continue answering questions but with a minimum effort. This means that these interviewees, instead of following the cognitive process described above, do not think as deeply about the meaning of the questions, and search their memory for the most “adequate” response for their interviewer.

A respondent lacking the necessary motivation to answer may change their attitude by the use of nonverbal communication techniques on the part of the interviewer in face-to-face interviews, or by the interviewer’s explanation of the advantages of cooperating in the research (Holbrook et al., 2003). On the contrary, a respondent could lose the motivation to participate in a survey when they feel the loss of intensity in a form of communication that lacks nonverbal communication, as it happens in telephone and self-administered surveys. In the specific case of telephone surveys, it must be taken into account that the interview process is very fast due to the need to avoid silences, which ends up generating bad feelings among those being interviewed, as they had agreed to collaborate because they believed that it would be a pleasant conversation (Green, Krosnick, & Holbrook, 2001; Holbrook et al., 2003). This hurried feeling, together with the possibility of doing other things while they are answering the questionnaire (multitasking), prevents them from carrying out the cognitive process necessary to respond to each question, which means that a larger number of questions do not accurately reflect what the respondent thinks.

In the self-administered questionnaire, there is a large social distance between the researcher and the person being interviewed that fosters the respondent’s honesty, thus reducing the number of “socially desirable” answers (Holbrook et al., 2003). The greater social distance characteristic of self-administered surveys varies depending on the medium used to answer: a physical questionnaire (paper) or an intangible questionnaire (electronic-virtual). The postal survey involves a closer relationship between the researcher and the respondents to the extent that the latter receive various materials (from the interviewer), such as a personalized letter of introduction, an envelope, a questionnaire, thank you cards, and so on. We could say that these materials are “borrowed” as, at least some, will have to be returned to the researcher once completed. Web surveys imply a greater social distance in that the respondent does not receive a “physical object.” This greater social distance reduces the social desirability and could encourage a sincere response (Green et al., 2001; Johnson, Fendrich, & Mackesy-Amiti, 2012; Kreuter, Presser, &Tourangeau, 2008), but, at the same time, it reduces the link between the respondent and the researcher. This link is reduced due to extrinsic characteristics of the Internet, such as multitasking (caused by having too many windows open) and the high speed of data processing. The greater social distance and the effects of the channel (the Internet) lead to the hypothesis that less effort is employed by those interviewed on the Internet to respond to a questionnaire, which would translate into a greater choice of first-option responses and a greater choice of affirmative answers (weak satisficing). These respondents would also show how a greater number of don’t know answers, lower differentiation in the use of scales, and a greater choice of easy answers (strong satisficing). In short, the satisficing theory has important implications for the issue at stake in this article, to the extent that the mode in which the questionnaire is administered may result in substantial differences in the quality of the responses.

Sample and Fieldwork

The data used in this article come from a research study (Moscoso et al., 2010) commissioned and funded by the Regional Government of Andalusia, and carried out by the Institute of Advanced Social Studies, a center belonging to the CSIC (Consejo Superior de Investigaciones Científicas—Spanish High Council for Scientific Research), an institution that is well known and highly regarded in Spain. The objective of the research was to understand the situation of people from Andalusia who were resident in other countries. The questionnaire contained 56 numbered questions with up to 116 variables.

More than half of the questionnaire (35 questions) was made up of forced choice questions presented vertically, the majority having five or two response choices, using round radio buttons in the web questionnaire (DeRouvray & Couper, 2002). There were also nine multiple-choice questions with square checkboxes in the web-based questionnaire, five check-all-that-apply questions, and the four remaining had a forced-choice format, with a yes/no choice for each item (Dillman et al., 2009; Smyth et al., 2005). There were no filter questions.

The questionnaire also had 12 open questions. In the open numerical questions, space was left blank in the postal questionnaire that was large enough to write several-digit numbers. In the web-based questionnaire, two of these questions were responded to with drop-boxes or a select list following Couper’s (2008) recommendations. The open questions that required a phrase were designed to collect up to three answers. Each answer had two lines available in the postal questionnaire. In the web-based questionnaire, a one-line text area (Couper, 2008; Dillman et al., 2009) was made available with no indication of space.

All these questions were included in a six-page questionnaire in whose design recommendations by experts in self-administered questionnaires were used (Dillman, 1978, 2008; Dillman et al., 2009; Mangione, 1995), following the unified mode construction principles (Dillman et al., 2009). No question—whether open or closed—would have the options “don’t know” or “no answer,” in a similar way as in previous pieces of research with similar aims (among others Fricker, Galesic, Tourangeau, & Yan, 2005). With the aim of avoiding effects caused by the different ways of viewing the questionnaire, it was decided that all the respondents would “view the same,” regardless of the mode chosen to respond. The web-based questionnaire allowed the possibility of moving from one question to another without having to answer the previous question and we did not use of the “forced answer” option (the interviewee could skip one or several questions).

The sample was selected randomly from people over the age of 18 with the right to vote in Andalusia who were resident abroad. The official updated reference framework for Spanish citizens living in other countries is the El Censo Electoral de Residentes en el Extranjero (The Electoral Census of Residents in Other Countries). From a universe of 144,007 emigrants from Andalusia who were resident in other countries in March 2008, 15,657 people were selected systematically, with the aim of achieving a sample size of at least 2,400 people. The members of the selected sample were contacted by post, and three channels were made available for them to answer the questionnaire: postal questionnaire (postage-paid), digital questionnaire, and CATI (computer-assisted telephone interview) paid for by the receiver.

The implementation procedures were analogous to those in similar research (Dillman et al., 2009, chap. 7; Ilieva, Baron, & Healey, 2002; Watson, Lissitz, & Rudner, 2006), although no monetary incentive was used nor was the mode of contact changed. Specifically, each person selected received an invitation envelope by post with a personalized letter of introduction, their ID number, a questionnaire, and a postage-paid envelope. The letter of introduction, addressed to a specific person and signed by the Vice-Councillor for Local Administration and Security of the Government of Andalusia (Viceconsejero de Gobernación del Gobierno de Andalucía), explained the objectives of the study, encouraged the addressees to cooperate, and referred to the confidentiality of their answers citing a Spanish law. It went on to explain that the questionnaire could be responded to by post (using the prepaid envelope), by Internet, indicating the questionnaire number (the link was shown in the letter)¹ or by telephone. Having received only 2,198 completed surveys, a second round of 3,698 questionnaires was sent out, finally achieving a total of 2,493 completed questionnaires.

Results

Response Rate

The 15,657 letters sent, of which 2,476 never reached their destination, generated 2,493 responses, which constitutes a cooperation rate (COOP2) of 18.9%. This is reduced to 15.9% when the response rate formula (RR6) is used (American Association for Public Opinion Research, 2011).

Having a sampling framework with data of the universe makes it possible to identify the differences between the theoretical sample and that actually obtained. A comparison between both distributions reveals a low number of European migrants and a larger number of residents in Latin America, with similar differences, but in the reverse (10%; that 63% of Andalusian migrants who live in Europe responded to 53% of the questionnaires; whereas out of the migrants living in Latin America, 30% responded to 41% of the questionnaires). There were few differences in the rest of the countries. The gender of the respondents was the same to the planned sample, while regarding age, a lower rate of participation was seen among the young and a greater participation among the old.

The analysis of the response obtained by each mode showed that the large majority of the questionnaires, 2,083, were received by post (83.6%), 359 by Internet (14.4%) and 51 people opted to reply by telephone (Diment & Garrett-Jones, 2007).

The effect of contacting by post, using a paper questionnaire inside the envelope, could explain the high number of people who replied using the paper questionnaire, but the low response of web questionnaires can also be explained by the questionnaire not being immediately accessible. The interviewee had to turn on the computer, connect to the Internet, type the web site, and enter the ID number to get identified. Many of the interviewees decide to postpone answering, and this produced “a break in the response process” (Dillman et al., 1994; Griffin et al., 2001). This “break in the response process” (Griffin et al., 2001) on many occasions results in failure to answer, as some respondents forget or lose interest.

Paying attention to the demographic profile of interviewee could explain this further (Miller, Kobayashi, Caldwell, Thurston, & Collett, 2002). The use of the mail survey was the main choice of those older than 65 years, people with lower education level, and those living in European countries. It is important to clarify that almost two of three had access to the Internet in their home and that half of the sample used it at least once a week. The web survey was mainly used by the under 44 age-group, with medium-high education level, and resident in Latin American countries (mainly for the reasons expressed above).

An analysis of the response quality revealed that 24 of the respondents had answered less than half of the questionnaires.² These questionnaires were eliminated, applying a much more stringent criterion than that used by Teclaw, Osatuke, Yanovsky, Moore, and Dyrenforth (2010), which included questionnaires with between only 51% and 80% completed responses and considered them as “incomplete.” Finally, the small number of people who chose the telephone survey prevented any meaningful conclusions, and therefore those questionnaires were eliminated from the analysis, which focused on the 2,419 people who answered the questionnaire by post or Internet, filling in at least half of the questionnaire.

Response Quality

Based on the Survey Satisficing proposals described in the first section, response quality will be operationally defined in terms of three criteria: (1) number of partial item non-responses to the questions in the questionnaire, differentiated between non-responses to open and closed questions; (2) length of answers to open questions; and (3) choice of first response options (primacy effect). It is important to take into account that our measures of quality do not cover quality in the sense of accuracy, honesty, or openness of responses.

We used significant differences in means and proportions to determine whether significant differences existed between each question in both the postal and the Internet modes of response.

Item Non-response: It is important to remember that the questions did not allow the respondent to choose a non-response category. This criterion records unanswered questions due to an absence of information, because the respondent did not want to answer or as a result of carelessness. Table 1 show that those who responded to the postal survey left more than 11 questions unanswered, nearly 5 more than in the web survey. This difference is basically due to the large number of non-responses to closed questions (5.4), which is 4 times lower in the web survey. In addition, better collaboration was observed in the web survey in the open questions with text, in line with what is found in most of the international literature on the subject (among others, Israel & Lamm, 2012; Millar & Dillman, 2012).

Table 1.

Number of Non-responses.

Number of Non-responses (Averages)
	Mail Survey	Web	t Value and p
Non-responses (total)	11.44	5.81	14.85**
Non-responses (closed questions)	5.40	1.29	14.38**
Non-responses (open numerical quest)	0.44	0.14	11.27**
Non-responses (open text questions)	5.60	4.39	6.81**
Percentage with respect to the total number of questions in the questionnaire***
Non-responses (total)	9.3%	4.9%	13.80**
Non-responses (closed questions)	5.2%	1.3%	14.02**
Non-responses (open numerical quest)	0.4%	0.1%	10.95**
Non-responses (open text questions)	4.5%	3.7%	6.54**

**Significant at .01.***Percentages represent an average of the item non-response rates for all respondents to each mode, for all items in the questionnaire.

Converting the number of non-responses into percentages with respect to the total number of answers to the questionnaire (Messer, Michelle, & Dillman, 2012; Millar & Dillman, 2012) provided more accurate information, and revealed that 9.3% of the postal survey questions were left unanswered, as opposed to 4.9% of those in the web survey. This makes it possible to verify that the percentage of non-response in the postal survey was shared almost equally between the closed and open questions; regardless of how easy it is to answer closed questions. In fact, the low number of non-responses to open questions was surprising, since the majority of studies have indicated that they tend to have poorer response (among others, Israel & Lamm, 2012).

The majority of the questionnaire was made up of single-response closed questions but, as mentioned earlier, there were also eight multiple-choice questions, four forced-choice format questions, and another four check-all-that-apply questions. Various pieces of research (among others, Dillman, Smyth, Christian, & Stern, 2003; Rasinski, Mingay, & Bradburn, 1994; Smyth, Dillman, Christian, & Mcbride, 2009; Smyth, Dillman, Christian, & O'Neill, 2010; Smyth et al., 2005; Stern, Smyth, & Mendez, 2012; Thomas & Klein, 2006) recommend the use of single-response closed questions, as they obtain the most answers and also prevent most respondents from choosing the first options provided in the answers to check-all-that-apply questions (primacy effect). The first column of Table 2, which shows the average number of non-responses collected for each question, differentiated by the mode used, shows a greater number of non-responses in the forced-choice questions.

Table 2.

Average Number of Non-responses to Multiple-Choice Questions.

	Average Number of Non-responses	Non-responses		Diff. Post
	Average Number of Non-responses	Post	Web	and Web (t Value and p)
Check-all-that apply
Q D7 (max 4***): health cover	0.064	0.075	0.000	5.61**
Q D8 (max 4): help/no help in paying for medicines	0.099	0.106	0.057	1.54
QG8 (max 6): with whom did you emigrate	0.033	0.035	0.023	0.41
Q G9 (max 4): who stayed	0.112	0.125	0.034	2.98**
Forced choice
Q B6 (10): home equipment	0.3260	0.373	0.051	8.46**
Q C8 (10): economic family problems	0.6238	0.716	0.082	9.76**
Q D2 (7): demand of services from institutions	0.6635	0.766	0.063	12.95**
Q D11 (4): Visual, hearing, mobility, or communication disability.	0.1215	0.140	0.012	6.54**

**Significant at .01. ***Number in brackets shows the maximum number of responses to each question.

Starting with the check-all-that-apply questions, the first question of Table 2 reveals that the question with the most possible answers (G8) was, logically enough, the one that obtained the best results. The non-response rates, as can be seen in the second and third column of Table 2, are always higher in the postal survey than in the web survey, although only two variables present a significant difference.

A study of the distribution of the non-responses answer unveils a greater selection of first response categories, except for the question “with whom did you emigrate” (G8) and “who stayed” (G9); as parents and siblings were placed as first and second choice, these were selected most often. This situation showed some differences with respect to the majority of the international literature on the subject, as it highlights that the primacy effect is more pronounced in these questions. In the other two questions, “health cover” (D7) and “help with the payment of medicines” (D8), all the response categories were identified, with the third and fifth being the most chosen.

Contrary to what is indicated by the majority of the literature, in this study it was observed that non-response rates increased considerably in the forced-choice questions, with the poorest response being that to the question about the demand for services provided by institutions, which had seven possible answers. As in the previous case, the web survey obtained more answers than the postal survey, showing significant differences in the four questions.

Regarding open numerical questions, it must be remembered that in two of these, drop-boxes were used (select list) in the web survey,³ leaving respondents to write whatever they wanted to in the postal survey. This generated very low rates of non-response (2.0% and 0.6% respectively) in the web survey, as opposed to 8% and 13.8% in the postal survey. The differences between both surveys reduced considerably when a (blank) space was left for the answer in the web survey, as was the case in the rest of the questions. The item non-response in the question concerning income reached 14.6% in the postal survey and 9% in the web survey; and the number of journeys to Andalusia was not answered by 3.5% and 1.7% of those interviewed, respectively. Considering only respondents born in Andalusia, 3.0% of those who used the postal survey did not answer the question concerning the year in which they left or when they started to live in that country, a percentage that went down to 0.6% in the web survey.

These differences between the modes of administering the survey could have been caused not so much by differences in the mode of response of the questionnaire, but rather by sociodemographic differences in the sample. With the aim of dispelling these doubts, a number of regressions were carried out to uncover whether differences in the non-response of both surveys remain even when controlling for the influence of sociodemographic variables (Table 3). After controlling for gender, age, educational level, being born in Andalusia, and country of residence (taking into account the continent), the number of unanswered questions was lower in the web survey (–0.113); with a greater decrease in closed questions.⁴ As shown in other studies (among others Messer et al., 2012), older people left more questions unanswered, and those born in Andalusia provided poorer responses to open questions.

Table 3.

Multivariate Analysis of Data Quality. Dependent Variable: Number of Unanswered Questions.

	Non-Response
	Total		Closed Questions		Open Questions
	Standardized b	t	Standardized b	t	Standardized b	t
Mode of survey (0 = postal)	–0.113**	–5.78	–0.126**	–6.26	–0.008	–0.49
Gender (0 = female)	–0.045**	–2.39	–0.028	–1.46	–0.076**	–4.66
Age	0.250**	10.53	0.249**	10.07	0.121**	5.81
Educational level	–0.076**	–3.30	–0.032	–1.32	–0.131**	–6.44
Born in Andalusia (0 = born outside)	0.119**	5.17	–0.020	–0.83	0.398**	19.73
Country of residence, sociopolitical grouping (reference: Central and South America)
European Union	0.067**	2.82	0.000	0.02	0.171**	8.39
Europe not EU	0.015**	0.73	–0.014	–0.64	0.065**	3.69
North America	0.003	0.17	–0.019	–0.92	0.047**	2.70
Others (Australia and North Africa)	0.072	0.019	0.019	0.96	0.044**	2.67
R ²	0.167		0.097		0.360
N	2,400		2,400		2,400

**Significant at .01.

A large number of research studies (among others, Couper & Bosnjak, 2010; Davidov & Depner, 2011; Denscombe, 2009; Fricker & Schonlau, 2002; Ilieva et al., 2002; Kwak & Radler, 2002; Reja et al., 2003) have indicated that web-based surveys, in addition to having a greater number of answers to open questions, obtain longer answers, which are also more detailed and more elaborate. To test this statement, the verbatim responses noted by each respondent were analyzed and subsequently checked for the length of answer by counting the number of characters. The longest answers were those referred to the questions about the respondent’s main problems or needs, followed by their occupation. Table 4 reflects the longest responses provided by the web survey for four of the five questions considered, although only three showed significant differences.

Table 4.

Length of answers to open questions.

Number of Characters of Responses to Open Questions
	Postal Survey	Web	t Value and p
Occupation (3 answers)	47.37	61.32	4.28**
Main problems or needs (3 answers max.)	48.51	68.53	5.39**
Reasons for staying in this country (three reasons)	43.09	49.54	2.14*
Main reasons for emigration (3)	31.30	21.14	4.47**
Reasons for choosing this country (3)	83.06	98.58	1.51

*Significant at .05. **Significant at .01.

Table 5.

Average Number of Characters per Word.

	Postal Survey	Web	t Value and p
Occupation (first answer)	6.98	7.69	4.90**
Main problem or need	4.93	6.18	7.45**
Main reason for staying in the country	5.63	5.93	3.09**
Main reason for emigrating	3.64	2.97	3.37**
Main reason for choosing that country	3.43	2.64	4.34**

**Significant at .01.

Table 6.

Primacy Effect in Questions With More Than Four Categories.

Choices of First Response Categories
	Postal Survey	Web	t Value and p
First category choices	5.46	6.10	4.62**
Second category choices	3.51	3.52	0.197
Choice of the first two categories	8.96	9.62	4.14**

**Significant at .01.

The length of the answers, measured by considering the number of characters in the words used to answer the questions, can also be used to consider the type of words used by each respondent. The information in Table 5 shows that those interviewed by way of the Internet used longer words, which we attribute to the use of a more efficient language, that is, a lower use of articles and prepositions.⁵ This is true in four of the five open questions used.

Primacy effect, choice of the first response category: To identify the presence of this effect the demographic questions and the four forced-choice dichotomy questions were discarded, leaving only the forced-choice questions and the five check-all-that-apply questions. The count of those choices with a value of 1 was slightly higher in the web survey, and this disappeared when second-category choices were included. When considering both together, the web survey showed a more pronounced primacy effect than the paper survey (see Table 6).

Conclusion and Discussion

Contrary to the initial proposals to the effect that the respondents interviewed on the web would only scan the text in the questions, the study showed the advantages of using web-based self-administered questionnaires. Although the primacy effect is higher in web surveys, these showed a low number of unanswered questions, more detailed answers to open questions, and longer answers to questions than those generated from paper questionnaires. The 11.4 unanswered questions in the postal survey (9.3% of the questionnaire) was reduced to less than 6 in the web-based survey, a decrease that is much greater when comparing the closed questions (5.40 and 1.29, respectively), as opposed to the tendency shown by other researchers (Denscombe, 2009). Another difference with respect to the findings by the majority of research studies is the fact that forced-choice questions showed greater non-response rates than check-all-that-apply questions. The differences in non-response persisted even when the influence of the demographic variables was eliminated. The five open questions with text showed longer answers from the web survey, around 63 characters more, the questions concerning “main needs or problems” showing the greatest difference. The respondents who answered the open questions on the web surveys used more efficient language, that is, they reduced the number of articles, focusing their answer on substantive words. In the open questions with numerical answers, the use of drop-box (select list) generated better response than the use of a blank space left in the paper questionnaire.

An analysis of other pieces of research that use self-administered web surveys together with surveys with an interviewer (face-to-face or telephone interviews) have shown a poorer completion rate of the former, attributable to factors such as multitasking (Heerwegh, 2009; Heerwegh & Loosveldt, 2008; Holbrook et al., 2003, p. 83), the reading of questions rapidly (scanning effect), and the greater social distance with the researcher, and different results from the web survey compared to the postal survey.

We consider that the way of inviting someone to participate in the survey could partly explain the excellent results from the web survey (Bradley, 1999; Dillman et al., 2009; Heerwegh, 2009; Heerwegh & Loosveldt, 2008; Holmberg et al., 2010; Muñoz Leiva et al., 2009). An electronic invitation sent by e-mail or an alert on the computer, received at a time when the user is using multiple applications, involves an interruption of the daily tasks that could result—on many occasions—in texts being scanned, and in inaccurate answers. On the contrary, an invitation to participate in a survey in the form of a letter received by ordinary post “forces” the respondent to turn on the computer (or another device) to fill in the questionnaire, which results in them paying more attention and having better motivation to answer the questionnaire. In this case, the person selected could choose to respond on paper or to do it by using the web. A detailed analysis of the information from the questionnaire shows that the respondents born in Spain responded mainly by mail, while the children of immigrants responded mainly online. Something similar happened with those who stated that they identified with their Andalusian roots. In light of this information, it seems that, at least in this study, those who were most motivated, most involved, chose to answer by mail.

These results confirm the findings of the majority of international research on the subject, although not many studies have been carried out on a general population. The need to count on populations with a high level of Internet access have led many researchers to use specific populations, such as company employees (Couper et al., 1999; Parker, 1992; Jones & Pitt, 2000), professional groups (Cobanoglu et al., 2001), students (Davidov & Depner, 2011; Denscombe, 2009), teachers (Converse et al., 2008), university graduates (Heerwegh, 2009; Heerwegh & Loosveldt, 2008; Holmberg & Lorenc, 2008; Kreuter et al., 2008; Kwak & Radler, 2002; Werner, 2005), and so on, with the result that—for many of them—caution is recommended when attempting to generalize the results to a wider population.

Finally, this study has several limitations. It is important to note that the population who received the letter with the questionnaire were born in Andalusia and, in many cases, have lived abroad for many years. Therefore, results may vary on the basis of different populations. The way of inviting participation, with a letter signed by the vice-councillor of the Government of Andalusia, may have increased the number of responses. The interviewees received only one letter. The response rate could significantly increase, if respondents were to be contacted several times, or if an incentive or a telephone reminder were provided.

Footnotes

Acknowledgment

With special thanks to Julian Thomas.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is a part of a wider research financed by the Ministerio de Economía y Competitividad (Government of Spain), ref. CSO2012-34257. This research note is a revised version of a paper presented at the 2012 Spanish Federation of Sociology (Federación Española de Sociología) meeting, Universidad Complutense, Madrid.

Notes

References

American Association for Public Opinion Research. (2011). Standard Definitions: Final dispositions of case codes and outcome rates for surveys. (7th ed.). Retrieved from http://www.aapor.org/Standard_Definitions2.htm

Boyer

K. K.

Olson

J. R.

Calatone

R. J.

Jackson

E. C.

(2002). Print versus electronic surveys: A comparison of two data collection methodologies. Journal of Operations Management, 20, 357–373. doi:10.1016/S0272-6963(02)00004-9

Bradley

(1999). Sampling for Internet surveys. An examination of respondent selection for Internet research. International Journal of Market Research, 41, 387–395.

Brecko

B. N.

Carstens

. (2006). Online data collection in SITES 2006: Papers versus web surveys – Do they provide comparable results? In Proceedings of the 2nd IEA International Research Conference. Washington, DC.

Cobanoglu

Warde

Moreo

P. J.

(2001). A comparison of mail, fax and web-based survey methods. International Journal of Market Research, 43, 441–452.

Converse

P. D.

Wolfe

E. W.

Huang

Oswald

F. L.

(2008). Response rates for mixed-mode surveys using mail and e-mail/Web. American Journal of Evaluation, 29 (1), 99–107.

Couper

M.P.

(2005). Technology trends in survey data collection. Social Science Computer Review, 23, 486–501. doi:10.1177/0894439305278972

Couper

M. P

. (2008). Designing effective web surveys, New York, NY: Cambridge University Press. doi:10.1017/CBO9780511499371

Couper

M. P.

(2011). The future of modes of data collection. Public Opinion Quarterly, 75, 889–908. doi:10.1093/poq/nfr046

10.

Couper

Blair

Triplett

(1999). A comparison of mail and e-mail for a survey in US. Statistical Agencies. Journal of Official Statistics, 15, 39–56.

11.

Couper

M. P.

Bosnjak

(2010). Internet surveys. In Marsden

P. V.

Wright

J. D.

(Eds.), Handbook of survey research (2nd ed., pp. 527–550). Bingley, England: Emerald Group Publishing Limited.

12.

Davidov

Depner

(2011). Testing for measurement equivalence of human values across online and paper and pencil surveys. Quality & Quantity, 45, 375–390. doi:10.1007/s11135-009-9297-9

13.

De Leeuw

E. D.

(2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21, 233–255.

14.

Denscombe

(2006). Web based questionnaires and the mode effect. An evaluation based on item non-response rates and data contents of near-identical questionnaires delivered in different modes. Social Science Computer Review, 24, 246–254. doi:10.1177/0894439305284522

15.

Denscombe

(2009). Item non-response rates: A comparison of online and paper questionnaires. International Journal of Social Research Methodology, 12, 281–291. doi:10.1080/13645570802054706

16.

DeRouvray

Couper

M. P.

(2002). Designing a strategy for capturing respondent uncertainty in web-based surveys. Social Science Computer Review, 20, 3–9.

17.

Dillman

D. A.

(1978). Mail and telephone surveys. New York, NY: Wiley.

18.

Dillman

D. A.

(2008). The logic and psychology of constructing questionnaires. In de Leeuw

E. D.

Hox

J. J.

Dillman

D. A.

(Eds.), International handbook of survey methodology (pp. 171–175). New York, NY: Lawrence Erlbaum Associates and European Association of Methodology.

19.

Dillman

D. A.

Kirsten

K. W.

Clark

J.R.

(1994). The influence of an invitation to answer by telephone on response to census questionnaires. Public Opinion Quarterly, 58, 557–68. doi: 10.1086/269447

20.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

Stern

M. J

. (2003). Multiple answer questions in self-administered surveys: The use of check-all-apply and forced-choice question formats. Retrieved from http://www.websm.org/uploadi/editor/File/Dillman_2002_Multiple_Answer_Questions.pdf.

21.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

(2009). Internet, mail, and mixed-mode surveys: The tailored design method (3rd ed.). New York, NY: John Wiley.

22.

Diment

Garrett-Jones

(2007). How demographic characteristics affect mode preference in a postal/web mixed mode survey of Australian researchers, Social Science Computer Review, 25, 410–417.

23.

Fowler

F. J.

Mangione

T. W.

(1990). Standardized survey interviewing: Minimizing interviewer-related error (Vol. 18). Thousand Oaks, CA: Sage.

24.

Fricker

Schonlau

(2002). Advantages and disadvantages of internet research surveys: Evidence tom the literature, Field Methods, 14, 347–367. doi:10.1177/152582202237725

25.

Fricker

Galesic

Tourangeau

Yan

(2005). An experimental comparison of web and telephone surveys. Public Opinion Quarterly, 69, 370–392. doi:10.1093/poq/nfi027

26.

Green

M. C.

Krosnick

J. A.

Holbrook

A. L

. (2001). The survey response process in telephone and face-to-face surveys: Differences in respondent satisficing and social desirability response bias. Retrieved from http://www.umich.edu/?nes/resources/techrpts/tech-abs/tech-ab62.htm.

27.

Griffin

D. H.

Fischer

D. P.

Morgan

M. T.

(2001). Testing an Internet response option for the American Community Survey. Paper presented at the annual meeting of the American Association for Public Opinion Research, Montreal, Canada.

28.

Groves

R. M.

(2012). Three eras of survey research. Public Opinion Quarterly, 75, 861–871. doi:10.1093/poq/nfr057

29.

Heerwegh

(2009). Mode differences between face-to-face and web surveys: An experimental investigation of data quality and social desirability effects. International Journal of Public Opinion Research, 21, 111–121. doi:10.1093/ijpor/edn054

30.

Heerwegh

Loosveldt

(2008). Face to face versus web surveying in a high Internet coverage population. Public Opinion Quarterly, 72, 836–846. doi:10.1093/poq/nfn045

31.

Holbrook

A. L.

Green

M. C.

Krosnick

J. A.

(2003). Telephone versus face-to-face interviewing of national probability samples with long questionnaires: Comparisons of respondent satisficing and social desirability response bias. Public Opinion Quarterly, 67, 79–125. doi:10.1086/346010

32.

Holmberg

Lorenc

(2008). Understanding the decision to participate in a survey and the choice of the response mode. Proceedings of Q2008 European Conference on Quality in Official Statistics.

33.

Holmberg

Lorenc

Werner

(2010). Contact strategies to improve participation via the Web in a mixed-mode mail and Web surveys. Journal of Official Statistics, 26, 465–480.

34.

Ilieva

Baron

Healey

N. M.

(2002). Online surveys in marketing research: Pros and cons. International Journal of Market Research, 44, 362–380.

35.

Israel

Lamm

A. J

. (2012). Item non-response in a client survey of the general public. Survey Practice, April . Retrieved from http://surveypractice.wordpress.com/2012/4

36.

Johnson

T. P.

Fendrich

M. E.

Mackesy-Amiti

M. E.

(2012). An evaluation of the validity of the Crowne-Marlowe need for approval scale. Quality and Quantity, 46, 1883–1896. doi:10.007/s11135-011-9563-5

37.

Jones

Pitt

(2000). Health surveys in the workplace. Occupational Medicine, 48, 556–558.

38.

Kreuter

Presser

Tourangeau

(2008). Social desirability bias in CATI, IVR, and web surveys: The effects of mode and question sensitivity, Public Opinion Quarterly, 72, 847–865. doi: 10.1093/poq/nfn063

39.

Krosnick

J. A.

(1991). Response strategies fr coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology, 5, 213–236. doi:10.1002/acp.2350050305

40.

Krosnick

J. A.

(1999). Survey research. Annual Review of Psychology, 50, 537–567. doi:10.1146/annurev.psych.50.1.537

41.

Kwak

Radler

(2002). A comparison between mail and web surveys: Response pattern, respondent profile and data quality. Journal of Official Statistics, 18, 257–273.

42.

Lozar Manfreda

Bosnjak

Berzelak

Haas

Vehovar

(2008). Web surveys versus other survey modes: A meta-analysis comparing response rates. International Journal of Market Research, 50, 79–104.

43.

Lozar Manfreda

Vehovar

(2002). Do mail and web surveys provide the same results? In Ferlijow

Mrvar

(Eds.), Development in social research methodology (pp. 149–169). Ljubljana, SI: Metodoloski Zvezki.

44.

Mangione

T. W.

(1995). Mail surveys: Improving the quality (Vol. 40). Thousand Oaks, CA: Sage.

45.

Messer

B. L.

Michelle

L. E.

Dillman

D. A

. (2012). Determinants of item nonresponse to web and mail respondents in three address-bases mixed-mode surveys of the general public, Survey Practice, 5. Retrieved from http://surveypractice.wordpress.com/2012/4.

46.

Millar

M. M.

Dillman

D. A

. (2012). Do Mail and Internet surveys produce different item nonresponse rates, Survey Practice, 5. Retrieved from http://surveypractice.wordpress.com/2012/4

47.

Miller

Kobayashi

Caldwell

Thurston

Collett

(2002). Citizen surveys on the web: General population surveys of community opinion. Social Science Computer Review, 20, 124–136. doi:10.1177/089443930202000203

48.

Moscoso

Moyano

Sesma

Ruiz

Domínguez

J. A.

Tiranti

Pasadas

Sotomayor

Trujillo

. (2010). Condiciones de Vida y Expectativas de Retorno de los Andaluces en el Exterior, IESA-CSIC / Consejería de Gobernación y Justicia. Junta de Andalucía. Retrieved from http://www.iesa.csic.es/publicaciones/detallarpublicacion/id/57

49.

Muñoz Leiva

Sánchez Fernández

Montoro Ríos

Ibañez Zapata

J. Á.

(2009). Improving the response rate and quality in web-based surveys through the personalization and frequency of reminder mailings. Quality and Quantity, 44, 1037–1052.

50.

Parker

(1992). Collecting data the e-mail way. Training and Development, 46, 52–54.

51.

Rasinski

K. A.

Mingay

Bradburn

N. M.

(1994). Do respondents really mark al that aplly on self-administered questions? Public Opinion Quarterly, 58, 400–408.

52.

Reja

Lozar Manfreda

Hlebec

Vehovar

(2003). Open-ended vs. close-ended questions in web questionnaires. Advances in Methodology and Statistics (Metodološki zvezki), 19, 159–177.

53.

Rossi

P. H.

Wright

J. D.

Anderson

A. B.

(1983). Sample surveys: History, current practice and future prospects. In Rossi

P. H.

Wright

J. D.

Anderson

A. B.

(Eds.), Handbook of survey research (pp. 1–20). New York, NY: Academic Press.

54.

Schaefer

Dillman

D. A.

(1998). Development of a standard e-mail methodology: Results of an experiment. Public Opinion Quarterly, 62, 378–397. doi:10.1086/297851

55.

Smyth

J. D.

Dillman

D. A.

Christian

L. M.

Mcbride

(2009). Open-ended questions in web surveys can increasing the size of answer boxes and providing extra verbal instructions improve response quality? Public Opinion Quarterly, 73, 325–337. doi:10.1093/poq/nfp029

56.

Smyth

J. D.

Dillman

D. A.

Christian

L. M.

O'Neill

A. C.

(2010). Using the Internet to survey small towns and communities: Limitations and possibilities in the early 21st century. American Behavioral Scientist, 53, 1423–1448. doi:10.1177/0002764210361695

57.

Smyth

J. D.

Dillman

D. A.

Christian

L. M.

Stern

M. J

. (2005). Comparing check-all and forced-choice question formats in web survey: The role of satisficing, depth of processing, and acquiescence in explaining differences (SESRC Technical Report 05–029). Pullman, WA. Retrieved from http://survey.sesrc.wsu.edu/dillman/papersweb/2005.html doi: 10.1093/poq/nfj007

58.

Stangl

. (2004). Mode effects on data quality: Benefits and drawbacks of mixed mode surveys. Paper presented at the 27th Centre for International Research on Economic Tendency Surveys Conference, Warsaw.

59.

Stern

M. J.

Smyth

J. D.

Mendez

(2012). The effects of item saliency and question design on measurement error in a self-administered survey. Field Methods, 24, 3–27. doi:10.1177/1525822X11419478

60.

Teclaw

Osatuke

Yanovsky

Moore

Dyrenforth

. (2010, October). Comparison of paper, web, and IVR responses in the veterans health administration survey. Survey Practice. Retrieved from http://www.surveypractice.org

61.

Thomas

R. K.

Klein

J. D.

(2006). Merely incidental?. Effects of response format on self-reported behavior. Journal of Official Statistics, 22, 221–244.

62.

Tourangeau

(1984). Cognitive sciences and survey methods. In Jabine

Straf

Tanur

Tourangeau

(Eds), Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73–100). Washingron, DC: National Academic Press.

63.

Tourangeau

Rasinski

(1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103, 299–314. doi:10.1037/0033-2909.103.3.299

64.

Tourangeau

Rips

L. J.

Rasinski

. (2000). The psychology of survey response. Cambridge, UK: Cambridge University Press. doi:10.1017/CBO9780511819322

65.

Watson

J. T.

Lissitz

R. W.

Rudner

L. M.

(2006). The influence of web-based questionnaire presentation variations on survey cooperation and perception of survey quality. Journal of Official Statistics, 22, 271–291.

66.

Werner

(2005). On the cost-efficiency of probability sampling based mail surveys with web response option. Linköping Studies in Statistics, no. 6. Doctoral Thesis. Linköping University.

67.

Wright

Marsden

P. V

. (2010). Survey research and social science: History, current practice and future prospects. In Marsden

P. V.

Wright

J. D.

(Eds.), Handbook of Survey Research (pp. 1–25). Bingley, UK: Emerald Group Publishing Limited. doi:10.1177/009430610503400458.