Cover Letter Reduces Response Rate

Abstract

French

The effect of whether to add or omit a cover letter at written surveys is examined by split-ballot experiments in surveys about victims of a crime in Thuringia (N = 10,184 distributed questionnaires, Austria (N = 804), Switzerland (N = 804) and Germany (N = 1180 + 9787). The survey form is the direct distribution of the questionnaires into the household boxes. The result is that a cover letter reduces the return rate by one tenth. In Austria and Switzerland is the reducing effect of the answering rate even greater. A cover letter discourages younger people from answering. This selection effect causes content distortions and for example, less deviant behavior is indicated. Adding a cover letter reduces the rate of victims by more than 3 percentage points. The omission of a cover letter is more effective in the presented survey form. It saves costs, increases the response rate and does not cause bias effects.

Keywords

survey split-ballot sample covering letter bias effect return rate

Introduction

In this article a split-ballot survey experiment is presented, which test the effect of an attached or non-attached cover letter to the return rate and response behavior in surveys. Mangione (1995: 63) notes that a cover letter is better, the shorter it is. In the ZUMA instructions “How to increase the response rate for postal surveys”, the length of the accompanying letter is also addressed: “In order to reduce the reading ‘costs’ of the asked person, the cover letter should not be longer than one page” (Porst, 2001: 3; similar to Raab–Steiner and Benesch, 2010: 49). The idea of the study presented below goes one step further. If there is a consensus that a cover letter is all the better, the shorter it is, would it not be optimal to use no cover letter? If the potential survey participant does not have to read any cover letter before the first question, the biggest read cost savings are realized. The research question is: How does the response rate of a written survey change if you do not attach a cover letter?

Cover letters are commonly used. The reasons for doing so are increasing the credibility of the survey, encouraging respondents to answer and providing a common frame for respondents. But reasons for the thesis that ‘no letter’ would be the ‘better cover letter’ could be also enumerated. A cover letter increases the costs for the respondents, rises the costs for the pollster and causes selection bias. Regarding higher costs for the respondents: A part of potential respondents is bored by reading the cover letter and throwing the examination sheet away, similar to a read–out advertisement. In the time someone reads the cover letter, he could better fill in the questionnaire. Analog to the rational argument from Dillman (1983); reading a cover letter increases the participation costs without increasing the individual benefit. b) As higher costs for the pollster are concerned: A cover letter must be printed on paper. In the case of an inkjet printer, paper and ink prices of 2.6 cents come together in addition to the questionnaire, with a 10,000 survey being 260 Euros. Cover letter produces a lot of more work. The folding and bagging of a survey with a cover letter takes considerably longer. According to own time measurements, 12 seconds per sheet are more needed. In the case of 10,000 questionnaires, this means 33 working hours or almost a workweek more. A cover letter is not good for the environment. It causes more paper and more transport expenditure in the survey implementation as well as more waste in the questionnaire storage. c) As for higher selection bias: “Do not use ‘filter’” points ZUMA to the questionnaire design in the above–mentioned instructions for backflow optimization (Porst, 2001: 4). But a cover letter is one of the strongest possible filters available. Persons who do not feel addressed by the cover letter leave the questionnaire unanswered. A cover letter influences directly the potential target group. The bad bias–error of a poll, that only those who have a higher interest in the subject respond, is formally induced by the cover letter. Bock/Schnapp report the state of research: “In the German (Schnell, 1997: 2008) as international (Groves and Couper, 1998) research on survey participation has been emphasized for some time that participation is primarily a function of the situation, but less firm beliefs for or against participation in surveys” (Bock and Schnapp, 2009: 253). Reading a cover letter is a situational hurdle that must be exceeded for spontaneous ad hoc participation.

With this abundance of arguments against a cover letter, it is surprising how naturally it is always attached to a survey (McGartland Rubio et al., 2003: 96; Häder, 2010: 242; Behnke et al., 2010: 241). And it is equally astonishing that hardly any work is known that once empirically verify the effect of removing a cover letter. There are hints for the effect of long cover letters. In the PISA student performance test, it is determined that the French test, which use 12% longer introduction text compared to English, reduce the results by 2% or 8 PISA points (Adams and Wu, 2002: 64ff.). The fact that there is no exploration of the effect of an omitted cover letter is all the more astonishing that there are a number of incentive experiments (overview work: Stähli and Joye, 2016; Meta-analysis: Singer, 2002; Singer and Ye, 2013). In the incentive investigations, the effects of rewards are considered. Incentives usually cost money for the survey performer. Previously, telephone cards (Arzheimer and Klein, 1998) were used but nowadays only cash (Stadtmüller and Porst, 2005; Mehlkop and Becker, 2007; Stadtmüller, 2009; Pforr et al., 2015). Omitting the cover letter reduces the cost of the survey. What effect does this have on survey participation? The aim of the submitted investigation is to close this research gap and to examine empirically: what is the effect of adding or not adding a cover letter?

Literature Review

It is noticeable that the letter in a written survey is so self-evident that it is often not mentioned at all. Neither in the book by Scholl (2013) “The Survey” nor in Kromrey et al. (2016) “Empirical Social Research” or Petersen (2014) “The Questionnaire in Social Research” is discussed the use of a cover letter. Where the cover letter is mentioned, it is declared unreflective to the obligatory canon of a written survey: “The written survey needs…An accompanying and introductory letter must orient the respondents…” (Atteslander, 2010: 158). The fact that most empirical social researchers consider a “personalized cover letter” (Hirschle, 2015: 137) to be normal is based on the orientation towards Dillmann (1983, 2008) and his total design method, which works with systematic exhortations of the interviewee to increase the return. Actually, a personalized cover letter with the “hint that the answers will be treated anonymously” (Mayer, 2009: 99) would be a contradiction in itself. Equally contradictory is the assurance of anonymity, while at the same time providing feedback on individual results (Kallus, 2010: 84). Anonymity should not be included as an empty formula in the covering letter, but only to assure “if this can really be guaranteed” (Raab-Steiner and Benesch, 2010: 50; similar to Schumann 2006: 129). So there is some dishonesty in the cover letter. It is similarly dishonest not to inform the respondent about the true subject of the survey (Mayer 2013: 100) in order not to induce any artificial effects. At both points – a bias by unbelieved anonymity and an interest–oriented participation – one recognizes that a cover letter triggers two effects that can lead to distortions. If, in the following, the omission of a cover letter is tested, one could therefore make the hypothesis that these distortion effects do not occur without a covering letter.

Because there is a long trend of decreasing participation rates in polls (Aust and Schroeder, 2009; Brick and Williams, 2013), it would be helpful to find out factors that increase the response rate. It is internationally known that: “The cover letter has been recognized as one of the factors that influence the response rate” (Sarantakos, 2013: 253). However, the willingness to experiment in the cover letter refers only to attempts to change the form of the salutation (Benini, 2000) or the color of the cover letter (Greer and Lohtia, 1994). A personalized salutation does not have an impact on the Dutch CAPI Labor Force Survey (Luiten, 2011). Personalization also proves to be irrelevant at Gendall (2005). Because personal speeches are closely linked to the cover letter, it could be hypothetically deduced that cover letters do not increase the return. At the Millennium Cohort Study in England, the transition in the covering letter to a very simple language leads to a minimal impact (Calderwood, 2014). It does not matter whether the signature on the letter is genuine or printed/scanned (Dodd and Markwiese, 1987; Reuband 2001). If the questionnaires have a number, shared sample experiments show that the nonresponse triggered by this is increased if these numbers are addressed in the cover letter (Bader et al., 2016).

Despite some experiments with changed letters, no international work can be found that checks the absence of the cover letter. In a comprehensive meta-analysis (Pit et al., 2014), which brings together a large number of field research attempts to increase the response rate, omitting a cover letter is not mentioned.

Method

In a representative written population survey of the citizens of Thuringia, 10,184 questionnaires are distributed in 88 villages and cities. Thuringia is a federal state in the middle of Germany. A small part is sent by post. The normal distribution is done by going from house to house. The questionnaires are inserted into the mailboxes of single–family houses and apartment blocks. They are including a return envelope “Porto pays receiver” with the recipient address of the University of Hamburg. For this method of written interrogation there are a number of examples (Schumann, 1990: 44ff.; Eilfort, 1994: 117; Schweer and Thies, 2000). It has similarities to the typical survey method in organizational research, where the questionnaires are distributed in company mailboxes (Brake, 2009: 393). The method used in Thuringia is not a drop–off survey, in which a questionnaire is handed to the interviewee after a personal interview (Hüfken, 2009). There is no eye contact interview. Personal contact is rare and is limited to random conversations at the front door, where the people are only asked to fill out the questionnaire.

In the distribution of questionnaires (as is customary) in Thuringia a cover letter is attached to most sheets (9184 from 10,184). In a smaller part, this letter is omitted for reasons of cost (1000 from 10,184). It turns out that the questionnaires without cover letters have a higher response rate than questionnaires with cover letters. This rather accidentally determined finding is the starting point for systematically examining the influence of the cover letter. Two improvements are made. The further empirical checks are carried out with intermingled samples. This means that every second distributed questionnaire is with a cover letter or without a cover letter. In addition, other countries are also involved to determine whether the possibly negative effect of the cover letter is based not only on geographical specificity.

Firstly 804 questionnaires intermixed 402 with and 402 without cover letter are distributed in Austria, more precisely in several parts of the city in Salzburg. The same survey test is carried out in Switzerland, in the case of Winterthur and Saint Gallen. As in Austria the survey design is the same split–ballot experiment.

In two further surveys with split samples, questionnaires are distributed with and without cover letters in Germany. However, for the first survey (N = 9787), the third–party variables are not controlled sufficiently, because no intermingled questionnaires are distributed. On the contrary, in West Germany, at the beginning, a cover letter is distributed and then, as the disadvantage of the cover letter is recognized, almost without cover letter. The questionnaires with cover letters are however almost thrown only in separate houses, because at the beginning a pure single house survey is thought of. The return from family houses is systematically higher, presumably because more members can bring the reply letter to the mailbox. In this respect, it is not possible to evaluate these results for the return rate. However, an evaluation is possible for the influence on the content result (see below).

In a second controlled distribution wave in Germany (N = 1180), intermeshing questionnaires are distributed in the classic split–ballot design. Every second questionnaire is with (N = 590) or without cover letter (N = 590). Because remaining arches of Thuringia are still left, it is distributed again in Thuringia. In this respect, the determined return rate is not representative for the whole of Germany.

Results

Response Rates with and without Cover Letter

In the four regions the split–ballot experiments by add or omit a cover letter lead to the following results:

The 9184 in Thuringia distributed/sent questionnaires with a cover letter achieve a response rate of 10.6%. The 1,000 sheets without a cover letter have a response rate of 11.6%. A cover letter reduces the response rate by one percentage point.

In Austria the distributed questionnaires with a cover letter produce a response rate of 5.5%. Without covering letter, the return rate is with 7.5% higher. That means that distribute the sheets without cover letter is 2 percentage points better than with cover letter. The disadvantage of the cover letter is even somewhat greater than in Thuringia. Speculatively, this could be due to the fact that the emphasis in the letterhead of the cover letter that it is a German survey is not an advantage in Austria.

In Switzerland again, a cover letter is a clear disadvantage. The return rate with cover letter is 8.0% and rises to 13.9% if you remove the cover letter. The return rate is 5.9 percentage points better without cover letter. The advantage of the questionnaires without cover letter is surprising. However, it can be proved here. If the regions of Saint Gallen and Winterthur are considered separately (then each N = 402), the answer rate with cover letter is 7.5% and 8.5% and increases in both parts to 13.9% if you omit the cover letter. Here, too, it could be a disadvantage of the cover letter that the population prefers to participate in surveys of their own country.

From the experience gained with the surveys in Austria and Switzerland, the arguments against cover letters from the introduction are added. The international bias argument: In cross–border surveys a cover letter may cause national distortions. Because surveys from their own country seem to be preferred, a cover letter from another country could reduce the return rate.

The result of the second investigation in Germany–Thuringia is very similar to the first result of Thuringia. The response rate with cover letter is 7.3% and rises to 8.5% without adding a covering letter. Distributing the survey sheet without cover letter leads to 1.2 percent points more return. 1.2 percentage points better response by omitting the cover letter is exactly the value, which is achieved a year before, if one withdraws the post–mailed questionnaires. The return flow rate in this survey is generally lower than in the first distribution waves around a year before. This is probably due to the weather. The cold partly snowy weather surely stops some people from the walk to the post–mailbox, in order to throw in the answer sheets. In addition, the surveying sheets look somewhat unsightly due to moisture storage.

Objections to the Results

What objections could be made to this empirical finding, especially to the big Thuringia survey? The strongest objection is that in Thuringia is not a sample of intermingled questionnaires with or without a cover letter distributed. Instead, the 1000 questionnaires without a cover letter are tossed across the country for a week in the first tour. The other questionnaires with cover letters are distributed in several distribution tours in the following months. Thus, possible other factors regarding the distribution are not adequately controlled. However, the regional spread is similar. And both the 1000 questionnaires without cover letter and the questionnaires with cover letters are distributed approximately half in single houses and block of flats. Learning from Thuringia, in the next other three split–ballot experiments, in Austria, Switzerland and Germany–Thuringia, the two versions with and without cover letter are intermingled.

One objection could be that in this written survey the method of distributing the questionnaires by hand deviates from the usual method of postal mailing (Atteslander and Kopp, 1999: 170). Of the total of 10,184 questionnaires among the population of Thuringia, 762 are sent by post (around 8%). These generate a response rate that corresponds roughly to the average of the questionnaires with cover letters. Third–party factors such as a dispatch to villages (slightly higher return rate) or in hard–to–reach outlying regions, the size of shipping envelopes and the quality of addresses (whether generated via telephone book or internet) play a greater role here. All postal mailed questionnaires are sent with cover letter. If you exclude the postal questionnaires, the distributed questionnaires with cover letters generate a response rate of 10.4% and are even 1.2 percentage points behind the 1,000 questionnaires without cover letter (11.6%).

A third possible objection could question the sample drawing. The 88 places in Thuringia are reached in four distribution waves, which run crosswise through Thuringia. A large spatial dispersion and a mixture of villages and cities are a matter of course. In the ADM–sample (i.e. the three–step random sample of the Association of German Market Research Institutes), which has been the selection basis of the leading market and research institutes since 1978, only 20–30 locations are included for the Allbus (General Population Survey of the Social Sciences) in Thuringia. In the presented Thuringia survey, the questionnaires are distributed in all houses structures. Thus the typical ADM sample error of a homogeneous clump (in one place only villas are questioned) is avoided (Althoff, 1993: 110; Häder, 2010: 156). In the Thuringia survey, each fieldwork phase is controlled and documented, thus fulfilling the conditions of a scientific survey (Atteslander and Kopp, 1999: 151). The 10,184 respondents from 1.053 million households (Thuringia Office for Statistics for 2014) indicate that in 1% of Thuringia households questionnaires are being thrown in. In election surveys only 1,000 of 82 million German citizens (= 0.001%) are surveyed. The representativeness of the sampling in the Thuringia survey should therefore be able to keep up with the ADM for Thuringia. Because there are demographic basic data for Thuringia, the representativeness can be secured by external validation. To assess only one variable: in the Thuringia survey, the proportion of responding women is 53%. In the official statistics it is quite similar to 51% (Thuringia Office for Statistics for 2014).

A fourth objection might be that the cover letter is not good. With a university letterhead, the assurance of anonymity, the mention of a contact person for inquiries, a (printed) handwritten signature and a short cover letter text, it satisfies the most important cover letter requirements formulated by Porst (2001: 2ff). The text¹ created by a group of three persons is indeed a matter of taste, but contains the usual appeals (for example, “your opinion is important to us”) and is “a motivating cover letter” (Stadtmüller, 2009: 169).

A fifth objection could relate to the overall low return rate. However, the return rate is not at all low. “Germany’s population is among the most survey–critical in the world, with very low response rates” (Pforr et al., 2015: 740). The response rate is determined by the distribution method, which works without de–anonymizing reminder and gross sample cleanup. Schweer and Thies (2000: 337) apply the same survey method and arrive to a response rate of 9.3%. Diekmann (2010: 516) find it realistic, with no reminding to get response rates of 5–20%. According to Walter et al. (2009: 50) return rates from 10–15% in written surveys are normal.

Discussion: Distortion Effects by Cover Letter

Cover Letter Leads to Content Response Distortions

In the introduction, it is assumed that a cover letter also influences the content of the survey. This aspect is empirically examined in the five presented surveys conducted with a two-divided sample – without and with cover letter. The question of the victim is asked, i.e. the respondents are asked if they have been victims of a criminal offense in the last 12 months.

As is probably the case in most researches, it is avoided to emphasize the survey background. Any mention of a topic in the cover letter would only encourage those who are interested in the subject (self-recruiting effect; Schnell et al., 2013: 351). In spite of a neutral text in the cover letter, an observer might assume that the survey is related to criminality by means of the letterhead of the cover letter “Institute for Criminological Social Research”. Naturally, victims of crime, i.e. persons who have recently been victims of a criminal offense, are more likely to participate in such surveys (Stadler, 1987: 78). It must be said that the Institute of Criminological Social Research is also located in the head of the questionnaire without cover letter. However, one might suspect that the official letterhead of the cover letter and the text trigger a larger influencing push.

The content distortion by a cover letter is obvious. In three of the four investigations, the proportion of the victim’s rate is considerable higher than in the other cover letter variant. However, the impact of the cover letter is reversed as expected. It reduces the victim's quota. Without cover letter the victimization rate is higher. If the respondents of all four surveys which answered the question of the victim were added together (i.e. N-weighted), the victim rate of 20.7% is calculated in the left column (questionnaires with cover letter) and 24.6% in the right column (without cover letter). That means, the cover letter reduces the victim quota by 3.9 percentage points.

Summaries of multiple studies are always subject to combination interaction effects. The variable sample size is a confounder in this evaluation (Bortz and Döring, 2016: 203). Here an N-weighting in table 2 leads to an over–concreting of the Thuringia results. The Thuringia survey however reflects the majority (3: 1) of the investigations. If the West German survey (table 3) would add, which lead to a similar result, the majority would be even 4: 1.

Table 1.

Return rate with/without Cover letter in Representative Written Population Surveys (Returned/Distributed)²

Region (distributed N)	With cover letter	Without cover letter
Thuringia (N = 10,184)	10.6% (969/9184)	11.6% (116/1000)
Austria (N = 804)	5.5% (22/402)	7.5% (30/402)
Switzerland (N = 804)	8.0% (32/402)	13.9% (56/402)
Germany-Thuringia (N = 1180)	7.3% (43/590)	8.5% (50/590)

Table 2.

Victim’s Rate for Distribution with/without Cover Letter for four Representative Population Surveys (Victims/Respondents)

	With cover letter	Without cover letter
Area	victim rate (N)	victim rate (N)
Thuringia	20.9% (199/951)	27.9% (31/111)
Austria	36.4% (8/22)	30.0% (9/30)
Switzerland	15.6% (5/32)	17.9% (10/56)
Germany-Thuringia	11.6% (5/43)	21.3% (10/47)
Total	20.7% (217/1048)	24.6% (60/244)

Table 3.

Victim’s Rate for Distribution with/without Cover Letter in West German Population Survey, N = 9787 Distributed Questionnaires (Victims/Respondents)

Area	With cover letter	Without cover letter
	victim rate (N)	victim rate (N)
West Germany	16.1% (10/62)	19.3% (162/839)

An explanation of why a cover letter reduces the victim's quota by more than 3 percentage points is difficult. Perhaps the potential fillers spot the crime questions more quickly on the questionnaire without cover letter (which begin after two political questions). Victims are more likely to be motivated. Or we could speculate that people who read more (even the cover letter) are less likely to be outside and are thus less often victims. In any case, the influence of the cover letter is not content–neutral, but distorts the result of the sample.

The influence without/with cover letter in the amount of 3.9 percentage points raises doubts whether the indication of confidence intervals for the assurance of a survey result is still up–to–date. These confidence intervals, which are sometimes supplemented in election surveys, cover only the random error. In fact, they should ensure that the true value is with 95% security within the confidence interval. What is the situation here? The random error calculated from the sample size (N=1048 respondents with cover letter) and a victim quota of 20.7% results in a confidence interval of +/– 2.5 percentage points:

+ / - 1.96 x {[0.207 x 0.793 / 1048]}^{0.5}

95% of all values should be in a range of 18.2% to 23.2%. If no cover letter had been attached, the value would be 24.6% victim quota. This value is outside the 95% confidence interval.

The 95% confidence of the confidence interval is a deceptive security. Without influence of the cover letter the same questionnaire would have been filled out in such a way that the value would then be outside of the confidence interval. In order to illustrate the empirical uncertainty of an investigation, the information about the cover letter or other fieldwork disturbances would be more important than the indication of fictitious–theoretical confidence intervals (see Schnell and Kreuter, 2000: 116; Behnke, 2007: 31; Gerber and Malhotra, 2008). Significance tests often only cover the smallest disturbance variable with the random error. Quatember (2005: 132) therefore speaks of “a veritable crisis of confidence in the significance tests”.

The second survey, Germany–Thuringia, is part of a Central Europe survey, in which further 9787 questionnaires are distributed in West Germany during these two years. To this respect, the generalization of Germany takes place, although only Thuringia is listed in table 3. The analysis of West Germany also reveals a more than 3 percentage point higher rate of victim without cover letter. This is similar to the average of table 2.

The victim quota of 16.1% with cover letter (10 victims / 62 respondents) for West Germany is 3.2 percentage points lower than the victim rate of 19.3% (162/839) without cover letter. In this respect, the N–weighted table 2 finding is confirmed, which shows 3.9 percentage points more victim quota if the cover letter is omitted.

However, the West German survey is not included in table 2, because without and with cover letter are not mixed together. Rather, it is distributed roughly distorted, because almost all the sheets with cover letters are thrown in single–family houses (this arises the return) and without the cover letter to a considerable extent in multi–family houses (with reduced return). In addition, the house distribution is done in nice weather, which increases the return, because the respondents then are more likely to walk to the post–mailbox.

If the West Germany survey was added in table 2, this would lead to a complete change in the result for an N weighting. It would come to the Simpson's paradox. Although both partial results show 3.9 and 3.2 percentage points higher victim rate without cover letter, an addition would lead to almost a reversal. This is due firstly to the fact that the victim rate in Thuringia and the three areas with cover letters is close to the victim quota of West Germany without cover letter. Secondly, in Thuringia, the majority of the cases are distributed with cover letters, while in West Germany the majority of them are distributed without cover letters. If both results are added, the value of the victim ratio of Thuringia without cover letter is drawn down very strongly, while the value of the victim ratio of Thuringia with cover letter hardly changes. In the summation, both had the victim rate of 20.5%. If one would take total German data instead of West German data, the Simpson's paradox would occur fully.

Cover Letter Leads to Selection Effects for Survey Participants

First, in the five examined samples, the effect without / with cover letter on gender effects is examined. The influence is minimal and is non–directional. In two of the cases under examination, the proportion of women taking part falls by a few percentage points, while it rises in three cases. Austria and Switzerland behave the other way round; the cover letter also has a different effect in Thuringia and West Germany. This would lead to the conclusion that omitting or adding a cover letter does not trigger systematic sexual distortion.

On the other hand, there is an obvious effect on the second most important demographic factor. In table 4 it is to be seen how large the proportion of the young participant cohort group is influenced by a cover letter:

Table 4.

Percentage of Younger Survey Participants in Distribution with/without cover Letter for Five Representative Population Surveys (16-34 Years)

Area	with cover letter	without cover letter
Thuringia	18.9%	29.8%
Austria	4.8%	6.7%
Switzerland	38.9%	31.3%
Germany-Thuringia	20.5%	20.5%
West Germany	13.3%	20.1%

It is almost unison that a cover letter reduces the proportion of younger participants. In Thuringia, 10.9 percentage points fewer young people are among the respondents. In Austria, 1.9 percentage points are less recent among responders. In Switzerland, their share is 7.6 percentage points lower. Only in Germany-Thuringia is no distinction detectable, which might has to do with the cold wind. In West Germany the cover letter reduces the proportion of 16 to 34-year-olds by 6.8 percentage points.

The proportion of 16 to 34 year–olds in the five areas varies with the distribution area. In Austria, the arches are only distributed in the newly built residential areas of Salzburg, which are close to the city center, which can hardly be afforded to younger residents. The very small proportion of young respondents in Austria, which is reduced only marginally from 6.7% to 4.8% by adding a cover letter, also explains why in Table 2 Austria is the only area in which the victim’s quota through a cover letter does not fall. In Switzerland the questionnaires are distributed in the student city of Saint Gallen, which explains the high percentage of younger respondents. The main result of the five areas of investigation is that a cover letter will lead to the fact that younger people will be deterred from the survey³.

Probably the computer games and internet dominated non–reading generation of the younger (Schnoor, 2009) is deterred by a cover letter, which means reading effort. They throw the questionnaire much faster in the trash with cover letter. Without cover letter they are animated by an immediate questionnaire filling.

This also shows how a cover letter probably works: it bores the potential participants and therefore has a deterrent effect on people who are accustomed to multi-media entertainment. Through a cover letter the proportion of the representative population is biased. It throws out the young and active, which stands for the future and for development. Correspondingly, the action values of the investigation are also reduced. An example of a cover letter changing the result is the rate of victim, which is reduced by 3.2 to 3.9 percentage points (see tables 2 + 3). The cover letter eliminates the younger and therefore more deviant survey participants from the survey population.

Conclusions

Although there are many arguments against a cover letter in written surveys (see introduction), there are so far no studies that systematically investigate the omission of a cover letter in surveys. Therefore in five surveys with shared samples the effect of a cover letter or no cover letter is tested.

In Thuringia (N=10,184 distributed questionnaires) the response rate is 1 percentage point higher if you eliminate the cover letter. This is confirmed through the second investigation in Germany–Thuringia (N=1180) one year later where the removing of the cover letter improves the response rate by 1.2 percentage points. The answer rate is in Austria (N=804) 2 percentage points higher and in Switzerland (N=804) 5.9 percentage points better if no cover letter is attached to the questionnaire. The return-reducing effect of a German cover letter is thus even greater in foreign countries (Austria and Switzerland).

Based on the realistic return rate of around 10%, a cover letter reduces the return by one tenth, abroad by even more (Austria by a quarter, Switzerland almost by half). The fact that a cover letter in Germany reduces the response rate by one–tenth is also evident in the study by Schweer and Thies (2000: 337), who work with the same survey distribution method but use a cover letter and achieve a reduced return rate of 9.3%. Although due to the small size of samples, we cannot expect association coefficient to be significant. We consider that the validity of this investigation depends from the fact that the victim quota and the age bias show the same tendency in almost all cases. The influence of omitted cover letter always goes in the same direction.

With such a negative influence of the cover letter it is perhaps no accident that Porst in his new 210-page workbook “questionnaire” mentions the cover letter only in a footnote (Porst, 2014: 37). If, as stated out in the introduction, the literature mainly determines situational factors for the survey participation, the cover letter proves to be situational disadvantage. To address the initial discussion about poll incentives: a cover letter is an anti–incentive. Omission of a cover letter is an incentive.

Cover letter results in a selective effect that could not be more unfavorable. A cover letter filters out the younger population, which is anyway underrepresented in surveys (telephone, face–to–face and in writing) due to many non–domestic activities (Kunz, 2010). Younger people shy away from reading. And when you have to start reading a letter before fill in the questionnaire, it is deterrent for those who would have to move disproportionately to participate. A cover letter worsens the existing age cohort distortion (Dillman et al., 2009: 11).

The content–impact of a cover letter is illustrated by the example of the victim quota. The rate of victim decreases by 3.2 to 3.9 percentage points when a cover letter is switched before the questionnaire. This is a considerable influence. If all the five surveys had been carried out without any cover letter, the rates of victim would be higher. A cover letter causes that surveys which measure crime indicate too low crime values.

Schnell and Kreuter (2000) work out that the difference in the victim quota between the multi–theme survey 1997 and the 1997 social science bus in the amount of several percentage points is reasoned in details of the undocumented fieldwork. A similar change effect would therefore also have the factor to carry out the survey with or without cover letter. The fact that a cover letter has such an influence on the result of the investigation is striking, because the cover letter is not even mentioned in most of the examination reports. Manipulating influences of the cover letter are thus a wide opened door.

For the future there is to be feared that the negative cover letter effect increase. Porst from the opinion–leading Center for Surveys Methods and Analyzes ZUMA (now part of GESIS) favors a kind of double cover letter for surveys (also Schnell et al., 2013: 353ff.; Scholl, 2013: 175). The title page on the multi–page questionnaire should also have a letter head similar to the cover letter and inform about the study. Häder (2010: 243) suggest that on the separate cover letter, the university institute is presented. On the title page of the questionnaire the commissioned opinion research institute is explained and on the back cover of the title page follows instruction and definition (Porst, 2014: 32–52). In the future, a potentially cross–interested survey person would have to overcome the hurdle of three sides of redundant reading. The first question would be on the fourth page. In their discussion of a good positioning of questions in the questionnaire, Faulbaum et al. (2009: 75ff.) have overlooked this aspect.

In the introduction, a number of arguments against cover letters in written surveys are given. Let us come back to three important ones: It is already indicated in the literature review that the respondents are treated dishonestly in the cover letter. The assured anonymity is often none, and the true subject of the survey is not openly communicated. In addition, the announced fill–in time is often a too low value. There are also quite direct manipulation approaches, e.g. respondents should be asked not to respond in a socially desirable way (Sarantakos, 2013: 251ff.). The latest return date is most often not the correct deadline. Not infrequently, even the survey organizer is disguised, in which the return is made to a neutral place (for example university), although the actual performer is a different one. Commercial advertising is camouflaged in the cover letter. Because a cover letter often fools the interviewees, it is morally questionable.

Cover letters sometimes encourage potential respondents to complain and this in turn can endanger the continuation of the survey. Complaints are a major disadvantage for studies with incentives. Incentives are considered to be one of the last possibilities of a return–quota guarantee, but which always have to be worked with cover letters. Complaints here are a big problem (Stähli and Joye, 2016, 435), which is possibly reported inadequately (Blom, 2016: 386). A large survey conducted by the Criminological Research Institute of Lower Saxony with 5 Euro incentives led to so many complaints that respondents even turn to the police (Wickert, 2014).

A cover letter destroys the respondent’s belief in the anonymity of the survey: the only way to secure anonymity in a survey is the door-to-door distribution method. In the case of a cover letter with names, some assume that the return questionnaire can somehow be put in relation to the name. Even the mention of a name in the cover letter confuses the interviewees: ‘Why was I selected?’ Only the distribution by walking from house to house, visibly for every neighbor, secures the data protection. Anonymity must be believed, not just assured (Kreutz and Titscher, 1974: 71). Contrary to all other forms of survey, by telephone, face–to–face, postal or online, only the mass distribution direct to the households secures anonymity and therefore guarantee honest answers. As the direct distribution of questionnaires is a cheap and functioning survey method, it is important to learn from the five presented studies. A cover letter should be omitted because it worsens the survey efficiency. These results do not apply to the postal mailing of questionnaires. There the effect of an omitted cover letter has yet to be found out through split-ballot experiments.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Adam

(eds) (2002) PISA 2000 Technical Report. Paris: OECD.

Althoff

(1993) Auswahlverfahren in der Markt–, Meinungs–, und empirischen Sozialforschung, (Band 19 in der Reihe Sozialwissenschaften). Pfaffenweiler: Centaurus.

Arzheimer

Klein

(1998) Die Wirkung materieller Incentives auf den Rücklauf einer schriftlichen Panelbefragung. ZA–Information 43, Nov 1998, 6–31.

Atteslander

Kopp

(1999) Befragung. In: Roth

Holling

Heidenreich

(eds) Sozialwissenschaftliche Methoden: Lehr– und Handbuch für Forschung und Praxis, 5. durchgesehene Auflage. München, Wien: Oldenbourg, 146–74.

Atteslander

(2010) Methoden der empirischen Sozialforschung, 13. Auflage. Berlin: Erich Schmidt Verlag.

Aust

Schröder

(2009) Sinkende Stichprobenausschöpfung in der Umfrageforschung – ein Bericht aus der Praxis. In: Weichbold

Bacher

Wolf

(eds) Umfrageforschung. Herausforderungen und Grenzen, (Sonderheft 9/2009 Österreichische Zeitschrift für Soziologie). Wiesbaden: Verlag für Sozialwissenschaften, 195–212.

Bader

Bauer

Kroher

(2016) Privacy concerns in responses to sensitive questions: a survey experiment on the influence of numeric codes on unit nonresponse, and misreporting. Methods, Data, Analyses: a Journal for Quantitative Methods and Survey Methodology 10: 47–72.

Behnke

(2007) Kausalprozesse und Identität. Über den Sinn von Signifikanztests und Konfidenzintervallen bei Vollerhebungen, (Arbeitskreis der DVPW “Empirische Methoden der Politikwissenschaft”). Beiträge zu empirischen Methoden der Politikwissenschaft. Teilgebiet: Statistik / Wissenschaftstheorie, Jg. 2, Nr. 3.

Behnke

Baur

Behnke

(2010) Empirische Methoden der Politikwissenschaft, 2. Aufl. Paderborn: Verlag Ferdinand Schöningh.

10.

Benini

(2000) Construction of Knowledge. Rome: Gnome.

11.

Blom

(2016) Survey Fieldwork. In: Wolf

Joye

Smith

Y–C

(eds) The SAGE Handbook of Survey Methodology. Los Angeles: SAGE Publications, 382–96.

12.

Bock

Schnapp

K–U

(2009) Kommentar zu Anna Schnauber und Gregor Daschmann: States oder Traits? Was beeinflusst die Teilnahmebereitschaft an telefonischen Interviews” (MDA 2008, 2, 97–123). Methoden – Daten – Analysen. Zeitschrift für empirische Sozialforschung 3(2), 249–59.

13.

Bortz

Döring

(2016) Forschungsmethoden und Evaluation, 5. überarb. Auflage. Berlin, Heidelberg: Springer-Verlag.

14.

Brake

(2009) Schriftliche Befragung, In: Kühl

Strodtholz

Taffertshofer

(eds) Handbuch Methoden der Organisationsforschung. Quantitative und Qualitative Methoden. Wiesbaden: Verlag für Sozialwissenschaften, 392–412.

15.

Brick

Williams

(2013) Explaining rising nonresponse rates in cross-sectional Surveys. The ANNALS of the American Academy of Political and Social Science 645, 36–59.

16.

Calderwood

(2014) Improving between-wave mailings on longitudinal surveys: a randomised experiment on the UK millennium cohort study. Survey Research Methods 8: 99–108.

17.

Diekmann

(2010) Empirische Sozialforschung. Grundlagen, Methoden, Anwendungen. 4/.20. Aufl. Reinbek bei Hamburg: Rowohlt.

18.

Dillman

(1983) Mail and other self-administrated questionnaires. In: Rossi

Wright

Anderson

(eds) Handbook of Survey Research. New York: Academic Press, 359–76.

19.

Dillmann

(2008) Total design method. In: Lavrakas

(eds) Encyclopedia of Survey Research Methods. Thousand Oaks, CA: SAGE Publications, 892–96.

20.

Dillmann

Phelps

Tortora

Swift

Kohrell

Berck

. (2009) Response rate and measurement differences in mixed-mode surveys using mail, telephone, Interactive Voice Response (IVR) and the Internet. Social Science Research 38(1): 1–18.

21.

Dodd

Markwiese

(1987) Survey response rate as a function of personized signature on cover letter. The Journal of Social Psychology 127: 97–98.

22.

Eilfort

(1994) Die Nichtwähler. Wahlenthaltung als Form des Wahlverhaltens. Paderborn, München, Wien, Zürich: Ferdinand Schöningh.

23.

Faulbaum

Prüfer

Rexroth

(2009) Was ist eine gute Frage? Die systematische Evaluation der Fragenqualität. Wiesbaden: Verlag für Sozialwissenschaften.

24.

Gendall

(2005) The effect of covering letter personalisation in mail surveys. International Journal of Market Research 47: 367–82.

25.

Greer

Lohtia

(1994) Effects of source and paper color on response rates in mail surveys. Industrial Marketing Management 23(1): 47–54.

26.

Gerber

Malhotra

(2008) Publication bias in empirical sociological research. Do arbitrary significance levels distort published results? Sociological Methods & Research, 37(1): 3–30.

27.

Groves

Couper

(1998) Nonresponse in Household Interview Surveys. New York: Wiley.

28.

Häder

(2010) Empirische Sozialforschung. Eine Einführung. 2. Aufl. Wiesbaden: Verlag für Sozialwissenschaften.

29.

Hirschle

(2015) Soziologische Methoden. Eine Einführung. Weinheim und Basel: Juventa.

30.

Hüfken

(2009) Teilnahme am ISSP. Bürger und Staat 2004 und Nationale Identität 2003. Drop-off – Erhebungen im Rahmen der Allgemeinen Bevölkerungsbefragung (ALLBUS) 2004. In: Weichbold

Bacher

Christof

(eds) Umfrageforschung. Herausforderungen und Grenzen (Sonderheft 9/2009 Österreichische Zeitschrift für Soziologie). Wiesbaden: Verlag für Sozialwissenschaften, 213–37.

31.

Kallus

(2010) Erstellung von Fragebögen. Wien: Facultas.

32.

Kreutz

Titscher

(1974) Die Konstruktion von Fragebögen. In: Koolwijk

Wieken–Mayer

. Techniken der empirischen Sozialforschung, 4. Band: Erhebungsmethoden: Die Befragung. München, Wien: Oldenbourg, 24–82.

33.

Kromrey

Roose

Strübing

(2016) Empirische Sozialforschung, 13. Aufl. Konstanz und München: UVK.

34.

Kunz

(2010) Mahnaktionen in postalischen Befragungen: empirische Befunde zu Auswirkungen auf den Rücklauf, das Antwortverhalten und die Stichprobenzusammensetzung. Methoden Daten Analysen 4: 127–55.

35.

Luiten

(2011) Personalisation in advance letters does not always increase response rates: demographic correlates in a large scale experiment. Survey Research Methods 5: 11–20.

36.

Mangione

(1995) Mail Surveys. Improving the quality. Thousand Oaks, London, New Delhi: SAGE Publications.

37.

Mayer

(2009) Interview und schriftliche Befragung. Grundlagen und Methoden empirischer Sozialforschung, 5. Aufl. München: Oldenbourg.

38.

Mayer

(2013) Interview und schriftliche Befragung. Grundlagen und Methoden empirischer Sozialforschung, 6. Aufl. München: Oldenbourg.

39.

McGartland Rubio

(2003) Objectifying Content Validity: Conducting a Content Validity Study in Social Work Research. Social Work Research 27, 94–104.

40.

Mehlkop

Becker

(2007) Zur Wirkung monetärer Anreize auf die Rücklaufquote in postalischen Befragungen zu kriminellen Handlungen. Theoretische Überlegungen und empirische Befunde eines Methodenexperiments. Methoden – Daten – Analysen. Zeitschrift für empirische Sozialforschung 1, 5–24.

41.

Petersen

(2014) Der Fragebogen in der Sozialforschung. Konstanz und München: UVK.

42.

Pit

Pyakurel

(2014) The Effectness of Recruitment Strategies on General Practitioner's Survey Response Rates – a Systematic Review. Medical Research Methodology 14: 76.

43.

Pforr

Blohm

Blom

Erdel

Felderer

Fräßdorf

. (2015) Are Incentive Effects on Response Rates and Nonresponse Bias in Large–Scale, Face–to–Face Surveys Generalizable to Germany? Evidence from Ten Experiments. Public Opinion Quarterly 79(3): 740–68.

44.

Porst

(2000) Praxis der Umfrageforschung, 2. Aufl. Stuttgart: Teubner.

45.

Porst

(2001) Wie man die Rücklaufquote bei postalischen Befragungen erhöht. Zuma How–to–Reihe. Mannheim: ZUMA.

46.

Porst

(2014) Fragebogen. Ein Arbeitsbuch. (Lehrbuch Studienskripten zur Soziologie). 4. Aufl. Wiesbaden: Verlag für Sozialwissenschaften.

47.

Quatember

(2005) Das Signifikanz–Relevanz–Problem beim statistischen Testen von Hypothesen. ZUMA Nachrichten Nr. 57, 29. Jg., November 2005, 128–50.

48.

Raab–Steiner

Benesch

(2010) Der Fragebogen. Von der Forschungsidee zur SPSS –Auswertung, 2. Aufl. Wien: Facultas wuv.

49.

Reuband

K–H

(2001) Möglichkeiten und Probleme des Einsatzes postalischer Befragungen. Kölner Zeitschrift für Soziologie und Sozialpsychologie 53(2): 307–33.

50.

Sarantakos

(2013) Social Research, 4th Ed. Basingstoke: Palgrave Macmillan.

51.

Schnell

(1997) Nonresponse in Bevölkerungsumfragen. Opladen: Leske Budrich.

52.

Schnell

Kreuter

(2000) Untersuchungen zur Ursache unterschiedlicher Ergebnisse sehr ähnlicher Viktimisierungssurveys. Kölner Zeitschrift für Soziologie und Sozialpsychologie 52(1), 96–117.

53.

Schnell

. (2013) Methoden der empirischen Sozialforschung. 10. Aufl. München, Wien: Oldenbourg.

54.

Schnoor

(Hg.) (2009) Jugendmedienforschung als empirisches Projekt. Wiesbaden: Verlag für Sozialwissenschaften.

55.

Scholl

(2013) Die Befragung, 3. Aufl. Konstanz und München: UVK.

56.

Schumann

(1990) Wahlverhalten und Persönlichkeit. Opladen: Westdeutscher Verlag.

57.

Schumann

(2006) Repräsentative Umfrage, 4. Aufl. München, Wien: Oldenbourg.

58.

Schweer

MKW

Thies

(2000) Kriminalität und Kriminalitätsfurcht. Eine empirische Untersuchung zum Kriminalitätserleben in der Bevölkerung. Kriminalistik 54(5): 336–42.

59.

Singer

(2013) The Use and Effects of Incentives in Surveys. ANNALS AAPSS 645, 112–41.

60.

Singer

(2002) The Use of Incentives to Reduce Nonresponse in Household Surveys. In: Groves

. Little RJA: Survey Nonresponse . New York: Wiley, 163–78.

61.

Stadler

(1987) Kriminalität im Kanton Uri. Eine Opferbefragung. Diss. Univ. Zürich, Entlebuch: Huber.

62.

Stadtmüller

Porst

(2005) Zum Einsatz von Incentives bei postalischen Befragungen. ZUMA How–to–Reihe Nr. 14, Mannheim.

63.

Stadtmüller

(2009) Rücklauf gut, alles gut? Zu erwünschten und unerwünschten Effekten monetärer Anreize bei postalischen Befragungen. Methoden – Daten – Analysen. Zeitschrift für empirische Sozialforschung, 3(2), 167–85.

64.

Stähli

Joye

(2016) Incentives as a possible measure to increase response rates. In: Wolf

Joye

Smith

Y-C

(eds) The SAGE Handbook of Survey Methodology, Los Angeles: SAGE Publications, 425–40.

65.

Walter

. (2009) Einführung in kriminologisch–empirisches Denken und Arbeiten, Stuttgart u.a.: Richard Boorberg Verlag.

66.

Wickert

(2014) Umfrage des KFN zu Kriminalitätsfurcht schürt Kriminalitätsfurcht. In: Blog Criminologia 15.02.2014.