Web and Mail Surveys

Abstract

This study aims to compare two widely used methods of original data collection in nonprofit research: web and mail surveys. We employ an experimental design to assign a web-based survey and a mail survey to nonprofit professionals working in human services organizations in New Jersey. We then compare responses generated from the two survey methods in terms of response rates and data quality. Our study finds that the mail survey achieved a significantly higher response rate than the web survey, and data obtained from the mail survey produced higher internal consistency than that obtained from the web survey. There was no difference between methods, however, in respondent characteristics, the completeness of the survey, and the percentage of missing items. Taken together, the findings suggest that a mail survey, although more costly, may have response-rate and data-quality advantages over a web survey as a methodology for gathering data from nonprofit organizations.

Keywords

mail and web surveys experimental comparison response rate data quality

Introduction

Survey methods are a widely used approach to collecting data on nonprofit organizations and the people who run them. A review of the most recent (2010 and 2011) issues of Nonprofit and Voluntary Sector Quarterly (NVSQ) demonstrates the popularity of survey methods in the field: of the 50 empirical studies (as opposed to review essays or theory pieces), more than half (30) used survey data of some kind. Among the various survey modes, web surveys (involving an emailed request to complete an online questionnaire) and mail surveys (involving self-administered paper questionnaires) appear to be the most frequently used modes to study nonprofits.

Because survey mode is an important factor in the success of a study, in terms of response rate and data quality, it is important to better understand the strengths and weaknesses of different modes (Groves et al., 2009). However, very few empirical studies have examined survey methods and related data quality issues in the context of nonprofit research.

The purpose of this study is therefore to compare two of the most common modes of surveying nonprofits, web and mail surveys, and to explore strategies that can help improve survey quality in the field. Following Cole’s (2005) study comparing mail and web-based survey methods in tourism research, this study is trying to address the following research questions: Does the web survey mode generate the same response rate as the traditional mail survey mode when gathering data on nonprofits? Do respondents to the web survey have the same organizational or/and personal characteristics as those who respond to the mail survey? Does the web survey have the same data quality as the regular mail survey, particularly in terms of item nonresponse? Do scales in the web survey have the same levels of internal consistency as those in the regular mail survey? And finally, do responses generated in the web survey have the same mean values on study variables as those generated in the mail survey?

Background

Web surveys have a number of advantages over traditional mail surveys. Financially, web surveys are usually associated with relatively lower logistical and personnel costs (Couper, 2000; Dolnicar, Laesser, & Matus, 2009; Fricker & Schonlau, 2002; Jones & Pitt, 1999; Kwak & Radler, 2002; Naus, Philipp, & Samsi, 2009). Previous studies also suggest that web surveys can be administered faster than conventional mail surveys, allowing more rapid analysis of the responses (Couper, 2000; Dolnicar et al., 2009; Jones & Pitt, 1999). A significant portion of the research on various survey methods has mainly focused on comparing web surveys and traditional mail surveys in three broad areas of inquiry: response rate, respondent profile, and data quality (Kwak & Radler, 2002).

In terms of survey response rate, which is critical to the representativeness and generalizability of survey findings, most studies find that paper-based methods tend to obtain higher response rates than their electronic equivalents (Groves et al., 2009). Fricker and Schonlau (2002) examined literature generated from two social sciences databases and found that web surveys generally do not achieve response rates equal to those of mail surveys. Manfreda, Bosnjak, Berzelak, Hass, and Vehovar (2008) review of 26 research articles comparing web survey and mail survey response rates identified from multiple sources revealed a similar pattern. A few studies have shown that web surveys can achieve comparable response rates to mail surveys (Kaplowitz, Hadlock, & Levine, 2004; McCabe, Boyd, Young, Crawford, and Pope, 2005; Weible & Wallace, 1998 in Manfreda et al., 2008). A review of tourism surveys conducted by Dolnicar and his colleagues (2009) found that web surveys demonstrated higher return rates than mail surveys. Six of the 26 papers reviewed by Manfreda et al. (2008) also reported higher response rates for web surveys. Hager, Wilson, Pollak, and Rooney (2003) surveyed a sample of nonprofit organizational mail survey studies published in an academic journal and found that, like other types of organizational surveys, surveys of nonprofit organizations have low response rates. There have been no published studies to date comparing response rates of different survey modes in nonprofit research.

Previous studies have attempted to examine demographic profiles of survey respondents to see whether web respondents have comparable characteristics to those of mail respondents. Friedman, Clusen, and Hartzell (2004) compared web and mail respondents to a health care survey and found that web survey respondents tend to be younger, White, male, active duty beneficiaries, and to live outside of the United States. Kwak and Radler’s (2002) survey to university students revealed that the web survey involved fewer female and younger respondents. Bech and Kristensen’s (2009) survey of an older population in Denmark revealed that female are less likely to respond to web surveys than males. However, Schillewaert and Meulemeester (2005) concluded that the difference between web and mail survey modes in terms of sociodemographics was only minor.

Survey researchers also have compared the quality of data generated from different survey modes, with the most frequently employed criteria of data quality being the number of incomplete or partially completed surveys and the percentage of missing items (item nonresponse). Web surveys have been found to have a lower rate of item nonresponse (Schaefer & Dillman, 1998). Cole (2005) reviewed some previous survey studies which consistently reported that web survey responses had fewer missing items than paper surveys. Other studies (Bech & Kristensen, 2009; Friedman et al., 2004) found web surveys not only have significantly fewer item nonresponses but also fewer “don’t know” answers than mail surveys. Some researchers (McCabe et al., 2005) reported that their study did not find statistical difference in data quality, as measured by the number of partially completed questionnaires and the overall rate of item nonresponse between the two survey modes.

In addition to the three areas discussed above, some studies have compared the substantive answers to survey questions obtained from web and mail surveys. However, findings from these studies are mixed. Although some have revealed that web surveys produce similar responses to those obtained from mail surveys, others have shown evidence of significant differences in responses generated from the two different survey modes. For example, Naus et al. (2009) did not find significant differences for variables and scales between web and mail surveys to female university students. Likewise, a survey of third- and fourth-grade students that asked about alcohol and tobacco use (McCabe et al., 2005) revealed very few statistical differences in the distributions of variables between the two survey modes. Results from a survey to travel retailers (Cole, 2005) showed that web and mail modes produced different reliability scores for multi-item scales, with web mode having higher reliability on some scales but lower reliability on others.

In sum, although web surveys offer cost and speed advantages over traditional mail surveys, findings from methodological research in various settings seem to suggest that web surveys may suffer from somewhat lower response rates. However, web surveys appear to have an advantage over mail surveys in terms of item nonresponse. And with respect to substantive differences in the distribution of study variables, the evidence comparing the two modes is mixed. But most of this methodological research has focused on surveys of individuals or households, rather than organizations, and none of the studies to date has explicitly looked at mode differences in the context of surveying nonprofit organizations. Thus, given the importance of surveys in empirical nonprofit research, combined with evidence of some important mode differences in other types of surveys, we believe it is important to examine these issues in the nonprofit context.

Method

Sample and Data

To compare web and mail survey modes in the context of nonprofit research, this study uses data from a study of nonprofit executive directors drawn from a sampling frame of New Jersey nonprofits obtained from GuideStar Premium. Human services and community improvement organizations were selected because of their visibility among all types of nonprofits, their traditional and relatively heavy reliance on government as a major funding source, and that “their provision of similar goods and services are primarily consumed by local residents” (Grønbjerg, 1993). These organizations include eight types: crime and legal related; employment; food, agriculture and nutrition; housing and shelter; public safety, disaster preparedness and relief; youth development; multipurpose human services; and community improvement and capacity building. Having obtained a sampling frame from GuideStar Premium, we verified the address, email address, and other contact information by checking other sources, such as the organization’s official website.

We downloaded from GuideStar Premium information on all organizations that meet the following four criteria: (a) the organization is located in the State of New Jersey; (b) the organization’s NTEE code is one of the eight groups identified as human services and community improvement organizations; (c) IRS subsection of the organization is defined as 501(c)3 Public Charity; and (d) the organization’s annual income (for the most recent record year available in the GuideStar Premium database) was at least US$200,000. In addition, we cleaned the resulting list by deleting organizations whose NTEE codes do not fall into any of the eight groups, removing duplicates, and reassigning organizations that were misclassified. Because the data were being collected primarily for a study looking at revenue diversification, the sample was further trimmed of nonprofits that had mostly fee-driven income, such as labor unions, day care centers, and senior/retirement housing organizations. Finally, we removed organizations that failed to report with the IRS since 2006, because of concerns that they may no long be in existence or that their more recent revenues had dropped below IRS reporting requirements (US$25,000). These various procedures yielded a final list sample of 1,115 organizations.

We were able to obtain the name and email address for 320 executive directors or chief executive officers, out of the total cleaned sample of 1,115 organizations. We randomly split these 320 organizations into two experimental groups: one assigned to the web survey (n = 160) and the other to the mail survey (n = 160). The remaining 795 organizations, for which no email address was available, were sent the mail survey as part of the larger study. However, for purposes of the methodological experiment reported here, the analysis focuses only on those 320 organizations with complete email contact information that could thus be randomly assigned to the two survey modes (Groves et al., 2009). Table 1 provides the distributions of organizations by NTEE category in the large sample as well as in this study sample (experimental sample).

Table 1.

Study Sample by NTEE Category

NTEE category	Large sample	Study sample
I. Crime & legal-related	46 (4.1%)	14 (4.4%)
J. Employment	48 (4.3%)	6 (1.9%)
K. Food, agriculture & nutrition	33 (3%)	10 (3.1%)
L. Housing & shelter	186 (16.6%)	31 (9.7%)
M. Public safety, disaster preparedness & relief	23 (2.1%)	4 (1.3%)
O. Youth development	95 (8.5%)	48 (15%)
P. Human services	527 (47.3%)	171 (53.3%)
S. Community improvement & capacity building	157 (14.1%)	36 (11.3%)
Total	1,115	320

Survey Instrument

The survey instrument addressed nonprofit financial strategies and organizational performance. The questionnaire was finalized after being piloted with a number of nonprofit professionals. The survey instrument comprised 35 mostly closed-ended questions. There are five sections in the questionnaire: Section 1 asked about basic organizational characteristics, including service field, geographic location, organizational size, organizational age, board size, number of board meetings each year, and number of fund-raising professionals. Section 2 asked about the organization’s funding environment, such as annual budget and revenue sources, and also the perceived difficulty in obtaining revenues. Section 3 asked respondents to share their fund-raising experiences and strategies. Section 4 asked respondents to self-report their organizational performance. And Section 5 asked demographic questions, such as gender, age, job title, and tenure with their organization. Pretesting suggested that the questionnaire took approximately 15 minutes to complete. Respondents were allowed to consult with their colleagues for assistance in answering survey questions.

The web and mail versions of the questionnaire were identical in terms of question content. We designed, programmed, and posted the web version of the questionnaire in SurveyMonkey (SurveyMonkey.com), then downloaded the questionnaire as a PDF for use in the mail survey. Thus, the mail survey and web survey had essentially the same format and were highly comparable visually. We did not make survey questions mandatory. We disabled “require answer to question” feature when programming survey questions, so web survey respondents were able to skip questions just like when they complete paper surveys. We also set up the web survey in a way that allowed respondents to move backward to a previous page. The respondents were therefore able to go back to review or edit their answers any time in the survey or after the survey was complete.

Administration of Mail and Web Surveys

Berry, Arons, Bass, Carter, and Portney (2003) suggested that nonprofit survey researchers make some logistical effort such as adopting multiple mailings to obtain good response rates. We used a modified version of Dillman, Smyth, and Christian (2009) method that involved two mailings of the complete questionnaire (plus a cover letter), with one reminder post card in between. First, we sent a paper questionnaire, along with a personalized cover letter (including the individual’s name) and a business reply envelope. Two weeks after the first survey mailing, we sent a reminder postcard to all nonrespondents. After another 2 weeks, we sent a second letter, a replacement questionnaire, and another business reply envelope enclosed to all organizations who had not responded by that time.

For the web survey group, we also made three contact attempts. First, we sent an initial email invitation with language very similar to that of the cover letter used in the mail survey, with only minor modifications in wording to fit the web survey situation. As with the cover letter in the mail survey, we personalized these email messages with the name of the individual. Respondents were instructed to click on an embedded URL address link provided in the email text that would direct them to the survey webpage hosted on SurveyMonkey. Six days after the initial invitation email, we sent a follow-up email to all nonrespondents as well as those who only partially completed the survey. After another 6 days, we sent a final follow-up email to all remaining nonrespondents. Again, the wording of these messages was essentially the same as the wording in the follow-up cover letters used in the mail survey.

Results

Using chi square tests and two-sample t-tests (for categorical and quantitative outcomes, respectively), we compared the results of the web survey and mail survey in terms of response rates, organization and respondent characteristics, item nonresponse, internal consistency of scales, and mean values for the substantive study variables.

Response Rates

The response rate is an important factor in assessing the value of organizational survey research findings (Baruch & Holtom, 2008). The overall response rate for the survey is 53.4% (171/320). A number of reasons might explain why the survey achieved such a respectable return. First, nonprofit organizations in New Jersey might identify strongly with this study which was conducted by researchers affiliated Rutgers, the State University of New Jersey. Second, we personalized survey invitations (in our experimental study). Berry and his colleagues (2003) suggested that letters and emails with a specific name are more likely to be perceived as important which worked for the response rate. Third, the survey was administered at a time when the American economy was experiencing the most severe recession since the Great Depression and nonprofits were financially stressed. Nonprofit executives were highly likely to be attracted to the purpose of the study—examining how certain revenue patterns could hopefully help nonprofits survive the economic downturn. Finally, we designed the survey to make it as respondent friendly as possible so it would be easier for respondents to answer the questions and complete the survey.

Consistent with previous studies in other settings (Couper & Miller, 2008), this study showed (in Table 2) that the mail survey achieved a much higher response rate (62.5%) than did the web survey (44.4%). This difference is statistically significant (p = .001) and substantively important as well, in terms of both the larger number of observations available for analysis and the more respectable overall rate. One of the major concerns of low response rates in surveys is that a lower response rate might harm the representativeness of the data due to nonresponse bias. To assess whether the generalizability of the collected data in the web survey is lower than that in the mail survey, we compared the characteristics of the responding organizations and those of survey respondents.

Table 2.

Comparison of Response Rates

	Web		Mail
	N	%	n	%	Chi sq	p
					10.56	.001
Responded	71	44.4	100	62.5
Not responded	89	55.6	60	37.5
Total	160	100	160	100

Organization and Respondent Characteristics

It is important and necessary to compare whether different survey modes generate responses from different types of respondents. Thus Table 3 shows the comparison of the web and mail survey respondents in terms of both organizational and individual respondent characteristics. The two groups of responding organizations are quite similar in terms of their budgetary size and service field. And the individuals are similar as well, in terms of gender, age, and tenure with their organizations. Of course, the groups were formed by random assignment and thus there should be no initial differences in such characteristics. But these results suggest that, although the response rate to the web survey was lower, there does not appear to be much of a systematic difference in the types of organizations and individuals who responded in each group, at least in terms of observed characteristics.

Table 3.

Comparison of Responding Organizations and Respondent Demographics

	Web	Mail
Variables	Mean (SD)	Mean (SD)	t	p
Organizational budget (in millions)	3.9 (6.2)	3.9 (6.8)	−0.001	.999

	n (%)	n (%)	Chi sq	p

Organizational service field			2.13	.953
HS	40 (56.3)	47 (47.0)
Youth	9 (12.7)	15 (15.0)
Housing	7 (9.9)	11 (11.0)
Community	7 (9.9)	13 (13.0)
All other	8 (11.2)	14 (14.0)

	Mean (SD)	Mean (SD)	t	p

Respondent’s tenure with current organization (years)	10.7 (8.8)	11.1 (4.9)	0.283	.777

	n (%)	n (%)	Chi sq	p

Respondent’s gender			0.004	.948
Male	35 (50.0)	48 (49.0)
Respondent’s age			2.54	.638
25-34	4 (5.6)	4 (4.1)
35-54	34 (47.9)	48 (49.0)
55 and above	33 (46.4)	46 (46.9)

Item Nonresponse

Questionnaire completeness is an important indicator of data quality that can be measured as (a) the percentage of fully (as opposed to partially) complete questionnaires, or (b) the average percentage of missing items (Dolnicar et al., 2009; Fricker & Schonlau, 2002; McCabe et al., 2005). The questionnaire in this study had a total of 94 response items, so a fully complete questionnaire has recorded response for all 94 items (including “not applicable (N/A)” as a recorded response). As Table 4 shows, only 21% of web survey responses and 20% of mail survey responses had fully complete questionnaires, but this difference between the groups is not statistically significant. Compared to high response rates (63% for mail survey and 44% for web survey); the percentages of fully complete responses are strikingly low. It needs to be noted that we counted as valid responses those that answered the survey to the last question. We however did not count as valid responses those that returned the paper survey but did not answer any questions or clicked on the email links but did not answer any questions. We also did not count as valid responses those that stopped abruptly after answering only the first questions at the beginning of the survey. One possible explanation of the low “fully complete response rate” is that our survey included 32 items that have “not applicable (N/A)” option. Some respondents might have not answered these questions if they considered them to be not applicable, and their nonresponse to these questions significantly reduced the “fully complete response rate.”

Table 4.

Comparison of Survey Completion

	Web	Mail
	n (%)	n (%)	Chi sq	p
Completion			0.032	.857
Fully completed	15 (21)	20 (20)
Partially completed	56 (79)	80 (80)

Similarly, there is no significant difference between the two groups in terms of the percentage of missing items, although the mean percent of missing items for mail survey respondents is smaller than that of web survey respondents (see Table 5). Thus, the two survey modes are quite similar in terms of the level of item nonresponse.

Table 5.

Comparison of Missing Items

% of missing items	Web	Mail	t	p
Mean	12 (7.4)	11.9 (5.33)	0.51	.613
Quantiles
Min	0	0
0.25	6	6
Mdn	11	11
0.75	16	16
Max	33	39

Internal Consistency of Scales

Internal consistency is an important measure of reliability that assesses the consistency of multiple items that are designed to measure the same construct (Remler & Van Ryzin, 2010; Trochim, 2001). A frequently used measure of internal consistency reliability is Cronbach’s alpha. A larger alpha value means the response to the scale items are more closely correlated with each other and thus a more consistent measure of the latent construct. In our case, the main study involved several multi-item scales that were developed to measure nonprofit characteristics that might potentially influence the level of revenue diversification and organizational performance. These scales include the following: managerial success (five items), internal development (three items), and external relationships (two items). Factor analysis was conducted on the entire sample to examine the dimensionality of each scale. Table 6 presents the Cronbach’s alpha for each scale separately for the web survey group and the mail survey group. The results show that the internal consistency (reliability) of all three scales was higher in the mail survey group than in the web survey group.

Table 6.

Reliability Coefficients of Scales in Web and Mail Surveys

	Web	Mail
Scale 1 (Managerial success)	.7990	.8769
Scale 2 (Internal development)	.8703	.9011
Scale 3 (External relationships)	.4135	.6488

Mean Values of Study Variables

We compared mean values of selected variables across the two different survey modes to see if there were any differences between the mail and web surveys in terms of the substantive levels of the study variables. Table 7 presents the mean values of the three scales just discussed, as well as the mean values for five single items that were the primary dependent variables for the original study. The results indicate few differences in these substantive variables across survey modes. Respondents to the web and mail surveys showed no significant difference on mean scores of the three scales. And the two groups of respondents did not differ significantly on most of the five major dependent variables, with the exception on “Funding Success.” Web survey respondents reported a higher level of funding success than mail survey respondents (p = .037).

Table 7.

Comparison of Means for Selected Study Variables

	N	Mean	SD	t	p
Scale 1 (managerial success)				1.125	.262
Web	68	0.151	0.831
Mail	98	−0.028	1.113
Scale 2 (internal development)				−0.082	.935
Web	71	−0.054	0.983
Mail	99	−0.042	1.053
Scale 3 (external relationships)				0.598	.551
Web	71	0.363	0.932
Mail	99	0.277	0.92
Funding variety				0.336	.738
Web	66	4.833	1.697
Mail	94	4.745	1.606
Revenue balance				−0.441	.660
Web	66	0.467	0.290
Mail	94	0.488	0.315
Financial stress				−0.115	.909
Web	69	6.406	2.439
Mail	98	6.449	2.364
Maintain programs and services				1.551	.123
Web	71	6.943	0.286
Mail	99	6.323	0.269
Future funding success				2.105	.037
Web	71	4.057	1.123
Mail	99	3.720	0.956

Costs

Finally, costs were examined. Each mail survey and accompanying letter required US$1.05 postage, with a total of US$299.05 for 285 surveys (initial and third mailings). Each postcard required US$0.28 postage, a total of US$43.96 for 157 postcards (second mailing). Each business reply required US$0.44, a total of US$44 for all 100 returned surveys. The total postage for the mail survey was US$443.01. Other logistical costs for mail survey included envelops, postcards, business reply envelops, mailing labels, and letter head papers. Postage costs did not apply to web survey participants. The only cost for web surveys was subscription to the online survey tool, which was US$16.99 per month, a total of US$51 for 3 months. Cost per response was US$4.43 for the mail survey (only postage is calculated) and US$0.72 for the web survey (only subscription is calculated).

Other costs pertaining to labor time are more difficult to determine (Greenlaw & Brown-Welty, 2009). The creation of the web survey sample required time to collect and verify relevant information. It also took a large volume of time to prepare surveys for mailing (i.e., printing questionnaires and postcards, printing mailing labels, and stuffing envelopes). In addition, once the survey was returned, the data were manually entered into a database, which consumed more time than with the web survey that automatically generated a database after the survey was closed. Although labor cost was not calculated, it was clear from our experience that more labor was dedicated to the mail survey administration than that to the web survey administration.

Discussion

Findings from this experimental study clearly show that a traditional mail survey of nonprofit professionals achieved a much higher response rate than did a web survey. There are several possible explanations for this finding. It could be that nonprofit directors view a mail survey request as more legitimate, perhaps because they are accustomed to getting important documents by mail or because they receive an overload of unsolicited requests by email. It is also possible that some nonprofit professionals may simply prefer to fill out the survey on paper, rather than at their computers. There is a growing concern about web surveys that rely only on email invitations because of the proliferation of email messages of all kinds and because of the ease with which many people ignore or discard unsolicited email messages (Dillman et al., 2009). Regardless of the reasons for the experimentally observed difference in response rates, these results clearly indicate that mail surveys are likely to produce higher response rates when surveying nonprofits and, if web survey methods are employed, extra efforts may well be needed to achieve comparable response rates.

Comparison of the profiles of responding organizations and respondents did not demonstrate any significant difference between the two modes. Although previous studies have reported certain demographic variations between mail and web survey respondents (Friedman et al., 2004; Kwak & Radler, 2002), these differences did not emerge in the context of nonprofit organizations and their directors in this study. Thus, it appears that there is no obvious, systematic difference or potential bias related to the types of organizations and individuals that are responding, or not responding, to the web survey. On a related note, when we compare our study sample to the larger nonexperimental sample, a number of noticeable differences deserve mention. In particular, the study sample is more likely to have a male respondent (significant at .001), more likely to have a respondent that is 55 or older (significant at .10), and is more likely to be a youth or community improvement organization (significant at .05). These differences suggest certain systematic differences between the study sample and the larger nonexperimental sample, which implies somewhat of a limitation on the generalizability of the experimental findings.

We found that our mail survey and web survey did not differ in terms of survey completeness and percentage of missing items, two important indicators of data quality. And in terms of the levels (means) of various substantive variables in the study, we found few differences between the two survey modes. However, we did find that scales in the web survey did not display the same level of internal consistency as those in the mail survey. This difference in reliability matters in the context of nonprofit research because many measures of organizational behavior involve multi-item scales. Moreover, the lower levels of reliability in the web survey imply that these variables contain more noise (random error) and thus analytical correlations or regressions based on these variables are likely to be attenuated (Remler & Van Ryzin, 2011).

The above findings should be interpreted with some caution and study limitations in mind. Our study involved a survey of nonprofit organizations in New Jersey only, and only those organizations operating in human services and community improvement areas. The finding of an advantage for mail surveys may not generalize to other states (or other countries) and other categories of nonprofit organizations. Another limitation in our study is coverage error that might arise from the fact that we used a sample of 320 organizations that had available personal email addresses, which is only a proportion of the original sample of 1,115. In other words, the study does not consider those organizations that are less electronically accessible and might have different approaches to responding to web and mail surveys from those included in our frame sample. Finally, response rates and other outcomes, such as internal consistency and means, also depend on the particular design and content of a survey (Groves et al., 2009). Other types of surveys, covering different substantive topics and involving different types of questions, could result in outcomes that vary from what we found in our study.

Having recognized these limitations, our study suggests some implications and directions for future methodological research involving nonprofit organizations. First, it would be important to replicate the kind of mode experiment reported here with other types of nonprofits, in other locations, and using other survey instruments. Second, future methodological studies could follow-up on the reasons why nonprofit directors seem to prefer mail surveys, if indeed this finding is confirmed. It would be desirable, for example, to conduct qualitative studies to explore nonprofit professionals’ attitudes toward different survey modes. Finally, we believe it would be extremely useful to test the potential of mixed-mode surveys for studying nonprofits, as such an approach allows the respondents to choose their particular mode preference (Shannon & Bradshaw, 2002). Mixed-mode surveys have been proposed to be as cost efficient and successful in improving survey response rates (Dillman et al., 2009; Greenlaw & Brown-Welty, 2009). Due to time constraint and lack of resources, this study was not able to use a multimode approach.

In summary, the results of our mode experiment involving nonprofits lends support for findings from other fields that web surveys generate lower response rates than mail surveys, yet with small differences between these modes in terms of data quality and in the distributions of respondent characteristics and substantive study variables. These results warrant further research to investigate the extent of mode effects on nonprofit survey research. It would be desirable to replicate this study with more general nonprofit populations and by using different study designs that could include mixed modes.

Conclusion

Comparing mail and web surveys in the nonprofit context is important because surveys are a basic source of data in the study of nonprofits and because mode differences may influence data quality and substantive findings in the field. Our experimental study found that the mail survey achieved a significantly higher response rate than the web survey, suggesting that mail surveys may produce more representative and generalizable results. Survey data obtained from the mail survey also produced measures with greater reliability than those obtained from the web survey. There was no difference between methods, however, in respondent characteristics, the completeness of the survey, and the percentage of missing items. As for some of the logistical concerns, our experience suggests that the mail survey was more expensive to administer and required more labor costs. Taken together, the findings imply that a mail survey, although more costly, may have response-rate and data-quality advantages over a web survey as a methodology for gathering data from nonprofit organizations.

Footnotes

Acknowledgements

The authors gratefully acknowledge Jeffrey L. Brudney and the anonymous reviewers for their helpful comments.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Bios

Weiwei Lin, PhD, is an assistant professor in the Department of Public Policy, University of Massachusetts Dartmouth. Her research focuses on nonprofit management, public and nonprofit performance measurement, and program and policy evaluation.

Gregg G. Van Ryzin, PhD, is an associate professor in the School of Public Affairs and Administration, Rutgers University. He has published widely on citizen satisfaction with public services, program evaluation, and survey research methods and is coauthor (with Dahlia K. Remler) of Research Methods in Practice (SAGE).

References

Baruch

Holtom

B. C.

(2008). Survey response rate levels and trends in organizational research. Human Relations, 61, 1139-1160.

Bech

Kristensen

M. B.

(2009). Differential response rates in postal and web-based surveys among older respondents. Survey Research Methods, 3(1), 1-6.

Berry

J. M.

Arons

D. F.

Bass

G. D.

Carter

M. F.

Portney

K. E.

(2003). Surveying nonprofits: A methods handbook. Washington, DC: Aspen Institute.

Cole

S. T.

(2005). Comparing mail and web-based survey distribution methods: Results of surveys to leisure travel retailers. Journal of Travel Research, 43, 422-430.

Couper

M. P.

(2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64, 464-494.

Couper

M. P.

Miller

P. V.

(2008). Web survey methods: Introduction. Public Opinion Quarterly, 72, 831-835.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

(2009). Internet, mail, and mixed method surveys: The tailored design method. Hoboken, NJ: John Wiley.

Dolnicar

Laesser

Matus

(2009). Online versus paper: Format effects in tourism surveys. Journal of Travel Research, 47, 259-316.

Fricker

R. D.

Schonlau

(2002). Advantages and disadvantages of internet research surveys: Evidence from the literature. Field Methods, 14, 347-367.

10.

Friedman

E. M.

Clusen

N. A.

Hartzell

(2004). The net effect: A comparison of internet and mail survey respondents. Retrieved from http://www.websm.org/uploadi/editor/1179950243Friedman_et_al_2004_Net_Effect.pdf

11.

Greenlaw

Brown-Welty

(2009). A comparison of web-based and paper-based survey methods: Testing assumptions of survey mode and response cost. Evaluation Review, 33, 464-480.

12.

Grønbjerg

K. A.

(1993). Understanding nonprofit funding: Managing revenues in social service and community development organizations. San Francisco, CA: Jossey-Bass.

13.

Groves

R. M.

Fowler

F. J.

Couper

M. P.

Lepkowski

J. M.

Singer

Tourangeau

(2009). Survey methodology (2nd ed.). Hoboken, NJ: John Wiley.

14.

Hager

M. A.

Wilson

Pollak

T. H.

Rooney

P. M.

(2003). Response rates for mail surveys of nonprofit organizations: A review and empirical test. Nonprofit and Voluntary Sector Quarterly, 32, 252-267.

15.

Jones

Pitt

(1999). Health surveys in the workplace: Comparison of postal, email, and word wide web methods. Occupational Medicine, 49, 556-558.

16.

Kaplowitz

M. D.

Hadlock

T. D.

Levine

(2004). A comparison of web and mail survey response rates. Public Opinion Quarterly, 68(1), 94-101.

17.

Kwak

Radler

(2002). A Comparison between mail and web surveys: Response pattern, respondent profile, and data quality. Journal of Official Statistics, 18, 257-273.

18.

Manfreda

K. L.

Bosnjak

Berzelak

Hass

Vehovar

(2008). Web surveys versus other survey modes—A meta-analysis comparing response rates. International Journal of Market Research, 50(1), 79-104.

19.

McCabe

S. E.

Boyd

C. J.

Young

Crawford

Pope

(2005). Mode effects for collecting alcohol and tobacco data among 3rd and 4th grade students: A randomized pilot study of web-form versus paper-form surveys. Addictive Behaviors, 30, 663-671.

20.

Naus

M. J.

Philipp

L. M.

Samsi

(2009). From paper to pixels: A comparison of paper and computer formats in psychological assessment. Computers in Human Behavior, 25, 1-7.

21.

Remler

D. K.

Van Ryzin

G. G.

(2011). Research methods in practice. Los Angeles, CA: Sage.

22.

Schaefer

D. R.

Dillman

D. A.

(1998). Development of a standard e-mail methodology: Results of an experiment. Public Opinion Quarterly, 62, 378-397.

23.

Schillewaert

Meulemeester

(2005). Comparing response distributions of offline and online data collection methods. International Journal of Market Research, 47,163-178.

24.

Shannon

D. M.

Bradshaw

C. C.

(2002). A comparison of response rate, response time, and costs of mail and electronic surveys. Journal of Experimental Education, 70, 179-192.

25.

Trochim

W. M. K.

(2001). The research methods knowledge base. Cincinnati, OH: Atomic Dog.