Social Desirability Bias and Mode Effects in the Case of Voting Behavior

Introduction

Social desirability bias in surveys is the tendency of some respondents to intentionally provide answers that are socially acceptable to interviewers, but the answers are not necessarily their true responses (Richman et al., 2003; Tourangeau and Yan, 2007; Tourangeau et al., 2000). It can be seen as a response error associated with the presence of an interviewer, sensitivity of survey questions, and certain social or psychological conditions. The needs for social approval or conformity to social standards lead the respondents to provide answers that are socially acceptable. Typical examples would be that people tend to report they visit the dentist or go to church regularly, or donate money or food to the charities (Presser and Stinson, 1998).

Efforts to reduce social desirability bias have been made and any change of the associated factors may reduce the motivation to misreport the unacceptable attitudes or behaviors (Tourangeau et al., 2000). For example, the presence of an interviewer often increases the possibility of social desirability bias due to interpersonal interactions. Therefore, self-administered interviews are often suggested to reduce such bias, as the anonymity of the respondents is increased.

On the other hand, it is not clear whether the level of social desirability will be different between telephone and face-to-face interviews, where interviewers are present in both modes of data collection. The latter is expected to obtain more socially desirable responses than the former, as it takes longer time and slower pace during the interview process. The rapport developed between respondents and interviewers encourages respondents to think thoroughly and provide answers that are socially acceptable (Tourangeau et al., 2000). However, recent studies in the USA and European countries have indicated that a higher level of social desirability bias is found in telephone respondents than in face-to-face interviews on studies of political attitudes and election (Díaz de Rada, 2011; Holbrook et al., 2003; Voogt and Saris, 2005). Such an issue is worthy of further examination as empirical findings have not reached consensus.

Voting is generally seen as a civic duty in democracy societies and also a socially desirable act (Huang, 2013; Voogt and Saris, 2005). The advantage of using voting behavior as an example for social desirability is the availability of voter turnouts of each election as the “true scores” to compare with survey results. It is common to find overreporting of votes in post-electoral surveys, which is associated with social desirability bias and can alter the effects of socio-demographic characteristics (Cassel, 2003; Selb and Munzert, 2013; Stocké and Stark, 2007; Zeglovits and Kritzinger, 2014). This study aims to examine mode effect on social desirability bias by comparing face-to-face interviews and CATI on voter turnout, given the divergent results associated with voting behavior in previous studies. The 2004 presidential election in Taiwan is used as an example and survey data collected several months after the election are analyzed to understand the level of social desirability in both modes.

Social Desirability and Interviewer-involved Data Collection

Social desirability bias can be attributed to personality, impression management, and conformity to social norms (Fu and Lu, 2012; Tourangeau and Yan, 2007). The degree of the bias may vary depending on cultures (Crown and Marlowe, 1960). The measures for social desirability usually include attitudes, beliefs, or behaviors on sensitive questions. Therefore, respondents tend to provide socially acceptable answers, as well as to deny socially disapproval attitudes or behaviors, when the issues are considered sensitive in the society. These issues include the use of drugs, psychological disorder, contagious disease, or voting behaviors (Bowling, 2005; Tourangeau and Smith, 1996; Tourangeau and Yang, 2007; Tourangeau et al., 2000). However, lack of “gold standards” of these attitudes and behaviors often makes it difficult to verify their answers. Electoral results and test scores are, then, often used to examine the level of such bias because there are known facts.

In addition to the sensitivity of survey questions, the presence of an interviewer is also associated with social desirability, as it is assumed that the respondents are concerned, at least to some extent, about the interviewer’s approval or disapproval of their answers (Tourangeau et al., 2000). Traditional views in social and personality psychology conceived the respondent’s need for approval as a personality trait but it is not necessarily contradictory to the view that considers the interviewing situations (Tourangeau et al., 2000). The influences of the presence of an interviewer and the topic of survey questions are intertwined on the respondent’s need for social approval on their answers. However, this need varies among individuals and is difficult to measure. The presence or absence of an interviewer and the nature of the questions under discussion are then often controlled in empirical studies to examine the level of social desirability bias. When the situations are similar, that is, when the interviewer is present or when the questions are sensitive, it seems reasonable to expect similar levels of social desirability. However, previous studies have demonstrated the opposite.

Meta-analysis based on existed survey data has found that social desirability bias is higher in face-to-face interviews than in telephone interviews, although the difference is rather small (De Leeuw and van der Zouwen, 1988; Holbrook et al., 2003; Presser and Stinson, 1998). Questions examined in the meta-analysis include attitudes toward surveys, voting behavior, and participation in religious activities. When using data from experimental design, social desirability bias is found to be higher in telephone surveys than in face-to-face interviews on voting behavior and mental health (Voogt and Saris, 2005; see also Bowling, 2005). Similar results are found in a pre-electoral survey on voting intention (Díaz de Rada, 2011).

With respect to the higher level of social desirability in telephone surveys on voting behavior, Voogt and Saris (2005) suggest that it may be attributed to memory effects, as demonstrated by Stocké and Stark (2007), due to the fact that face-to-face interview is conducted few months later than telephone interviews in the data collection period. In addition, those who participated in face-to-face interviews often have little interests in politics and, therefore, they may report not having voted, even if they did. It is also possible that the rapport developed during lengthy face-to-face interviews may have encouraged respondents to admit things that may not be socially acceptable (Holbrook et al. 2003).

Previous studies examining mode effects on social desirability for voting behavior between modes are mainly in Western countries. As culture is considered a key condition for social desirability bias (Crowne and Marlowe, 1960), it is of interest to investigate the issue in Taiwan where overreporting of votes is found (Selb and Munzert, 2013). Existed survey data using nationally representative samples are employed for this study as the first attempt.

Research Design

To compare social desirability bias between telephone and face-to-face interviews, the actual voter turnout of the Taiwan presidential election in 2004 is used as the true scores. Data for telephone interviews are part of the first wave of the 2004 Taiwan Social Image Survey (TSIS) (Chiu, 2004), collected from the end of May to mid-June in 2004 using CATI. A three-stage stratified systematic sampling approach was used to select respondents aged 18 years and older in Taiwan. The first stage was to divide listed residential numbers into 316 district clusters at the city/municipality level. Next, a systematic sampling method was used to select numbers based on the principle of population proportional to size (PPS) in each of the clusters with the last two digits being replaced by random numbers. After the telephone numbers were dialed, a probability-based within-household sampling approach developed by Hung (2001) was employed to select one qualified individual within each household. A total of 1,211 complete cases are obtained with a response rate of 34.11 percent (RR5).

Data for face-to-face interviews using paper questionnaires (PAPI) are drawn from the 2004 Taiwan Social Change Survey (TSCS), module of Citizenship (Chang and Fu, 2006), which includes core questions of the International Social Survey Programme (ISSP). Surveys are delivered to individuals aged 18 years and older within each selected municipality from July to September in 2004. Using three-stage stratified sampling based on the urbanization level of townships as the primary sampling unit (PSU), PPS sampling method was employed in the first two stages—township and village or li under townships, respectively. Finally, household-registered residents in each village/li were systematically selected to obtain a representative sample of Taiwan’s population. A total of 1,781 complete cases are obtained with a response rate of 44 percent.

For the purpose of comparison, question items that are included in both of CATI and PAPI surveys, as well as those that are relevant to voting behavior or political attitudes, are used in this study. Direct measures of social desirability bias are unfortunately not available in these surveys. Since neither of the surveys was designed for the purpose of comparison, some of the item wordings are slightly different but the response labels for the selected questions are the same. Both of the datasets are weighted by age, gender, and geographic areas of the population for further analysis.

Voting behavior as an indirect measure of social desirability is measured using the following question: Who did you vote for the 2004 presidential election in March? Since there were only candidates from two major parties for the 2004 presidential election, the response categories are candidates’ names of Democratic Progressive Party (DPP) and Nationalist Party or Kuomintang (KMT), as well as cast invalid vote, did not vote, no voting right, and refusal. These two political parties differ mainly in their policies on the relationship between Taiwan and China while both are seen as leaned-to-the-right in politics.

Attitudes related to policies of the two political parties include restrictions on issuing citizenship ID to women from China and East Asian countries who married a Taiwanese and live in Taiwan, frequency of contacts with foreigners, and experience of traveling abroad. Selection of these variables also takes into account their availability in the two data sources. Variables associated with socio-demographic characteristics include respondent’s age, gender, education, marital status, father’s and mother’s ethnicity, and residential area. Parents’ ethnicity distinguishes those whose parents migrated from China in the late 1940s, mainly with the KMT government, after World War II. The majority of these migrants (so called Mainlanders) and their descendants were politically affiliated with KMT and may have the different national identity from others in Taiwan. More historical background about Taiwan’s nationalistic politics can be found in Chang (2005). The influence of ethnicity on the support of political parties in Taiwan is also discussed in Huang (2005).

While both of the data sets are weighted by population characteristics, analyzed variables are first compared across groups to examine the dis/similarity of the two samples. Multinomial logistic regressions are employed on voting behavior separately for the two samples to examine the effects of the associated variables. Furthermore, data from the two samples are combined and a variable for data sources, with face-to-face interviews as the reference, is added in the final model to estimate the impact of data collection mode on voting behavior.

Sample Description

Descriptive statistics of CATI and PAPI samples are reported in Table 1. As indicated the mean age of the respondents is 43.57 years old for CATI sample and 42.39 years old for PAPI sample and both have equal proportions of males and females. The majority of the respondents in both samples were married and with a secondary school degree. With respect to parents’ ethnicity, the proportions of parents identified as Minnan in PAPI sample are higher than those in CATI sample. On the other hand, the proportions of those who identified their parents as Mainlanders are almost twice higher in CATI sample as in PAPI sample. The distribution of residential areas is the same between the two samples, which can be attributed to population weight.

OPEN IN VIEWER

Table 1.

Descriptives of the CATI and PAPI samples

	CATI	PAPI
Variable	Mean/%	Mean/%
Age	43.57 (0.48)	42.39 (0.39)
Gender-Male	50.5%	50.5%
Marital status***
Single	25.0%	30.5%
Married/cohabited	67.9%	60.9%
Divorced, separate, widowed	7.1%	8.6%
Education***
Elementary or less	17.5%	23.8%
Secondary school	41.4%	40.7%
Some college or more	41.1%	35.5%
Father’s ethnicity***
Minnan	67.8%	75.1%
Hakka	13.5%	15.5%
Mainlander	16.0%	8.3%
Others	2.6%	1.1%
Mother’s ethnicity***
Minnan	72.7%	78.7%
Hakka	14.8%	15.0%
Mainlander	9.0%	4.6%
Others	3.6%	1.7%
Residential area
North region	41.6%	41.6%
Middle region	25.0%	25.0%
South region	28.7%	28.7%
East region	4.8%	4.8%
Experience of travelling abroad***
Yes	36.7%	43.1%
No	63.3%	56.9%
Contact with foreigners***
Never	63.0%	51.4%
Once a year or less	9.2%	21.6%
At least once a month	12.3%	12.1%
Almost everyday	15.4%	14.9%
Citizenship for Chinese brides***
Almost no restriction	21.3%	32.5%
Some restriction	36.7%	38.3%
Strictly restriction	42.0%	29.2%
Citizenship for Southeast Asian brides***
Almost no restriction	20.0%	35.6%
Some restriction	43.3%	39.8%
Strictly restriction	36.7%	24.6%
n	1211	1781

*: p<.05; **: p<.01; ***: p<.001.

The experience of traveling abroad, contact with foreigners, and attitudes toward foreigner brides from China or Southeast Asia also differed significantly between CATI and PAPI samples. More respondents in PAPI sample had experience of traveling abroad than those in CATI sample. On the other hand, higher proportions of the CATI respondents never have contact with foreigners, while indicating some or strict restriction on issuing citizenship for Chinese or Southeast Asian brides.

The survey results of voting behavior and the actual voter turnout are presented in Table 2. The actual voter turnout for the 2004 presidential election is reported in the rightmost column, showing DPP the winning party with marginal success. When comparing the voting behavior between the two samples, more CATI respondents voted for KMT or indicated that they did not vote, while more PAPI respondents voted for DPP. When the responses to voting behavior are transformed to voter turnout, in PAPI sample the percentage of votes for the winning party DPP and the turnout rate are higher than the actual voter turnout, which is quite common from post-electoral surveys. The reported voter turnout in CATI sample, however, is contrary to the actual turnout, with overreporting of votes for KMT than DPP. Refusal rates for the two samples are similar, with 11.5 percent in CATI and 10.4 percent in PAPI, respectively.

OPEN IN VIEWER

Table 2.

Voting behavior and voter turnout for the 2004 presidential election*

	CATI		PAPI
Voting behavior	%	Turnout	%	Turnout	Voter turnout
DPP	29.9	46.95%	40.8	58.36%	50.11%
KMT	33.8	53.05%	29.1	41.64%	49.89%
Did not vote	19.4		11.7
Invalid vote	3.3	4.81%	2.7	3.64%	2.54%
Refusal	11.5		10.4
No voting right/Don’t know	2.2		5.3
Voter turnout rate		77.51%		86.12%	80.28%
n	1211		1781

*: Respondents who did not vote and those who did not have the right to vote were excluded from the calculation of turnout rates for CATI and PAPI samples.

Mode Effect on Voting Behavior

Separate models for voting behavior are first analyzed for each of the samples. The winning party DPP is used as the reference category in the multinomial logistic regression models. As shown in Table 3, variables that have the significant effect on voting behavior generally indicate the same direction of influence in CATI and PAPI samples, despite of the significant difference in the voter turnout. In both samples, females are more likely to vote for KMT over DPP than males. In addition, those whose father are identified as Minnan or Hakka, those whose mother are identified as Minnan, and those who prefer more restriction on issuing citizenship to Chinese brides are less likely to vote for KMT over DPP than their counterparts.

OPEN IN VIEWER

Table 3.

Regression coefficients of voting behavior: CATI and PAPI

		CATI			PAPI
Variables	KMT	Refusal	No vote	KMT	Refusal	No vote
Intercept	4.82***	3.44***	2.40*	4.08***	1.83*	−.83
Age	−.02*	−.03**	−.02	−.01	−.02*	.01
Gender
Female	.37*	−.17	.41	.29*	−.05	.41*
Male	0	0	0	0	0	0
Marital status
Single	−.40	.47	−.13	.09	−.38	.42
Married/cohabited	−.17	.03	.55	.21	−.62*	.80*
Divorced, separate, widowed	0	0	0	0	0	0
Education
Primary school or less	−.13	.53	.63	−1.03***	−.72*	−.22
Secondary school	.03	.12	−.15	−.41**	−.25	.01
Some college or more	0	0	0	0	0	0
Father’s ethnicity
Minnan	−.13***	−.91*	−.75	−1.43***	−.11*	−.42
Hakka	−.16***	−.67	−.56	−1.30**	−.10	.38
Mainlanders/others	0	0	0	0	0	0
Mother’s ethnicity
Minnan	−.87*	−.31	−.16	−1.47**	−.50	−.96
Hakka	−.53	−.66	−1.09	−1.01	−.23	−1.69*
Mainlanders/others	0	0	0	0	0	0
Residential area
North region	−.20	−.33	−.99	−.12	.41	.29
Middle region	−.42	−.49	−.75	−.51	.10	−.68
South region	−.52	−.86	−1.26*	−.81**	.10	−.21
East region	0	0	0	0	0	0
Have traveled abroad	−.54**	−.02	−.29	−.24	.27	−.09
Contact with foreigners	−.04	−.04	−.06	−.06	−.08	−.01
Citizenship for Chinese brides	−.83***	−.65**	−.33	−.74***	−.44*	−.37
Citizenship for East Asian brides	.18	.17	−.27	.37*	.34	.21
Pseudo R-square	.228			.216

*: p<.05; **: p<.01; ***: p<.001.

In regard to those who refused to answer whether and who they have voted, similar patterns are found between the two samples as well. Older respondents, those who identified their father as Minnan, and those who prefer more restriction on issuing citizenship to Chinese brides are more likely to vote for DPP than being refusal to the question. Respondent’s marital status, residential area, the experience of traveling abroad, and attitudes toward issuing citizenship to Southeast Asian brides are also found to have significant effects on voting behavior in either CATI or PAPI samples. It is noticeable that educational level is found to have the inconsistent influence on voting behavior for the two samples. PAPI respondents who have a secondary school degree or lower are less likely to vote for KMT over DPP than those who have some college or higher degree. However, a different pattern is found among CATI respondents. Pseudo r-square of the model is .228 for CATI and .216 for PAPI, respectively.

Data from both samples are pooled to estimate the impact of mode on voting behavior. The results of multinomial logistic regression are reported in Table 4, with DPP as the reference category. It is indicated that CATI respondents are more likely to vote for KMT or refuse to answer the question over voting for DPP than their PAPI counterparts. The effects of other variables remain the same as they are in the model for separate samples while some of the direction differs from that in the separate model. The effect of education is found only among those with a degree of primary school or less. These respondents are less likely to vote for KMT over DPP than their counterparts. Pseudo r-square of the model is .219.

OPEN IN VIEWER

Table 4.

Regression coefficients estimating the impact of mode on voting behavior

Variables	KMT	Refusal	No vote
Intercept	3.91***	2.08**	.14
Mode
CATI	.38***	.85***	.26
PAPI	0	0
Age	−.01*	−.03***	−.00
Gender
Female	.31**	−.12	.39**
Male	0	0	0
Marital status
Single	−.06	−.02	.25
Married/cohabited	−.09	−.37	.71*
Divorced, separate, widowed	0	0	0
Education
Primary school or less	−.75***	−.19	.02
Secondary school	−.23	−.09	−.06
Some college or more	0	0	0
Father’s ethnicity
Minnan	−1.38***	−.99***	−.62
Hakka	−1.39***	−.76*	−.14
Mainlanders/others	0	0	0
Mother’s ethnicity
Minnan	−1.14***	−.46	−.55
Hakka	−.71*	−.49	−1.23*
Mainlanders/others	0	0	0
Residential area
North region	−.08	.14	−.12
Middle region	−.40	−.12	−.59
South region	−.62*	−.31	−.59
East region	0	0	0
Have traveled abroad	−.35**	.16	−.17
Contact with foreigners	−.05	−.05	−.03
Citizenship for Chinese brides	−.76***	−.52***	−.33*
Citizenship for East Asian brides	.23*	.28	.03
Pseudo R-square	.219

*: p<.05; **: p<.01; ***: p<.001.

Concluding Remark and Discussion

Mode effect has been extensively examined on sensitive questions (Holbrook et al., 2003; Presser and Stinson, 1998; Rockwood et al., 1997; Tourangeau and Yan, 2007). When an interviewer is present, it is expected that people in face-to-face surveys are more likely to overreport socially desirable behavior than in telephone surveys. The results in voting as a socially desirable behavior, however, did not find support to this claim.

Using data from CATI and PAPI surveys, the self-reported voter turnouts from both surveys are not consistent with the actual voter turnout. The turnout rate is higher in face-to-face interviews than in telephone interviews, which indicates that social desirability bias is high in the former. Respondents tend to manage their impression as faking conformity to the social norm when an interviewer is present in person (Fu and Lu, 2012). The overreporting of the votes in face-to-face when comparing to the actual voter turnout rate proves that voting is seen as a civic duty in a democracy and also a socially desirable behavior (Huang, 2013; Voogt and Saris, 2005).

The voter turnout from PAPI sample is consistent with previous findings that the respondents are partial to the elected. To the contrary, the CATI respondents report more votes for the defeated candidates than the elected ones. Such a result may be attributed to the disproportional Mainlanders of parents’ ethnicity in the CATI sample. Therefore, it is possible that CATI respondents are simply telling the truth by showing their support to KMT. Limitation of this study is the use of existed surveys that are not designed for the comparative purpose and may not include important variables. Future research can take into account direct measures of social desirability when investigating mode effect on voting behavior.