Social Desirability and Undesirability Effects on Survey Response latencies

Introduction

The validity of responses to sensitive questions has been a topic in survey research for several decades. Within the context of sensitive questions, the effects of social desirability are generally the most often looked at type of response effect. Social desirability refers to the tendency of respondents to overstate positive behaviours or characteristics and understate negative ones (cf. Holtgraves, 2004: 161).

Various attempts have been made to assess the extent to which responses are a source of bias in survey results, and to develop ways to avoid having results coloured by social desirability. Besides classical survey methods such as anonymous interview settings, the implementation of need for social approval- and trait desirability-scales in surveys, some techniques designed specifically to encourage respondents to answer truthfully are the randomized response or the item count, faking instructions or bogus pipeline techniques, for example. However, doubts have been cast in terms of the effectiveness of these techniques in eliciting more valid responses (see for example Holbrook and Krosnick, 2010a; Holbrook and Krosnick, 2010b; Wolter and Preisendörfer, 2013). Therefore, researchers have turned to other techniques to identify socially desirable responses as well as to gain a better understanding of why and how people answer in a socially desirable way. One such technique involves analyzing paradata collected about the survey process, often in the form of response latencies to answers. In this way, response latencies are used as proxies, working with cognitive information processing theoretical frameworks, to infer information processing modes. Often longer response latencies are taken as evidence that the respondent was, among other things, grappling with inconsistent information, taking more information into account before forming a judgement, or stopping to “edit” their response in a socially desirable way (cf. Mayerl, 2013: 3). In terms of studies looking at response latencies and socially desirable response behaviour, several researchers have found evidence to support this “editing” hypothesis (cf. Holtgraves, 2004; Silber et al., 2013) as well as “faking good”-effects (e.g. Holden and Hibbs, 1995) leading to slower response latencies. However, others have found conflicting evidence in which socially desirable responses were actually linked to faster response latencies (cf. Amelang and Müller, 2001; Kohler and Schneider, 1995; Mayerl and Urban, 2008).

This paper therefore looks to contribute to the still unclear research field. After the relatively complicated process of preparing raw reaction times into adjusted response latencies, and controlling for item- and respondent-related effects, we look specifically at the effects of trait desirability and need for social approval on response latencies. In doing so, we hope to observe patterns, moving us closer to developing ways to model socially desirable response behaviour. Our data was collected within the research project EVA3PLUS ¹ which looks at preservice teachers’ attitudes towards experiments in biology and chemistry lessons.

We will begin by outlining our research question and secondary interests. Following that, we will look at the theoretical background with a review of response effects and their impact on survey results. After that, we will look at several current models of response behaviour and at ways in which response effects can be integrated into these models. On the basis of these theoretical deliberations, as well as past empirical evidence, we will be able to formulate our hypotheses. After an overview of the data and operationalization we will move on the empirical testing of the hypotheses showing the empirical results, and finally discuss the substantive findings and their implications for future research.

Research Question

This paper looks to contribute to advancing the use of paradata, specifically item response latencies, in identifying socially desirable responses. By using an external measure for establishing the (un)desirability of the characteristics covered in our instrument and by using the shortened Crowne-Marlowe (1960) scale for respondent need for social approval, it is possible for us to establish a measure of the sensitivity of the situation. Using this information, we then observe how response latencies act. The research question can therefore be formulated as: is it possible to observe patterns in response latencies based on either item-based trait desirability or respondent-based need for social approval? Furthermore, the strength and direction of effects of social desirability on response latencies will help to clarify a field that has been characterized by conflicting findings and interpretations. The secondary research question will be then: how does the strength and direction of social desirability affect response latencies?

Theoretical Background

Questions can be sensitive because of 1) their intrusiveness: such questions usually involve topics that are not seen as appropriate to talk about generally in conversation and can lead to a “none of your business” reaction from the respondents. They can deal with taboo topics such as sharing one’s income. Questions can be perceived as sensitive also if the respondent 2) becomes wary that the disclosure may lead to negative consequences if third parties gained access to the information. This is often a problem concerning questions asking the respondent to divulge illegal behaviour such as drug use. Finally, questions can be perceived as sensitive if they 3) ask respondents to express potentially socially (un)desirable attitudes or behaviour (cf. Tourangeau and Yan, 2007: 860). This third element is obviously linked to the second in the way that admitting to drug use could both lead to negative consequences due to its illegality, as well as disapproval from the interviewer. Both of these last two elements presuppose that there are clear social norms surrounding the attitudes or behaviour being expressed (for more in-depth discussion, see Tourangeau and Yan, 2007).

As for socially desirable responding, it is generally seen as a result of three factors: the item’s trait desirability (which involves the social norms surrounding a survey question), the respondent’s need for social approval (the topic-independent tendency of some individuals to seek social approval more than others) and the anonymity of the situation (in conditions of anonymity, the respondent can expect to receive neither approval nor disapproval for their answers). A basic rational choice model stipulates that the likelihood of eliciting a socially desirable response increases with the item’s trait desirability and the respondent’s need for social approval but decreases under conditions of anonymity (cf. Stocké, 2004: 304).

This rational choice-based approach can be criticized in several ways. For one, it takes an unrealistic view of respondents carefully weighing their expected utility before answering every question. Furthermore, it is unrealistic to assume anonymous surveys eliminate social desirability. As others have shown in research based on methods designed to ensure absolute anonymity such as the randomized-response and item count techniques, respondents still tend to report inaccurately even in conditions of anonymity (cf. Preisendörfer and Wolter, 2014). We agree generally with the sentiment that expectancy of consequences should logically increase socially desirable responding, but see this as less of a dichotomous, black-and-white condition. Furthermore, it is conceivable that the expectancy of consequences can be an inner-, personality based prerequisite for socially desirable reporting in terms of avoiding cognitive dissonance, for example. In fact, Esser covers this topic shortly in an article from 2000 in which he cites Elster: “norms do not need external sanctions to be effective. When norms are internalized, they are followed even when violation would be unobserved and not exposed to sanctions” (Elster, 1989, cited by Esser, 2000: 137).

Although classical rational choice-based models can be helpful for an economical conceptualization of the conditions that promote socially desirable responding, they are dragged down by a few of the criticisms discussed above and do not provide a good basis on which to build hypotheses about the behaviour of response latencies in conditions of social desirability. Several other types of models lend themselves better to the task at hand. In fact, several authors have attempted to apply different cognitive models to the explanation of response behaviour and the identification of response effects. Tourangeau and Rasinski (1988) as well as Strack and Martin (1987) looked at socially desirable responding using a phase model of information processing. Krosnick (1991) as well as Schaeffer (2000), Belli et al. (2001), Holtgraves (2004) and Kaminska and Foulsam (2016) have looked at the topic from an optimizing/satisficing perspective. Fazio’s MODE-model (Fazio, 1990) and Esser’s Model of Frame Selection (Esser, 2010) have also been applied, see also Mayerl (2009, 2013). What these models all have in common is the assumption that responding can truly reflect attitudes, or it can be a result of multiple response effects. Furthermore, according to common dual-mode models of cognitive processes (cf. Chaiken and Trope, 1999; Smith and DeCoster, 2000) latent attitudes can cause responses in what we would characterize as “true” values either based on a deliberate attitude generation (much in the way modeled by classical rational choice theory), or a spontaneous activation of already held attitudes. These last two points have implications for the present research. For one, referring back to the first point, response effects can be both topic-dependent and topic-independent. Effects such as acquiescence bias, tendency to the middle category, etc. reflect response effects that are independent of the current topic. Socially desirable responding, on the other hand, is topic-dependent and hinges on the strength and direction of norms surrounding an item (trait desirability) in combination with the respondent’s sensitivity to social desirability in terms of need for social approval (Mayerl, 2013: 7). Secondly, and more related to the application of response latencies as a proxy for socially desirable responding, spontaneous activation of already held, chronically accessible attitudes should result in shorter response latencies (cf. Fazio, 1989, Bassili and Fletcher 1991). Along what is often referred to as “deliberate controlled” process mode, relevant information must first be accessed, evaluated, translated to a scale value causing longer response latencies (cf. Yan and Tourangeau, 2008).

The discussion above can be summed up as follows: respondents can be assumed to behave in either “deliberate-controlled” or “automatic-spontaneous” process modes (although the terminology can vary from model to model: controlled-deliberate mode would equate to an optimizing behaviour, for example) and responses can either reflect latent attitudes or merely response effects. Response effects can be characterized as either topic-independent or topic-dependent. Finally, response latencies along the deliberate-controlled process mode should generally be longer than along the automatic-spontaneous mode.

With all this in mind, the task of explicitly incorporating social desirability into the model however remains incomplete. Several of the above mentioned authors have spent time tackling this problem. Tourangeau and Ransinski (1988) and Strack and Martin (1987) envision socially desirable responding as an “editing” process that occurs at the very end of the model, immediately before entering the final value. Respondents either activate previously generated attitudes or form attitudes on the spot, come to a “true” value and, depending on situation, take a moment to edit their response to conform to what is seen as socially desirable. The editing process occurs regardless of the “mode” of responding (based on whether a judgment of the attitude object is already stored in memory). The implication of this model is that the editing process takes time and therefore socially desirable responses may be associated with noticeably longer response latencies.

Both the optimizing/satisficing and dual process models of Krosnick, Fazio and Esser conceive of both editing processes and automatic, norm-conforming responding. The implication of this is that socially desirable responses can occur very quickly without much hesitation, as well as due to a deliberate editing process. This obviously complicates the relationship between socially desirable responding and response latencies. The next section will discuss the hypothetical model which incorporates much of the discussion above into a model for using response latencies as an indicator for socially desirable responses.

Hypothesis Model

Based on the findings of the authors referenced above, we can discern two main hypothetical implications. Tourangeau and Rasinski and Strack and Martin see responding to survey attitude questions as a four-phase model in which the respondent interprets the question and identifies an attitude object, retrieves relevant information, beliefs and feelings about the object, forms a judgment and enters the response (cf. Tourangeau and Rasinski, 1988). Between the judgment formation and the entering of the answer, pressure to respond in a socially desirable way will trigger an editing phase in which the respondent takes time to consider whether what he or she is about to answer is socially desirable in order to present themselves in a positive light.

On the other hand, the satisficing/optimizing-, MODE- and Frame Selection-Models all have in common the differentiation that, for various reasons, some responses occur in a deliberate, rational fashion while others occur rather spontaneously without much deliberation. We will use Hartmut Esser’s Frame Selection Model to illustrate our thoughts on this, although both other mentioned models could be used to similar effect. The Frame Selection Model stipulates that behaviour is influenced by “frames”, or as Fazio might refer to them “cues”, that, in specific situations, elicit attitude associations between objects and previously stored attitudes. The degree to which that object and situation can be “matched” with a previously stored attitude is then referred to as attitude accessibility (cf. Esser, 2000: 141). A match between object and situation will lead to an automatic activation of either attitude consistent, stimulus-reaction style, or normative behaviour, often referred to as a script. If a sufficient match cannot be found and if the respondent has the motivation, takes the necessary effort and has the opportunity to do so he or she responds in a deliberate-controlled, rationally utility maximizing way (cf. Esser, 2000: 144).

These two models point to explicitly different expectations. The phase model suggests strong norms around sensitive topics should illicit longer response times as the respondent edits his or her response while the frame selection and other dual process models suggest strong norms should ensure attitude matches, enabling the respondent to respond automatically and quickly in a norm-conforming way. However, as Tourangeau himself anticipates that both spontaneous and deliberate modes of responding are possible, and that the editing process can take place along both cognitive routes (cf. Tourangeau and Yan, 2007: 877), we believe that response latencies are mainly determined by mode of processing and that the editing process is at most a minor effect that will have only a small impact on latencies.

Sensitive questions (either due to their intrusiveness, threat of disclosure, the anonymity of the interview situation or the respondent’s own sensitivity or need of social approval) can be seen as a frame or cue that informs the respondent about the logic of the situation triggering an awareness for socially acceptable responses. The more obvious the social norm surrounding an attitudinal or behavioural object, the more likely the script of a topic-specific answering heuristic will be employed.

In terms of response latencies, as suggested in our theoretical framework, the more obvious the social norm surrounding the item (both strongly negatively and positively rated items in terms of trait desirability), the more likely the automatic script of socially desirable responding should be employed and the shorter the response latencies should be. Figure 1 illustrates this. Methodologically speaking, we expect an inverted U-shaped effect of trait desirability on response latencies (a significant positive linear effect and a significant negative quadratic effect). Our study involves surveying student teachers, so in this example since it is fairly clear that teachers should like working with young people or that teachers should not be nervous speaking in front of a crowd, respondents simply answering in a socially desirable fashion should be able to do so quickly. More ambiguous social norms should tend to take more time to answer because it is not clear what the “socially” correct answer is. ² For example, should a teacher be funny? Should they be able to cope well with disappointment? While it is likely desirable to be funny in general, admitting to not being funny does not necessarily imply that one is unfit for the career they have studied for more than five years to follow. The lack of a clear social norm means respondents will not be able to respond quickly in a socially desirable manner.

OPEN IN VIEWER

Figure 1.

Hypothesized effect of trait desirability of an item on response latencies when answering to these items.

Method

Response latencies are used as proxies for different mental processes (cf. Mayerl, 2013: 3). In our study, we use response latencies as a proxy measure of information processing mode with respect to the degree of elaboration (see in the same line of reasoning for example Carlston and Skowronski, 1986; Gibbons and Rammsayer, 1999; Mayerl, 2013; Schaffner and Roche, 2016; Sheppard and Teasdale, 2000; Shiv and Fedorikhin, 2002). With this research, we are attempting to use response latencies for the identification of socially desirable responding. Computer-assisted survey modes like CASI (e.g. web surveys), CATI or CAPI enable the (half-) automatic collection of additional context information of surveys. Survey researchers call this additional information paradata (e.g. Couper and Kreuter, 2013). Paradata cover all computer-assisted automatically conducted information about the response process. In most cases these data are non-reactive and thus not consciously biased by respondents. This includes item-level data like “mouse click” data or time stamps (response times), survey-level data like duration or contact information, and in the case of web surveys server-side as well as client-side data. Recently, response time itself has been in the focus of survey research papers (e.g. Couper and Kreuter, 2013; Mayerl, 2013; Olson and Smyth, 2015; Yan and Tourangeau, 2008).

There are different possibilities for recording response latencies depending on the survey design. CATI and CAPI surveys involve active reaction time measurement by the interviewer who manually stops a timer when the respondent has answered and then must validate in a second-step whether the measurement was valid or not. Passive, or “latent” reaction time measurement is made possible in web-surveys in which timestamps are recorded automatically to record various events. Passive reaction time measurement involves measuring the entire question answering process, including reading the question and entering the answer (cf. Mayerl, 2013: 3). Passive reaction time measurement almost always entails more bias than active time measurements because the lack of an interviewer means it is impossible to ensure that respondents are not distracted, have not left the room or do not have multiple programs open simultaneously (Mayerl and Urban, 2008: 16-17).

We therefore decided on a rather elaborate survey method. We purchased several dozen tablet PCs upon which “kiosk” software ³ installed to prevent other apps and internet webpages being opened during the survey situation. Screens and resolutions were identical, and the questionnaire was optimized for the tablet format. Switching between landscape and portrait formats or zooming in on the screen among other manipulations were locked down. Respondents were given styluses to improve the tablets recognition of “taps” and to improve comparability. Preservice teachers in Rhineland-Palatinate complete the final phase of their education at various locations around the state. All the preservice teachers at a particular location were surveyed at the same time in a room together. One researcher was present throughout the entire filling-out process to assist with technical difficulties (mostly internet connection problems) but remained otherwise passive. Anonymity procedures were explained at the beginning of each survey by the researchers.

The respondents were asked to answer 30 questions related to their self-assessed suitability for the teaching profession. With this instrument, we attempted to elicit socially desirable response behaviour from the student teachers; the rationale being that student teachers may be hesitant to admit qualities that suggest unsuitability. The questions covered 10 sub-dimensions of teacher personalities (see appendix 1 for an overview of the dimensions), with three items per sub-dimension. In each sub-dimension, two items were positively worded (statement indicating suitability for profession) and one was negatively worded (indicating unsuitability). The 10 sub-dimensions were presented to the respondent in a random order and the items within the sub-dimensions were also randomly ordered from first to third item presented on screen. The instrument was taken from a survey by Herlt and Schaarschmidt from 2007. Following the pretest several items were slightly modified from the original questions (cf. Herlt and Schaarschmidt, 2007).

Response latencies are recorded using timestamps that are automatically triggered by actions taken by the respondents. A timestamp is recorded as soon as the website loads completely on the screen, which can be referred to as “event” number 1 (e1). The respondent presumably then begins to read the instructions and question text. Once he or she has decided on an answer and has tapped the screen to indicate the desired scale value, a second timestamp is recorded (e2). Along with the timestamp, several other pieces of crucial information are saved in the dataset: the scale value, the item being referenced, the event’s relative order compared to preceding events, what type of event is being recorded (the loading of the page, a tap on the item-scale, a tap on the “continue/back” button, etc.). If the respondent then moves on to the next question and taps the screen to indicate the desired scale value for the next item, event 3 (e3) is saved with a timestamp. The same goes for e4. If the respondent then decides to go back and change their answer for the first item, a new timestamp is recorded (e5), which is, of course, saved in the dataset with a reference to the first item. The response latency for most answers is then the difference between the preceding event and the tapping on the item-scale. For the first item, however, because the hypothetical respondent changed their answer, the overall response latency is calculated as the time between the page loading (e1) and the first answer for item 1 (e2): Δt1, plus the time difference between e4 and e5: Δt4. The response latency therefore represents an estimate of the time the respondent spent concentrating on each item.

There are obvious deficits to this method that bear noting. For one thing, the response latency for the first item on the screen also encapsulates the time spent on reading the instructions at the top of the page. This solution was a compromise based on the technical procedure used to collect the timestamps. In order to completely control for the time spent reading the instructions, each page would have to be loaded in a 2-step process: the first tap on “continue” would load the screen with only the instructions displayed, and then a second tap on the screen would be needed to signal the questions to be displayed. The already very long questionnaire (30-40 minutes) would become even more trying if the respondent was forced to tap the screen twice on each page for questions to be displayed. Furthermore, varying internet speeds and the fact that the tablets sometimes do not register initial taps leads to respondents often tapping several times and several millisecond-long latencies being recorded (which were eliminated as outliers, see below). The two-step loading process would inevitably lead to even more errors and frustration amongst the respondents. To control for this effect somewhat, a control variable for the position of the item on the screen was included in the model (the item displayed closest to the top of the screen therefore tends to register longer response latencies due to the time spent often also reading the instructions).

Furthermore, the problem just discussed is exacerbated by the fact that we cannot rule out whether respondents begin reading the entire screen first, or whether they start reading from the bottom or middle, when filling out the questionnaire. Our response latency measurement technique is thankfully not effected regardless of which item the respondent answers in what order. The fact remains, however, that the measurement method assumes respondents cognitively handle each item individually and in a one-at-a-time fashion. Without the use of some sort of eye tracking method, it simply cannot be guaranteed that this is the case.

In terms of the analytical design, the statistical model will attempt to explain response latencies based on item trait desirability – the more unambiguous the social norm surrounding the item, the more likely an automatic-spontaneous norm-conforming mode (read: socially desirable) will be activated, and the shorter the response latencies will be.

We test the model using a cross-classified multilevel linear model. This is due to the structure of the data. Each respondent is asked to answer the 30 questions about their suitability for the teaching profession and each of the 30 questions is answered by multiple respondents. The response latencies can therefore be seen as nested at once within respondents and items – certain respondents will respond generally faster or slower than others, and certain items will cause respondents to take more or less time responding to them. Furthermore, the study was a panel study in which the 30 characteristic items were administered in up to two out of three waves of the survey. It is important here to note that response latencies are then also theoretically nested within different waves of the same respondent. However, for a multilevel analysis, generally more than 30 (or at least 20) macro-level units are needed for stable results and in this case we have only two. Therefore we cannot integrate the panel wave as an additional macro-level in which response latencies would be nested. The fact that the response latencies of the same respondent at two different times may not be independent of each other means we resort to the more primitive method of including a dummy variable in the multilevel model. In this way, we can hold any effects constant that may arise from respondents answering the same questions again.

Data and Operationalization

The research presented here resulted from the research project EVA3PLUS conducted at the University of Kaiserslautern in Germany. It is an interdisciplinary project conducted by the Biology and Chemistry Didactics Departments together with the Department for Empirical Social Research. The substantive goal of the research is to assess student teachers’ attitudes and behaviour with regards to conducting experiments in the biology and chemistry classrooms following a university educational reform. The project is a longitudinal panel-study with computer assisted self-interview (CASI) web-based tablet questionnaires with three survey waves taking place at intervals of six months. The project attempts to conduct a complete sample of all chemistry and biology student teachers at the Gymnasium-level (a university/college preparation-level secondary school form in Germany) in Rhineland-Palatinate from mid-2014 to the end of 2016. In total, the sample size is 550 with 357 individual respondents participating between one to three times.

The 30 teacher characteristic items were partially modified from an instrument by Herlt and Schaarschmidt, 2007. The instrument contains 10 constructs ranging from “interaction with younger people”, “humour”, “didactic abilities” and so on. Each construct contained three items; two of which were formulated positively, one formulated negatively. The items were measured on a 7-point scale ranging from 1 “does not apply to me at all” to 7 “applies fully and completely to me”.

Respondent-related variables included a baseline speed variable to adjust for generally slow or fast respondents (the baseline speed is established based on response latencies for the sociodemographic variables) as well as a dummy variable for whether or not it was the second time the respondent participated.

For the item-related variables we will specifically be looking at factors that must be controlled for in order to evaluate the cognitive processes irrespective of several formal characteristics of the questions. A variable for the item length in terms of syllables was included. Furthermore, the item position was taken into account in terms of its overall position out of thirty and its appearance on screen through the use of dummies for being either the first or last of three items presented at once.

Finally, in terms of the variables of central substantive interest, the respondent’s need for social approval and the item’s trait desirability were modeled.

In order to assess the trait desirability of the instrument’s items, a small pencil-and-paper survey of students at the TU Kaiserslautern was conducted (n=77). The students were asked to assess how strongly the presented characteristics of teachers are expected societally. The scale ranged from −4: “extremely undesirable” to +4: “extremely desirable” with 0 as the middle category: “neutrally seen”. The mean scores that resulted from the survey can be found in the appendix. The expected nonlinear relationship between trait desirability and response latency was accounted with the inclusion of a quadratic term (the simple correlation between the trait desirability variable and its quadratic term was not large; r=0.230).

The respondents’ need for social approval was measured using an index of two items from the Crowne-Marlowe social desirability scale (cf. Crowne and Marlowe, 1960: 351). The items used were “No matter who I’m talking to, I’m always a good listener” and “I am always courteous, even to people who are disagreeable”. The trick to this scale is that these items should not realistically apply to just about anyone. Respondents that nevertheless answer strongly in the affirmative have a strong need for social approval, otherwise they would be comfortable admitting this normal and human behaviour. The index created was additive, displaying satisfactory characteristics (α=0.621). The descriptive statistics of the items are also found in the appendix.

Empirical Analysis

Table 1 gives an overview of the dependent variable, the response latencies. Due to the large degree of non-normality of the distribution, and in order to eliminate outliers as suggested by Mayerl and Urban (2008), the top and bottom 5% of the distribution was eliminated. This resulted in a mean response latency of 4.64 seconds, a standard deviation of 4.16 and a range of 1.88 to 11.86. The skewness and kurtosis values show the adjustment resulted in a more normally distributed variable. Due to the fact that response latencies are often skewed right (which is indeed the case with our distribution), taking the natural log of the variable is also advisable when applying statistical procedures that are based on the assumption of normality of the data (cf. Mayerl and Urban, 2008: 82).

OPEN IN VIEWER

Table 1.

Descriptive statistics dependent variable

	N	Mean	Med.	Std. Dev.	Var.	Skew.	Kurt.	Min.	Max.
Response latency (seconds per item)	9713	5.04	4.13	4.08	16.64	6.45	89.42	0.08	105.10
Response latency, outliers removed (<5%, >95%)	8769	4.64	4.16	2.00	3.99	1.20	1.21	1.88	11.86
Response latency, log outliers removed (<5%, >95%)	8769	1.45	1.42	0.40	0.16	0.29	−0.46	0.63	2.47

The first step in the empirical analysis was to establish a baseline model with only the second-level group variables as determinants to establish interclass correlations (ICCs). Table 2 provides an overview of this model. ICCs express the amount of the overall variance that can be traced back to the macro-level variables. The residual variance is therefore the variance not explained by the macro-level variables. ICCs are determined based on the simple calculation of the macro-level variance divided by the overall variance. The table shows the respondent is responsible for roughly 13% (0.523 ÷ (3.2079 + 0.5230 + 0.2944) = 0.1299) and the item for 7% (0.2944 ÷ (3.2079 + 0.5230 + 0.2944) = 0.0721) of the overall variance of the response times. Generally, if more than 5% of the overall variance can be traced by to macro-level variables, they should be included in the form of a multilevel analysis (cf. Hox, 2010: 244). This leads to the conclusion that both group variables are empirically and substantively important.

OPEN IN VIEWER

Table 2.

Interclass correlations

	Variance	Std. Error	Wald Z	Sig.	Confidence Interval 95%		ICC
					Lower	Upper
Residual	3.2079	0.0495	64.853	0.000	3.1124	3.3063
Respondent	0.5230	0.0509	10.266	0.000	0.4321	0.6331	0.1299
Item	0.2944	0.0789	3.373	0.000	0.1741	0.4979	0.0721

Table 3 shows the cross-classified random-intercept fixed-slope multilevel model. The unstandardized parameter estimates are shown in the column Estimate. The scaling of the independent variables are shown in parenthesis in the column Parameter in order to aid the interpretation of the unstandardized coefficients. The dependent variables is the outlier treated response latency. Parameter estimates can therefore be interpreted in seconds. As was to be expected, the longer the question is in terms of syllables, the more time was needed to respond, the first item on screen took longer to respond to (due partially to the instructions also being presented at the top of the screen), and the respondent’s baseline speed has a significant positive effect on response latencies: the slower the respondent is in general, the longer the response latencies on the 30 characteristics questions.

OPEN IN VIEWER

Table 3.

Fixed parameter estimates, dependent variable: response latencies, outliers removed ⁴

Parameter	Estimate	Std. Err.	DF	T-statistic	Sig.	Confidence (95%)
						Lower limit	Upper limit
Intercept	2.1443**	0.4951	185.525	4.331	0.000	1.1675	3.1210
Repeat respondent (0,1)	−0.1301	0.1276	150.468	−1.019	0.310	−0.3821	0.1220
Respondent baseline speed (metric)	0.1462**	0.0371	149.390	3.938	0.000	0.0728	0.2195
Number of syllables (metric)	0.0728**	0.0098	30.124	7.464	0.000	0.0529	0.0927
Item position in questionnaire (1-30)	−0.0077	0.0048	30.296	−1.597	0.121	−0.0175	0.0021
Item first on screen (0,1)	1.3011**	0.0992	29.898	13.121	0.000	1.0985	1.5036
Item last on screen (0,1)	0.2762**	0.0992	29.756	2.785	0.009	0.0736	0.4789
Trait desirability (−4 - +4)	−0.0714**	0.0206	29.562	−3.456	0.002	−0.1135	−0.0292
Trait desirability x Trait desirability (0-16)	0.0287	0.0190	29.639	1.511	0.141	−0.0101	0.0674
Need for social approval (1-7)	0.0376	0.0775	157.100	0.486	0.628	−0.1154	0.1906

** p<0.01, * p<0.05, + p<0.10

The substantively interesting coefficients have to do, however, with the item trait desirability and respondent’s need for social approval. The model results show the respondent’s need for social approval had no significant effect on response latencies. ⁵ Also, the coefficient for the squared trait desirability term was not significant meaning the expected inverted-U shape illustrated in figure 1 did not appear. The most interesting result has to do with the statistically significant negative coefficient for the original trait desirability variable. What is important to keep in mind in interpreting this result is that the trait desirability was measured on a bipolar scale from −4 (strongly undesirable) to +4 (strongly desirable). This means we can expect values in the positive range to become smaller: i.e. the more desirable an item is, the shorter the response latency. On the other hand, the more undesirable the item, the longer the response latencies become. This implies two completely separate effects depending on whether the trait is desirable or undesirable. Contrary to our expectation (shown in Figure 1) that extremely desirable and undesirable items should lead to the same result (shorter response latencies), we actually observe two separate patterns for desirable and undesirable items (see Figure 2).

OPEN IN VIEWER

Figure 2.

Response latency by trait desirability (all other effects held constant).

Conclusions and Discussion

The study attempted to explore the impact of item-based trait desirability and respondent-based need for social approval on response latencies. We attempted to apply a dual process theoretical approach to these ideas and argued that sensitive questions will frame the responses to come in the sense of a response set or “short-lived bias” as opposed to a response style (cf. Tourangeau and Yan, 2007; Paulhus, 2002; DeMaio, 1984). In this way, the fact that only the trait desirability (item-specific response effect) was statistically significant and not the respondent’s need for social approval (as a bias consistent across time and questionnaires, again see Tourangeau and Yan, 2007) is understandable. We assumed that within this frame, the mode of response will be determined by the strength of the social norm surrounding the behaviour or attitude.

Taking the work of Sudman and Bradburn (1974) as an example, the social norms of the items were measured externally in a separate study. We did not find the inverted U-shape that we had expected and the quadratic term included in the model was not statistically significant. Rather, our findings suggest that respondents react differently depending on whether the trait (item) was considered to be desirable or undesirable. The more desirable a trait was rated, the shorter the respondents took to answer the question. The more undesirable the trait, on the other hand, the longer they needed. Going back to Esser’s Model of Frame Selection for an explanation of these results, it would seem as if increasingly clear socially desirable norms lend themselves to an automatic norm-conform script. The interspersion of undesirable attitudes and behaviours interrupts the automatically activated frame “being a good teacher” and its associated script of response behaviour (responding in agreement), possibly forcing them into a controlled-deliberate response mode. Faced with a question of whether the respondent is now a “bad teacher”, a deliberate controlled response mode may be activated. In this case, the script does not match the new frame, increasing motivation to consider the response and find an appropriate script.

In terms of dual-process models (e.g. Fazio, 1990, see the discussion above), motivation and opportunity are the main determinants of cognitive processing modes: when both motivation and opportunity are given, information is processed in a deliberative mode. Thus, the undesirability of an item seems to enhance motivation for elaborative processing, while desirability decreases motivation. In this case, two effects may be responsible: firstly, the undesirable trait and the interruption of the automatic script of agreement may influence the respondent to confront and admit an uncomfortable truth about themselves, or secondly the respondent may want to mask the fact that he or she actually agrees to an undesirable item which, as Tourangeau and Rasinski (1988) state, tends to require more cognitive effort. In both cases, the interruption of the frame and its associated script leads to a higher level of motivation to consider the answer and, accordingly, longer response latencies.

In fact, a revised theoretical model which distinguishes between desirability and undesirability may actually better conform to the ongoing discussion about explanatory models of social desirability than as originally presented. As some have argued (for example Sudman and Bradburn, 1974; Paulhus, 2002), socially desirable responding is a strategy of dealing with sensitive questions. This would support our conception that the sensitivity of the question should trigger a socially desirable frame in the sense of Esser’s model of frame selection. Once acting on the cues of this socially desirable frame, socially desirable attitudes or behaviours can elicit two distinct response modes. The findings shed light on the importance of trait desirability, and by extension, item wording when using response latencies as a proxy for socially desirable responding. The difference between agreeing with the statement that “Spending time with teenagers is a lot of fun” and disagreeing with the statement “Teenagers tend to annoy me quickly” or another set of manifest indicators is evidenced by the diverging response latencies. We suggest that the conflicting results in the field of response latencies as proxies for socially desirable responding may in part be traceable to the different effects of socially desirable vs. undesirable item-traits.

This paper attempted to contribute to the still conflicted research involving the use of response latencies in identifying socially desirable responses. Our findings suggest that the direction of an item’s trait desirability plays a role in determining the response mode as evidenced by the significant negative effect coefficient of the trait desirable variable. The most undesirable traits lead to slower responses. With increasing desirability, the response latencies become shorter. On a bipolar scale this indicates diverging effects for desirable and undesirable traits.

This finding highlights our impression that the research field remains in need of a systematic evaluation as it is characterized by a lack of clear and intersubjective terminology. The difference between a sensitive question and a socially desirable question is not always clear (“Sensitive questions is a broad category that encompasses not only questions that trigger social desirability…” Tourangeau and Yan, 2007: 859); there are various instances in which either trait desirability, social desirable belief, social norm, and minority opinion were used seemingly interchangeably. For example, Stocké refers to trait desirability as the “relative social desirability of the attitude-position” (cf. Stocké, 2004: 310) making the differentiation between social desirability as a type of biasing response effect and as a characteristic of either or all of the respondent, item or interview situation a difficult proposition. When Tourangeau and Yan write: “This conception of sensitivity presupposes that there are clear social norms regarding a given behavior or attitude” (Tourangeau and Yan, 2007: 860), it would suggest only survey items with high trait desirability can be sensitive questions, to cite another example. Our argument is, however, that sensitivity acts as a frame, and that the direction and magnitude of an item’s trait desirability informs which script is carried out. Surely, a question dealing with such topics as homosexuality or refugees can be seen as sensitive even if the attitude or behaviour suggested in the survey item can vary between innocuous and extreme; socially desirable and undesirable.

Empirically, we see several areas in which operationalization should be investigated which may help untangle the various conflicting findings. Besides the direction of desirability, the ambiguity or rather clarity of the social norm is another indicator which may play a role in influencing the response mode of the respondent. This refers back to our original hypothesis expressed in figure 1 in which we expected ambiguous norms to cause longer responses. Here, an individual respondent-related measure of trait desirability as well as the standard deviation of the aggregate item value (as an indicator of the degree of consensus involving a trait) may be important to model. Such as of yet under-examined aspects may be partly responsible for the conflicting findings in which researchers have found evidence for social desirability leading to both shorter and longer latencies. In the article cited above by Silber et al. (2013), for example, the researchers created indices of both socially desirable and undesirable items of attitudes towards minority groups and reported in all but one of the eight models non-significant coefficients for the response latency variable. Taking our findings into consideration, it is plausible that this would be due to an “averaging-out” of effects. Based on our results, we would expect the item “Muslim culture fits well into our Western world” should illicit faster responses, while slower responses should be expected with a question such as “Immigration to Germany should be prohibited for Muslims” (cf. Silber et al., 2013: 132).

The next step of our research will involve attempting to recreate these results with a new sample. Furthermore, our findings did not support the initial hypothesized relation between trait desirability and response latencies. Further research will look at the causal mechanisms responsible for seemingly interrupting the script of automatic-spontaneous responses a-priori rather than offering post-hoc explanations for the conflicting results.

Another area which we wish to investigate involves looking at the actual item values entered by the respondents. In this way, we hope to establish not only that social desirability influences response latencies, but also to develop ways of using response latencies, together with other types of paradata (value changes, corrections of previous questions in order to preserve continuity and so on) to correct for socially desirable response bias. The difficulty of this goal involves establishing ways to actually identify lies and under/overestimations. In addition, further research on the use of response latencies in combination with item-count or randomized-response techniques is needed to get a deeper understanding of cognitive processes underlying social desirable response behaviour.