Personalized Feedback in Web Surveys

Abstract

Web surveys technically allow providing feedback to the respondents based on their previous responses. This personalized feedback may increase respondents’ motivation and possibly the accuracy of responses. While past studies mainly concentrate on the effects of providing study results on future response rates, thus far survey research lacks theoretical and empirical contributions on the effects of personalized, immediate, feedback on response behavior. To test this, we implemented a randomized trial in the context of the Berlin Aging Study II (BASE-II) in 2014, providing feedback regarding the respondents’ personality tests (Big Five Personality Inventory) to a subgroup of the sample. Results show only moderate differences in response behavior between experimental and control groups. However, we find that respondents who received personalized feedback report higher levels of satisfaction with the survey.

Keywords

personalized feedback web surveys online surveys measurement error survey satisfaction Big Five Personality Inventory

Introduction

Web surveys are increasingly replacing traditional modes of data collection, such as face-to-face interviewing. This holds true for Germany, the context under investigation, as well as in many other Western societies. Although web-based surveys still face serious methodological difficulties, for instance, with regard to selection bias (e.g., Schonlau, Van Soest, Kapteyn, & Couper, 2009), they also offer important advantages: Web surveys are usually less costly than either in-person interviews or mail surveys (Couper & Miller, 2008) and offer researchers a straightforward way to integrate complex and technically demanding survey instruments. For example, survey researchers have integrated videos (Fuchs & Funke, 2007) and interactive response options (e.g., sliders, see Buskirk, Saunders, & Michaud, 2015; Funke, 2016) in their online surveys.

Technological advancement has not only changed the ways in which we conduct surveys in the social sciences, but it has also paved new ways for citizens to study themselves. Movements such as the so-called quantified self (e.g., Swan, 2013) highlight the growing wish of many to learn something about themselves by self-monitoring personalized information. This article suggests that survey research should harness this intrinsic motivation of individuals to increase their self-knowledge by enhancing procedures to provide feedback from the collected personal survey data to the respondent. For instance, after collecting information about the respondent’s body weight and height, the web survey system can calculate and display the respondent’s body mass index. We argue that personalized feedback may motivate respondents to respond more accurately in surveys. Despite those possible benefits, very few studies have actually provided this type of feedback in their web survey applications, likely in part because of concerns it may negatively affect measurement. So far, there is almost no research on the potential benefits and costs of providing personalized feedback.

We seek to address this gap in the literature by investigating the potential advantages and disadvantages of providing personalized feedback within an online survey. Using a randomized trial, we assess (1) whether (the advance notice of) the feedback decreases undesired response behavior, such as item nonresponse, response styles, low reliability, socially desirable responding, or corrective answers and (2) whether the feedback affects respondent satisfaction with the survey.

Background and State of Research

Feedback in Surveys

Providing feedback to respondents and study participants is common practice in various research disciplines, including medicine, psychology, and sociology. Four main dimensions of feedback can be separated: scope, purpose, timing, and form.

Scope

The scope of the feedback relates to the level of aggregation of information provided. While many studies report sample-based findings to respondents (i.e., overall study results), other studies provide individual feedback to respondents (i.e., individual health status). We define personalized feedback as the provision of information to respondents based on their individual responses during the interview.

Purpose

Reasons for researchers to offer feedback are quite divergent. First, feedback has been used to influence respondent and participant behavior. This type of feedback is usually found in health-related or medical intervention studies (see DiClemente et al., 2001). For instance, Larimer et al. (2007) test whether personalized feedback based on responses about individual drinking behavior affects future drinking behavior. Second, feedback has been provided for research-ethical considerations. For instance, in the context of studies including medical checks, researchers are ethically, if not legally, obligated to provide individual feedback to participants. Third, feedback has been used to influence respondent’s survey participation behavior. Especially in the context of panel surveys, such as the Socio-Economic Panel Study (SOEP; see Wagner, Frick, & Schupp, 2007), study results and selected publications are regularly provided to respondents. Here, the feedback is used to increase the respondent’s engagement with the study and to maintain the respondent’s willingness to participate in (future) surveys and waves. Fourth, especially in web surveys, feedback is used as a type of technical assistance in order to increase the quality of responses. Researchers may display previous answers in targeted follow-up questions (a type of “dependent interviewing”; e.g., Jäckle & Lynn, 2007), include consistency checks, or provide interactive, real-time feedback on responses (e.g., Conrad, Couper, Tourangeau, & Galesic, 2005) in order to reduce the cognitive burden of answering to complex survey questions. Finally, feedback may be used to increase the respondents’ motivation to provide accurate and thoughtful answers during a single web survey (Göritz & Luthe, 2013).

Timing

Feedback varies with respect to the timing when it is provided to respondents. In the case of general feedback based on overall study results, the feedback is often sent to respondents via mail or e-mail weeks after the interview has been completed. In the case of web surveys and other forms of computer-assisted interviewing, feedback may be provided at the end of, or even during, the interview.

Form

The information constituting the feedback may be presented in different ways such as text, tables, and graphs.

Effects of Personalized Feedback on Respondent behavior in Web Surveys

Even though providing general feedback on study results to respondents is common practice in many ongoing panel surveys (e.g., German Internet Panel [GIP]: Blom, Gathmann, & Krieger, 2015; Longitudinal Internet Studies for the Social sciences [LISS]: Scherpenzeel & Toepoel, 2014; SOEP: Wagner et al., 2007), only few studies have experimentally tested the effects of either personalized or general feedback on participation rates and response behavior. This seems especially surprising in the context of web surveys, since the implementation of personalized feedback is quite a simple matter from a technical point of view.

A number of studies focus on the effects of providing personalized feedback on participation in panel surveys. Marcus, Bosnjak, Lindner, Pilischenko, and Schütz (2007) find that for individuals who are not especially interested in the survey topic, personalized feedback provided after an initial web survey increases response rates in follow-up online surveys. This is in line with Bälter, Fondell, and Bälter (2011), who observed higher response rates in follow-up surveys for respondents provided with personalized feedback on responses regarding energy expenditure during the first interview.

As to the effects of offering general, nonpersonalized feedback, Scherpenzeel and Toepoel (2014) find no effects on future participation when online panel respondents were provided with real-time answer distributions at the end of a web questionnaire.

Furthermore, two studies focus on the effects on participation rates when offering feedback in e-mail invitations to web surveys. Göritz and Luthe (2013) found no effect of offering general study results on response rates. Similarly, results by Angelovska and Mavrikiou (2013) revealed no effect of offering personalized feedback neither.

To our knowledge, only one study examines effects of feedback on actual response behavior and data quality: Göritz and Luthe (2013) study the effect on retention (noncompletion) rates and did not find evidence of a positive effect of offering general study results on completion rates in an online panel. Moreover, the authors find no significant difference in responses when analyzing the nondifferentiation of answers (straightlining behavior) between respondents who were offered feedback and those who were not.

Hence, no known study looks at the effects of personalized feedback on response behavior and data quality during an interview. We try to fill this gap by investigating effects of immediate (within-survey), personalized, feedback on response behavior and data quality in web surveys. We augment the list of indicators of response behaviors with item nonresponse rates, response times, internal consistency of answers, socially desirable responding, and corrective answers. Moreover, we evaluate the effect of personalized feedback on respondents’ satisfaction with the survey.

Theoretical Background and Hypotheses

One crucial aspect when conducting surveys is to motivate respondents to provide accurate and thoughtful responses in order to maximize reliability and validity. It is well documented, however, that response behaviors do not always live up to this ideal and that errors occur throughout the response process, starting from the understanding of a survey question and ending with selecting an answer (Tourangeau, Rips, & Rasinski, 2000).

In particular, satisficing, also known as “shortcutting” or just providing a satisfactory rather than an optimal answer, is associated with a set of response behaviors that are said to be closely linked to the motivation of respondents. According to Krosnick (1991), satisficing occurs in situations when the burden and complexity of survey questions exceeds the cognitive effort and motivation of respondents. Satisficing may take effect through different response strategies (see Krosnick, 1991, p. 215): Respondents may simply choose the first answer option offered in a question (acquiescence).¹ Some may choose to refuse an answer or respond with “don’t know” rather than giving a substantial answer (item nonresponse). When answering multiple items, they may choose the same answer over and over again (straightlining). In addition, respondents may speed through the questionnaire rather than carefully reading and thinking about questions and answer options (response time). Finally, satisficing may also result in haphazard misreporting and, therefore, lower reliability of answers (internal consistency).

Hypothesis 1: Respondents interested in informative, personalized feedback also need to provide accurate answers. As an advance notice of upcoming personalized feedback of test results may increase the motivation of respondents, we expect that it also reduces item nonresponse, straightlining, speedy answering, and increases internal consistency of answers.

Socially desirable responding refers to the tendency of respondents to adjust their answers toward social norms in order to give positive self-descriptions and appear in a good light (see Krumpal, 2013, for a recent overview). In the context of web surveys, social desirability bias is likely to be due to the self-deceptive component of the concept (see Paulhus, 1991): Some respondents may tend to adjust their answers in a more positive and desirable direction in order to feel better about themselves.

Offering personalized feedback may affect the occurrence of socially desirable responses in two contradictory ways. On the one hand, an advance notification of the feedback is expected to increase the respondent’s motivation to answer accurately in order to receive meaningful feedback.

Hypothesis 2a: The anticipation of personalized feedback is expected to decrease the respondents’ tendency to choose answer options that are socially desirable.

On the other hand, the anticipation of the feedback may also provoke self-deception tendencies in some respondents since he or she is facing an imminent situation (the upcoming feedback), either making him or her feel good or bad. Negative feedback may even result in respondents reaccessing and manipulating their initial responses to obtain the desired self-description.

Hypothesis 2b: Personalized feedback increases subsequent manipulations of answers after the feedback is displayed.

Above and beyond response behavior, as personalized feedback is likely to be perceived as interesting and novel, respondents may experience the questionnaire more positively, which in turn may have a positive effect on subsequent panel participation.

Hypothesis 3: Personalized feedback increases respondents’ positive evaluation of the survey experience.

Data and Method

The Berlin Aging Study II (BASE II)

We test the effects of personalized feedback on response behavior in a randomized trial implemented in a web survey of the BASE II (Bertram et al., 2014). BASE II is a longitudinal study that was launched in 2008 that covers 1,600 elderly respondents (most between the ages of 60 and 80) and a reference group of 600 young respondents (most of them between the ages of 20 and 35). The BASE-II study focuses on healthy aging, with respondents participating in regular medical checkups as well as mental and motoric testing at centralized test sites. In 2008, 2009, 2012, and 2014, participants and their household members received an additional 45-min individual questionnaire containing questions on their economic situation, social relations, and biographical information. The experiment on personalized feedback was implemented in the 2014 wave. The experimental sample consisted of 843 individuals (48% female; 58% secondary/tertiary education; 70 years on average in the elderly group [median = 71] and 32 years on average in the reference group [median = 31]).

Experimental Setup and Feedback on Personality

We implemented personalized feedback using a 15-item version of the well-established Big Five Personality Inventory (Gerlitz & Schupp, 2005; McCrae & Costa Jr., 1987). The inventory was used to calculate personality profiles that we provided to respondents once they finished the personality testing. We expect that the information on personality traits is intriguing for many respondents: While this information is most likely to be perceived as interesting, respondents usually have only very limited information on their performance and individual scores in standardized, scientific personality tests.

All respondents were randomly allocated to a treatment group (feedback, n = 439) and a control group (no feedback, n = 404; see Table 1). Right before answering the personality inventory, the treatment group received advanced notice informing them that they would receive personalized feedback regarding their personality and would be able to compare their own personality profile against German population averages (Figure A1 in the Appendix). After answering the questions (Figure A2 in the Appendix), respondents were informed that they had just completed a well-established scientific survey instrument (Figure A3 in the Appendix). The personalized feedback was displayed immediately after this additional information page. We made use of visual representations for each of the five personality dimensions using scales with verbal labels at both ends of the scale (Figure A4 in the Appendix). Individual scores were displayed by using red dots placed on the respective scales. In addition, a population average derived from the SOEP (Wagner et al., 2007) was displayed by using blue dots on the scales. Finally, at the end of the questionnaire, respondents were asked to evaluate and report their satisfaction with the survey based on evaluation items (Figure A5 in the Appendix).

Table 1.

Experimental Setup: Big Five Personality Inventory.

Experimental Group	Advance Notice Feedback	Big Five Inventory	Personalized Feedback	Evaluation Questions
Treatment (n = 439)	✓	✓	✓	✓
Control (n = 404)	✖	✓	✖	✓

Data Analysis

In order to investigate whether (the advance notice of) personalized feedback affects response behavior as well as satisfaction with the survey, we made use of t-tests on differences between both experimental groups. More specifically, we compared (a) item nonresponse, (b) straightlining,² (c) response time, and (d) internal consistency as indicators of satisficing. As indicators of social desirability, we compare the means of answers on the items of the personality test in light of previous research on the susceptibility of these items toward social desirability bias. A second indicator is the tendencies of subsequent adjustments of answers measured by paradata. Furthermore, we test the effect of personalized feedback on respondents’ evaluation of the survey by performing t-tests on 4 evaluation items. Finally, multivariate linear regression models establish the robustness of results and address possible group-specific effects.

Results

Effects of the Advance Notice of Feedback on Response Behavior

Performing various comparisons of the measures between experimental groups, we do not find effects of the advance notice of upcoming personalized feedback on response behavior on the Big Five Inventory.

First, item nonresponse rates were comparatively low in both groups. Only seven individuals in the feedback group (1.6%) and eight individuals in the control group (2.0%) were associated with a missing value in at least one of the 15 Big Five Inventory items (t = −.42, p = .67). Second, the experimental groups did not differ with respect to their tendency to apply straightlining response behavior (feedback: 11.6%, SE = .02, no feedback: 9.0%, SE = .2; t = 1.28, p = .20) on the inventory. Third, no differences were observed with respect to average response times (feedback: 87.1 s, SE = 1.61, no feedback: 87.8, SE = 1.77; t = .31, p = .76; the top 2% were excluded from this analysis). Fourth, we compared levels of internal consistency (reliability) of responses between the groups. Table 2 displays Cronbach’s αs for each of the Big Five dimensions across experimental groups. We applied a bootstrapping approach and compared 95% confidence intervals. No significant differences were observed for any of the five dimensions.

Table 2.

Internal Consistency of Responses.

	Cronbach’s α		Δ^a
Big Five Scale	Advance Notice	No Advance Notice
Extroversion	.66	.70	–.05 (ns)
Agreeableness	.40	.39	.01 (ns)
Conscientiousness	.58	.63	–.05 (ns)
Neuroticism	.67	.68	–.01 (ns)
Openness	.64	.69	–.05 (ns)

^a95% CI overlap, bootstrapping, 1,000 replications.

Testing whether the advance notice of personalized feedback affects socially desirable responding, Table 3 reports mean scores across the five dimensions of the Big Five Inventory. We find differences across the experimental groups in the case of agreeableness (p < .05). Respondents receiving notice of the upcoming feedback report lower levels of agreeableness. While this finding tentatively suggests more honest answers in the feedback group, we do not find evidence for an effect on social desirability bias in the four other dimensions.

Table 3.

Effects of Advance Notice on Substantial Responses in Big Five Inventory.

	Mean Scale Score
	Advance Notice	No Advance Notice	t-Test
Big Five Scale	$\bar{X}$ (SE)	$\bar{X}$ (SE)	FB-noFB
Extroversion	4.84 (.06)	4.77 (.06)	0.94
Agreeableness	5.11 (.05)	5.24 (.05)	2.02**
Conscientiousness	5.51 (.05)	5.53 (.05)	–0.27
Neuroticism	3.64 (.06)	3.50 (.06)	–1.58
Openness	5.08 (.06)	5.07 (.06)	0.10

*p < .10. **p < .05. ***p < .01.

Paradata on mouse clicks (responses and buttons) allowed us to investigate whether respondents—after receiving their personalized feedback—reaccessed the questions and adjust their initial responses. Only two individuals (.24%) made use of the back button to revisit the personality questions. One of these two individuals modified 2 of the 15 items. Thus, the provision of feedback did not lead respondents to alter their initial answers.

Personalized Feedback and Survey Evaluation

We included a set of survey evaluation questions at the end of the questionnaire. Four of these items relate to the respondents’ enjoyment and satisfaction with the survey. Table 4 displays mean values across both experimental groups along with results of the corresponding t-tests.

Table 4.

Satisfaction With the Survey.

	Feedback	No Feedback	t-Test
Item	$\bar{X}$ (SE)	$\bar{X}$ (SE)	FB-noFB
Survey was fun	5.19 (.07)	5.00 (.07)	1.88*
Learned something (about me)	4.14 (.09)	3.60 (.09)	4.30***
Topics/contents boring (–)	2.67 (.07)	2.85 (.08)	–1.70*
Contribution to science	5.27 (.07)	5.30 (.08)	–0.24

*p < .10. **p < .05. ***p < .01.

We observe a significant difference between the groups for 3 of the 4 items. Respondents who received personalized feedback evaluated the survey as more fun (t = 1.88, p < .10), less boring (t = −1.70, p < .10), and were more likely to report that they learned something about themselves (t = 4.30, p < .01). No significant effect was observed—as one may expect—for the subjective contribution to science.

Individual Characteristics of the Feedback and Survey Evaluation

It seems plausible that respondents who perceive their feedback as flattering or pleasant may evaluate the survey as more enjoyable. Since each individual feedback on personality scores was supplemented with a population average, respondents were able to compare their own scores with a benchmark. Moreover, the Big Five scales reflect a more or less (socially) desirable and undesirable range (Paulhus, 2002). Thus, we tested whether differences between individuals’ scale scores compared to (1) the population averages displayed and (2) the most desirable end (maximum/minimum) of the scale affected survey evaluation in the group of respondents receiving personalized feedback.

Table 5 displays the results of multiple multivariate linear regressions. First (Model 1), each of the 4 evaluation items were separately regressed on the average difference between individual scores and population averages displayed across all five personality dimensions. In the second set of models (Model 2), each of the 4 evaluation items were regressed on the average difference between individual scores and the more desirable scale ends across all five dimensions. The respondents’ personality scores, gender, age (three groups), and education (high school vs. rest) were included as controls. In both sets of models, we observe a significant (and quite plausible) effect for 1 of the 4 evaluation items: Respondents who experienced larger differences between their individual scores and (1) the population averages and (2) the most desirable end of the scale were less likely to report that the survey was boring.

Table 5.

Individual Feedback Characteristics and Survey Satisfaction: Multivariate Linear Regressions.

		Evaluation Items
		1	2	3	4
		Fun	Learned	Boring	Science
Model	Variable	b (SE)	b (SE)	b (SE)	b (SE)
1	Mean difference to population	–0.19 (0.20)	–0.13 (0.26)	–0.70*** (0.21)	–018 (0.23)
	Average
2	Mean difference to desirable scale ends	–0.22 (0.29)	–0.17 (0.38)	–1.02*** (0.31)	–0.20 (0.33)
	N	431	430	429	429

Note. Regression coefficients (b) and standard errors (SE) are given in parentheses; Controls: gender, age, education, and Big Five Personality Scale scores.

*p < .10. **p < .05. ***p < .01.

Summary and Discussion

In this article, we investigated whether personalized feedback on personality traits affects response behavior and respondents’ satisfaction with a web survey. Based on a randomized trial, potential effects were examined using a variety of indicators of response behaviors and different domains of enjoyment with the survey.

We found only minor effects of the advance notification of feedback on responses to the Big Five Personality Inventory. Thus, contrary to what was expected, the results do not point to an increase in data quality through the announcement of upcoming feedback. Fortunately, we also do not find evidence for manipulations and adjustments of answers after the feedback was presented to respondents. Finally, we observe a positive effect of feedback on respondent satisfaction with the survey. Respondents who received personalized feedback were more likely to rate the survey as interesting and fun, less boring, and reported more frequently that they had learned something about themselves. Thus, despite the lack of evidence for beneficial effects on response quality, offering personalized feedback may affect participation in future waves as the survey is perceived as more enjoyable and fun. The individual characteristics of the feedback displayed had only minor effects on the reported enjoyment with the survey.

Even though we implemented a randomized trial, our study faces some limitations concerning the generalizability of the results. First, the study population consists of a self-recruited sample of residents of the larger metropolitan area of Berlin and all respondents had already participated in two earlier waves that included both medical checks and personal interviews. Thus, it seems plausible that our web survey respondents were generally highly motivated to take part in the survey. As a consequence, personalized feedback may have had only a little effect on responses since the motivation to answer accurately and thoughtfully may have been already comparatively high. Moreover, respondents were already used to receive feedback during medical examinations and checkups in previous waves. Second, we only implemented feedback for personality traits. Even though we think that feedback on personality traits is likely to be perceived as novel and important to many, other topics and ways to present the feedback may reveal stronger effects on response behavior and survey evaluation.

Interesting fields for further research exist, especially within the context of panel studies. On the one hand, personalized feedback may increase not only satisfaction with the current interview but also participation rates in follow-up waves. At the same time, though, personalized feedback in one wave may introduce forms of reactivity and panel conditioning in later waves of a longitudinal survey. Moreover, future studies may investigate and compare different types of feedback, covering multiple topics and feedback characteristics as well scope and form of the feedback.

Footnotes

Appendix

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Angelovska

Mavrikiou

P. M.

(2013, 7). Can creative web survey questionnaire design improve the response quality? (Working Paper 131). The Netherlands: Amsterdam Institute for Advanced Labour Studies, University of Amsterdam.

Bälter

Fondell

Bälter

(2011). Feedback in web-based questionnaires as incentive to increase compliance in studies on lifestyle factors. Public Health Nutrition, 15, 982–988.

Bertram

Böckenhoff

Demuth

Düzel

Eckardt

Lil

S.-C.

… Steinhagen-Thiessen

(2014). Cohort profile: The Berlin Aging Study II (BASE-II). International Journal of Epidemiology, 43, 703–712.

Blom

A. G.

Gathmann

Krieger

(2015). Setting up an online panel representative of the general population: The German Internet panel. Field Methods, 27, 391–408.

Buskirk

T. D.

Saunders

Michaud

(2015). Are sliders too slick for surveys? An experiment comparing slider and radio button scales for smartphone, tablet and computer based surveys. Methods, Data, Analyses, 9, 229–260.

Conrad

F. G.

Couper

M. P.

Tourangeau

Galesic

(2005). Interactive feedback can improve the quality of responses in web surveys. Paper presented at the 60th annual conference of the American Association for Public Opinion Research, May 12–15, 2005, Miami Beach, FL. Retrieved from http://www.amstat.org/sections/srms/Proceedings/y2005/Files/JSM2005-000938.pdf

Couper

M. P.

Miller

P. V.

(2008). Web survey methods. Introduction. Public Opinion Quarterly, 72, 831–835.

DiClemente

Mirinilli

Singh

Bellino

(2001). The role of feedback in the process of health behavior change. American Journal of Health Behavior, 11, 217–227.

Fuchs

Funke

(2007). Video web survey: Results of an experiment comparison with a text-based web-survey. Paper presented at the Association for Survey Computing’s fifth international conference on the impact of technology on the survey process, September 2007, Southampton, England.

10.

Funke

(2016). A web experiment showing negative effects of slider scales compared to visual analogue scales and radio button scales. Social Science Computer Review, 34, 244–254.

11.

Gerlitz

J.-Y.

Schupp

(2005). Zur Erhebung der Big-Five-basierten Persnlichkeitsmerkmale im SOEP. Dokumentation der Instrumententwicklung BFI-S auf Basis des SOEP Pretests 2005 (Research notes 4). DIW Berlin, Germany: German Institute for Economic Research.

12.

Göritz

A. S.

Luthe

S. C.

(2013). How do lotteries and study results influence response behavior in online panels? Social Science Computer Review, 3, 371–385.

13.

Jäckle

Lynn

(2007). Dependent interviewing and seam effects in work history data. Journal of Official Statistics, 23, 529–551.

14.

Krosnick

(1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.

15.

Krumpal

(2013). Determinants of social desirability bias in sensitive surveys: A literature review. Quality & Quantity, 47, 2025–2047.

16.

Larimer

Lee

Kilmer

Fabiano

Stark

Geisner

… Neighbors

. (2007). Personalized mailed feedback for college drinking prevention: A randomized clinical trial. Journal of Consulting and Clinical Psychology, 72, 285–293.

17.

Marcus

Bosnjak

Lindner

Pilischenko

Schütz

(2007). Compensating for low topic interest and long surveys. A field experiment on nonresponse in web surveys. Social Science Computer Review, 25, 372–383.

18.

McCrae

Costa

Jr. (1987). Validation of the five-factor model of personality across instruments and observers. Journal of Personality and Social Psychology, 52, 81–90.

19.

Paulhus

(1991). Measurement and control of response bias. In Robinson

Shaver

Wrightman

(Eds.), Measures of personality and social psychological attitudes (pp. 17–59). New York, NY: Academic Press.

20.

Paulhus

(2002). Socially desirable responding: The evolution of a constructs. In Braun

Jackson

Wiley

(Eds.), The Role of constructs in psychological and educational measurement (pp. 49–69). Mahwah, NJ: Lawrence Erlbaum.

21.

Scherpenzeel

Toepoel

(2014). Informing panel members about study results: Effects of traditional and innovative forms of feedback on participation. In Callegaro

Baker

Bethlehem

Gritz

A. S.

Krosnick

J. A.

Lavrakas

P. J.

(Eds.), Online panel research: A data quality perspective (pp. 192–214). New York, NY: Wiley.

22.

Schonlau

Van Soest

Kapteyn

Couper

(2009). Selection bias in web surveys and the use of propensity scores. Sociological Methods & Research, 37, 291–318.

23.

Swan

(2013). The quantified self: Fundamental disruption in big data science and biological discovery. Big Data, 1, 85–99.

24.

Tourangeau

Rips

L. J.

Rasinski

(2000). The psychology of survey response. Cambridge, England: Cambridge University Press.

25.

Wagner

Frick

Schupp

(2007). The German Socio-Economic Panel Study (SOEP) – Scope, evolution and enhancements. Schmollers Jahrbuch, 127, 139–169.