In this study, 5,257employees provided upward feedback ratings for 759 target managers who had the option of having their subordinates rate them using a traditional paper-and-pencil (opscan) response mode or using the company’s intranet. Preliminary analyses showed mean online ratings were more favorable than were mean paper-and-pencil ratings (d = .38). The authors then conducted rater- and ratee-level hierarchical regression analyses in which they controlled for rater and ratee characteristics. Results showed that (a) the association between response mode and upward feedback ratings was largely artifactual, with response mode accounting for 1% or less of the variance in upward feedback ratings after controlling for rater and ratee characteristics; (b) ratee characteristics accounted for much more of the variance in upward feedback ratings than did rater characteristics; and (c) response mode was unrelated to the number of missing responses.
Get full access to this article
View all access options for this article.
References
1.
American Educational Research Association, American Psychological Association, National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington DC: American Educational Research Association.
2.
Antonioni, D. (1994). The effects of feedback accountability on upward appraisal ratings. Personnel Psychology, 47, 349-356.
3.
Booth-Kewley, S., Edwards, J. E., & Rosenfeld, P. (1992). Impression management, social desirability, and computer administration of attitude questionnaires: Does the computer make a difference? Journal of Applied Psychology, 77, 562-566.
4.
Church, A. H. (2001). Is there a method to our madness? The impact of data collection methodology on organizational survey results. Personnel Psychology, 54, 937-969.
5.
Dillman, D. (2000). Mail and Internet surveys: The tailored design method (2nd ed.). New York: John Wiley.
6.
Donovan, M. A., Probst, T. M., & Nelson, N. (2001, April). Web-based attitude assessments: Does technology affect equivalence? Paper presented at the 16th annual conference of the Society for Industrial and Organizational Psychology, San Diego, CA.
7.
Fenlason, K. J. (2000, April). Multiple data collection methods in 360-feedback programs: Implication for use and interpretation. Paper presented at the 15th annual conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
8.
Hatten, J. T. (2000, April). An employee survey: Dual processes. Paper presented at the 15th annual conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
9.
Hezlett, S. A. (2000, April). Employee attitude surveys in multi-national organizations: An investigation of measurement equivalence. Presented at the 15th annual conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
10.
Kiesler, S., & Sproull, L. S. (1986). Response effects in the electronic survey. Public Opinion Quarterly, 50, 402-413.
11.
King, W. C., & Miles, E. W. (1995). A quasi-experimental assessment of the effect of computerizing noncognitive paper-and-pencil measurements: A test of measurement equivalence. Journal of Applied Psychology, 80, 643-651.
12.
Kraut, A. I. (1999). Organizational surveys: Coming of age for a new era. In A. I. Kraut & A. K. Korman (Eds.), Evolving practices in human resource management (pp. 302-327). San Francisco: Jossey-Bass.
13.
Kraut, A. I. (2001). An e-mail letter to a friend. The Industrial-Organizational Psychologist, 38(4), 37-39.
14.
Lautenschlager, G. J., & Flaherty, V. L. (1990). Computer administration of questions: More desirable or more social desirability? Journal of Applied Psychology, 75, 310-314.
15.
Magnan, S. M., Lundby, K. M., & Fenlason, K. J. (2000, April). Dual media: The art and science of paper and internet employee survey implementation. Paper presented at the 15th annual conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
16.
McFarland, L. A., Ryan, A. M., & Paul, K. B. (1998, April). Equivalence of an organizational attitude survey across administration modes. Paper presented at the 13th annual conference of the Society of Industrial and Organizational Psychology, Dallas, TX.
17.
Mehta, R., & Sivadas, E. (1995). Comparing response rates and response content in mail versus electronic mail surveys. Journal of Market Research Society, 37, 429-439.
18.
Rosenfeld, P., Booth-Kewley, S., Edwards, J., & Thomas, M. (1996). Responses on computer surveys: Impression management, social desirability, and the big brother syndrome. Computers in Human Behavior, 12, 263-274.
19.
Ryan, A. M., Chan, D., Ployhart, R., & Slade, L. A. (1999). Employee attitude surveys in a multinational organization: Considering language and culture in assessing measurement equivalence. Personnel Psychology, 52, 37-58.
20.
Schaefer, D., & Dillman, D. (1998). Development of a standard e-mail methodology: Results of an experiment. Public Opinion Quarterly, 62, 378-397.
21.
Spera, S. D. (2000, April). Transitioning to Web survey methods: Lessons from a cautious adopter. Paper presented at the 15th annual conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
22.
Spera, S. D., & Moye, N. A. (2001, April). Measurement equivalence between paper and Web survey methods in a multinational company. Paper presented at the 16th annual conference of the Society for Industrial and Organizational Psychology, San Diego, CA.
23.
Sproull, L. S. (1986). Using electronic mail for data collection in organizational research. Academy of Management Journal, 29, 159-169.
24.
Stanton, J. M. (1998). An empirical assessment of data collection using the Internet. Personnel Psychology, 51, 709-725.
25.
Tabachnick, B. G., & Fidell, L. S. (1983). Using multivariate statistics. New York: Harper & Row.
26.
Yost, P. R., & Homer, L. E. (1998, April). Electronic versus paper surveys: Does the medium affect the response? Paper presented at the 13th annual conference of the Society for Industrial and Organizational Psychology, Dallas, TX.
27.
Young, S. A., Daum, D. L., Robie, C., & Macey, W. H. (2000, April). Paper vs. web survey administration: Do different methods yield different results? Paper presented at the 15th annual conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
