Wide availability of networked personal computers within organizations has enabled new methods for organizational research involving presentation of research stimuli using Web pages and browsers. The authors provide an overview of the technological challenges for collecting organizational data through this medium as a springboard to discuss the validity of such research and its ethical implications. A review of research comparing Web browser–based research with other administration modalities appears to warrant guarded optimism about the validity of these new methods. The complexity of the technology and researchers’ relative unfamiliarity with it have created a number of pitfalls that must be avoided to ensure ethical treatment of research participants. The authors highlight the need for an online research participants’ bill of rights and other structures to ensure successful and appropriate use of this promising new research medium.
Get full access to this article
View all access options for this article.
References
1.
Abbate, J. (1999). Inventing the Internet. Cambridge, MA: MIT Press.
2.
Aguinis, H., & Henle, C. A. (in press). Ethics in research. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology. Malden, MA: Blackwell.
3.
Allred, L. J. (1986). Sources of non-equivalence between computerized and conventional psychological tests. Unpublished doctoral dissertation, Johns Hopkins University.
4.
Anderson, S. E., & Gansneder, B. M. (1995). Using electronic mail surveys and computermonitored data for studying computer-mediated communication systems. Social Science Computer Review, 13, 33-46.
5.
Atchison, C. (1998). Men who buy sex: A preliminary description based on the results from a survey of the Internet-using population. Unpublished doctoral dissertation, Simon Fraser University, Burnaby, Canada.
6.
Azar, B. (2000). A Web experiment sampler. Monitor on Psychology, 31(4), 46-47.
7.
Bartel-Sheehan, K., & Grubbs-Hoy, M. (1999). Flaming, complaining, abstaining: How online users respond to privacy concerns. Journal of Advertising, 28(3), 37-51.
8.
Bertot, J. C., & McClure, C. R. (1996). Electronic surveys: Methodological implications for using the World Wide Web to collect survey data. Proceedings of the ASIS Annual Meeting, 33, 173-185.
9.
Birnbaum, M. H. (1999). Testing critical properties of decision making on the Internet. Psychological Science, 10, 399-407.
10.
Bosnjak, M. (2000, May). Participation in non-restricted Web surveys: A typology and explanatory model for non-response. Paper presented at the 55th annual conference of the American Association for Public Opinion Research, Portland, OR.
11.
Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, 125-144.
12.
Cho, H., & LaRose, R. (1999). Privacy issues in Internet surveys. Social Science Computer Review, 17, 421-434.
Davidson, S. (1999). From spam to stern: Advertising law and the Internet. In D. W. Schumann & E. Thorson (Eds.), Advertising and the World Wide Web (pp. 233-263). Mahwah, NJ: Lawrence Erlbaum.
15.
Gilmore, C., Kormann, D., & Rubin, A. D. (1999). Secure remote access to an internal Web server. IEEE Network, 13(6), 31-37.
Jones, S. G. (Ed.). (1999). Doing Internet research: Critical issues and methods for examining the Net. Thousand Oaks, CA: Sage.
18.
Kantor, J. (1991). The effects of computer administration and identification on the Job Descriptive Index (JDI). Journal of Business & Psychology, 5, 309-323.
19.
Kaye, B. K., & Johnson, T. J. (1999). Taming the cyber frontier: Techniques for improving online surveys. Social Science Computer Review, 17, 323-337.
20.
Kiesler, S., Siegel, J., & McGuire, T. (1984). Social psychological effects of computer-mediated communication. American Psychologist, 39, 1123-1134.
21.
Kiesler, S., & Sproull, L. S. (1986). Response effects in the electronic survey. Public Opinion Quarterly, 50, 402-413.
22.
King, N. (2000). What are they thinking? PC Magazine, 19(4), 163-178.
23.
Krantz, J. H., Ballard, J., & Scher, J. (1997). Comparing the results of laboratory and World-Wide Web samples on determinants of female attractiveness. Behavior Research Methods, Instruments, and Computers, 29, 264-269.
24.
Krantz, J. H., & Dalal, R. (2000). Validity of Web-based psychological research. In M. H. Birnbaum (Ed.), Psychological experimentation on the Internet (pp. 35-60). San Diego: Academic Press.
25.
Lautenschlager, G. J., & Flaherty, V. L. (1990). Computer administration of questions: More desirable or more social desirability? Journal of Applied Psychology, 75, 310-314.
26.
Lowman, R. L. (Ed.). (1998). The ethical practice of psychology in organizations. Washington, DC: American Psychological Association.
27.
Luong, A., & Rogelberg, S. G. (1998). How to increase your survey response rate. The Industrial-Organizational Psychologist, 36(1), 61-65.
28.
McBrien, B. D. (1984, November). The role of the personal computer in data collection, data analysis, and data presentation: A case study. Paper presented at the ESOMAR Conference, Nice, France.
29.
Michalak, E. E., & Szabo, A. (1998). Guidelines for Internet research: An update. European Psychologist, 3(1), 70-75.
30.
Morrow, R. H., & McKee, A. J. (1998). CGI scripts: A strategy for between-subjects experimental group assignment on the World-Wide Web. Behavior Research Methods, Instruments, and Computers, 30, 306-308.
31.
Musch, J., & Reips, U.-D. (2000). A brief history of Web experimenting. In M. H. Birnbaum (Ed.), Psychological experimentation on the Internet (pp. 61-87). San Diego, CA: Academic Press.
Oakes, W. (1972). External validity and the use of real people as subjects. American Psychologist, 27, 959-962.
34.
Ognianova, E. V. (1997). Audience processing of news and advertising in computer-mediated environments: Effects of the content providers perceived credibility and identity. Unpublished doctoral dissertation, University of Missouri, Columbia.
35.
Oliver, D. (1999). Sams teach yourself HTML 4 in 24 hours (4th ed.). Indianapolis, IN: Sams.
36.
Pasveer, K. A., & Ellard, J. H. (1998). The making of a personality inventory: Help from the WWW. Behavior Research Methods, Instruments, and Computers, 30, 309-313.
37.
Pettit, F. A. (1999). Exploring the use of the World Wide Web as a psychology data collection tool. Computers in Human Behavior, 15, 67-71.
38.
Rogelberg, S. G., & Luong, A. (1998). Non-response to mailed surveys: A review and guide. Current Directions in Psychological Science, 7, 60-65.
39.
Rogelberg, S. G., Luong, A., Sederburg, M. E., & Cristol, D. S. (2000). Employee attitude surveys: Exploring the attitudes of noncompliant employees. Journal of Applied Psychology, 85, 284-293.
40.
Rosenthal, R. (1991). Meta-analytic procedures for social research. Newbury Park, CA: Sage.
41.
Rosenthal, R. (1994). Science and ethics in conducting, analyzing, and reporting psychological research. Psychological Science, 5, 127-134.
42.
Schaefer, D. R., & Dillman, D. A. (1998). Development of a standard email methodology: Results of an experiment. Public Opinion Quarterly, 62, 378-397.
43.
Schillewaert, N., Langerak, F., & Duhamel, T. (1998). Non-probability sampling for WWW surveys: A comparison of methods. Journal of the Market Research Society, 40, 307-323.
44.
Schmidt, W. C. (1997). World-Wide Web survey research: Benefits, potential problems, and solutions. Behavior Research Methods, Instruments, & Computers, 29, 274-279.
45.
Schmidt, W. C. (2000). The server side of psychology Web experiments. In M. H. Birnbaum (Ed.), Psychological experimentation on the Internet (pp. 285-310). San Diego, CA: Academic Press.
46.
Schmidt, W. C., Hoffman, R., & MacDonald, J. (1997). Operate your own World-Wide Web server. Behavior Research Methods, Instruments, & Computers, 29, 189-193.
47.
Schwarz, N., Groves, R. M., & Schuman, H. (1998). Survey methods. In D. T. Gilbert, S. T. Fiske, & G. Lindzey (Eds.), The handbook of social psychology (Vol. 2, 4th ed., pp. 143-179). Boston: McGraw-Hill.
48.
Sieber, J. E. (1992). Planning ethically responsible research: A guide for students and internal review boards. Newbury Park, CA: Sage.
49.
Simsek, Z., & Veiga, J. F. (2000). The electronic survey technique: An integration and assessment. Organizational Research Methods, 3, 92-114.
50.
Smart, R. (1966). Subject selection bias in psychological research. Canadian Psychologist, 7, 115-121.
51.
Smith, M. A., & Leigh, B. (1997). Virtual subjects: Using the Internet as an alternative source of subjects and research environment. Behavior Research Methods, Instruments, & Computers, 29, 496-505.
52.
Sproull, L. S. (1985). Using electronic mail for data collection in organizational research. Academy of Management Journal, 29, 159-169.
53.
Stanton, J. M. (1998a). An empirical assessment of data collection using the Internet. Personnel Psychology, 51, 709-725.
54.
Stanton, J. M. (1998b). Validity and related issues in Web-based hiring. The Industrial-Organizational Psychologist, 36(3), 69-77.
55.
Supovitz, J. A. (1999). Surveying through cyberspace. American Journal of Evaluation, 20, 251-263.
56.
Swoboda, W. J., Muhlberger, N., Weitkunat, R., & Schneeweiss, S. (1997). Internet surveys by direct mailing: An innovative way of collecting data. Social Science Computer Review, 15, 242-255.
57.
Teo, T.S.H., Lim, V.K.G., & Lai, R.Y.C. (1997). Users and uses of the Internet: The case of Singapore. International Journal of Information Management, 17, 325-336.
58.
Thomas, B., Stamler, L. L., Lafreniere, K., & Dumala, D. (2000). The Internet: An effective tool for nursing research with women. Computers in Nursing, 18, 13-18.
59.
Thomas, J. (1996a). When cyber-research goes awry: The ethics of the Rimm “cyberporn” study. Information Society, 12(2), 189-197.
60.
Thomas, J. (1996b). Introduction: A debate about the ethics of fair practices for collecting social science data in cyberspace. Information Society, 12(2), 107-117.
61.
Tse, A.C.B. (1998). Comparing the response rate, response speed and response quality of two methods of sending questionnaires: E-mail vs. mail. Journal of the Market Research Society, 40, 353-361.
62.
Walsh, E. O. (1998, August). The value of the journalistic identity on the World Wide Web. Paper presented at the conference of the Association for Education in Journalism and Mass Communication, Baltimore.
63.
Walsh, E. O., McQuivey, J. L., & Wakeman, M. (1999, November). Consumers barely trust Net advertising. Cambridge, MA: Forrester Research.
64.
Witmer, D. F., Colman, R. W., & Katzman, S. L. (1999). From paper-and-pencil to screen- and- keyboard: Toward a methodology for survey research on the Internet. In S. G. Jones (Ed.), Doing Internet research: Critical issues and methods for examining the Net (pp. 145-162). Thousand Oaks, CA: Sage.
65.
Yost, P. R., & Homer, L. E. (1998, April). Electronic versus paper surveys: Does the medium affect the response? Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, Dallas, TX.
66.
Zickar, M. J. (2000). Modeling faking on personality tests. In D. R. Ilgen & C. L. Hulin (Eds.), Computational modeling of behavior in organizations: The third scientific discipline (pp. 95-113). Washington, DC: American Psychological Association.
