Using Facebook to Supplement Participant Pools for Class Research Projects

Abstract

In-class research projects are a valuable way of providing research experience for undergraduate students in psychology. This article evaluates the use of online social networks to supplement sample recruitment for in-class research projects. Specifically, this article presents a systematic analysis of seven student research projects that recruited through social networks and a traditional participant pool. Data from these studies suggest that the social network and participant pool samples were very similar in participant characteristics and overall levels of the dependent measures. Similarly, the magnitude and direction of the effect sizes were very similar across the studies. Results suggest that online recruiting may be a viable way of supplementing sample sizes while also providing additional opportunities to address learning goals related to statistical analyses. However, the pedagogical benefits of increasing sample size through online recruiting must be considered in conjunction with the potential ethical and methodological limitations of recruiting through online social networks.

Keywords

research methods statistics social media

Students in high-quality psychology programs are expected to encounter research methodology throughout the curriculum (American Psychological Association [APA], 2011; Dunn et al., 2010). In-class research projects are one effective strategy for teaching and reinforcing important methodology skills (Vespia, Wilson-Doenges, Martin, & Radosevich, 2012). However, these types of projects frequently do not involve students in all phases of the research process (Perlman & McCann, 2005). Condensing the research process into a single semester is extremely difficult and may limit the range and complexity of student projects. In this article, I evaluate an approach to streamlining the data collection phase through social network recruiting.

In-class research projects typically rely on samples of convenience. In many cases, participants may be recruited through departmental participant pools. In addition to the substantive problems with the generalizability of university participant pool samples (Henrich, Heine, & Norenzayan, 2010), class projects may be limited by the size of the pool. Supplementing participant pool recruitment with social networking sites may provide a low-cost, time-efficient alternative for data collection. However, the potential benefit of this approach depends on the ways in which samples recruited online might differ from the standard participant pool. Prior research has documented that Facebook users may differ on a variety of dimensions, including demographic characteristics, motivations for use, and identity presentation (R. E. Wilson, Gosling, & Graham, 2012). In this study, I evaluate the use of Facebook recruiting as a supplement to traditional participant pool samples for in-class research projects. Specifically, I examine whether the two samples differed in participant characteristics, overall levels of the dependent variables, and the pattern of results (i.e., magnitude and direction of experimental effects).

Method

Courses

Data for this study came from two sections of an Advanced Research in Psychology course that I taught in consecutive semesters (Fall, 2012; Spring, 2012). The course meets 5 hr per week and is the required capstone experience for all majors. All sections of the course place a primary emphasis on the development of students’ knowledge and skills related to research methodology. Each section is typically “themed” around a specific topic area. The focus of my sections was health-related stigma, with an emphasis on stigma related to psychological disorders. All of the assigned readings, class assignments, and research projects were directly or indirectly related to health-related stigma.

Projects

A major component of the Advanced Research course is a semester-long empirical research project. Working in small groups (two to four students), students conceptualize, design, and collect data on a topic of their choice. The final product is a formal APA-style manuscript along with a 12-min oral presentation at a biannual department research conference. In total, there were nine group projects over the two sections. For the purposes of this article, I analyzed data from the seven projects that collected usable information on recruitment sources. Table 1 contains a brief overview of the projects with abbreviated information about the major variables included in the study. Because each group designed its own study, the range of demographic information and dependent measures varied from project to project. However, the thematic focus of the course resulted in a fair amount of overlap in the variables measured. For instance, several projects measured relevant constructs using adapted measures of stigmatizing attributions (Corrigan, Markowitz, Watson, Rowan, & Kubiak, 2003), social distance (Martin, Pescosolido, Olafsdottir, & Mcleod, 2007), and level of familiarity (Corrigan, Green, Lundin, Kubiak, & Penn, 2001).

Table 1.

Summary of Individual Studies and Key Variables Included in the Analysis.

Study Label	Project Description	Independent Variable(s)	Dependent Variable(s)
Study 1 (homeless)	Homelessness and mental illness stigma	Label (mentally Ill, homeless, mentally Ill and homeless, neither)	Stigmatizing attributions (anger, pity, help, danger, fear, avoidance, segregation, coercion, blame)
Study 2 (adoption)	Attitudes toward same sex parents and adoption	Parent sex status (heterosexual, gay male, lesbian)	Attitudes toward gay and lesbian parents (perceived effects on child adjustment)
Study 3 (schizophrenia)	Effects of antistigma video messages for schizophrenia	Antistigma message type (contact-based, educational, simulation)	Stigmatizing attributions (avoidance, danger)
Study 4 (cancer)	Effects of type of cancer and patient gender on stigma	Type of cancer (breast, lung, skin); patient gender (male, female)	Stigmatizing attributions (blame, pity, avoidance)
Study 5 (weight)	Labels and weight loss motivation on stigma	Label (fat, overweight); motivation (yes, no)	Social distance, stigmatizing attributions (blame)
Study 6 (family/autism)	Effect of diagnostic disclosure on stigma toward children with autism and their parents	Stigma target (child vs. parents), disclosure (yes, no)	Stigmatizing attributions (blame, avoidance, fear, pity, dangerousness, and help)
Study 7 (age/autism)	Effects of age and level of familiarity on stigma of autism	Age (child, adult), level of familiarity (low, high)	Stigmatizing attributions (pity, fear, social distance)

Recruitment Procedures

All projects received formal institutional review board (IRB) approval¹ prior to beginning data collection. Student researchers recruited participants for their studies through several sources. Groups began by recruiting through the departmental participant pool. However, our department limits the size of samples recruited through the participant pool (i.e., 60 participants per project) to balance available participants across all projects. In my classes, students were permitted to recruit beyond the participant pool provided they clearly indicated in their consent materials that participation for online samples does not count for course credit. Once they had collected data through the participant pool, student researchers began recruiting through social media. Each student in the group posted a Facebook “event” using the title of the study from the consent form and a link to the materials. All projects were completed entirely online. To identify the samples, each group included a “recruitment source” question at the end of their materials to identify whether participants came from the participant pool, Facebook, or some other source (e.g., referral from a friend).

Analysis

Because of the large number of variables involved in each study, I focused on a subset of the analyses conducted for each project. Specifically, I only evaluated the effects of the student researchers’ primary independent variable. I did not examine other independent variables, interaction effects, or potential moderators. I focused primarily on dependent measures that were central to the student researchers’ main hypotheses, with an emphasis on variables that were common to multiple studies (e.g., attributions of blame and pity).

At the end of the semester, students submitted their raw data along with their final papers. Using their data, I computed effect sizes (i.e., Cohen’s d) for their primary analyses by entering relevant statistical information into effect size calculators provided by Cumming (2011) and D. B. Wilson (2013). I have emphasized effect sizes and confidence intervals (95%) throughout the results. The limitations of null-hypothesis significance testing (NHST) are well documented (Kline, 2004). However, where it might be informative, I have also included standard NHST results.

Results

Sample Characteristics

A comparison of the age and gender distributions for the Facebook and participant pool samples is presented in Table 2. With regard to gender, there was little difference in the proportion of men to women in the two samples. Only two of the seven studies showed a significant difference in gender distribution, but those differences were in opposite directions. There were more consistent differences in the age of the two samples. As might be expected, Facebook samples tended to be older than those recruited through the participant pool, which consists primarily of traditional college-age students. One other background characteristic that might be important methodologically and conceptually is the participant’s level of familiarity with the target group (e.g., prior experience with Autism, etc.). Three projects used a formal measure of participants’ level of familiarity (Corrigan et al., 2001), which assigns participants a total score based on prior experience with the targeted population. The level of familiarity between the two samples did not differ significantly in any of these studies (all p’s > .05; all d’s < .20).

Table 2.

Demographic Characteristics of Online and Participant Pool Samples.

	Facebook			Participant Pool
Variable	%M	%F	N	%M	%F	N	p ^a
Gender^b
Study 1 (homeless)	21.0	79.0	143	44.7	55.3	38	.004
Study 2 (adoption)	27.5	72.5	51	28.0	72.0	50	1.000
Study 3 (schizophrenia)	23.8	76.2	21	46.8	53.2	47	.108
Study 4 (cancer)	25.0	75.0	44	8.0	92.0	50	.046
Study 5 (weight)	20.5	79.5	151	12.5	87.5	40	.270
Study 6 (family/autism)	29.4	70.6	17	20.9	79.1	43	.511
Study 7 (age/autism)	20.0	80.0	25	23.3	76.7	43	1.000
Age	M	SD	N	M	SD	N	t	p
Study 1 (homeless)	22.65	6.94	147	20.55	6.21	38	1.69	.092
Study 2 (adoption)	N/A^c	N/A	N/A	N/A	N/A	N/A	N/A	N/A
Study 3 (schizophrenia)	22.29	3.58	21	19.38	1.05	47	3.65^d	.001
Study 4 (cancer)	N/A^c	N/A	N/A	N/A	N/A	N/A	N/A	N/A
Study 5 (weight)	23.07	7.68	150	19.65	1.27	40	5.19^d	<.001
Study 6 (family/autism)	20.71	1.05	17	19.51	1.26	43	3.46	.001
Study 7 (age/autism)	22.36	7.33	25	19.73	3.72	41	1.93	.058

Note. N/A = not applicable.

^aTwo-tailed p value using Fisher’s exact test. ^bPercentage based only on participants who self-identified as male or female. ^cSurvey included categories for age only. ^dUsed separate variance estimate for t-test calculation.

Levels of Stigma

To assess differences in overall levels of stigma between the two samples, I plotted the mean differences for three stigmatizing attributions measured in several of the studies. Attributions were all measured using adaptations of Corrigan’s Attribution Questionnaire (Corrigan et al., 2003). Scores for each attribution were scaled to reflect the participant’s mean item score on a 7-point rating scale. Figure 1 shows a forest plot of mean differences in pity, blame, and avoidance. Errors bars reflect the 95% confidence interval for the difference in attribution means. Overall, there was little difference in the levels of each attribution across the two samples. There was a statistically significant difference between the two samples in only 1 of the 12 comparisons (i.e., Study 3, avoidance). The mean estimated effect size using a random effects model was very small (i.e., a mean difference of .049 on a 7-point scale) suggesting that the Facebook and participant pool samples reported very similar levels of these attributions.

Figure 1.

Forest plot of mean differences in attribution ratings between Facebook and participant pool samples.

Effect Sizes

A more direct test of the effects of supplementing samples through social media is the magnitude and direction of experimental effects. If experimental manipulations result in different effects (i.e., in size or direction), then student researchers’ conclusions become more complicated. For instance, null results for one of the subsamples may mask differences in the other. To analyze the relationship between recruitment source and effect sizes, I compared effect sizes for the student researchers’ primary experimental variables in the Facebook and participant pool samples. Figure 2 shows the effect sizes for the Facebook and participant pool samples side by side for multiple dependent measures from the seven studies included in this analysis. Across the multiple comparisons, there is substantial overlap and consistency in both magnitude and direction of the effects for the two samples, suggesting that the effects of experimental manipulations were comparable across recruiting sources.

Figure 2.

Comparison of effect sizes (with 95% confidence intervals) for Facebook and participant pool samples.

Discussion

The results of this analysis suggest that supplementing participant pool samples with recruitment through social networks is not likely to dramatically change the direction or size of experimental effects for class research projects. For the seven projects in my analysis, the sample characteristics, overall levels of stigma, and primary experimental effects were very similar regardless of the source of the sample. The similarity between online and participant pool samples is consistent with prior research (Smith & Leigh, 1997). Whether this similarity makes online recruiting more or less desirable may depend on the learning goals for the experience. On one hand, these data suggest that supplementing with online samples may be a simple way of boosting sample size quickly and easily. The ability to increase sample size using online sources may have several potential benefits to student projects. First, it may allow students to get adequate sample sizes more quickly, which may create additional time for literature review early in the semester or data analysis near the end of the semester. Second, the larger sample size may better support learning goals related to statistical analysis. Larger samples may provide sufficient power for additional analyses (e.g., analysis of potential moderators). Across the seven studies in my course, sample size increased by at least 40% with three of the samples more than doubling in size. In my experience with class projects, many students fail to find statistically significant effects and quickly conclude that their sample was not large enough, without consideration of effect size or direction. Having larger samples may eliminate the sample size limitation and challenge students to look for alternative, more conceptually important explanations for their null results.

The addition of a second recruiting source may also add interpretive complexity for more advanced students. Even though most of the results in the current analysis appear to be similar across samples, future students may find some important differences in their studies. These differences can lead to interesting questions about how the samples differ and why these differences might influence interpretation. This level of analysis supports higher level learning outcomes specified in the APA Guidelines for Undergraduate Majors in Psychology (APA, 2013). For instance, one of the baccalaureate indicators for the learning goal of conducting basic psychological research (2.4) is for students to evaluate the effectiveness of quantitative research methods and use quantitative analyses to evaluate hypotheses (APA, 2013). Providing sufficient data for students to explore a range of questions can allow students to evaluate the strengths and limitations of statistical analyses in addressing research questions.

Adding a second recruiting source may also increase some forms of diversity in the overall sample. Although most sample characteristics did not differ across sources in my students’ projects, there was a significant difference in age across the two samples. Depending on the nature of the study, variability in age might be very important. In the case of mental illness stigma, age and generational differences are not uncommon and may be important to examine (Crisp, Gelder, Rix, Meltzer, & Rowlands, 2000). However, if broader sample diversity is important for a study, the results from the current analysis suggest that recruiting through Facebook may not make a big difference over the participant pool. The two samples for the seven studies examined were similar in most respects. This homogeneity is not surprising, given that Facebook contacts are, to a large degree, self-selected.

Methodology Limitations

The results of this analysis are limited in several important ways that instructors should consider when deciding whether to supplement recruiting with social networks. First, the generalizability of these results may be limited because most of the experimental manipulations in the current analysis had relatively small effects (i.e., less than ½ SD; see Figure 2). In my experience, this is not unusual for class projects; students often must use experimental manipulations that are modest in scope. It is possible that sample differences may become more salient for more substantial effects. Also, each of the studies included in this analysis used relatively simple experimental designs (e.g., 2 × 2 factorial). Five of the studies used vignette-based approaches and the other two used brief exposure to videos as the experimental manipulation. Although these approaches are common in stigma research (Link, Yang, Phelan, & Collins, 2004), they are not necessarily representative of the range of projects that undergraduate students in psychology courses might explore. Finally, comparison of the background characteristics in this analysis was limited to a small number of variables that were common across studies (i.e., age, gender, and level of familiarity). Facebook and participant pool samples may differ on a range of other important variables (Ryan & Xenos, 2011; R. E. Wilson et al., 2012) that might influence the potential utility of recruiting through social networking sources.

Pedagogical Considerations

Traditionally, many psychology departments support faculty and student research through participant pools. Researchers using these pools must consider potential threats to validity including the timing of sign up (Aviv, Zelinski, Rallo, & Larsen, 2002), self-selection into studies (Jackson, Procidano, & Cohen, 1989), and participant pool contamination (Klein & Cheuvront, 1990). Recruiting participants through Facebook and other online sources is also likely to have significant limitations. For instance, Facebook participants are not compensated like those in participant pools. Moreover, they are known to the experimenters, which may increase the risk of response sets (e.g., socially desirable responding), the need for privacy precautions, and the potential for social coercion. Because recruiting through social networks like Facebook requires use of internet resources for data collection, instructors should supplement their coverage of research ethics with material specific to Internet research (e.g., Buchanan & Williams, 2010; Hoerger & Currell, 2012).

Decisions about sampling strategies provide opportunities for students to examine methodological trade-offs. Boosting sample size may improve statistical power, but may require more careful attention to issues of effect size and moderator variables. Recruiting through online samples may increase some forms of diversity but not others and it may present important ethical issues (e.g., privacy, social coercion). Therefore, when requiring students to collect data for class research projects, instructors will need to consider not just the feasibility of various sampling methods, but how these methods align with their learning goals.

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

Note

References

American Psychological Association. (2011). Principles for quality undergraduate education in psychology. Washington, DC: Author. Retrieved from http://www.apa.org/education/undergrad/principles.aspx

American Psychological Association. (2013). APA guidelines for the undergraduate psychology major: Version 2.0. Washington, DC: Author. Retrieved from www.apa.org/ed/precollege/undergrad/index.aspx

Aviv

A. L.

Zelinski

J. M.

Rallo

Larsen

R. J.

(2002). Who comes when: Personality differences in early and later participation in a university subject pool. Personality and Individual Differences, 33, 487–496. doi:10.1016/S0191-8869(01)00199-4

Buchanan

Williams

J. E.

(2010). Ethical issues in psychological research on the Internet. In Gosling

S. D.

Johnson

J. A.

(Eds.), Advanced methods for conducting online behavioral research (pp. 255–271). Washington, DC: American Psychological Association.

Corrigan

P. W.

Green

Lundin

Kubiak

M. A.

Penn

D. L.

(2001). Familiarity with and social distance from people who have serious mental illness. Psychiatric Services, 52, 953–958. doi:10.1176/appi.ps.52.7.953

Corrigan

P. W.

Markowitz

F. E.

Watson

Rowan

Kubiak

M. A.

(2003). An attribution model of public discrimination towards persons with mental illness. Journal of Health and Social Behavior, 44, 162–179. doi:10.2307/1519806

Crisp

A. H.

Gelder

M. G.

Rix

Meltzer

H. I.

Rowlands

O. J.

(2000). Stigmatisation of people with mental illnesses. The British Journal of Psychiatry, 177, 4–7. doi:10.1192/bjp.177.1.4

Cumming

(2011). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York, NY: Routledge.

Dunn

D. S.

Brewer

C. L.

Cautin

R. L.

Gurung

R. A. R.

Keith

K. D.

McGregor

L. N.

… Voigt

M. J.

(2010). The undergraduate psychology curriculum: Call for a core. In Halpern

D. F.

(Ed.), Undergraduate education in psychology: A blueprint for the future of the discipline (pp. 47–61). Washington, DC: American Psychological Association.

10.

Henrich

Heine

S. J.

Norenzayan

(2010). The weirdest people in the world? Behavioral and Brain Sciences, 33, 61–83. doi:10.1017/S0140525X0999152X

11.

Hoerger

Currell

(2012). Ethical issues in Internet research. In Knapp

S. J.

Gottlieb

M. C.

Handelsman

M. M.

VandeCreek

L. D.

(Eds.), APA handbook of ethics in psychology, Vol 2: Practice, teaching, and research (pp. 385–400). Washington, DC: American Psychological Association.

12.

Jackson

J. M.

Procidano

M. E.

Cohen

C. J.

(1989). Subject pool sign-up procedures: A threat to external validity. Social Behavior and Personality, 17, 29–43. doi:10.2224/sbp.1989.17.1.29

13.

Klein

Cheuvront

(1990). The subject-experimenter contract: A reexamination of subject pool contamination. Teaching of Psychology, 17, 166–169. doi:10.1207/s15328023top1703_6

14.

Kline

R. B.

(2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.

15.

Link

B. G.

Yang

L. H.

Phelan

J. C.

Collins

P. Y.

(2004). Measuring mental illness stigma. Schizophrenia Bulletin, 30, 511–541.

16.

Martin

J. K.

Pescosolido

B. A.

Olafsdottir

Mcleod

J. D.

(2007). The construction of fear: Americans’ preferences for social distance from children and adolescents with mental health problems. Journal of Health and Social Behavior, 48, 50–67. doi:10.1177/002214650704800104

17.

Perlman

McCann

L. I.

(2005). Undergraduate research experiences in psychology: A national study of courses and curricula. Teaching of Psychology, 32, 5–14. doi:10.1207/s15328023top3201_2

18.

Ryan

Xenos

(2011). Who uses Facebook? An investigation into the relationship between the Big Five, shyness, narcissism, loneliness, and Facebook usage. Computers in Human Behavior, 27, 1658–1664. doi:10.1016/j.chb.2011.02.004

19.

Smith

M. A.

Leigh

(1997). Virtual subjects: Using the Internet as an alternative source of subjects and research environment. Behavior Research Methods, Instruments & Computers, 29, 496–505. doi:10.3758/BF03210601

20.

Vespia

K. M.

Wilson-Doenges

Martin

R. C.

Radosevich

D. M.

(2012). Experiential learning. In Schwartz

B. M.

Gurung

R. A. R.

(Eds.), Evidence-based teaching for higher education (pp. 77–97). Washington, DC: American Psychological Association.

21.

Wilson

D. B.

(2013). Practical meta-analysis effect size calculator. The Campbell Collaboration. Retrieved February 11, 2014, from http://www.campbellcollaboration.org/resources/effect_size_input.php

22.

Wilson

R. E.

Gosling

S. D.

Graham

L. T.

(2012). A review of Facebook research in the social sciences. Perspectives on Psychological Science, 7, 203–220. doi:10.1177/1745691612442904