Fast,Free,and Targeted

Abstract

Recruiting participants is a necessary step in many studies. With the advent of online research techniques, scientists are looking for new places where participants can be recruited online, in order to overcome the limitations of current sources and avoid the issues associated with sample overuse. The social media website “Reddit” is a potential source for recruitment, as it allows for free and rapid data collection from large samples, while enabling researchers to target specific populations when needed. The ability to recruit for free is especially important because it enables students and early career researchers, for whom even low recruitment costs can be prohibitive, to benefit from the opportunity of conducting research that they otherwise would not be able to. The current article therefore aims to bring this prospective, untapped resource to the attention of the research community. The article discusses current online recruitment sources and their limitations, provides an overview of Reddit, validates its use for research purposes, examines participation data from previous studies which recruited through Reddit, highlights its advantages and limitations as a participant pool, and suggests guidelines that can improve recruitment and retention rates for scientists looking to use Reddit for their research.

Keywords

computer-assisted survey research online participant recruitment online surveys Reddit Amazon Mechanical Turk (MTurk)

Introduction

Recruiting participants is an integral part of many scientific studies. Currently, online recruiting and testing is becoming commonplace, with several analyses showing that surveys conducted over the Internet provide results that are as valid as more “traditional” methods (Birnbaum, 2004; Evans & Mathur, 2005; Gosling, Vazire, Srivastava, & John, 2000; Miller & Dickson, 2001; Wilson & Dewaele, 2010; Woo, Kim, & Couper, 2015). Furthermore, in order to ensure the validity of studies using online surveys, new protocols and tools have been developed by researchers, such as the checklist for reporting results of Internet e-surveys developed by Eysenbach (2004), and Couper’s (2008) book Designing Effective Web Surveys. Due to the validation and subsequent proliferation of online research methods, researchers are now looking for new online venues where they can recruit participants (Buhrmester, Kwang, & Gosling, 2011; Lane, Armin, & Gordon, 2015; Morgan, Jorm, & Mackinnon, 2013; Pedersen & Nielsen, 2016).

There are a number of websites which researchers tend to use as their go-to place for online recruitment and testing. The most prominent is Amazon Mechanical Turk (MTurk), a crowdsourcing Internet marketplace, where workers receive wages in return for performing tasks online (Berinsky, Huber, & Lenz, 2012; Buhrmester et al., 2011; Casler, Bickel, & Hackett, 2013; Paolacci, Chandler, & Ipeirotis, 2010). Other often-used venues include social media sites such as Facebook and Twitter, where researchers either advertise their study for a fee via the sites’ advertising platforms, or use their social connections in order to reach participants (Casler et al., 2013; Johnson, Mueller, Williams, & Gutmann, 2014; Kosinski, Matz, Gosling, Popov, & Stillwell, 2015). Additional viable source are Google AdWords, an advertisement-based service; online forums related to the topic at hand; online community noticeboards, such as Craigslist; e-mail lists; and more (Johnson et al., 2014; Kosinski et al., 2015; Morgan et al., 2013). Each of these sources has its limitations, and the effectiveness of each recruitment method must be weighed according to the particular demands of each study. MTurk, for example, may not be efficacious for researchers looking to target certain demographic groups or special interest groups which do not have a strong presence on the site (Samuels & Zucco, 2013). Advertisements on Facebook and Google on the other hand are often not cost effective, and researchers are not always able to obtain sufficiently large samples when using them (Morgan et al., 2013; Valdez et al., 2014).

There are also some more general issues with existing recruitment methods. In pay-for-participation platforms such as MTurk, for instance, researchers often find it necessary to exclude a large number of workers who provide questionable data, as shown by Chandler, Mueller, and Paolacci (2014). In their analysis, they found that 33.3% of studies which used MTurk to recruit participants reported excluding workers based on the quality of their responses, and that excluded workers accounted for an average of 15% of the studies’ samples (range 3–37%). Furthermore, the analysis by Chandler et al. (2014) shows that this represents only a portion of the studies which excluded workers, and that there are likely additional studies that did so without reporting it. Although there are a number of methods which can be used in order to cope with low-quality participants, such as trap questions and post hoc exclusion, the effectiveness of these methods is often questioned (Chandler, Mueller, & Paolacci, 2014; Downs, Holbrook, & Peel, 2012). This is especially true because increased manipulation of the data for the sake of improving its quality may inadvertently inflate researchers’ degrees of freedom (Simmons, Nelson, & Simonsohn, 2011). The primary reason for low-quality data is the monetary compensation, which gives participants an incentive to cheat, either by participating in a study that they are unqualified for, or by attempting, to answer the survey as fast as possible without paying attention to the questions (Gadiraju, Kawase, Dietze, & Demartini, 2015; Galais & Anduiza, 2014; Hinderer & Nielsen, 2003; Lane et al., 2015). In addition, the use of monetary compensation can cause other issues for researchers. For example, the payment mechanisms for both advertisements as well as for pay-for-participation platforms often require a significant investment of time in order to deal with related bureaucracy. Such bureaucracy can even be downright prohibitive to researchers in some cases. MTurk, for example, currently requires a U.S. billing address for those who wish to recruit on it, which can be problematic for scientists outside the United States (“Amazon Machanical Turk FAQ,” 2016).

Overall, there is no single online source that serves as an end-all solution to participant recruitment, particularly as the demands from such a source would vary significantly across studies. In addition, even if such a source existed, methodological issues can often arise around repeated overuse of the sample (Berinsky et al., 2012; Chandler et al., 2014; Paolacci & Chandler, 2014; Samuels & Zucco, 2013). On MTurk, for example, a minority of highly active workers perform the majority of tasks, and experienced workers often become familiar with classic paradigms and tasks used in the behavioral sciences (Chandler et al., 2014; Fort, Adda, & Cohen, 2011; Paolacci & Chandler, 2014). Therefore, it is in the interest of the research community to examine new directions and expand the current repertoire of recruitment sources.

As such, this article proposes that scientists can use Reddit, a popular social media website, as an effective and versatile source for recruiting participants. In comparison with other recruitment methods, the use of Reddit offers significant benefits, as its structure allows for free and rapid recruitment of large samples, and for the targeting of specific groups when necessary. The fact that participants can be recruited at Reddit for no cost is an especially important advantage. The first and most obvious benefit of this is that it can help researchers conserve funding and save money. In addition, even minor payments often require a significant investment of time in order to deal with bureaucracy, which is not necessary in the case of Reddit. Finally, free recruitment helps students and early career researchers conduct research that they otherwise would not be able to, because for some of them, even low-recruitment costs can be entirely prohibitive. The importance of increasing the availability of these kinds of opportunities cannot be overstated, and experience has shown time and time again that these individuals, who comprise the future members of the research community, strongly benefit from added research opportunities (Agarwala & Teitelbaum, 2010; Kinkead, 2003; Lopatto, 2004; Namageyo-funa et al., 2014).

In order to expand on this new source and increase the feasibility of using Reddit for recruitment, the current article provides an overview of Reddit’s structure, analyzes recruitment and participation data from previous studies which used Reddit for recruitment, highlights the advantages and limitations of using it for research, and lists guidelines for researchers interested in using Reddit as an efficient research tool.

Overview of Reddit

Reddit is a social media website where users share and discuss various stories and links, while voting on submitted content. As such, users can choose to give either a positive vote, which will increase a post’s visibility, or a negative vote, which will decrease a post’s visibility. This provides a form of a community-based moderation, where content with a lot of positive votes is significantly more visible than content which receives a lot of negative votes. The website itself is composed of an interlinked community of subforums, called “subreddits,” which cover an expansive array of topics. The use of subreddits has important implications for researchers, because it enables them to target special interest groups and specific demographics when necessary. Table 1 lists a few of Reddit’s 10,000+ active subreddits, in order to illustrate Reddit’s scope and diversity.

Table 1.

A Selection of Active Subreddits, Together With Their Subscriber Counts.

Subreddit Name	Subscriber Count
ADHD	40,393
Depression	117,598
Fitness	4,290,970
Language Learning	58,229
Music	7,999,833
Philosophy	3,950,997
Sex^a	599,227
Stop Smoking	38,009
Two X Chromosomes^b	3,659,230

Note. Subscriber count is based on each subreddits’ official statistics as of September 15, 2015.

^aFor discussions of all facets of sexuality and sexual relationships. ^bA subreddit for topics that are relevant to the experience of “girlhood” (“TwoXChromosomes FAQ,” 2015).

As far as usage, during August 2015, Reddit had over 200 million unique visitors from 208 countries, who together viewed nearly 8 billion pages. On a regular day of activity, Reddit had more than 3.5 million logged in users who cast nearly 27 million votes (Reddit, 2015a). Analyses of Reddit’s user demographics show that approximately 90% of users are under the age of 35, with a mean age of 24.7 years (Bogers & Wernersen, 2014; Duggan & Smith, 2013; Reddit, 2011). In terms of gender distribution, there is some disagreement between older studies, which found a large majority of male users, and newer studies, which generally found either a smaller difference or a near equal representation of both genders (Bogers & Wernersen, 2014; Dou et al., 2015; Duggan & Smith, 2013; Reddit, 2011, 2015b). A study on the demographics of adult U.S. Reddit users, who comprise approximately half of the active users on the site, showed that these users are relatively representative of the general adult U.S. population, particularly when controlling for age (Barthel, Stocking, Holcomb, & Mitchell, 2016; Reddit, 2015b; Taylor & Keeter, 2010). There are, however, some key differences. Based on the comprehensive study by Barthel et al. (2016), adult U.S. Reddit users are more likely to be politically liberal than the general U.S. population (41% liberal compared to 25%) and less likely to be conservative (21% compared to 35%), although they are as likely to be politically moderate (38% and 39%). In addition, the same study shows that 29% of users have an annual income of under US$30,000 (compared with 21% in the general population), while 34% are at the US$30,000–US$74,999 range (compared with 35%) and 35% are at the above US$75,000 range (compared with 44%). Furthermore, the study shows that these users are more educated than the general population, with 36% holding a college degree (compared with 28%), 42% having some college education (compared with 31%), and 21% having only high school education or less (compared with 41%). Racially, 63% are White non-Hispanics (compared with 65%), 10% are Black non-Hispanics (compared with 12%), 14% are Hispanics (compared with 15%), and 13% are other types of non-Hispanics (compared with 8%; all data from Barthel et al., 2016).

Although user demographics differ to varying degrees between the different subreddits, there is unfortunately no database which aggregates demographic information for each subreddit individually. However, in some cases, a subreddit’s community may conduct an independent user survey, which could contain valuable demographic information, as well as questions that are relevant to its subject matter. One fitness subreddit, for example, conducted a survey with over 7,000 responses. The questions covered basic aspects such as age, gender, and ethnicity and also more subject-oriented questions on exercise, weight loss, self-image, and eating habits (“Fitness,” 2014). Similarly, a survey in a language learning subreddit included questions on language proficiency, learning strategies, and study habits, in addition to basic demographic questions (“Language Learning,” 2015).

Previous Studies Using Reddit

There are currently very few published articles where participants were recruited through Reddit, making it difficult to assess its efficacy as a source for recruitment. However, the author of the current article used Reddit to recruit participants for a study on the effects of goal-oriented instructions in learning tasks (Shatz, 2015). Recruitment took place through a posting on a subreddit named “Language Learning”, which called for participation in a short survey. No monetary incentive was offered. The truncated mean duration for answering the survey was 7 min, while the mean duration was 14 min; this includes cases where the survey was left open for several hours. The survey opened with a qualifying question, in order to ensure that participants did not have proficiency in the target foreign language; this requirement was also stated in the posting’s title. Those who “failed” it were logged out of the survey and prevented from retaking it by the software. This was followed by basic background questions and a timed foreign-vocabulary memorization task. The survey ended with another questionnaire, which included Likert-type scale items and decision-making questions.

In total, 962 participants started the survey. The majority of participants (848, 88.1%) started the survey during the initial posting day. This number decreased to 91 (9.5%) on the second day and to 23 (2.4%) on the third day, after which no new participants started the survey. This response pattern is consistent with a previous study on Reddit’s usage, which showed that the majority of users’ interaction with new postings occurs within approximately 24 hours of the initial posting (Moyer, Carson, & Carson, 2015).

In total, 669 participants (69.5%) completed the survey. This represents a relatively high attrition rate, which could be indicative of a self-selection bias among participants (Hooghe, Stolle, Maheo, & Vissers, 2010; Umbach, 2004). However, nearly all attrition occurred before participants started the experimental task. Of the 962 participants who started the survey, 926 (96.3%) passed the qualifying question. Of those who qualified, 844 (91.1%) answered the initial background questions; and of these, 676 (80.1%) started the task. Out of those who started the task, 654 (96.7%) completed it, and of those who completed it, 634 (96.9%) also completed the final questionnaire. This indicates that the vast majority of attrition occurs because participants lose interest in the beginning of the survey and not because they have difficulty coping with the task. This is in line with the general browsing patterns of Reddit users, a majority of whom tend to exit outgoing links from Reddit after a short amount of time (Moth, 2014; Russel, 2015). Thus, while there is attrition among participants, which is problematic from a methodological perspective, it does not seem to be a greater issue here than it is for other source of recruitment (Hooghe et al., 2010; Umbach, 2004).

Of the 634 participants who completed the entire survey, 526 (83.0%) were used in the final sample, meaning that 108 (17.0%) were excluded for the following reasons:

Seventy nine (73.1%) of the exclusions were due to technical issues. This consisted primarily of 66 cases where the time limit wasn’t imposed in the learning task. There were also seven cases where it wasn’t possible to confirm whether the time limit was imposed and, six cases where there were issues with the score calculations.

Twenty five (23.1%) of the exclusions were due to participants indicating that they are not currently learning a foreign language or did not start learning one in the past 5 years. This additional qualifying question appeared in the final questionnaire only, as the data from the survey were used for two separate studies.

Four (3.7%) of the exclusions were due to the participants providing incomplete data in the questionnaire.

Overall, only a small portion (0.6%) of the initial sample was excluded due to incomplete data, which indicates that this was not a significant issue in the present study. On the other hand, the high number of technical issues is problematic, and it is unknown what was the cause of this. However, it appears that this issue would be less crucial in surveys that do not impose a strict time limit.

The present sample is markedly different than the ones used in previous studies on the topic of goal orientation, based on a comparison with the three studies cited in the original article by Shatz (2015). These studies were authored by Bell and Kozlowski (2002); Steele-Johnson, Beauregard, Hoover, and Schmidt (2000); and VandeWalle, Cron, and Slocum (2001). First of all, the three studies had an average sample of N = 142 (range: 102–199), which is significantly smaller than the Reddit sample. Second, the present study had 68.4% male to 31.6% female gender distribution, while the two studies which reported participants’ gender distribution had a majority of female participants: 58.0% in Kozlowski et al.’s (2001) study and 60.3% in Steele-Johnson et al.’s (2000) study. This represents a slightly more gender-balanced distribution than the Reddit sample. However, as there was a significant representation of both genders in the Reddit sample, this is unlikely to have impacted the findings. In addition, all the three previous studies used samples consisting solely of undergraduate college students from U.S. universities. Although this is a commonly used convenience sample, the reliance on college students for experiments can lead to systematic biases in findings, an issue known as the “narrow data base problem” (Hooghe et al., 2010; Kam, Wilking, & Zechmeister, 2007; Lane et al., 2015; Sears, 1986). The Reddit sample, by comparison, was relatively more diverse and thus circumvented this problem to a degree. This diversity can be estimated via participants’ ages. In the Reddit samples, participants’ mean age was 22.75 years (Standard deviation = 6.34, Min = 12, Max = 53). On the other hand, participants’ age was listed in only one of the previous studies, which stated that 88.8% of participants were between 18 and 21 (Bell & Kozlowski, 2002). This suggests that the Reddit sample was more diverse and did not consist only of college students, although a more detailed study of participants’ background is required. It is likely that such a study will also find points of homogeneity among Reddit users. As such, the present comparison is not intended to suggest that the Reddit sample is perfectly representative of the population. Rather, the goal is to show that the Reddit sample is as viable as the samples recruited by more traditional means.

It is difficult to contrast the results of the Reddit study with those of previous studies, as there is generally significant variation in the findings of studies on the topic of goal orientation. This is evident, for example, in a review which examined the relationship between outcome orientation and task performance and found that different studies reported either a negative, a nonsignificant, or a positive relationship between the two (VandeWalle, Cron, & Slocum, 2001). However, the results of the present study fit within a modern theoretical framework, and there were no findings which were suspect. There are also some supporting statistics for the validity of the data. First, there were no outliers in terms of performance score; this was defined as scores more than 1.5 interquartile ranges above the third quarter or below the first quarter (Navidi, 2008). Second, Cronbach’s alpha was found to be similar to that of traditional-sample alphas. The scale used for this comparison was a set of questions measuring participants’ language risk taking developed by Ely (1986). It was chosen as it was the only scale in the questionnaire which consisted of more than one or two questions, and because it was the only scale which was also used in other studies. In the present study, alpha was .733, while the original article reports an initial alpha of .65 and an alpha of .74 in a follow-up study, both of which used a sample of college students recruited in person (Ely, 1986). The similarity in alphas provides a positive indication as to the reliability of data collected on Reddit. Overall, the most notable limitation of the Reddit study was the short length and subsequent limited complexity of the survey, which took only a few minutes to complete. In comparison, the other studies on the subject sometimes included multiple sessions, spread out over a long period of time. While this limitation is not inherently problematic, it does mean that for studies that require a more extended interaction, Reddit may not be a viable source for recruitment.

Although this illustrates a case where participants were recruited for a study involving an experimental task, Reddit can also been used for other types of studies. For example, Workman’s (2014) article on the formation of safe spaces in gendered online communities, relied primarily on a qualitative, questionnaire-based survey of Reddit users, consisting of closed and open-ended questions. Here too, no monetary incentive was offered to participants. The survey was posted twice, 5 days apart, to a subreddit called “TwoXChromosomes,” which is intended for discussions of the experiences of “girlhood” (“TwoXChromosomes FAQ,” 2015). Responses slowed significantly after 7 days since the initial posting, and stopped entirely after 9. During this period, 1,185 participants started the survey, and 498 (42.0%) of them completed it. Unfortunately, participants’ demographics and participation patterns were not made available in this case.

Although these studies shed some light into recruitment on Reddit, it is important to remember that different studies may achieve different results when attempting to recruit on Reddit. Unfortunately, aside from these two studies, I was not able to find other published article which recruited participants directly on Reddit and which discuss participants’ demographics and response patterns. As more studies will appear, it will be possible to provide a more in-depth analysis of recruitment on Reddit.

Advantages and Limitations

As shown so far, using Reddit to recruit participants is advantageous for several reasons. First, studies can potentially recruit large samples in a short amount of time. Second, by targeting particular subreddits, it is possible to recruit participants from specific demographics and special interest groups when necessary. Finally, studies can generally be conducted for free, as users are often willing to participate in studies even when no monetary incentive is offered. This is especially important for students and early career researchers, for whom even low costs can be prohibitive.

However, there are also limitations to recruitment on Reddit. First, there is always the possibility of low recruitment or retention rates, although the risk of this happening can be mitigated by adhering to the guidelines listed in the following section. Second, the recruitment process relies on users’ self-selection based on the posted selection criteria. This could potentially be an issue, as it opens the possibility that unqualified people might cheat to participate. Such behavior is impossible to prevent entirely and is indeed a frequent problem in online surveys. However, this issue is mitigated in the case of Reddit, as the lack of monetary compensation removes the incentive to cheat to a large degree (Gadiraju et al., 2015; Galais & Anduiza, 2014; Hinderer & Nielsen, 2003). More importantly, using Reddit for recruitment is not a catch all solution, as it may be inappropriate for certain types of studies, and particularly those that involve long and complex procedures. Rather, it is intended to serve as a supplementary source together with existing solutions, in order to allow researchers to overcome some of the limitations of current sources, and avoid the issues associated with sample overuse. In addition, when using Reddit to recruit participants, any issues and limitations associated with online recruitment in general are also relevant. These include ethical considerations, required technical expertise, and various errors, in terms of coverage, sampling, measurement, and nonresponse, some of which have been discussed directly in the article (Birnbaum, 2004; Umbach, 2004). However, these issues can be dealt with through proper methodology and experimental design, and do not prevent Reddit in particular and online sources overall from being valid for research purposes (Birnbaum, 2004; Couper, 2008; Evans & Mathur, 2005; Eysenbach, 2004; Gosling et al., 2000; Miller & Dickson, 2001; Pedersen & Nielsen, 2016; Wilson & Dewaele, 2010).

Guidelines for Effective Recruitment

By following certain guidelines, it is possible to improve the odds of success when using Reddit to recruit participants.

Find an appropriate subreddit for your study. This is akin to finding the proper journal to submit your article to, and as such, fit and reach are two factors which must be taken into consideration. In terms of fit, you should post your survey to a subreddit where users are likely to be interested in participation because the general topic of the study is relevant to that subreddit’s subscribers. For example, if the study is related to a new vocabulary teaching technique, it is better to post it in a subreddit devoted to language learning than in a subreddit dedicated to hockey. This is an important consideration because choosing the topically appropriate subreddit can significantly affect the success of your posting (Lakkaraju, Mcauley, & Leskovec, 2013; Levell, 2015; Sherice, 2011). Fortunately, searching for the appropriate subreddit is a simple process, and can be done either directly through Reddit’s search function or via an external search engine (e.g., Google). However, it should be noted that the importance of fit could also be an issue, as it means that the majority of respondents will likely be those that are very interested in the topic at hand and are thus less representative of the overall population. This should be taken into consideration before posting and during the data analysis stage, as it might cause a selection bias. In terms of reach, on the other hand, the more subscribers a subreddit has the better, as more users will view your posting, leading to increased recruitment rates (Baxter, 2016; Gilbert, 2013). Both the subscriber count as well as the number of active users currently browsing the subreddit are conveniently available in each subreddit’s sidebar.

In some cases, posting to a single subreddit may not generate sufficient responses. When necessary, it is possible to post your survey on more than one subreddit, although it is important to analyze the data in order to determine whether this influenced the results. Singer, Flock, Meinhart, Zeitfogel, and Strohmaier (2014), for example, posted a survey to two different subreddits in their longitudinal study of Reddit’s evolution. They found no significant difference in answer patterns between the two subreddits, leading them to report the results in aggregate.

When posting a call for participants, the title of your posting should clearly highlight the nature of the survey, while using language appropriate for the target subreddit’s community (Gilbert, 2013; Lakkaraju et al., 2013; Patterson, 2015). There is no set length requirement; be concise while ensuring that readers can understand what the posting is about (Lakkaraju et al., 2013). If you are unsure how to phrase your title, examine previous posts in that subreddit in order to get an idea of the preferable style.

Increase exposure to your posting by optimizing the time of the day in which you post as well as the day of the week (Gilbert, 2013; Lakkaraju et al., 2013; Moyer et al., 2015). Generally, the recommended posting time is around 9 AM Eastern Standard Time, although this can vary by subreddit; traffic data for specific subreddits are available online through a number of free and simple sources (Bard, 2016; Olson, 2015; Reddit, 2015c). When it comes to day of the week, it’s best to determine the average amount of time that a post spends on a subreddit’s front page by looking at the age of posts there and then consider this information together with traffic data for the subreddit. For example, Table 2 contains user traffic data for “AskReddit,” a popular subreddit with over 9.5 million subscribers (“Top Subreddits,” 2015). In order to maximize exposure to unique users, the optimal posting time for a posting which is likely to stay on the subreddit’s front page for a single day is Tuesday, while Monday is preferable if a posting is likely to remain on the front page for 2 days or more. More important, however, is the fact that in either case, a weekend posting is likely to lead to suboptimal results.

Note that each subreddit’s demographics may also vary throughout the day. The proportion of North American users, for example, is likely to be higher during certain hours. However, as posts are likely to receive exposure for more than 24 hr, the effects of this on participants’ demographics are likely to be relatively minor.

Keep surveys as short and as concise as possible. The longer the survey, the higher the participants’ attrition rate, particularly if no incentive is offered (Freeman, 2002; Kosinski et al., 2015; Marcus, Bosnjak, Lindner, Pilischenko, & Schutz, 2007; Umbach, 2004). This advice is widely accepted in relation to the use of web surveys in general, and the recommended maximal length for surveys ranges between no more than 10–20 min (Marcus et al., 2007; Umbach, 2004). Furthermore, when the survey can be considered short (generally meaning that it takes 10 min or less to complete), then it can be beneficial to explicitly state that the survey is short in the posting as shown by Crawford, Couper, and Lamias (2001). However, Crawford et al. (2001) also show that it is important not to misrepresent the survey’s completion time, in order to avoid a high rate of attrition among participants, in addition to the ethical issues involved. This is particularly important in the case of recruitment on Reddit in order to avoid community backlash (Levell, 2015).

After posting the link to the survey, continue to interact with the community and try to answer any questions that they might have in the comments. Being open to discussion gives you more credibility as a researcher, while increasing the number of comments associated with your posting and attracting users’ attention. This, in turn, can increase participation rates in the study (Bulygin, 2013; Levell, 2015; Naik, Sachin, & Kuri, 2015; Patterson, 2015).

Table 2.

Traffic Data for the “AskReddit” Subreddit by Day of the Week.

Day	Unique Users	Page Views
Monday	1,397,689	5,511,413
Tuesday	1,484,332	5,656,443
Wednesday	1,392,436	5,539,740
Thursday	1,392,141	5,355,108
Friday	1,227,035	4,414,647
Saturday	1,016,675	3,880,263
Sunday	1,024,819	4,127,675

Note. Adapted from Reddit (2015b).

Conclusions

Researchers can use Reddit to effectively recruit participants for scientific studies. While Reddit does have some limitations and may not be suitable for all studies, the site is nonetheless a valuable source for recruitment because it allows for free and rapid data collection from large samples, while enabling researchers to target specific populations when necessary. When using Reddit for recruitment, researchers should keep in mind the abovementioned guidelines in order to optimize their posting, thereby improving recruitment and retention rates among participants.

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

References

Agarwala

Teitelbaum

(2010). Trends in funding for dissertation field research: Why do political science and sociology students win so few awards? PS: Political Science & Politics, 43, 283. Retrieved from http://doi.org/10.1017/S1049096510000156

Amazon Machanical Turk FAQ. (2016). Retrieved April 19, 2016, from https://requester.mturk.com/help/faq (Archived at: http://www.webcitation.org/6gsxHQrw7)

Bard

(2016). RedditLater. Retrieved April 20, 2016, from https://www.redditlater.com/ (Archived at: http://www.webcitation.org/6guVj9aE1)

Barthel

B. Y. M.

Stocking

Holcomb

Mitchell

(2016). Nearly eight-in-ten Reddit users get news on the site. Washington, DC: Pew Research Center.

Baxter

(2016). The Web marketer’s guide to Reddit. Retrieved March 1, 2016, from https://builtvisible.com/web-marketers-guide-to-reddit/ (Archived at: http://www.webcitation.org/6fgg1iiVd)

Bell

B. S.

Kozlowski

W. J.

(2002). Goal orientation and ability: Interactive effects on self-efficacy, performance, and knowledge. Journal of Applied Psychology, 87, 497–505. Retrieved from http://doi.org/10.1037//0021-9010.87.3.497

Berinsky

A. J.

Huber

G. a.

Lenz

G. S.

(2012). Evaluating online labor markets for experimental research: Amazon.com’s mechanical turk. Political Analysis, 20, 351–368. Retrieved from http://doi.org/10.1093/pan/mpr057

Birnbaum

M. H.

(2004). Human research and data collection via the Internet. Annual Review of Psychology, 55, 803–832. Retrieved from http://doi.org/10.1146/annurev.psych.55.090902.141601

Bogers

Wernersen

(2014). How ‘social’ are social news sites? Exploring the motivations for using Reddit.com. In iConference 2014 Proceedings (pp. 329–344). Retrieved from http://doi.org/10.9776/14108

10.

Buhrmester

Kwang

Gosling

S. D.

(2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5. Retrieved from http://doi.org/10.1177/1745691610393980

11.

Bulygin

(2013). How to get massive traffic from reddit to your website. Retrieved February 29, 2016, from http://trendblog.net/how-to-get-traffic-from-reddit/ (Archived at: http://www.webcitation.org/6ffGUHmRf)

12.

Casler

Bickel

Hackett

(2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29, 2156–2160. Retrieved from http://doi.org/10.1016/j.chb.2013.05.009

13.

Chandler

Mueller

Paolacci

(2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–30. Retrieved from http://doi.org/10.3758/s13428-013-0365-7

14.

Couper

M. P.

(2008). Designing effective web surveys. New York, NY: Cambridge University Press.

15.

Crawford

S. D.

Couper

M. P.

Lamias

M. J.

(2001). Web surveys: Perceptions of burden. Social Science Computer Review, 19, 146–162.

16.

Dou

Cho

ElTayeby

Choo

Wang

Ribarsky

(2015). DemographicVis: Analyzing demographic information based on user generated content. In 2015 IEEE Conference on Visual Analytics Science and Technology (VAST) (pp. 57–64). IEEE. Retrieved from http://doi.org/10.1109/VAST.2015.7347631

17.

Downs

J. S.

Holbrook

Peel

(2012). Screening participants on mechanical turk: Techniques and justifications. Paper presented at the Annual Conference of the Association for Consumer Research, Vancouver, Canada.

18.

Duggan

Smith

(2013). 6% of online adults are reddit users. Pew Internet & American Life Project Washington, DC.

19.

Ely

C. M.

(1986). An analysis of discomfort, risktaking, sociability, and motivation in the L2 classroom. Language Learning, 36, 1–25.

20.

Evans

J. R.

Mathur

(2005). The value of online surveys. Internet Research, 15, 195–219. Retrieved from http://doi.org/10.1108/10662240510590360

21.

Eysenbach

(2004). Improving the quality of web surveys: The checklist for reporting results of Internet e-surveys (CHERRIES). Journal of Medical Internet Research, 6, e34. Retrieved from http://doi.org/10.2196/jmir.6.3.e34

22.

“Fitness”. (2014). /r/fitness survey results. Retrieved April 17, 2016, from https://docs.google.com/forms/d/1MU9xE6nv71UnYFgUVcrHYh-nTkaLVYRXMs4RAcOFgpM/viewanalytics#start=publishanalytics (Archived at: http://www.webcitation.org/6gow2h8xA)

23.

Fort

Adda

Cohen

K. B.

(2011). Amazon mechanical turk: Gold mine or coal mine? Computational Linguistics, 37, 413–420. Retrieved from http://doi.org/10.1162/COLI_a_00057

24.

Freeman

(2002). Effective recruitment and retention for online surveys. Proceedings. IEEE International Professional Communication Conference, 509–512. Retrieved from http://doi.org/10.1109/IPCC.2002.1049135

25.

Gadiraju

Kawase

Dietze

Demartini

(2015). Understanding malicious behavior in crowdsourcing platforms. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems—CHI ‘15 (pp. 1631–1640). Retrieved from http://doi.org/10.1145/2702123.2702443

26.

Galais

Anduiza

(2014). “You cheated on me!” Causes and consequences of cheating in online surveys. Paper presented at Visions in Methodology (VIM), McMaster University, Hamilton, Canada.

27.

Gilbert

(2013). Widespread underprovision on Reddit. In Proceedings of the 2013 conference on computer supported cooperative work (pp. 803–808). ACM. Retrieved from http://doi.org/10.1145/2441776.2441866

28.

Gosling

S. D.

Vazire

Srivastava

John

O. P.

(2000). Should we trust web-based studies? A comparative analysis of six preconceptions about Internet questionnaires. The American Psychologist, 59, 93–104. Retrieved from http://doi.org/10.1037/0003-066X.59.2.93

29.

Hinderer

Nielsen

(2003). 234 tips and tricks for recruiting users as participants in usability studies. Fremont, CA: Nielsen Norman Group.

30.

Hooghe

Stolle

Maheo

V. -A.

Vissers

(2010). Why can’t a student be more like an average person?: Sampling and attrition effects in social science field and laboratory experiments. The Annals of the American Academy of Political and Social Science, 628, 85–96. Retrieved from http://doi.org/10.1177/0002716209351516

31.

Johnson

K. J.

Mueller

N. L.

Williams

Gutmann

D. H.

(2014). Evaluation of participant recruitment methods to a rare disease online registry. American Journal of Medical Genetics, Part A, 164, 1686–1694. Retrieved from http://doi.org/10.1002/ajmg.a.36530

32.

Kam

C. D.

Wilking

J. R.

Zechmeister

E. J.

(2007). Beyond the “narrow data base”: Another convenience sample for experimental research. Political Behavior, 29, 415–440. Retrieved from http://doi.org/10.1007/s11109-007-9037-6

33.

Kinkead

(2003). Learning through inquiry: An overview of undergraduate research. New Directions for Teaching and Learning, 2003, 5–18. Retrieved from http://doi.org/10.1002/tl.85

34.

Kosinski

Matz

S. C.

Gosling

S. D.

Popov

Stillwell

(2015). Facebook as a research tool for the social sciences. American Psychologist, 70, 543–556. Retrieved from http://doi.org/10.1037/a0039210

35.

Kozlowski

S. W. J.

Gully

S. M.

Brown

K. G.

Salas

Smith

E. M.

Nason

E. R.

(2001). Effects of training goals and goal orientation traits on multidimensional training outcomes and performance adaptability. Organizational Behavior and Human Decision Processes, 85, 1–31. Retrieved from http://doi.org/10.1006/obhd.2000.2930

36.

Lakkaraju

Mcauley

Leskovec

(2013). What’s in a name? Understanding the interplay between titles, content, and communities in social media. In Proceedings of the Seventh International AAAI Conference on Weblogs and Social Media (pp. 311–320). Cambridge, MA: AAAI Press.

37.

Lane

T. S.

Armin

Gordon

J. S.

(2015). Online recruitment methods for web-based and mobile health studies: A review of the literature. Journal of Medical Internet Research, 17, e183. Retrieved from http://doi.org/10.2196/jmir.4359

38.

“Language Learning.” (2015). /r/languagelearning survey results. Retrieved April 17, 2016, from https://docs.google.com/forms/d/1JTQXerGlVdPsC9vb0LoV9pWpfjZbxh-LTwDRZIPB6Cg/viewanalytics (Archived at: http://www.webcitation.org/6gow1w7qg)

39.

Levell

(2015). Cracking Reddit: How to get thousands of visitors to your campaign with reddit marketing. Retrieved February 29, 2016, from http://startupbros.com/cracking-reddit-how-to-get-thousands-of-visitors-to-your-campaign-with-reddit-marketing/ (Archived at:http://www.webcitation.org/6fe226nCU)

40.

Lopatto

(2004). Survey of undergraduate research experiences (SURE): First findings. Cell Biology Education, 3, 270–277. Retrieved from http://doi.org/10.1187/cbe.04-07-0045

41.

Marcus

Bosnjak

Lindner

Pilischenko

Schutz

(2007). Compensating for low topic interest and long surveys: A field experiment on nonresponse in web surveys. Social Science Computer Review, 25, 372–383. Retrieved from http://doi.org/10.1177/0894439307297606

42.

Miller

T. W.

Dickson

P. R.

(2001). International journal of on-line market research. International Journal of Electronic Commerce, 5, 139–167. Retrieved from http://doi.org/10.1080/10864415.2001.11044213

43.

Morgan

A. J.

Jorm

A. F.

Mackinnon

A. J.

(2013). Internet-based recruitment to a depression prevention intervention: Lessons from the mood memos study. Journal of Medical Internet Research, 15, e31. Retrieved from http://doi.org/10.2196/jmir.2262

44.

Moth

(2014). Is referral traffic from Reddit of any value to publishers? Retrieved April 18, 2016, from https://econsultancy.com/blog/64725-is-referral-traffic-from-reddit-of-any-value-to-publishers/ (Archived at: http://www.webcitation.org/6grHBof9b)

45.

Moyer

Carson

S. L.

Carson

R. T.

(2015). Determining the influence of Reddit posts on wikipedia pageviews. In Ninth International AAAI Conference on Web and Social Media (pp. 75–82). Oxford, UK: AAAI Press.

46.

Naik

Sachin

A. S.

Kuri

(2015). Classification of posts on Reddit (CSE 255). San Diego: University of California.

47.

Namageyo-funa

Rimando

Brace

A. M.

Christiana

R. W.

Fowles

T. L.

Davis

T. L.

… Sealy

D.-A.

(2014). Recruitment in qualitative public health research : Lessons learned during dissertation sample recruitment. The Qualitative Report, 19, 1–17.

48.

Navidi

W. C.

(2008). Statistics for engineers and scientists. Boston, MA: McGraw-Hill Higher Education.

49.

Olson

(2015). A data-driven guide to creating successful reddit posts, redux. Retrieved July 19, 2015, from http://www.randalolson.com/2015/01/11/a-data-driven-guide-to-creating-successful-reddit-posts-redux/

50.

Paolacci

Chandler

(2014). Inside the turk: Understanding mechanical turk as a participant pool. Current Directions in Psychological Science, 23, 184–188. Retrieved from http://doi.org/10.1177/0963721414531598

51.

Paolacci

Chandler

Ipeirotis

(2010). Running experiments on amazon mechanical turk. Judgment and Decision Making, 5, 411–419. Retrieved from http://doi.org/10.2139/ssrn.1626226

52.

Patterson

(2015). So you want to “market on Reddit.” Retrieved February 29, 2016, from http://marketingland.com/want-market-reddit-144396 (Archived at: http://www.webcitation.org/6fe2TjMJt)

53.

Pedersen

M. J.

Nielsen

C. V.

(2016). Improving survey response rates in online panels: Effects of low-cost incentives and cost-free text appeal interventions. Social Science Computer Review, 34, 229–243. Retrieved from http://doi.org/10.1177/0894439314563916

54.

Reddit. (2011). Who in the world is reddit? Results are in …. Retrieved July 17, 2015, from http://www.redditblog.com/2011/09/who-in-world-is-reddit-results-are-in.html

55.

Reddit. (2015a). About Reddit. Retrieved September 17, 2015, from https://www.reddit.com/about (Archived at: http://www.webcitation.org/6bbj70z5g)

56.

Reddit. (2015b). Audience and demographics. Retrieved February 29, 2016, from https://reddit.zendesk.com/hc/en-us/articles/205183225-Audience-and-Demographics (Archived at: http://www.webcitation.org/6ffMscqLT)

57.

Reddit. (2015c). Traffic statistics for /r/AskReddit. Retrieved July 19, 2015, from https://www.reddit.com/r/AskReddit/about/traffic

58.

Russel

(2015). Getting on the front page of Reddit. What does it mean for your organization? Retrieved April 18, 2016, from http://attentiv.com/front-page-of-reddit/ (Archived at: http://www.webcitation.org/6grHBPLtJ)

59.

Samuels

Zucco

(2013). Using Facebook as a subject recruitment tool for survey-experimental research. SSRN 2101458. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2101458

60.

Sears

D. O.

(1986). College Sophomores in the laboratory: Influences of a narrow data base on social psychology’ s view of human nature. Journal of Personality and Social Psychology, 51, 515–530. Retrieved from http://doi.org/10.1037//0022-3514.51.3.515

61.

Shatz

(2015). The negative impact of goal-oriented instructions. Educational Studies, 41(5), 476–480. doi:10.1080/03055698.2015.1043982

62.

Sherice

(2011). The marketer’s guide to Reddit. Retrieved February 29, 2016, from https://blog.kissmetrics.com/reddit-marketing-guide/ (Archived at: http://www.webcitation.org/6fe1PB7Om)

63.

Simmons

J. P.

Nelson

L. D.

Simonsohn

(2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. Retrieved from http://doi.org/10.1177/0956797611417632

64.

Singer

Flock

Meinhart

Zeitfogel

Strohmaier

(2014). Evolution of Reddit: From the front page of the Internet to a self-referential community? In Proceedings of the companion publication of the 23rd international conference on world wide web companion (pp. 1–6). International World Wide Web Conferences Steering Committee. Retrieved from http://doi.org/10.1145/2567948.2576943

65.

Steele-Johnson

Beauregard

R. S.

Hoover

P. B.

Schmidt

A. M.

(2000). Goal orientation and task demand effects on motivation, affect, and performance. Journal of Applied Psychology, 85, 724–738. Retrieved from http://doi.org/10.1037//0021-9010.85.5.724

66.

Taylor

Keeter

(2010). Millennials: A portrait of generation next: Confident, connected, open to change. Washington, DC: Pew Research Center.

67.

Top Subreddits. (2015). Retrieved from http://redditmetrics.com/top (Archived at: http://www.webcitation.org/6bRKUU3u4)

68.

TwoXChromosomes FAQ. (2015). Retrieved September 14, 2015, from https://www.reddit.com/r/TwoXChromosomes/wiki/faq (Archived at: http://www.webcitation.org/6bXW2SsDq)

69.

Umbach

P. D.

(2004). Web surveys: Best practices. In Porter

S. R.

(Ed.), Overcoming survey research problems (pp. 23–38). San Francisco, CA: Jossey-Bass. Retrieved from http://doi.org/10.1002/ir.98

70.

Valdez

R. S.

Guterbock

T. M.

Thompson

M. J.

Reilly

J. D.

Menefee

H. K.

Bennici

M. S.

… Rexrode

D. L.

(2014). Beyond traditional advertisements: Leveraging Facebook’s social structures for research recruitment. Journal of Medical Internet Research, 16, e243. Retrieved from http://doi.org/10.2196/jmir.3786

71.

VandeWalle

Cron

W. L.

Slocum

J. W.

(2001). The role of goal orientation following performance feedback. The Journal of Applied Psychology, 86, 629–640.

72.

Wilson

Dewaele

J. M.

(2010). The use of web questionnaires in second language acquisition and bilingualism research. Second Language Research, 26, 103–123. Retrieved from http://doi.org/10.1177/0267658309337640

73.

Woo

Kim

Couper

M. P.

(2015). Comparing a cell phone survey and web survey of university students. Social Science Computer Review, 33, 399–410. Retrieved from http://doi.org/10.1177/0894439314544876

74.

Workman

(2014). Formation of safe spaces in gendered online communities: Reddit and “the front page of the Internet.” (Master’s thesis). Texas Christian University, TX. Retrieved from https://repository.tcu.edu/bitstream/handle/116099117/4558/Workman_tcu_0229M_10510.pdf?sequence=1