Using Reddit as a Source for Recruiting Participants for In-Depth and Phenomenological Research

Introduction

Adequate participant recruitment is a perpetual challenge within qualitative research methods such as focus groups or in-depth interviews. Eligible participants may not wish to travel to the research site, may not see the value of the research, or may feel that their participation requires too much time and effort (Patel et al., 2003). University-led studies tend to recruit from the sample most available to them—i.e., university students—though these participants represent a small minority of the global population (Henrich et al., 2010b). As a result, a large proportion of research knowledge generated comes from populations that are Western, educated, wealthy, and industrialized (Arnett, 2008). Despite the fact that such characteristics are not present in most people across the world, authors persist in their claims of the generalizability of results using these samples (Henrich et al., 2010a).

In recent years, online recruitment has become an increasingly common alternative to traditional participant recruitment. Online recruitment and data collection have been empirically supported to provide samples and results that are comparably valid to research conducted offline (Birnbaum, 2004; Evans & Mathur, 2005; Wilson & Dewaele, 2010; Woo et al., 2015). Online recruitment allows researchers to easily reach participants (Buhrmester et al., 2011), easily collect data (Shapiro et al., 2013), and gain access to varied population pools (Rouse, 2015). Global Internet usage has continued to grow in recent years (Silver, 2019), which allows for recruitment in larger areas of the world or targeted to geographic areas that are far away from the research teams. In response, scholars have begun investigating and evaluating various Internet platforms for their potential as sources of data-rich participants for research activities (Buhrmester et al., 2011; Lane et al., 2015).

Social networking platforms and sites have therefore become the focus of emerging literature in online recruitment. For example, Facebook has an incredibly large and diverse user base, with nearly 3 billion site visits per month (Dixon, 2022). When compared to traditional mass media recruitment, paid Facebook research advertisements generate more user impressions for less money (Whitaker et al., 2017). Facebook has also been praised for its ability to recruit from hard-to-reach populations (Sikkens et al., 2017; Whitaker et al., 2017). Social networking sites such as Facebook also allow researchers to seek out their participant pool directly, build trust and rapport necessary for engaging in research activities, and to gate-keep their own personal information for both safety and rigor purposes (Sikkens et al., 2017).

Online recruitment and data collection are not without their challenges. For example, studies using one of the most popular online recruitment sources, Amazon Mechanical Turk (MTurk), have repeatedly needed to discard data for being unsuitable or unusable (Chandler et al., 2014). Further, since MTurk users represent a very specific sector of Internet users, it may not be a viable option for recruiting members of specialized populations (Samuels & Zucco, 2013). More popular, broader-reaching platforms such as Facebook or Twitter present their own obstacles. In order to reach large numbers of users, research teams must know how to make connections through potential participants’ social networks or otherwise pay money to engage in targeted advertising (Johnson et al., 2014; Kosinski et al., 2015; Whitaker et al., 2017). Like offline research, online research recruitment generally benefits from promising some sort of compensation for participations; however, such compensation can lead to poor data quality or attempts to please the researchers (Gadiraju et al., 2015; Lane et al., 2015), and efforts to mitigate these circumstances have been criticized for their relative ineffectiveness (Chandler et al., 2014). Depending on the participant populations being sought and the study goals, recruiting from social media sites such as Facebook may not generate representative samples (Whitaker et al., 2017). Researchers are therefore in continual need of inexpensive, reliable, honest sources for online recruitment which offer the greatest eligible participant pool and highest-quality data.

One platform with potential to meet these criteria is Reddit. Reddit is a message board website where users can post text, media, or external links, as well as interact with one another through comments and direct messages. A hallmark feature of Reddit is its voting system, whereby users can give posts (or comments) a positive or negative vote, with the resulting scores determining the level of visibility the content has for other users (Massanari, 2017). The site is divided into a multitude of communities separated by topic area, called “subreddits”, and an individual user’s main landing page is a collection of the highest-voted posts from the various subreddits they are subscribed to. Reddit is an extremely popular website, currently ranked 19^th globally in popularity (2022) with tens of millions of unique users logging in and engaging with content daily (Reddit, 2020). Due to its large user base and influence over broader online culture, Reddit calls itself “the front page of the Internet”. Individual Reddit communities are also highly specialized, with tens of thousands discrete subreddits (Shatz, 2017).

Empirical research has therefore begun to use Reddit as a source for participant recruitment (Jamnik & Lane, 2017; Richard et al., 2021; Shatz, 2017). Reddit-based data are high-quality and have demonstrated equivalent psychometric validity and when compared to prior in-person research (Jamnik & Lane, 2017). As Reddit recruitment or data analysis has become more common, multiple studies have described the benefits and challenges associated with recruitment from Reddit, noting that Reddit’s broad demographics and highly specific subreddits offer the ability to communicate with members of distinct communities or affinities quickly, easily, and in sufficient numbers (Amaya et al., 2021; Proferes et al., 2021; Shatz, 2017). However, these studies primarily describe or employ quantitative, survey-based methods, or content analyses on previously-existing user content generated and collected with minimal participant interactions on the part of the research teams. One recent study described the viability of Reddit as a location to recruit participants for qualitative online focus groups, though these focus groups were created and supported asynchronously using Reddit’s message board features (Richard et al., 2021).

Given the growing popularity in using Reddit to access research participant populations, there is a need to systematize the various ways it can be used for research purposes (Amaya et al., 2021; Proferes et al., 2021). Currently, literature is only beginning to emerge detailing the use of Reddit as a primary source to recruit participants for synchronous, in-depth qualitative interviews conducted off-site. Through the example of a social work research project which drew participants from a specialized Reddit community, this paper describes the processes, challenges, and lessons learned from using Reddit for participant recruitment. In doing so, this discussion seeks to guide and empower future researchers to pursue Reddit as an opportunity to access high-quality data in niche populations.

Methods

The research study was an exploratory phenomenological study into the experiences of current or former intimate partners of men who were involved with The Red Pill (TRP). TRP is a community largely organized on Reddit and which advocates the use of systematic manipulation as a seduction technique (Bratich & Banet-Weiser, 2019; pk_atheist, 2012). TRP is primarily intended for heterosexual men trying to seduce heterosexual women, and its systems advocate a type of hypermasculinity which prioritizes being an ‘alpha’ at all times (Ging, 2019). The name “The Red Pill” comes from a sequence in the film The Matrix where swallowing a red pill allows the protagonist to become aware of the unpleasant truths of the world (The Wachowskis, 1999); TRP philosophy maintains that the ugly truth men need to realize in this world is that feminism gives women a monopoly of power over who is able to engage in romantic and sexual relationships (Ging, 2019). TRP claims that its tactics will specifically empower adherents to circumvent the roadblocks to sexual and romantic success presented by feminism, which it refers to as women’s “sexual strategy” (pk_atheist, 2012).

The two subreddits chosen for recruitment were The Blue Pill and Ex-Red Pill. The Blue Pill is a community where users post topics and discourse from TRP, discussing and satirizing the elements deemed most ridiculous (Reddit, 2013). Ex-Red Pill is a subreddit where users who had been part of, or adjacent to, TRP process their exit from that community and its worldview (Reddit, 2014). These subreddits were selected in accordance with the steps outlined in Shatz (2017), which state that a chosen subreddit must have adequate fit and reach. Both The Blue Pill and Ex-Red Pill were determined to have adequate fit because they require community members to have familiarity with TRP and potentially even have been (or known) participants. Both subreddits were of a size believed to be large enough to find enough participants for a small qualitative study without being so large that the research team would be overwhelmed by responses. Recruitment occurred in two periods, between March-April 2019 and November-December 2020. The initial period recruited solely from The Blue Pill, while the second period added Ex-Red Pill in order to increase user reach and engagement.

The research team began the recruitment process by creating a Reddit username separate from any team members’ personal accounts and only for use during research activities. Using this account, the team posted a thread on The Blue Pill to determine if there was sufficient community interest and eligibility to support recruitment. This was viewed by the research team as a digital form of outreach, similar to methods employed for assessing feasibility of recruiting participants from hard-to-reach populations in-person (Alvarez et al., 2006; Stewart et al., 2020). Enough positive responses were observed in the interest thread to suggest that recruitment could, in fact, be sustained via The Blue Pill. The outreach thread was subsequently deleted and the research team directly messaged the community moderators, explaining the project’s aims and procedures while requesting permission to use their subreddit for recruitment. Once full moderator permission was obtained, the research team again published a thread containing the official recruitment information and protocols.

Users were instructed to directly message the research team’s user account if they believed themselves to be eligible and were interested in participating. Direct message conversations were used to answer questions, assuage users’ concerns around safety or anonymity, arrange schedules for the in-depth interviews, and exchange non-Reddit contact information to be used when conducting the interviews. The research team established a Skype account specific to research activities which allowed for participants’ choice of traditional phone or internet-enabled voice calls. Using this Skype information, team members conducted one-time, in-depth, semi-structured interviews with participants.

Findings

A total of 6 interviews were conducted with eligible participants recruited from The Blue Pill and Ex-Red Pill. Participants all identified themselves as heterosexual women who were ex-partners of Red Pill men and who had become users of the included subreddits as a result of their experiences with their former partners. Four participants were located within the United States, while two were located in Europe. In addition, one interview was scheduled and had begun when the participant identified himself as a heterosexual man from the Dominican Republic who had been friends with, but never in a romantic relationship with, a Red Pill man. The interview was completed but his data were excluded from the final analysis, as they did not meet research goals. However, this experience is elaborated in greater detail below as part of the discussion regarding navigation of website features.

Although an adequate number of participants were recruited for the study, the recruitment experience also yielded a range of opportunities and challenges. This information can be beneficial to future studies attempting to recruit from Reddit, particularly as related to internal community response, the Institutional Review Board approval process, navigating site features, and external responses to recruitment.

Discussion

Previous research has indicated that recruitment from Reddit for academic research is feasible and offers high-quality data so long as certain steps and guidelines are followed (Amaya et al., 2021; Shatz, 2017). However, there have been few studies describing the application of these steps for the recruitment of participants in synchronous, in-depth qualitative research. The case example presented here demonstrates that recruiting individual qualitative participants from specialized Reddit communities is feasible and acceptable. The lessons observed in this process can be applied to making future research recruitment operate smoothly and accessibly.

The experiences of recruiting from Reddit also broadly echo prior research into recruitment from social media and networking sites such as Facebook, with certain key differences. The message-board communication and specialized community nature of Reddit means that researchers do not need to seek out participants or directly initiate communication. Once contact is made, however, the process of using direct messages to build rapport and coordinate research activities is quite similar to recruiting specialized populations from Facebook (Sikkens et al., 2017). Reddit recruitment is a free alternative to recruiting through targeted advertising, although without the outcome metrics available through advertising on sites like Facebook it can be difficult to know if recruitment is having the necessary level of reach or interaction (Shatz, 2017; Whitaker et al., 2017). Much like recruitment from Facebook, it is also important that researchers using Reddit create user accounts which are neutral to potential participants and/or are otherwise unconnected to any personal accounts they may have (Sikkens et al., 2017).

Overall community response in this study indicates the potential for using Reddit for recruiting active qualitative participants in future studies. However, not all subreddit communities can sustain participant recruitment. In his guidelines for Reddit recruitment, Shatz (2017) states that “the more subscribers a subreddit has, the better” (p. 544). For qualitative recruitment, this may not be as true. While a chosen subreddit must have adequate fit and reach, a larger user base may lead to a larger amount of necessary work for the study team to parse replies, screen potential participants, and navigate encounters with hostile outsiders such as experienced during this project. Targeting a more popular or larger subreddit could encourage these events to occur more frequently, as there is a higher likelihood of posts being viewed by outside users who are not subscribed to those subreddits. Reddit’s voting features may expose non-subscribed users to this content on their feeds if it gains popularity, and users also have the ability to share or “cross-post” the content in other communities. In addition, subreddits may not wish to participate in recruitment due to fear of repercussions, desire for anonymity, antagonism toward the idea of being studied, or any other reason. As such, we highly recommend for future studies to familiarize themselves with any existing guidelines a subreddit has regarding research, and to always request permission from moderators/administrators of a chosen subreddit prior to posting any calls for participation.

The Blue Pill and Ex Red Pill communities tolerated recruitment in their space very well. Users did exhibit a natural protectiveness, which was to be expected given the subject area. Men associated with communities related to TRP have been observed to harass, stalk, or even commit violence against women they believe to have wronged them (Bratich & Banet-Weiser, 2019; Ging, 2019). Despite this, the community seemed encouraged by the notion that the academic community would pay attention to their experiences as valuable knowledge, and users appeared to regard this as outweighing any inherent risks associated with participations. Community moderators and administrators were efficient, polite, and supportive of the process, which facilitated the research team’s ability to reach this unique participant pool. Nevertheless, researchers should maintain readiness and willingness to share information such as university credentials with their potential participants, but should do so in ways that protect their own safety and privacy. For example, the researcher may consider offering their IRB reference ID or a general institutional contact point (such as a study-specific email) if they do not feel comfortable offering their own personally identifiable information.

Researchers should practice flexibility within Reddit communities chosen for recruitment, understanding that the way that information (such as eligibility) has been conveyed may be misunderstood, glossed over, or may necessitate additional effort to connect with interested site users. It cannot be presumed that all users will be inherently familiar with the nuances of research procedures or will intuit what a research team is intending to accomplish. This project demonstrates the benefits of continuous dialogue with community participants, as well as clarifications and modifications when necessary. In the case of the male participant whose friend was a Red Piller, the research team had never considered verifying gender or romantic relationship status after users had self-identified their eligibility, and the team had incorrectly presumed that this user had fully understood eligibility requirements before making initial contact. Likewise, the team nearly missed connecting with users expressing interest due to site-wide messaging feature changes. Future research projects should therefore consider collecting baseline information before moving into formal data collection, as well as confirming that they have explored all possible avenues for participants to make contact before moving on from that community.

This research also demonstrates the necessity of ongoing collaborations with university IRBs in order to better navigate participant recruitment from Reddit and similar websites. Online participant recruitment, particularly via social media or platforms such as Reddit, is still a new phenomenon. It is clear that IRBs have been working tirelessly to ensure ethical research practices and the safety of participants while supporting the value of using the Internet to access small or difficult-to-reach populations. However, informed consent language suggesting the threat of external hacking or data theft may undermine participants’ trust in the efforts made by research teams to preserve their confidentiality, thereby damaging rapport. As demonstrated by this research, these types of consent scripts do not necessarily account for the personal histories of study participants nor the trauma which could be triggered by hearing these consequences. Consent language such as required in this study may also reflect poorly on research teams, as neglecting to acknowledge the high improbability of having data stolen and personally identified could lead participants to believe that the research team is not technologically literate or is potentially mishandling data safety protocols. We therefore propose that researchers and IRBs engage in ongoing conversations and partnerships which can satisfy the safety and ethics of human participant research while also preserving the integrity and trustworthiness of the specific research activities.

Despite the clear benefits to using Reddit for recruiting qualitative or in-depth research participants, it is also possible that bad-faith actors from outside of the groups being studied may try to intercede and disrupt the research process. Though this can often occur with in-person participant recruitment, affordances of technology such as anonymity and ease of contact may allow users to feel more empowered to be antagonistic than they would offline (Suler, 2004). During the recruitment process for this project, self-identified Red Pill users repeatedly sent messages to the research team’s Reddit account rebuffing the rigor and validity of the research being carried out. This research team considered these types of messages to be a collateral nuisance and varied in response based on how threatening they perceived a given message to be. Researchers recruiting from Reddit should recognize the likelihood of these types of messages and likewise act accordingly. However, it must be reiterated that members of Reddit and other similar online message board communities have been observed to mobilize harassment and threat campaigns, as well as in rare cases commit deadly acts of violence, against people they perceived as enemies (Bratich & Banet-Weiser, 2019). It is therefore of the utmost importance that future researchers recognize this risk and take appropriate steps to protect themselves and their participants’ personally identifiable data, as well as having communication and safety plans in place with administration should any types of these consequences occur.

Conclusions

Beyond recruiting participants for survey-driven or content analysis research, Reddit offers fertile ground for recruiting niche or specialized populations for in-depth qualitative data collection. Though it can offer researchers free and rapid access to large amounts of unique data, researchers should maintain respect and deference to community organization and etiquette when entering their space. The barriers and facilitators described as part of this research may help guide future researchers to recruit Reddit-based participants safely and ethically while yielding high-quality, trustworthy results.